Self-Supervision Enhanced Feature Selection with Correlated Gates
Authors: Changhee Lee
Fergus Imrie
Mihaela van der Schaar
Venue: ICLR 2022 Spotlight (2022)
Keywords: Feature Selection
Feature Importance
Self-Supervised Learning

Abstract: Discovering relevant input features for predicting a target variable is a key scientific question. However, in many domains, such as medicine and biology, feature selection is confounded by a scarcity of labeled samples coupled with significant correlations among features. In this paper, we propose a novel deep learning approach to feature selection that addresses both challenges simultaneously. First, we pre-train the network using unlabeled samples within a self-supervised learning framework by solving pretext tasks that require the network to learn informative representations from partial feature sets. Then, we fine-tune the pre-trained network to discover relevant features using labeled samples. During both training phases, we explicitly account for the correlation structure of the input features by generating correlated gate vectors from a multivariate Bernoulli distribution. Experiments on multiple real-world datasets including clinical and omics demonstrate that our model discovers relevant features that provide superior prediction performance compared to the state-of-the-art benchmarks in practical scenarios where there is often limited labeled data and high correlations among features.

Published as a conference paper at ICLR 2022
SELF-SUPERVISION ENHANCED FEATURE SELECTION WITH CORRELATED GATES
Changhee Lee"
Fergus Imrie UCLA, USA imrie@uee a edu
Mihaela van der Schaar University of Cambridge, UK UCLA, USA
Chung-Ang University, Korea changhee eeeccuu. ac. kr
Alan Turing Institute, UK mv4728 cam. ac. uk
ABSTRACT
Discovering relevant imput features for predicting target variable is a key scien- tific question. However, in many domains, such as medicine and biology, feature selection is confounded by scarcity of labeled samples coupled with significant correlations among features. this paper, we propose a novel deep learning ap- proach to feature selection that addresses both challenges simultaneously. First, we pre-train the network using unlabeled samples within a self-supervised learn- ing framework by solving pretext tasks that require the network to learn informa- tive representations from partial feature sets. Then, we fine-tune the pre-trained network to discover relevant features using labeled samples. During both train- ing phases, we explicitly account for the correlation structure of the input features by generating correlated gate vectors from a multivariate Bernoulli distribution. Experiments multiple real-world datasets including clinical and omics demon- strate that our model discovers relevant features that provide superior prediction performance compared to the state-of-the-an benchmarks in practical scenarios where there is often limited labeled data and high correlations among features.
INTRODUCTION
High-dimensional datasets are increasingly prevalent a variety of fields, including critical do- mains such medicine and biology. Discovering features responsible for the target variable is essential but challenging step the analysis of such data. For example, while next generation sequencing can detect the expression of tens of thousands of genes per sample (e.g., RNA-seq), many genetic disorders stem from the variation in only a few groups of related genes (Jackson et al., 2018). Identification of such disease-related factors crucial for the design of therapeutic treat- ments. Furthermore, feature selection has additional benefits, including reduced experimental costs, greater interpretability, and improved model generalization (Min al., 2014
Ribeiro et al. 2016).
However, the high-dimensional and often noisy nature of such data prevents relevant features from being readily discovered. Moreover, in many domains there is scareity of label information due to cost, privacy reasons, experimental study design. While deep learning approaches have resulted improvements in feature selection (e.g.. Yamada et al. 2020);(Lemhatte et al. (2021), see Related Work), such methods typically assume access to many labeled samples. With low sample size, feature selection methods, in particular those employing deep learning, have shown a propensity to overfit high- digh-dimensional data (Kuncheva et al. 2020).
This issue is further exacerbated by the inherent structure of such data. Many thigh-dimensional) datasets exhibit substantial inter-correlations multicollinearity - -i.., there exist features that (highly) correlated among themselves which impacts the performance of feature selection methods (Chong & Jun 2005; Katrutsa & Strijov| 2017]] Belsley al. 2005). In particular, this structure can cause parameter estimates to be unstable (Dobson & Barnett 2018) and, in the extreme, can prevent variable effects from being separated, confounding the problem of feature selection (Meloun al., 2002). Existing deep learning- baaning-based methods for feature selection do not (explicitly) consider the correlations between features which will often result in the selection of redundant features.
"Equal contribution

---

Published as a conference paper at ICLR 2022
When labeled samples scarce, large numbers of unlabeled samples are often available (e.g., Perez-Riverol et al. (2019). However, the importance of unlabeled data has been overlooked in feature selection despite its potential to prevent the model selection process from overfitting by allowing informative feature representations to be learned. Therefore, the development of methods for feature selection that can exploit both labeled and unlabeled data is of great practical importance.
Contributions. We propose a novel method for feature selection that addresses two key challenges in real-world scenarios: limited labeled data and correlated features. We use a self-supervised ap- proach to train an encoder using unlabeled data via two pretext tasks: feature vector reconstruction and gate vector estimation. This pre-conditions the encoder to learn informative representations from partial feature sets, aligning the self-supervision with the model selection process of the down- stream feature selection task. In addition, we introduce a novel gating procedure that accounts for the correlation structure of the input features. This ensures the pretext tasks remain challenging by preventing the model from memorizing trivial relations between features. Moreover, unlike previous deep learning- -ased feature selection methods, the correlated gate vectors encourage method to select the most relevant features by making multiple correlated features compete against each other.
We validate our approach through experiments on synthetic and multiple real-world datasets, in- cluding clinical and omics, where only a small number of labeled samples are available. Our model discovers relevant features that provide superior prediction performance compared to state-of- the-hhaaa art benchmarks, and we corroborate these features with supporting medical and scientific literature.
RELATED WORK
Feature Selection Methods. Feature selection is well-studied problem, with number of proposed
solutions including wrapper (Kohavi & John 1997) and filter (Liu & Setiono 1996 Kira & Rendell 1992) methods. Recent advances in deep learning provide elegant way of training embedded feature selection methods by jointly learning to perform feature selection while training a prediction network (Huang et al.] 2020). These methods learn to perform the non-differentiable process of selecting feature subsets by approximating it either with Lasso or elastic net penalization (Li et al., 2016), using an MCMC sampling approach (Liang et al., 2018), or more recently with continuous relaxation using independent Gaussian random variables Yamada et al. (222). However, supervised feature selection methods can fail identify relevant features when limited labeled samples are available due to overfitting (Kuncheva et al. 2020), impacting their suitability in many real-world scenarios. Moreover, these methods do not consider the underlying correlation structure of the input features which can be problematic when selecting relevant features among (highly) correlated ones.
Relatively few feature selection methods use unlabeled samples. Abid et al. (2019) used auto- encoders to identify pre-specified number of features that are sufficient for reconstructing the data. Lindenbaum et al. (2020) improved the well-known Laplacian score (He et al., 2005) by selecting feature subsets that better capture the "local structure" of the data. However, both approaches are fully unsupervised and, without the guidance of label information, can fail to identify features rel- evant the target outcome. While number of semi-supervised feature selection methods have been proposed (Sheikhpour et al.] 2017), they have typically been extensions of traditional methods, such Laplacian score with modified affinity scores using labels (Zhao et al. 2008) or manifold regularization based on linear SVMs (Dai et al. 2013). To the best of our knowledge, this is the first deep learning framework that fully utilizes both labeled and unlabeled samples for feature selection.
Self-Supervised Learning. Self-supervised learning methods create (weak) supervised signals from unlabeled data, employing contrastive learning or pretext task(s) to provide surrogate labels. While self-supervised learning has found success in computer vision (Chen et al.) 2020) and natural lan- guage processing (Devlin et al., 2018), the tabular domain, which is most relevant in feature se- lection, has been largely neglected. Methods for tabular data seek to reconstruct data either based on a corrupted sample alone (Vincent et al.) 2008; Arik & Pfister 2019 Yin et al. 2020) or with knowledge of which entries have been corrupted (Pathak et al. 2016). Recently, Yoon et al. (2020) jointly learn to recover the original sample and predict the mask vector used to corrupt the data.
While our pretext tasks are also reconstruction based, we propose a novel method for generating the gate vector used to produce the input feature vector. This is particularly important when there exists substantial correlation between features (see Section 5). All existing methods have used indepen-
2

---

Published as a conference paper at ICLR 2022
dent, uniform sampling; however, this allows highly correlated features to be readily reconstructed since it is likely that not all such features will be corrupted. We prevent this by incorporating the correlation structure into the gating procedure. In this work, we employ pretext tasks that mirror the goal of the feature selection process: the input to the encoder is subset of the features selected random. Employing partial feature sets for self-supervised learning encourages the encoder to learn better representations of these partial feature sets. This in turn benefits learning both the model selection function and the feature selection step since they are also trained using partial features sets.
PROBLEM FORMULATION
Let X = (....... Xp) E XP and Y E y be random variables for the high-dimensional input features (e.g., gene expressions) and the target outcome (e.g., disease traits) whose realizations are denoted as x (#1.-.., ;zp) and y, respectively. Throughout the paper, will often use lower- case letters to denote realizations of random variables.
Emhedded feature selection aims to select subset, S C [p). of features that are relevant for pre- dicting the target as part of the model selection process. Denote * be any point not in X' and define Xs = ( Then, given X X², the selected subset of features can be denoted Xs E X's where TSS* I& if k E S and æ& = * if k & S. Let f Xs + ) be function in F that takes as input the subset Xs and outputs Y. Then, selecting relevant features can be achieved by solving the following optimization problem:
minimize FEF, SCIp)
Exp-oxx [e ((x)))
subject
|S| S
where 5 constrains the number of selected features and by (v. = - Ec Ye log for C-way classification tasks (i.e., y = (1.... ,C} and (y = lly - 1/113 for regression tasks (i.e.,
) = R) P Unfortunately, the combinatorial problem in (1 becomes intractable for high- high-dimensiooal data as the search space increases exponentially with p. Hence, we instead focus on a relaxation by converting the combinatorial search in ( into search over the space of possibly correlated binary random variables. It is worth highlighting that unlike existing work (e.g.. Yamada et al. (2020); Yoon et al. (2019)), we do not assume independence among these random variables.
Let M = (M, Mp) E {0. 132 be binary random variables governed by distribution pM. whose realization is referred to as the gate vector, for indicating selection of the corresponding features. Then, the selected features given gate vector m can be written as
A m x + (1 m) x
where x = Excpx [x] and @ indicates element-wise multiplication. Here, we replace not-selected features with their means to resolve any issue when feature having zero value (e.g., turned-off genes) has specific meaning. Finally, we can approximately achieve [ by jointly learning the model selection and the gate vector distribution pm based the following optimization problem:
minimize Expy-xx Em-pas (x (v, f(x) + -||m||o|
(3)
f. pm
where f is implemented as a neural network and (3 is balancing coefficient that controls the number of features to be selected.
Challenges. In practice (especially in the healthcare domain), there are two main challenges that confound selecting relevant features for predicting the target: (i) inter-correlation or multicollinear- ity, which is the existence of features that are (highly) correlated among themselves, and (ii) the absence of sufficient labeled samples. These challenges make embedded feature selection vulner- able to overfitting. More specifically, model selection (i.e.. learning f) and feature selection (i.e., learning Par) are conducted jointly. Either one being overfit to correlated or noisy irrelevant features will end up discovering spurious relations and poor prediction. For instance, if the network is bi- ased toward irrelevant features, the selection probability (i.e., importance) of those features will be increased as if those features were "discriminative" or "predictive", and vice versa.
1 To enable neural networks to be trained with varying feature subsets, we set * = R. For y = {1,- 11.... C), we will nccasionally abuse notation and write ye denote the e-th element of the one-hot encoding of y when clear in the context.
3

---

Published as a conference paper at ICLR 2022
Reaure Vittr Esteuter (1p.
Entoder
Gate Vester bii (1ay)
7
Subset Gonerator
Supervition Phase
Fre-trolied
Predicoo (())
Subset Sonerator
(4) Festure Subeet Genaralor
(3) Superddiion
An illustration of SEFS. The selection probability in the Feature Subset Generator is fixed and treated as hyper-parameter during Self-Supervision Phase while it is updated during Supervision Phase using reparameterization trick.
Figure 1:
4 METHOD: SELF-SUPERVISION ENHANCED FEATURE SELECTION
In this section, describe novel feature selection method, which refer to Self-supervision Enhanced Feature Selection (SEFS). To address the challenges described above, we propose a novel gate vector generation process modeled by multivariate Bernoulli distribution where dependencies among gates are transferred from the correlation structure of the input features, and a two-step training procedure that utilizes both labeled and unlabeled samples.
Our approach is defined by the selection probability π, that governs the gate vector generation process, and the model selection function f. that we decompose into encoder and a predictor fø. i.e., f f o fo. as illustrated Figure To utilize both labeled and unlabeled samples, these components are updated via a two-step training procedure:
Self-Supervision Phase: the encoder fo is pre-trained to learn favorable representations for fea- selection using unlabeled samples via solving two pretext tasks specifically designed to take only subsets of the features as input.
Supervision Phase: the learned representations from the Self-Supervision Phase are adopted to solve the objective in 3 i.e. updating the encoder fe. the predictor fo, and the selection probability π for discovering task-relevant features using labeled samples.
4.1 MULTIVARIATE BERNOULLI GATE VECTORS USING GAUSSIAN COPULA
Instead of assuming independence in the gating mechanism we model the gate vector distribu- tion PM as a multivariate Bernoulli distribution where dependencies among the gates are determined by the correlation structure of the input features. To this goal, we Gaussian copula (Nelsen) 2007) to construct valid multivariate Bernoulli distribution. The Gaussian copula is multivariate cumulative distribution function (CDF) of random variables (U1,... Up) the unit cube [0, 1]2 with uniform marginals, where U2 ~ Uni form(0, 1) for E [p]. Formally, given the correlation matrix R [-1, 1]2 that captures the correlation structure X, the Gaussian copula defined as:
CU ..., Up) = ( ( ... (U,))
where PR stands for the joint CDF of multivariate Gaussian distribution with mean zero vector and correlation matrix R, and Φ is the inverse CDF of the standard univariate Gaussian distribution.
Define π (...... Tp) E [0, 1] to be the selection probability that governs the multivariate Bernoulli distribution used to generate the gate vector m. Then, given CR, we can generate the gate vector from multivariate Bernoulli distribution, which is denoted as m Mul t Bern ( <T; R.), based on the correlated random variables (U1,... Up) that preserve the correlation structure of the input features; formally, m = 1 if U I गy and m, if U3 > मy for E [p].
Using the correlation structure of the input features when generating the gate vector has the follow- ing two advantages: (i) during the Self-Suuervision Phase, correlated gate vectors encourage the network not to rely trivial signals from highly correlated features when solving the pretext tasks, which may lead to inefficient pre- pre-training and sub-optimal downstream performance; (ii) during the Supervision Phase, correlated gate vectors encourage the network to select only the most rele-
4

---

Published as a conference paper at ICLR 2022
vant features by increasing the chance that correlated features selected jointly and thus compete against each other.
4.2
PHASE
Self-supervised learning uses unlabeled samples to automatically generate a (weak) supervisory signal by solving relevant pretext tasks in the absence of labeled data. The learned representations can then be used to solve downstream tasks. To this goal, we begin by defining two pretext tasks motivated by recent success in sulf-supe learning for tabular data (Yoon et al., 2020 The pretext tasks to be jointly solved are: (i) reconstructing the original input x based on a randomly selected subset of input features ž, and (ii) simultaneously estimating the gate vector m that defined which features were selected and which features were not (i.e., replaced by their mean value).
To solve these pretext tasks, we formally define the encoder fa and auxiliary network compo- nents hug hws that are temporarily employed during the Self-Supervision Phase:
Encoder, fo XP - 2, that takes as input either the selected subset of features the original feature vector x, and outputs latent representations z E Z.
Feature vector estimator, hug 2 + XX, that takes So (x) input and outputs a vector x which is an estimate of the original input x.
Gate vector estimator, hugg Z + [0, 1J", that takes = (x) input and outputs vector In which is prediction of which features have been selected (i.e., my = 1) and which features have been replaced by the mean (i.e., m = 0).
When generating the subset of features x, each feature is selected based on the multivariate Bernoulli distribution with an equal probability π,, i.e., nं æ for [p]. which throughout the Self- Sulf-Superviiion Phase. The adoption of an equal selection probability allows us to make no assumptions about the relative feature importance, which not known priori.
fixed
The purpose of the Superviservi Phase is to pre-train the encoder. As such, both estimators take as input the output the same encoder fo, which the only component that will retained in the Supervision Phase. Formally, given the latent representations z fo(x), the reconstructed input feature and the gate vector are defined as & & heg fo (X) and r - hw2 fx(),, respectively, The encoder and the two estimators are trained jointly based on the following objective function:
minimize 0,01,,82
Ex-px moppa [x (x, x) + a (m, m)
(5)
where a is coefficient chosen to balance between the two pretext task losses: (x(xx)) = ||x- * and la (m, m) Σ my log mh + (1 mg.) log(1
m)))
Compared to previous work Yoon et al. 2020), the correlated gating procedure in the proposed method prevents the pretext tasks from being solved by only exploiting trivial relationships among features by increasing the probability that highly-correlated features are masked simultancously. Further, the pretext tasks directly mirror the Supervision Phase: in both phases, the input to the encoder is subset of the features x. This trains the encoder to produce better representations of partial feature sets which in turn prevents the model selection process in the Supervision Phase from overfitting to spurious, noisy features, benefiting feature selection.
4.3 SUPERVISION PHASE
During the Supervision Phase, the encoder, predictor, and the selection probability, which refer to as the feature selection network, are jointly updated to solve the feature selection objective (3). The predictor is formally defined as follows:
Predictor, fo Z + D. that takes as input the latent representations of the selected subset fo (x), and outputs predictions the target outcome. Together with the
of features, i.e., encoder, the model selection function is given by f = fo o fo.
To account for the correlations between features,
generate gate vectors using multivariate Bernoulli distribution. This is important since the correlated gate vectors encourage the network to select only the most relevant features by making multiple correlated features compete against
5

---

Published as a conference paper at ICLR 2022
each other. However, solving 3 directly using such gate vectors is intractable since Par (m) has differentiable closed-form expression. Instead, we adopt a continuous relaxation of the multivariate Bernoulli distribution (Wang & Yin 2020) by applying the reparameterizztion trick to the correlated uniform random variables (U1,".. Up) that preserve the correlation structure R.
(TI,... ;Tp)) and the multivariate uniform random variables (UI,-..,5, from Gaussian copula CR. we can generate relaxed gate vector m = (mi,,-.. ;mp) E (0, 1)2 based on the following reparameterization trick (Wang & Yin 2020):
Formally, given selection probability
my =a a (+ (log T& log(1 mk) + log U -log(1 U&))
(6)
where (x) = (1 + cxp(-x)) is the sigmoid function. Such a relaxation is parameterized by π, a pre-specified covariance matrix R. and temperature hyper-parameter (0, oo). Similar the Relaxed Bernoulli distribution (Maddison et al. 2017). (6 is differentiable with respect to æ.
In practice, the gate vector generation process in 66 can be handled deterministic transformation of samples from a standard multivariate Gaussian random variable. More specifically, we first draw samples from the multivariate Gaussian distribution v (O, by v (U1, Up) = Le where ~ (0, I) and L is a lower triangular matrix with positive diagonal entries, which is derived by the Cholesky factorization of the covariance matrix, i.e. R LL"" Next, we generate correlated ((v)) for k [p]. Finally, we can generate a relaxed multivariate Bernoulli variable nH E (0, 1) by plugging in Ug and Ty into (6).
uniform random variables
Under the continuous relaxation, the regularization term in (3) that induces sparsity of selected features can simply derived Emapx ||mi|o = Ed=1 P(Uk S Tk) = Eb Tk- Overall, we can rewrite our objective
minimize 0
Ex.r-xr. (. lx (v. 1o0 m m m ) 3
(7)
k=1
The model selection function f fo se and the selection probability are updated jointly via gradient descent. Pseudo-code of SEFS can be found in Appendix
EXPERIMENTS
In this section, we evaluate the performance of SEFS and multiple feature selection methods using synthetic and several real-world datasets. Further experiments together with detailed information regarding all experiments, benchmarks, and datasets can be found in the Appendix.
Benchmarks. We compare SEFS with 7 feature selection methods including 3 conventional meth- ods and 4 state-of-the- -ftt---rt- methods. The conventional methods include LASSO regularized lin- ear/logistic model (Lasso, [ibshirani (1996)), extremely randomized tree ensembles (Tree, Geurts let al. (2006)), and Laplacian Score that preserves locality structure (L-Score, He et al. all.]((20)))) (2005)). State- of-the-art deep learning methods include Bayesian neural networks for feature selection (BNNsel, (Liang et al. (2018), feature selection based on continuous relaxation (STG, Yamada et al. (2020). unsupervised feature selection based on gated Laplacian (DUFS, [indenbaum al. (2020). We also include an extension of STG to the semi-supervised setting by augmenting the loss with a re- construction task that estimates the original features from the gated inputs (STG (SS)). Among the benchmarks, three are able to use both labeled and unlabeled samples: DUPS is fully unsupervised, while both L-Score and STG (SS) are semi-supervised approaches that explicitly use the label.
To highlight our novel two-step training process, we also include variant of our method that is trained without employing the Self-Supervision Phase, which denoted as SEFS (no SS).
Performance Metrics. For the synthetic experiments, we use the true positive rate (TPR) to as- sess whether discovered features are truly relevant compared to the ground-truth. With real-world data, however, it is not possible to calculate the TPR since the ground truth relevance is often not known. Hence, we instead evaluate the prediction performance the discovered features obtained by different feature selection methods. For classification tasks, operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC), and for regression tasks, mean squared error (MSE). This is indirect way of assessing the relevance of the
the area under the receiver

---

Published as a conference paper at ICLR 2022
Table 1: AUROC and AU- RPC (mean + -I) given the learned representa- tions for 20=00, n. n0=1000.
56P5 Dn
SET5 (AL)
Orreg)
Tour Tosttve Rut
0.20
0.00
00
Representation
AUROC
SEFS (12 55, Indea
SE65 9V
0.0
SEFS (AE)
0.85100.33
SEP2
SEFS (indep)
0.88
0:20
300 1975
2000
4003
SEFS
0.92-0.02
Samples
(a) TPR vs Labeled Samples
(b) TPR Unlabeled Samples
Representation
AUPRC
Figure 3: Results on the Block-Structure, Noisy Two-Moons dataset. Average TPRs with with (a) varying numbers of labeled samples given n, 100= and (b) varying numbers of unlabeled samples given m==10,
SEES (AF)
0.80
SEFS (indep)
0.87=00.33
SEFS
0.91=20.03
discovered features the target tasks: we first conduct feature selection based each method and then train a multi-layer perceptron (MLP) to perform predictions on top of the featwre-selected data. Training separate MLP isolates the effect of having different model selections such linear models for Lasso, ensemble trees for Tree, and deep neural networks for BNNsel, STG and SEFS and thereby provides a fair comparison of the discovered features.
Experimental Setup. For each dataset, we divide the available samples into labeled and unlabeled. Labeled samples are randomly split into training and testing sets. Within the training set, n; labeled samples from the training set are used to train the feature selection methods and the entire labeled samples are used evaluate the discovered features. As result, while features are selected based on limited sample sizes, a significantly larger number of labeled samples are used to validate the discovered features. Details of the data construction process is given in Appendix C
Implementation. Implementation details, including hyper-paramend and the construction of co- variance matrix R, together with details for all benchmarks can found in Appendix
5.1 SYNTHETIC: BLOCK- STRUCTURED NOISY TWO-MOONS DATASET
To motivate and confirm our intuition of applying sulf-supernision and correlated gate vectors in feature selection, we begin by evaluating SEFS via a set of synthetic experiments that addresses two key challenges- i.e., limited labeled data and correlated features in worldworld scenarios.
Dataset Description. We consider the well-known 2-dimensional "Two-Moons" dataset with addi- tive noise with variance a² = 0.1. The outcome label (i.e., y = {0, is generated solely based on these 2 features (i.e., I and 22)). We augment 8 auxiliary feature dimensions that are irrelevant to target outcomes by sampling from the standard Normal distribution. Finally, we introduce a cor- related block-structure by augmenting each feature with 9 highly-correlated features (correlation: 0.94) by injecting small white noise (i.e., (0, 0.3?)), yielding 100 features in total.
Quantitative Analysis. In Figure 3 a). compare the average TPR with varying numbers of labeled samples ny = {10, 20, 40, 60, 80} while fixing the number of unlabeled samples n. = 1000. We define æ1 (or 222) correctly discovered if and only if 21 (or 222) has the first second highest feature importance. SEFS benefits from learning the data structure using unlabeled samples and provides significant gain over all baselines (see Appendix AppendixD.I] for
full comparison). The majority of benchmarks struggle to identify x in particular due to z2 being more discriminative than æ and therefore noisy features that are correlated with I2 are often selected (Table S.4. We highlight the benefit of the correlated gating procedure by comparing SEFS (no SS) and SEFS (no SS, indep), a variant that uses the independent Bernoulli distribution to generate gate vectors. Our novel gating procedure significantly improves the TPR even in the fully supervised setting (Figure 3
We further investigate the utility of the Self-Supervision Phase SEFS. To this goal, we intro- duce two counterparts that replace our self-supervised training with (i) conventional auto-encoder (SEFS (AE)) and (ii) variant of our model with an uncorrelated gating procedure using indepen- dent Bernuuli random variables (SEFS (indep)). In all cases, we employ the same feature selection process as SEFS to isolate the impact of seff-supervision. In Figure 3)b, we evaluate these variants in terms of the TPR with varying number of unlabeled samples n. 200, 500, 1000, 2000. and 4000 while fixing the number of labeled samples n, = 10. (i.c., 5 for each label). Our method sig- nificantly outperforms the other variants, improving the TPR over SEFS (no SS) even with relative
7

---

Published as a conference paper at ICLR 2022
small number of unlabeled samples. Moreover, we further evaluate the quality of the learned repre- sentation by assessing the discriminative performance of the learned representations given only the ground-truth features (i.e., setting m, m2 and my 0 for k & 21) ). As shown in Table I the proposed gating procedure in SEFS helps the encoder to learn meaningful structure and results in the most performant representations, further validating the self-supervised training procedure.
5.2 CLINICAL: UKCF DATASET
Table 2: AUROC and AUPRC (mean
+ -5)-CI) for |S| 10 discovered features for the UKCF dataset.
Dataset Description. The UK Cystic Fibrosis registry (UKCFF records annual follow-ups for 6,754 adults over the period 2008-2015. We include patients with obser- vations in at least three consecutive years. Each patient is associated with p=245 clinical variables (11 static and 38 time-varying features), including demographies, ge- netic mutations, clinical tests, and therapeutic management. We set respiratory failure (death or lung transplant) in the next 5 years as the label.
Methods
AUROC
AUPRC
Lasso
0,767=00054
0,409=200083
Tree
0,807-0.036
0.463+0.066
BNNsel
0.650=00001
0,269=0.069
STG
0.781=00048
0.440=000082
DUTS
0.799-10.079
0.398:10.062
L-Soore
0.668:100010
0.213:10.017
STG (SS)
0.810=20.036
0.477=00045
SEPS (no SS) SEFS
0.785-10:044
0.412:40.083
Quantitative Results. In Table we compare the predic-
0.846=20013
0,532=20.027
tion performance of feature selection methods with ny = 32, n. = 4754 and |S| 10. As expected, SEFS benefits from learning the data structure utilizing the unlabeled samples and discovers features that provide significantly greater discriminative performance over all other methods (Table 2 Fully supervised feature selection methods struggle to discover relevant features with only a few labeled samples. The benefit of sele-supe can be clearly seen by comparing SEFS with SEFS (no SS). Our results also highlight that fully unsupervised feature selection methods (i.c., L-Score and DUFS) struggle to discover relevant features these methods cannot select features that effectively distinguish the target outcomes without the guidance label information.
Two interesting features identified by SEFS but that were not discovered by other methods were PI Allele and 2 that are related to pancreatic functions. More specifically, pancreatic function studies have demonstrated that (types of) mutations in the CFTR gene, which causes CF, are highly associated with dysfunction in organ systems and the pathology of CF patients (Sosnay et al., [2013 Gibson-Corles et al.) 2016). Further details of the most frequently discovered features and supporting evidence of their clinical relevance can be found in Appendix D.2
Qualitative Results.
5.3 PROTEOMICS: CCLE DATASET
Table 3: Frequently discovered
Dataset Description. We study the response of heterogeneous cancer cell lines to 11 different drugs where the goal is to iden- tify proteins associated with the cell line response based pro- teomic measurements from the Cancer Cell Line Encyclope- dia (CCLE. Barretina et al. (2012)) CCLE is small dataset containing 899 cancer cell lines (i.c., samples) described by 1-196 protein expressions. The real-valued drug response (i.c.. ) = R) is available for 458 samples and is missing for the remaining 441 samples (thus unlabeled). To benefit from self- supervised learning, we integrate the RPPA measurements 7.329 samples from The Cancer Genome Atlas (TCGA) cre- ating overall 7,770 unlabeled samples (i.e, n. 7770). While the two datasets are not generated from the same distribution, expect some benefit from learning basic correlations among features, even if not all correlations from CCLE are preserved.
features by SEFS for Panobinos- tat (CCLE). Features in blue are not chosen by SEFS (no SS). Supporting references provided in Appendix D.3.
Rank
Proteins
Ref.
1
Caveolia-1
2
YAP PS127
3
PRAS40-pT246
VHL
5
Sre-pY416
7
14.3-3.9
Fibromectin
GSK3.
10
MSH2
Quantitative Results. In Figure 4 compare the ranking of SERS and the benchmarks across 11 drugs (n: 20, nh, = 7770, S 10). SEFS is the best performing method for 9 drugs and is always in the top 3 (median rank: average rank: 1.36). Despite the majority of unlabeled data originating from a different source, SEFS outperforms SEFS (no SS) in every experiment. While
https: http://wwwwaccccccooo
8

---

Published as a conference paper at ICLR 2022
5E15
SE1S (1o 551
BMhsel
STG
DU15
L-Scome
Lasso
10-0223901 #F2341066
PLX472C Pac
Figure 4: Ranking of the average prediction performance for |S] 10 discovered features across different drugs in the CCLE dataset. (Rank: best, 9 worst.)
Table 4: AUROC and AURPC (mean + 95%-CCI given | discovered features for the PBMC dataset.
SEFS
0.7
Lasso
Methods
AUROC
AUPRC
0.5
Lasso
0,703-0.033
0.705=0.001
Tree
0,767=0.044
0.788=00.047
BNNsel
0.676:10.035
0.686110.035
0.2
STG
0.001
0.777-10.071
DUFS
0.734=2.002 029
0.742:20.034
L-Score
0.699:10.009
0.688:10.009
0.0
STG (SS)
0.794=10.065
0.814:20.099
Expression
SEFS (no SS)
0.768-10.055
0.783.10.057
Figure 5: Comparison of discovered features using Lasso,
SEFS
0.884:10.013
0.901
Tree, and SEFS for the PBMC dataset with n/-20.
we would expect further gain from more similar unlabeled data, our results highlight that there is potential benefit even when unlabeled data is only partially related to the labeled samples.
Qualitative Results. Now we have demonstrated the superior predictive performance of the features discovered using SEFS, we seek validate these features in the scientific literature. Here, we focus analysis on panobinostat, histone deacetylase (HDAC) inhibitor used in the treatment of several cancers. We found strong supporting evidence for of the top ranked features selected by SEFS (Table see Appendix Jppendix(D.3fool fo3 for full details). In addition, the remaining feature (fibronectin) is strongly implicated in cancer, but we did not find literature specifically relating to panobinostat or HDAC-mediated pathways. We provide supporting evidence for other drugs in Appendix D.3
5.4 TRANSCRIPTOMICS PBMC DATASET
Dataset Description. We focus on distinguishing sub- cub-populations of T-cells (i.e., y = {0, 1)), namely naive and regulatory T-cells, from purified populations of peripheral blood monocytes (PBMCs based on transcriptomic measurements (i.e., mRNA sequence). The PBMC dataset con- sists 20,742 samples described by p=19256 protein-coding genes. We primarily used nı = 20, ny 15557, with the remaining 1,5 labeled samples used to evaluate the discovered features.
Quantitative and Qualitative Results. Similarly to the performance on other datasets, SEFS sig- nificantly outperforms all benchmarks (Table 4 Figure S.4 displays only features selected with frequency 0.4. It is immediately apparent that only SEFS consistently identifies the same fea- tures, while Lasso and Tree typically select different features, demonstrating the robustness of our approach. We provide supporting evidence for the importance of these features in Appendix D.4 D.4]
CONCLUSION
In this paper, we proposed SEFS, self-supervised feature selection framework that is able to lever- age the abundant quantities of unlabeled data that often available, achieving state-of the-art performance when limited labeled data is available. Using synthetic data, we motivate and confirm benefit feature selec-
why self-supervision and novel gating procedure tion methods. Through experiments on multiple real-world datasets, we validate that our model discovers features that provide superior prediction performance and corroborate the vast majority of the discovered features with supporting medical and scientific literature. As with all feature selec- tion methods, the discovered features based SEFS should be further experimentally verified or evaluated by domain expert prior to deployment in practice.
intuition
https:: /uspportt 10xeeoomi cs com/ wwt.em/iinnlleeee geell-ganeeeee www.gggeeeepppeiiii/aaaaaassseee

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank anonymous reviewers as well as members of the vanderschaar-lab for many insightful comments and suggestions. CL was supported through the IITP grant funded by the Korea govern- ment (MSIT) (No. 2021-0-01341, Al Graduate School Program, CAU). FI and MvdS are supported by the National Science Foundation (NSF), grant number 1722516. MvdS is additionally supported by the Office of Naval Research (ONR).
ETHICS STATEMENT
Like all methods for selecting features or providing the importance of features from observational data, SEFS relics on an assumption (i.c., the important features subset should be enough to provide good prediction power). Consequently, all identified features should undergo additional evaluation or verification by domain experts prior to deployment in practice. this work, we highlight di- rect/indirect impact of our method through experiments including the UKCF, CCLE, and PBMC datasets. The use of these datasets was in accordance with the guidance of the respective data providers and domain experts.
real-world clinical and omics datasets
REPRODUCIBILITY STATEMENT
The source code of SEFS and the synthetic dataset are provided in the Supplementary Material and is also available at at : qi thub. ctm/ ch1 2h188856/35 (/55//FEF) SEFS All details of our method and the experimental setup are included either the main manuscript or the Appendix: description of the method can be found in Section 4 pseudo-code for SEFS is described in Appendix A Appendix B
provides implementation details for both SEFS and the benchmarks, Appendix C provides details about the real-world datasets.
REFERENCES
A. Abid, M. Balin, and J. Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In Proceedings of the 36th International Conference on Machine Learning (ICML 2019), 2019.
Sercan Ömer Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. arXiv preprint arXiv:1908.077 07442, 2019.
Barretina, G. Caponigro, and N. Stransky et The Cancer Cell Line Encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature, 483:603-600 2012.
David A Belsley, Edwin Kuh, and Roy E Welsch. Regression diagnostics: Identifying influential data and sources of collinearity, volume 571. John Wiley & Sons, 2005.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning (ICML. 2020), 2020.
Il-Gyo Chong and Chi-Hyuck Jun. Performance of variable selection methods when mul- ticollinearity is present. Chemometrics and Intelligent Laboratory Systems, 78(1):103-112 12, 2005. ISSN 0169-7439.
doi: http://loo.agglll.ooooocccceaoaa www. http://cceccccccccecoo.ooooooeia
URL https:
Kun Dai, Hong- Yi Yu, and Qing Li. A semisupervised feature selection with support vector machine. Journal of Applied Mathematics, 64: 14111,1158, 2013.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 110.04800, 2018.
Annette J Dobson and Adrian G Barnett. An introduction to generalized linear models. CRC press, 2018.
10

---

Published as a conference paper at ICLR 2022
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine Learning, 63(1):3 (2, 2006.
Katherine N. Conson-Corley, David K. Meyerholz, and John Engelhardt. Pancreatic pathophysi- ology in cystic fibrosis. The Journal of pathology, 238(2)331 2016.
X. He, D. Cai, and P. Niyogi. Laplacian score for feature selection. In Proceedings of the 18th Conference on Newral Information Processing Systems (NIPS 2005), 2005.
Yingkun Huang, Weidong Jin, Zhibin Yu, and Bing Li. Supervised feature selection through deep neural networks with pairwise connected structure. Knowledge-Based Systems, 204, August 2020.
Maria Jackson, Leah Marks, Gerhard H.W. May, and Joanna B. Wilson. The genetic basis of disease. doi: 10. (0.2//2/B0000 70053.
Essays in Biochemistry, 62(5):643-723, 2018.
Alexandr Katrutsa and Vadim Strijov. Comprehensive study of feature selection methods to solve multicollinearity problem according evaluation criteria. Expert Systems with Applications, 76: 1-11, 2017. ISSN 0957-4174.
doi: http://ww......aaaaaaaa////////.
URL https:
7/www. www.saccencedireeccoo com
Kenji Kira and Larry Rendell. practical approach to feature selection. In Machine learning proceedings 1992, pp. 249-256. Elsevier, 1992.
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial Intelligence, 97 10016/5000004777777700444X
(1):273-324, 1997. doi:
Ludmila I. Kuncheva, Clare E. Matthews, Álvar Arnaiz-González, and Juan J. Rodríguez. Feature selection from high-dimensional data with very low sample size: A cautionary tale, 2020.
Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. Lassonet: Neural networks with feature spar- sity. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS 21), pp. 10-18, 2021.
Y. Li. C. Y. Chen, and W. W. Wasserman. Deep feature selection: theory and application identify enhancers and promoters. Journal of Computational Biology. 23(5):322-33) 336, 2016.
Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, 113(523):955-977 2018.
O. Lindenbaum, U. Shaham, J. Svirsky, E. Peterfreund, and Kluger Differentiable unsupervised feature selection based on gated laplacian. arXiv preprint arXiv:2007.047288 2000.
Huan Liu and Rudy Setiono. A probabilistic approach to feature selection filter solution. In Proceedings of the 13th International Conference on Machine Learning (ICML 1996), 1996.
C. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: continuous relaxation of discrete random variables. In Proceedings of the 5th International Conference on Learning Rep- resentations (ICLR 2017), 2017.
Milan Meloun, Jifí Militky, Martin Hill, and Richard Brereton. Crucial problems in regression modelling and their solutions. Analyst, 127(4):433-450, 2002.
Fan Min, Qinghua Hu, and William Zhu. Feature selection with test cost constraint. Interna- tional Journal of Approximate Reasoning, 55(1, Part 2): 167-179, 2014. ISSN 0888-613X. URL https: :/tp:://wwww wtt.ec..iienncreett....
doi: http://ccoeee/rrreeee///////1220 com/
R. B. Nelsen. An Introduction to Copulas. Springer, 2007.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexci Efros. Context en- coders: Feature learning by inpainting. In Computer Vision and Pattern Recognition (CVPR), 2016.
Yasset Perez-Riverol. Andrey Zorin, and Gaurhari Dass et al. Quantifying the impact of public omics data. Nature Communications, 10(3512), August 2019.
11

---

Published as a conference paper at ICLR 2022
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2016), 2016.
Razieh Sheikhpour, Mehdi Agha Sarram, Sajjad Gharaghani, and Mohammad Ali Zare Chahooki. A survey on semi-supervised feature selection methods. Pattern Recognition, 64: 141-158, 2017.
Patrick R Sosnay, Karen R Siklosi, Fredrick Van Goor, Kyle Kaniecki, Haihui Yu, Neeraj Sharma, Anabela S Ramalho, Margarida D Amaral, Ruslan Dorfman, Julian Zielenski, David L. Masica, Rachel Karchin, Linda Millen, Philip J Thomas, George P Patrinos, Mary Corey, Michelle H Lewis, Johanna Rommens, Carlo Castellani, Christopher M Penland, and Garry R Cutting. Defining the disease liability of variants in the cystic fibrosis transmembrane conductance regula- tor gene. Nature genetics, 45(10): 160-1167, 2013.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288. 1996.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pp. 1096-1103. ACM, 2008. doi: 10. 145/1390156. 1390294.
Xi Wang and Junming Yin. Relaxed multivariate bernoulli distribution and its applications to deep generative models. In Proceedings the 26th Conference Uncertainty in Artificial Intelligence (UAl 20), pp. 432, 2020.
Y. Yamada, O. Lindenbaum, S. Negahban, and Y. Kluger. Feature selection using stochastic gates. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sehastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8413-8426. Association for Computational Linguisties, 07 2020. doi: 18653/v aclp2//.......iin...
Jinsung Yoon, James Jordon, and Mihaela der Schaar. INVASE: Instance-wise variable lection using neural networks. In Proceedings of the 7th International Conference on Learning Representations (ICLR 2019), 2019.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self- and semi-supervised learning to tabular domain. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
Jidong Zhao, Ke Lu, and Xiaofei He. Locality sensitive semi semi-supervised feature selection. Neuro- computing, 71:1842-a -889,, 2008
12

---

Published as a conference paper at ICLR 2022
A
PSEUDO-CODE OF SEFS
SEFS trained via a two-step training procedure. We provide pseudo-codes for the Self-Supervision
Phase in Algorithm and for the Supervision Phase in Algorithm 2
The source code for SEFS is available in https:://ggthhii httpp//ccccccccccc.5/83/3
Algorithm Pseudo-code for the Self- Self-Supervision Phase of SEFS
Input: Dataset D,, {x' ==1+ coefficient a, selection probability n,
mini-batch size Temb, learning rate
Output: SEFS parameter (
Initialize parameters (0, 1/1, 1/22) Compute the mean value: X
Thu x²
Compute the correlation matrix: R. where Raj = 1C2yl and Okj
Vou Ojs
repeat
Sample a mini-batch of Temb unlabeled data samples: {x" Jimi De for i- 1.... Amb do
Sample gate vector: m MultiBern(T: R)
Generate feature subset: X <- m O x"
+
m) OX
Estimate feature vector: X + hwr fx(x')
Estimate gate vector: m +- hig (x')
end for
Update the encoder parameter
Rmmb
8 0
V
1x x*) Las .m"
7mmb
Update the feature vector estimator parameter 4/1:
1 + 1/1
1)
V
Σ fx(xx, xt1)
nmmb
Update the gate vector estimator parameter ths
Touh
1/2 + r/2
V/
a Las (mm'm,, mn)
Numb
i=
until convergence
B IMPLEMENTATION DETAILS
The hyper-parameters of SEFS and those of the benchmarks are chosen via a grid search. For all methods, we use 20% of the overall training set as the validation set, which will then be unseen for training feature selection methods with chosen hyper- paperr-rramters...
Once feature selection methods are trained utilizing small subset of the overall training set as illustrated Figure S.I we train MLPs (with layers and 100 nodes) with feature-selected data. As described in Section 5, training separate MLP isolates the effect of having different model selections and thereby provides fair comparison among the discovered features.
It is worth highlighting that generating correlated gate vectors be computationally burdensome for high- digh-diimnsional data due to the matrix multiplication in Algorithm 3 and 4 To avoid such an issue, first apply thresholding (e.g., 0.7 for the PBMC dataset) to the correlation matrix R. group
13

---

Published as a conference paper at ICLR 2022
Algorithm 2 Pseudo-code for the Supervision Phase of SEFS
Input: Dataset Dj {x', y² J pre-trained encoder 8, coefficient B, mini-batch size nlmb, learning rate !)
Output: SEFS parameters (0, 0, T)
Initialize parameters (Ø, T Compute the mean value: Via Ei=1 x
Compute the correlation matrix: R where RRs = Ca,il and Okj
Ene (ur, - -255
Vouu-Cs
repeat
Sample a mini-batch of Remb labeled data samples: {xt, )/m for i 1,
Rmm Dr
7mmb do
Sample gate vector: m ~ Relaxed Molxxed-MuliiBBennn π:; R)
Generate feature subset: X m O x' + (1 m x
Predict outcome given the feature subset: 8 + fo o fo (xx)
end for
Fine-tune the encoder parameter 0:
Pmbb
0 - 1VVo
Σ (x()r
Rmmb
Update the predictor parameter di Φ с Φ
V
Σ
fr
1lmb
1=1
Update the selection probability parameter n::
π 4- π
V
(r
N.ma 1=1
1=1
until convergence
Algorithm Pseudo-code for Mult Mult (BllniBerr(r;R)) R)
Input: selection probability म, correlation matrix R
Output: correlated gate vector m Draw standard normal sample: e ~ (0,
Compute L Cholesko weeeeeekk-eecoosssiiis
Generate multivariate Gaussian vector v = Le
for 1, I.... .P do
Apply element- wise Gaussian CDF: uy & (vv))
Generate correlated gate: my = 1 (ug I mp)
end for
features based on the agglomerative clustering sure, and then conduct block-wise matrix multiplication. Overall, the generated gates for features within the same group will maintain the correlation structure. This significantly reduces the compu- tational complexity Algorithm 3 and as it scales quadratically with respect to the largest block size (i.e., the number of features grouped in the largest block). Thus, if the largest block size remains the same, the complexity of generating the correlated gate vectors will only increase linearly with the feature dimension (since the number of blocks would increase linearly).
6 Jusing the correlation matrix as the similarity mea-
"mplementedd using Python package s kit-eean
14

---

Published as a conference paper at ICLR 2022
Algorithm 4 Pseudo-codk for Relaxed--WulttBernnnnn (Wang & Yin 2020
Input:
selection probability æ correlation matrix R, temperature
correlated gate vector ın Draw standard normal sample: e ~ N/ I)
Output:
Compute L = weweessyy--oeosiiiieeeeee (R)(R)
Generate a multivariate Gaussian vector v Lc for k 1, =P do
Apply element- wise Gaussian CDF: uy = ((v))
Apply reparameterization trick (Maddison et al. 2017):
loo (logmT Te - log(11 - T&) + log un - log(1 - -mxa)
ma
where (xx) T+exp(- a))
end for
Throughout the experiments, training SEFS takes approximately 30 minutes to I hour for each phase (similar to that of STG) on single GPU machine
B.1 SEFS
The overall training process of SEFS consists of different network components that are imple- mented as neural networks: (i) encoder fa. (ii) predictor fş, (iii), feature vector estimator h1g and (iv) gate vector estimator h2g.. We use a fully-connected network as the baseline architecture for all the network components.
For the Supervuperiiioon Phase, coefficients (a, π) and the ber of hidden units, the number of nodes, and the number of layers of the encoder. the feature vector estimator, and the gate vector estimator are chosen utilizing grid search. We choose hyper- parameter values that achieve the lowest validation loss.
includinng the num-
For the Supervision Phase, coefficient |9 and the hyper- parameters including the number of nodes and the number of layers- - of the predictor are chosen utilizing a grid search (note that the hyper- parameters of the encoder is fixed after the Semi-Supervision Phase). We choose hyper-parameter values that achieve the best validation performance by using the outputs the predictor (i.c., 2 f fe(x)).
The permitted values of coefficients (a, 3, л) and the hyper-parameters are listed in Table S.I
B.2 BENCHMARKS
Throughout the experiments, compared SEFS with feature selection methods ranging from ventional approaches such as Lasso (Tibshirani, 1996) and Tree (Geurts et al. 2006) to the state-of-the-art approaches such STG (Yamada et al. 2020) and DUFS (Lindenbaum et al. 2020 Further descriptions of the benchmarks and implementation details are as follows:
Lassc Lasso is well-known embedded method whose objective is to minimize the prediction loss (1.c., cross-entropy loss for classification tasks and mean squared error for regression tasks) while enforcing the l--pnalty to achieve sparsity among input features [Tibshiran] 1996). We implement Lasso regression regression tasks and Logistic regression with 1 -penalty for clas- sification tasks. The bapar-parameter to control the input sparsity is chosen from {0.001, 0.01,
0.1, 1, 10, 100).
Tred ExtraTree Geurts et 2006 is an ensemble of decision trees that has become a popular choice for classification and regression tasks for tabular data. We use the feature importance
"The specification of the machine is: CPU Intel Core 17-8700K, GPU NVIDIA GeForee GTX 1080T;, and RAM - 64GB DDR4
'mplementee using Python package sc secikit-lean
Implementedd using Python package sci lear
15

---

Published as a conference paper at ICLR 2022
Table S.I: Hyper-paramettrss of SEFS.
Block
Set of Hypee-Parrametrs
Initialization
Xavier Glorot & Bengio 2010
Optimization
Adam Kingma Ba
Mini-batch size'
32
Non-lincann Non-lincarity
ReLu
Semi-Supervision Phase
Learning rate (7)
{0.0001, 0.001, 0.01}
No. of hidden units
{10, 30, 50, 100}
No. of layers
[1.2.3]
No. of nodes
(10, 30, 50, 100, 300, 500)
Coeff. a
(0.01, 0.1, 1.0, 10, 100)
Coeff.
(0.2, 0.4, 0.6, 0.8)
Supervision Phase
Temperature r
1.0
Dropout
0.3
Learning rate (7, 722)
(0.0001, 0.001, 0.01) [1,
No. of layers
No. of nodes
[10, 30, 50, 100, 300, 500}
Coeff. B
{0.01, 0.1, 1.0, 10, 100}
We used vu as the mini-batch size for the cases with the number of labeled samples smaller than 32, i.c., min(ni, 32).
obtained by ExtraTree to conduct feature selection. The number of trees and the maximum depth are chosen from {10, 50, 100, 300, 500) and {1, 3, 4), respectively.
L-oor The Laplacian Score (L-Scoree) (He et al. 2005) is an unsupervised (filter) feature selection method that quantifies the importance of input features by the ability to preserve local structure the data, which captured by the Laplacian matrix. We further modify the affinity score between pair of labeled samples following the description in (Sheikhpour et al. (2017), i... the affinity score becomes 1 if the two samples have the same label and otherwise. We construct the Laplacian matrix based on the Gaussian kernel using both labeled and unlabeled samples with 10-nearest neighbors and setting the kernel bandwidth as the median Euclidean distance.
BNNse BNNsel is an embedded feature selection method based on Bayesian neural networks that overcomes the non-differentiabiiiy in the selection process by utilizing MCMC sampling [Liang al. 2018). The number of hidden units is chosen among {3 (default value), 5, 10, 30, 50} with a lixed prior probability 0.025 (default value).
STG is an embedded feature selection method using deep neural networks that over- comes the con-differentiabee process of selecting feature subsets via continuous relaxation using Gaussian random variables (Yamada et al., (020).. We use a colleecondeeted network as the base- line architecture where the number of nodes and the number of layers are chosen among (10, 30, 50, 100, 300, 500} and (1,2,3,455), respectively. The hyper-parameter to control the sparsity of input features chosen among {0.001, 0.01, 0.1, 1, 10, 100).
STO 12
STG (SS): We propose an extension of STG to the semi-supervised setting. To this goal, we introduce reconstruction task in addition to the prediction task of the original STG. More specifically, we utilize an auxiliary network, i.e., decoder, which reconstructs the original input features given the gated features by minimizing the reconstruction loss. The overall network is trained based on the combination of the prediction loss and the reconstruction loss. (Here, we adopt hyper-parameter
E {1., 0.1, 0.01, 0.001} to balance the two losses). We use fully- connected network as the baseline architecture where the number of nodes and the number of layers are chosen among {10, 30, 50, 100, 300, 500) and (1,2,3,4.5), respectively. The hyper-
We explicitly implement wtw.pppleeeettttSSeeaeeeet L-Score for semi-supervised setting based on the description in Sheikhpour et al.
2017)
httee///ddar..bb/rr////////a//
html
http:://gtthbbb.cmmmm/ggggg////
16

---

Published as a conference paper at ICLR 2022
Table S.2: Summary statistics of the dataset.
Data Source
UKCF
PBMC
CCLE
Lapa
AZD6244
PD
PLX
Erlo Paeli
Topo
Data Type
elinical resp.
manseriptomies eell-type class
proteomics drug response
Target Outcome
Feature Dimension
285
19256
196
No. Labeled Samples (FS-Train)
32
25
27
29
Labeled Samples TTaii)
1000
1296
146
223
225
227
229
229
No. Labeled Samples (Eval--Test)
1000
3889
146
223
225
227
228
229
Unlabeled Samples
Trin: Triotecan, PLX: PLX4720, L-83455, Fano: Panohinouta, AZD: AZD6244,, Erlo: Erlotimb, Pact:: Pacliaael
Lapatinib, PD: FDD0325901, PF: PF2341066,
Topo :Tpootecan
(5
with Pu503j ansersc|E331
from
und unfitcckeiarrpls 1454
Iibeled Cirel
uriebshodsamglla
tegpiegaagles for celonceel
tortingsookk evaluciion (3689)
teitting camp
for eviluation 11.100
for wonerntttnii..om
ping comples
Audtumese 2200
for fecture fe selaction
(a) UKCF
(6)PBMC
[c] CCLE Lopatinib
Figure S.I: Illustration of dataset splits for (a) UKCF, (b) PBMC, and (c) CCLE Lapatinib datasets.
230
330
Beatures
789
X00
rcatures
features
features
UKCF dataset
(b) PBMC dataset
(c) CCLE dataset
Figure S.2: The correlation structure of (a) UKCF, (b) PBMC, and (c) CCLE datasets. For the PBMC dataset, the number of features are reduced after thresholding.
parameter control the sparsity of input features chosen among (0.001, 0.01. 0.1, 1. 10, 100}.
DUFSI3 DUFS is an unsupervised feature selection method that improves the Laplacian score (He et al., 2005) by replacing it with a gated variant computed on a subset of features (Linden-) baum et al.] 2020). We use the parameter-frre version of DUFS which construct the Laplacian matrix with 2-nearest neighbors and setting the kernel bandwidth as the median Euclidean dis- tance. Both labeled and unlabeled samples used for training.
C DETAILED DATA DESCRIPTIONS
In Section 5 of the manuscript, we evaluate SEFS and benchmarks with multiple healthcare datasets from three different sources, i.e., UKCF (clinical), CCLE (proteomics), and PBMC (tran- scriptomics). The summary statistics of these datasets are provided S.22
hhttp::://gtthub.comm//fff/
17

---

Published as a conference paper at ICLR 2022
Throughout the experiments on the real-world healtheare and omics datasets, we consider a practical scenario for feature selection under low labeled data regime where only a small number of labeled samples are available while having sufficient amount of unlabeled sample. To this goal, we struct datasets hased on the following steps: First, we obtain enough unlabeled samples by utilizing samples without labels that are readily present in the original data (e.g., UKCF dataset has censored patients and CCLE dataset has samples without drug response), treating a portion of labeled samples as unlabeled (e.g., UKCF and PBMC datasets), and by integrating samples from different sources (e.g., CCLE dataset). Second, we randomly split the labeled samples into training and testing sets. Here, both the training and the testing sets are used for evaluating discovered features from differ- ent feature selection methods. This assures that the discovered features generalize well to unseen samples. Third, we randomly select small subset of training samples for training different feature selection methods. The overall data construction steps are illustrated in Figure S.1
Correlation Structure. The correlation structure of the datasets are illustrated in Figure S.2 We construct the correlation matrix R as defined in Algorithm and 2 formally, each element of R -
is given Raj iCuil where Ckj Σ
It is worth highlighting that the PBMC dataset contains many features that have low correlation with other features; thus, after thresholding at 0.7, only 426 features with high correlation remain.
VCu--Ca
To quantify the amount of collinearity, we calculated the variance inflation factor (VIF), which is an index that measures how much the variance (the square of the estimate's 's standard deviation) of an estimated regression coefficient is increased because of collinearity. While there is not strict cut-off indicating collinearity, general rule of thumb is that VIFs exceeding 10 are signs of serious multicollinearity (Menard [2001] VVitinghoff et al., 2011] James al.) (013). We note this a conservative cut-off, with guidelines suggesting that VIIs greater than 5 Menard 2001 Vittinghoff ct al. 201T) indicate considerable collinearity. In Table S.3] we provide the VIFs of features in each dataset; here the VIFs for the PBMC dataset are not available due to the computational complexity. As seen in the table, there are strong signs of multicollincariy in both datasets with the VIFs for 23.5% of the features in the UKCF dataset and all the features in the CCLE dataset exceeding 10.
Table S.3: VIFs for the tested real-worke datasets.
Datasets
Mean
Min
Max
VIF >> 10
UKCF CCLE
11.33 80.81
1.04
391.84
23.50%
15.00
462.97
100%
D ADDITIONAL EXPERIMENTS
D.1 SYNTHETIC:
NOISY TWO-MOONS DATASET
Quantitative Results. Table S.4 shows the average TPRs for all benchmarks. As can be seen in the table, SEFS outperforms all benchmarks, with the majority of methods struggling to identify Feature in particular. This is due to Feature being more discriminative than Feature and therefore the noisy features that are correlated with Feature 2 are often selected. It is worth highlighting that L-Score, DUFS, and STG (SS) fail to discover discriminative features almost all cases due to the correlation structure of the data that cannot be addressed by the similarity metrics or reconstruction employed by these methods.
D.2 CLINICAL: UKCF DATASET
Quantitative Results. Figure S.3 displays features discovered with frequency > 0.4, where we consider a feature is discovered (for all three feature selection methods) if that feature is within the top 20 highest feature importance. It is immediately apparent that SEFS consistently identifies the same features, while Lasso and Tree typically select different features, demonstrating the robustness of our approach.
18

---

Published as a conference paper at ICLR 2022
Table S.4: The average TPRs on the Two-Moons dataset.
(ny 1y:20 20, m0=0000)
(nt (My=80, 80, - 1000)
Methods
Feature
Feature
Average
Feature
Feature
Average
Lasso
0.45
0.98
0.72
0.77
1.00
0.89
Tree
0.02
0.46
0.24
0.05
0.88
0.47
STG
0.14
0.96
0.55
0.21
1.00
0.61
DUFS
0.02
0.03
0.03
0.01
0.02
0.02
L-Score
0.00
0.00
0.00
0.00
0.00
STG (SS)
0.09
0.55
0.09
0.94
0.52
SETS (no SS)
0.68
0.99
0.84
0,98
1,00
0.99
SEFS
0.92
1.00
0.96
1.00
1.00
1.00
1.0
Lasso
Tree
Discovery Irequency
0.6
0.2
FEV pred (233)
FEV1 pred (yr2)
FEV1 pred (wrl)
FEV, (yr3)
FEvi % (9r2)
% (yrl)
Allele
Pi Allele
Chr PAE (yrd)
IV Abx Home (yr3)
Collstin (yr1)
Figure S.3:
Comparison discovered features based on Lasso, Tree, SEFS for the UKCF dataset with ny = 32.
Qualitative Results. Other than PI Allele and that are related pancreatic functions as de- scribed in Section 5.2, SEFS consistently discovered features related to the lung function scores (i.e., FEV, % pred and FEV1).
worth highlighting that, features related the lung function scores are frequently selected despite their relatively high correlation. We conjecture that the tra- jectory of the lung functions scores measured at different times (with the highest frequency of the latest measurement) plays an important role for predicting the respiratory failure supported in Tablor-Robinson et al. (2012) Adler Liou (2016).
D.3 PROTEOMICS: CCLE DATASET
compare the prediction performance of 5 and 10 discov- ered features (i.c., [S] = 5 and 10) for the 11 drugs AZD6244, Erlotinib, Irinotecan, L-685458, Lapatinib, PD-0325901, PF2341066 PLX4720. Paclitaxel, Panobinostat, and Topotccan reported in the manuscript. Despite the majority of unlabeled data originating from a different source, our method benefits from learning the underlying data structure from unlabeled samples. SEFS consis- tently displays improvements (except for Erlotinib with |S] 5 and 10, L-685458 with [S] 5, and PF2341066 with |S] and 10) performance for different drugs and varying numbers of discovered features, outperforming SEFS (no SS) in every experiment. While we would expect fur- ther gain from more similar unlabeled data, our results highlight the effectiveness of sulf-supervision when the unlabeled data is only partially related to the labeled samples.
Quantitative Results. In Table S.5
Qualitative Results. the main manuscript, we validated the features identified by SEFS for panobinostat in the scientific literature. In Table [S6 we provide details of the references for panobi- nostat and also provide supporting evidence for the selected features for lapatinib and irinotecan.
Lapatinib is drug used primarily to treat breast cancer, well other solid tumours. It is dual tyrosine kinase inhibitor which inhibits the epidermal growth factor receptor (EGFR) and human epidermal growth factor receptor 2 (HER2) receptors. In total, we found supporting literature for
19

---

Published as a conference paper at ICLR 2022
Table S.5: Comparison of the MSE (mean + 95%-CII given discovered features for the CCLE dataset. (Lower the better.)
Methods
AZD6244
Eriotinib
Irinotecan
1-485488
[S] =
[8]
[8] =
[8] = 10
[S] =
[S] 10
[8] 5
[S| = 10
Lasso
1.444:000
1.49410.00
0.22:100222
02
1.4
1.416:20.89
0.305:001
0:315=001
Thee
1.426:1:0:04
0.407:10002
1.385
1.340:400.0
0.293=0011
0.309=0011
BNNsel
1.394:00.04
1.112::0.04
0.400:10..01
0.413 0.01
0.36
1.35444...3
0.28::0001
0.305 0.01
STG
1.385 0.004
1.455
0.419:4-0.01
0.446=0.02
0.09
1.347+00.33
0.295=00.01
0.316=0001
DUPS
1427400.04
1.114-00001
0.420=100.01
0.M2=000
1.373+00.03
1.323-+-0.0
0.311=0001
0.33=001
S.Scrre
1.396-0000
1.800:10.04
0.453:1-.02
0.447=001
1.499.0.044
1.384410.04
0.298:0001
0.309=0001
(SS)
1.393-00.04
1.166:-0.04
0.411=40.001
0.444=0001
L.09
1.327440.04
0
0.295 0.01
SPPS SS)
1.37620004
1.099:0.004
0.408:10.001
0:400
1.4142003
1.335:20044
0.284:2011
0:312-001
SEPS
0.63
1403-10.03
0.404:10002
0.428:00002
0.38:0004
1.29110004
02822:001
0:307=001
Methods
Lapatialb
PD-032591
PF2341066
PL
[S]
[S]
[S]
[S]
[S]
[S]
[8]
[S] 10
0:45020001
0.450-10.02
2.216:2::05
0.
0.365:0.01
0.365:2001
0447-0022
0:420=002
Tree
0.11:200022
0.440-1:0:22
2.196:1:004
0.178=0005
0.34
0.370-10.00
0339:0002
0416=0002
BNNsel
0.457-00002
0.474:1:.002
2.176:20000
0.178=0005
0.04
0.363:10.01
0.397.0002
0.14=0002
STG
0.438:-0.02
0.464-1-0.02
2.210:4-0.06
0.179=00.06
0.350-0000
0.369
0.403=00001
0.412:0.02
DUPS
0.466200001
0.460:2:0001
2.186:20005
0.233-0004
0.359-20001
0.376:1-0:91
0414:0011
0.42--001
L-Score
0.499.0.01
0:001
2.455110.04
0.269=000
0.349-10.01
0.384:10001
0,22200011
0.429-0001
STG (SS)
0.435:20000
0.456:1:000
2.184:10.00
2.083=0.005
0.344:5000
0.373:1:0.01
0.409:0.02
0.410=0022
SEPS 88)
0:304-00:02
0.424100.02
2.171:10006
06
0:345:0.011
0.371:2:0011
0.39620001
0:40=2002
SEPS
0.392:20.02
0.4051:0.02
2.168:100005
2.06
0.344:2.01
0.360:10.01
0.396:0.01
0.396=00022
Methods
Paclitaxel
Panobinostat
Topotocan
[8]
[8]
[8]
[S]
[S]
[S]
Lasu
1.89=0003
1.899-00003
0.651+0000
0:33
1.533:00002
1.530:200.02
Tree
1633-2003
1.626::0:03
0.547:::.02
0.53
1.41
1.401
BNNsel
1.66620000
1.629:10.03
0.574:10.02
0.552_0002
1.05
1.415:10.95
STG
1.664:00003
1.60-110:04
0.573-1-0:02
0.539-10001
1.03--00003
1.425:2-0.03
DUPS
1.66:20.04
1.6252:0044
0.607:20002
0.575=0002
1.443-20.03
1,439-200003
L-Score
167552003
1699110.08
0.624110.01
0.5542001
1.501
1,491:10002
STG (8S)
1618220.0
16622::000
0.541::0.02
0.522:0.02
1.40820.03
1.393
SEPS 55)
1.61420.04
1.041:0003
0.522:1:.02
0:50620001
1.416:00033
1.407-2:0.88
SEPS
1.05
1.587:1:0.0
0.512:10.02
0.49
1.405
1.309-10.03
of the top 10 ranked features selected by SEFS (Table S.6b). with five such features not proposed by SEFS (no SS). The other two features selected by SEFS are both established cancer biomarkers with evidence of their importance in breast cancer, but we did not find literature specifically relating them to Lapatinib.
Irinotecan is a chemotherapy agent used primarily to treat colon cancer in addition to small cell lung cancer. All of the top 10 features identified by SEFS had supporting scientific literature for their impact on the mechanism and effectiveness of irintecan or colon cancer (Table S.66). of which six were not proposed by SEFS (no SS).
D.4 TRANSCRIPTOMICS: PBMC DATASET
Qualitative Results. In Figure S.4 we compare the frequency of the discovered features based on SEFS with that of the discovered features based on Lasso and Tree. The figure displays only the features that were selected with frequency equal to greater than 0.4 by one of the methods. is worth highlighting that SEFS (no SS) does not discover any feature with frequency equal to or greater than 0.4, which shows the importance of the Self-Supervision Phase. The lack of consistency of discovering features suggests that Lasso and Tree could be overfit to spurious relations in different splits of the data. Only 2 features are discovered at least 40% of the time by either Lasso or Tree (one feature each), compared to 9 for SEFS.
SEFS almost always discovers S100A4 and IL32 as relevant features. Both of these genes have been shown to be only expressed, or preferentially expressed, in certain types of T-cells S100A4: Weatherly et al.] (2015) and IL32: Goda et al. (006)) which strongly validates the selection of these features. These features are also the most frequent features discovered by Tree, but with much lower frequency, while Lasso does not discover either feature, despite their high relevance. Similarly, while DUSPI (also known as MKP-/) is selected by Lasso, Tree, and SEFS, only our
20

---

Published as a conference paper at ICLR 2022
Table S.6: Frequently discovered proteins (features) by SEFS for the CCLE dataset. Features in blue were not proposed by SEFS (no SS).
(a) Panobinostat
Rank
Proteins
Ref.
Caveolin-1
(Deb et al 2016
YAP-pS122
Heinemann et al 2015
PRAS40-p1246
Gallgger ct a. 2018
VHIL
Kilac el al 201
Src-pY416
Kostynnak sl. 2002
TAZ
Loc ct al
14-3-3-B
Wang 2000
Fibronectin
GSK3-pS9
Rahmani al
2014
10
MSH2
Li al
2020)
(b) Lapatinib
(c) Irinotecan
Rank
Proteins
Ref.
Rank
Proteins
Ref.
Caveolin-1
Qian el
2019
Lck
larashima et
2001
2
HER2-pY1228
Medina &
2008
Stathmin
Peng
2010
5
VHI.
3
(8-Catemin
Sarlo el
2010
6
mTOR
Gayle al.
2012
4
Bel-2
Gupta
2007
7
53BP1
Ll a Gahe
2012
5
Ste.pY416
Petipree & Larsen
2013
8
N-Ras
2019
6
Smad4
Wong cl al.
2020
9
14-3-3_8
7
YAP-pS127
Noguchi ct 2018
8
Claudin-7
Constantinou el al. 2018
8
Bel-xL
(Ler el ml 2019)
9
Rah25
(Cheng ct al. 2013
9
P13K-p85
Köizumi ct al.
2005
10
GSK3.pS9
Duda et al. 2020
E-Cadherin
Bendardal et
2019
method discovers this feature in the majority of the experiments, despite its importance for T-cell activation and function (Zhang et al. 2009).
In addition, we also find evidence the importance of JUN Riera-Sans Behrens) 2007 SH3BGRL3 (Deng et al. 2006), CCR7 (Schneider et al. (007), and ribosomal genes RPST3, RPL21, and RPS3A (Procaccini et al. 2016).
SEFS
Lasso
Discovery Irequency
0.6
0.5
0.4
0.3
0.0
5100A4
DUSP1
1.32
AP513
RPL21 SHIBGHL3 RPSJA
CCR/
Expression
Figure S.4: Comparison of discovered features using Lasso, Tree, and SEFS for the PBMC dataset with 20 (i.e., 10 labeled samples for each label).
D.5 CORRELATED VS INDEPENDENT GATE VECTOR GENERATION IN FEATURE SELECTION
In this subsection, we investigate how the correlated gate vectors using a multivariate Bernoulli dis- tribution in SEFS improves the feature selection process over independently generated gate vectors. To this goal, in Figure S.5 we provide trajectories of selection probabilities for the ground-truth fea- ture 22 and its that the results are generated only utilizing the Supervision Phase. As claimed in the manuscript, the correlated gate vectors encourage the network to select only the most relevant feature by making
noisy features with respect to the number of training iterations. Note
21

---

Published as a conference paper at ICLR 2022
Selection Probabiitty
0.6
00
1000 2000 3000
4000 5000 Iteration
6000
7000 0000 9006
1000 2009 3000
6000
9000 5000
5000
7000 8000
Iteration
(a) Correlated gate vectors
(b) Independent gate
Figure S.5: The trajectory of 's with respect to the number of iterations for the ground-truth relevant feature z2 (in the blue line) and its block-corrllated noisy features (in other colors) using (a) correlated gate vectors using multivariate Bernoulli distribution and (b) independent gate vectors using a independent Bernoulli distribution.
10
Ava. Discoverd Features
100
87.65
70.29
Postive Rate
0.8
0.6
1.52
0.62
(a) vs true positive rate
(b) | vs No. of Discovered Features
Figure S.6: Comparison of (a) the TPR performance and (b) the number of discovered features with Tk > 0.5, with respect to different values of B8 respectively.
multiple correlated features compete against each other. This is highlighted in Figure However, when independent gate vectors used, it is less likely make correlated features to compete each other and increase the selection probabilities of the correlated noisy features because these features still informative about the target compared to the augmented features i.e., (Iz, ,10) and their correlcode features that are purely noisy.
D.6 SENSITIVITY ANALYSIS ON COEFFICIENT 8
In this subsection, we provide sensitivity analysis using the Block-Structured Noisy Two-Moons dataset to the effect of coefficient 3 that controls the sparsity of the selected features. Figure S.6
shows (a) the true discovery rate as defined in Section 5. where we define # (or x2) as correctly discovered if and only if (or 222) has the first the second highest feature importance and (b) the number of features discovered whose selection probability is higher than 0.5 (i.e., Tr > 0.5).
Here, multiple instances of SEFSwith different I's trained utilizing n4 20 labeled samples (i.e., 10 labeled samples for each label) and pre-trained with n., = 1000. The results are averaged 100 iterations described in Section 5.1 of the manuscript.
Figure (.6(a) a) shows that SEFS trained with setting 8 = 5, 10 provides the highest TPR. However, as can be seen Figure (.)(b). setting the coefficient too small (here, B S. 1) will end up discovering too many irrelevant features as if they were relevant i.c., having many features whose selection probability above certain level and will mislead the discovery relevant features. Contrarily, setting the coefficient too high (here, 3 10) can restrict the selection process and may fail to identify relevant features.
22

---

Published as a conference paper at ICLR 2022
APPENDIX REFERENCES
Frederick R. Adler and Theodore G. Liou. The dynamics of disease progression in cystic fibrosis. PLOS ONE, 11(6):1-17, 06 2016.
Riyad Bendardaf, Fatemeh Saheb Sharif-Askari, Narjes Saheb Sharif-Askari, Kari Syrjänen, and Seppo Pyrhönen. Cytoplasmic e-cadherin expression is associated with higher tumour level of vegfa, lower response rate irinotecan-based treatment and poorer prognosis in patients with metastatic colorectal cancer. Anticuncer Research, 39(4) 2019. ISSN 0250-7005. doi: 10.218873/mticamres.. 13305. URLbttss::/ 4/1953
Kw Cheng, R. Agarwal, S. Mitra, and Gb Mills. Rab25 small gtpase mediates secretion of tumor necrosis factor receptor superfamily member 1lb (osteoprotegerin) protecting cancer cells from effects trail. Journal of genetic syndromes & gene therapy, 4:1000153, 2013. doi: 10.4172/ 1157-7412.1000153.
Chloe Constantinou, Savvas Papadopoulos, Eirini Karyda, Athamasios Alexopoulos, Niki Agnanti, Anna Batistatou, and Haris Harisis. Expression and clinical significance of claudin-7, pdl-I, pten, c-kit, c-met, c-myc, alk, ck5/6, ck17, p53, egfr, ki67, p63 in triple-negative breast cancer-a single centre prospective observational study. In vivo (Athens, Greece), 32(2):303-31 311, 2018
Moonmoon Deb, Dipta Sengupta, Swayamsiddha Kar, Sandip Kumar Rath, Subhendu Roy, Goutam Das, and Samir Kumar Patra. Epigenetic drift towards histone modifications regulates cavl gene expression in colon cancer. Gene, 581(1):75 84, 2016. doi: https://doi.org// http://00011//.eeeeee20000 01.029.
Y. J. Deng, Z. X. Huang, C. J. Zhou, J. W. Wang, You, Q. Song, M. M. Xiang. B. Y. Zhong, and F. Hao. Gene profiling involved in immature cd4+ lymphocyte responsible for systemic lupus erythematosus. Molecular Immunology, 13(9)::497-150)) 2006.
Przemyslaw Duda, Shaw M. Akula, Stephen L. Abrams, Linda S. Steelman, Alberto M. Martelli, Lucio Cocco, Stefano Ratti, Saverio Candido, Massimo Libra, Giuseppe Montalto, Melchiorre Cervello, Agnieszka Gizak, Dariusz Rakus, and James A. McCubrey. Targeting gsk3 and associ- ated signaling pathways involved in cancer. Cells, 95):11110, Apr 2020. ISSN 2073-4409. doi: 10.33900/ell:905111000
Mirco Galiè. Ras as supporting actor in breast cancer. Frontiers in oncology, 9:1 9:1199-1199 1199, Nov 2019. ISSN 2234-943X. doi: 2338%//onc.221..00 199.
Stuart J Gallagher, Dilini Gunatilake, Kimberley A Beaumont, Danae M Sharp, Jessamy C Tifffen, Anja Heinemann, Wolfgang Weninger, Nikolas K Haass, James Wilmott, Jason Madore, Pe- ter M Ferguson, Helen Rizos, and Peter Hersey. Hdac inhibitors restore braf-inhibitor sensitivity by altering pi3k and survival signalling in a subset melanoma. International Journal of Cancer, doi: https://doi.ogg oogg 10. 101 cellsi...1s1..000......
142(9))1926-1999)) 1937, 2018.
Sylvia S. Gayle, Samuel I.. M. Amold, Ruth M. O' O'Regan, and Rita Nahta. Pharmacologic inhibition of mtor improves lapatinib sensitivity in pere-overexpressing breast cancer cells with primary trastuzumab resistance. Anti-cancer agents in medicinal chemistry. 12(2):151-162, Feb 2012.
ISSN 1875-5992, doi: 10.2174//77122011799955000
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine Learning, 63(1):3-42, 2006.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AIS- TATS 2010), 2010.
Chiho Goda, Taisuke Kanaji, Sachiko Kanaji, Go Tanaka, Kazuhiko Arima, Shigeaki Ohno, and Kenji Izuhara. Involvement of IL-32 in activation -nwraatioo-nnuueddd cell death in cells. International Immunology, 18(2):233-240, 2006. doi: 10. 10000///mmimm//xxh39.
23

---

Matko Kalac, Luigi Scotto, Enrica Marchi, Jennifer Amengual, Venkatraman E. Seshan, Govind Bhagat, Netha Ulahannan, Violetta V. Leshchenko, Alexis M. Temkin, Samir Parekh, Benjamin Tycko, and Owen A. O'Connor. HDAC inhibitors and decitabine are highly synergistic and associated with unique gene-expression and epigenetic profiles in models of DLBCL. Blood, URL Published as a conference paper at ICLR 2022
Vinay Gupta, Yuzhuang S Su, Christian G Samuelson, Leonard F Liebes, Mare C Chamber- lain, Florence M Hofman, Axel H Schönthal, and Thomas C Chen. Irinotecan: a potential new chemotherapeutic agent for atypical malignant meningiomas. Journal of neurosurgery, 106(3):455-4 462, March 2007. ISSN 0022-3085. doi: 10.3111f1s.2007.16334555 URL.
https://dof wor.//oor////..o 3171/ jns 2007 www.://.............. 1.4.3.455
Nanae Harashima, Koji Tanaka, Teruo Sasatomi, Kanako Shimizu, Yoshiaki Miyagi, Akira Yamada, Mayumi Tamura, Hideaki Yamana, Kyogo Itoh, and Shigeki Shichijo. Recognition of the lek tyrosine kinase as tumor antigen by cytotoxie t lymphocytes of cancer patients with distant metastases. European Journal of Immunology, 31(2):323 332, 2001. doi: https://oi.org///
X. He, D. Cai, and P. Niyogi. Laplacian score for feature selection. In Proceedings of the 18th Conference on Neural Information Processing Systems (NIPS 2005), 2005.
Anja Heinemann, Carleen Cullinane, Ricardo De Paoli-Iseppi, James S. Wilmott, Dilini Gunatilake, Jason Madore, Dario Strbenac, Jean Y. Yang, Kavitha Gowrishankar, Jessamy C. Tiffen, Rab K. Prinjha, Nicholas Smithers, Grant A. McArthur, Peter Hersey, and Stuart J. Gallagher. Combining bet and hdac inhibitors synergistically induces apoptosis of melanoma and suppresses akt and yap signaling. Oncotarget, 6(25)221507-2152,,, Aug 2015. ISSN 1949-2553. URL https: 7/pubmed. ncbi nlm. nih gcw/bc687/22
G. James, D. Witten, T. Hastie, and R. Tibshirani R. An Introduction to Statistical Learning: With Applications in Springer, 2013.
118(20)5506-55166 11 2011. ISSN 0006-4971. doi:
https: org. 10. w111/11oooooo 2011 02 336891
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), 2015.
Naoki Koizumi, Etsuro Hatano, Takashi Nitta, Masaharu Tada, Nobuko Harada, Kojiro Taura, Iwao Ikai, and Yasuyuki Shimahara. Blocking of pi3k/akt pathway enhances apoptosis induced by sn- 38, an active form of cpt- 11, in human hepatoma cells. Iut J Oncol, 26(5):1301-130 May 2005. doi: 10.3892/1jo.26.5..333
URL htthttps: /ow. org/ 10. 3892 /ttp9//ijj... 26 5.101
Calley L. Kostyniuk, Scott M. Dehm, Danielle Batten, and Keith Bonham. The ubiquitous and tissue specific promoters of the human sre gene repressed by inhibitors of histone deacetylases Oncogene, 21(41) Sep 2002. ISSN 1476-5594, doi: 10. 1033///..oc..1257778 URL https::/ddo..rrg/ 1038 s 1205787
Boah Lee, Jeong A. Min, Abdullateef Nashed, Sang-Ok Lee, Jae Cheal Yoo, Seung- Wook Chi, and Gwan-Su Yi. A novel mechanism of irinotecan targeting mdm2 and bel-xl. Biochemical and Biophysical Research Communications, 514(2):518-522 2019. ISSN 0006-291X. doi: hwwp://wwww.eeo.......o....o....
URL https: wtwp://wwwww/.... sci enceddieeddi rect.ec. .omm
Wen-Ying Lee, Pin-Cyuan Chen, Wen-Shin Wu, Han-Chung Wu, Chun-Hsin Lan, Yen-Hua Huang, Chia-Hsiung Cheng, Ku-Chung Chen, and Cheng Wei Lin. Panobinostat sensitizes kras-mutant non-small-cell lung cancer to gefitinib by targeting taz. International Journal of Cancer, 141(9):
1002/jc..0888 RRLhhttps://oniinellll 1921 doi: 010.0..100//031 30888
www.ey...omoo/////// http://///aa////////
Shiqin Li, Bingbing Shi. Xinli Liu, and Han Xiang An. Acetylation and deacetylation of DNA repair proteins in cancers. Frontiers in Oncology, 10, Oct 2020. doi: 10.3389/foom..220053330000
U .org 33899/1..333/// fenc. 2020. 573502
24

---

Published as a conference paper at ICLR 2022
Xiaoyan Li, Bing Xu, Meena S. Moran, Yuhan Zhao, Peng Su, Bruce G. Haffty, Changshun Shao, and Qifeng Yang. 53bpl functions as a tumor suppressor in breast cancer via the inhibition of nf-xb through mir- 146a. Carcinogenesis, 33(12):5533-2600 Dec 2012. ISSN 0143-3334. doi: 10. 10.10999anccinbbg228...
/ttp:s://dooooee/ oog/ 100.0..009300 1093/ 1t99///aaiing http///accnnnn///////////
Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, 113(523)9555-972 2018.
O. Lindenbaum, U. Shaham, J. Svirsky, E. Peterfreund, and Y. Kluger. Differentiable unsupervised feature selection based on gated laplacian. arXiv preprint arXiv:2007.04725 2020.
C. J. Maddison, A. Mnih, and Y. W. Tch. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the 5th International Conference on Learning Rep- resentations (ICLR 2017), 2017.
Patrick J. Medina and Susan Goodin. Lapatinib: A dual inhibitor of human epidermal growth factor receptor tyrosine kinases. Clinical Therapeutics, 30(8):1426-1447, 2008
S. Menard. Applied Logistic Regression Analysis. SAGE Publications, Inc, 2001.
Satoshi Noguchi, Akira Saito, and Takahide Nagase. Yap/taz signaling molecular link between fibrosis and cancer. International journal of molecular sciences, 19(1 19(11):3674, Nov 2018. ISSN 1422-0067. doi: 10.3390/fjm191113 19113344
Xing-Chen Chen Peng, Feng-Ming Gong, Meng Wei, Xi Chen, Ye Chen, Ke Cheng, Feng Gao, Feng Xu, Feng Bi, and Yin Liu. Proteomic analysis of cell lines to identify the irinotecan resistance proteins. Journal of Biosciences, 35(4):557-564, 2010.
Amelie Petitprez and Annette K. Larsen. Irinotecan resistance is accompanied by upregulation of egfr and src signaling in human cancer models. Current Pharmaceutical Design, 1955:958-964, 2013. URL http: /wwww wwww.eurres eurekase ect com/ ccom/node//e www.oooe/lss0 005///1555///// //rtiice
Claudio Procaccini, Fortunata Carbone, Dario Di Silvestre, Francesca Brambilla, Veronica De Rosa, Mario Galgani, Deriggio Faicchia, Gianni Marone, Donatella Tramontano, Marco Corona, Carlo Alviggi, Antonio Porcellini, Antonio La Cava, Pierluigi Mauri, and Giuseppe Matarese. The proteomic landscape of human ex vivo regulatory and conventional cells reveals spe- cific metabolic requirements. Immunity, 44(2):406 421, Feb 2016. ISSN 1097-4180. doi: 10.1016/j.immml 1mmmum.2006.01.0000
Xian-Ling Qian, Yi-Hang Pan, Qi- Yuan Huang, Yu-Bo Shi, Qing- Yun Huang, Zhen-Zhen Hu, and Li-Xia Xiong. Caveolin-I: multifaceted driver of breast cancer progression and its application in clinical treatment. OncoTargets and therapy, 12: 12:1539-1552, Feb 2019. ISSN 1178-6930. doi: 1111117/
Mohamed Rahmani, Mandy Mayo Aust, Elisa Benson, LaShanale Wallace, Jonathan Fried- berg, and Steven Grant. Pi3k/mtor inhibition markedly potentiates hdac inhibitor activity in nhl cells through bim- and mcl-1 -dep--deeen mechanisms in vitro and in vivo. Clinical Can-
cer Research. 484 4860, 2014. doi:
14-0044. URL
//clincancerresco aac.
aacr jowroool..oooooo.. htt.://ww.aaraaa...rooooooaara/a
Lluís Riera-Sans and Axel Behrens. Regulation of a cell development by the activator protein transcription factor c-jun. The Journal of Immunology, 17809):560-57700 2007.
Maher S. Saifo, Donald R. Rempinski Jr., Youcef M. Rustum, and Rami G. Azrak. Targeting the oncogenic protein beta-catenin to enhance chemotherapy outcome against solid human cancers. Molecular cancer, 9:310-310, Dec 2010. doi: 10.1 116///76-4598-999
Martin A. Schneider, Josef G. Meingassner, Martin Lipp, Henrietta D. Moore, and Antal Rot. Cer7 is required for the in vivo function of cd4+ ed25+ regulatory t cells. Journal of Experimental Medicine, 204(4):735-745 Mar 2007. ISSN 0022-1007. doi: 10. N0.10044/jm.2000 20161e.m...00555.... URL
https:://doi.... /ttp::////oii.o/10....1..1. 1084/ 100.0jem.....mm.m jem. 20061405
Razich Sheikhpour, Mehdi Agha Saramm, Sajjad Gharaghani, and Mohammad Ali Zare Chahooki.
A survey semi-supervised feature selection methods. Pattern Recognition, 64:141-15 141-158. 2017.
25

---

Published as a conference paper at ICLR 2022
David Etylor-RRbinsee Margaret Whitehead, Finn Diderichsen, Hanne Vebert Olesen, Tania Pressler, Rosalind L Smyth, and Peter Diggle. Understanding the natural progression in %fevl decline in patients with eystic fibrosis: longitudinal study. Thorax, 67(10):860-80)) 866, 2012.
Robert Tibshirani. Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
E. Vitinghoff, D. V. Glidden, S. C. Shiboski, and C. E. McCulloch. Regression Methods in Bio- statistics: Linear, Logistic, Survival, and Repeated Measures Models. 2011.
A. H. Wang, M. J. Kruhlak, Wu, N. R. Bertos, M. Vezmar, I. Posner, D. Bazett-Jones, and X. J. Yang. Regulation of histone deacetylase 4 by binding of 14-3-3 proteins. Molecular and cellular biology, 20(18))6904-639 6912, Sep 2000.
Xi Wang and Junming Yin. Relaxed multivariate bemoulli distribution and its applications to deep generative models. In Proceedings the 26th Conference on Uncertainty in Artificial Intelligence (UAI 20), pp. 425-432, 2020.
Kathleen Weatherly, Marie Bettonville, David Torres, Arnaud Kohler, Stanislas Goriely, and Michel Braun. Functional profile of s100a4-deficient cells. Immunity, Inflammation and Disease, 3(4):431-444, 2015. doi: https://doi.org/10.100000
Chen Khuan Wong, Arthur W. Lambert, Sait Ozturk, Panagiotis Papageorgis, Delia Lopez, Ning Shen, Zaina Sen, Hamid M. Abdolmaleky, Balázs Györffy, Hui Feng, and Sam Thiagalingam. Targeting rictor sensitizes smad4-negative colon cancer irinotecan. Molecular Cancer Re- search, 18(3):41-422, 2020. ISSN 1541-7786. doi: 10. 10.1118//541-7788.MME. 19-CR-9 URL https: /er.//rr.////e/ aacr www.crr.accooorraaaaaoaaa journals org. hrt/ wttp://oooottt//////11..
Y. Yamada, O. Lindenbaum, Negahban, and Kluger. Feature selection using stochastic gates. In Proceedings of the 37th International Conference on Machine Learning (ICMI. 2020), 2020.
Yongliang Zhang, Joseph M Reynolds, Seon Hee Chang, Natalia Martin Martin-Orozco, Yeonseok Chung, Roza Nurieva, and Chen Dong. Mkp-1 is necessary for cell activation and function. Journal of Biological Chemistry, 284(45):00115-30824 2009.
26


Reviews:
[{"review_id": "zg85FDH__Cl", "reviewer": "Reviewer_gJvk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated and the method is clearly presented with all its technique points delivered. Several concerns during my reading are well addressed: the Bernoulli distribution reparameterization, the lasso regularizer term's relaxation, the ablation study on SS phase, the performance gain over other baselines etc.\n\n\nIn section 3, the formulation around Eq1, Eq2 may have contradicting ideas on how to set values for features not selected. It reads around Eq 1, features not selected will take an out-of-distribution value \"*\", while in Eq 2, features not selected will take their mean values. Do authors have any comments or corrections here? Readers may be confused by the two formulations. \n\n\nAnother similar tabular data prediction task is CTR prediction for ads recommendation, this method may have potential to be applied there.", "summary_of_the_paper": "This paper presents a novel feature selection method for tabular data prediction. It tackles two challenges by respective novel designs, i.e. labeled data scarcity issue by an unsupervised self-supervision phase, and feature correlation issue by multivariate Bernoulli gate vector with learnable correlation matrix. The experiments on one synthetic and two medicine/biology datasets demonstrate effectiveness.", "summary_of_the_review": "This paper is impressive. Its technique is novel and clear. Experiment is convincing and solid. A clear accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cwG5NT2WYOC", "reviewer": "Reviewer_uaxu", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments: \nI think the dataset descriptions could have been earlier, it’s helpful to read them prior to some of the methods descriptions to better understand the approach. \n\nGiven the focus on genetic data in the introduction it may be helpful to cite RELIEF-based approaches. \n\nThe introduction mentions WGS and the high dimensionality inherent to it but the datasets used are several orders of magnitude less features. Either the introduction should be updated to better reflect the datasets used (e.g., not full sequence data) or the potential application to genotypes (~2M features) or sequencing should be explored. Importantly, it seems this would require a streaming approach or other algorithmic changes if the data did not fit into GPU memory. In any event, It would be helpful to see the authors thoughts of about whether these methods could be combined using dimensionality reduction as pre- or post-processing in addition to, instead of as a replacement for their approach (as in Table 1). \n\nThe name - SEFS (no SS) isn't consistent from table 1. Why isn't SEFS (no SS) shown in table 1? My understanding is that it is neither the autoencoder or independent Bernoulli approach, but I was confused about why it couldn't be included in the initial experiment. \n\nI was confused about why Table 1 does not include any of the benchmark approaches from Table S4? At first I thought the description of performance comparison in terms of both TPR and downstream predictive performance was incorrect because I did not see the supplemental table. The discussion about traditional methods performing poorly on feature 1 seems important enough to be included in the core manuscript as it provides intuition about the advantages of the author's approach. \n\nMinor comments:\nFigure 4's legend is a little confusing - lower the better, but the rank is lowest at the top of the figure. I believe this could be easily clarified. \n\nLanguage should be clarified / copy-edited prior to publication \nJust as a single example - Changing via to by simplifies this sentence in abstract: \nFirst, we pre-train the network using unlabeled samples within a self-supervised learning framework via solving pretext tasks that require the network to learn informative representations from partial feature sets. \n\nSupplement - SEFS outperforms all benchmarks - benchmarks is misspelled\n", "summary_of_the_paper": "This work aims to use self-supervision to identify the most useful features for downstream tasks particularly in the context of correlated features. This is an important aspect that is lacking from many feature selection approaches commonly used within the highly correlated datasets of healthcare. This approach - using both feature vector reconstruction and gate vector estimation appears to provide better results for both true structure estimation in a  dataset with known truth as well as downstream predictive performance in real world data sets. ", "summary_of_the_review": "I found this work to be focused on an important problem as well as technically interesting and correct. I believe the authors could substantially improve the presentation of the work (e.g., condensing intro sections, reworking the order and going deeper into results - both quantitative presentation as well as the interpretation in the discussion). I believe the authors could do this during the camera-ready phase and therefore recommend this work be accepted with this feedback prior to publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-wLgY1vBI4S", "reviewer": "Reviewer_K8kD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Equation (6) was borrowed from previous work, which makes me wonder which part of Section 4.1 is the novel contribution from this work. In addition, the last paragraph of Section 4.1 where two advantages are listed is quite hard to follow.\n\n2. Perhaps I missed this, how does one compute the correlation structure R based on the the inputs? It is stated as `pre-specified`, is it dataset or problem dependent?\n\n3. Regarding the experiments, most of the baselines are rather weak models. Could you elaborate among previous work which are those that also used neural networks? \n\n4. Overall what will the model behave if none of the features are dispensable? How does it scale with the dimensionality of the data?", "summary_of_the_paper": "This work presents some additional tweaks and tricks in using neural networks for feature selection. The main contributions include an inclusion of a parameterized masking function that selects the feature and a self-supervised component for pretraining. The overall proposal makes sense and there are strong experimental results that justifies the merits. ", "summary_of_the_review": "The experiments are rather comprehensive and illustrative. I look forward to get some feedback from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a feature selection method to identify features for downstream supervised tasks, focused on addressing challenges with sample scarcity and feature correlations.  The proposed approach is highly motivating in biological and medical applications.  Reviewers pointed out various strengths including potential high impacts in biomedical applications, technical novelty and significance, and comprehensive and illustrative experiments.  The authors adequately addressed major concerns raised by reviewers.
Decision: Accept (Spotlight)