Feature Selection in the Contrastive Analysis Setting
Authors: Ethan Weinberger
Ian Connick Covert
Su-In Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: Feature selection
contrastive analysis
computational biology

Abstract: The goal of unsupervised feature selection is to select a small number of informative features for use in unknown downstream tasks. Here the definition of ``informative'' is subjective and dependent on the specifics of a given problem domain. In the contrastive analysis (CA) setting, machine learning practitioners are specifically interested in discovering patterns that are enriched in a target dataset as compared to a background dataset generated from sources of variation irrelevant to the task at hand. For example, a biomedical data analyst may wish to find a small set of genes to use as a proxy for variations in genomic data only present among patients with a given disease as opposed to healthy control subjects. However, as of yet the problem of unsupervised feature selection in the CA setting has received little attention from the machine learning community. In this work we present CFS (Contrastive Feature Selection), a method for performing feature selection in the CA setting. We experiment with multiple variations of our method on a semi-synthetic dataset and four real-world biomedical datasets, and we find that it consistently outperforms previous state-of-the-art methods designed for standard unsupervised feature selection scenarios. 

Under review as a conference paper at ICLR 2022
FEATURE SELECTION IN THE CONTRASTIVE ANALY- SIS SETTING
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of unsupervised feature selection is to select small number of informa- tive features for use unknown downstream tasks. Here the definition of "infor- mative" is subjective and dependent on the specifics of a given problem domain. In the contrastive analysis (CA) setting, machine learning practitioners are specif- ically interested in discovering patterns that enriched in target dataset compared to a background dataset generated from sources of variation irrelevant the task at hand. For example, biomedical data analyst may wish to find small set of genes to use as a proxy for variations in genomic data only present among patients with given disease opposed to healthy control subjects. How- ever, as of yet the problem of feature selection in the CA setting has received little attention from the machine learning community. In this work present CFS (Contrastive Feature Selection), method for performing feature selection in the CA setting. We experiment with multiple variations our method on a semi-synthetic dataset and four real-world biomedical datasets, and we find that it consistently outperforms previous state-of-the-ee methods designed for standard unsupervised feature selection scenarios.
INTRODUCTION
In many scientific domains, it is increasingly common to see high dimensional datasets consisting of observations with many features. However, not all features are created equal: oftentimes, noisy information-pont features may obscure the natural structures underlying dataset and impair the performance of machine learning algorithms on downstream tasks. such, it is often desired to select small number of informative features and consider only those features when performing future analyses. For example, biologists may seek to measure the expression levels of only a small number of informative genes and then use these measurements in variety of future prediction tasks, such as disease subtype prediction or cell type classification. Measuring only subset of features may also yield number of other benefits, such reduced experimental costs, enhanced interpretability, and better generalization of models trained to perform downstream tasks.
While feature selection is relatively straightforward when prior knowledge the structure of the data is available (e.g. class labels in supervised setting), such information is often not available a priori and thus unsupervised methods must be used. In the unsupervised setting, feature selection algorithms seek to select features that are useful for arbitrary downstream tasks. A diverse array of prior work exists on unsupervised feature selection, with different methods taking different ap- proaches to quantify feature quality. For example, some methods are designed to select features that provide good clustering results (Mitra et al. [002 Lu et al.) 2007), while others are designed to pick
features that preserve local structures in the data (y & Brodley 2004 Boutsidis et al. 2009 Cai et al. 2010).
In many domains, data analysts are specifically interested in patterns that are enriched in one dataset, referred to as the target, as compared a second related dataset, referred to as the background. Such target and background dataset pairs arise naturally in many settings; data from healthy controls ver- sus a diseased population, from pre-intervention and post-intervention groups, or signal-free versus signal-contaiingg recordings all form natural background and target pairs (Abid et al.) 2018). In each these scenarios, the target dataset likely shares some set of uninteresting nuisance variations with the background dataset, such population level variations sensor noise.

---

Under review as a conference paper at ICLR 2022
Target Images
Background Images
Concrete Autoencoder
CFS (Pretrained)
2
3
2
6
9
(a)
bb)
(c)
(d)
Figure 1: Visualizing CFS applied to the Grassy MNIST dataset select 20 features. (a): This semi-synthetic target dataset was constructed by superimposing handwritten digits from the MNIST dataset onto images of grass from ImageNet. Our goal here is to select features that allow us to classify images by digit. (b): For our background dataset, we used separate set of images of grass. (c, d): Features selected (yellow) by a concrete autoencoder (c), a state of the art feature selection method, and by a variant of our method CFS (d). Features selected by CFS are concentrated towards the center of the images, where our superimposed digits lie. On the other hand, features selected by the concrete autoencoder more diffuse throughout the image.
Unfortunately, when these nuisance variations comprise the majority of the overall variance in a dataset, standard feature selection methods may not select the features that best reflect the variations of interest. For example, suppose we wanted to collect gene expression measurements from cancer patients for only a subset of genes better understand cancer-related differences between patients, such as differences between molecular subtypes of cancer. Understanding such differences is of great importance to the cancer research community, as intra-cancer variations are closely related to disease progression and treatment response (Stanta & Bonin 2018). With knowledge on different varieties of cancer rapidly evolving and new treatments consistently being released at a fast pace (Kinch] 2014), it is unlikely that would have access to single set of informative labels that would enable us to use supervised techniques for this task. Unfortunately, selecting features using fully unsupervised methods is also not straightforward, as the variations in this data be broken down into two components: those shared across healthy and cancerous tissue (e.g. variations due to demographic factors), and those only present in cancerous tissue. If the cancer-specific variations are subtle compared to the shared variations, standard unsupervised feature selection algorithms may not select a suitable set of genes for our analyses.
Isolating these salient variations present only in a target dataset is the subject of contrastive analysis (CA: Zou et al. (2013)) and a number of algorithms have been proposed that extend unsupervised learning methods to the CA setting. For example, Abid et al. (2018) extended principal component analysis to the CA setting, and many recent works (Li et al., 2020; Severson et al., 2019, Abid & Zou 2019: Ruiz et al. 2019 Jones et al. (021)) have focused on developing methods for learning contrastive latent variable models. However, feature selection in the CA setting has not been studied in depth by previous work.
The contributions of this paper are the following:
1. We introduce the problem of feature selection in the CA setting (Section 2 We do so by connecting feature selection to the specific data generating process assumed in CA.
2. We propose CFS (Contrastive Feature Selection), a method for feature selection that specifi- cally selects features useful for downstream CA tasks (Section f. At high level, CFS aims to select set of features that best reflect variations enriched in a target dataset but which are not present in the background dataset.
3. We validate our approach through extensive experiments (Section 5 We begin by apply- ing CFS to semi-synthetic dataset to better understand when it may succeed as well as potential failure cases. We then apply CFS to four real-world biomedical datasets and find that it consistently outperforms standard fully unsupervised feature selection algorithms.
2

---

Under review as a conference paper at ICLR 2022
Target
Background
Zj
s'
bj
1
Data generating processes for target and background samples. Background latent variables z are common to the two processes. A second set of salient latent variables are used to generate target points, while these variables (and not the background variables) are used to generate labels for target points. Observed values shaded, and square nodes represent constant values.
Figure 2:
fixed for background points. The salient variables
PROBLEM FORMULATION
We now describe the problem of feature selection in the CA setting. Recall that the goal of CA is to isolate variations only present in a target dataset from those present in both the target and background datasets. To capture this idea, we assume
data be modeled according to the following process: let X = {x,) E Rnxd denote our target dataset consisting of samples of interest. We assume that our æ; are drawn i.dd. from a random process that depends on two sets of latent variables 2; and 8,, We use 8 to refer to variables that are salient in our analysis, while denotes background variables that are not relevant to us. Our observed æ; are then drawn from function of our two sets of latent variables, i.e.:
p( st).
We also assume that each data point æ; is associated with labels yi drawn from another function that depends on 8, i.e.:
y: ~
That is, our labels depend solely on the salient latent variables, and not on the background variables. Our goal is to select some subset of features 5 S {1,2..... d} of size |S] = k that can be used to infer y. Were our y known in advance, selecting appropriate features would be straightforward using standard supervised feature selection techniques. However, we assume that y is not given priori, and depend on the salient latent variables s.
must instead select feature set S suitable for arbitrary downstream tasks that
When run a target dataset alone, standard unsupervised feature selection algorithms will not necessarily select features that contain much information about 8 (or, by extension, future labels y). To isolate these salient variations, we can leverage background dataset B {b, 1m E mx for which we assume that samples are generated only from the irrelevant latent factors with the salient variables fixed to some constant s'. We note that both our target and background samples are assumed to be drawn from the same data generating process p(-). Moreover, we do not assume that we have the same number of samples in the target and background datasets (i.e., of m), and do not assume that there is any special relationship between samples and j from the target and background datasets, respectively. We depict this data generating process Figure 2
Leveraging such a background dataset amounts to form of weak supervision that arises naturally in many real-world settings. For example, suppose we wanted to analyze data from patients receiving experimental treatment in clinical trial to better understand differences in how the treatment impacts the patients who receive it. In this case, a suitable background dataset would consist of data from control patients drawn from similar population but administered a placebo instead of real
3

---

Under review as a conference paper at ICLR 2022
treatment, and our salient latent factors of interest would correspond to variations present only in patients that received the real treatment.
3 RELATED WORK
The idea of CA has previously been used extend number of unsupervised learning methods. IZou et al. (003)) initially adapted mixture models for CA. Recent work (Abid al. 2018) adapted principal component analysis the contrastive setting so that a set of salient principal components are learned that summarize variations of interest in a target dataset while ig-
Contrastive Analysis.
noring uninteresting variations. Other recent works (Li et al. 2020 Abid & Zou 2019; Severson
et al. 2019 Ruiz et al., 2019; Jones et al. 2021) have since developed probabilistic latent variable models for CA. As part of their work, Severson et al. (2019) experimented with adding sparsity prior to one of their proposed contrastive latent variable models to perform feature selection. How- ever, this penalty was designed for use only fully supervised scenarios; the penalty was manually tuned by observing how well pre-labelled classes of target points separated in the resulting latent space, and this procedure is thus not suitable for use in the scenario described in Section 2
We note that CA can be seen specific case of the problem of learning disentangled representa-
tions (Bengio 2013), and which is the subject of much previous work (Higgins et al., 2017 Kim &
Mnih 2018 Chen et al. [201 80118110 Mathieu et al. 2019 Lopez et al. [018 Kumar al. 2018 Burgess
et al. 2018 Locatello et al.) 2019). However, unlike the general case, in which each feature encodes
semantically meaningful attribute and is invariant to the other features, in CA only require that background and salient features are disentangled; moveover, in the CA setting we are able to leverage background dataset form of weak supervision not available the general case, thus making our problem more tractable.
Finally, we note that CA unrelated the similarly named field of contrastive representation learn- ing (Chen et al. 2020a Schroff et al., 2015 Chen et al. 20206 Chopra et al. 2005).
Feature Selection. Feature selection has been extensively studied the machine learning literature. Much of this work has been devoted to selecting features in the supervised setting, in which each sample associated with a label that can be used to guide the selection process. Such methods can be divided broadly into three categories. Filter methods (Battiti 1994 Peng et al.) 2005 Estévez
jet al. 2009 Song et al.) 2007 2012: Chen et al., 2017) remove irrelevant features before model training by computing per- feature relevance scores based on some statistical measure. Wrapper methods (Kohavi & John 1997, Stein et al.) 2005 Zhu et al. 2007 Reunanen 2003 Allen (013)
assess the quality of subsets of features by training new models for each subset. Finally, embedded methods [Tibshiran]
1996; Yamada et al. 2020 Li et al.) 2016; Scardapane et al. 2017: Feng &
Simon 2017) jointly learn a model while selecting an optimal subset of features.
In this work we focus on feature selection in settings where we do not have access to per-sample labels. In such scenarios we must select set of features that are useful for arbitrary downstream tasks, and a variety of criteria have been proposed to perform fully unsupervised feature selection, including: selecting clusters of similar features (Mitra et al. 2002 Lu et al.) (007). using spectral
information to find features that best preserve the local structure of the data (He et al., 2005 Linden-
baum al. 2020), finding features that preserve local discriminative information (Yang et al. (001])
and selecting features that best preserve local clustering structure (Dy & Brodley 2004 Boutsidis
al. 2009 Cai et al. 2010).
Despite this diversity of approaches, to our knowledge no previous works have addressed feature selection for CA as formulated in Section and taken advantage of the weak supervision available in this setting.
4 METHOD
We aim to select ls features that are informative of the salient latent variables s (and thus, any potential future labels y) in our data generating process from Section 2 Such a procedure must be performed delicately, as standard unsupervised feature selection methods may select features that capture much of the overall variance in dataset but not the (potentially subtle) variations due
4

---

Under review as a conference paper at ICLR 2022
Background
Reconstruction
Reprosentation
b
(2(0)
(x(b)
b
9x
9x
fo
Target Selector
(a
C
C (boget)
a S
Figure 3: CFS model architecture. Here illustrate the general architecture of CFS-based meth- ods for feature selection. We use our background dataset to learn an embedding function 9s that only captures variations due the latent variables that present in both the background and target datasets. We also train a concrete selector layer to select features that best reflect the salient latent variables used generate target points. This selector layer trained along with reconstruc- tion function fe that is trained to reconstruct target data points using the concatenation of the output of Is and the features selected by our gates, and to reconstruct background data points using the concatenation of the output of Is and a zero vector. Here x refers to a target sample restricted to the set of features
To tackle our problem, we thus need a way to isolate the l features that best capture the variations in dataset specifically due to To do so, will use the following intuitive idea: suppose have some function Ø=: Rdd RR'K with k < d that takes in of our target data points æ; input and returns a low dimensional representation of z; that captures variations due to but does not contain information on s,. We call this function a background representation function. Assuming we have such a Iz. we then select set of features S that minimize the error of function fo: R + Rd parameterized by 9 that is trained to use our background representations as well as our selected features reconstruct the original target data points. That is, we attempt to optimize
arg min ([]fo(y=(01), ax x. 12.
where æ. denotes the target data point restricted to the set of features S, In other words, we select the of & features S that capture the variations present in our target dataset but which are missed by 9z. We call our approach CFS (Contrastive Feature Selection).
To implement CFS, must make choices. First, assuming already have some Oz. need method for selecting our set of features S. To do so, we leverage previously proposed stochastic gating layers (Yamada et al., 2020 Baln et al., 2019. Lindenbaum et al. 2020) developed as part of recent state of the art feature selection methods. Such methods select features by optimizing continuous approximations of discrete variables at the input layer of a neural network; selected variables are multiplied with 1 and allowed pass through the gates, while rejected features multiplied with a O. In our experiments we train a concrete selector layer (Balın et al.) 2019) to select features that, when concatenated with the low-dimensional representation from Øz and fed to fo. allow us to reconstruct target data points faithfully; this idea depicted in Figure B and we refer the reader Appendix A for details the concrete selector layer.
Next, we need a procedure for learning 92. Learning such a function is not straightforward, as requires JN to capture only variations due z, and not those due 8; if information leaks from

---

Under review as a conference paper at ICLR 2022
into Iz. are less likely to select the features that provide the most information However, with the help of a background dataset assumed to be generated only from 2, we may be able to successfully isolate these variations. In this work we explore multiple potential approaches for learning 9z:
1. Joint: We let 9N take the form of a multi-layer perceptron, and train it jointly with our con- crete selector layer and reconstruction function So using both our target and background datasets. When reconstructing target data point using fe. we concatenate the output of Is with the features selected by our concrete selector layer. On the other hand, when recon- structing a background data point, we concatenate the output of Oz with zero vector meant to represent the absence of salient variations. Such an approach is highly expressive and likely to lead to good reconstructions of our data. However, without additional constraints on 9= also possible that information will leak between Oz and our concrete selector layer, leading our model to select lower-quality features. To test this hypothesis, we also experiment with two other approaches for learning Iz designed to reduce the possibility of information leakage.
2. Pretraining: We first train standard autoencoder solely
background dataset. We then use our autoencoder's encoder network as freeze its weights, and then learn So and our concrete gates layer as described for the Joint model. By fixing the weights of 9z. we hope to prevent it from encoding salient variations when applied to target data points.
3. Gates: We let take the form of a second concrete selector layer and otherwise train it using the same approach as our Joint model. By constraining Iz to perform feature selection, opposed to general representation learning, we hope to reduce the possibility for information to leak between J and our target-point-specfie concrete selector layer.
5 EXPERIMENTS
Here we validate our CFS approach by applying to a number of datasets. We begin by reporting re- sults from a suite of experiments on a semi-synthetic Grassy MNIST dataset. We then report results obtained from four publicly available collections of biomedical data. For each of these collections, points in the target dataset have ground-truth class labels. However, distinguishing between these classes may be difficult using features selected by algorithms not designed for the CA setting. In our experiments we compared our proposed training schemes against set of fully unsupervised feature selection methods to illustrate the benefits of our weakly supervised approach. These baseline meth- ods include: the Laplacian score (He et al.] 22055) MCFS (Cai et al.. al.) (2110). PFA (Lu et al. (007)), the concrete autoencoder (CAE; Balın al. (2019). ) and the recently proposed gated Laplacian score (Lindenbaum et al., 2020). We refer the reader to Appendix B for details on dataset preprocessing and Appendix for details model implementation and training. For all experiments, divide our data using an 80-20 train-test split. Our code for downloading and preprocessing these datasets well implementations of our models are available www www.placett der. com
5.1 GRASSY MNIST
The Grassy MNIST dataset, originally introduced in Abid et al. (2018), is constructed by superim- posing pictures with the "grass" label from ImageNet (Deng et al.j 2009) onto handwritten digits from the MNIST dataset (LeCun et al. 2010). For each MNIST digit, randomly chosen grass image was cropped, resized to be 28 by 28 pixels, and scaled to have double the amplitude of the MNIST digit before being superimposed onto the digit. Our goal here is to classify images based on their digit class. For a background dataset, we used a separate set of images consisting only of grass. We note that, although Grassy MNIST is image dataset, the MNIST digits in each image are centered and thus we can treat each pixel as a separate feature as has been done with the stan- dard MNIST dataset in previous works on feature selection (Balin et al. 2019 Yamada et al. 2020 Lindenbaum et al. 2020).
Cooee available in supplementary materials and will he made public upon acceptance.

---

Under review as a conference paper at ICLR 2022
0.9
0.8
Accurccy
08
(Pretranee)
Concrete Autsencoder
(faste))
02
Superiisd Concrate
0.1
Background reprosentation dimension
Rabo of Grass Dight
(b)
Figure 4: Downstream classification accuracies and sensitivity analyses of CFS on Grassy MNIST: (a): We quantitatively assess the quality of each method's selected features by training an extremely randomized trees classifier (Geurts et al.] (006) to classify Grassy MNIST images by their digit class using varying numbers of selected features k. We report mean accuracies over five initializations, and error bars correspond to standard deviation. For all tested values of k, we find that our Pretrained and Gates CFS models outperform state of the art unsupervised feature selection algorithms. (b): We fix k 20 and vary the size of each CFS variant's background representation to understand how this hyperparameter impacts feature quality. Our Pretrained and Gates CFS varia- tions continue outperform baseline methods for all background representation sizes. (c): We vary the relative contribution of grass noise to the dataset and assess the performance of each method when used to select l 20 features.
5.1.1 DOWNSTREAM ACCURACY
We began our evaluation by using each of our proposed CFS models to select features for varying values of & between 10 and 50 with fixed background representation size K' 20. We evaluated the features selected by each method by training classifiers to predict the digit (i.e., 0-9) in cach image. That is, for given method and value of k; extracted the set of features S selected by that method. The resulting data matrix X was then used to train an extremely randomized trees classifier (Geurts et 2006), variant of random forests used by previous works (Drotár et al.) 2015 Balın et al. 2019) to evaluate feature selection methods. We emphasize that these classifiers were not used select features, but only evaluate the quality of the features selected by each method. We report the accuracy of the classifiers on held out test set in Figure 4a We also report results from two other baseline methods that attempt to leverage the weak supervision in the CA setting. That is, we report results from simple filter baseline that selects the features with the great difference in variances in the target dataset vs. background dataset ("Variances")) and for supervised concrete autoencoder ("Supervised Concrete") trained to distinguish between target vs. background points. We also report results for our fully unsupervised feature selection baseline methods. To reduce visual clutter, we only report results for the best performing fully unsupervised baseline (CAE) in Figure 4a and provide results for the other baselines in Appendix D.I We also visualize the features selected by each method in Figure [and Appendix D.2
We find that our Pretrained and Gates CFS models outperform all baseline methods for all values of k. On the other hand, we find that our Joint CFS model did not outperform our baseline methods. Such a result potentially indicates that our Joint approach fails to properly disentangle salient and background variations when selecting features.
5.1.2 SENSITIVITY ANALYSES
To better understand the robustness of CFS in various settings, we next conducted experiments mea- suring how CFS's 's performance was affected by changes in our experimental setup. First, we sought to understand how varying the hyperparameter k (i.e. the size of the background representation) affected CFS variants' performance. To do so, fixed the number of selected features & 20 and varied the background representation size between 10 and 50. We then report the accuracy of extremely randomized trees classifiers trained the features selected from these experiments in
7

---

Under review as a conference paper at ICLR 2022
Figure 4b As baseline, we also include the concrete autoencoder and two other weakly super- vised baselines trained to select 20 features. For all background representation sizes, we find that our Pretrained and Gates CFS variants continue outperform fully unsupervised feature selection methods while our Joint variation underperforms baseline models.
We next sought assess the how the performance of CFS varies with respect the amount of background noise vs. salient variation. To do so, we varied the scale of the background images of grass used construct Grassy MNIST. Starting with scale of (i.e., our target dataset was the standard MNIST dataset), we increased the value of our scale parameter in increments of 0.5 until reaching 4 (i.e., the amplitude of the background was four times that of the digits). For each increment, we used our CFS models as well as our basclines to select lo = 20 features and evaluated the quality of these features using the same approach described previously. We report our results in Figure 4c] We find that our Pretrained and Gates CFS models consistently outperform baseline approaches for non-zero levels of noise. This result continues to hold even as the level of noise in our data greatly surpasses that of the salient factors of variation. On the other hand, we find that our Joint CFS approach again fails to outperform baseline methods.
5.1.3 QUANTIFYING DISENTANGLEMENT
Finally, we sought to better understand why our Pretrained and Gates CFS variants outperformed standard unsupervised methods while our Joint variant failed to do so. In particular, in these ex- periments we measured how well each of our CFS variants disentangled salient and background variations; that is, we sought to measure how much salient information leaked into each CFS vari- ant's background representations, and how much background information leaked into each variant's y selected target features. We expect that methods that struggle to disentangle the two sources of variation would be more likely to select suboptimal features for CA tasks.
To quantify this disentanglement, we reused the CFS models trained produce Figure 4a We first extracted the target features selected by each model, and then trained a multilayer perceptron to reconstruct background dataset samples using only these target features. If method has successfully disentangled background and target variations, thereby selecting a set of target features without much background information, we would expect its selected features result in higher error on this task. We report the mean squared eгror of reconstructions for a held out test set in Figure [5a We find that the features selected by our Pretrained and Gates CFS variants consistently result in higher reconstruction errors than do the features selected by our Joint variant.
Next, instead of extracting the target features selected by each method, we obtained the background representations 92 (xxx,) for every x; in our target dataset. We then trained multilayer perceptrons to reconstruct target data points using these background representations. Better disentanglement again corresponds to higher error on this task. We report our results for this experiment in Figure 5b) We find that the representations Icarned by our Pretrained and Gates CFS variants result in higher reconstruction eπror compared to those learned by the Joint model. Taken together, these results indicate that our Gates and Pretrained models are indeed more effective at disentangling salient and background variations than our Joint model. Moreover, when combined with our downstream classification results from Figure 4 these results indicate that achieving such disentanglement is crucial to selecting suitable features for CA tasks.
5.2 EVALUATION ON REAL- WORLD DATASETS
We next applied CFS to four real-world datasets. Here we compare our proposed training schemes against the same set of baselines used in our Grassy MNIST experiments. In our evaluations we used each method to select k 20 features for each dataset, except for the Mice Protein dataset for which we used k = 5 because of its lower dimensionality. Due to its poor performance on Grassy MNIST, we did not apply the Joint CFS approach to these datasets and only experimented with the Pretrained and Gates variants. We include brief deseriptions of our datasets below in Section 5.2.1 along with table containing the number of samples and features for each dataset in Appendix E
8

---

Under review as a conference paper at ICLR 2022
2.04
0.35
CPS (Pretrinne)
9022
CP5 (Gates)
0.330
0.955
2018
2014
0.000
2014
0415
2012
0010
30
50
Background regresentatio dimension
(a)
(b)
Figure 5: Quantifying disentanglement of background and salient variations. (a): We train multilayer perceptrons to reconstruct buckground samples using the larget features selected by each CFS variant. method that successfully disentangles target and background variations should result in higher reconstruction error on this task. (b): We train multilayer perceptrons to reconstruct target samples using each CFS variant's background representations. Once again, higher reconstruction error corresponds to better disentanglement.
5.2.1 DATASET DESCRIPTIONS
Epithelial cell infection response. We constructed our target dataset by combining two sets of gene expression measurements from Haber et al. (2017). These datasets consist of gene expression measurements of intestinal epithelial cells from mice infected with either Salmonella or Heligmoso- moides polygyrus (H. poly). Here our goal is separate cells by infection type. As background dataset we used measurements collected from healthy cells released by the same authors.
Bone marrow cell treatment response. Here combined datasets from Zheng et al. (2017) containing gene expression measurements from a patient with acute myeloid leukemia (AML) before and receiving a blood cell transplant. Our goal here is to learn representation that separates pre and post transplant measurements. As background dataset we used expression measurements from a healthy control patient that were collected as part of the same study.
Mice Protein Expression. This dataset (Higuera et al. 2015 Ahmed et al. 2015) consists of protein expression levels from healthy control mice and mice that have developed Down syndrome. Each mouse was injected with memantine, subjected to shock therapy, received both of these treatments, or none of them. Our goal here was to classify mice with Down syndrome based on their treatment regimens. We used data from healthy mice that did not receive any treatment
background.
Human Activity Recognition Using Smartphones. This dataset consists of sensor data collected from smartphones mounted on the waist of study participants as they performed various physical activities (c.g. walking, sitting, laying down), with each data point corresponding to single activity. The goal of this task to classify samples by the activity performed during collection. Here our background dataset consisted of measurements taken while participants were laying down.
5.2.2 RESULTS
To evaluate each method's performance on these datasets, we once again extracted the features se- lected by each method and trained extremely randomized trees classifiers to classify target data points. We report the results of these experiments Table T We find that our CFS methods yield consistently higher quality features than do baseline methods, with the sole exception of our pre- trained CFS variant on the Mice Protein dataset. We also repeated the same evaluation procedure but with gradient boosted trees implemented in XGBoost Chen & Guestrin (2016) as our down- stream classifier. We report these results in Appendix [ and find that they are consistent with our results in Table || These results further demonstrate that Pretrained and Gates CFS variants choose features more suitable for downstream CA tasks than do standard unsupervised feature se- lection methods.

---

Under review as a conference paper at ICLR 2022
Dataset
Lap
MCFS
PFA
CAE
Gated Lap
CFS (Pretrained)
CFS
(Gates)
Epithelial Cell
0.593
0.606
0.596
0.834
0.638
0.904
0.897
Leukemia Treatment
0.708
0.853
0.588
0.934
0.644
0.970
0.952
Mice Protein 0.865
0.820
0.838
0.793
0.784
0.847
0.973
Smartphone Activity 0.846
0.912
0.906
0.923
0.904
0.937
0.961
Table 1: Performance of CFS and baseline methods on real-world datasets. Here we show the classification accuracies of classifiers trained on varying numbers of features selected by CFS and various baseline methods (higher better). For each method we selected k 20 features (except for the mice protein dataset, for which we use k = 5 because the data is lower dimensional). We used an extremely randomized trees classifier and report performance on held out test set.
DISCUSSION
In this work we considered the problem of feature selection in the contrastive analysis setting. In this setting we are specifically interested in discovering features that reflect salient variations enriched in some target dataset compared to a background dataset. To tackle this problem we proposed CFS (Section 4. On both simulated and real-world datasets we found (Section 5 that features selected by CFS to superior performance on downstream CA tasks across multiple data modalities as compared to those selected by fully unsupervised feature selection methods. Like other methods designed for CA, CFS requires that the user have access to background dataset generated from factors of variation that uninteresting to the user. Collecting such data may not always be feasible and does impose an additional burden on the user; however, such background datasets are routinely collected in many real-world problem domains (e.g. part of clinical trials) that naturally suited for CA. We believe that CFS represents an important first step towards addressing feature selection in the CA setting, and hope that be used basis for future work.
REFERENCES
Abubakar Abid and James Zou. Contrastive variational autoencoder enhances salient features. arXiv preprint 1XXi::992.006011 2019.
Abubakar Abid, Martin J Zhang, Vivek K Bagaria, and James Zou. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature communications, 9(1):17, 2018.
Md Mahiuddin Ahmed, A Ranjitha Dhanasekaran, Aaron Block, Suhong Tong, Alberto CS Costa, Melissa Stasko, and Katheleen J Gardiner. Protein dynamics associated with failed and rescued learning in the ts65dn mouse model of down syndrome. PloS one, 10(3):e01 19491, 2015.
Genevera I Allen. Automatic feature selection via weighted kernels and regularization. Journal of Computational and Graphical Statistics, 22(2):284 2999 2013.
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning, pp. 444 453. PMLR, 2019.
Roberto Battiti. Using mutual information for selecting features in supervised neural net learning.
IEEE Transactions on neural networks, 5(4):537-550, 1994.
Yoshua Bengio. Deep learning of representations: Looking forward. In International conference on statistical language and speech processing, pp. 1-37. Springer, 2013.
Christos Boutsidis, Petros Drineas, and Michael W Mahoney. Unsupervised feature selection for the k-means clustering problem. In Advances in Neural Information Processing Systems, pp- 153-161, 2009.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in B-vae. arXiv preprint arXiv: 1r04 03599, 2018.
10

---

Under review as a conference paper at ICLR 2022
Deng Cai, Chiyuan Zhang, and Xiaofei He. Unsupervised feature selection for multi-cluster data. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 313-333322 2010.
Jianbo Chen, Mitchell Stern, Martin J Wainwright, and Michael I Jordan. Kernel feature selection
via conditional covariance minimization. arXiv preprini /70.00177000166 2017.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In NeurlPS, 2018.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kormblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong cumissuperiieed learners. arXiv preprint arXiv:2006. 10029, 2020b.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively. with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'OS) volume 1, pp. 539-546. IEEE, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Peter Drotár, Juraj Gazda, and Zdenek Smékal. experimental comparison of feature selection methods on two-class biomedical datasets. Computers in biology and medicine, 66:1-10, 2015.
Jennifer G Dy and Carla E Brodley. Feature selection for unsupervised learning. Journal of machine learning research, 5(Aug):845-889, 2004.
Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. Normalized mutual infor- mation feature selection. IEEE Transactions on neural networks, 20(2):189-2011 2009.
Jean Feng and Noah Simon. Sparse-input neural networks for high-dimensional nonparametric regression and classification. arXiv preprint arXiv: 2017.
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine learning, 63(1):3-42, 2006.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
Adam L Haber, Moshe Biton, Noga Rogel, Rebecca Herbst, Karthik Shekhar, Christopher Smillie, Grace Burgin, Toni M Delorey, Michael R Howitt, Yarden Katz, et al. A single-cell survey of the small intestinal epithelium. Nature, 551 (768003333333333 2017.
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. Advances in neural information processing systems, 18. 2005.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
Clara Higuera, Katheleen J Gardiner, and Krzysztof J Cios. Self-organizing feature maps identify proteins critical to learning in a mouse model of down syndrome. PloS one, 10(6):c01211 2015.
11

---

Under review as a conference paper at ICLR 2022
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softnax. arXiv preprint 1rXiv:1611.01144 2016.
Andrew Jones, F William Townes, Didong Li, and Barbara Engelhardt. Contrastive latent variable modeling with application to case-control sequencing experiments. arXiv:2102.06731, 2021.
arXiv preprint
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp.
PMLR, 2018.
Michael S Kinch. An analysis of fda-approved drugs for oncology. Drug discovery today, 19(12): 1831-1835, 2014.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes. arXiv preprint
Ron Kohavi and George H John. Wrappers for feature subset selection. Artificial intelligence, 97 (1-2):273-324, 1997.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference Learning Representations, 2018.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://samm.leuu..cmm//dmmmmmmit 2, 2010.
Didong Li, Andrew Jones, and Barbara Engelhardt. Probabilistic contrastive principal component
analysis. arXiv preprint arXiv:2012.07977, 2020.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P'rrevino, Jiliang Tang, and Huan
Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):94, 2018.
Yifeng Li, Chih- Yu Chen, and Wyeth W Wasserman. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23())322-333,, 2016.
Ofir Lindenbaum, Uri Shaham, Jonathan Svirsky, Erez Peterfreund, and Yuval Kluger. Differentiable unsupervised feature selection based on gated laplacian. arXiv preprint arXiv:2007.04778, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints auto- encoding variational bayes. Advances in Neural Information Processing Systems, 31, 2018.
Yijuan Lu, Ira Cohen, Xiang Sean Zhou, and Qi Tian. Feature selection using principal feature analysis. In Proceedings of the 15th ACM international conference Multimedia, pp. 301-304, 2007.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712. 2016.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. International Conference on Machine Learning, pp. 4402-4412. PMLR, 2019.
Pabitra Mitra, CA Murthy, and Sankar K. Pal. Unsupervised feature selection using feature similar- ity. IEEE transactions on pattern analysis and machine intelligence, 24(3):301-312, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-peeformance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
12

---

Under review as a conference paper at ICLR 2022
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based mutual information criteria of max mux--eeendency, max-relevance, and seidddddud....... IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226-123) 1238, 2005.
Juha Reunanen. Overfitting in making comparisons between variable selection methods. Journal of Machine Learning Research, 3(Mar): 2003.
Adrià Ruiz, Oriol Martinez, Xavier Binefa, and Jakob Verbeek. Learning disentangled representa- tions with reference-based variational autoencoders. arXiv preprint arXiv:1901.08534 2019.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regu- larization for deep neural networks. Neurocomputing, 241:81-89, 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the recognition, pp. 815-823. 2015.
the IEEE conference on computer vision and pattern
2015.
Kristen A Severson, Soumya Ghosh, and Kenney Ng. Unsupervised learning with contrastive latent variable models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4862-4869, 2019.
Le Song, Alex Smola, Arthur Gretton, Karsten M Borgwardt, and Justin Bedo. Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning, pp. 823-833-00 2007.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
Giorgio Stanta and Serena Bonin. Overview on clinical relevance of intra-tumor heterogeneity. Frontiers in medicine, 5:85, 2018.
Gary Stein, Bing Chen, Annie S Wu, and Kien A Hua. Decision tree classifier for network intrusion detection with ga-based feature selection. In Proceedings of the 43rd annual Southeast regional Conference-Volune 2, pp. 136-141, 2005.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
F Alexander Wolf, Philipp Angerer, and Fabian Theis. Scanpy: large-scale single-cell gene ex- pression data analysis. Genome biology, 19(1):1-5, 2018.
Yutaro Yamada, Ofir Lindenbaum. Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pp. 10648-10659, PMLR, 2020.
Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. L2, 1-norm regularized dis- criminative feature selection for unsupervised. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
Grace XY Zheng, Jessica Terry, Phillip Belgrader, Paul Ryvkin, Zachary Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications, (11:1-12,2011,
Zexuan Zhu, Yew-Soon Ong, and Manoranjan Dash. Wrapper- feature selection algorithm using memetic framework. IEEE Transactions Systems, Man, and Cybernetics, Part B (Cy- bernetics), 37(1):70-76, 2007.
James Yang Zou, Daniel Hsu, David C Parkes, and Ryan Prescott Adams. Contrastive learning using spectral methods. Proceedings Advances in Neural Information Processing Systems, 2013.
13

---

Under review as a conference paper at ICLR 2022
A THE CONCRETE SELECTOR LAYER
The concrete selector layer, introduced in Balın et al. 2019). is based on concrete random variables Maddison al. 2016 Jang et al. 2016). concrete random variable can be sampled produce continuous approximation of a ont vector. To sample d-dimensional concrete random variable, one first samples d-dimensional vector of i.i.d. samples from a Gumbel distribution (Gumbel 1954 g. Each element of our concrete variable is then defined as
exp((log a, + +g)/T)
m, Σ exp((log ar + gx)/T)
Here our temperature parameter T E (0, 00) controls the extent to which our one-hot vector is relaxed. In the limit 7 - 0, our concrete random variable outputs one-hot vectors with m, = 1 with probability / Σ k= axg. Concrete random variables are differentiable with respect to their parameters a via the reparameterization trick (Kingma & Welling 2013).
We can concrete random variables to select features through concrete selector layer. The concrete selector layer selects features in the following manner. For each of the k nodes in the selector layer, we sample a d-dimensional concrete random variable m i kl. The ith node in the selector layer ()) outputs æ ()) As + this inner product simply becomes equal to one of the input features. After the network is trained, the concrete selector layer is replaced by an arg max layer for which the output of the ith neuron is Zarg max, a,"
Before training, the selector's parameters a are initialized as small positive values to encourage the layer to explore different combinations of input features. As the layer is trained, the values of og become more sparse as the layer becomes more confident particular choices of features.
As proposed in Balm et al. (2019), we train our concrete selector layers using simple annealing schedule for the temperature parameter T. That is, we begin training with a high temperature To and gradually decay the temperature until we reach a final temperature TB each epoch according to first-order exponential decay T(b) To(Tp/ /To)b//B where T(b) is the temperature at epoch b and the total number epochs.
B DATASET PREPROCESSING
Here we provide preprocessing details for each of the datasets used in Section 5.2 Jupyter notebooks with implementations of our preprocessing workflows can be found in the supplementary materials.
Grassy MNIST. We first constructed the original version of Grassy MNIST as described in Abid et al. 2). For each MNIST digit, randomly chosen "grass" image from ImageNet was cropped, resized to be 28 by 28 pixels, and scaled to have double (except for the results presented in Figure 4c for which the scale parameter was varied) the amplitude of the MNIST digit before being super- imposed onto the digit. For background dataset, we used a separate set of images consisting only of grass. This dataset was used generate the results in Figure 4al
we generated multiple versions of Grassy MNIST using the same procedure described above but with varying the scale of the grass relative to the MNIST digits. In our experiments we varied this scale parameter from zero to four in increments of 0.5.
To produce the results in Figure 4c
Epithelial Cell. We began by downloading GSE92332 SawmHellmllllaeeea UMIcounts txt from
https: // www. ncbi mlm. n h.
acc. cgi ?acc=GSR92332 Each col- umn this file corresponds to a single cell sample, while each row corresponds to the expression level of given gene. For each cell, we extracted its condition (i.c., healthy control discasc sta- tus) from that cell's label in the first row of the file. In particular, in our experiments we used cells labelled as Controll rol, Sa Salmonella. la, and Hpoly Day1 0; cells with other labels were discarded.
Following standard practices in single cell RNA-seq analyses, we first normalized the counts such that each cell had a total count of 10000 after normalization. We then took the logarithm of these normalized counts + 1. For these two normalization steps we used the normal rimaliie total and loglp po functions implemented in the Scanpy python package Wolf et al. 2018).
14

---

Under review as a conference paper at ICLR 2022
Leukemia
Treatment Response. We downloaded set of scRNA-seg gene expres- sion measurements from a leukemia patient pre and post-tramsplant from https:
pre transplant/ tar.ttarr.oo and https:// tttp_ppossttts_llaaaaaa
am http://70..acaaaaggooooaaaaaaaaa
cf. F..100 htt.f//....ooorece.ccccc///aaaaa otg.apahhhhaaaaaaaaaaaaoaaaaaaaa
am1027_post htt__ppssttraappaaaa___eeeeeeeee Btctaan__fiitteeee
______________________________ het.reee_eeeeeee____tiiieeeeeeee taa. respectively, A
set of measurements from healthy control patient were also downloaded from https://cf.
O0x.gppon..sssccooo/mmmaeeeeells Erozen_
frozenn Btc____mm____________ healthy.hhhh....... ch http_/cc_aattt_eeeee__ tar.iiiees.trr.g.......
We then preprocessed our data in a similar manner as done in previous work (Li et al. 2020). That is, we took the log of expression counts + I, removed any gene features or cell samples consisting entirely of zeros, and filtered down the list of gene features for each file to those that were shared among all the files.
Mice Protein. We downloaded Cotte
x1s from databases 00342/ and converted file. Rows I through 77 of this file were used as our features while rows 78 through 81 were used to determine class labels. For a given sample, if feature value was labeled as missing, we imputed the value of that feature using the mean value for that feature across all samples. As done in previous work (Balm et al. 2019) we then scaled the values of each features to lie between zero and one.
ccs.uct: www/eddmmm/////m htt.///aannieeeemeeeeeeeeeeeeeee
CSV
Our background dataset was then chosen to be mice that were healthy, did not undergo shock therapy (label S S/C) and which received an injection of saline solution. Our target dataset then consisted of mice that did undergo shock therapy (label C/ These target mice were divided into four subclasses based on: whether they were trisomic (i.e.. had Down Syndrome) and whether they received an injection of Memantine or a placebo saline solution.
Smartphone Activity. We downloaded train. csv and test csv from https: // www. kaggle www.comm/uaoo.o comm/ueimlle /hwmannn/mmmmmma cetimaan-cttivinn activity recognitiono with- smartphones combined the samples from both files to produce a single dataset (we note that we later performed an 80-20 train-test-spiie on this combined dataset). For our background dataset, we used samples with the class label LAYING, and the remainder of the original dataset was used as our target dataset. Following previous work (Balm et al. (019) we scaled the values of each feature to lie between zero and one.
We
C IMPLEMENTATION DETAILS
All of our CFS models as well as other methods trained using SGD-based optimization routines (Concrete Autoencoder, Gated Laplacian) were implemented using PyTorch (Paszke al.) 2019) using the Pytorch Lightning API. For all datasets and experiments we used a minibatch size of 128 for methods trained using SGD-based optimization routines.
2
For our concrete autoencoder implementation, we used two hidden layers of size 512 with ReLU activation functions for all datasets except the mice protein dataset. When training on the mice protein dataset, we used a single hidden layer of size 512 due to the dataset's low dimensionality. At the start of training, we set the temperature of the concrete layer to 10, and gradually annealed the temperature to final value of 0.1 using the exponential decay schedule described in Appendix A As in Balın et al. (2019), we trained our concrete autoencoders until the mean of the concrete samples exceeded 0.99.
For all CFS variants, we let our reconstruction function fo be multilayer perceptron with two hidden layers of size 512 with ReLU activation functions; the only exception was models applied to the mice protein dataset, for which we used only a single layer of size 512. The temperatures of the target feature concrete selector layers were all tuned in the same way as concrete autoencoder implementation. For our Joint CFS models, we let 9 be a multilayer perceptron with a single hidden layer of size 128. For our Pretrained CFS models, we first trained autoencoder consisting of an encoder with a single hidden layer of size 128 and a decoder with the same architecture in reverse. The encoder network's weights were then frozen and it used Jz- Finally, for our Gates
httts:://g
15

---

Under review as a conference paper at ICLR 2022
CFS variant we let 9z be second concrete selector layer for which the temperature parameter was adjusted as described for our concrete autoencoder implementation. We trained all CFS variants Grassy MNIST,
until the mean of the concrete samples exceeded 0.99. For our experiments unless otherwise specified, we used a background representation size 20. For our real-world datasets, set k' = 20 except for the Mice protein dataset for which set k' due the lower dimensionality of the dataset.
We trained the concrete autoencoder and gated Laplacian score baselines using the optimizers rec- ommended by their respective authors. That is, for the concrete autoencoder we used ADAM with a learning rate of 0.001, 81 0.9 and B2 0.999, while for the gated Laplacian score we used SGD with learning rate of I. For all CFS variants we used ADAM with the same hyperparameter settings as for the concrete autoencoder. For the Gated Laplacian, we manually tuned the hyperparameter so that the method selected the desired number of features for given experiment.
For the reconstruction experiments presented Figure we used a multilayer perceptron with two hidden layers of size 512. These MLPs were trained using ADAM with learning rate of 0.001, 3 = 0.9 and 32 = 0.999.
For the Laplacian score and MCFS, we used their mentations and ran them with the default hyperparameter choices. For PFA, we used the pub- licly available implementation from Balm et al. (2019) in the pfa.selector function avail- able at
Li et al. 2018) imple-
exper expeac/mmmnttsseeeeeereeeeoaaaaa http:tteeecccpprrrriiceo
To evaluate the downstream classification performance of a subset features, we trained extremely randomized tree classifiers using the ExtraTreesClan sossiiiess fier implementation available in sci felkit-lea rn with the n eatttorss parameter set to 100.
To alleviate potential issues with overfitting, randomly divided each dataset into training and test sets using an 80-20 split. The training data was used to select features and train classifiers for downstream tasks, while the test data was only used evaluate the performance of the classifiers.
D RESULTS FOR ADDITIONAL BASELINES ON GRASSY MNIST
D.1 QUANTITATIVE RESULTS FOR ADDITIONAL BASELINES
In Figure 6 we provide the results of the Grassy MNIST experiments depicted in Figures 4a and 4c for additional fully unsupervised feature selection methods. We found that for both set of exper- iments, the concrete autoencoder (CAB) tended to outperform all other baselines tested. No fully unsupervised baselines was able to match the performance of the Pretrained or Gates CFS variants on these experiments.
D.2 VISUALIZATIONS OF SELECTED FEATURES
In Figure 7 we provide visualizations of the k 20 features selected by each of our CFS variants and fully unsupervised baselines not already depicted in Figure Qualitatively, we observe that the features selected by the Pretrained and Gates CFS variants of high quality; both methods select set of features spread out near the center of the image. On the other hand, some of our unsupervised baselines (MCFS, PFA. CAE) and the Joint CFS variant select many features near the edge of the images, where no digit information is present. Moreover, our spectraliifforration--ann (Laplacian score, Gated Laplacian) tend select small number of clusters of co-located features, limiting the ability of downstream classifiers to distinguish between the ten digit classes.
methods
E DATASET DETAILS
In Table 3
we provide additional details (number of samples and features) for each of the real-world
datasets used Section 5.2
feature
16

---

Under review as a conference paper at ICLR 2022
Guted Laptacian
0.6
0.5
04
0.4
(a)
Figure 6: Applying fully unsupervised feature selection methods to Grassy MNIST: (a): We measure the quality of features selected by fully unsupervised methods as done in Figure 4a We find that the concrete autoencoder (CAE) performs the best out of our baseline methods. (b): We fix the number of features & = 20, vary the ratio of noise to digit when constructing Grassy MNIST, and then evaluate the quality of selected features by each unsupervised baseline in Figure 4c Here too we find that CAE tends to perform the best out of our fully unsupervised methods.
(Gates)
CtS
Gated Laplacian
(a)
(b)
(c)
(d)
(c)
Figure 7: Additional visualizations of Grassy MNIST results. Here we provide visualizations of the features selected by each of our baseline unsupervised feature selection methods when run to select 20 features. We find that our features that diffused around the images, with many selected features being close to the edges where digits are never present. On the other hand, our spectral-informatte informmtiinn- based methods (c. d) tend to select localized clusters of features. which (as demonstrated by our quantitative results), do not capture enough variations to successfully classify images based on digit.
methods tend pick
F DOWNSTREAM CLASSIFICATION REULTS USING GRADIENT BOOSTED TREES
17

---

Under review as a conference paper at ICLR 2022
Dataset
Features (d)
Train size (target)
Train size (background)
Test size
Classes
Epithelial Cell
15215
3584
2592
897
2
Leukemia Treatment
12079
6318
1985
1580
2
Mice Protein
444
108
Smartphone Activity 561
6684
1555
1671
5
Table 2: Characteristics of real-world datasets used for empirical evaluation. Each dataset con- tains both background dataset, generated from irrelevant sources of variation, and target dataset, generated from both the irrelevant source of variation as well as the salient source of variation used generate class labels. We note that CFS uses both target and background data points select features, while all baseline methods are run solely on the target data points.
Dataset
Lap
MCFS
PFA
CAE
Gated Lap
CFS (Pretrained)
CFS
(Gates)
Epithelial Cell
0.649
0.606
0.625
0.842
0.672
0.892
0.887
Leukemia Treatment
0.716
0.855
0.592
0.933
0.663
0.951
0.968
Mice Protein
0.792
0.721
0.775
0.721
0.658
0.838
0.874
Smartphone Activity
0.829
0.876
0.906
0.908
0.873
0.936
0.940
Table 3: Downstream classification performance features selected by CFS and baseline methods real-world datasets assessed using gradient boosted tree classifiers. Here we show the classification accuracies of classifiers trained on varying numbers features selected by CFS and various baseline methods (higher is better). For each method selected k 20 features (ex- cept for the mice protein dataset, for which we use k = because the data lower dimensional). Here used the XGBoost Chen & Guestrin (016)) implementation of gradient boosted tree classifier and report performance on held out test set. We find our results to be consistent with those found using extremely randomized trees classifiers (i.e., our CFS variants outperform the fully unsupervised methods).
18


Reviews:
[{"review_id": "y80Uew2t8HZ", "reviewer": "Reviewer_vsNM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments:\n- The evaluation only uses extremely randomized trees as the classifier, which is not ideal for a feature selection evaluation as the randomness means that it's hard to know if the whole feature space was useful. Some kind of regularized gradient boosted tree, or other non-linear but less randomized classifier would also be helpful in evaluating the selection procedure. Or even an MLP given the feature selection system is in pytorch.\n- There are fairly simple filter baselines for contrastive analysis once the problem has been introduced. For example selecting features which have different variances or entropies as compared to the background dataset, or selecting features which maximise the KL divergence between p(x_b) and p(x_t).  Without these baselines it's hard to say whether it's the additional supervision from the background/target difference or the specific techniques proposed in the paper which improve performance.\n- Could the authors expand on the difference between the pretrained and gated systems with reference to the experimental results? It's not clear in what circumstances one technique should be preferred over the other, neither experimentally nor theoretically. Presumably the pretrained method took longer as there were two separate training runs, but there are no runtimes reported either.\n- Overall the presentation of the contrastive analysis problem is well motivated, the dataset choice and descriptions are helpful and the paper is well written.\n\nQuestions:\n- How big was the background set for grassy-MNIST? \n- Are the grass images sampled with replacement or without replacement, and if without replacement is it the same sample across both background and target datasets? Are any grass images repeated?", "summary_of_the_paper": "The paper presents a weakly supervised feature selection algorithm which selects features which have different amounts of signal compared across two datasets (a background or control dataset, and the dataset of interest). This allows it to sidestep the requirement for labels to select features for a downstream task or group of tasks. It uses a concrete selection layer to find relevant features with three different approaches for separating them from the background features.", "summary_of_the_review": "The problem the paper investigates is interesting, the algorithm seems like a sensible extension of unsupervised techniques to the weakly supervised setting.  The experimental study has weaknesses, but still justifies the proposed technique.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cxskacpXlL", "reviewer": "Reviewer_ft7b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths. The reported accuracy of the experimental results is extremely good. \n\nWeaknesses. In general, there are a number of unclear points in the paper. Fig. 3: \"captures variations due to our uninteresting latent variables z\", what do you mean by that? Do you mean that $g_z$ is not useful?\n\nNot clear sentence: \"A diverse array of prior work exists on unsupervised feature selection, with one major differentiating factor between methods being how to define the “usefulness” of a feature.\" \n\nThe feature selection is performed using the existing stochastic gating layers, so, the feature selection novelty is absent. The final framework is quite complex, is not extremely clearly described. \n\nIn the experimental section, it is mentioned that to assess the quality you train an extremely randomised tree classifiers. Am I right that this part is not drafted on Figure 3? You perform the feature selection (the framework on Fig. 3), and then a classifier (in your case randomised trees) are applied to learn the predictive model?\n", "summary_of_the_paper": "Summary. The paper considers the problem of feature selection in the contrastive analysis setting. In particularly, the authors are interested to discover features that reflect salient variations enriched in some target dataset compared to a background dataset. A new method, called Contrastive Feature Selection, is proposed.", "summary_of_the_review": "The paper presents a combination of an existing feature selection method (called stochastic gating layers) and a classifier (to test the models performance). There is a lack of novelty in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L6l_qT1gi7f", "reviewer": "Reviewer_Mgox", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1) Modeling the latent vector as salient and background latent vector is interesting. \n2) The proposed algorithm does not use too much supervision and only relies on the knowledge of whether a data point belongs to background or target datasets.\n\nCons:\n\n1) One of the main weaknesses of the proposed method is the choice of k. It is not clear as to how one would choose the number of salient features. \n\n2) I have doubts about the basic problem formulation. The paper talks about contrastive analysis, but completely ignores the recent work on contrastive learning in ML community that uses strong data augmentations and ranking loss functions to generate visual representations that work well for a wide variant of downstream tasks, e.g., Chen et al. 2020 SimCLR.\n\n3) The paper also has some similarity with the notation of isolating factors of variation and the disentanglement literature, e.g., Locatello et al. 2019. In some sense we can think of the underlying problem as identifying the factors of variation in the target dataset that is not present in the background dataset. There is a large literature on disentanglement and representational learning that is relevant to this work. \n\nFrancesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ratsch, Sylvain Gelly, Bernhard Scholkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsu-pervised learning of disentangled representations. InInternational Conference on MachineLearning, 2019.\n\n4) There is no error bars or statistics for the results reported in Fig. 4. The baseline used is just a concrete autoencoder. Stronger baselines would make the paper strong. \n\n5) The experiment to measure disentanglement is a bit inconclusive. The main idea is that the target features should do poorly in generating the background dataset since it captures both the background and the salient latent variables. The poor reconstruction could be due to many other reasons in addition to capturing salient variables.\n\n6) For grassy MNIST, I am surprised that the authors use a simple MLP architecture. We typically use CNNs and ResNets for imaging data. The use of simple MLP seems like it may not learn much about complex factors of variation. It would be good to provide details on the experimental parameters such as batch size, learning rate, and number of epochs. \n", "summary_of_the_paper": "This paper is about feature selection in a weakly supervised setting using a background and target dataset and the goal is to learn features that are specific to the target dataset. The paper formulates the problem using a latent vector for data generation. There is a background latent vector z and a salient vector s. For the background model, the salient vector s is fixed where it has variations specific to the target. The general idea of this paper is to learn an encoding function that takes the input data and generates a low dimensional embedding g_z to captures the variation in z and not the variations in s. We define a loss function that takes this z along with the features to reconstruct the original target points. The main challenge here is that some of the salient features may leak through g_z and we want to avoid this. The paper presents different methods to learn these functions, and the validation is done on a semi-synthetic dataset and a biomedical dataset.", "summary_of_the_review": "Overall the paper addresses and important problem, but the proposed methods and the experimental results need significant improvement and clarification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CxPhNQD1ppH", "reviewer": "Reviewer_UFq8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is communicating the proposed approach and conducted exercises in a very clearly and concise manner. Mentioned related work is relevant and credits to prior studies are given appropriately. Both the formulation and solutions proposed appears novel.\n\nEmpirical evaluation is performed on augmented MNIST problem (synthetically fused digits with images of grass), where accuracy is measured for different hyperparameter settings, as well as on 4 unaltered biomedical datasets. Proposed approach (in two out of three flavors) showed superior performance in terms of accuracy. Baseline feature selection methods used are exclusively unsupervised, while proposed approach has \"unfair advantage\" of being trained in a \"Contrastive Analysis\" setup. That is, proposed Contrastive Feature Selection had a certain insight into downstream problem of classification. It would have been interesting (and highly relevant) to see how would the method compare to the supervised feature selection approaches (e.g using target and background sets as a binary classification problem).\n\n", "summary_of_the_paper": "A new formulation of selecting the top k informative features, when two sets of samples are available: one containing background patterns and patterns of \"potential\" interest, and the other having background patterns only. Proposed objective is quadratic error minimization between the original example and vector obtained using reconstruction function applied on background embedding vector concatenated with k selected original features. Three different ways of learning the reconstruction and background functions was proposed and evaluated on several datasets (synthetic and real world ones) against unsupervised baselines.", "summary_of_the_review": "Even though I think the approach can be valuable, my main concern is the motivation to use it when background and target sets are already known. How does its utility compares to supervised feature selection methods? Can Contrastive Feature Selection be applicable to the cases when the target and background samples are not given (or clearly separated)?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received a majority voting of rejection. During the internal discussion, all reviewers insisted their original scores. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the initial recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Research Problem**

In this paper, the authors consider a novel scenario that feature selection in the contrastive setting, where an extra *background* dataset is utilized to remove the background noisy features. However, this problem can be easily handled with a fully supervised feature selection method, where the samples in the *target* datasets are annotated as 1 and the samples in the *background* datasets are annotated as 0. Therefore, the research problem addressed in this paper is not novel. Reviewer UFq8 and ft7b held the same opinion. 

**Technical Points**

The technical part could be more informative. The whole framework is based on auto-encoder based self-reconstruction, where the feature selection is finished by the recent CAE model. In my eyes, the major contribution of this paper lie in learning $g_z$, the background representation function. To achieve this, the authors proposed three strategies, *joint*, *pretraining* and *gates*. The *pretraining* idea does not involve any information from the target dataset, where the background representation function is a general one and it has no relationship with the target dataset. I believe the concept of background should be defined based on the target dataset. The *joint* idea suffers from the information leak, which was pointed by the authors. We can also see the inferior performance of the joint model, comparing with two other models. Unfortunately, the philosophy of *gates* is unclear.

**Experimental Evaluation**

(1) The authors only compared with one supervised method on the semi-synthetic dataset. No results of supervised methods on real-world datasets were reported. (2) The performance with different numbers of selected features were not reported.
Decision: Reject