Feature Selection in the Contrastive Analysis Setting
Authors: Ethan Weinberger
Ian Connick Covert
Su-In Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Feature selection
contrastive analysis
computational biology
representation learning
information theory
TL;DR: We propose a method for selecting features that best capture variations in enriched in one "target" dataset compared to a related background dataset

Abstract: Contrastive analysis (CA) refers to the exploration of variations uniquely enriched in a _target_ dataset as compared to a corresponding _background_ dataset generated from sources of variation that are irrelevant to a given task. For example, a biomedical data analyst may wish to find a small set of genes to use as a proxy for variations in genomic data only present among patients with a given disease (target) as opposed to healthy control subjects (background). However, as of yet the problem of feature selection in the CA setting has received little attention from the machine learning community. In this work we present contrastive feature selection (CFS),
a method for performing feature selection in the CA setting. We motivate our approach with a novel information-theoretic analysis of representation learning in the CA setting, and we empirically validate CFS on a semi-synthetic dataset and four real-world biomedical datasets. We find that our method consistently outperforms previously proposed state-of-the-art supervised and fully unsupervised feature selection methods not designed for the CA setting. An open-source implementation of our method is available at https://github.com/suinleelab/CFS.

Feature Selection in the Contrastive Analysis Setting
Ethan Weinberger
Ian C. Covert
Paul Allen School of Computer Science
Department Computer Science Stanford University Stanford, CA 94305
University of Washington
Seattle, WA 98195 eweinêcs, www.wwwwt..nnn..o... edu
icovert@stanford, edu
Su-In Lee
Paul G. Allen School of Computer Science
University of Washington Seattle, WA 98195
Sesinleee@cs..wshhnngtee edu
Abstract
Contrastive analysis (CA) refers to the exploration variations uniquely enriched in a largel dataset as compared to corresponding background dataset generated from sources variation that are irrelevant to a given task. For example, biomedi- cal data analyst may wish to find small set of genes to use proxy for variations in genomic data only present among patients with a given discase (target) as op- posed healthy control subjects (background). However, of yet the problem of feature selection in the CA setting has received little attention from the machine learning community. In this work present contrastive feature selection (CFS), a method for performing feature selection in the CA setting. We motivate our ap- proach with novel information-theoretie analysis of representation learning in the CA setting, and we empirically validate CFS on a semi-synthetic dataset and four real-world biomedical datasets. We find that our method consistently outperforms previously proposed state-of-the-ar supervised and fully unsupervised feature selection methods not designed for the CA setting. An spen-s implementation of our method is available at https:://// //gi trigthhu.... com/ wwtp///unleeeeabbeem
Introduction
In many scientific domains, it increasingly common to analyze high-dimensional datasets consisting of observations with many features 2 2 3). However, not all features are created equal: noisy, dataset and impair the performance of machine learning algorithms on downstream tasks. As such, it is often desired to select small number of informative features to consider subsequent analyses. For example, biologists may seek to measure the expression levels of a small number of genes and use these in variety of future prediction tasks, such as disease subtype prediction or cell type classification. Measuring only subset of features may also yield other benefits, including reduced experimental costs, enhanced interpretability, and better generalization models trained with limited data.
information-pove features can obscure important structures
While feature selection is relatively straightforward when supervision guides the process (e.g., class labels), such information often unavailable in practice. In the unsupervised setting, we can select features for specific objectives like preserving clusters [
5 or local structure in the data [ or we can aim simply reconstruct the full feature vector g However, these approaches are likely to focus on factors that dominate in the dataset, even if these structures are uninteresting for a specific
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

downstream analysis. A third setting that lies between supervised and unsupervised is contrastive analysis 9 T0 ITI which we describe below.
In the contrastive analysis (CA) setting, we have two datasets conventionally referred to as the target and background [TO]. The target dataset contains observations with variability from both "interesting" (c.g., due to a specific disease under study) and "uninteresting" (e.g., demographic-related sources. On the other hand, the corresponding background dataset (e.g., measurements from healthy patients with similar demographic profiles) is assumed to only contain variations due to the uninteresting sources. The goal of CA is then isolate the patterns enriched in the target dataset and not present in the background for further study. Here, the target- barkkoud dataset pairing provides a form of weak supervision to guide our analysis, and carefully exploiting their differences can help us focus on features that are maximally informative of the target-specific variations.
For example, suppose our goal were measure gene expression in cancer patients for subset of genes to better understand molecular subtypes of cancer. Identifying factors specific to cancerous tissues is of great importance to the cancer research community, as these intra-cancer variations are essential to study discase progression and treatment response [12). However, selecting an appropriate set of genes not straightforward, as unsupervised methods are likely be confounded by variations due to nuisance factors such as age, ethnicity, or batch effects, which often dominate the overall variation in gene expression and other omics datasets (genomics, transcriptomics, proteomics) [TO] 13 14. Moreover, fully supervised methods may not be applicable as they require detailed labels that may not be available a priori (c.g., due to the rapid rate of development and testing of new cancer treatments [5)). Thus, methods that can effectively leverage the weak supervision available in CA may be able to select more appropriate features for given analysis.
Isolating salient variations is the focus of many recent works on CA. including many that generalize
unsupervised representation learning methods to the CA setting ( [1 17 [8 19 [3. This work aims to develop a principled approach for contrastive feature selection, which represents an important use case not considered in prior work
The remainder of this manuscript is organized as follows: we begin (Section 2) with an introduction to the contrastive analysis setting and the unique challenges therein, followed by discussion of related work (Section 3). We then proceed propose CFS, a feature selection method that leverages the weak supervision available in the CA setting (Section 4). CFS selects features that reflect variations enriched in a target dataset but are not present in the background data, and it does so using a novel two-step procedure that optimizes the feature subset within neural network module. Subsequently, we develop a novel information-theorede to justify our proposed method (Section 5): under mild assumptions about the data generating process, we prove that our two-step learning approach maximizes the mutual information with target-specific variations in the data. Finally, we validate our approach empirically through extensive experiments on semi-synthetic dataset introduced in prior work as well as four real-world biomedical datasets (Section 6). We find that CFS consistently outperforms supervised and unsupervised feature selection algorithms, including state-of state-of-the-ar methods that it builds upon.
characterization of representation learning in the CA setting
2 Problem formulation
We now describe the problem of feature selection in the CA setting. Our analysis focuses an observed variable æ E Rdd that we assume depends on two latent sources of variation: a salient variable that represents interesting variations (e.g., disease subtype or progression), and separate background variable z that represents variations that are not relevant for the intended analysis (e.g.. gender demographic information). Thus, assume that observations are drawn from the data distribution p(x) = = p(x 2, s) 2, 8 3)dzss which we depict using a graphical model in Figure
Our goal is to select small number of features that represent the relevant variability in the data, or that contain as much information as possible about s. To represent this, we let S C [d] = f1......d dd denote subset of indices and æs =| {æ; E S) subset of features. For a given value & < d, we aim to find S with |S] = l where 2g is roughly equivalent to s, or is similarly useful for downstream
We note that contrastive analysis (CA) is unrelated to contrastive learning in the sense of SimCLR 20 CLIP (20. or other sulf-supe learning methods [22]. We retain the name used in prior work 9 10 16 17

---

Target
Background
2
x
71
Figure 1:
Data generating process for target and background samples. Observed values æ are generated from two latent variables: salient variable s and background variable z. Both are active the target dataset, while the salient variable is fixed
in the background data.
tasks (e.g., clustering cells by disease status). We state this goal intuitively here following prior work [TT) but later frame our objective through the lens of mutual information (Section)
What distinguishes CA from the supervised setting is that we lack labels to guide the feature selection process [23]. We could instead perform unsupervised feature selection [3), but this biases our selections towards variations that dominate the data generating process; thus, in real-world genomics datasets, we may inadvertently focus demographic features that irrelevant Instead, the CA setting provides a form of weak supervision via two separate unlabeled datasets (II]. First, we have target dataset Dr consisting of samples with both sources of variation, or which follow the data distribution p(a:) defined above (Figure []. Next, we have a background dataset D. that consists of samples with salient variation; this may represent single fixed disease status, or perhaps no disease, and we formalize these as samples from p(æ: 8 = s') for a fixed value s'. We note that individual samples from the target and background datasets are not paired and do not assume the same number of samples from the target and background datasets (i.c., [De] # [Db )). By analyzing both datasets, we hope to distinguish sources of variability specific to rather than z.
specifie disease.
The challenge we must consider is how to properly exploit the weak supervision provided by separate target and background datasets. To identify a maximally informative feature set æg, we must effectively disentangle the distinct sources of variation in the data, and we wish to do so while allowing for different dataset sizes [D,] irrelevant information z than salient factors 8, and with no need for additional labeling. Our work
and [D,|
greater variability in driven by
does by building on previous work contrastive representation learning [10 16 18 and neural 23 while deriving insights that apply beyond the feature selection problem.
feature selection
3 Related work
Contrastive analysis Previous work has designed number of representation learning methods to exploit the weak supervision available in the CA setting. Zou et al. (9] initially adapted mixture models for CA, and recent work adapted PCA to summarize target-specific variations while ignoring uninteresting factors of variability (10). Other recent works developed probabilistic latent variable
17 197 including a method based on VAEs (T]] Severson et al. [16] experimented with adding a sparsity prior to perform feature selection within their model, but the penalty was designed for fully supervised scenarios: the penalty was manually tuned by observing how well pre-labeled target samples separated in the resulting latent space, which is not suitable for the scenario described in Section 2 where such labels are unavailable. Our work thus the first to focus on feature selection in this setting, and we provide an information-theoreti analysis for our approach that generalizes to other forms of representation learning the CA setting.
models [8 16
Feature selection Feature selection has been extensively studied in the machine learning literature. see (24 25 for recent reviews. Such methods can be divided broadly into filter 27], wrapper [28 29 and embedded [30 31 methods, and they are also distinguished by whether they require labels (i.e., supervised versus unsupervised). Recent work has introduced mechanisms for neural feature
34 35), which learn feature subset jointly with an expressive model and allow a flexible choice of prediction target and loss function. Our work builds upon these methods, which represent the current state-of-the-art in feature selection. However, unlike prior works that
selection 332 3 23
3

---

Step 1)
E Do
Step
E E
D
(background)
(target)
Freeze Weights
Background Representation Encoder
Background Representation Encoder
Target Feature Selector
g
b
b
IS
Background
Target Reconstruction Network
Reconstruction
h
Network
C
C
Figure 2: Overview contrastive feature selection (CFS). Our approach consists of two-step optimization procedure. In the first step (left), we train an autoencoder which we assume is generated solely from irrelevant variations due to z. In the second step (right), train feature selector layer using best reflect salient variations due to 8, the selector layer's s output concatenated with the output of the fixed encoder g, and then passed to learned reconstruction function f. Ideally, the feature set zg chosen by the selector will capture only the factors of variation unique to the target data.
background dataset,
target dataset. To encourage the selection of features that
focus on either supervised
23
34
or unsupervised feature selection
3
36
we focus on contrastive
dataset pairs and consider how to properly exploit the weak supervision this setting.
4 Proposed method: contrastive feature selection
We aim to select k features within Rd that provide maximum information about the salient latent variable s in our data generating process (Section 2 Such a procedure must be performed carefully, standard unsupervised methods may optimize for selecting features that capture much of the overall variance in dataset but not the (potentially subtle) variations of interest due to 8.
To do
begin with the following intuitive idea: suppose have priori variable b E R with < d that serves as a proxy for (i.c., summarizes all the uninteresting variation in the data). We assume for now that b is already available, and that it should explain significant portion of x's variance, so that any remaining variance is due solely to factors of interest. We can then aim to find a complementary set of features that capture the remaining variations in not explained by b, and do so by learning the feature set in conjunction with a function that reconstructs the full vector æ. The reconstruction function f: gé+l + Rd uses both and the learned feature set æ inputs, and we can learn f and S jointly as follows:
arg min E 0, D-D,
(l, xs; x
We call this approach contrastive feature selection (CFS). Notably, eq. ( requires data with both background and salient variations, and thus it uses only the target dataset D, Intuitively, optimizing this objective will select the set of features S that best capture the variations present in the target dataset but which are not captured by the variable b.
To implement CFS must make two choices: (I) how to optimize the feature set C [d]. and (2) how to obtain b. First, assuming we already have b, we can leverage recent techniques for differentiable feature selection to learn S 23 3 Such methods select features by optimizing continuous approximations of discrete variables at the input layer of neural network: selected variables are multiplied with and allowed to pass through the gates, while rejected features are multiplied with 0. In our implementation of CFS, we use the Gaussian- based relaxation of Bernuulii

---

random variables termed stochastic gates (STG), proposed by Yamada et al. [23 denote a random vector of STG values parameterized by E IR4, so that samples of G are defined as
Let G [0,
G₁ = max(0, min(1, 1, un + ()),
where J is a learnable parameter and ( is drawn from (0, By multiplying each feature by its corresponding gate value G; and incorporating a regularization term that penalizes open gates (i.e., the number of indices with μr > 0), feature selection can be performed in fully differentiable manner. Leveraging the STG mechanism, we obtain the following continuous relaxation of eq. D
arg min 14.0
I [1f(b, x O G:0) x
(3)
where @ denotes the Hadamard product, À is hyperparameter than controls the number of selected features, and & is the standard Gaussian CDF (23).
Next, we must decide how to obtain b so that it captures irrelevant variations due to z, and ideally not those due to the salient variable s. We propose to learn it as a function of æ, so that b = g o)
and g Rd + TR! Intuitively, this is where we can leverage the background dataset, because contains only uninteresting variations due to We learn this background representation function in conjunction with a reconstruction network h TR. + Rd using the following objective:
arg min E 6.7
[1(g((;;) - x012
Previous representation learning methods for CA, such as the contrastive VAEs of Abid and Zou [ and Severson et al. [16). also considered learning two representations; however, they learned both representations jointly by training simultaneously with target and background data. In our feature selection context, this joint training setup would entail optimizing with the sum of eq. (3) and eq. 4 so that g(x; Φ) is updated based on both D, and D,. This is potentially problematic, because the background representation b g(æ; σ) may absorb salient information that then cannot be learned by Øg. Indeed, subsequent work observed that such joint training procedures may fail to properly segregate shared and target-specific variations, with target-specifie variations being captured by the background representation function and vice versa (37).
In order to avoid this undesirable "leakage" of salient information into our background representation, we thus separate our objectives into a two-stage optimization procedure. For given target- dataset pair, we use the first stage to train autoencoder network using only background samples (eq. (4). In the second stage, we then use the encoder network b g(æ; Φ) as our background proxy, freeze its weights, and learn a feature set Vg by optimizing eq. using only target samples. We depict this procedure graphically in Figure 2 In the following section, we further motivate this two-stage approach by developing in the CA setting, which illustrates why this approach compares favorably to joint contrastive training [1] 16), well fully unsupervised feature selection approaches 3
cetmmmation-thhorrtte formulation of representation learning
5 Information-theoreti analysis
In this section, we motivate our approach from Section 4 through framework of mutual information maximization. To justify learning two representations that are optimized in separate steps, we frame each step as maximizing specific ultimate goal of representing the salient variation via feature set æss.
ceermation-thooretin objective, which we then relate to our
We simplify analysis by considering that all variables (æ,, 2) all discrete rather than continuous, which guarantees that they have non-negative entropy (38). The learned representations deterministic functions of æ. and we generalize second step be an arbitrary variable a rather than a feature set æs. Thus, our analysis applies even beyond the feature selection setting.
analysis by letting the representation from the
While we focused on the STG layer in this work, CFS can be implemented with other choices of selection layers (e.g. the concrete selector layer of (3). We refer the reader Appendix A for further details.
5

---

5.1 Analyzing our approach
Before presenting our main results, we state two sets of assumptions required for our analysis. Regarding the data generating process (see Figure [ we require mild conditions about the latent variables being independent and about each variable being explained by the others.
data distribution.) We assume that the latent variables are roughly independent, that æ is well-explained by the latent variables, and that each latent variable is well-explained by and the other latent variable. That is, we assume a small constant e 0 Oucch that [s:; z) S 6, (ax 8,2) S, H(s xt 2) S 6, and ( x. x8,8) S
Assumption 1. (Assumptions
Intuitively, we require independence that salient information is not learned in the first step with b and then ignored by a: the other conditions guarantee that s can be recovered at all via æ. We also require an assumption about the learned representations, which is that they do not affect the independence between the latent variables 8, 2. This condition is not explicitly guaranteed by our approach, but it also is not discouraged, so we view as mild assumption.
learned representations.) We assume that 8, remain roughly independent after conditioning on the learned representations a.b. That is, we assume a small constant O such that (s;; b) and (s;; aa) c.
Assumption 2. (Assumptions
With these assumptions, we can now present our analysis. First, we re-frame each step of our approach in an information- fashion:
The first step in Section 4 is analogous maximizing the mutual information I(b; |s s'). We do this for only a single value s' in practice, but for our analysis we consider that this is similar to [[b; s). which would be achieved if we optimized b over multiple background datasets with different values. This holds because [(b; æ 8) E, [I b; x
s = s)].
The second step in Section is analogous to maximizing the mutual information ((a, b; ar). However, by treating b as fixed and rewriting the objective as I(a, b, æ) [(a; x b) + [(b; x))
we see that we are fact only maximizing [(a; x: b). or encouraging a to complement b.
Finally, our true objective is for to be equivalent to 8, which we view as maximizing I(a: s). If their mutual information is large, then a is as useful as in downstream prediction tasks [38] The challenge in CA optimizing this when we lack labeled training data.
Now, we have the following result that shows our two-step procedure implicitly maximizes I(a; 8), which is our true objective in contrastive analysis (proof in Appendix
Theorem 1. Under Assumptions 7 and 2 learning b by maximizing I(b; 8) and learning a by
maximizing I(a; x b) yields the following lower bound on the mutual information I(a; s):
1a;; b) + 1 l(b;
(()) - 4e S. (aa:s).
The result above shows that our true objective is related to [(b; x | 8) and ((a; x represent the two steps of our learning approach. Specifically, the sum of these terms provides lower bound on I(a; s). so maximizing them implicitly maximizes I(a; s) (similar to the ELBO in variational inference (39). Within the inequality, H(z) acts rough upper limit T(b; x s). and the constant from Assumptions [aand contributes contrbuutt to looseness in the bound, which becomes tight in the case where O. This result requires several steps show, but is perhaps intuitive that letting serve as a proxy for z and then learning a as a complementary variable helps recover the remaining variation, or encourages a large value for sa)
b). which
5.2 Comparison to other approaches
We now compare our result for our two-stage approach with two alternatives: (1) simpler one- stage "joint" training approach similar to previously proposed contrastive VAEs [6] and (2) an unsupervised learning approach similar to the concrete autoencoder of Balın et al. 3.
First, the joint training approach learns a and b simultaneously rather than separately, which we can view as optimizing one objective corresponding to each dataset (background and target). Following the argument in Section 5.1 the background data is used roughly to maximize [(b; x s). However,

---

the target data is used to maximize I(a, b; æ) with respect to both and b rather than just a. Thus, we can modify the inequality in eq. (5) obtain the following lower bound on I [(a;
I(a, x) + 1(b ()
[ x
(z))) 4e S [ s).
(6)
Maximized during training
This follows directly from Theorem and the definition of mutual information [38]. By maximizing I (a, b; (b; x) ( b) rather than just / (a; x: | 6), we may inadvertently reduce the lower bound through (b;; æ) and therefore limit the salient information that a is forced to recover. Indeed, our experiments show that the joint training approach results in lower-quality features compared to our proposed two-stage method (Section ()-
Next, we consider fully unsupervised methods that learn single representation a. Such methods do not leverage paired contrastive datasets, so we formalize them as maximizing the mutual information [(a; m) which is natural when is used reconstruct S 40) Intuitively, if most variability in the data is due to irrelevant variation in 2, these methods are encouraged to maximize T(a; z) rather than 1 (a; s). This can be seen through the following result (see proof in Appendix B
Theorem When we learn a single representation a by maximizing [(a; Φ), we obtain the following simultaneous lower bounds on T(a; s) and [(a; z):
((a; x) H (x) + / (x; 8) < I(a; 8) I(a; a:) Hxx) + [(:; 2) T(a: z).
The implication of Theorem 2 that if explains more variation than 8 in the observed variable æ, or if we have [x:; x) [(x; s), then we obtain a higher lower bound for [(a; z) than [(a; s). This is undesirable, because the mutual information with x roughly decomposes between s and z: as we show in Appendix B Assumptions
[ aadd implly that we have [L(a; s) + I(a; z) I(a: x) S 2e.
We conclude that under this information- theorrttc perspective and several mild assumptions, we can identify concrete benefits to two-stage approach that leverages paired contrastive datasets.
5.3 Caveats and practical considerations
There are several practical points to discuss about our analysis, but we first emphasize the aim of this section: our goal is to justify two-stage approach that leverages paired contrastive datasets, and to motivate performing the optimization in these steps separately rather than jointly. Our approach provides clear practical benefits (Section ( and this analysis provides lens understand why that is, even if it does not perfectly describe how the algorithm works in practice.
The caveats to discuss are the following. (1) In practice, we minimize mean squared error for continuous variables rather than maximizing mutual information; however, the two are related, and Appendix C shows that minimizing mean squared error maximizes a lower bound on the mutual information. (2) Our analysis relates to an arbitrary variable a rather than a feature set Øz. This is not an issue, and in fact implies that our results apply more general representation learning and could improve existing methods [T]. (3) The assumptions we outlined may not hold in practice (see Assumptions T and 2). However, this represents a legitimate failure mode rather than issue in our analysis: for example, if the latent variables are highly dependent, the second step may not recover meaningful variation from because this will be captured to some extent by b in the first step.
6 Experiments
Here, we empirically validate our proposed CFS approach by using it to select maximally informative features in multiple bagget-bcckrouudd dataset pairs that were analyzed in previous work. We begin by considering a semi-synthetic dataset, Grassy MNIST [10), which lets us control the data generation process and the strength of uninteresting variations; we then report results on four publicly available biomedical datasets. For each prgge-bbcckround dataset pair, samples in the target dataset have ground-truth class labels related to the salient variations, which we use to evaluate the selected features. However, distinguishing these classes may be difficult using methods not designed for CA.
7

---

Tbroet
Backaround
CFS Pretrained
CAF
DUPS
D
7
STG
LasseNitt
84
6
Figure 3: a-b, Example target (a) and background (b) images from the Grassy MNIST dataset. Our goal here is to select the set of features that best capture the turget-dataset--pecifi variations due to the MNIST digits. c-d, l- 20 features selected by CFS (c) versus unsupervised (CAE DUFS 36 and supervised (STG 23 LassoNet (4 baselines (d).
To illustrate the benefits of leveraging the weak supervision available in the CA setting, compared against unsupervised feature selection methods applied to our target datasets. In particular, we considered two recently proposed methods that claim state-of-the-art results in fully unsupervised settings: the concrete autoencoder (CAE) [3] and differentiable unsupervised feature selection (DUFS) [36)- Moreover, illustrate that naive adaptations of previous methods designed for fully supervised settings are not sufficient to fully exploit the weak supervision available in the CA setting, we compared against the state-of-the-an fully supervised methods STG trained to simply classify samples as target versus background.
23 and LassoNet [34]
Finally, to illustrate the importance of carefully learning CFS's background representation function (as discussed in Cantions[ann[]] we experimented with three variations of CFS. First, we performed the two-stage approach proposed in Section 4 where g is pretrained on background data (CFS- Pretrained). Second, we tried learning 9 jointly with our feature set (CFS-Joint), similar to previous work representation learning for CA [11]6]] Finally, in an attempt to combine the benefits of our proposed approach (i.e., improved separation salient and irrelevant variations) with the simplicity of joint training (i.c., the ability carry out optimization in single step), we experimented with modification of the joint training procedure: for target data points, we applied a stop-gradient operator the output of the background representation function g (CFS-SG). By doing so, g is only updated for background data points and thus is not encouraged to pick up target-specific variations.
We refer the reader to Appendix D for details model implementation and training For all experiments we divided our data using an 80-20 train-test split, and we report the mean standard error five random seeds for each method.
6.1 Semi-synthetic data: Grassy MNIST
The Grassy MNIST dataset, originally introduced in Abid et [10] is constructed by superimposing pictures with the "grass" label from ImageNet [41] onto handwritten digits from the MNIST dataset 42.. For each MNIST digit, randomly chosen grass image was cropped, resized be 28 by 28 pixels, and scaled to have double the amplitude of the MNIST digit before being superimposed onto the digit (Figure 3). For a background dataset we used a separate of grass images with no digits (Figure 3p), and our goal here is to select features that best capture the target-specific digit variations.
We began by using CFS variants with fixed background dimension I 20 and baseline models to select = 20 features. Qualitatively, the features selected by CFS (Figure 3D; Figure S1a-b) concentrate around the center where digits are found, as desired. On the other hand, features selected by baselines (Figure ()) are undesirably located far from the center where digits are never found (CAE), are clustered in only subset of locations where digit-related variations can be found (STG, DUFS), or both (LassoNet). These results illustrate the unsuitability of fully unsupervised methods for CA, as such methods may select features that contain minimal information on the target- specifcc salient variations. Moreover, these results illustrate that naive adaptations of supervised methods are not suitable for CA, as features that are sufficient to discriminate between target and background samples do not necessarily contain maximal information on the target-specific salient variations.
Coode for reproducing results is available in supplementary materials.

---

Downstream accuracy
Dowesstree accurecyy
Downstream accuracy
CAR JUn
DUPS (U)
STG 15)
Lassolett
CPS-trutraind
21
CPS-jornt (C)
20
Number of featunes
Ratio of noise to digit
Backgraund
Figure 4: a, Quantitative performance comparison of CFS versus baseline supervised (S) and unsupervised (U) methods on Grassy MNIST for varying numbers of features k. b, We vary the relative contribution of grass noise to the dataset and assess each method's performance when used select k- = 20 features. c, We select k = 20 features on the original Grassy MNIST dataset and vary CFS's background representation dimension / understand the impact of this hyperparameter.
Mice Protein
Epithelial Call
AML Treatment
ECCITE-s09
0.85
0.53
083
0.92
Downstreenn
075
035
045
212
10 -I-
20
48
Number features
48
Number features
32
16
Number festures
-I--
CAE (U)
DUFS (U)
STG (5)
I NastoNet (5)
CFS-Protrained (C)
I
CFS-SG (C)
-I-
CFS-joint (C)
Figure 5: Benchmarking CFS and baseline supervised (S) and unsupervised (U) methods on four
real-world biomedical datasets.
We then evaluated the quality of features selected by each method by training random forest classifiers to predict the digit (i.e., 0-9) in each image, using only the le selected features. We report the accuracy of the classifiers on a held out test set in Figure 4 for varying values of k. We find that our pretrained and stop-gradient CFS models outperform baseline methods, with minimal gap between them. On the other hand, CFS-Joint selected lower quality features and did not always outperform baselines, reflecting the pitfalls of joint training discussed in Sections 4 We next assessed how CFS's performance varies with respect to the ratio of background noise vs. salient variation by varying the scale of grass noise versus MNIST digit: when selecting & 20 features each increment, we found that CFS continued to outperform baselines across noise scales (Figure 4 Finally, we assessed the impact of the background representation size I for our CFS variants by selecting & 20 features on the original Grassy MNIST dataset with varying values of l, and we found that our results were largely insensitive this hyperparameter (Figure 4c). We also found that this phenomenon was consistent across varying choices of ls (Figure S3
6.2 Real-world datasets
We next applied CFS and baseline methods to four real-world biomedical datasets: protein expression levels from trisomic and control mice that were exposed to different experimental conditions (Mice Protein) [3). single-cell gene expression measurements from mice intestinal epithelial cells exposed to different pathogens (Epithelial Cell) 44. single-cell gene expression measurements from patients with acute myeloid leukemia before and after a bone marrow transplant (AML Treatment) [4]. and gene expression measurements from a high-throughput single-cell gene knockout perturbation screen (ECCITE-seq) [14]. For each target dataset, obtained corresponding background collected as part of the same study. We refer readers to Appendix E for further details on each dataset.
Feature quality was again assessed by training random forest models to classify target samples into known subgroups of target points, and we report classification accuracy on a held-out test set for varying numbers of features in Figure Once again we find that pretrained and stop-gradient

---

CFS variants outperformed baseline fully supervised and unsupervised methods, demonstrating that carefully exploiting the weak supervision available the CA setting can lead to higher-quality features for downstream CA tasks. Furthermore, we find that CFS-Joint's performance was inconsistent across datasets, and that sometimes underperformed baseline models. This phenomenon further illustrates the need to carefully train the background function 9 as discussed in Sections 4 to ensure the selection of features that are maximally informative for CA.. Moreover, we found that these results were consistent for other choices of downstream classifier model, including XGBoost [46] and multilayer perceptrons (Figure S4). Using the mice protein dataset, we also performed further experiments assessing the importance of the background dataset for CFS's performance. In particular, we assessed the performance of our pretrained and gradient stopping CFS variants as the number of background samples varied, and we also compared against variant of CFS where the background encoder was fixed at a random initialization (i.e.. the background encoder was not updated during training). We found that CFS's performance initially improved as the number of background samples increased before saturating (Table (S2). and we also found that the randomly initialized CFS variant exhibited significantly worse performance for lower numbers of features (Table (S3).
Finally, to further illustrate the utility of our method for exploring real-world biomedical datasets, we performed deeper analysis of the specific genes selected by CFS on the Epithelial Cell pathogen infection dataset, and we present these results in Appendix In short, we found that CFS tended to select genes that exhibited subtle cell- cell-type-spp sppesscccific responses pathogen infection. On the other hand, features selected by baseline methods were sufficient for distinguishing between control and infected cells, but did not capture these more subtle sell-fipe-specifi effects. This result thus illustrates CFS' potential to facilitate new biological insights.
7 Discussion
In this work we considered the problem of feature selection in the contrastive analysis (CA) setting. Our goal is to discover features that reflect salient variations enriched in a target dataset compared to background dataset, and we tackled the problem by proposing CFS (Section 4 method we motivated with a new corrmation-thhored
analysis of representation learning in the CA setting (Section 5). Experiments on both simulated and real-world datasets show that features selected by CFS led to superior performance on downstream CA tasks compared to those selected by previous feature selection methods. CFS represents a promising method to identify informative features when salient variations are subtly expressed in unlabeled contrastive datasets, and our theoretical analysis may prove useful for developing and comparing other learning methods designed for the CA setting.
Acknowledgements
This work was funded by NSF DBI- and DBI-1759487 (E.W., I.C. and S.-LL.), NIH R35- PM-118838 and ROI-NIA-AG-061133 132 (E. (., I.C. and S.-.L.). E.W. was supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2140004.
References
[1] Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine Learning, 46:389-422, 2002.
[2] Aravind Subramanian, Rajiv Narayan, Steven M Corsello, David D Peck, Ted E Natoli, Xi- aodong Lu, Joshua Gould, John Davis, Andrew A Tubelli, Jacob K Asiedu, et al. next generation connectivity map: L1000 platform and the first 1,000,000 profiles. Cell, 171(6): 1437-1452, 2017.
[3] Muhammed Fatih Balı, Abubakar Abid, and James Zou, Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference Machine Learning, pages 444-453. PMLR, 2019.
[4] Pabitra Mitra, CA Murthy, and Sankar K. Pal. Unsupervised feature selection using feature similarity. IEEE Transactions on Puttern Analysis and Machine Intelligence, 24(3):301-31,, 2002.
10

---

[5] Yijuan Lu, Ira Cohen, Xiang Sean Zhou, and Qi Tian. Feature selection using principal feature analysis. In Proceedings of the 15th ACM International Conference on Multimedia, pages 301-304, 2007.
[6] Jennifer G Dy and Carla E Brodley. Feature selection for unsupervised learning. Journal of Machine Learning Research, 5(Aug):845-88) 2004.
[7] Deng Cai, Chiyuan Zhang, and Xiaofei He. Unsupervised feature selection for multi-cluster data. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 333-342, 2010.
[8] Ahmed K Farahat, Ali Ghodsi, and Mohamed S Kamel. efficient greedy method for unsupervised feature selection. In 2011 IEEE 1 Ith International Conference on Data Mining. pages 161-170. IEEE, 2011.
[9] James Yang Zou, Daniel Hsu, David C Parkes, and Ryan Prescott Adams. Contrastive learning using spectral methods. Proceedings of Advances in Neural Information Processing Systems, 2013.
[10] Abubakar Abid, Martin Zhang, Vivek K Bagaria, and James Zou. Exploring patterns enriched dataset with contrastive principal component analysis. Nature Communications, 9(1):1-7, 2018.
[11] Abubakar Abid and James Zou. Contrastive variational autoencoder enhances salient features. arXiv preprint arXiv: 1902. 00000,4661,1.... 2019.
[12] Giorgio Stanta and Serena Bonin. Overview clinical relevance of intra-tumor heterogeneity. Frontiers in Medicine, 5:85, 2018.
[13] Ethan Weinberger, Chris Lin, and Su-In Lee. Isolating salient variations of interest in single-cell data with contrastivevi. Nature Methods, pages 1-10, 2023.
[14] Efthymia Papalexi, Eleni P Mimitou, Andrew W Butler, Samantha Foster, Bernadette Bracken, William M Mauck III, Hans-Hermann Wessels, Yuhan Hao, Bertrand Z. Yeung, Peter Smibert, et al. Characterizing the molecular regulation of inhibitory immune checkpoints with multimodal single-cell screens. Nature Genetics, 53(3):322-331, 2021.
[15] Michael S Kinch. An analysis of FDA-approved drugs for oncology. Drug Discovery Today, 19 (12):1831-1835, 2014.
[16] Kristen A Severson, Soumya Ghosh, and Kenney Ng. Unsupervised learning with contrastive latent variable models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4862 4869, 2019.
[17) Adrià Ruiz, Oriol Martinez, Xavier Binefa, and Jakob Verbeek. Learning disentangled repre- sentations with reference-based variational autoencoders. arXiv preprint arXiv: 2019.
[18] Didong Li, Andrew Jones, and Barbara Engelhardt. Probabilistic contrastive principal compo- nent analysis. arXiv preprint arXiv:2012.07977, 2020.
[19] Andrew Jones, F William Townes, Didong Li, and Barbara E Engelhardt. Contrastive latent variable modeling with application to case-control sequencing experiments. arXiv preprint arXiv:2102.06731 2021.
[20] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-16077 PMLR, 2020.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR, 2021.

---

[22] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. theoretical analysis of contrastive unsupervised representation learning. In Interna- lional Conference on Machine Learning, pages 5628-5637. PMLR, 2019.
[23] Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pages 10648-10659. PMLR, 2020.
[24] Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6))1-45, 2017.
[25] Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang. Feature selection machine learning: A new perspective. Neurocomputing, 300:70-79, 2018.
[26] Roberto Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks, 5(4):537-550, 1994.
[27] Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of nax-dependency. max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, (7())1226-1233 2005.
[28] Ron Kohavi and George John. Wrappers for feature subset selection. Artificial Intelligence. 97(1-2):273-324, 1997.
[29] Gary Stein, Bing Chen, Annie S Wu, and Kien A Hua. Decision tree classifier for network intrusion detection with ga-based feature selection. In Proceedings of the 43rd Annual Southeast Regional Conference-Von vinierennee-ollume 2, pages 136-141, 2005.
[30] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
[31] Yifeng Li, Chih- Yu Chen, and Wyeth W Wasserman. Deep feature selection: theory and application identify enhancers and promoters. Journal of Computational Biology, 23(5): 322-336, 2016.
[32] Jean Feng and Noah Simon. Sparse-input neural networks for high-dimensional nonparametric regression and classification. arXiv preprint arXiv: 2017
[33] Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaic, and Emily B Fox. Neural granger causality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4267 2021.
[34] Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. LassoNet: A neural network with feature sparsity. The Journal of Machine Learning Research, 22(1))5633--661, 2021.
[35] Covert, Wei Qiu, Mingyu Lu, Nayoon Kim, Nathan White, and Su-In Lee. Learning to maximize mutual information for dynamic feature selection. arXiv preprint 27X5522301.00557. 2023.
[36] Ofir Lindenbaum, Uri Shaham, Jonathan Svirsky, Erez Peterfreund, and Yuval Kluger. Differentiable unsupervised feature selection based gated laplacian. arXiv preprini arXiv:2007.04728, 2020.
[37] Ethan Weinberger, Nicasia Beebe- and Su-In Lee. Moment matching deep contrastive latent variable models. arXiv preprint arXiv:2202. 10560, 2022.
[38] Thomas M Cover. Elements of Information Theory. John Wiley & Sons, 1999.
[39] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6113 2013.
[40] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504 2006.
12

---

[41] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255 leee, 2009.
[42] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. www.u..co//xdb//nnssttttt 2, 2010.
[43] Clara Higuera, Katheleen J Gardiner, and Krzysztof J Cios. Self-organizing feature maps identify proteins critical to learning in mouse model of down syndrome. PloS One, 10(6): c0129126, 2015.
[44] Adam L Haber, Moshe Biton, Noga Rogel, Rebecca H Herbst, Karthik Shekhar, Christopher Smillie, Grace Burgin, Toni M Delorey, Michael R Howitt, Yarden Katz, et al. A single-cell survey of the small intestinal epithelium. Nature, 551(7680)3333-33333 2017.
[45] Grace XY Zheng. Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature Communications, 8(1):1-12, 2017.
[46] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGKDD International Conference Knowledge Discovery and Data Mining, pages 785-794, 2016.
[47] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 1000v:161..00722 2016.
[48] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiio with gumbel-softmal softmax. arXiv preprini arXiv: 16611.1144 144, 2016.
[49] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1954.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026-8037, 2019.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[52] Md Mahiuddin Ahmed, A Ranjitha Dhanasekaran, Aaron Block, Suhong Tong, Alberto CS Costa, Melissa Stasko, and Katheleen Gardiner. Protein dynamics associated with failed and rescued learning in the ts65dn mouse model of down syndrome. PloS One, 10(3):e01 19491, 2015.
[53] Malte D Luecken and Fabian Theis. Current best practices in single-cell ma-sse analysis: a tutorial. Molecular systems biology, 15(6):87746, 2019.
[54] F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome Biology, 19(1):1-5, 2018.
[55] Eleni P Mimitou, Anthony Cheng, Antonino Montalbano, Stephanie Hao, Marlon Stoeckius, Mateusz Legut, Timothy Roush, Alberto Herrera, Efthymia Papalexi, Zhengqing Ouyang, et al. Multiplexed detection of proteins, transcriptomes, clonotypes and crispr perturbations in single cells. Nature Methods, 16(5):409-412, 2019.
13

---

A Choice of the feature selection layer
For the experiments presented in the main text, we implemented CFS with the STG feature selection layer proposed in Yamada et al. (23]. However, CFS does not have to be implemented with STG, and can be implemented with any choice of differentiable feature selection layer. Here we describe in
more detail another potential choice of feature selection layer: the concrete selector layer introduced Balın et al.
The concrete selector layer is based on concrete random variables 47 A concrete random variable can be sampled produce continuous approximation of one-hot vector. To sample d-dimensional concrete random variable, one first samples d-dimensional vector of i.id. samples from Gumbel distribution [49 g. Each element of our concrete variable is then defined as
exp((log a + gg)/T))
m, Σk=1 exp((log ae gx)/T)
Here temperature parameter T (0, o0) controls the extent to which one-hot vector is relaxed. In the limit 7 - O, our concrete random variable outputs one-hot vectors with m; = with probability / Σ 1= agg. Concrete random variables are differentiable with respect to their parameters via the reparameterization trick [39].
The concrete selector layer then selects features in the following manner. For each of the & nodes in the selector layer, we sample a d-dimensional concrete random variable m E {l..... k}. The ith node the selector layer (*) outputs æ (n) As T + this inner product simply becomes equal to one of the input features. After the network is trained, the concrete selector layer is replaced by an arg max layer for which the output of the ith neuron is Targ max, o,"
Before training, the selector's parameters Q, initialized small positive values encourage the layer to explore different combinations of input features. As the layer is trained, the values of o; become more sparse as the layer becomes more confident particular choices of features.
Similar to B we can train concrete selector layers with CFS using simple annealing schedule for the temperature parameter T. That is. begin training with high temperature To and gradually decay the temperature until we reach a final temperature T3 at each epoch according to a first-order exponential decay (bb) = (T (o) b/B total number of epochs.
where T(b) is the temperature at epoch and B is the
In Figure S2 we present the results of applying CFS with the concrete selector layer select features on the Grassy MNIST dataset. As with our original CFS implementation with the STG layer, we found that the pretrained and selected features around the center of images as desired.
CFS variants implemented with the concrete layer
14

---

B Proofs
In this section, we re-state and then prove our theoretical results from the main text.
Theorem 1. Under Assumptions 7 and 2 learning b by maximizing [(; x 8) and learning by maximizing I(a;3 b) yields the following lower bound on the mutual information T(a; 8):
aa; x b) + T(b;
H(2) 4c S T(a;8)
Proof. In the first step of our learning algorithm, we learn the representation by maximizing T(b; æ s). To show how this relates to maximizing [b;; z), consider the following joint mutual information, which
decompose in ways using the chain rule of mutual information (38
[(b 8, x) [b;; x) + [b;; 8
x) + [(b; x 8, 8) + I(b; = x 8).
[(: + I(b; æ
Rearranging terms, we can see the following relationship between [(b; a: 8) and [b;; z):
I(b; x 8) [bb 2) [(b: 8, 1,
I(b: z)
I(b: 8)
I(b; = x.
62
6G
We can show that each of the 8g terms is bounded near zero due to our assumptions:
81 [[b; 8, < (( 8 2) <e
+ di E [0,e]
82 [b,; 8 z) - (b; = [(s; b) 1(8;2)
+ 82 [-,,
83 T(b; 8) H( 22 <
dg E [
Thus, objective in the first step has the property that / [b; 8) R l(b; x). more precisely,
[l x s) 2e - [(z:b)= - I(b: x s) 2c.
(8)
Regarding the second step of the learning algorithm, we can use a similar argument to show relationship between [(a; z) and 8(a;8):
[(a; I(a;s) + [(a;æ 882)
[(a;21)) [()s)>;;=+ z)
I(a; 8 x 2)
Because we lack access to during this step, we must maximize T(a; x: b) instead of T(a: x 2). Fortunately, we can show that these are related using a similar mutual information decomposition:
I(a; z, x. I(;; b) + [(a; æ
b) + I(a; z xx b) =1 2) + / (a; z) [(a; b x,
Rearranging terms, get the following:
I(a; a b) [(a; x
x) + I(a; b
Φ, 2)
1(a;z) 1o--aaas ((abb)
-a(a; x.....
(10)
A.
As
Aa
15

---

Putting the results from eq. 9 and eq. (TO) together. we have the following:
{a; b) [(a; 8) A + - Ao
(1I)
Combining our assumptions with the result from eq. (8) we can show that the A; terms are bounded follows:
A [(a; a 5 (
AΑ I(a; -[== = A2 E [-c,
A3 I(a; s x, [()
4 I(a;b b ==0))
A5 [a; z) [(a;b) = [(a b) I(a: 2) + As E ( [b; s) 2c, H(z) I(b; |s) 2c)
A a(; n, H( b) H(z) - (1z)) As E (()) T(b: s) + 2c]
This yields the following two-sided inequality, which connects the empirical objective / /(a; a b) to implicit objective [(;; s):
(a;; 8) (b; 8) H(=) H(b) 6 S 1(a; b) I(a; 8) I(b; x 8) H()) + 4e.
Focusing on one-sided version of the result, we can see that the two empirical objectives [(a; æ b) and [b; æ s) control the underlying objective [(a; s) as follows:
1a; b) + I(b - H(z) 4e S ((a;s).
Theorem When we learn a single representation a by maximizing I(a; x)), we obtain the following simultaneous lower bounds on I(a; s) and [(a; z):
I(a; x) H (x) + [xx; s) < [(a: I(a; x)) (x)) [(:; =) [(a; z).
Proof. Consider the following joint mutual information, which decompose in two ways:
(a; x. [(a; s) + I(a; x I(a; x)) + [(a; x))
Rearranging terms, we can see the following relationship between / ((a; s) and I ((a; w):
T(a; 8) = [(a;; a:) + T(a; 8 () - [(a; x s)-
Bounding the remaining terms, we arrive the following two-sided result:
I(a; x) - (x s) < ((a; I I [(a; x) + H(s x).
Our result in Theorem 2 focuses the left-hand side and uses the definition of [(x;; 8). An identical argument can be used to show the same result for T(a;; z).
16

---

Proposition 1. Under Assumptions I and 2 the representation a has mutual information with a that decomposes additively between and z:
S [ a) [(a; s) I(a; 2) s 2c.
Proof. Consider the mutual information [(a; a:) for representation a generated as a function of æ, which rewrite using the data processing inequality and chain rule:
I(a; a)) (a: x. s. [(a: + [(a: = s) + [a; s,z)
={{a;3) + ((;;)) + (1(a; = 8) [a;;)) + [a; 8, (12)
Assumption implies that - I(a s.z) < H( s.2) S c for the final term. For the subtraction term, we may expect that I(a; x s) = [(a; x) because are roughly independent. We can formalize this idea using another mutual information decomposition:
I(a, 8; z) I(a; z) + [(s; z a)
2(s;2) [(a; = 8)
+ T(a; 2 8) T(a; 2) [(s; = a) [(s; z).
Assumptions and 2 imply that [[(a; 8) I(a: 2)1 S e. Substituting this into eq. final result:
12 yields
-E S [(a;z) I(a; 8) I(a; x) S 2E.
17

---

C Connection between mean squared error and mutual information
When predicting given arbitrary representation a, our loss function based on the reconstruction function is the following:
min E [llg(a) 21 x=
To minimize the loss, the best possible prediction is given by the function g(a) =E[a a). and our loss becomes the trace the expected conditional variance:
E
[|
x+]3]
Tr [
[Var(æ: a)l)
a
Optimizing the mean squared error error with respect to a can thus be understood as minimizing the trace of z's expected conditional variance.
Next, we can show that conditional variance connected to the mutual information [(a; æ). First, for a fixed value of where has conditional variance Var(æ a), we have the following upper bound on the differential entropy,
(( a) - log det (Var(æ | a)) + log()
where der((Va(2: a)) denotes the determinant the covariance matrix. This result follows from the fact that the entropy for random variables with fixed covariance is maximized by Gaussian distribution [38]. Next, assuming that Var(æ a) is positive definite, or that Var(æ: a) > 0, we can modify the bound to use the trace rather than the log-determinant of the covariance matrix:
1 Tr((Var 2
d
((w a) S
a)) log(2.).
Taking the expectation both sides with respect to a, we obtain the following relationship between the conditional entropy and expected conditional variance:
(E [ar( a))) + d log(2x).
( a) S Tr
Finally, using the relationship between the mutual information T(a; a:) and conditional entropy H(x a), we can identify the following connection between the minimum mean squared error and the mutual information achieved by
d. H(x)
2 (2g)
2 min E za
[Ilg(a) =0|²] < I(a;x))
Thus, minimizing the mean squared error with respect to can be understood as maximizing lower bound on the mutual information I(a;)) This shows that our empirical procedure, which involves minimizing mean squared error, is connected our theoretical results that are stated in terms of mutual information.
We note that this bound can be vacuous if (xx) < O, which is possible for differential entropy. because we trivially have I(a;; .) 2 even for continuous random variables. The result should therefore not be interpreted as providing a precise estimate of the mutual information: it instead demonstrates the rough intuition that improving the mean squared error related to increasing the mutual information.
18

---

D Implementation details
All experiments were peformed on system running CentOS 7.9.2009 equipped with an NVIDIA RTX 2080 TI GPU with CUDA 11.7.
CFS models were implemented using Py PyTorch [50 (version 1.13) with the PyTorch Lightning AP For all CFS variants we let our reconstruction function f be a multilayer perceptron with two hidden layers of size 512 with ReLU activation functions. For Joint and stop-gradient (SG) CFS models, we let y be a multilayer perceptron with a single hidden layer of size 128. For our Pretrained CFS models, first trained autoencoder consisting of encoder with single hidden layer of size 128 and a decoder with the same architecture in reverse. The encoder network's 's weights were then frozen and it was used as g. All CFS models were trained using the PyTorch implementation of the Adam [5]] optimizer with default hyperparameters. Batch sizes of 128 were used for all experiments.
Hyperparameters for the STG layer for the N (0, o²) distribution used for drawing noise to a = 0.3, and the regularization parameter was tuned that the desired number of gates were left open (i.e., 14: 0) after convergence.
set described in Yamada al. 23 That is, set the
We reimplemented the unsupervised concrete autoencoder (CAE), differentiable unsupervised feature selection (DUFS), and stochastic gates (STG) baselines in PyTorch Similar to CFS, for the CAE's reconstruction function we used a multilayer perceptron with two hidden layers of size 512. As in Balı al. 3]. we optimized the CAE using Adam [51] with default hyperparameters and set the temperature of the CAE's s concrete selector layer following Balın et al. [3] i.c., at the start of training, set the temperature of the concrete layer to 10, and gradually annealed the temperature
final value of 0.1 using the exponential decay schedule described in Appendix A As in [3, we trained our concrete autoencoders until the mean of the concrete samples exceeded 0.99.
As in Lindenbaum et al. [36], we trained DUFS with an SGD optimizer with learning rate of using the authors' proposed "parameter-free" loss function (i.e., eq. of Lindenbaum et al. (36)) until convergence. For given number of desired features the features with the highest gate values were then selected described in Lindenbaum et al. [36].
For the supervised STG baseline [23), used two hidden layers of 512 units. To adapt STG, which was designed for fully supervised scenarios, to the contrastive analysis setting, we trained it on the binary classification problem distinguishing target versus background points. We optimized STG using Adam with default hyperparameters, For all experiments with STG, the regularization parameter À was tuned such that STG selected the desired number of features.
For all experiments with the supervised LassoNet model 34 we used the authors' Python package As with STG, we adapted the supervised LassoNet model to the contrastive analysis setting by training on the binary classification problem of distinguishing target versus background points. The regularization parameter À was tuned select the desired number of features, and all other hyperparameters were left at their default values in the LassoNet package.
19

---

E
Dataset descriptions
Here we provide additional details on the real-world datasets considered in our empirical evaluation of CFS in the main text. We also summarize these datasets in Table ST Our code for preprocessing these datasets can be found at www. www.placeholder.. com
Mice protein expression This dataset 43 52
consists of protein expression levels from healthy control mice and mice that have developed Down syndrome. Each mouse was injected with me- mantine, subjected to shock therapy, received both of these treatments, none of them. Our goal here was to classify mice with Down syndrome based on their treatment regimens. We used data from healthy mice that did not receive any treatment as background. As done in previous work (e.g.. [3), features were min-max normalized. Raw data was downloaded from https://archiver ics. uci actu/mmmmachine htt////chheeeeeennnn databases/00342)
Epithelial cell infection response We constructed target dataset by combining two sets of gene expression measurements from Haber et al. [44]. These datasets consist of gene expression measure- ments of intestinal epithelial cells from mice infected with either Salmonella or Heligmosomoides polygyrus (H. poly). Here our goal is to separate cells by infection type. As a background dataset we used measurements collected from healthy cells released by the same authors. As is standard for single-cell gene expression measurements (see e.g., (53) for a review), raw count values were lbbaay-size-normmaizze
account for differences in sequencing depth across cells. Normalized counts r were then transformed according to log(æ + 1). All preprocessing steps for this dataset were carried out using scanpy [54], a standard Python package for analyzing single-cell RNA-seq data. Gene expression count files were downloaded from the NIH Gene Expression Omnibus at https://www. .cbi. nlm. nih. wot..ogg//goo/eeee//eeeeo /acp:/accccccoo cgi GCCC=GS22233
AML treatment Here we combined datasets from Zheng et al. 45 containing single-cell gene expression measurements from two patients with acute myeloid leukemia (AMI) be- fore and receiving a blood cell transplant. Our goal here is to learn a representation that separates pre and post transplant measurements. As background dataset we used expres- sion measurements from two healthy control patients that were collected as part of the same study. Raw counts were normalized described above for the epithelial cell infection re- sponse dataset. Files containing measurements from the first patient pre- and post- prst-transplant can be found htttta/e_b_mmarrccseeeoo http://o...aaaaaooaaaaoaaaaaaaar
100pe////s......ggoomiicc.c. com httpa///escaeeeeaaaaeeee/e///...
027_
and http:///phaaaaaaaaoaooo/ooooaaaa
respec-
tively; from the second patient pre- and com/ filteree__eene_oonnttiiieee
post-transplant httpt://tt..c.... 101genomm.01c
and https: http///ff...ggmmmmcccccco///oo/ samples/
cell htt_::ppp_..110aaa10333_ppaaaaa1 03n 005__osst________ pta.pps_trraapplat___leeeee.e
gene_
bene__bc_matriccsstar...g tawpiccss.tarrree. wtw.:ettt..tt.....oooaaeeeecceie respectively, and from the two healthy control patients at
and
https://ctt hanorr//hccrnn......maaaaaaaaooe
tareeestaar.. tar.gz
ECCITE-seq perturbation sereen ECCITE-seq (55] is a technique that allows for multimodal
readouts of single-cell perturbation screens. Here obtained ECCITE-seq dataset from Papalexi ct al. [14] that measured how various gene knockout perturbations affected the gene expression profiles of cells from the THP1 cancer cell line. Our goal here was to classify cells into one of three clusters of perturbations with similar effects identified by Papalexi et al. [1]. As a background dataset we used measurements from control cells with perturbation collected part of the same study. Raw gene expression counts were normalized as described above for the epithelial cell infection response dataset. Gene expression count files and associated metadata were downloaded from the NIH Gene Expression Omnibus at hhtpp:: /wwww/wwww..om/ ncbi. nlm. nih. gov/ wwoo/goo/quee qetee////uu/eeeeeeeeeeeeeeee http//cccccccc////// Pacc=GSE1500656
'Code available in supplementary materials and will be made public upon acceptance.
20

---

Table S1: Summary of real-workd datasets considered in the main text.
Train size
Train size (background)
Dataset
Features (d)
Test size
Classes
(target)
Grassy MNIST
784
48,000
48,000
12,000
10
Mice Protein
77
444
135
111
Epithelial cell
15,215
3,584
3,240
897
AML treatment
12,079
9,919
4,457
2,480
2
ECCITE-Seq 18,649
14,674
2,386
3,669
3
F
Deeper analysis of Epithelial Cell dataset results
To illustrate how the features selected by CFS may lead to new biological insights, we performed an additional analysis of the specific gene features selected by CFS versus supervised methods for the mice intestinal epithelial cell infection dataset from Haber et al. [44) In this dataset gene expression levels were compared between control cells (background) and cells exposed either the bacteria
Salmonellaor the parasite H. poly (target). For this analysis we compared the features selected by CFS and STG, the rest-performing supervised baseline in our quantitative experiments, with the number of features & set to 20.
We first considered the overlapping genes selected by both methods, and we found that both methods selected genes involved in the inflammatory response (e.g. Cd74) and previously studied defense responses pathogens (e.g. Reg3b. Reg3g). We next considered the genes selected only by CFS and not by STG. We found that CFS tended to select genes (Figure S5p) with clear differences in expression patterns between the two pathogens (e.g. Gm42418), including a number of enterocyte marker genes (e.g. Apoal, Guca2b, Fabpl) that were upregulated in Salmonella cells compared to H. poly cells. Notably, the enterocyte markers exhibited clear bimodal pattern their expression levels for Salmonella-infecte cells.
Upon further inspection found that the observed upregulation of these genes due combination of two phenomena. First, as noted in Haber et al. [44], Salmonella-infection resulted in increase in the number of enterocytes relative healthy and II. poly-infected populations. Despite the increase in the number of enterocytes, the distribution of expression levels for these marker genes were notably similar for control and Salmonella enterocytes (Figure (Figure[S5p). On the other hand, we found (Figure (55)) that these genes' expression levels were substantially upregulated in Salmonella- infeetedd non-enterocytes compared control non-enterocytes. This cell-type-speciff response was not noted in Haber et al. [44] and this finding thus illustrates the potential for CFS to uncover insights that may be missed with standard workflows.
On the other hand, the genes selected by STG (but not CFS) consisted immune response genes (e.g. Lgals9, Krt19) with altered expression compared to controls, but which did not exhibit other notable patterns among the perturbed cells (Figure(S3) These results indicate that supervised methods may select features that can distinguish target vs. background data, but which do not capture subtle phenomena within the target data.
21

---

G
Supplementary Figures
CFS-SG
CFS-joint
Figure S1: Qualitative results for stop-gradient (SG) and joint training CFS variants on the Grassy MNIST dataset for k 20 features.
CFS-SG (Concrete)
CFS-Pretrained (Concrete)
CFS-Joint (Concrete)
Figure S2: Qualitative results for CFS implemented with the concrete selection layer of Balm et al. 3 on the Grassy MNIST dataset for k = 20. We find that the gradient stopping (GS) and pretrained CFS variants select features concentrated towards the center as desired, while features selected by the joint training CFS variant are more spread out away from the center.
22

---

k=10
k=15
k-20
0.80
CFS (Pretrained)
Downstream accuracy
0.70
CFS (SG)
CFS (joint)
Downstream accuracy
0.70
Dewnstream accuracy
0.70
0.88
0.85
0.65
0.30
0.50
0.90
0.45
0.45
0.45
20
30
Background representation dim.
Background representation dim.
Background representation dim.
km25
km
30
0.80
0.80
0.75
0.75
Dowistreem accuracy
Downstream accuracy
0.70
0.60
0.60
0.38
0.55
0.40
0.40
15
25
10
15
20
30
Background representation dim.
Background representation dim.
Figure S3: Varying background representation dimension for various numbers of selected features k on Grassy MNIST.
Hice
Eathellal Call
b
Probein
3.00
6.65
0.0
6:00
0.
0.0
0.7
6:18
6.55
35
15
0.755
0.05
0.55
0.0%
3.00
0.006
0.70
nime
0.00
0.6%
0.72
0.55
6.2
Mamber festuree
Rumber of Seatures
Namber Seaures
Manieer fenttre
LassoNet
CFS-Pretrained
CFS-SG
CFS-joint
Figure $4: Downstream classification results using (a) XGBoost and (b) neural networks as the downstream classifier.
23

---

Figure S5: (a) Sample of gene features selected by CFS. (b) Visualizing cell-type-specife changes in gene expression in response to Salmonella infection for genes selected by CFS. (c) Sample of genes selected by the best performing supervised feature selection method STG.
24

---

H
Supplementary Tables
Table S2: Assessing the quality of features selected by CFS for varying numbers of background samples on the mice protein dataset for features. Feature quality was assessed by training a random forest classifier distinguish between known groups of target samples, and we report the mean accuracy + standard error over five random seeds on a held out test set. We find that feature quality initially increases the number of samples increases, with saturation achieved around 50 background samples for this dataset.
Number of Background Samples
CFS (Pretrained)
CFS (GS)
10
0.902 + 0.021
0.902 + 0.021
20
0.912 0.012
0.912 0.012
30
0.920 + 0.014
0.922 + 0.012
50
0.934 0.012
0.933 0.014
70
0.932 + 0.009
0.936 + 0.012
135
0.935 + 0.011
0.935 0.015
Table S3: Comparing the quality of features selected by the CFS architecture with fixed, randomly initialized background encoder (CFS (Random)) versus our pretrained and gradient stopping (GS) CFS variants on the mice protein dataset. For comparison we also include the results from STG, our best performing non-contrastive baseline. Feature quality was assessed by training a random forest classifier to distinguish between known groups of target samples, and report the mean accuracy + standard error over five random seeds on held out test set. We find that CFS (Random) significantly underperforms our pretrained and GS CFS variatns for lower numbers of features, with comparable performance to STG.
Number of Features
CFS (Random)
STG
CFS (Pretrained)
CFS (GS)
5
0.888 + 0.004
0.865 + 0.006
0.935 = 0.011
0.935 + 0.015
10
0.918 + 0.003
0.923 + 0.007
0.993 = 0.005
0.989 + 0.007
15
0.970 + 0.002
0.968 + 0.004
0.995 = 0.004
0.993 + 0.007
20
0.985 + 0.005
0.980 + 0.005
0.989 = 0.009
0.998 + 0.002
30
0.995 + 0.001
0.991 + 0.003
0.998 = 0.002
0.996 + 0.002
25


Reviews:
[{"review_id": "7vaAvLEwdx", "reviewer": "Reviewer_BGi6", "summary": "They employed a contrastive analysis method for feature selection on the target dataset, which provides variation not found in the background dataset. Contrary to previous contrastive analysis that typically involves joint modelling of background and target datasets to identify shared variables, they drew from information-theoretic analysis to demonstrate that a two-step pre-training model outperforms joint modelling.", "questions": "In the real dataset, I noticed that all classes are larger than 1. What happens if there's only one class? Can the method still find variation that distinguishes it from the background? Regarding AML treatment, if the goal is to identify features that separate post-treatment from pre-treatment, could we use pre-treatment as the background and consider the selected features from post-treatment as the ones that have changed? Alternatively, would the method identify the same features as the current settings, which require healthy patients for the background?\n\nAs for scenarios where the number of features exceeds 30, do all methods eventually achieve similar accuracy? From Figure 5, I noted that STG performs similarly to CFS in the Mice Protein and Epithelial Cells datasets, though it is slightly worse in the AML and ECCITE_seq datasets. Could you provide information on the accuracy when the number of features is increased to 50?\n\n", "limitations": "Same as above weakness. Hard to satisfy the assumption that independent features. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Feature selection is a common requirement in health data to enhance interpretability. This paper clearly explained why the method proposed by them has a better performance than previous joint modelling and have good downstream accuracy when the number of features is limited.", "weaknesses": "That method needs strong assumptions that may not hold in the real world. Assumption one states that the latent variables s, z are roughly independent, but the genes in scRNA-seq (usage case in its real data experiments) are mostly highly correlated. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully examining our work and providing their feedback. We’re thrilled to hear that the reviewer found that our manuscript “clearly explained” why CFS achieved superior performance compared to previously proposed feature selection methods. We have responded to general concerns shared across reviewers in our main response, and in the space below we respond to the reviewer’s specific concerns.\n\n**Clarifying the scope of our contribution:** The reviewer noted in their summary that a major contribution of our manuscript was demonstrating (both empirically and via our information-theoretic analysis) that our proposed two-step training procedure can lead to improved segregation of shared and target-specific variations, and we gratefully thank the reviewer for acknowledging this contribution.\n\nHowever, we also wanted to clarify that the contribution of our manuscript goes beyond just proposing the two-step training procedure. Indeed, to our knowledge, our manuscript is the **first work to consider the problem of feature selection in the contrastive analysis setting**, and our proposed CFS architecture is the first model specifically designed to take advantage of the weak supervision available in this scenario\n\nWe will make this point more clear in a revised version of the manuscript.\n\n**AML/ECCITE-seq results for more than 30 features:** As requested by the reviewer, we performed additional experiments assessing the performance of CFS and baseline methods for >30 features on the AML and ECCITE-seq experiments. Below we provide results for $k=40, 50$ features as well as $k=30$ for comparison. We found that CFS continued to exhibit superior performance compared to baseline methods for the AML and ECCITE-seq datasets, though the magnitude of the difference between CFS and the best-performing baseline methods did narrow as the number of features increased.\n\nAML:\n\n| $k$ |CAE (U) | DUFS (U) | STG (S) | LassoNet (S) | CFS-Joint (C) | CFS-Pretrained (C) | CFS-SG (C) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 30 | $0.934 \\pm 0.004$ | $0.688 \\pm 0.016$ | $0.943 \\pm 0.002$ | $0.669 \\pm 0.003$ | $0.932 \\pm 0.003$ | $0.965 \\pm 0.002$ | $\\boldsymbol{0.972 \\pm 0.001}$ |\n| 40 | $0.938 \\pm 0.002$ | $0.722 \\pm 0.017$ | $0.925 \\pm 0.008$ | $0.956 \\pm 0.002$ | $0.932 \\pm 0.003$ | $0.971 \\pm 0.000$ | $\\boldsymbol{0.973 \\pm 0.000}$ | \n| 50 | $0.952 \\pm 0.003$ | $0.754 \\pm 0.015$ | $0.942 \\pm 0.003$ | $0.964 \\pm 0.001$ | $0.932 \\pm 0.003$ | $\\boldsymbol{0.976 \\pm 0.001}$ | $0.972 \\pm 0.001$ | \n\nECCITE-seq:\n\n| $k$ | CAE (U) | DUFS (U) | STG (S) | LassoNet (S) | CFS-Joint (C) | CFS-Pretrained (C) | CFS-SG (C) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 30 | $0.857 \\pm 0.002$ | $0.743 \\pm 0.015$ | $0.860 \\pm 0.002$ | $0.835 \\pm 0.015$ | $0.849 \\pm 0.001$ | $\\boldsymbol{0.879 \\pm 0.002}$ | $0.874 \\pm 0.004$ |\n| 40 | $0.862 \\pm 0.003$ | $0.746 \\pm 0.014$ | $0.867 \\pm 0.003$ | $0.866 \\pm 0.010$ | $0.855 \\pm 0.003$ | $0.881 \\pm 0.001$ | $\\boldsymbol{0.883 \\pm 0.002}$ |\n| 50 | $0.866 \\pm 0.002$ | $0.749 \\pm 0.014$ | $0.870 \\pm 0.003$ | $0.869 \\pm 0.007$ | $0.858 \\pm 0.000$ | $\\boldsymbol{0.882 \\pm 0.002}$ | $0.879 \\pm 0.003$ |\n\n\n**What happens with only one class in the target dataset?:** We thank the reviewer for raising this point and for allowing us to clarify some potential points of confusion. First, we want to emphasize that CFS is designed for weakly supervised scenarios where we know only whether a data point is from the target or background dataset (so detailed class labels and/or a specific number of classes are not available). The goal of CFS is then to select features that best reflect target-specific variations without the aid of more detailed labels. Such a capability is valuable for analyzing many forms of biomedical data. For example, when analyzing large-scale single-cell genetic knockout perturbation screens, we may wish to identify features with a wide variety of responses to different perturbations (and which could subsequently be used to categorize the individual perturbations) without detailed knowledge ahead of time of how the various perturbations affect each feature.\n\nTo accomplish this, CFS selects features that are most helpful for reconstructing target points given their low-dimensional background representations. To validate CFS’s performance, we assessed how well the features selected by CFS could distinguish between known ground truth classes of target points. However, when there is only one ground truth class of target points, the reconstruction objective will still encourage it to select features that exhibit variations unique to the target dataset (and which may distinguish target vs. background data). To illustrate this point, we trained CFS models on a modified version of the Grassy MNIST dataset, in which the target dataset consisted solely of images with the “1” digit. We found (**Fig. 3** in the rebuttal PDF) that CFS indeed selected features only in foreground regions where the “1” may be found. Similarly, if, as suggested by the reviewer, pre-treatment cells from the AML dataset were used as a background with post-treatment cells as the target, CFS would likely select features that distinguish pre- versus post-treatment cells."}, {"review_id": "m5SqsPtIVl", "reviewer": "Reviewer_xS67", "summary": "The authors tackle the important problem of feature selection in a contrastive setting. Supervised and unsupervised feature selection problems have been extensively studied and have many high-impact applications. In this work, the authors design an algorithm for finding features that are more enriched in a target set than a background set (contrast). This type of comparison is widely used in biology, for example, in differential expression gene (DEG) analysis, where biologists look for differentially expressed genes across different medical conditions. The proposed method is based on a two-step procedure; in the first step, an autoencoder is trained to learn the latent information required to represent the background data. In the second step, a feature selection module is trained to identify the set of features required to capture the additional information needed to represent the target data. To sparsify the feature selection module, they use the recently proposed stochastic gates, which are continuously relaxed random variables. The new approach is motivation using a mutual information perspective. Several real and semi-synthetic datasets are used to evaluate the approach.  ", "questions": "Some parts of the paper are very similar to [37], for example, two first sentences in the abstract and figure 1. Propper credit should be given here.\nWas the 80-20 percent split performed for FS or only for the classification part? This is not clear from the text.\nWhat are the architectures used for learning the background? Target? And also for STG and CAE?\nHow are those tuned? \nWhile the authors demonstrate the merits of the approach for improving class separation (without access to class labels), I’m missing an evaluation that is tied to the motivation. Specifically, an example is that the approach is beneficial for DEG. As mentioned, this could be done using synthetic or real data with known gene markers. This would substantially improve the paper.\n", "limitations": "The limitations of the method are not discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is important with many applications in bioinformatics. The proposed is new and seems highly appropriate for the suggested problem. The experimental results highlight that the approach can lead to advantages compared to existing schemes. The authors propose a few variations for training the method and compare their quality. The benefits of the two-stage approach are justified using mutual information under mild assumptions. ", "weaknesses": "Conceptually the problem could be solved using supervised feature selection; the authors also use supervised FS models as baselines. I understand that in terms of some applications, the setting is different. However, this is not well explained in the paper. Specifically, the authors should add information and results on how the selected features differ for the contrastive analysis case compared with the supervised (or unsupervised) case.  Accuracy is important, but interpretability is vital in the examples motivated by the authors. This could be done using synthetic data or by benchmarking the method (and others) on a previously studied DEG problem. This will significantly improve the paper and help increase its impact and use in bioinformatics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for examining our work and providing their feedback. In the space below we respond to the reviewer’s specific concerns.\n\n**Train/test split:** The training points from the 80/20 test split were used to select features using CFS and train the subsequent classifiers, while the test points were only used to evaluate the performance of the final downstream classification models. We will clarify this  in the revised manuscript.\n\n**Clarifying the applications of contrastive analysis:** We thank the reviewer for pointing out the potential confusion on this point. To clarify, in the contrastive analysis setting a data analyst is interested in isolating patterns enriched in target samples compared to a corresponding set of background samples generated from “uninteresting” sources of variation. Previous works have shown that isolating these variations from confounding sources of variations shared with the background can lead to new biological insights, including in large-scale genetic perturbation screens [1, 2], and on autism spectrum disorder from MRI scans [3]. While previous works have focused on representation learning (e.g., via variational autoencoders) for CA, to our knowledge CFS is the first method designed for feature selection in the CA setting, which may provide increased interpretability.\n\nAs done in our manuscript, fully supervised selection methods can in theory be naively applied to this problem by selecting features that discriminate between target and background points. However, as seen in our qualitative Grassy MNIST results in Figure 3 of the original manuscript, features that discriminate between the target and background often do not reflect the full spectrum of target-data-specific salient variations. Thus, methods designed to better exploit the weak supervision in CA may select features that reflect a wider variety of target-dataset-specific phenomena (e.g. fine-grained target-dataset-specific cellular states), and lead to additional insights.\n\nTo illustrate this point, we provide the results of a deeper analysis of the features selected by CFS  for the mice intestinal epithelial cell infection dataset from Haber et al. [4] below, and we will make sure to clarify these nuances in the final manuscript.\n\n**Deeper analysis of features selected by CFS vs. supervised methods for epithelial cell dataset:** To illustrate how the features selected by CFS may lead to new biological insights, we performed an additional analysis of the specific gene features selected by CFS versus supervised methods for the mice intestinal epithelial cell infection dataset from [4]. In this dataset gene expression levels were compared between control cells (background) and cells exposed to either the bacteria Salmonella or the parasite H. poly (target). For this analysis we compared the features selected by CFS and STG, the best-performing supervised baseline in our experiments, with the number of features $k$ set to 20. We refer the reviewer to the rebuttal PDF for additional supporting figures.\n\nWe first considered the overlapping genes selected by both methods, and we found that both methods selected genes involved in the inflammatory response (e.g. *Cd74*) and previously studied defense responses to pathogens (e.g. *Reg3b*, *Reg3g*). We next considered the genes selected only by CFS and not by STG. We found that CFS tended to select genes (**Fig. 4a** in the rebuttal PDF) with clear differences in expression patterns between the two pathogens (e.g. *Gm42418*), including a number of enterocyte marker genes (e.g. *Apoa1*, *Guca2b*, *Fabp1*) that were upregulated in Salmonella cells compared to H.poly cells. Notably, the enterocyte markers exhibited a clear bimodal pattern in their expression levels for Salmonella-infected cells.\n\nUpon further inspection we found that the observed upregulation of these genes was due to a combination of two phenomena. First, as noted in [4], Salmonella-infection resulted in an increase in the number of enterocytes relative to healthy and H.poly-infected populations. Despite the increase in the *number* of enterocytes, the distribution of expression levels for these marker genes were notably similar for control and Salmonella enterocytes (**Fig. 4b** in rebuttal PDF).  On the other hand, we found (**Fig. 4b** in PDF) that these genes’ expression levels were substantially upregulated in Salmonella-infected non-enterocytes compared to control non-enterocytes. This cell-type-specific response was not noted in [4], and this finding thus illustrates the potential for CFS to uncover insights that may be missed with standard workflows.\n\nOn the other hand, the genes selected by STG (but not CFS) consisted of immune response genes (e.g. *Lgals9*, *Krt19*) with altered expression compared to controls, but which did not exhibit other notable patterns among the perturbed cells (**Fig. 4c** in rebuttal PDF). These results indicate that supervised methods may select features that can distinguish target vs. background data, but which do not capture more subtle phenomena within the target data.\n\nThis extended analysis will be included in our revised manuscript.\n\n**Similar language to [37]**: We thank the reviewer for pointing out this issue. These sentences providing basic background info on CA were included when drafting early versions of the manuscript and accidentally remained in our submission. They will be rewritten and/or given proper credit in the final version. Thank you again for pointing out this issue!\n\n[1]: Jones, A., et al. Contrastive latent variable modeling with application to case-control sequencing experiments\n\n[2]: Weinberger, E., Lin, C. & Lee, SI. Isolating salient variations of interest in single-cell data with contrastiveVI\n\n[3]: Aglinskas, A. et al., Contrastive machine learning reveals the structure of neuroanatomical variation within autism\n\n[4]: Haber, A.L., et al. A single-cell survey of the small intestinal epithelium\n"}, {"review_id": "PFN2ZpZyk4", "reviewer": "Reviewer_SoSQ", "summary": " The paper aims to solve the problem of “salient” feature selection in a Contrastive Analysis setting, termed “Contrastive Feature Selection.” The authors' primary focus lies in uncovering distinctive features that highlight salient variations present in a specific target dataset that are absent in a standard background dataset. The paper discusses & compares the limitations of popular feature selection methods both in supervised & unsupervised methods settings and further attempts to formulate a “Contrastive Feature Selection” method to extract those salient features. The paper provides an initial empirical formulation based on intuitions, then formalizes the idea using principles of Information Theory (bounds on Mutual Information, Entropy)  and finishes with empirical results on semi-synthetic (vision) and real-world (medical) datasets. Overall, the paper has made a novel attempt at feature selection in a weak supervision setting using a background and target dataset, with the goal being to learn features specific to the target dataset.", "questions": "1. Is adding experiments based on Weakness Section Points 1-2 possible?\n2. Also, is it possible to provide a theoretical explanation or intuitive insight why the two variants of CFS-SG and CFS-Pretrained result in a similar performance, but the top performer changes when tested across datasets?  Is it related to some factor like dataset sizes/ number of classes for classification?\n", "limitations": "1. Limited evaluation of real-life classification datasets. All datasets had <= ten classes.\n2. Validation of the selected feature performance when using different classifier models.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper introduces a \"contrastive feature selection\" method. The authors provide experimental validation with popularly used baselines (both supervised+unsupervised)  employed in feature selection. \n2. The paper also provided intuitive, experimental, and mathematical proofs (based on Information Theoretic bounds) to justify their proposed method.\n3.  Another strength of the proposed approach is it relies only on weak supervision (whether a data sample is part of a background/target dataset).\n4. The results show the superior performance of the proposed feature selection method for downstream tasks.\n5. Code provided for reproducibility. ", "weaknesses": "1. The authors use only \"Random Forest Classifiers\" for their downstream tasks. It would be a good idea to add comparative results (perhaps in the Supplementary) using other popular classifiers (like Decision Trees, SVM, ANNs, XGBoosts, etc.) to show that the feature selection works across different classifiers. \n2. The paper shows empirical results on the \"Grassy-MNIST\" and four other medical datasets. However, results on more practical classification tasks (>10 classes) would make more sense. Also, it would be interesting to see the performance variation for a particular dataset like GrassyMNIST as background vs. target dataset size is varied.   \n3. A more extensive ablation of varying dimensions of the latent {k,l} is missing.\n4. The overall organization of the paper can be improved. It feels a bit unorganized going through the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for examining our work and providing their feedback. We’re thrilled to hear that the reviewer found that CFS represents a “novel attempt at feature selection” and that our results  show “the superior performance of the proposed feature selection method”. We have responded to general concerns shared across reviewers in our main response, and In the space below we respond to the reviewer’s specific concerns.\n\n**Additional downstream classifiers:** As requested by the reviewer, we have re-computed our downstream classification metrics using two alternative classifiers: XGBoost and neural networks, and we provide the results for the real-world biomedical datasets in **Fig. 1** in the one-page rebuttal PDF. We found that the results with these new classifiers were largely consistent with the original results using random forests, and we will include these new results in the final version of our supplementary materials.\n\n**Additional ablations of background representation dimension vs. number of features:** We thank the reviewer for this suggestion. To satisfy the reviewer’s concerns, we performed further experiments on the Grassy MNIST dataset measuring CFS’s performance (as measured by downstream classification accuracy for the selected features) for each combination of background representation dimension $l \\in [10, 15, 20, 25, 30]$ and number of selected features $k \\in [10, 15, 20, 25, 30]$.  We present these results in **Fig. 2** in the rebuttal PDF document. We found that, for a given number of target features $k$, performance was largely consistent with respect to the choice of background representation dimension $l$.\n\n**Evaluations on \"more realistic\" (>10 classes) datasets:** We thank the reviewer for raising this point. Despite our efforts, we were unable to find a dataset that is both suitable for contrastive analysis (i.e., has an obvious target versus background) and which contains more than 10 ground truth class labels. However, we emphasize that even for datasets where the downstream classification task should be relatively “easy” (e.g. only two ground truth classes), CFS often substantially outperformed supervised and unsupervised baselines. For example, the features selected by the pretrained and gradient stopping CFS variants for the epithelial cell dataset with $k = 5$ led to downstream classification accuracy > 97.5%, while the next best baseline model achieved 91.8%. \n\nMoreover, to further demonstrate the utility of CFS in real-world scenarios, in the final manuscript we will include new results from a detailed case study analyzing the features selected by CFS when applied to the mice intestinal epithelial cell infection dataset from Haber et al. [1] (see response to reviewer **xS67** for details). We found that the features selected by CFS reflected cell-type-specific responses to infections that were not discussed in the original study. This result thus suggests that CFS may aid in the exploration of realistic/real-world datasets compared to standard analysis workflows.\n\n**When to choose pretraining versus gradient stopping?:** We thank the reviewer for raising this point. We do not have a precise explanation for when pretraining versus gradient stopping should perform better for different datasets; however, we note that performance was usually comparable between both CFS variants (i.e., overlapping error bars). We also note that from a user’s perspective, this choice may simply be regarded as a hyperparameter to be tuned e.g. based on performance on a validation set. For example, the user could compare whether the features selected by pretraining or gradient stopping lead to better reconstruction error on validation data.\n\n**Improved organization of the paper:** We thank the reviewer for their constructive criticism re: the organization.  To improve the flow of the manuscript, we will add a more detailed outline of the organization of our paper to the introduction, as well as improve transitions between sections. We are happy to include any other specific suggestions on this front for the final version of our manuscript.\n\n**Impact of target versus background dataset size:** We first note that the datasets considered in our manuscript had a variety of background to target dataset size ratios, and CFS consistently outperformed baseline methods on all of these datasets. For example, the mice protein dataset had 135 background samples to 444 target samples, while the epithelial cell dataset had a nearly 1 to 1 target to background sample ratio; we refer the reviewer to Appendix E of the supplementary materials for full details for each dataset. In addition, to better understand CFS's performance with respect to different numbers of target versus background samples, we assessed the quality of features selected by CFS on the mice protein dataset with $k = 5$ for varying numbers of background samples, where 135 is the full number of background samples (and this dataset contained 435 target samples):\n\n| Num background samples | CFS (Pretrained) | CFS (GS) |\n|:---:|:---:|:---:|\n| 10 | $0.902 \\pm 0.021$ | $0.902 \\pm 0.021$ | \n| 20 | $0.912 \\pm 0.012$ | $0.912 \\pm 0.012$ | \n| 30 | $0.920 \\pm 0.014$ | $0.922 \\pm 0.012$ | \n| 50 | $0.934 \\pm 0.012$ | $0.933 \\pm 0.014$ | \n| 70 | $0.932 \\pm 0.009$ | $0.936 \\pm 0.012$ | \n| 135 | $0.935 \\pm 0.011$ | $0.935 \\pm 0.015$ | \n\nWe observed that CFS's performance improved initially as more background samples were introduced, with saturation achieved at ~50 background samples. These results provide further evidence that CFS can select high quality features even when the number of background samples is small compared to the number of target samples (435 in this case). Similar results were obtained for other datasets with varying values of $k$, and these will be included in the updated supplementary materials.\n\n[1]: Haber, A.L., et al. \"A single-cell survey of the small intestinal epithelium.\" Nature 551.7680 (2017): 333-339"}, {"review_id": "JfIwCYxfAp", "reviewer": "Reviewer_nNUw", "summary": "The paper studied feature selection in the contrastive analysis setting. There are two types of datasets, the background dataset and the target dataset. The background dataset contains variations due to the uninteresting sources. The target dataset contains observations with variability from both ``interesting'' and ``uninteresting'' sources. The goal of the problem is to select interesting variables. Since interesting variables only exist in target variable, the feature selection is performed on the target dataset. To make use of the background dataset, the embedding parameters learned by the auto encoder on the background dataset is shared with the target dataset in feature selection.\n\nThe problem is with practice interest. But the exact layers of background representation encoder, the background reconstruction and the target reconstruction network are not specified. The paper vaguely mentioned that it is similar to auto-encoder. Since the architecture is the main contributions of the paper, it is better to specify the networks.\n\nThe theoretical analysis is about mutual information $I$, which is related with multiple variables. However, $I$ is not defined in the paper. \n\nFeature selection selects features that are most related with the target. However, the experiments are based on MNIST. There is no analysis about what features are selected. Though the motivation of the problem is about biomedical study, there is no experiments on related area. ", "questions": "1. What is the definition of the mutual information $I$?\n2. What is the impact of the population of background dataset and target dataset?\n3. What does the networks $g$, $h$ and $f$ look like?\n4. Is the feature space of the background dataset is a subset of the feature space of the target dataset?\n5. What is the scale of population and feature dimension to make the proposed method work?", "limitations": "1. The theoretical analysis is not clear which means the performance is not guaranteed.\n2. Assume the background dataset is in large scale, how to select the background dataset related to the target dataset is important but not discussed here.\n3.There are many assumptions to implement the algorithms in real-world applications, such as the population between background and target datasets, the number of selected features $s$ compared with the global feature space.\n", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written. The motivation is strong. The paper proposed a method, made related analysis and conducted some experiments.\n ", "weaknesses": "1. The architecture is not clear to me, specially the representation / reconstruction networks.\n\n2. There is no ablation study if there is no background dataset, the background representation encoder is initialized by random distribution. \n\n3. There is no analysis about the impact of different population of background dataset and target dataset.\n\n4. There is not enough experiments to validate the performance of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for examining our work and providing their feedback. We’re thrilled to hear that the reviewer found our work “well-written” with “strong” motivation. In the space below we respond to the reviewer’s specific concerns:\n\n**Evaluations beyond MNIST:** The reviewer noted that, despite our method being motivated by biomedical applications, our evaluation of CFS consisted solely of experiments on MNIST. **We believe that the reviewer’s claim here results from a misunderstanding.** In Figure 5 of our submission, we provide results from applying CFS and baseline methods to **four real-world biomedical datasets**, including three single-cell RNA sequencing datasets and a proteomics dataset. In these experiments, we found that CFS consistently outperformed baseline methods.\n\n**Definition of mutual information (MI):** We thank the reviewer for pointing out this omission. The MI at an intuitive level quantifies the amount of information obtained about the value of one random variable $X$ from observing the value of a second variable $Y$. More formally, for two discrete random variables $X$ and $Y$, the MI $I(X; Y)$ is defined as:\n\n$$ I(X; Y) = \\sum_{y \\in \\mathcal{Y}} \\sum_{x \\in \\mathcal{X}} P_{X,Y}(x, y)\\log\\frac{P_{X,Y}(x,y)}{P_{X}(x)P_{Y}(y)} $$\n\nwhere $P_{X,Y}$ is the joint probability mass function of $X$ and $Y$ and $P_{X}$ and $P_{Y}$ are the marginal probability mass functions of $X$ and $Y$.\n\nIn our revised manuscript we will include an intuitive description of the mutual information, as well as a formal definition with references to texts (e.g. Cover and Thomas [1]) that provide a deeper introduction to the subject.\n\n**Feature space of target vs. background dataset:** We thank the reviewer for raising this potential point of confusion. CFS (and CA methods more generally) require that the feature space for target and background samples are identical. For example, in the single-cell RNA sequencing datasets considered in our study the same genes were measured for background and target samples. We will clarify this in the final manuscript.\n\n**Required scale of population and feature dimension:**  We thank the reviewer for raising this point. We found empirically that CFS worked across a variety of population and feature dimension sizes. For example, we found that CFS achieved state of the art results on both low-dimensional datasets (e.g. the mice protein dataset with 77 features) as well as the higher dimensional datasets (e.g. the ECCITE-seq dataset with 18,649 features). Similarly, we found that CFS achieved state of the art results across datasets with varying numbers of samples, from hundreds (mice protein) to thousands (epithelial cell and AML treatment) to tens of thousands of samples (Grassy MNIST, ECCITE-seq). These results thus suggest that CFS is effective across a variety of feature dimensions and dataset sizes.\n\n**Ablation without background dataset:** As suggested by the reviewer, we computed the downstream classification of features selected by CFS with a randomly initialized encoder. We provide the results for the mice protein dataset below, and we will include additional results in the supplementary materials for the final manuscript. We found that randomly initializing the encoder led to a significant drop in performance for lower numbers of features. Notably, we found that CFS with random initialization had comparable performance to the supervised STG method (results included below); this is not entirely surprising, as CFS with a randomly initialized background encoder is nearly identical to the STG method.\n\n| k | CFS (Random) | STG | CFS (Pretrained)  | CFS (GS) |\n|---|---|---|---|---|\n| 5 | $0.888 \\pm 0.004$ | $0.865 \\pm 0.006$ | $0.935 \\pm 0.011$ | $0.935 \\pm 0.015$ |\n| 10 | $0.918 \\pm 0.003$ | $0.923 \\pm 0.007$ | $0.993 \\pm 0.005$ | $0.989 \\pm 0.007$ |\n| 15 | $0.97 \\pm 0.002$  | $0.968 \\pm 0.004$ | $0.995 \\pm 0.004$ | $0.993 \\pm 0.007$ |\n| 20 | $0.985 \\pm 0.005$ | $0.980 \\pm 0.005$ | $0.989 \\pm 0.009$ | $0.998 \\pm 0.002$ | \n| 30 | $0.995 \\pm 0.001$ | $0.991 \\pm 0.003$ | $0.998 \\pm 0.002$ | $0.996 \\pm 0.002$ |\n\nSimilar results were obtained for other datasets and will be included in the final manuscript.\n\n**Impact of target versus background dataset size:** We first note that the datasets we considered in our manuscript had a variety of background to target dataset size ratios, and CFS consistently outperformed our baselines on these datasets. For example, the mice protein dataset had 135 background to 444 target samples, while the epithelial cell dataset had a nearly 1 to 1 target to background ratio; we refer the reviewer to Appendix E of the supplementary materials for full details for each dataset. In addition, to better understand CFS's performance with respect to different numbers of target vs. background samples, we assessed the quality of features selected by CFS on the mice protein dataset with $k = 5$ for varying numbers of background samples, where 135 is the full number of background samples (and this dataset contained 435 target samples):\n\n| Num background samples | CFS (Pretrained) | CFS (GS) |\n|:---:|:---:|:---:|\n| 10 | $0.902 \\pm 0.021$ | $0.902 \\pm 0.021$ | \n| 20 | $0.912 \\pm 0.012$ | $0.912 \\pm 0.012$ | \n| 30 | $0.920 \\pm 0.014$ | $0.922 \\pm 0.012$ | \n| 50 | $0.934 \\pm 0.012$ | $0.933 \\pm 0.014$ | \n| 70 | $0.932 \\pm 0.009$ | $0.936 \\pm 0.012$ | \n| 135 | $0.935 \\pm 0.011$ | $0.935 \\pm 0.015$ | \n\nWe observed that CFS's performance improved initially as more background samples were introduced, with saturation achieved at ~50 background samples. These results provide further evidence that CFS can select high quality features even when the number of background samples is small compared to the number of target samples (435 in this case). Similar results were obtained for other datasets with varying values of $k$ and will be included in the final manuscript.\n\n[1]: Cover, Thomas M. Elements of information theory. John Wiley & Sons, 1999.\n\n"}]

Author rebuttal:
We thank the reviewers for their thoughtful comments. We’re thrilled to hear that reviewers found CFS a “novel attempt at feature selection in a weak supervision setting”, “new and highly appropriate” (xS67), and “with practical interest” (nNUw). We’re also thrilled to hear that reviewers found our paper “well-written” (nNUw) with “strong” motivation (nNUw), and that our method was “clearly explained” (BGi6).

We respond to to the reviewers’ comments individually, and we also highlight our responses to some shared concerns below. Overall, we have made a number of helpful additions to the paper and we hope you will consider raising your scores. 

**Clarification of architectural details (xS67, nNUw):**

For all experiments, the background representation encoder $g$ was implemented as a multilayer perceptron with input size equal to the number of features, one hidden layer, and an output dimension specified by the hyperparameter $l$ (the “background representation dimension”). The background reconstruction networks for the pretrained variant of CFS (i.e., $h$) were implemented with the same architecture in reverse.

The target reconstruction function $f$ was implemented as a network with two hidden layers. As inputs, the target reconstruction function $f$ took the concatenated background representation $\mathbf{b}$ and selected target features $x_{S}$, with output dimension equal to the original number of features.

We used ReLU activation functions for all the networks. For CFS, CAE, and STG, the sizes of all hidden layers were set to 512 for experiments with MNIST and 128 for the tabular biomedical datasets, as done in previous work with the CAE [1]. Due to the large number of models being trained, we did not tune the hidden layer size.


**Clarifying the role of the assumptions in Section 5 (BGi6, nNUw):**

In Assumption 1 of the analysis in Section 5, we assume that the salient and shared latent variables ($\mathbf{s}$ and $\mathbf{z}$, respectively) are roughly independent. As noted by the reviewers, this assumption may not hold in realistic datasets. For example, variations in gene expression relating to a cell's response to a given pathogen (i.e., target-specific salient variations governed by $\mathbf{s}$), such as those considered in the epithelial cell dataset from Haber et al. [2] used in our manuscript, may depend on that cell’s cell type (governed by the shared latent variables $\mathbf{z}$).

We thus emphasize that this assumption is not a requirement for applying CFS; rather, this assumption was made to aid in our analysis of the joint training procedures proposed in previous works on contrastive analysis versus our newly proposed two-step procedure. Indeed, with the aid of this simplifying assumption, we were able to prove why the two-step approach may lead to superior performance, as well as uncover a previously unappreciated potential failure case of previous joint contrastive training strategies.

This finding was empirically validated first on our semi-synthetic Grassy MNIST dataset, where we knew a priori that the salient and shared latent factors were fully independent, and our two-step procedure consistently outperformed joint training. Similarly, we found that our two-step procedure continued to outperform joint training on the four real-world biomedical datasets where it was not guaranteed that $\mathbf{s}$ and $\mathbf{z}$ would be independent. These additional empirical results thus suggest that the conclusions from our information-theoretic analysis may continue to hold even in cases when $\mathbf{s}$ and $\mathbf{z}$ exhibit realistic, nontrivial dependence.

**Additional ablations (SoSQ, nNUw):** 

Based on the reviewers’ suggestions, we added additional ablation studies to better understand how CFS’s performance varies with respect to the method’s hyperparameters, and the properties of the target and background datasets. In particular, we

* Further experimented with the impact of varying the background dimension $l$ for different numbers of selected features $k$
* Assessed how performance varied with changes in the size of the background dataset versus the target dataset
* Assessed the impact on performance of randomly initializing the background representation encoder $g$ without training it on any background data.

We refer the reviewers to our responses to individual reviews for more details.


**In-depth analysis of the features selected by CFS on real-world data (xS67, nNUw)**: 

To illustrate how CFS may aid in uncover new biological insights, we performed further analysis on the features selected by CFS for the mice intestinal epithelial cell infection dataset from Haber et al. [2]. In short, we found that CFS uncovered cell-type-specific changes in gene expression for a subset of genes in response to *Salmonella* infection that were not discussed in the original study. This finding thus illustrates the potential of CFS to lead to new biological insights that may be overlooked by standard analysis workflows. Due to space constraints we have placed the full details of this analysis in our individual response to reviewer **xS67**.

[1]: Covert, I., et al. "Learning to maximize mutual information for dynamic feature selection." International Conference on Machine Learning. PMLR, 2023.

[2]: Haber, A.L., et al. "A single-cell survey of the small intestinal epithelium."


Meta-review: My recommendation is to accept this paper.

The paper proposes a method for doing feature selection in contrastive analysis, which is widely used in some applied fields, e.g., biomedical. Confident reviewers noted the potential broad utility of the method. I would encourage the authors to incorporate reviewer comments on additional results and paper organization in the camera ready.
Decision: Accept (poster)