SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {b}) + I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) - H (\boldsymbol {z}) - 4 \epsilon \leq I (\boldsymbol {a}; \boldsymbol {s}).
$$
Proof. In the first step of our learning algorithm, we learn the representation  $\pmb{b}$  by maximizing  $I(\pmb{b};\pmb{x} \mid s)$ . To show how this relates to maximizing  $I(\pmb{b};\pmb{z})$ , consider the following joint mutual information, which we can decompose in two ways using the chain rule of mutual information [38]:
$$
\begin{array}{l} I (\boldsymbol {b}; \boldsymbol {s}, \boldsymbol {z}, \boldsymbol {x}) = I (\boldsymbol {b}; \boldsymbol {z}) + I (\boldsymbol {b}; \boldsymbol {s} \mid \boldsymbol {z}) + I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \\ = I (\boldsymbol {b}; \boldsymbol {s}) + I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) + I (\boldsymbol {b}; \boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {s}). \\ \end{array}
$$
Rearranging terms, we can see the following relationship between  $I(\pmb{b}; \pmb{x} \mid s)$  and  $I(\pmb{b}; z)$ :
$$
I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) = I (\boldsymbol {b}; \boldsymbol {z}) + \underbrace {I (\boldsymbol {b} ; \boldsymbol {x} \mid \boldsymbol {s} , \boldsymbol {z})} _ {\delta_ {1}} + \left(\underbrace {I (\boldsymbol {b} ; \boldsymbol {s} \mid \boldsymbol {z}) - I (\boldsymbol {b} ; \boldsymbol {s})} _ {\delta_ {2}}\right) - \underbrace {I (\boldsymbol {b} ; \boldsymbol {z} \mid \boldsymbol {x} , \boldsymbol {s})} _ {\delta_ {3}}.
$$
We can show that each of the  $\delta_{i}$  terms is bounded near zero due to our assumptions:
$$
\begin{array}{l} \delta_ {1} = I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \leq H (\boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \leq \epsilon \quad \Rightarrow \delta_ {1} \in [ 0, \epsilon ] \\ \delta_ {2} = I (\boldsymbol {b}; \boldsymbol {s} \mid \boldsymbol {z}) - I (\boldsymbol {b}; \boldsymbol {s}) = I (\boldsymbol {s}; \boldsymbol {z} \mid \boldsymbol {b}) - I (\boldsymbol {s}; \boldsymbol {z}) \quad \Rightarrow \delta_ {2} \in [ - \epsilon , \epsilon ] \\ \delta_ {3} = I (\boldsymbol {b}; \boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {s}) \leq H (\boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {s}) \leq \epsilon \quad \Rightarrow \delta_ {3} \in [ 0, \epsilon ] \\ \end{array}
$$
Thus, our objective in the first step has the property that  $I(\pmb{b}; \pmb{x} \mid s) \approx I(\pmb{b}; z)$ , or more precisely,
$$
I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) - 2 \epsilon \leq I (\boldsymbol {z}; \boldsymbol {b}) \leq I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) + 2 \epsilon . \tag {8}
$$
Regarding the second step of the learning algorithm, we can use a similar argument to show a relationship between  $I(\boldsymbol{a}; \boldsymbol{x} \mid \boldsymbol{z})$  and  $I(\boldsymbol{a}; \boldsymbol{s})$ :
$$
I (\boldsymbol {a}; \boldsymbol {x} \mid z) = I (\boldsymbol {a}; \boldsymbol {s}) + \underbrace {I (\boldsymbol {a} ; \boldsymbol {x} \mid \boldsymbol {s} , z)} _ {\Delta_ {1}} + \left(\underbrace {I (\boldsymbol {a} ; \boldsymbol {z} \mid \boldsymbol {s}) - I (\boldsymbol {a} ; \boldsymbol {z})} _ {\Delta_ {2}}\right) - \underbrace {I (\boldsymbol {a} ; \boldsymbol {s} \mid \boldsymbol {x} , z)} _ {\Delta_ {3}}. \tag {9}
$$
Because we lack access to  $z$  during this step, we must maximize  $I(\boldsymbol{a}; \boldsymbol{x} \mid \boldsymbol{b})$  instead of  $I(\boldsymbol{a}; \boldsymbol{x} \mid z)$ . Fortunately, we can show that these are related using a similar mutual information decomposition:
$$
\begin{array}{l} I (\boldsymbol {a}; \boldsymbol {z}, \boldsymbol {x}, \boldsymbol {b}) = I (\boldsymbol {a}; \boldsymbol {b}) + I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {b}) + I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {b}) \\ = I (\boldsymbol {a}; \boldsymbol {z}) + I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {z}) + I (\boldsymbol {a}; \boldsymbol {b} \mid \boldsymbol {x}, \boldsymbol {z}). \\ \end{array}
$$
Rearranging terms, we get the following:
$$
I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {b}) = I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {z}) + \underbrace {I (\boldsymbol {a} ; \boldsymbol {b} \mid \boldsymbol {x} , \boldsymbol {z})} _ {\Delta_ {4}} + \left(\underbrace {I (\boldsymbol {a} ; \boldsymbol {z}) - I (\boldsymbol {a} ; \boldsymbol {b})} _ {\Delta_ {5}}\right) - \underbrace {I (\boldsymbol {a} ; \boldsymbol {z} \mid \boldsymbol {x} , \boldsymbol {b})} _ {\Delta_ {6}}. \tag {10}
$$
Putting the results from eq. (9) and eq. (10) together, we have the following:
$$
I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {b}) = I (\boldsymbol {a}; \boldsymbol {s}) + \Delta_ {1} + \Delta_ {2} - \Delta_ {3} + \Delta_ {4} + \Delta_ {5} - \Delta_ {6}. \tag {11}
$$
Combining our assumptions with the result from eq. (8), we can show that the  $\Delta_{i}$  terms are bounded as follows:
$$
\Delta_ {1} = I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \leq H (\boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \leq \epsilon \Rightarrow \Delta_ {1} \in [ 0, \epsilon ]
$$
$$
\Delta_ {2} = I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {s}) - I (\boldsymbol {a}; \boldsymbol {z}) = I (\boldsymbol {s}; \boldsymbol {z} \mid \boldsymbol {a}) - I (\boldsymbol {s}; \boldsymbol {z}) \Rightarrow \Delta_ {2} \in [ - \epsilon , \epsilon ]
$$
$$
\Delta_ {3} = I (\boldsymbol {a}; \boldsymbol {s} \mid \boldsymbol {x}, \boldsymbol {z}) \leq H (\boldsymbol {s} \mid \boldsymbol {x}, \boldsymbol {z}) \leq \epsilon \Rightarrow \Delta_ {3} \in [ 0, \epsilon ]
$$
$$
\Delta_ {4} = I (\boldsymbol {a}; \boldsymbol {b} \mid \boldsymbol {x}, \boldsymbol {z}) \leq H (\boldsymbol {b} \mid \boldsymbol {x}) = 0 \Rightarrow \Delta_ {4} = 0
$$
$$
\Delta_ {5} = I (\boldsymbol {a}; \boldsymbol {z}) - I (\boldsymbol {a}; \boldsymbol {b}) = I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {b}) - I (\boldsymbol {a}; \boldsymbol {b} \mid \boldsymbol {z}) \Rightarrow \Delta_ {5} \in [ - H (\boldsymbol {b}) + I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) - 2 \epsilon , H (\boldsymbol {z}) - I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) + 2 \epsilon ]
$$
$$
\Delta_ {6} = I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {b}) \leq H (\boldsymbol {z} \mid \boldsymbol {b}) = H (\boldsymbol {z}) - I (\boldsymbol {z}; \boldsymbol {b}) \Rightarrow \Delta_ {6} \in [ 0, H (\boldsymbol {z}) - I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) + 2 \epsilon ]
$$
This yields the following two-sided inequality, which connects the empirical objective  $I(\pmb{a}; \pmb{x} \mid \pmb{b})$  to our implicit objective  $I(\pmb{a}; \pmb{s})$ :
$$
I (\boldsymbol {a}; \boldsymbol {s}) + 2 I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) - H (\boldsymbol {z}) - H (\boldsymbol {b}) - 6 \epsilon \leq I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {b}) \leq I (\boldsymbol {a}; \boldsymbol {s}) - I (\boldsymbol {b}; \boldsymbol {x} \mid \boldsymbol {s}) + H (\boldsymbol {z}) + 4 \epsilon .
$$
Focusing on a one-sided version of the result, we can see that the two empirical objectives  $I(\pmb{a}; \pmb{x} \mid \pmb{b})$  and  $I(\pmb{b}; \pmb{x} \mid \pmb{s})$  control the underlying objective  $I(\pmb{a}; \pmb{s})$  as follows:
$$
I (\boldsymbol {a}; \boldsymbol {x}) - H (\boldsymbol {x}) + I (\boldsymbol {x}; \boldsymbol {z}) \leq I (\boldsymbol {a}; \boldsymbol {z}).
$$
Proof. Consider the following joint mutual information, which we can decompose in two ways:
$$
\begin{array}{l} I (\boldsymbol {a}; \boldsymbol {x}, \boldsymbol {s}) = I (\boldsymbol {a}; \boldsymbol {s}) + I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {s}) \\ = I (\boldsymbol {a}; \boldsymbol {x}) + I (\boldsymbol {a}; \boldsymbol {s} \mid \boldsymbol {x}). \\ \end{array}
$$
Rearranging terms, we can see the following relationship between  $I(\pmb{a}; \pmb{s})$  and  $I(\pmb{a}; \pmb{x})$ :
$$
I (\boldsymbol {a}; \boldsymbol {s}) = I (\boldsymbol {a}; \boldsymbol {x}) + I (\boldsymbol {a}; \boldsymbol {s} \mid \boldsymbol {x}) - I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {s}).
$$
Bounding the remaining terms, we arrive at the following two-sided result:
$$
I (\boldsymbol {a}; \boldsymbol {x}) - H (\boldsymbol {x} \mid \boldsymbol {s}) \leq I (\boldsymbol {a}; \boldsymbol {s}) \leq I (\boldsymbol {a}; \boldsymbol {x}) + H (\boldsymbol {s} \mid \boldsymbol {x}).
$$
Our result in Theorem 2 focuses on the left-hand side and uses the definition of  $I(\pmb{x}; \pmb{s})$ . An identical argument can be used to show the same result for  $I(\pmb{a}; \pmb{z})$ .
Proposition 1. Under Assumptions 1 and 2, the representation  $\mathbf{a}$  has mutual information with  $\mathbf{x}$  that decomposes additively between  $s$  and  $z$ :
$$
- \epsilon \leq I (\boldsymbol {a}; \boldsymbol {x}) - I (\boldsymbol {a}; \boldsymbol {s}) - I (\boldsymbol {a}; \boldsymbol {z}) \leq 2 \epsilon .
$$
Proof. Consider the mutual information  $I(\pmb{a}; \pmb{x})$  for a representation  $\pmb{a}$  generated as a function of  $\pmb{x}$ , which we can rewrite using the data processing inequality and chain rule:
$$
\begin{array}{l} I (\boldsymbol {a}; \boldsymbol {x}) = I (\boldsymbol {a}; \boldsymbol {x}, \boldsymbol {s}, \boldsymbol {z}) = I (\boldsymbol {a}; \boldsymbol {s}) + I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {s}) + I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}) \\ = I (\boldsymbol {a}; \boldsymbol {s}) + I (\boldsymbol {a}; \boldsymbol {z}) + (I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {s}) - I (\boldsymbol {a}; \boldsymbol {z})) + I (\boldsymbol {a}; \boldsymbol {x} \mid \boldsymbol {s}, \boldsymbol {z}). \tag {12} \\ \end{array}
$$
Assumption 1 implies that  $0 \leq I(\boldsymbol{a}; \boldsymbol{x} \mid s, z) \leq H(\boldsymbol{x} \mid s, z) \leq \epsilon$  for the final term. For the subtraction term, we may expect that  $I(\boldsymbol{a}; z \mid s) \approx I(\boldsymbol{a}; z)$  because  $z, s$  are roughly independent. We can formalize this idea using another mutual information decomposition:
$$
\begin{array}{l} I (\boldsymbol {a}, \boldsymbol {s}; \boldsymbol {z}) = I (\boldsymbol {a}; \boldsymbol {z}) + I (\boldsymbol {s}; \boldsymbol {z} \mid \boldsymbol {a}) \\ = I (s; z) + I (a; z \mid s) \\ \Rightarrow I (\boldsymbol {a}; \boldsymbol {z} \mid \boldsymbol {s}) - I (\boldsymbol {a}; \boldsymbol {z}) = I (\boldsymbol {s}; \boldsymbol {z} \mid \boldsymbol {a}) - I (\boldsymbol {s}; \boldsymbol {z}). \\ \end{array}
$$
Assumptions 1 and 2 imply that  $|I(\pmb{a}; \pmb{z} | s) - I(\pmb{a}; \pmb{z})| \leq \epsilon$ . Substituting this into eq. (12) yields our final result:
C Connection between mean squared error and mutual information
When predicting  $\pmb{x}$  given an arbitrary representation  $\pmb{a}$ , our loss function based on the reconstruction function  $g$  is the following:
$$
\min _ {g} \underset {\pmb {x a}} {\mathbb {E}} \left[ | | g (\pmb {a}) - \pmb {x} | | ^ {2} \right].
$$
To minimize the loss, the best possible prediction is given by the function  $g(a) = \mathbb{E}[\pmb{x} \mid a]$ , and our loss becomes the trace of the expected conditional variance:
$$
\underset {\boldsymbol {x} \boldsymbol {a}} {\mathbb {E}} \left[ | | \mathbb {E} [ \boldsymbol {x} \mid \boldsymbol {a} ] - \boldsymbol {x} | | ^ {2} \right] = \operatorname {T r} \left(\underset {\boldsymbol {a}} {\mathbb {E}} \left[ \operatorname {V a r} (\boldsymbol {x} \mid \boldsymbol {a}) \right]\right).
$$
Optimizing the mean squared error error with respect to  $\pmb{a}$  can thus be understood as minimizing the trace of  $\pmb{x}$ 's expected conditional variance.
Next, we can show that the conditional variance is connected to the mutual information  $I(\pmb{a}; \pmb{x})$ . First, for a fixed value of  $a$  where  $\pmb{x}$  has conditional variance  $\operatorname{Var}(\pmb{x} \mid a)$ , we have the following upper bound on the differential entropy,
$$
H (\boldsymbol {x} \mid a) \leq \frac {1}{2} \log \det  \left(\operatorname {V a r} (\boldsymbol {x} \mid a)\right) + \frac {d}{2} \log (2 \pi e),
$$
where  $\operatorname{det}(\operatorname{Var}(\boldsymbol{x} \mid a))$  denotes the determinant of the covariance matrix. This result follows from the fact that the entropy for random variables with fixed covariance is maximized by a Gaussian distribution [38]. Next, assuming that  $\operatorname{Var}(\boldsymbol{x} \mid a)$  is positive definite, or that  $\operatorname{Var}(\boldsymbol{x} \mid a) > 0$ , we can modify the bound to use the trace rather than the log-determinant of the covariance matrix:
$$
H (\boldsymbol {x} \mid a) \leq \frac {1}{2} \operatorname {T r} (\operatorname {V a r} (\boldsymbol {x} \mid a)) + \frac {d}{2} \log (2 \pi).
$$
Taking the expectation of both sides with respect to  $\mathbf{a}$ , we obtain the following relationship between the conditional entropy and expected conditional variance:
$$
H (\boldsymbol {x} \mid \boldsymbol {a}) \leq \frac {1}{2} \operatorname {T r} \left(\mathbb {E} _ {\boldsymbol {a}} [ \operatorname {V a r} (\boldsymbol {x} \mid \boldsymbol {a}) ]\right) + \frac {d}{2} \log (2 \pi).
$$
Finally, using the relationship between the mutual information  $I(\pmb{a}; \pmb{x})$  and conditional entropy  $H(\pmb{x} \mid \pmb{a})$ , we can identify the following connection between the minimum mean squared error and the mutual information achieved by  $\pmb{a}$ :
$$
H (\boldsymbol {x}) - \frac {d}{2} \log (2 \pi) - \frac {1}{2} \left\{\underset {g} {\min} \underset {\boldsymbol {x a}} {\mathbb {E}} \left[ | | g (\boldsymbol {a}) - \boldsymbol {x} | | ^ {2} \right] \right\} \leq I (\boldsymbol {a}; \boldsymbol {x}).
$$
Thus, minimizing the mean squared error with respect to  $\pmb{a}$  can be understood as maximizing a lower bound on the mutual information  $I(\pmb{a};\pmb{x})$ . This shows that our empirical procedure, which involves minimizing mean squared error, is connected to our theoretical results that are stated in terms of mutual information.
We note that this bound can be vacuous if  $H(\pmb{x}) < 0$ , which is possible for differential entropy, because we trivially have  $I(\pmb{a};\pmb{x}) \geq 0$  even for continuous random variables. The result should therefore not be interpreted as providing a precise estimate of the mutual information; it instead demonstrates the rough intuition that improving the mean squared error is related to increasing the mutual information.
All experiments were performed on a system running CentOS 7.9.2009 equipped with an NVIDIA RTX 2080 TI GPU with CUDA 11.7.
CFS models were implemented using PyTorch [50] (version 1.13) with the PyTorch Lightning  $\mathrm{API}^4$ . For all CFS variants we let our reconstruction function  $f$  be a multilayer perceptron with two hidden layers of size 512 with ReLU activation functions. For our Joint and stop-gradient (SG) CFS models, we let  $g$  be a multilayer perceptron with a single hidden layer of size 128. For our Pretrained CFS models, we first trained an autoencoder consisting of an encoder with a single hidden layer of size 128 and a decoder with the same architecture in reverse. The encoder network's weights were then frozen and it was used as  $g$ . All CFS models were trained using the PyTorch implementation of the Adam [51] optimizer with default hyperparameters. Batch sizes of 128 were used for all experiments.
Hyperparameters for the STG layer were set as described in Yamada et al. [23]. That is, we set the  $\sigma$  for the  $\mathcal{N}(0,\sigma^2)$  distribution used for drawing noise to  $\sigma = 0.5$ , and the regularization parameter  $\lambda$  was tuned so that the desired number of gates were left open (i.e.,  $\mu_i > 0$ ) after convergence.
We implemented the unsupervised concrete autoencoder (CAE), differentiable unsupervised feature selection (DUFS), and stochastic gates (STG) baselines in PyTorch. Similar to CFS, for the CAE's reconstruction function we used a multilayer perceptron with two hidden layers of size 512. As in Balin et al. [3], we optimized the CAE using Adam [51] with default hyperparameters and set the temperature of the CAE's concrete selector layer following Balin et al. [3]: i.e., at the start of training, we set the temperature of the concrete layer to 10, and gradually annealed the temperature to a final value of 0.1 using the exponential decay schedule described in Appendix A. As in [3], we trained our concrete autoencoders until the mean of the concrete samples exceeded 0.99.
As in Lindenbaum et al. [36], we trained DUFS with an SGD optimizer with learning rate of 1 using the authors' proposed "parameter-free" loss function (i.e., eq. 9 of Lindenbaum et al. [36]) until convergence. For a given number of desired features  $k$ , the  $k$  features with the highest gate values were then selected as described in Lindenbaum et al. [36].
For the supervised STG baseline [23], we used two hidden layers of 512 units. To adapt STG, which was designed for fully supervised scenarios, to the contrastive analysis setting, we trained it on the binary classification problem of distinguishing target versus background points. We optimized STG using Adam with default hyperparameters. For all experiments with STG, the regularization parameter  $\lambda$  was tuned such that STG selected the desired number of features.
For all experiments with the supervised LassoNet model [34], we used the authors' Python package. As with STG, we adapted the supervised LassoNet model to the contrastive analysis setting by training it on the binary classification problem of distinguishing target versus background points. The regularization parameter  $\lambda$  was tuned to select the desired number of features, and all other hyperparameters were left at their default values in the LassoNet package.
<sup>4</sup>https://github.com/PyTorchLightning/pytorch-lightning
<sup>5</sup>https://github.com/lasso-net/lassonet
E Dataset descriptions
Here we provide additional details on the real-world datasets considered in our empirical evaluation of CFS in the main text. We also summarize these datasets in Table S1. Our code for preprocessing these datasets can be found at www.placeholder.com. $^6$
Mice protein expression This dataset [43, 52] consists of protein expression levels from healthy control mice and mice that have developed Down syndrome. Each mouse was injected with memantine, subjected to shock therapy, received both of these treatments, or none of them. Our goal here was to classify mice with Down syndrome based on their treatment regimens. We used data from healthy mice that did not receive any treatment as a background. As done in previous work (e.g., [3]), features were min-max normalized. Raw data was downloaded from https://archive.ics.uci.edu/ml/machine-learning-databases/00342/.
Epithelial cell infection response We constructed our target dataset by combining two sets of gene expression measurements from Haber et al. [44]. These datasets consist of gene expression measurements of intestinal epithelial cells from mice infected with either Salmonella or Heligmosomoides polygyrus (H. poly). Here our goal is to separate cells by infection type. As a background dataset we used measurements collected from healthy cells released by the same authors. As is standard for single-cell gene expression measurements (see e.g., [53] for a review), raw count values were library-size-normalized to account for differences in sequencing depth across cells. Normalized counts  $x$  were then transformed according to  $\log (x + 1)$ . All preprocessing steps for this dataset were carried out using scany [54], a standard Python package for analyzing single-cell RNA-seq data. Gene expression count files were downloaded from the NIH Gene Expression Omnibus at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE92332.
AML treatment Here we combined datasets from Zheng et al. [45] containing single-cell gene expression measurements from two patients with acute myeloid leukemia (AML) before and receiving a blood cell transplant. Our goal here is to learn a representation that separates pre and post transplant measurements. As a background dataset we used expression measurements from two healthy control patients that were collected as part of the same study. Raw counts were normalized as described above for the epithelial cell infection response dataset. Files containing measurements from the first patient pre- and post-transplant can be found at https://cf.10xgenomics.com/samples/cell-exp/1.1.0/aml027_pre_transplant/aml027_pre_transplant-filtered_gene(bc_matrices.tar.gz and https://cf.10xgenomics.com/samples/cell-exp/1.1.0/aml027_post_transplant/aml027_post_transplant_filtered_gene(bc_matrices.tar.gz, respectively; from the second patient pre- and post-transplant at https://cf.10xgenomics.com/samples/cell-exp/1.1.0/aml035_pre_transplant/aml035_pre_transplant-filtered_gene(bc_matrices.tar.gz and https://cf.10xgenomics.com/samples/cell-exp/1.1.0/aml035_post_transplant/aml035_post_transplant_filtered_gene(bc_matrices.tar.gz, respectively; and from the two healthy control patients at https://cf.10xgenomics.com/samples/cell-exp/1.1.0/frozen_bmmc Healthy_donor1/frozen_bmmc Healthy_donor1_filtered_gene(bc_matrices.tar.gz and https://cf.10xgenomics.com/samples/cell-exp/1.1.0/frozen_bmmc Healthy_donor2/frozen_bmmc Healthy_donor2_filtered_gene(bc_matrices.tar.gz.
ECCITE-seq perturbation screen ECCITE-seq [55] is a technique that allows for multimodal readouts of single-cell perturbation screens. Here we obtained an ECCITE-seq dataset from Papalexi et al. [14] that measured how various gene knockout perturbations affected the gene expression profiles of cells from the THP1 cancer cell line. Our goal here was to classify cells into one of three clusters of perturbations with similar effects identified by Papalexi et al. [14]. As a background dataset we used measurements from control cells with no perturbation collected as part of the same study. Raw gene expression counts were normalized as described above for the epithelial cell infection response dataset. Gene expression count files and associated metadata were downloaded from the NIH Gene Expression Omnibus at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE153056.
Code available in supplementary materials and will be made public upon acceptance.
Table S1: Summary of real-world datasets considered in the main text.
Dataset	Features (d)	Train size (target)	Train size (background)	Test size	Classes
Grassy MNIST	784	48,000	48,000	12,000	10
Mice Protein	77	444	135	111	4
Epithelial cell	15,215	3,584	3,240	897	2
AML treatment	12,079	9,919	4,457	2,480	2
ECCITE-seq	18,649	14,674	2,386	3,669	3
F Deeper analysis of Epithelial Cell dataset results
To illustrate how the features selected by CFS may lead to new biological insights, we performed an additional analysis of the specific gene features selected by CFS versus supervised methods for the mice intestinal epithelial cell infection dataset from Haber et al. [44]. In this dataset gene expression levels were compared between control cells (background) and cells exposed to either the bacteria Salmonellaor the parasite  $H.$  poly (target). For this analysis we compared the features selected by CFS and STG, the best-performing supervised baseline in our quantitative experiments, with the number of features  $k$  set to 20.
We first considered the overlapping genes selected by both methods, and we found that both methods selected genes involved in the inflammatory response (e.g.  $Cd74$ ) and previously studied defense responses to pathogens (e.g.  $Reg3b$ ,  $Reg3g$ ). We next considered the genes selected only by CFS and not by STG. We found that CFS tended to select genes (Figure S5a) with clear differences in expression patterns between the two pathogens (e.g.  $Gm42418$ ), including a number of enterocyte marker genes (e.g.  $Apoa1$ ,  $Guca2b$ ,  $Fabp1$ ) that were upregulated in Salmonella cells compared to  $H.$  poly cells. Notably, the enterocyte markers exhibited a clear bimodal pattern in their expression levels for Salmonella-infected cells.