SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Upon further inspection we found that the observed upregulation of these genes was due to a combination of two phenomena. First, as noted in Haber et al. [44], Salmonella-infection resulted in an increase in the number of enterocytes relative to healthy and  $H.$  poly-infected populations. Despite the increase in the number of enterocytes, the distribution of expression levels for these marker genes were notably similar for control and Salmonella enterocytes (Figure S5b). On the other hand, we found (Figure S5b) that these genes' expression levels were substantially upregulated in Salmonella-infected non-enterocytes compared to control non-enterocytes. This cell-type-specific response was not noted in Haber et al. [44], and this finding thus illustrates the potential for CFS to uncover insights that may be missed with standard workflows.
On the other hand, the genes selected by STG (but not CFS) consisted of immune response genes (e.g. Lgals9, Krt19) with altered expression compared to controls, but which did not exhibit other notable patterns among the perturbed cells (Figure S5c). These results indicate that supervised methods may select features that can distinguish target vs. background data, but which do not capture more subtle phenomena within the target data.
G Supplementary Figures
Figure S1: Qualitative results for stop-gradient (SG) and joint training CFS variants on the Grassy MNIST dataset for  $k = 20$  features.
Figure S2: Qualitative results for CFS implemented with the concrete selection layer of Balin et al. [3] on the Grassy MNIST dataset for  $k = 20$ . We find that the gradient stopping (GS) and pretrained CFS variants select features concentrated towards the center as desired, while features selected by the joint training CFS variant are more spread out away from the center.
Figure S3: Varying background representation dimension for various numbers of selected features  $k$  on Grassy MNIST.
Figure S4: Downstream classification results using (a) XGBoost and (b) neural networks as the downstream classifier.
Figure S5: (a) Sample of gene features selected by CFS. (b) Visualizing cell-type-specific changes in gene expression in response to Salmonella infection for genes selected by CFS. (c) Sample of genes selected by the best performing supervised feature selection method STG.
H Supplementary Tables
Table S2: Assessing the quality of features selected by CFS for varying numbers of background samples on the mice protein dataset for  $k = 5$  features. Feature quality was assessed by training a random forest classifier to distinguish between known groups of target samples, and we report the mean accuracy  $\pm$  standard error over five random seeds on a held out test set. We find that feature quality initially increases as the number of samples increases, with saturation achieved around 50 background samples for this dataset.
Number of Background Samples	CFS (Pretrained)	CFS (GS)
10	0.902 ± 0.021	0.902 ± 0.021
20	0.912 ± 0.012	0.912 ± 0.012
30	0.920 ± 0.014	0.922 ± 0.012
50	0.934 ± 0.012	0.933 ± 0.014
70	0.932 ± 0.009	0.936 ± 0.012
135	0.935 ± 0.011	0.935 ± 0.015
Table S3: Comparing the quality of features selected by the CFS architecture with a fixed, randomly initialized background encoder (CFS (Random)) versus our pretrained and gradient stopping (GS) CFS variants on the mice protein dataset. For comparison we also include the results from STG, our best performing non-contrastive baseline. Feature quality was assessed by training a random forest classifier to distinguish between known groups of target samples, and we report the mean accuracy  $\pm$  standard error over five random seeds on a held out test set. We find that CFS (Random) significantly underperforms our pretrained and GS CFS variatns for lower numbers of features, with comparable performance to STG.
Number of Features	CFS (Random)	STG	CFS (Pretrained)	CFS (GS)
5	0.888 ± 0.004	0.865 ± 0.006	0.935 ± 0.011	0.935 ± 0.015
10	0.918 ± 0.003	0.923 ± 0.007	0.993 ± 0.005	0.989 ± 0.007
15	0.970 ± 0.002	0.968 ± 0.004	0.995 ± 0.004	0.993 ± 0.007
20	0.985 ± 0.005	0.980 ± 0.005	0.989 ± 0.009	0.998 ± 0.002
30	0.995 ± 0.001	0.991 ± 0.003	0.998 ± 0.002	0.996 ± 0.002