PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction

Authors: Aaron Wenteler, Martina Occhetta, Nikhil Branson, Magdalena Huebner, William Dee, Victor Curean, William Connell, Alex Hawkins-Hooker, Siu Pui Chung, Yasha Ektefaie, Amaya Gallagher-Syed, César Miguel Valdez Córdova
Primary area: datasets and benchmarks
Keywords: foundation models, benchmark, single-cell biology, perturbation effect prediction

Abstract:
_In silico_ modeling of transcriptional responses to perturbations is crucial for advancing our understanding of cellular processes and disease mechanisms. We present PertEval-scFM, a standardized framework designed to evaluate models for perturbation effect prediction. We apply PertEval-scFM to benchmark zero-shot single-cell foundation model (scFM) embeddings against simpler baseline models to assess whether these contextualized representations enhance perturbation effect prediction. Our results show that scFM embeddings do not provide consistent improvements over baseline models, especially under distribution shift. Additionally, all models struggle with predicting strong or atypical perturbation effects. Overall, this study provides a systematic evaluation of zero-shot scFM embeddings for perturbation effect prediction, highlighting the challenges of this task and revealing the limitations of current-generation scFMs. Our findings underscore the need for specialized models and high-quality datasets that capture a broader range of cellular states. Source code and documentation can be found at: https://anonymous.4open.science/r/PertEval-C674/.

Decision: reject
Reviewer scores: 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730576235812

thread_type:
official_review

<thread_object>
creator: reviewer_yfbt

<content>
summary:
The paper introduces an evaluation framework for scRNA-seq foundation models applied to the downstream task of predicting the transcriptomic effect of unseen genetic perturbations. In order to adapt foundation models to this task, the authors first extract gene embeddings from a variety of foundation models by simulating a genetic knock-out in unperturbed cells and then embedding both unperturbed and in-silico perturbed cells using the foundation model. Then, they fit MLP adapters on top of these embeddings with a root mean squared error loss. They include two baselines to compare against: the mean baseline, (presenting no variation between different perturbations) and an MLP adapter fit on a simple co-expression based embedding.

The authors then proceed to evaluate these models. At its heart, the proposed evaluation suite relies on two previously established metrics, the mean squared error (MSE) over the top 20 differentially expressed genes (DEGs) per perturbation and the MSE over 2000 highly variable genes (HVGs). On top of this, a spectral splitting technique (SPECTRA) is applied to simulate the effects of increasingly stronger distributional shifts between train and test data, and the top 2000 HVG MSE is averaged over different splitting strengths to yield the area under the SPECTRA performance curve (AUSPC) metric. The authors overall find that foundation model embeddings show a marginal, but not stastitically significant improvement over simple baselines in these metrics.

Finally, they investigate properties of individual perturbations and of the different foundation models used that could explain the observed performance. First, they show that larger effect sizes as measured by the energy distance of the perturbation to the control population and uncommon effect distributions are correlated with higher MSE per perturbation. Second, they explore the effect of overlap between foundation model pre-training corpus and the downstream perturbation dataset on the overall performance, finding mostly a correlation with robustness at a very stringent data splitting regime, with little overall impact otherwise.

strengths:
* **Impactful question:** Predicting the effect of unseen perturbations is at the heart of understanding the causal impact of genes and therefore an important biological question. Therefore, the scope of the paper, evaluating the impact of scRNA-seq foundation models on this task, is an important research topic.
* **Manuscript quality:** The manuscript is overall well-written and clearly organized.
* **Novelty:** The two elements that are highlighted as novel, simulating challenging datasets with SPECTRA and investigating contextual alignment, are, to my knowledge, novel in this domain.
* **Publishing negative findings:** The authors overall report negative findings as to the impact of foundation models on the task in question, which is is often  disincentivized by the scientific community. I find this commendable since it can help the community to make progress and improve these tools.

weaknesses:
* **Benchmarking on inappropriate dataset**
    * Most importantly, the authors rely on a **single and quite small** (<300 overall perturbations) perturbation dataset for all of their evaluation, namely [Norman (2019)](https://www.science.org/doi/full/10.1126/science.aax4438). Despite the overall solid presentation, this leaves the impression of a rushed and unfinished work. At the very least, I would like to see results on [Replogle (2022)](https://www.cell.com/cell/fulltext/S0092-8674(22)00597-9?dgcid=raven_jbs_aip_email) included (>10,000 perturbations), but a comprehensive evaluation framework should ideally look at most available Perturb-seq datasets out there to be able to draw any insightful conclusions.
    * More than half of the perturbations in Norman (2019) are combinations of genetic perturbations. Figure E1 shows that the simplest data splits contain >200 perturbations, suggesting that these combinations were included in the analysis. However, I don't see any mention of how combinations are handled by the MLP adapters, and the authors never show any evaluation pertaining to the effect of combinations, as done, for example, in [Roohani (2024)](https://www.nature.com/articles/s41587-023-01905-6).
    * The Norman et al dataset contains CRISPRa perturbations, resulting in overexpression of the perturbed genes, i.e., a **gain-of-function perturbation**. The foundation model embeddings employed by the authors simulate **loss-of-function perturbations**. On a biological level, this is highly unintuitive. While it may still lead to meaningful and useful embeddings, this is a potential failure mode that could explain the overall findings of the paper.
* **Dataset splitting is not high-impact:** Since evaluation is performed on individual datasets (or more specifically, a single dataset here), generalizing to unseen perturbations could be construed as an experimental design question: can I generalize from the observed perturbations to unseen ones, and if so, which ones should I collect data for? Since Perturb-seq can already applied to the order of thousands of perturbations, if the task is to generalize to arbitrary perturbations, a **random selection of genes is always an admissible design**. While novel, that means that making the prediction task artifically harder through the SPECTRA split might overestimate the difficulty of the problem in an actual experimental design context. Moreover, the dataset that the authors consider is already very small and is further massively reduced (see Figure E1, less than 30 perturbations in the final split). Overall, I am not convinced that this splitting technique should be an integral part of the eval pipeline.
* **Correlative investigations of model performance are not convincing**
    * The observed correlation of E-distance to MSE (Figure 3a) is interesting, but not completely convincing to me. At a minimum, could the authors compute a measure of correlation in this plot? Moreover, it should be able to not only assess this question observationally, but also causally through ablation studies. Does changing the splits to specifically include similar E-distance perturbations change the observed performance? Overall, all this seems to say is that outliers are harder to predict.
    * Similarly, the different contextual alignment examples in Figure 5, while novel, do not actually show any good stratification of results (except for the 0.7 split which is very small, and even there it might be exacerbated by the box-plot excluding outliers for scBERT). Therefore, I'm not convinced as to why this is the right formulation to consider for assessing how appropriate the pre-training dataset is for the downstream task.
* **Summary and rating:** The above weaknesses by far outweigh the strenghths, leading to a recommendation to **reject**.

questions:
* It is not clear to me how the mean baseline is computed. Is it the mean over all perturbations? This is not immediate from the notation in Equation (6).
* The error bars in Figure 2 are very large. Could this be improved by averaging over more model splits?
* In Figure 3, how are the MSEs per perturbation calculated over splits? Are they averaged here? Would it make more sense to investigate this on a per split basis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594584092

thread_type:
official_review

<thread_object>
creator: reviewer_3e1k

<content>
summary:
This paper introduces PertEval-scFM, a framework for evaluating how well single-cell foundation models (scFMs) can predict gene expression changes in response to genetic perturbations. The authors benchmark zero-shot embeddings from various scFMs against simple baseline models, finding that scFMs don't consistently outperform baselines. They indentify narrow training distributions as one of the potential culprits and suggest improvements.

strengths:
- Creation of a standardized evaluation framework (PertEval-scFM) with well-documented codebase is good for field progress
- I like that there's multiple complementary evaluation methods (E-distance, AUSPC), including some novel metrics
- Evaluation includes all major single-cell foundation models with appropriate baselines
- I like how the paper provides constructive criticism of current models while suggesting concrete paths forward, by doing the distribution shift effect analysis.

weaknesses:
- Limited dataset scope: Heavy reliance on single dataset (Norman et al.), restricted to CRISPRa perturbations in K562 cell line make me concerned about how well this generalizes
- My main issue with the paper is that there seem to be big parameter count disparaties betw. the MLPs because of the different embedding dimensions.
        - Base embedding dimensions vary widely (200 to 3,072)
        - Results in vastly different MLP sizes between models - by my calculations, scFoundation's MLP ends up with around 25M parameters while scBERT's has less than 0.5M
        - Such dramatic differences in model capacity could significantly affect training dynamics and make fair comparison difficult and this doesn't seem significantly considered / addressed. Though the authors tuned the hypers individually for each embedding.
- Minor note: In Table 1 scGPT is a Transformer, FlashAttention is just a way of efficiently implement normal attention.

questions:
1. Could you elaborate on the methodological choices around MLP probe architecture? Specifically:
    - How sensitive are results to the choice of architecture?
    - How does the varying parameter count due to different embedding dimensions affect fair comparison between models?
    - Have you considered approaches to normalize model capacity across different embeddings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612821246

thread_type:
official_review

<thread_object>
creator: reviewer_sabf

<content>
summary:
This work introduces a benchmark framework for the evaluation of single-cell FMs in the context of genetic perturbation prediction. Multiple scFMs (scBERT, Geneformer, scGPT, UCE, and scFoundation) are compared to simple baselines through a linear probing approach and evaluated across multiple metrics aggregated from previous works. Results highlight the limitations of scFMs in this context, with negligible improvements over the baselines. Additionally, the authors highlight the need for better datasets across a broader and more diverse range of cellular states to better assess predictive models.

strengths:
- The topic is relevant, as the prediction of genetic perturbations has critical biological applications and therapeutic potential, and previous work has shown how scFMs generally provide limited benefits in this context.
- The manuscript is easy to follow, and the figures are clear.
- A good number of scFMs is considered in this study. 
- The work introduces a general, open-source benchmarking framework.

weaknesses:
- Setting: the major limitations of this work are its scope and predictive methodology. This work focuses on a very specific setting of "zero-shot" scFM embeddings for perturbation effect prediction. However, this is not necessarily the only (or the best) approach to leverage scFMs for this task. Notably, previous works on scFMs also include perturbation effect prediction results and often focus on a different setting. For example, scGPT, in the original publication (Cui et al., 2024), leverages finetuning for this task, reporting promising results. This leads to important limitations: 1) while it is possible to compare multiple methods in this study, it is not possible to compare results to those included in the original publications, and 2) given that this is the only approach tested in this paper for all methods, it is difficult to say if it is generally good enough for the task. Indeed, it could be possible that the way perturbed embeddings are generated (line 133) leads to out-of-distribution (and, in turn, low-quality) scFMs embeddings for all methods. Overall, this limits the generality of the conclusions of this work.
- Novelty: the novelty of this work appears to be limited, as it leverages previously-defined evaluation metrics, and both the dataset, task, and models have been extensively studied in previous works. 
- Data: the benchmark relies on a single, small dataset. This further restricts the generality of the conclusions, potentially limiting their robustness.


Overall, this work presents a general framework that, while potentially relevant, still has critical limitations in terms of modeling techniques explored and datasets analyzed.

questions:
- The authors should include additional datasets, discussing trends across them.
- The authors should include multiple modeling paradigms, for example based on fine-tuning or exploring the space of different architectures for linear probing.
- The authors should include additional baselines, in particular specialized methods for perturbation effect prediction (e.g., GEARS), to better contextualize the results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712340268

thread_type:
official_review

<thread_object>
creator: reviewer_jl1w

<content>
summary:
This work presents PertEval-scFM for evaluating zero-shot single-cell foundation models (scFMs) in predicting cellular responses to genetic perturbations, using the Norman et al. dataset (2019).  Specifically, PertEval-scFM evaluates zero-shot embeddings from several existing scFMs—such as scGPT, scBERT, Geneformer, UCE, and scFoundation against baseline models based on gene expression matrix in the distribution shift scenario. The results suggest that zero-shot scFM embeddings do not consistently outperform simpler baselines, such as mean expression and MLP baselines across different evaluation metrics. As the sparsification probability increases, the performance of scFM embeddings declines more sharply than baseline models, suggesting that scFMs may be sensitive to distribution shifts. particularly in strong/atypical scenarios, with limited generalizability in zero-shot contexts of perturbation effect prediction.

strengths:
**More scFMs in benchmarking and distribution shift scenarios**: Compared to existing works on gene perturbation benchmarking,  PertEval-scFM considers more available updated scFMs and introduces distribution shift scenarios to evaluate the generalization of scFM better.

**Comprehensive Metrics**: By combining AUSPC, E-distance, and contextual alignment,  PertEval-scFM provides a detailed analysis of model robustness, distribution shift sensitivity, and alignment between pre-training and application-specific datasets.

**Reproducibility**: The PertEval-scFM code has been provided transparently and accessiblely, with the potential for further development and use by the community.

weaknesses:
**Novelty and Significance**: As the authors have noted in the introduction section, benchmarking scFMs on genetic perturbation tasks is not new, with similar conclusions being drawn.  The current benchmark indeed falls short in considering only one Perturb-seq dataset, making the results here rather preliminary. Using the datasets provided in existing benchmark works, such as Wu et al. (2024) could enhance the benchmarking breadth of current work. The author may also elaborate on addressing the key findings of PertEval-scFM by introducing distribution shift, especially in a biological context -- for example,  more quantitative and systematic analysis on the strong/atypical perturbations, and biological mechanism explanations/insights evidenced by bioinformatics analysis.

**The MLP Structure Selection**:  In PertEval-scFM， one hidden layer structure was used as an MLP probe to model the perturbation effects, and it served as a baseline. It's not clear how this selection influences the results since it's possible that the mapping between scFM embeddings and perturbation is highly complicated. Robustness tests could be added here. It is also insightful to investigate the role of scFM embeddings in more advanced or state-of-the-art deep learning-based prediction models such as GEARS[1] or CELLOT[2].

**Absence of Fine-Tuning Component**: While the zero-shot test provides insights into scFM performance, real-world applications frequently involve fine-tuning to adapt models. Without a fine-tuning component, the evaluation lacks a practical measure of scFM potential in real-world perturbation prediction settings, especially where significant distribution shifts are possible. Incorporating fine-tuning into the analysis, or at least comparing zero-shot performance to fine-tuned baselines, would yield a more comprehensive understanding of scFM effectiveness in applied perturbation prediction.

Overall, while the paper is not recommended to be accpeted by ICLR at current form, I would like to increase the score if the concerns are adequately addressed by the authors.

[1]Roohani, Y., Huang, K. & Leskovec, J. Predicting transcriptional outcomes of novel multigene perturbations with GEARS. Nat Biotechnol 42, 927–935 (2024). https://doi.org/10.1038/s41587-023-01905-6

[2]Bunne, C., Stark, S.G., Gut, G. et al. Learning single-cell perturbation responses using neural optimal transport. Nat Methods 20, 1759–1768 (2023). https://doi.org/10.1038/s41592-023-01969-x

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732193619930

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General rebuttal

comment:
**General Introduction**

We would first like to thank all reviewers for their time, as well as for their insights and comments on our work. We note all reviewers valued the scope of single-cell Foundation Models (scFM) evaluated in our Benchmark PertEval-scFM, corresponding to most currently available models (JL1w, SABf, 3E1k). The suite of comprehensive metrics was also noted as a strength: we incorporate AUSPC, E-distance and contextual alignment into our framework, with the aim of providing standardised evaluation protocols for a rapidly expanding field (JL1w, 3E1k). In particular, we highlight the importance of quantifying model robustness and sensitivity to distribution shift in a field with critical biological/therapeutic applications (JL1w, 3E1k). This highlights a crucial yet often overlooked aspect of machine learning evaluations: the importance of generalizability across diverse and unseen scenarios. Integrating the AUSPC and E-distance to our pipeline to quantify these aspects has, to our knowledge, not been proposed elsewhere to date (3E1k, yfBT). Finally, we are grateful that the reproducibility and transparency of our work is recognised by the reviewers (JL1w, SABf, 3E1k, yfBT), with the code base and single-cell embeddings of our benchmark being maintained and added to as a resource for our research community. 

**Aim**

The aim of this paper is to evaluate single-cell foundation models in a zero-shot context. This is because we want to quantify the information content of the single-cell embeddings in the context of perturbation prediction. The hypothesis is that due to the extensive pre-training on single-cell (sc) data these models have undergone, the embeddings obtained should contain biologically relevant information. Perturbation effect prediction is a major biological problem and knowing whether the models currently available are learning inherent biological principles pertaining to this task is an important research consideration. Identifying and understanding any limitations will guide the development of better models and help advance the field.

**Concerns**

In each following comment, we address the concerns which were raised by all reviewers and respond to individual concerns as comments to the corresponding reviews:
1. Fine-tuning
2. Additional Baseline
3. MLP parameter count
4. Dataset
5. SPECTRA and Dataset Splitting
</content>
</thread_object>

</thread>
<thread>
creation date:
1732814606892

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for acknowledging our hard work and for their detailed feedback, which has helped us enrich and expand our work. Taking on board all your comments, we have made substantial additions to the revised PDF, which we describe below:

**1 - Additional Baseline.** We have included **GEARS** as a **task-specific baseline**, enriching the contextual relevance of our comparisons. See **Section 2.2.2**, **Tables 2** and **3** and **Section 3.2** in the PDF.

**2 - Dual-gene perturbation.** We have extended our analysis to encompass not only single-gene perturbations but also **combinatorial perturbations (dual-gene)** from the Norman dataset. See **Section 2.1.2**, **Tables 2** and **3** and **Section 3.2** in the PDF. This is a non-trivial extension which provides a broader and more comprehensive evaluation of our framework. 

**3 - Additional Dataset.** Crucially, we have incorporated the **Replogle et al. (2022) dataset** [1] to our analysis. See **Section 2.4.1** and **Tables 2** and **3** for results in the PDF. Preprocessing this dataset (~2,000 perturbations) presents a significant computational burden, which we have been working hard to overcome. All data preprocessing is complete, and we have developed a fully functional featurization pipeline. Preliminary results, spanning **three models and two baselines across five SPECTRA splits**, are included as a demonstration of our progress. We anticipate completing the full set of results within one week.

**4 - MLP parameter count.** We included an analysis on the effect of MLP parameter count by training a range of MLPs with increasing parameter count. See **Section 2.2.1** and **Table D1**.

On the topic of fine-tuning, we maintain the original focus of our work: assessing the **zero-shot information content** of scFM embeddings for perturbation prediction. While fine-tuning is indeed a valuable direction, our primary aim is to **establish a framework for evaluating biologically meaningful perturbation effects in a zero-shot setting**, thereby probing the inherent information content of the embeddings. We therefore believe fine-tuning all the scFMs incorporated into our benchmark is well beyond the scope of this paper. Furthermore, this would require extensive computational and financial resources which we do not currently have at our disposal - and indeed which many people do not have, highlighting the importance of this framework and these results for the research community. 

We hope that by incorporating these additional baselines, datasets and experiments, we have managed to address your concerns and that you will take another global look at our submission. We remain committed to advancing this work and we thank you again for your time and your thoughtful feedback.

[1] Replogle, J. M. et al. Mapping information-rich genotype–phenotype landscapes with genome-scale Perturb-seq. Cell 185, 2559–2575 (2022).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733163976177

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle reminder for reviewers - End of discussion period summary

comment:
We want to conclude this rebuttal period by sincerely thanking all the reviewers for their thoughtful feedback and insights. We have successfully obtained further Replogle results. Specifically, we have trained and evaluated most models from split 0.1 to 0.7, including mean and gene expression baselines. The results so far obtained are in line with what we have described in the manuscript, giving further weight to the conclusions reached in the paper. We think that with the inclusion of this dataset, we have addressed the majority of the concerns brought up by the reviewers in their initial reviews, including: 

- **MLP parameter count**: Investigation into the effect of model capacity as parameterised by parameter count on our results 
- **Dual-gene perturbations**: Inclusion of dual-gene perturbations evaluated across all of the considered settings
- **Statistical significance**: Extension of Figures 3a and 3b to include dual gene perturbations and Spearman $\rho$. 
- **Additional Baseline**: Addition of GEARS baseline across single- and dual-gene perturbations in all of the considered evaluation settings 
- **Additional Dataset**: Inclusion of the Replogle dataset (most recent checkpoint included below)
- **Clarifications**: We have justified our initial focus and preference for the Norman dataset, highlighting its higher-quality perturbation signal. Furthermore, we have elaborated on the utility of leveraging a zero-shot framework to probe scFM embeddings and have provided a detailed explanation of why we consider fine-tuning is beyond the scope of this work.

We would like to respectfully invite the reviewers to consider if these additions and the extra work carried out during the rebuttal period have sufficiently addressed their concerns. Thank you again for all of your valuable feedback.

**Replogle results checkpoint 02/12/2024**

| Models | Split 0.1 | Split 0.2 | Split 0.3 | Split 0.4 | Split 0.5 | Split 0.6 | Split 0.7 |
|------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| Mean baseline | 0.21263 | 0.22914 | 0.22276 | 0.21774 | 0.2132 | 0.236 | 0.23873 |
| Raw Baseline | 0.21121 | 0.22697 | 0.22077 | 0.21537 | 0.21198 | 0.23183 | 0.23545|
| scGPT | 0.21145 | 0.22601 | 0.21988 | 0.21542 | 0.21098 | 0.23229 | 0.23334 |
| scFoundation | 0.21075 | 0.226 | 0.21989 | 0.21543 | 0.21102 | 0.26631| 0.23341|
| Geneformer | 0.2107 | 0.226 | 0.21988 | 0.21543 | 0.21098 | 0.24864 | 0.23337 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a benchmarking framework for single-cell foundation models on the downstream task of predicting the outcomes of genetic perturbations. The paper has many strengths: the topic is important, the writing is high-quality, and the evaluation is robust in some ways (in particular, it uses a comprehensive set of metrics). However, the use of a single, relatively small dataset was a serious concern for multiple reviewers. Some of the reviewers found the scope of the tasks to be relatively narrow and the absence of a finetuning component to be an issue. 

I very much appreciate the authors' responses during the rebuttal period (in particular, the use of the GEARS baseline). However, in the end, we believe it would be better for the authors to consider additional datasets and potentially broaden the paper's scope. Because these additions would be too substantial to pass without an additional round of reviews, I am recommending rejection this time around. We encourage the authors to improve the paper using the feedback in the reviews and to submit it to a different venue.