PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Authors: Aaron Wenteler
Martina Occhetta
Nikhil Branson
Victor Curean
Magdalena Huebner
William Dee
William Connell
Siu Pui Chung
Alex Hawkins-Hooker
Yasha Ektefaie
César Miguel Valdez Córdova
Amaya Gallagher-Syed
Venue: ICML 2025 poster (2025)
Keywords: benchmark
single-cell biology
perturbation effect prediction
Primary area: applications->health_medicine
TL;DR: Using PertEval-scFM, we benchmark zero-shot single-cell foundation model (scFM) embeddings and find that they do not outperform simpler baselines for perturbation effect prediction.

Abstract: *In silico* modeling of transcriptional responses to perturbations is crucial for advancing our understanding of cellular processes and disease mechanisms. We present PertEval-scFM, a standardized framework designed to evaluate models for perturbation effect prediction. We apply PertEval-scFM to benchmark zero-shot single-cell foundation model (scFM) embeddings against baseline models to assess whether these contextualized representations enhance perturbation effect prediction. Our results show that scFM embeddings offer limited improvement over simple baseline models in the zero-shot setting, particularly under distribution shift. Overall, this study provides a systematic evaluation of zero-shot scFM embeddings for perturbation effect prediction, highlighting the challenges of this task and the limitations of current-generation scFMs. Our findings underscore the need for specialized models and high-quality datasets that capture a broader range of cellular states. Source code and documentation can be found at: https://github.com/aaronwtr/PertEval.

PertEval-scFM:: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Aaron Wenteler +11
Martina Occhetta +1 Nikhil Branson 122 Victor Curean Magdalena Huebner'
William Dee
William Connell 4 Siu Pui Chung' Alex Hlawkins-Hooke Yasha Ektefaie
César Miguel Valdez Córdova $ 778 Amaya Gallagher-Syed :
Abstract
ment of biological foundation models. Among these, single- cell foundation models (scFMs) leverage vast amounts of un- labeled transcriptomic single-cell RNA sequencing (scRNA- seq) data to learn contextualized representations through self-supervised pre-training (Ericsson et al., 2022). Fine- tuning the resulting model on labeled data enhances the performance downstream applications, such as cell-type classification, gene regulatory network inference, and the prediction of cellular responses to perturbations (Yang et al., 2022; Kedzierska et al., 2023; Theodoris et al., 2023; Rosen et al., 2023; Cui et al., 2024; Wen et al., 2023; Hao et al., 2023).
In silico modeling of transcriptional responses to perturbations is crucial for advancing our under- standing of cellular processes and disease mech- anisms. We present PertEval-scFFM, standard- ized framework designed to evaluate models for perturbation effect prediction. We apply PertEval- scFM to benchmark zero-shot single-cell founda- tion model (scFM) embeddings against baseline models to assess whether these contextualized representations enhance perturbation effect pre- diction. Our results show that scFM embeddings offer limited improvement over simple baseline models in the zero-shot setting, particularly un- der distribution shift. Overall, this study provides a systematic evaluation of zero-shot scFM em- beddings for perturbation effect prediction, high- lighting the challenges of this task and the limita- tions of current- generation scFMs. Our findings underscore the need for specialized models and high-qualing datasets that capture broader range of cellular states. Source code and documenta- tion can be found at: https:/ : aaronwt::/erttat httpps////iee /github..... com/
A perturbation refers any intervention event leading to the phenotypic alteration of a cell. Perturbation response prediction can provide invaluable insights into cellular mech- anisms and disease progression, facilitating the mapping of genotype to phenotype and the identification of potential drug targets (Lotfollahi et al., 2019). Numerous models, here referred to as narrow perturbation prediction models (NPPMs), have been developed specifically for this task (Gavrillidis et al., 2024). However, perturbation response prediction is challenging task, as demonstrated by the difficulty of models to improve consistently over simpler baseline methods (Wu et al., 2024; Branson et al., 2024; Ahlmann-Fltze et al., 2024).
Recently, there has been a concerted effort to evaluate bio- logical foundation models. The Therapeutic Data Commons, an open science initiative, has curated some datasets, models and benchmarks for single-cell analysis (Velez-Arce et al., 2024). Additionally, Wu et al. (2024) and Ahlmann-Eltze et al. (2024) show that simple baseline models perform com- parably to scFMs in predicting transcriptomic response to perturbations. However, their analysis does not account for distribution shift and focuses only on predictions for highly variable genes, many of which show little to no effect in response to a perturbation (Nadig et al., 2024).
1. Introduction
Inspired by the success of foundation models in fields such as natural language processing (Devlin al., 2019; Brown al., 2020; OpenAI, 2024) and computer vision (Dosovit- skiy et al., 2021), there has been an increase in the develop-
"Equal contribution "Core contributor &Senior author 'Queen Mary University London UUniversity of Oxford 'University of Medicine and Pharmacy of Cluj-Napoca 'University of Cal- ifornia. San Francisco 'University College London Harvard University 'Mila *MeGill University. Conrespondence to: Aaron Wenteler <a. wa.wenteler@e www.tlerr@qmulla..uus Martina Occhetta <m. cm..cchetta@ @ qmul.ac.uk>
Yet, distribution shift is a well-documented issue with scRNA-seq data (Boiarsky et al., 2023; Marklund et al., 2020), which often hinders the deployment of models that appear perform well during evaluation. Distribution shift
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PertEval-scFM
can occur consequence of inherent technical and biolog- ical noise, abundant in scRNA-seq data, and, while scFMs have been proposed to mitigate such problems, there have been conflicting reports on their ability to do so (Theodoris et al., 2023; Cui et al., 2024; Wu et al., 2024). This high- lights the need for a comprehensive benchmark to evaluate their limitations and failure modes, specifically for distribu- tion shift.
and g corresponds to the number of genes in the dataset (see Appendix A.I).
2.1.1. DATA PREPARATION
Briefly, during our pre-processing we normalize and log- transform the raw expression count matrix C. We then select the top 2,000 highly variable genes (HVGs), v, ob- taining a reduced control matrix C' E R Addition- ally, we identify the top 20 differentially expressed genes (DEGs) for each perturbation to ensure that our evaluations capture biologically relevant gene expression changes (see Appendix A.2).
1.1. Contributions
Here, we present PeRtEval-SCFM to address these research
gaps by providing:
2.1.2. DATA FEATURIZATION
A standardized framework for evaluating biologically meaningful perturbation effect prediction a zero-shot setting. The source code and documentation can be found on our GitHub.
To generate the input features for our baselines, we randomly select 500 cells from C to form pseudo-bulk sample C. To combat noise and sparsity issues, we calculate the average expression across C and repeat this process Np times. The resulting basal gene expression vectors can then be paired with perturbed cells, resulting in control expression feature matrix X. E IR",** (see Appendix C.I).
Integration of a spectral graph theory method - SPEC- TRA (Ektefaic et al., 2024) - that allows us to assess model generalizability under distribution shift, a cru- cial consideration for real-world applications of scFMs.
Single-cell foundation model embeddings. To construct the control cell embeddings, we then feed our input matrix X into the scFM:
A toolbox of comprehensive metrics, providing a de- tailed analysis of model performance, focusing on as- sessing robustness and sensitivity to distribution shifts.
fsFM Ze.
Z. E Xe
(I)
2. PertEval-scFM
where e is the embedding dimension. To simulate genetic perturbations in silico, we adopt a universal strategy of nul- lifying the expression of a targeted gene. This approach motivated by prior work demonstrating that discrete manip- ulations of rank-order vectors yield biologically meaningful shifts in cell embeddings (Theodoris et al., 2023) and by our own experiments showing that alternative representations such as modeling a CRISPRa perturbation by expression doubling - not improve performance (see Appendix D. Furthermore, these representations ensure consistent test- ing conditions across all models, given the current lack of standardized methods for generating comparable in silico representations of perturbations. Perturbed cell embeddings Zp E IR are therefore generated by setting the expres- sion counts of perturbed genes to zero in cells exposed to that perturbation. The final input for the MLP probe is then formed by concatenating the control and perturbation embeddings (see Appendix C.2):
PertEval-scFM is designed to assess the zero-shot informa- tion content of scFM embeddings for perturbation effect prediction. To achieve this goal, we obtain zero-shot embed- dings from five pre-trained scFMs across four datasets, then train a multi-layer perceptron (MLP) probe for each scFM. This method mirrors established probing techniques to as- sess the semantic content of embeddings (Alain & Bengio, 2018; Jin et al., 2019). This approach enables fair evaluation of the base information content of embeddings across mod- els, as it evaluates representation quality while removing confounding effects introduced by task-specific prediction heads (Tenney et al., 2019; Radford et al., 2021). In Figure I we present an overview of the pipeline, composed of three parts: data pre- processingg training, and evaluation.
2.1. Data Pre-Processing
To interrogate cellular response to perturbations, we use high-dimensional Perturb-seg screens, which combine single-cell RNA sequencing with netic perturbations, enabling systematic profiling of tran- scriptional landscapes at single-cell resolution (Dixit et al., 2016). Perturb-seq data consists of transcriptomic data for unperturbed control cells C E R and perturbed cells P E R9 where ne and np correspond to the number of control and perturbed cells being measured respectively,
ZudtM Ze + Zp
(2)
ge-
Raw expression data. To serve as a baseline against which to compare the performance of the scFM embeddings, we use our input matrix X,. Here, we model single-gene perturbations by calculating the gene co- ro-expression matrix Go E RV between the perturbed genes and the highly
2

---

PertEval--cCM
PRE-PROCESS
TRAIN
EVALUATE
Data preparation
Featurization
Modeling distribution shift
Probing
Evaluation toolkit
Perturb-seq data
AAUSPC
Train-test splits SPECTRA
"I" Beseline Model
MSE (4)
starsppic
Spartilicationn probabilty
0.5
Pertateedods)
O
0.5 Spersification probability (x)
E-distance
Contectual alignment
plamic lecaal
D
Ox
Our
cample
Normon ot at. 2819
0313
MLP probes
Replogl=
Figure 1. PertEval-scFM framework (left to right) data pre-pro pre-processing, training of MLP probes under different sparsification conditions;
evaluation of trained models with AUSPC, E-distance and contextual alignment metrics.
variable genes in Xe- Similarly, for double-gene pertur- bations, we calculate the co-expression matrices for the individual perturbations, and then average them to obtain Ge. We then concatenate the control and perturbation em- beddings to form the final input for the MLP probe (see Appendix C.
with the SPECTRA framework. All other training config- urations, hyperparameters, and preprocessing steps remain at their default values. To ensure consistency across our benchmark, we train GEARS from scratch without using pretrained weights. While GEARS a task-specific model, it shares the same supervised training setup as the other methods we evaluate. The key distinction lies in how rep- resentations handled: GEARS learns representations end-to-end, while our probes evaluate the information con- tent of fixed inputs (transcriptomic counts or zero-shot scFM embeddings) under controlled conditions.
ZGR = X ① Go.
(3)
2.2. Baseline Models
We establish baseline models against which compare the performance of the MLP probes trained with ScFM embed- dings.
Mean baseline. The mean bascline assumes that pertur- bation has little effect on the perturbed cell's gene expres- sion. This reflects the biological reality that most perturba- tions result in small changes in gene expression, providing simple biologically plausible null model highlighting the challenge inherent in distinguishing meaningful perturba- tion effects from background variability in single-cell data. The predicted perturbation effect, ô, is then simply com- puted as the deviation of the cell's gene expression, X from the mean gene expression of all cells in the same con- text, X as defined by 5 X X
MLP baseline. The MIP baseline uses log-normalized raw gene expression data directly as input. This allows us to ensure that any performance differences can be traced back to the semantic information introduced into the embed- dings by the scFM. The perturbation effect is predicted as follows:
(Zaz) We + b,) + b2,
(4)
where dimensions of parameters y correspond to W, E hxx22 W2 gexh, bj R and b2 IR..
2.3. Training
GEARS baseline. To benchmark zero-shot scFMs against existing task-specific models, we implement GEARS, state-of-the-art model that integrates raw gene expression data with known biological priors via a graph-based archi- tecture (Roohani et al., 2023). We reproduce the original implementation, modifying only the train-test splits to align
1. MLP PROBE FOR PERTURBATION EFFECT PREDICTION
MLP probe design. A 1-hidden layer MLP was selected as a probe for its flexibility and simplicity in handling var- ious types of data representations. For each perturbation,
3

---

PertEval-scFM
the MLP learns the log fold change perturbation effect 6, defined as:
magnitude, and (iii) contextual alignment to measure how the overlap between pre-training and fine-tuning datasets influences model performance. Together, these metrics pro- vide a robust basis for comparative scFM evaluation while capturing distinct aspects of predictive performance in tran- scriptomic perturbation response modeling.
P X
(5)
where P E 1R X represents the perturbed gene expression matrix. The MLP probe predicts the perturbation effect, denoted by ò, described by the following equation:
2.4.1. AUSPC
80 (Z = ReLU + b, + +b2
(6)
To evaluate robustness under distribution shift, we adapt the approach introduced by Ektefaie et al. (2024) and define the AUSPC as:
The model parameters 0 include the weight matrices W1 E hx22 and W2 E TRex,, where h corresponds to the dimen- sion of the hidden layer, and the bias vectors b E I& and b2 E R
AUSPC = J. Frmm
((5) ds
where Ø(s) is the MSE as function of the sparsification probability used to define each train-test split. Integrating the MSE across yields single performance metric that reflects a model's ability to generalize under increasing distribution shift. The integral is approximated with the trapezoidal rule (see Appendix F.2).
MLP probe parameters. To assess whether parameter count impacts model performance, we analyze the sensitiv- ity of MLP probes to varying model capacities. We train a series of MLP probes with increasing parameter counts on both raw gene expression data and scFM embeddings. The results, summarized in Table E2, show no meaningful relationship between probe capacity and prediction perfor- mance. Additional details are provided in Appendix E.2.
Motivated by the observation that simple baselines often perform surprisingly well in perturbation prediction, we in- troduce the AAUSPC metric. This metric anchors a model's s robustness to the mean baseline. The AAUSPC is defined as:
2.3.2. MODELING DISTRIBUTION SHIFT
To assess the robustness of the MI.P probes when using either gene expression data or scFM embeddings, we im- plement SPECTRA (Ektefaie et al., 2024), graph-based based method that partitions data into increasingly challenging train-test splits while controlling for cross-split overlap be- tween the train and test data.
AAUSPC - (db(s) - &m (s)]ds
(8)
Here, O represents the MSE of the mean baseline, and @m is the MSE of the model being evaluated. positive AAUSPC indicates that the model outperforms the baseline, while negative value suggests the opposite. This metric provides clear measure of a model's generalizability improvement over simply predicting the mean perturbation effect.
In SPECTRA, edges within the graph represent sample-to- sample similarity. The connectivity of the similarity graph is controlled by the sparsification probability (s). For each split, this connectivity is adjusted by stochastically remov- ing edges with probability s. We introduce the constraint < mmax where Smax is empirically chosen to ensure suffi- cient number of samples in both the train and test sets. After sparsification, the train and test sets are sampled from dis- tinct subgraphs. As the sparsification probability increases, the degree of similarity between the train and test sets de- creases, making it harder for the model to generalize to unseen perturbations effectively (see Appendix F).
2.4.2. E-DISTANCE
We use the E-distance, introduced by Peidli et al. (2024), to quantify the difference between perturbed and control cell gene expression profiles (see Appendix G.I). This metric captures both within within-group group variability and distributional dif- ferences, offering a robust measure of perturbation effect strength. With E-distance, we can better analyze the char- acteristics of perturbations that models handle well versus those they struggle with, providing context for model per- formance-ppartiullat particularly in cases where outlier perturbations may not be immediately apparent using traditional metrics.
2.4. Evaluation
Following the empirical findings from Ji et al. (2023), we adopt Mean Squared Error (MSE) as our primary evalu- ation metric. In addition, address the current lack of standardized evaluation metrics for perturbation effect pre- diction, we propose using three complementary metrics: (i) Area Under the SPECTRA Performance Curve (AUSPC) to quantify model generalization across distribution shifts, (ii) E-distance to quantitatively measure perturbation effect
2.4.3. CONTEXTUAL ALIGNMENT
While pre-training dataset size is often linked to improved downstream model performance, recent research empha- sizes the critical role of data quality over dataset size (El- Nouby et al., 2021; Fournier et al., 2024). We therefore sug- gest the inclusion of a contextual alignment metric, which
4

---

PertEval-scFM
quantifies the similarity between the pre-training and fine- tuning datasets, and its effect model performance. We calculate the cross-split overlap between the pre-train and fine-tune datasets using cosine similarity to determine how representative the pre-training data is the fine-tuning data (see Appendix H. l).
across all models. However, scFMs showed steeper performance drop at higher sparsification levels compared to the MLP baseline, suggesting that the zero-shot embeddings are less robust to distribution shifts.
Norman double-gene. The ranking pattern observed for single-gene perturbations persisted for double-gene pertur- bations, with GEARS and the MLP baseline performing better than the scFMs. Most scFM models displayed similar performance, with a notable exception of scGPT, which ex- perienced a significant drop in performance, moving from rank 3 to rank 8. This increased the performance spread, with a larger AAUSPC difference (AAUSPC range of 0. 0.00980 for double-gene perturbations compared to 0. 0.00126 for single-gene), highlighting greater variability in model performance for the more complex perturbations. In line with our expectations, increasing the sparsification probabil- ity led to higher MSE across all models.
2.5. Models and Datasets
2.5.1. SINGLE-CELL FOUNDATION MODELS
PertEval-scFM currently evaluates the performance of the following five scFMs: scBERT (Yang et al., 2022), Gene- former (Theodoris et al., 2023), scGPT (Cui et al., 2024), scFoundation (Hao et al., 2023) and UCE (Rosen et al., 2023). See Appendix B.1 for details on their architecture and pre-training data.
2.5.2. DATASETS
Norman. PertEval-scFM is applied to the 105 single-gene and 91 double-gene perturbation datasets derived from a Perturb-seq screen in K562 cells from Norman al. (2019). These datasets contain strong CRISPRa perturbation signals, well as baseline expression for unperturbed cells, which allows for the systematic evaluation of model performance in predicting the effects of genetic perturbations at single- cell resolution.
Replogle K562. For the K562 cell line, GEARS main-
tained its superior performance with an AUSPC of 0.0082, followed by scGPT and UCE, both with an AUSPC of 0. 1384. The MI. baseline only ranked fifth with an AUSPC of 0.1420. Apart from GEARS, none of these differences were statistically significant, as indicated by the overlapping error bars in Figure 2b.
Replogle. Additionally, we apply our framework to the two single-gene perturbation datasets from Replogle et al. (2022), which profile transcriptomic responses CRISPRi- mediated genetic perturbations in both K562 (2,058 pertur- bations) and RPE1 (2,394 perturbations) cells. Compared to the Norman dataset, the overall perturbation effect signal in Replogle et al. (2022) is less pronounced, despite the con- siderably larger number perturbations (Peidli et al., 2024). For additional details on the datasets, see Appendix A.
+0.31
+0T
3. Results
3.1. Evaluation across 2,000 HVGs
In our initial evaluation, assess the models' ability to predict the effect of perturbations on the top 2,000 HVGs. We present our evaluation results in Table 1, Table and Figure and discuss the results for each dataset below.
Norman single-gene. For single-gene perturbations, GEARS achieved the highest performance with an AUSPC of 0.00815, significantly outperforming all scFM models. This is followed by the MLP baseline at 0.0448, which is slightly better than the scFM models. Performance differ- ences between the scFMs were minimal, suggesting similar capabilities in predicting perturbation effects. As the sparsi- fication prohability s increased from 0.1 to 0.7, MSE values
Mariede
Manboan
-2.07
+507
05
Figure 2. Average AUSPC (L) across sparsification probabilities for each model with standard error bars. (a) Norman single-gene (left) and double-gene (right) perturbation (b) Replogle K562 (left) and RPE1 (right)
5

---

PertEval-scFM
Table 1. Perturbation effect prediction evaluation across 2,000 HVGs. Models are listed in the specified order.
MSR (10-3)
Dataset
Model
$0.11
$0.2
$0.33
$8.44
$0.5
S0.6
$
(10SPC((10-)
"AAUSPC 2)
Rank
GEARS
202
0.937
0.177
1,120
0.167
0.378
1.750 0.401
1.0077
1,000 0.257
0.815 4-000
3.7968
buseline
6.288
0.282
64400
6.609
0.705
0.584
5.984 0.458
6.502
1.277
7,065 1,022
4.484++00 0-0.99
0.1280
2
Norman single
Mean baselime
6.177 1: 0.264
5:960
1: 0.621
6.497 1 0,513
6.219
1 0.308
6.659
1 0.154
7,413 1: 1,038
8:430 1. 0.540
1 46.221.0.117
5
Geneformer
6.257 + 0.049
6.132 +: 0.63
6.565
0.530
6.305
0.300
6,500 +1 0,140
7.382 + 1.155
8.525 = 0.494
4.61 + 0.300
-0.0396
8
ScBERT
0.316
0.356
6.761
0.765
6,363
0.544
5.924
6.451 1.200
8,488 0.558
4.537 0.266
0.0748
sehoundation
6.421
= 0.317
6.366
+: 0.356
6,793
4: 0.764
6,440
= 0.538
5.919
= 0.417
6.705 = 1.183
8,601
= 1.537
4.594 #: 0.246
0.0179
5
scGPT
6.237 a: 0.218
0,340
à: 0.008
6,765
0.428
6,363 a 0.345
5.926 à 0.174
6,400 4 1,144
8,505
2: 1,020
4.525 4: 0.255
008663
3
6.258
6.132
0.307
1,065
8,479
0.312
-0:0355
Normun double gene
GEARS
0.783
0.044
0.960 +
1,153 0.049)
1,250
0.280
1,467 + 0.351
1.223
0:147
18800 0.287
0,508 0.028
4.254
MLP baseline
5:261 + 0.100
5,913 + 0.255
5.728 +: 0,402
6.635
= 0.161
7.675 # 0.953
6,050 = 0.763
5.198 = 115993
4:253 # 01073
0.002
2
Mein havelline
5.257 +-0 102
5.910 + 0.255
5,722 0.401
6.644
0.167
7.674 + 0.962
6.071 + 0.772
5.201 + 0.594
4:255 + 0.073
Geneformer
5.514
6.145 0.188
6.029
0,458
0.257
7,937 1
1,246 0.707
5,179
0.088
-0.248
sBBERT
5.515 0.006
6.159 0. 0.19
6.022
0.465
6.757
0.281
7,999 1: 1,240
6.493 0.736
7,110 0.579
4.533 1: 0.081
-0.278
seFoundation
5.564 1: 0.051
6.173 1: 0.196
6,050 1: 0.462
6.755
1: 0.279
7,944 1: 1.186
6.382 1 0.876
5.238 = 0,578
4.432 1: 0.467
-0.177
seGiPT
0.067
6,023
0.464
0.257
0.182
14,906
5.184 0.132
40:929
UCB
5,514 0.066
6.145
6.029
0,460
0.289
7,939
1.184
0.831
5,612 =: 0.665
4,435
.0.180
Table Perturbation effect prediction evaluation across 2,000 HVGs. Models are listed in the specified order.
MSF
Dataset
Model
so.1
5
S.0.3
90.44
0
5
$
LAUSCC
AAASSP (10-3)
Rank
GEARS
0.0006 a: 0,0005
0.0102 1: 0.0005
0.0129 a: 0.0003
0.0133 1: 0,0005
0.0150 1 0,000
0,0169 :: 0.0016
0.0175
0.0009
0:0082
0.0001
13044
Replogle K342
MLP bascline
0.2125 = 0.0007
0.2345 à 0.0013
0.2343 à: 0.0055
0.2369
0,0084
0,2437 à: 0,0151
0.2445 :: 0.0348
0,2799 2: 0.0279
0,3420
0.0077
-0,3353
Mean bastline
022129 + 0.0007
0,2222 + 0.0017
0.2319 + 0.0091
0.2152
0.0089
0.2156
+ 0.0051
0,2435 + 0.0067
0.2200
0.0947
0.1386
0.0076
Gendleeme
0.224
0.0028
0.0014
0.515
0.0000
0,2412
0.005
0.0047
0,2526 0.0000
0.3630
0.0830
0.1475
0.0056
-0.6867
ScBEK
0.2071
0.0014
02003
0.009
0.2060
0.2002
0.002
0,204
0.2021
0,0055
0.2012
00000
0.1228
scFoundtion
02386
0.0033
0.2566
0.0009
0.2882
0.0002
0,2644
0.0151
0,288
0.0141
0,2506
0.0778
0.3639
0.031
0.1556
00013
SGGPT
0,2107 0.0
0.2245
0.0016
0.2287
0.0086
0.2223
0.0081
0,321 1 0:0213
0,2390 1: 0.0245
0.2635
0.0827
0.1384
0,0080
0,0221
UCE
02114 = 0.0007
0.2245
+ 0.0016
(.2287 + 0.0085
0.2223
+ 0.0081
0,2321 + 0.0012
0.2391
+ 0.0245
0,2635
0.0826
0.1384
= 0.0000
0,02211
GEARS
00154
0.004
D.188 + 0.0001
000277 + 0.0005
0.0154
0.0018
0,0285 + 0.0026
0.8126 + 0.0048
0.0454 + 0.0077
0.0157
0.005
11.88
Replingle NPRE
MLP bascline
0.2088 = 0.0033
020077
+ 0.007
0.2345 : 0.0011
0,2056
0,0053
0,2101 +: 0,0053
0,2167
+ 0.0073
0,2112 +: 0.0042
0,1252
0.0124
0
Mean baseline
0,2177
0.0015
0,2215 0.002
0.2190
0.0013
0,224
0.0087
0,2201
0:01
0,2153
0.0106
0.2246
0.0101
0,1341
0.0055
Genefermer
0.0019
0.2002
0.0085
0.0037
0.1251
0.0033
<cBRRT
02080 3: 0.0029
0,2066
0.003
0.0019
0.2062
0,0054
02113
0.0059
0,2195
0,0070
0.2178
0,0088
0.1258
0.0027
seFoundatio SGGPT
020154
0.0021
0.205%
0.0044
0.2048
0.0020
0.2063
0.0056
021113 0.0064
0.2173
0.0068
0.2101
0.0035
0.1253
0.0023
0.8764
0.2053
=: 0
0.2056
0.0023
0.2047
0.0019
0.2062 a: 0,0055
0,2112
1: 0.0065
0.2171
#: 0,0067
0.2103
0,0038
0.1253
0.0023
0.5768
UCC
0.2053
+ 0.0021
0.2055 + 0,0023
0 2048 0.0019
0.2062 + 0.0054
0.2115
+ 0.0054
0,21 + 0.0066
0.2/04
0.0037
0.1251
0.0024
0.8791
Replogle RPE1. In the RPEI cell line, GEARS again out- performed all other models with AUSPC of 0.0157, fol- lowed by the Mean baseline (0.1251) and the MLP baseline (0. (0.1252). The scFMs exhibited very similar performance, with AUSPC values tightly clustered around 0.1255 and no significant differences observed.
turbation. Poor performance in this setting would indicate that the semantic information encoded in zero-shot scFM embeddings does not meaningfully contribute to predict- ing perturbation effects. Lastly, we evaluate the impact of contextual alignment on performance for two of the scFMs.
1. EVALUATION ACROSS DEGS
Overall, GEARS outperforms all zero-shot scFMs and base- line models by an order of magnitude, suggesting that its architecture and training paradigm allow it to better cap- ture underlying biological processes and generalize more effectively across a variety of perturbation types. This under- scores the potential value of incorporating stronger inductive biases into the perturbation effect prediction task and sug- gests that current scFMs, which rely solely on a masked pre-training objective, may primarily capture average per- turbation effects in the absence of fine-tuning.
We evaluate the models on predicting the effect of pertur- bations on the top 20 DEGs per perturbation, providing a more stringent test of model performance than the full set of 2,000 HVGs. Indeed, genetic perturbations typically al- ter the expression of a limited subset of genes within the transcriptome (Figure A4), hence models predicting mean gene expression can still achieve low MSE values across 2, 2,000 HVGs. The results of the evaluation across DEGs are displayed in 3.
Norman single-gene. For single-gene perturbations, GEARS achieved the highest performance (AUSPC 0.266), significantly outperforming all scFMs and baseline models. The MLP baseline ranked second (0.342), followed closely by UCE (0.334), which was the best-performing scFM. As the sparsification probability increased, MSE values worsened across all models, with Geneformer and SCGPT exhibiting the steepest decline in performance. scBERT performed best among scFMs across most sparsity levels
3.2. A Additional Experiments
We conduct additional experiments on the Norman single- and double-gene perturbation datasets, which exhibit stronger perturbation effects and thus provide a clearer sig- nal for evaluating more challenging prediction tasks. Specif- ically, we assess the models' performance when restricting the evaluation to the top 20 DEGs, which typically capture the majority of the transcriptional response to a genetic per-
6

---

PortEval-scFM
Table Perturbation effect prediction evaluation across the top 20 DEGs per perturbation. Note that for double-gene perturbations split 0.5. there were not enough perturbations that passed our quality control to define multiple replicates.
MSF
Thitaset
Model
$0.1
50.2
50.4
50.5
$0.6
50..7
[ LAUSPC
AAASSPC (10- ?)
Rank
GEARS
0.254 a: 0.024
0.215 =: 0.837
0.314 a: 0,001
0.256 1: 0.055
0.341 =: 0,014
0.082 #: 0.194
0.888 4: 0.265
0.266 = 0.018
7,967
MI.P gene expression
01466 + 0.051
0.48
0.074
0.456 0.029
0.477 0.042
0.521 = 0.077
0.513 0.123
0.622 + 0.172
0.34 - 0.013
0,312
Normun single
Mcaa beseline
0.479 0.050
0.444
0.489 0.053
0,492
0.144
0.525
0.126
0.004 0.142
0.345 0.011
Geneformer
0.464 + 0.048
04644
0.481 0.052
0,75
0.022
0.483
0.046
0.483
0.902 +.0.20
0.31
00014
-0.664
scBERT
0.469 + 0.050
0.444 0.077
0.451 + 0.051
0.755 + 0.042
0.482 = 0.045
0.499
+ 0.117
0.608 + 0.149
0.33 - 0.011
0.878
s:Foundation
0.502 1: 0.052
0.466
0.077
0,489 0.056
0,469
0.040
0.486
0,046
0,567 1: 0.060
0/638 1 0.166
0.350
-0011
-0.455
scCPT
0.463 0.046
0.464
0.047
0.485
105
0.878 +.3449
0.377
-0.66
UCE
0.463 1. 0.048
0.164
0.077
0,482 1 0.053
0.476 1 0.012
0.485
0,047
0.484 1. 0.104
0.624 1 0.162
0.334 0.012
1,078
GEARS
0.211 + 0.032
0.200 + 0.013
0.2% + 0.052
0.425 + 0.01
0.385
0.0"
0.473
D. 109
0.422 + 0.07
01233
0.010
20,9
MLP expression
0.484 1: 0.046
0:538
0.052
0,585
0.061
0,618 1:18-1100 0.048
0.690
0.0"
0,552 1: 0.049
0.500 1 0.056
0.371
00,009
15.1
Norman doubll
Mean
0.54p 0.055
0.580
0.074
0.653
0.777
0:497 0.066
0.522
Geneformer
0.527 1. 0.055
0.550 1. 0,069
0.603 0.076
0,561 1 1.0055
0,706 0,0"
0.623 1 0.054
0,487 1 0.088
0.009 0.008
11.3
seBERT
0,528 1: 0.056
0.550 : 0.069
0,596
0.071
0.661 1: 0.40
0,740 :: 0.0"
0.622 1: 0.049
0.681 1: 0.066
0418 0.008
10.4
scFoundation
0.534 0.057
0.554 0.070
0,606
0.003
0,656
0.045
0.683 0,0"
0,621
0.497 0.51
0.410
0.008
11.2
scGPT
0.527 0.056
0.550
0.597
0.072
0,673
0.724
0.0"
1,911 0.329
0.500
UCE
0.527 :: 0.055
0.550
0,009
0.001
0,072
0.656
0.043
0.726 = 0.0
0.024 = 0.053
0,506 #: 0.048
0.410 0,007
11.2
(AAUSPC 0.00878), while UCE provided the most stable results throughout (AAUSPC 0.0108). These results sug- gest that, while some scFMs exhibit marginal advantages over the mean baseline, the performance gaps remained minimal, with UCE (best) only outperforming Geneformer (worst) by 4.8%.
ability to generalize more extreme cases.
Figure 3b further illustrates how perturbation strength varies across train-tes splits for both single- and double-gene per- turbations. At higher sparsification probabilities, low E- distance perturbations become less frequent, while stronger perturbations appear more often. This aligns with previous observations that model performance declines as sparsity increases, models are increasingly challenged to predict relat relatively rare, and strong perturbation effects. Two exam- ples illustrate this trend: AHR (Figure 4a), a perturbation with low E-distance, exhibited a narrow effect range (-0.1 to 0.25) and was predicted with relatively high accuracy. In contrast, CEBPE (Figure 4b), which produced a broader and more pronounced perturbation effect (-0.5 to 1), was predicted with significantly lower accuracy.
Norman double-gene. similar ranking pattern was ob- served for double-gene perturbations, where models outper- formed the mean baseline but showed no clear advantage over the MLP. GEARS again demonstrated superior per- formance (AUSPC 0. D,2 AAUSPC 0.299), surpassing the MLP baseline (AUSPC 0.371, AAUSPC 0.151) and all scFMs. The performance gap widened for more complex perturbations, suggesting that GEARS' architecture and training paradigm allow better model gene interactions and their impact on the effect of perturbations.
However, deviations from this trend suggest that the mag- nitude of the perturbation alone does not fully determine prediction difficulty. CEBPA (Figure 4d), despite being a strong perturbation, was predicted with relatively high accuracy. This could be attributed to its localized effect a small subset of genes, with long tail of mildly or
Overall, this evaluation proves more challenging, evidenced by the order of magnitude increase in MSE (see Ap- pendix I.4). However, the results again suggest that zero- shot scFM embeddings provide limited advantage over sim- ple baseline approaches in this setting. Consistent with previous findings, ScFM models struggled to generalize to perturbation -ppeciiittioo-eisiiiiii expression shifts, further reinforcing their limitations in predicting biologically relevant perturba- tion effects.
3.2.2. H- DISTANCE
We analyzed the relationship between perturbation strength and model performance using E-distance to measure pertur- bation effect magnitude. The results, shown in Figure 3a, confirm that models generally perform worse when predict- ing perturbations with higher E-distance, which corresponds to stronger perturbation effects. This pattern was consistent across all models, for both single-gene and double-gene perturbations, supporting the hypothesis that training data biased toward moderate perturbation effects limits a model's
(a)
(b)
Figure 3. (a) MSES for all test perturbations as function of the E-distance. The predictions displayed the averaged across all scFMs. (b) The E-distance of all test perturbations stratified per split as function of the sparsification probability. The mean of the E-distance per split is included in red.
7

---

PertEval-scCMM
14
120110 819 19 880 19000 488
Figure 5. MSE as a function of the pre-train and fine-tune data cross-split overlap for scGPT and scBERT.
Figure 4. Predictions of models across the top 20 DEGs for 4 per- turbations from different splits. Subcaptions indicate perturbation name, spansification prohability. The predictions are included as colored dots, and the target perturbation effect displayed as dashed line.
significant role than dataset scale alone. Notably, scGPT's pre-training corpus is an order of magnitude larger than scBERT's, reinforcing the idea that increasing dataset size does not necessarily compensate for a lack of alignment with the downstream task. For this analysis we focus on scGPT and scBERT, however assessing this phenomenon across a broader range of models and perturbation datasets is nec- essary to fully understand its impact on perturbation effect prediction. Future studies should explore how curating pre- training datasets to better reflect perturbation-specific distri- butions influences model performance, particularly in cases where strong or rare perturbations are represeereee in the training corpus.
non-affected genes. Conversely, IKZF3 (Figure 4c), which elicited a weaker overall perturbation, was predicted with lower accuracy-likely due to its atypical effect distribution (Appendix I.7). These results suggest that a model's capa- bility to predict perturbation effects depends not only the magnitude of the perturbation, but also on its distribution.
Taken together, these findings highlight the importance of a more balanced representation of perturbation effects during training. Ensuring that training data covers a diverse range of perturbation magnitudes and distributions could improve model generalization. Zero-shot scFM embeddings alone do not address this challenge, reinforcing the need for targeted strategies to enhance model robustness for perturbation ef- feet prediction.
3.3. Limitations
A key limitation of our study is that perturbations are mod- eled exclusively as knockouts, regardless of their original experimental context. This constraint aligns with prior work and reflects an inherent limitation of current foundation models, as not all architectures support the modeling of gene activation. More broadly, the lack of standardized approach for representing perturbations in scFMs limits the interpretability and generalizability of findings across studies. To advance the field, there is a pressing need for consensus on how to represent perturbations, ensuring they can robustly capture diverse range of modalities.
3.2.3. CONTEXTUAL ALIGNMENT
Previous work by Cui et al. (2024) demonstrated that the performance of zero-shot scFM models in cell-type annota- tion tasks is highly dependent the overlap between their pre-training datasets and the downstream task data. To inves- tigate whether this reliance on contextual alignment extends to perturbation effect prediction, we analyzed the similarity between pre-training corpus and fine-tuning dataset, as well as its impact on model performance.
4. Conclusion
PertEval-scFM addresses the current lack of consensus in benchmarking models for perturbation effect prediction by introducing modular evaluation toolkit with diverse met- rics, designed assess and interpret model performance. Notably, framework accounts for distribution shift, factor often overlooked in previous studies. We applied PertEval-scFM to evaluate whether zero-shot scFM embed- dings provide an advantage over raw gene expression data for perturbation effect prediction. Our results show that current-generatio scFM embeddings, when used zero-shot,
In Figure 5, we compute the contextual alignment between the pre-training datasets of scGPT and scBERT and the Norman dataset. scBERT exhibits higher alignment score (0.718) compared to scGPT (0.606), indicating that its pre- training corpus is approximately 19% more similar to the Norman dataset. Despite both models displaying compara- ble MSE across splits, scBERT demonstrates greater robust- ness, suggesting that contextual alignment plays a more
8

---

PertEval-scFM
do not consistently outperform simple baselines when pre- dicting the effects of perturbations 2,000 HVGs or the top 20 DEGs. The E-distance and contextual alignment metrics allow us to further contextualize our results and identify the failure modes of our models. We plan to main- tain PertEval-scFM as an open and extensible benchmarking suite, designed to support diverse use cases and facilitate the evaluation of perturbation models.
Impact Statement
This paper aims to advance the field of machine learning by providing a standardized framework for benchmarking perturbation effect prediction models. Our work has the potential to improve the evaluation of computational models used in drug discovery and precision medicine by ensuring rigorous and reproducible assessment of their predictive capabilities. By highlighting the limitations of current zero- shot single-cell foundation models (scFMs) in perturbation effect prediction, we encourage future research to develop more robust and biologically informed architectures.
Future work. Although our results highlight the limita- tions of using zero-shot scFM embeddings for perturbation effect prediction, we remain optimistic about the wider po- tential of scFMs. Key open questions include how to best represent perturbations in silico and how to fully leverage large-scale pre-training data to improve prediction accuracy. Existing cell atlases capture only a small fraction of the hu- man phenoscape- the full range of states cell can occupy (Fleck et al., and often exclude enturbationninduced states. Moreover, specialized models must be designed to fully leverage large-scale datasets for predicting transcrip- tomic responses to perturbations. The superior performance of GEARS, which incorporates inductive biases tailored to perturbation prediction, exemplifies the importance of model architectures that explicitly encode relevant biolog- ical priors. Future work may benefit from exploring how such priors can be integrated into the scFM architecture to improve the utility of zero-shot representations of single-cell RNA-seq data for perturbation effect prediction.
While findings do not have immediate ethical concerns, the broader application of machine learning in biological and clinical settings requires careful consideration of bi- ases in training data and the potential misinterpretation of model predictions. Ensuring that predictive models gener- alize across diverse biological contexts is crucial to avoid- ing misleading conclusions that could impact downstream biomedical applications. Future efforts should focus on increasing the diversity and representativeness of training datasets, as well as fostering transparency in model evalua- tion, to maximize the real-world utility of these approaches.
References
Ahlmann-Eltze, C., Huber, W., and Anders, S. Deep learning-basee predictions of gene perturbation effects do not yet outperform simple linear methods, September 2024.
Computational Requirements
A single MLP probe requires 1 NVIDIA A100-PCIE-40GB GPU (using 12 cores) for training. Runtime depends on the hidden dimension of the probe, which is around 5 to 30 minutes for the smallest to biggest probes, respectively.
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, November 2018. 1XXi:1610.01644 [stat].
Bengio, Y., Courville, A., and Vincent, P. Representation Learning: A Review and New Perspectives. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 35 (109/TPAMI.0013.55
Acknowledgments
The authors would like to thank Martino Mansoldo, Ionelia Buzatu, Pascal Notin, Andreas Bender, Kristof Szalay, Yuge Ji, Charlotte Bunne, Patrick Schwab, Djorde Miladinovic, Francesco Piatti, Michael Barnes, Claudia Cabrera, and Wei Wei for their helpful feedback and discussions.
1898- 2013. doi:
Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. Algo- rithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
A. A.W., M.O., W.T.D. and S.P.C. receive funding from the UKRI/BBSRC Collaborative Training Partnership in AI for Drug Discovery and Queen Mary University of London. N.B. and A.G.S. receive funding from the Wellcome Trust [218584/2/19/Z]
Boiarsky, R., Singh, N., Buendia, A., Getz, G., and Son- tag, D. A Deep Dive into Single-Cell RNA Sequencing Foundation Models, October 2023.
Branson, N., Cutillas, P. R., and Bessant, C. Understand- ing the Sources of Performance in Deep Drug Response Models Reveals Insights and Improvements, November 2024. Pages: 2024.06.05.597337 Section: New Results.
This research utilised Queen Mary's Apocrita HPC facility, supported by QMUL Research-IT (King et al., 2017).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.

---

PertEval-scFM
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Bemnrr, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901. Curran Associates, Inc., 2020.
Ericsson, L.,, Gouk, H., Loy, C., and Hospedales, T. M. Self-Supervised Representation Learning: Introduction, advances, and challenges. IEEE Signal Processing Mag- azine, 39(3))42-60 62, May 2022. ISSN 1558-0792. doi: 10.1 1099MMM.202..1333435 Conference Name: IEEE Signal Processing Magazine.
Fleck, J. S., Camp, J. G., and Treutlein, B. What is a cell type? Science, 38 (6666))7333-734 August 2023, doi: 10. 1126/ccence. adf6 162. Publisher: American Association for the Advancement of Science.
Chen, G. Ning, B., and Shi, T. Single-Cell RNA-Seq Technologies and Related Computational Data Analysis. Frontiers in Genetics, 10, April 2019. ISSN 1664-8021. doi: 10.33899fgene.20199 9.00177.
Fleming, S. J., Chaffin, M. D., Arduini, A., Akkad, A.-D., Banks, E., Marioni, J. C.,, Philippakis, A. A.. Ellinor, P. T., and Babadi, M. Unsupervised removal of sys- tematic background noise from droplet-based based single-cell experiments using CellBender. Nature Methods, 20(9): 1323-1335, September 2023. ISSN 1548-7105. doi: 00000///1552-022--1199009 lishing Group.
Publisher: Frontiers.
Cui, H., Wang, C.. Maan, H., Pang, K., Luo, F., Duan, N., and Wang, B. scGPT: toward building foundation model for single-cell multi-omics using generative AI. Nature Methods, pp. 1-11, February 2024. ISSN 1548-7105. doi: 10. 1000//4152-022--220---000 Publisher: Nature Publishing Group.
Publisher: Nature Pub-
Fournier, Q., Vernon, R. M., Sloot, A. v. d., Schulz, B., Chandar, S., and Langmead, C. J. Protein Language Models: Is Scaling Necessary?, September 2024.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and Memory-Efficieen Exact Attention with 10-Awareness, June 2022. arXiv:2205.144 14135 [cs].
Gavrilidis, G. I., Vasileiou, V., Orfanou, A., Ishaque, N., and Psomopoulos, F. A mini-review on perturbation modelling across single-cell omic modalities. Com- putational and Structural Biotechnology Journal, 23: 1886-1896, December 2024. ISSN 2001-0370. doi: 10. 11100664.cshj.222444.00000
Devlin. J.. Chang. M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding, May 2019, arXiv:1810.0440
[cs].
Dixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby- Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., Adamson. B., Norman, T. M., Lan- der, E. S., Weissman, J. S., Friedman, N., and Regev, A. Perturb-Seq: Dissecting Molecular Circuits with Scalable Single-Cell RNA Profiling of Pooled Genetic Screens. Cell, 167(7): 1005533--66661111 December 2016. ISSN 0092-8674. doi: 10. 00.101/6..ell.2016..1
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Song, L., and Zhang, X. Large Scale Foundation Model on Single-cell Transcriptomics, June 2023.
Heumos, L., Ji, Y., May, L., Green, T., Zhang, X., Wu, X., Ostner, J., Peidli, S., Schumacher, A., Hrovatin, K., Müller, M., Chong, F,, Sturm, G., Tejada, A., Dann, E., Dong, M., Bahrami, M., Gold, I., Rybakov, S., Nam- saraeva, A., Moinfar, A., Zheng. Z., Roellin, E., Mekki, I., Sander, C., Lotfollahi, M., Schiller, H. B., and Theis, F. J. Pertpy: an end-to-end framework for perturbation analysis, August 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J.. and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. arXiv:2010.11922 [cs].
Du, J. Jia, Dai, Y., Tao, C., Zhao, Z., and Zhi, D. Gene2vee: distributed representation of genes based co-expression. BMC Genomics, 20(1):82, February 2019. ISSN 1471-2164. doi: 10. 10.11866112864-018-522
Ji, Y., Green, T. D., Peidli, S., Bahrami, M., Liu, M., Zappia, L.. Hrovatin, K., Sander, C., and Theis, F. J. Optimal distance metrics for single-cell RNA-seq populations, December 2023.
Ektefaie, Y., Shen, A., Bykova, D., Marin, M., Zitnik, M., and Farhat, M. Evaluating generalizability of artificial in- telligence models for molecular datasets, February 2024.
Jin, Q., Dhingra, B., Cohen, W. W., and Lu, X. Probing Biomedical Embeddings from Language Models, April 2019.
El-Nouby, A., Izacard, G., Touvron, H., Laptev, I., Je- gou, H., and Grave, E. Are Large-scale Datasets Neces- sary for Sueervieed Pre-training?, December 2021. Axiv:2112.110:. [es].
Kedzierska, K. Z., Crawford, L., Amini, A. P., and Lu, A. X. Assessing the limits of zero-shot foundation models in single-cell biology, November 2023.
10

---

PertEval-scFM
King, T., Butcher, S., and Zalewski, L. Apocrita High Per- formance Computing Cluster for Queen Mary University of London. March 2017. doi: (.5288//eenod043330
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving Language Understanding by Generative Pre- Pre-Training. arXiv, 2018.
Kingma, D. and Ba, Adam: A Method for Stochastic Optimization, January 2017. arXiv:1412.00. [es].
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transfer- able Visual Models From Natural Language Supervision, [cs].
Lotfollahi, M., Wolf, F. A., and Theis, F. J. scGen predicts single-cell perturbation responses. Nature Methods, 16 (8):715-721, August 2019. ISSN 1548-7105. doi: 10. 1038/s41592-019-04400 Publisher: Nature Publishing Group.
February 2021.
Regev, A., Teichmann, S. A., Lander, E. S., Amit, I., Benoist, C., Birney, E., Bodenmiller, B., Campbell, P., Carninci, P., Clatworthy, M., and others. The human cell atlas. elife, 6:e27041, 2017. Publisher: eLife Sciences Publications, Ltd.
Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Beery, S., Leskovee, J., Kundaje, A., and others. Wilds: A benchmark of in-the- wild distribution shifts. arXiv preprint arXiv:2012.07421, 2020.
Replogle, J. M., Saunders, R. A., Pogson, A. N., Hussmann, J. A., Lenail, A., Guna, A., Mascibroda, L., Wagner, J., Adelman, K., Lithwick- Yanai, G., Iremadze, N., Oberstrass, F., Lipson, D., Bonnar, J. L., Jost, M., Nor- man, T. M., and Weissman, J. S. Mapping information- rich genotype-phenotype landscapes with genome-scale Perturb-seq. Cell, 185(14):2559-2577) 2577..228, July 2022. ISSN 0092-8674. doi: 10. 10.10016).cell.2222.5...
Nadig, A., Replogle, J. M., Pogson, A. N., McCarroll, S. A., Weissman, J. S., Robinson, E. B., and O'Connor, I.. Transcriptome-wide characterization of genetic perturba- tions, July 2024.
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge, A. Y.,
Xu, A., Jost, M., Gilbert, L. A., and Weissman, S. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455) August 2019. doi: 10.1126/science..e aax4438. Publisher: American Association for the Advancement of Science.
Roohani, Y., Huang. K., and Leskovee, Predicting transcriptional outcomes of novel multigene perturba- tions with GEARS. Nature Biotechnology. pp. 1- 9, August 2023. ISSN 1546-1699 1696. doi: 10.1038/ s4 1-15877-23--0995-66 Publisher: Nature Publishing Group.
OpenAL. GPT-4 Technical Report, March 2024. arXiv:2303.088: 0.8774 [es].
Rosen, Y., Roohani, Y., Agarwal, A., Samotorčan, L., Con- sortium, T. S., Quake, S. R., and Leskovec, J. Universal Cell Embeddings: A Foundation Model for Cell Biol- ogy, November 2023. Pages: 1.22.11.28.6891111 Section: New Results.
Peidli, S., Green, T. D., Shen, C., Gross, T., Min, J., Garda, S., Yuan, B., Schumacher, L. J., Taylor-King. J. P., Marks, D. S., Luna, A., Blüthgen, N., and Sander, C. scPer- turb: harmonized single-cell perturbation data. Nature Methods, 21(3):531-540, March 2024. ISSN 1548-7105. doi: 10. 10038444192-022-02144yy Publisher: Nature Publishing Group.
Spielman, D. A. and Teng. S.-H. Spectral Sparsification of Graphs, July 2010. arXiv:0808.4 [cs].
Svensson, V., Vento-Torma R., and Teichmann, S. A. Expo- nential scaling of single-cell RNA-seq in the past decade. Nature Protocols, 13(4):599-60) 604, April 2018. ISSN 1750- 2799. doi: 10. 00.1038/pprot.211..444 Publisher: Nature Publishing Group.
Program, C. S.-C. B., Abdulla, S., Aevermann, B., Assis, P., Badajoz, S., Bell, S. M., Bezzi, E., Cakir, B., Chaffer, J., Chambers, S., Cherry, J. M., Chi, T., Chien, J., Dorman, L., Garcia-Nieto, P., Gloria, N., Hastie, M., Hegeman, D., Hilton, J., Huang, T., Infeld, A., Istrate, A., Jelic, I, Katsuya, K., Kim, Y. J., Liang, K., Lin, M., Lom- bardo, M., Marshall, B., Martin, B., McDade, F,, Megill, C. Patel, N. Predeus, A., Raymor, B., Robatmili, B., Rogers, D., Rutherford, E., Sadgat, D., Shin, A., Small, C., Smith, T., Sridharan, P., Tarashansky, A., Tavares, N., Thomas, H., Tolopko, A., Urisko, M., Yan, J., Yeretssian, G., Zamanian, J., Mani, A., Cool, J., and Carr, A. CZ CELLXGENE Discover: A single-cell data platform for scalable exploration, analysis and modeling of aggregated data, November 2023.
Tenney, L,, Xia, P., Chen, B., Wang, A., Poliak, A., Mc- Coy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., and Pavlick, E. What do you learn from context? Probing for sentence structure in contextualized word representations, May 2019. weXiv:1905.000
[cs].
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., AI Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng. Z., Liu, X. S., and Ellinor, P. T. Transfer learn- ing enables predictions in network biology. Nature, 618

---

PertEval--sCFM
(7965):616-624, June 2023. ISSN 0028-0836, 1476-4687. doi: 10. 00.10088/41556--23--6119999949
Trapnell, C. Defining cell types and states with single- cell genomics. Genome Research, 25(10):1491-1498,
October 2015. ISSN 1088-9051, 1549-5469. doi:
10.1101/gr. 190595.1 Company: Cold Spring Har-
bor Laboratory Press Distributor: Cold Spring Harbor Laboratory Press Institution: Cold Spring Harbor Labora-
tory Press Label: Cold Spring Harbor Laboratory Press Publisher: Cold Spring Harbor Lab.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Velez-Arce, A., Huang, K., Li, M. M., Lin, X., Gao, W., Fu, T., Kellis, M., Pentelute, B. L., and Zitnik, M. TDC-2: Multimodal Foundation for Therapeutic Science, June 2024.
Wen, H., Tang, W., Dai, X., Ding, J., Jin, W. Xie, Y., and Tang, J. CellPLM: Pre-training of Cell Language Model Beyond Single Cells, October 2023.
Wenteler, A., Cabrera, C. P., Wei, W., Neduva, V., and Barnes, M. R. AI approaches for the discovery and validation of drug targets. Cambridge Prisms: Preci- sion Medicine, 2, January 2024. ISSN 2752-6143. doi:
10.1017/pcm.2024..
Wolf, F. A., Angerer, P., and Theis, F. J. SCANPY: large- scale single-cell gene expression data analysis. Genome Biology, 19(1):15, February 2018. ISSN 1474-760X. doi: 10.1186/s1 121186//133099-07-11882222
Wu, Y., Wershof, E., Schmon, S. M., Nassar, M., Osiński, B., Eksi, R., Zhang, K., and Graepel, T. PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis, August 2024. arXiv:2408. 10669 [cs, q-bio, stat].
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., and Yao, J. scBERT as a large-scale pretrained deep language model for cell type annotation of single- cell RNA-seq data. Nature Machine Intelligence, 4(10): 852 866, October 2022. ISSN 2522-5839, doi: 10.
1038/644255-022-0554--2 Publisher: Nature Publishing Group.
12

---

PertEval--sFM
Appendix
A. Single-Cell Transcriptomies Data
The advent of single-cell RNA sequencing technology (scRNA-seq) has revolutionized our understanding of cellular heterogeneity and dynamic biological processes (Chen et al., 2019). Unlike traditional bulk sequencing methods, which average signals across large populations of cells, scRNA-seq technologies enable the study of gene expression single-cell resolution. This granularity provides unprecedented insights into complex mechanisms of development, differentiation, and disease progression (Trapnell, 2015; Svensson et al., 2018; Fleck et al., 2023). The broad-scale application potential of scRNA-seg technology has led to the generation of large-scale datasets, such as the Human Cell Atlas (Regev et al., 2017) and the CellxGene Census (Program et al., 2023), which collectively span millions of cells and most sources of primary tissue.
A.1. Perturb-Seq Data
Perturb-seq integrates scRNA-seq with CRISPR-based perturbations to profile gene expression changes in response to specific genetic modifications at the single-cell resolution (Dixit et al., 2016). By systematically perturbing genes and measuring the resulting transcriptomic changes, Perturb-seq data provides a detailed map of cellular responses specific genetic modifications. These datasets, such as those generated by Norman al. (2019) and Replogle et al. (2022), allow researchers to explore the relationships between gene perturbations and cellular phenotypes in digh-siiensional space, providing invaluable insights into gene regulatory networks and cellular behavior and allowing the identification of potential drug targets (Wenteler et al., 2024).
A. THE NORMAN DATASET
The dataset from Norman et al. (2019) represents one the most comprehensive Perturb-seq resources available. It profiles transcriptional responses to 200 single-gens gene and double-gene perturbations in the human K562 leukemia cell line, using pooled CRISPRa screening and scRNA-seq. This dataset captures gene expression data from thousands of individual cells, each subjected to either a control or perturbation, providing ideal testing ground for models designed to predict perturbation effects. The Norman dataset includes both perturbed and unperturbed cells, allowing for systematic evaluation of model performance in predicting the effects of genetic perturbations at single-cell resolution.
Table Al. Overview of the Norman dataset
Characteristic
Description
Cell type
K562 (human leukemia cells)
Total number perturbations
196
Number single-gene perturbations
105
Number of double-gene perturbations
91
Perturbation method
CRISPRa
Number of control cells
122000
Number of cells
~ 100,00
Sequeneing platform
10x Genomics Chromium
Gene expression data
Single-cell RNA-seq
Number of genes measured
20,000+
Reference
Norman et al. (2019)
A. 1.2. THE REPLOGLE DATASET
The datasets from Replogle et al. (2022) provide another comprehensive Perturb-seq resource, offering large-scale map of transcriptional responses to pooled
perturbations. This datasets profile over 4000 genetic perturbations across multiple cell lines, including K562 (human leukemia) and RPE1 (human retinal pigment epithelial) cells, using scRNA-seq. Its expansive coverage of individual perturbations enables the evaluation of models designed to predict
13

---

PertEval-scCM
transcriptional responses in diverse cellular contexts. The Replogle dataset captures single-cell gene expression data under wide range of experimental conditions, providing an excellent benchmark for assessing the robustness and generalizability of predictive models. Furthermore, its inclusion of both perturbed and control cells facilitates systematic performance comparisons.
Table A2. Overview of the Replogle dataset
Characteristic
Deseription
Cell types
K562 (human leukemia cells),
RPE1 (human retinal pigment epithelial cells)
Total number perturbations
4,452
Number single-gene perturbations
2,058 (K562), 2,394 2,394 (RPE1)
Perturbation method
CRISPRi
Number of cells
~310,,000 (K562), 225,0000 (RPEI)
Sequencing platform
NovaSeg 6000, Ultima Genomics
Gene expression data
Single-cell RNA-seq
Number of genes measured
20,000+
Reference
Replogle et al. (2022)
A.2. Single-Cell Data Pre-Processing and Quality Control
The datasets were downloaded and pre-processed using ScPerturb (Peidli et al., 2024), PertPy (Heumos et al., 2024), and ScanPy (Wolf al., 2018). As scFMs utilize raw gene expression counts, two versions of the datasets are stored intermally: an AnnData object containing raw expression counts, used to generate embeddings with scFMs, and an AnnData object with pre-processed gene expression values, used to train the baseline models.
Pre-processing involves normalizing the raw gene expression counts by the total number of counts for each gene to account for differences in sequencing depth and ensure comparability across samples. This was performed us- ing the scanpy. PP. www.anooeetttttttooo.......eeeee a (adata) method with default settings. Next, the normalized counts were log- tredssrnnfor with scanpy pp. loglp (adata) to stabilize variance and make the data more amenable to downstream analysis. Finally, the top 2,000 highly variable genes were selected for training, using the ipert_adata, a_top_ggnes-2000 function.
scanpy pp. www.hhpyyp.ppp..o.pe...p.......m beiighyyvvriable_gee.
A.3. Quality Control Plots
To ensure the robustness and reliability of analyses derived from the Norman and Replogle datasets used in this study, quality control (QC) steps were applied to evaluate the integrity and consistency of the data. These QC plots provide visual assessment of key metrics such as the number cells per gene and the number of genes detected per cell, both critical indicators of dataset quality.
The number of cells per gene (Figures Ala, A2a, A3a) reveals how consistently individual genes are captured across cells, identifying genes widely expressed (potentially housekeeping or essential genes) and filtering out low-quality genes only detected in few cells. The number of genes detected per cell (Figures A1b, A2b, A3b) highlights the overall quality of cell-level data, with higher numbers indicating more comprehensive transcriptome coverage.
These distribution of perturbation effect plots (Figure A4) illustrate the range and variability of transcriptional changes induced by the perturbations, offering insights into the sensitivity of gene expression to these manipulations in each dataset.
14

---

PertEval--cFFM
10a
7000
10a
6000
Number of calls pet gane
10a
10a
Number genes
3000
10a
2000
10a
1000
(a)
(h)
Figure AI. Quality control plots for the Norman dataset. (a) The number of cells per gene. This indicates how often an individual gene measured across cells. Genes that present in many cells might be housekeeping genes essential genes. Because many genes present in only few cells, only genes present in minimum cells were considered. (b) The number of genes detected per cell across all datasets. This offers insights into the distribution of genes among cells and indicates how representative the measurements of single-cell transcriptomes.
7000
6000
Number of colls per gene
5000
1.0
Number of gencs
3000
2000
10a
Replogle K562
Raplogle KS62
(b)
Figure A2. Quality control plots for the Replogle K562 dataset. (a) The number of cells per gene. (b) The number of genes detected per cell across all datasets.
15

---

PertEval--cFM
Number of colls per gone
10"
Number of genes detected per
3000
2000
1000
Replogle RPE1
Replogle RPE1
(a)
(h)
Figure A3. Quality control plots for the Replogle RPEI dataset. (a) The number of cells per gene. (b) The number of genes detected per cell across all datasets.
Condition
6
Control
expression
4-
2
1-
norman
replogle_KS5 k562
replogie_rper
Figure A4. Overall distribution of mean expression in control and perturbed cells. The distributions are relatively similar for both conditions, indicating that most perturbations have limited effect the expression of most genes.
16

---

PertEval-scFM
B. Models
B.1. Single-Cell Foundation Models (scFMs)
Single-cell foundation models (scFMs) are trained on broad single-cell data using large-scale self-supervision, allowing them to be adapted (i.e., fine-tuned) for wide range of downstream tasks. Most scFMs use variants of the Transformer (Vaswani et al., 2017) architecture to process embedded representations of input gene expression data. However, they differ in input data representation, model architecture, and training procedures. Here, provide brief overview of the scFMs included in PertEval-scFM.
Table BI. Overview of the scFMs included PertEval-scFM.
MODEL NAME
ARCHITECTURE
PRE-TRAINING OBJECTIVE
# OF CELLS
ORGANISM
EMB. DIM.
SCBERT
TRANSFORMER
MASKED LANGUAGE MODELING (MLM)
~ MILLION
HUMAN & MOUSE
200
GENEFORMER
TRANSFORMER
MASKED LANGUAGE MODELING (MLM)
~0 MILLION
HUMAN
256
SCGPT
TRANSFORMER
SPECIALIZED ATTENTION-MASKING MECHANISM
<-33 MILLION
HUMAN
512
UCE
TRANSFORMER
MASKED LANGUAGE MODELING (MLM)
333 MILLION
8 SPECIES
1,280
SCFOUNDATION
TRANSPORMER
READ-DEPTH AwARe (RDA) MODELING
55 MILLION
HUMAN
3,072
Geneformer. Geneformer (Theodoris et al., 2023) employs six transformer units, each consisting of self-attention layer and an MLP layer. The model is pre-trained on Genecorpus-30M, which comprises 29.9 million human single-cell transcriptomes from a broad range of tissues obtained from publicly available data. Before feeding the data into the model, gene expression values are converted into rank value encodings. This method provides a non-parametric representation of each single-cell transcriptome by ranking genes based on their expression levels each cell and normalizing these ranks within the entire dataset. Consequently, housekeeping genes, which are ubiquitously highly expressed, are normalized to lower ranks, reducing their influence. Rank value encodings for each single-cell transcriptome then tokenized, allowing genes to be stored ranked tokens instead of their exact transcript values. Only genes detected within each cell are stored, thus reducing the sparsity of the data. When input into the model, genes from each single-cell transcriptome are embedded into a 256-dimension space. Cell embeddings can also be generated by averaging the embeddings of each detected gene in the cell, resulting in a 256-dimensional embedding for each cell. The model is pre-trained using a masked learning objective, masking a portion of the genes and predicting the masked genes, which is intended to allow the model to learn gene network dynamics.
ScBERT. ScBERT (Yang et al., 2022) adapts the BERT architecture (Devlin et al., 2019) for single-cell data analysis. A transformer is used as the model's backbone. The input data is represented as sequence of gene expression values for each cell, where cells are constructed from gene expression value tokens. Gene embeddings are generated from the sum of two embeddings, where the first represents the gene's ' binned log-scale expression level, and the second is generated with gene2vec (Du et al., 2019) and specifies the gene's identity. The model is pre-traine pre-trained via imputation on 5 million cells using masked learning objective- masked gene expression values are predicted as function of the other gene embeddings in the cell. In the paper, scBERT is fine-tuned for cell type annotation.
scFoundation. scFoundation (Hao et al., 2023) employs x xTrimogene backbone model, scalable transformer- architecture that includes an embedding module and an asymmetric encoder-decoder. The embedding module converts continuous gene expression scalars into high-ding -igh-niiensional vectors, allowing the model to fully retain the information from raw expression values, rather than discretizing them like other methods. The encoder is designed to only process nonzero and nonmasked gene expression embeddings, reducing computational load and thus enabling the application of "vanilla transformer blocks to capture gene dependency without any kernel of low-rank approximation" These encoded embeddings are then recombined with the zero-expressed gene embeddings at the decoder stage to establish transcriptome- -ide embedded representations. This backbone approach can then be built upon additional architectures which specialized for specific tasks i.e., GEARS (Roohani et al., 2023) for perturbation response prediction. scFoundation is pre-trained using wead-depth-aware (RDA) modeling, an extension of masked language modeling developed to take the high variance in read depth of the data into account. The raw gene expression values are pre- prepprcesseed using hierarchical Bayesian downsampling in order to generate the input vectors, which can either be the unchanged gene expression profile or where downsampling has resulted in a variant of the data with lower total gene expression counts. After gene expression has been normalized, raw
17

---

PertEval-scFM
and input gene expression count indicators are represented as tokens which concatenated with the model input, allowing the model to learn relationships between cells with different read depths. Pre-training used data from over 50 million single cells sourced from a wide range of organs and tissues originating from both healthy and donors with a variety of diseases and cancer types.
ScGPT. scGPT (Cui et al., 2024) follows a similar architectural and pre-training paradigm to scBERT. However, scGPT bins genes according to their expression, ensuring an even distribution across each bin. It uses random gene identity embeddings and incorporates an additional "condition embedding" to store meta-information and differentiate each gene. Along with gene embeddings, scGPT trains cell token to summarize each cell. Instead of the long-range Performer architecture, scGPT processes embeddings via Wlash-Attention (Dao et al., 2022) blocks. The model implements a generative masked pre-training using causal masking strategy inspired by OpenAl's GPT series (Radford et al., 2018). scGPT is pre-trained on 33 million human cells and fine-tuned wide suite of downstream tasks, including cell type annotation, genetic perturbation response prediction, batch correction, and multi-omic integration.
Universal Cell Embeddings (UCE). Universal Cell Embeddings (UCE) (Rosen et al., 2023) is trained on a large compendium of single-cell RNA-seq datasets from multiple species, including human, mouse, mouse lemur, zebrafish, pig, rhesus macaque, crab-eating macaque, and western clawed frog, to create a universal embedding space for cells. The model converts the transcriptome of a single cell into an sspression-weightee represents these genes by their protein products using a large protein language model. This representation then fed into a transformer model. UCE pre-trained in a self-supervised manner with contrastive learning objective, where similar cells are mapped to nearby points in the embedding space, and dissimilar cells mapped to distant points. This training paradigm enables UCE to provide high-quality embeddings that facilitate various downstream analyses. Benchmarks carried out by Rosen et al. (2023) in zero-shot framework shown that UCE outperforms Geneformer (Theodoris et al., 2023) and scGPT (Cui et al., 2024), as well as cell annotation models such as VI and scArches, in cell representation tasks.
sample of its corresponding genes and then
B.2. scFM Embedding Generation
In this section, we detail the process of generating embeddings for each foundation model in a zero-shot context using pre-trained models with frozen weights. For some models, pre-trained checkpoints are available and can be directly utilized, while others require initial pre- pre-training. By freezing model weights, we ensure that the embeddings represent the learned features from the initial training phase, without further adaptation to the specific perturbation prediction task, This approach allows us to evaluate the inherent quality and utility of the pre-trained representations for downstream applications in biological research.
Geneformer. To generate embeddings for Geneformer (Theodoris al., 2023), we downloaded the repository, including pre-trained model checkpoints, from Hugging Face. For control cells, we pre-processed the raw expression files to ensure the correct naming of columns and then fed them into the Geneformer tokenizer (Trans www.scriptooo www.ptomeeoeeiiieeeee... zer). Once the dataset had been tokenized, we extracted embeddings using the pre-trained checkpoint (6-layer model) with the EmbExt ractor method. For the perturbation data, we loaded the data and iterated through it in order to remove perturbed genes, simulating their deletion. The perturbed cells were then tokenized, and embeddings were extracted for each perturbed cell using the same functions.
scBERT. To generate emeddings for ScBERT (Yang et al., 2022), we first downloaded the checkpoint and data shared in the scBERT GitHub repository. The environment was set up using the ScBERT-reusablle GitHub repository. For the raw expression counts, the genes were aligned using Ensembl Homo sapiens gene information. Log-normalization was performed and cells with less than 200 expressed genes were filtered out. For the perturbation data, the gene expression value was set to 0 to simulate perturbation, and embeddings were generated using the predict ct py script.
scFoundation. To generate scFoundation embeddings (Hao et al., 2023), we initialized the scFoundation class shared at the official scFoundation GitHub repository. The O1B-resolution pre-trained model checkpoint was loaded and the embeddings were generated while setting the input_type = singlecell and tithighres = f1 to indicate no read depth differences between unperturbed and perturbed cells. The embeddings were then generated using the get_embeddd: ngs function.
18

---

PertEval-scFM
scGPT. To generate embeddings for scGPT (Cui al., 2024) installed the seGPT python package. We downloaded and used the whole-human scGPT model for embedding. For control cells, used the scGPT embed_da datddaa a function to generate the embeddings from the raw expression values. This function tokenises the data before feeding it through the model. For the perturbation data, we removed the perturbed genes, to simulate their deletion. The embeddings for the perturbed cells were then generated using the seGPT embed_dat.a function.
Universal Cell Embeddings (UCE). To generate cell embeddings for UCE (Rosen et al., 2023), we ran the eval_single_annde batngglleanndttaaaaa py script provided in the UCE GitHub repository. Model weights for the 33-layer model and the pre-computed protein embeddings were downloaded separately from figshare. The script takes as input an h5ad raw expression file with variable names set as gene_symbole The script was with default parameters, except for the filter argument which was to Fa se, order skip additional gene and cell filtering step. No further pre-processing was required to generate embeddings for control cells. For in vitro perturbed cells, the raw count value of the perturbed gene was explicitly set to zero for each condition prior to model inference, and saved as a h5ad file. The output of the script was an identical h5ad file with the input, except for cell-level embeddings that are stored in the Anndata obsm 'x_uce' ] slot.
19

---

PertEEal--cFM
C. Featurization
C.1. Single-Cell Expression Data Featurization
To generate the input features for raw single-cell expression data, we begin with the control matrix C' E R" consisting of n. unperturbed single-cell transeriptomes across highly variable genes (see Appendix A.2). From this matrix, we form a & pseudo-bulk sample C, which aggregates expression values from groups of cells within the same sample, in order to reduce sparsity and noise. Formally, let C = {cr ) denote the set of randomly sampled cells from C. The average expression value Cj for each cell j is then calculated by averaging the expression across the pseudo-bulked cells:
E Cia c== = |C| GEE
C
Vj (1p
(CI)
Using this basal expression, we construct the input matrix X R",xr, which has the same dimensions of the perturbed transcriptomic matrix P I# (i.e. what we want to predict), where ny is the number of perturbed cells. The input matrix Xe is generated by sub-sampling from Cj. ensuring that the dimensions are consistent between the input and the target output.
This approach ensures that input-target pairs are consistently defined for all training examples, as the dimensions of X. E R2,,** x align with the target matrix P. Representing input expression at pseudo-bulked basal levels helps mitigate sparsity issues caused by limited gene coverage in individual single-cell measurements from the original dataset. However, this method introduces a trade-off by reducing the heterogeneity of the input gene expression. As result, some salient single-cell signals, such as those related its initial state, may be diminished. However, inferring cellular states based solely gene expression data is inherently challenging, given the many confounding factors and technical noise present in single-cell datasets (Fleming et al., 2023). Therefore, conventional machine learning models should not be expected to perform this task with high fidelity to begin with.
1. MLP BASELINE
To generate the full set of input features for the MIP,, we must encode the identity of each perturbation alongside capturing basal gene expression. Let P PP denote the set of perturbable genes, and let D {di,.... d represent all highly variable genes.
To evaluate the models' ability to generalize to unseen perturbations, it is important to incorporate information about gene interactions within specific cell type. This allows the models to learn gene interaction networks, helping to extrapolate effects from known perturbations to novel ones.
To achieve this, we construct a u-dimensional correlation vector for each perturbable gene by calculating the Pearson correlation between its basal expression and that of all other genes, including itself. By including the auto-correlation of the perturbable gene, we explicitly encode the identity of the gene to be perturbed. The resulting feature vector for each perturbable gene, ge RR, captures the correlations between its basal expression and the basal expression of all highly variable genes. Aggregating these correlation vectors for all perturbable genes produces the matrix Ge E R where the perturbation in each row corresponds to the transcriptomic state observed in T.
Finally, the control gene expression matrix Xg is concatenated with the perturbation correlation matrix Ge to construct the complete input feature matrix:
ZGE X + Ge
(22)
Here, ZGE E Rnx22 represents the input feature matrix, where each row combines the log-normalized basal expression values of cell with the corresponding perturbation correlation features. This procedure is applied to both the training and testing sets, to generate ZGE.. and Zer...
C.2. Single-Cell Foundation Model Embedding Featurization
To generate embeddings from a pre-trained single-cell foundation model (scFM) with frozen weights, we begin by mapping raw gene expression counts to transcriptomic embeddings. Let ferm : Rd. -> R represent the function that transforms raw
20

---

PertEval--cFM
expression data into embedding for each cell.
To construct the control cell embedding, we feed the raw expression vector x for each of the ne control cells into the scFM:
(x)) Ze
(C3)
The embedding vectors then subsampled to create Ze Rupxer, where ny matches the number of perturbed cells and the dimension of Ze aligns with the target output matrix.
An in silico perturbation embedding is then generated by nullifying the expression of the perturbed genes across all control cells in which it is expressed, up to maximum of 500 cells. The nullification process, denoted by N ( pt). adjusts the gene expression vector according to the requirements of the scFM model in use. The nullification function can defined as : _" x N, + R', where IR" represents the space of the gene expression vector, and N, denotes the set of natural numbers from to v, corresponding to the indices of genes in x(. the scFM requires setting the perturbed gene's expression zero, = U. However, some scFMs filter out non-expressed genes during tokenization (scGPT), train on ranked gene token representations instead of expression values (Geneformer). In these cases, the perturbed gene must be removed from the control gene expression vector, resulting in | = 1. Nonetheless, the perturbation embedding x is constructed as follows:
Sa N (X(. 2!
(C4)
The perturbation embeddings for all cells form the matrix Zp E Kc., It is trivial to extend the above framework to combinatorial perturbations, where the nullification function accepts multiple perturbations and nullifies the associated gene expression values.
The final cell embedding is then obtained by concatenating the control embedding Ze with the perturbation embedding Zp:
ZWFM = Z + Zp
(C5)
This approach differs from raw expression featurization, where co-expression patterns are explicitly encoded to model perturbations. In the scFM embedding fcaturization, in silico perturbation simulates the changes caused by gene perturbation. We hypothesize that the embeddings generated by scFMs inherently encode co-expression relationships, aligning with their pre-training objective based on masked language modeling.
In this study, zero-shot embeddings are generated using five different scFMs (Table BI). Inference for each scFM is tailored to the specific idiosyncrasies of the model in question. Detailed information on all the scFMs used can be found in Appendix 1.
21

---

PertEval-scCM
D. Perturbation Representation Experiment
D.I. Doubling Gene Expression Perturbation Representation Results
Here we assess whether simulating perturbations via an silico knockout (scBERT versus an upregulation strategy (scBERT+) leads to appreciable differences in embedding quality on the Norman CRISPRa dataset. By applying each perturbation scheme for embedding generation using seBERT, and computing the mean squared error (MSE) of the predicted post-perturbation expression across range of sparsification probabilittess (S 0.1-0.7), directly compare knockout against a gain-of-function representation obtained by doubling the perturbed gene' expression. As shown in Table DI and Figure DI, both approaches yield nearly identical performance, indicating that our brockout-baseed perturbation protocol provides a fair and robust benchmark for scFM comparison while avoiding the arbitrary magnitude choices and re-ranking challenges inherent to upregulation simulations.
Table MSE + standard deviation for Norman single-gens gene embeddings generated with scBERT using an silico knockout (scBERT vs. upregulation strategy (scBERT+) +) across sparsification probabilities.
MODEL
0.1
S 0.2
S 0.3
S 0.4
0.5
$ 0.6
S 0.7
SCBERT- 0.0630 + 0.0031
0.0634 + 0.0062
0.0676 +: 0.0051
0.0636 0.0031
0.0592
0.0015
0.0645 + 0.0107
0. 0.0849 + O. 0.0060
SCBERT+ 0.0640 0.0038
0.0620 0.0038
0.0658 0.0077
0.0610 0.0009
0.0659
0.0046
0.0744 0.0122
0.0853 0.0064
MSE for Norman single gene embeddings generated with scBERT.
using an knockout (scBERT-) vs. an upregulation (scBERT+) strategy
0.10
Mean MSE
0.06
0.05
0.2
0.7
Sparsification Probability
Figure DI. MSE for Norman single-gene embeddings generated with scBERT using an in silico knockout (scBERT-, purple) vs. an upregulation strategy (scBERT+, light blue) sparsification probabilities. Error bars represent standard deviations.
22

---

PertEval--sFM
E. MLP
E.1. MLP Parameter Count
The architectural capacity of neural networks, as reflected by the number of trainable parameters, can significantly impact their performance. This is particularly relevant when probing embeddings with variable dimensions, such as those derived from single-cell foundation models (scFMs). To explore the relationship between parameter count and model performance, trained MLPs of varying sizes on gene expression data. This allowed expressiveness of the MLP affects its ability to capture biologically relevant patterns. The results, summarized in Table E2, demonstrate that parameter count has negligible influence on the mean squared error (MSE) for both training and validation datasets. Similar experiments conducted using seFoundation and scBERT embeddings reinforce these findings. Notably, despite increasing the parameter count, the scFM embeddings do not provide additional biologically relevant information for enhanced performance. These results suggest that embedding sizes do not inherently constrain the MLP probes and that the observed performance differences across models are unlikely to be influenced by disparities architectural capacity.
evaluate whether increasing the
Table E2. Train and test results with MLPs of increasing parameter count
Trainable parameters (million)
Training data
train/MSE
val/MSE
1.6
Raw gene expression
0.057067
0.057642
3.2
Raw gene expression
0.058670
0.057493
6.3
Raw gene expression
0.0567448
0.057424
12.7
Raw gene expression
0.056724
0.057428
1.6
scFoundation embeddings
0.060780
0.060260
3.2
scFoundation embeddings
0.0604400
0.059910
12.6
scFoundation embeddings
0.059570
0.059050
0.2
scBERT embeddings
0.061040
0.061426
1.0
scBERT embeddings
0.061046
0.061428
8.0
ScBERT embeddings
0.061040
0.061421
E.2. Hyperparameter Optimization
To train the MLP probes, we used root mean square error (RMSE) as the objective function and the Adam optimizer (Kingma & Ba, 2017). Model performance evaluated independent test set comprising unseen perturbations. The objective function to be minimized is:
[()) = Σ (T-
(T))
X.); - 50())))
(EI)
where j indexes each cell and Tıg denotes the batch size.
Hyperparameters were selected using the tree-structured Parzen estimator (TPE) tuning algorithm (Bergstra et al., 2011).
This optimization was performed on the first train-test split, which contains the largest training set. Given the computational demands of exhaustive parameter sweeps, we focused optimizing the hyperparameters using the gene expression data as reference.
An initial search across different numbers hidden layers revealed that this parameter had no substantial effect on model performance. Therefore, a single hidden layer was used throughout the experiments to maintain model simplicity. The learning rate, however, was found to significantly influence performance and was thus adjusted for the models trained using the scFM embeddings. Following the manifold hypothesis, set the hidden dimension half of the input dimension (Bengio et al., 2013).
23

---

PortEval-scFM
F. SPECTRA
The SPECTRA framework addresses a critical gap in the evaluation of machine learning models for biological tasks, particularly in the context of single-cell datasets like Perturb-seq, where distribution shifts are common. Unlike random splits, which may either overestimate or underestimate model performance due unquantified train-test similarity, SPECTRA systematically controls and quantifies train-test dissimilarity. This enables nuanced assessment of model robustness across spectrum of difficulty levels, including challenging scenarios that better simulate real-world variability.
By generating train-test splits based on defined sparsification probability, SPECTRA not only evaluates models under diverse conditions but also provides framework to contextualize performance results and compare robustness across datasets and tasks. Importantly, SPECTRA's methodology does not artificially inflate task difficulty; instead, highlights inherent redundancies and patterns within the dataset. This approach ensures that performance assessments reflect meaningful generalization.
Moreover, SPECTRA is not just an evaluation tool but a valuable guide for experimental design. For example, in Perturb-seq. SPECTRA can reveal if a model generalises well within pathways but struggles across pathways, guiding researchers on which genes to test next, optimizing experimental efforts. Random splits, by contrast, obscure such insights due to high train-test similarity.
SPECTRA's ability to expose weaknesses in generalizability, simulate biologically realistic distribution shifts, and guide experimental design makes it high-impact addition to evaluation pipelines. This is particularly crucial for tasks like perturbation effect prediction, where navigating complex, real-world variability is a fundamental challenge.
F.1. Evaluating Model Robustness under Distribution Shift in Single-Cell Data with SPECTRA
To construct the spectral graph for single-cell data, sample-to-sample similarity between distributions is calculated using the L2 norm, denoted by || ll. of the log Ip-fold change between the mean perturbation expression vector, P,, and the mean control gene expression vector,
S c) = || log(p, + 1) log(E + 1)ll
(FI)
If two samples sufficiently similar, an edge will be inserted in the spectral graph. A series of train-test splits are then generated by sparsifying the initial graph. Train and test instances are sampled from distinct subgraphs for each split, with decreasing mean pairwise similarity between the two sets, The sparsification of the initial graph is attenuated by a sparsification probability (s), which is the probability that an edge between two samples will be be dropped. Mathematically, SPECTRA employs a graph sparsification technique similar to what is described in Spielman & Teng (2010). A practical limitation of the current implementation of SPECTRA lies in its tendency to unevenly distribute perturbations of similar magnitudes across the training and test splits while minimizing cross-split overlap. This uneven distribution engenders class imbalances that become increasingly pronounced at higher sparsification probabilities. Consequently, this imposes trade-off between induced class imbalance and simulated distribution shift. Empirical observations on datasets indicate that the sparsification probability threshold at which the class imbalance remains manageable is approximately 0.7. Beyond this threshold, the deleterious effects of class imbalance as well as low sample numbers begin to outweigh the benefits of reduced cross-split overlap.
Indeed, for the Norman dataset, Figure Fla illustrates a rapid decrease in the number of training and testing samples as the sparsification probability increases. This is expected, as a higher sparsification probability leads to increasingly disconnected subgraphs to draw samples from. Furthermore, Figure Flb confirms that SPECTRA can simulate distribution shift by showing a corresponding decrease in similarity between the samples sparsification probability rises. Subsequently, we train and test models on each SPECTRA split and plot the MSE as function of the decreasing cross-split overlap. The area under this curve is defined as the AUSPC, which serves as a measure of model generalizability under distribution shift.
Similarly to the dathin- case outlined above, the cross-split overlap can be used to measure the similarity between- datasets, in this case between the scFM pre-train and our fine-tune datasets for scBERT and scGPT. This approach allows us to investigate the impact of pre-training data the quality of scFM embeddings. Further details provided in Section H.1.
24

---

PertEval--sCFM
Train
Test
200
150
Number of samples
100
50
0
0.1
0.2
0.4
0.5
0.6
0.8
0.9
Sparsification probability
(a)
4.5
-5.0
-5.5
Tran-oo sample ainilarlyy [-12 norm
-6.0
6.5
7.0
-7.5
-8.0
-8.5
0.0
0.1
0.4
0.7
1.0
Sparsificatio probabilty
b
Figure F1. (a) Number of samples in train and test as a function of the sparsification probability. (b) Cross-split overlap as function of the sparsification prohability.
25

---

PertEval--sFF
F.2. Implementation Details of the AUSPC
The AUSPC is defined by Equation 7. For numerical evaluation, the integral is approximated using the trapezoidal rule with sparsification probabilities s; E .o.,,.......0.777...
AUSPC f(0) =
((s) ds
(F2)
(0;s; + d(3+1))
==
where d denotes the step size of the sparsification probability (0.1 in this case) and & represents the metric of interest, (MSE).
The AAUSPC is subsequently derived by calculating this value for both the baseline and the model independently. and then subtracting the AUSPC of the model from that of the baseline. For simplicity, we use the notation Φ; = ((s)).
To quantify the uncertainty associated with the AUSPC, uncertainty propagation is utilised, wherein the AUSPC is assumed to be a non-linear function of the metric of interest, (()). For uncertainty propagation in this context, the following equation employed:
n-
Of Doi
2
Joi
(F3)
where represents the total error associated with the AUSPC and de, denotes the error associated with the MSE for split i.
The partial derivative 0f is calculated using the definition of f given in Equation F2: døc
0f
0; +
diti
06:
Oti
0Φ;
(F4)
0f
06:
Substituting this result into Equation F3 yields:
0²
(F5)
d
Σ
do:
2
The algorithmic implementation is given in Algorithm 1.
26

---

PertEval-scFM
Algorithm I Calculate AUSPC and its associated error
1: function TRAPEZOIDALAUSPCIC.. 8)
2: AUSPC <- np. trapz 8)
3: return AUSPC
4: end function
5: function CALCULLATEDEETAAUUSPCCC,,,, Ob. Pm. Tb. Om 8)
6: AUSPC, < TRAPEZODDALAUUSSPCCCC (06, 8)
7: AUSPC, f- TRAPEZOIDALAUSPC((
8: d + s[1] s(0)
Assuming uniform step size
9:
σb (
10:
mm Eg(ommm. ))
11: AAUSPC + AUSPC, AUSPCm
12:
return AAUSPC, Tb. mm
13:
end function
27

---

PertEval-scFM
G. E-Statistics
G.1. Using E-Distance and Differential Gene Expression Analysis to Evaluate Significant Perturbations
While examining transcriptome-wide wide, aggregated perturbation effects provides valuable insights, it lacks the granularity needed to assess model's ability to reconstruct perturbation effects at the gene level. To address this limitation, energy statisties (E-statistics) are employed evaluate and select significant perturbations in single-cell expression profiles. Subsequently, differential gene expression analysis is carried out identify the top 20 differentially expressed genes which then used evaluate individual perturbations.
Perturbation effects are quantified using the E-distance, which compares mean pairwise distances between perturbed and control cells. Let X' E (x1..... X.,) } and y E (yI,....Y.,, be two distributions of cells in different conditions with na and ni cells respectively, where X;, Yi Immh refer to the transcriptomes for cell i. Now the between-distribution distance xxy and the
distances xx and oyy can be defined as:
6xy
d( d(xi.y))
na
1hh
The
Th d(xi, x,)
0x
(G1)
1Ab σy d(y..y))
where :) is the squared Euclidean distance. The E-distance, E, is then defined as:
E(X,)) = 26xy - ox ay
(G2)
The E-test, a Monte Carlo permutation test, used assess the statistical significance of observed E-distances. This test generates null distribution by randomly permuting perturbation labels 10,000 times, comparing the observed E-distance against this distribution to yield an adjusted p-value, calculated using the Holm-Sidak method. This p-value can then be used to select which perturbations result in a perturbation effect that is significantly different from the control.
Before E-statistics are calculated, the data is pre-processed. The number of cells per perturbation is subsampled to 300, following the 200-500 range proposed by Peidli et al. (2024). Perturbations with fewer than 300 cells are excluded from downstream analysis. For the Norman dataset, this threshold excludes 20 perturbations, leaving 186 perturbations. One additional perturbation (BCL2LII) is excluded by the E-test as not significant.
For significant perturbations, the top 20 differentially expressed genes between perturbation and control are selected for evaluation. This approach is based the observation that genetic perturbations tend to significantly affect only fraction of the full transcriptome, while the remainder remains close to control expression (Nadig et al., 2024). This allows us to evaluate whether the predicted perturbation effect aligns with the experimental observations specifically for individual perturbations. The data is pre-processed for differential gene expression testing as described in Appendix A.2. Differential gene expression calculation performed using the Wilcoxon rank sum implemented in scanpy tl rank Laaaaanene___aaaaaaaaaaaaaaaeeea gene_groups.
28

---

PertEval--cFM
H. Contextual Alignment
H.1. Calculating Contextual Alignment between Pre-Train and Fine-Tune Datasets
To evaluate the influence of pre-training on the efficacy of scFM embeddings, we estimate the contextual alignment between the datasets used for pre-training and those used for tinettunig. We expect that enhanced model performance correlates with greater overlap between these datasets. Following the instructions outlined the scGPT GitHub, we obtained the complete pre-training cell corpus for scGPT from the CellXGene Census. As for scBERT, the pre-training dataset is derived from PanglaoDB and provided by the authors. The scBERT and scGPT datasets contain 1.4 million and 33 million cells, and 16,906 and 60,664 features respectively.
To carry out the contextual alignment experiment, we first ensure alignment between the paired datasets based on common genes. We normalize the fine-tuning dataset to a total read count of 10,000 over all genes and apply log I-trransformation. Additionally, we filter the data to include the same set of 2,061 highly variable genes that are used the fine-tuning process (see Appendix A.2). Following these steps, we obtain two too-trainngfiin-tuniinn common gene sets, 1,408 for scBERT + Norman and 2,044 for scGPT Norman.
To quantify the alignment, we compare gene expression profiles between the fine-tuning and pre-training datasets by computing cosine similarity scores, which are advantageous due to their insensitivity to expression magnitude. This comparison generates a dense score matrix of dimensions Ninetune x Npre-train For a subset of Npe-train used in at least one train-test split, an aggregate cross-split overlap calculated to evaluate the impact of different pre-training/fine-tuii dataset configurations on model performance.
Initially, matrix S E RR Nuxeus Nac is constructed, where each element St represents the cosine similarity between the i-th cell in the fine-tuning dataset and the j-th cell in the pre-training dataset. From this, we derive a binary similarity matrix B of the same dimensions with entries bej. The matrix is constructed as follows:
big = -10
if af 2 4 20, otherwise,
(H1)
where μ and a are the mean and standard deviation of the cosine similarities computed across 100,000 randomly sampled cell pairs. Based on this established threshold, B represents whether each fine-tuning cell significantly overlaps with each pre-training cell.
To quantify the alignment for each fine-tuning cell, we aggregate over the pre-training dimension matrix for each fine-tuning cell, resulting in a vector f where each component f. is given by:
ft
Big
(H2)
N Npre-train
Here, S E R.Nauuee represents the fraction of the pre-training dataset that is similar the i-th fine-tuning cell.
To conduct the sensitivity analysis, we define a threshold т, which represents the minimum fraction of the pre-training dataset that fine-tuning cell must be similar to in order to be considered significantly aligned. T is varied within the range of 0 to 0.1% of N Mpre-trin- For each value of r, we calculate the proportion of fine-tuning cells that meet or exceed this threshold, thus generating a series of values:
Ninaues
P(T)
Σ ((ST)
(H3)
Ninettue
where 1 is the indicator function that evaluates if the condition is true and otherwise.
The sensitivity curve is then plotted as p(T) versus т. The area under this curve reflects the overall cross-split overlap of the fine-tuning dataset relative to the pre-training dataset, as visualized in Figure HI.
29

---

PertEval-scCM
1.0
ScBERT
ScGPT
0.8
0.6
p(τ)
0.4
0.2
0.0
0.00
0.02
0.04
0.06
0.08
0.10
T
Figure HII. Plot of the probability that cell from the pre-train dataset is similar to cell from the fine-tune dataset function of т, the similarity threshold at which two cells are considered similar based their cosine similarity.
30

---

PertEval--sCFM
1. Supplementary Figures
I.I. Perturbation Effect Prediction Evaluated across 2000 HVGs
1.1.1. NORMAN SINGLE-GRES GENE
MSE Spattiia Proteiiitiis and Models
AUSEC Screech Modt
0,0627
0.0550
EXPPT
0.004
0.0614
609766
DOAM
0.2931
DICKS
00636
6,945
1040
aFoenbiice
0.0622
0.0119
DC644
0.011
Meas bandine-
0,0618
0,0598
6,0500
00422
0,2656
6.0741
048-1
Graefrmer
0.001
D.66000
0.0015
0.
0.2
0.3
0.4
0.6
ML bancii
Mox hardiae
Speritication Protabllyy
68
(4)
Figure 11. Predictions of single-gens perturbation effect for the Norman dataset evaluated across 2,000 highly variable genes for 8 train-test splits of increasing difficulty. (a) MSE for all prediction models. Experiments carried out in triplicate for each model. The heatmap shows the mean MSE values (1). (b) Average AUSPC (1) across sparsification probabilities for each model with standard error bars.
1.12. NORMAN DOUBLE-GENE
MS2 Arrrsppnniiii Priteiiiiie Models
AUSIC Screcct Modi
MLL Ivedlin-
0.1555
0.0941
COTTS
DCMM
0,078
CAADK
DIC500
GIGHT
0.0851
0
0A00
BC677
0050
00675
(1) (1131
aFoundtice-
0.085
0.0827
CU0D5
DC876
0.0744
030338
DCS04
-0040
Nastine.
0066
6,907
DC5000
UCE-
0.065L
0035
Graefemer
0.0851
0.0614
0.005
DON74
0.0744
0.724
DCS1X
0.1
0.2
0.3
0.4
0.5
0.6
677
Mea: baclar
0010
Speritication Probubikky
XE.
(SS((L)
Figure 12. Predictions of double-gen gene perturbation effect for the Norman dataset evaluated across 2,000 highly variable genes for 8 train-test splits increasing difficulty. (a) MSE for prediction models, Experiments were carried out triplicate for each model. The heatmap shows the mean MSE values (L). (b) Average AUSPC (L) across sparsification probabilities for each model with standard error bars.
31

---

PertEval-scFM
1.1.3. REPLOGLE K562
MSI Arne Spariisation Iréaailitee
AUSIC Meddl
0.77
scBERT
6.00144
020033
6.30994
0.20333
0.0061
6.00224
020122
WXPPT
020070
0222414
0.23700
0 023330
0.31004
6.29033
D 0776113
10.5
117 (113880
AFoundaiinn
623965
0.23764
Moa huectine
C21219
0,2211
6:21199
0.22500
0:21964
0.24195
UCE-
021142
0.22448
0.23869
0.22230
0:23008
6.27904
0 265555
1033
03
0.3
Spacification
44
0.7
3
MLP baados
45
Figure 13. Predictions single-gene perturbation effect for the Replogle K562 dataset evaluated across 2,000 highly variable genes for train-test splits of increasing difficulty. (a) MSE for all prediction models. Experiments were carried out in triplicate for each model. The heatmap shows mean MSE values (L). (b) Average AUSPC (J) across sparsification probabilities for each model with standard error bars.
1.1.4. REPLOGLE RPE1
MSI Acricc Spendficteeitiiii Pootamiiiieeeddd Metzis
AUSIC for each Model
0.140
ScBERT
0.20923
0.20552
0.29474
0.20223
0.21129
022088
0:21783
WOPT
000059
0.001%
0.36474
20430
0,21124
021718
6.20044
0.1325
200526
6.25811
Mias haecline
021867
022116
0.21001
6:221M
0.23029
0:21AX
0:22160
UCE
0,20929
0.2055
0.34475
0,21128
021706
6.21041
0.1250
-.1225
Gcnefomer-
0.20476
0.1
Gendomm
LCE
bunlizz
Spoctticatonn Prebohilty
MSDI 1)
Figure 14, Predictions single-gene perturbation effect for the Replogle RPE1 dataset evaluated across 2,000 highly variable genes for train-test splits of increasing difficulty. (a) MSE for all prediction models. Experiments carried out in triplicate for each model. The heatmap shows the mean MSE values (J). (b) Average AUSPC (J) across sparsification probabilities for each model with standard error bars
32

---

PertEval--sFFM
1.2. SPECTRA Performance Curves
1.2.1. NORMAN SINGLE-GENE
300
300
4001
0.1
Baseline
ScBERT
304
300
4.50
6.0
(c) seGPT
(d) Geneformer
300
407-
Spunticctia
Spetfictio
(c) UCE
(f) scFrundation
Figure 15. MSE as a function of the sparsification probability for the different models, tested on Norman single-gene effect perturbation prediction. These functions are used calculate to calculate the AUSPC, which is here shaded in blue.
33

---

PertEval--cFMM
1.2.2. NORMAN DOUBLE-GENE
300
3072
3000
Baseline MLP
ScBERT
1000
Spentlisia
(c) scGPT
(d) Geneformer
14h
Spentlodie
Speericatie
(c) UCE
( scFoundation
Figure 16. MSE function of the sparsification probability for the different models, tested on Norman double-gene perturbation effect prediction. These functions are used calculate calculate the AUSPC, which is here shaded blue.
34

---

PertEval--cFFM
1.2.3. REPLOGLE K562
Baseline MLP
Mean baseline
300
Spentlosie Prileil
(c) scGPT
(d) Geneformer
0.2
6.7
14B
023
414
Spentlodiee
Speeficatie
(e) UCE
( scFoundation
Figure 17. MSE function of the sparsification probability for the different models, tested on the Replogle K562 dataset. These functions are used calculate calculate the AUSPC, which is here shaded blue.
35

---

PertEval-scFM
1.2.4. REPLOGLE RPE1
4104
320
300
6301
3220
325-
Baseline MLP
Mean baseline
3230
1322
13.5
1180
10.0
4300
10.0
8300
1200
100
22
Spentfisii
(c) seGPT
(d) Geneformer
1250
10.7%
3210
100
Spentliciee
Spenriictie
(c) UCE
(i) scFoundation
Figure 18. MSE function of the sparsification probability for the different models, tested on the Replogle RPEI dataset. These functions are used calculate calculate the AUSPC, which is here shaded blue.
36

---

PertEval-scFM
1.3. MSE for all Models Compared to Mean Baseline across 2000 HVGs
1.3.1. NORMAN SINGLE-GENE
(a) Baseline MLP
(b) ScBERT
(c) scGPT
(d) Geneformer
(c) UCE
(1 scFoundation
Figure 19. MSE a function the sparsification probability for different models. This depiction of the curves that used calculate the AAUSPC.
37

---

PertEval--cFFM
1.3.2. NORMAN DOUBLE-GENE
4000
Baseline MLP
(b) scBERT
(c) scGPT
(d) Geneformer
(e) UCE
(f) scFunddation
Figure 110. MSE as function the sparsification probability for the different models. This depiction the curves that are used calculate the AAUSPC.
38

---

PertEval--sCFM
1.3.3. REPLOGLE K562
Spackkaki
Baseline MLP
(b) Geneformer
134
105
(c) scBERT
(d) scFoundation
(c) seGPT
(0 UCE
Figure II1. MSE as function of the sparsification probability for the different models. This is depiction of the that used calculate the AAUSPC.
39

---

PertEval-scFM
1.3.4. REPLOGLE RPE1
Spackaaki
Baseline MLP
(b) Geneformer
305
(6) scBERT
(d) scFoundation
(c) seGPT
(1 UCE
Figure 112. MSE as function the sparsification probability for the different models. This depiction the curves that used calculate the AAUSPC.
40

---

PertEval-scFM
1.4. Comparison between Mean Baseline Performance Predicting Perturbation Effect on the top 2000 HVGs and the top 20 DEGs
HVGx
114
MSE
Sparsification Probability
Figure I13. Comparison of the performance of the mean baseline the highly variable genes (HVGG) differentially expressed genes (DEGs) task across different sparsification probability train-test splits for Norman single-gene perturbation effect prediction.
BVG
08
MSE
Spassiicctionn Probubility
Figure 114. Comparison of performance mean baseline on the highly variable genes (HVGs) vs. differentially expressed genes (DEGs) task across different sparsification probability train-test splits for Norman double-gen gene perturbation effect prediction.

---

PertEval--sCFM
I.5. Perturbation Effect Prediction Results across the top 20 DEGs
1.5.1. NORMAN SINGLE-GENE
MSB SpmeSpppntiatiin ProeeMiiioocall and Midels
Screech Modi
0400
0.0612
B3969
69764
-0310
><BERT
0,083
0.5649
6/0088
0.5657
0.4776
6.712
0.7604
0.114
(1 (1/200V
Moas bawdinn-
0,475
CA188
0.9971
0.5925
0.112
-0275
UCE-
0.647
0.5947
6.6033
0.5607
0.539
6,676
0.353
0.300
0.6
MLP suctio
Gazzem
0.300
Figure 115. Predictions genle-genne perturbation effect Norman dataset evaluated across top differentially expressed genes for 8 train-test splits of increasing difficulty, (a) MSE for all prediction models. Experiments were carried out in triplicate for each model. heatmap shows the mean values (L). (b) Average AUSPC (1) across sparsification probabilities each model with standard error bars.
1.5.2 NORMAN DOUBLE-GENE
MSB Armw Spactisttion Probhilitzs Mntris
AUSIC for awch Medil
EXGIT
0.699
0.684
0.741
07011
0.7238
03200
2.1322
06754
0,7525
0.564
0.7855
(1) 245OV
aFoendiicn
0.904
0.3644
0.5900
0,4533
0.65
Moai bastinn...
6.6957
0.7078
0.7557
0.666
0.797
0,0000
0.5854
0.40
UrB
066
0.6052
0.7989
0.801
0.7357
0.517
0035
Gaoderor
Spenidiartoo ProbeMlly
MSB
Figure 116. Predictions of double- double-gene gene perturbation effect for the Norman dataset evaluated across the top 20 differentially expressed genes for train-test splits of inereasing difficulty. (a) MSE for all prediction models. Experiments carried out in triplicate for each model. The heatmap shows the mean MSE values (L). (b) Average AUSPC (L) across sparsification probabilitics for each model with standard error bars.
42

---

PertEval-scFM
1.6. MSE for all Models compared to Mean Baseline across 20 DEGs
1.6.1. NORMAN SINGLE-GENT
SpecEadion Pubukiio
Apust Kuaie Potuki
(a) Baseline MLP
(b) ScBERT
Speckaatin
Spacc
(c) scGPT
(d) Geneformer
(c) UCE
(1 scFoundation
Figure 117. MSE as a function of sparsification probability the different models evaluated across top 20 differentially expressed genes. This is a depiction of the curves that are used to calculate the AAUSPC.
43

---

PertEval-scFM
1.6.2. NORMAN DOUBLE-GENE
Specifatio Tribuliiyy
(a) Baseline MLP
(b) scBERT
(c) scGPT
(d) Geneformer
(e) UCE
(f) scFonndation
Figure 118. MSE as function of sparsification probability the different models evaluated across top 20 differentially expressed genes. This is a depiction of the curves that are used to calculate the AAUSPC.
44

---

PertEval-scFFM
1.7. Mean Post-Perturbation Expression Profiles for IKZF3 and CEBPA
CEBPA
IKZF3
109
Mean post- perturbation expression
10-1
10-2
10-3
2500
5000
7500
10000
12500
15000
17500
Gene
Figure 119. Post-perturbation mean expression profiles for IKZF3 and CEBPA. The y-axis has been log traassfoooomed for visual clarity.
45


Reviews:
[{"review_id": "R1yPnA3h7F", "reviewer": "Reviewer_xRpb", "summary": "The paper titled \"PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction\" presents a standardized framework called PertEval-scFM to evaluate single-cell foundation models (scFMs) for predicting perturbation effects. The study focuses on assessing whether zero-shot scFM embeddings can enhance the prediction of transcriptional responses to perturbations compared to simpler baseline models. The key findings are:\n1.\tBenchmarking Framework: PertEval-scFM provides a systematic evaluation framework to compare scFM embeddings against baseline models in predicting perturbation effects. It includes three datasets to test model performance under different conditions.\n2.\tPerformance of scFM Embeddings: The results show that scFM embeddings do not consistently outperform simpler baseline models, especially when there is a distribution shift. \n3.\tChallenges in Predicting Strong or Atypical Effects: The study highlights that all models, including scFM embeddings, struggle with predicting strong or atypical perturbation effects. This suggests that current models may lack the ability to generalize well to unseen or extreme perturbations.\nOverall, the paper provides valuable insights into the challenges of using zero-shot scFM embeddings for perturbation effect prediction and highlights the need for advancements in both model development and dataset quality to address these limitations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough evaluation and constructive feedback. Below, we address each of the points raised.\n\n**Perturbation representation**\n\nWe address the limitation of simulating perturbations via gene knockouts in our response to Reviewer **pVrS** and refer you to it. You can also see the results of a preliminary experiment with upregulated embeddings at this [link](https://drive.google.com/file/d/1jju-RECJcVANDfUj9s5-oobKPFSNQ6_9/view?usp=drive_link). \n\n**Fine-tuning**\n\nOne of the main goals of our benchmarking approach is to assess the zero-shot information content of pre-trained embeddings. If the models encode meaningful biological information for perturbation prediction, this would be apparent without task-specific adaptation. If performance gains only appear through fine-tuning, this challenges the premise that these models inherently learn generalizable representations.\n\nMLP probes are commonly used as an approach to answer the question of information content of embeddings in NLP and CV, as they evaluate representation quality while removing confounding effects pertaining to task-specific prediction heads, which introduce inductive biases \\[1, 2, 3\\]. We also note that previous work has investigated the performance of fine-tuned versions of scGPT and scFoundation \\[4]. This study found that simple linear baselines still outperformed these models, suggesting that fine-tuning does not fully address their limitations. Because fine-tuning performance is addressed in other studies and because it fundamentally goes against our approach of establishing existing information content, we do not include fine-tuning in our study design. The findings we present therefore highlight that zero-shot scFM embeddings do not contain useful biological information pertaining to the perturbation prediction task, which in and of itself is an important finding.\n\n**Discrepancy with scGPT results**  \n\nSeveral factors may explain the discrepancies between obtained results: scGPT evaluates model performance **after fine-tuning** on perturbation data, reporting high performance, whereas we focus on evaluating zero-shot information content. We also evaluate robustness under distribution-shift, which is not considered in scGPT. We view our study as complementary to scGPT rather than contradictory. It highlights current limitations in zero-shot scFMs and clarifies the gap between pre-training and real-world application. We believe that these complementary perspectives can help guide future improvements so that models better capture perturbation effects.\n\n**Evaluation**\n\nMSE was selected as the primary metric due to its strong biological grounding and demonstrated effectiveness in capturing perturbation effect, as opposed to Pearson Correlation \\[5\\]. Furthermore, we aimed to provide a toolbox of comprehensive and complimentary metrics, which include: AUSPC (distribution shift robustness), E-distance (perturbation magnitude), and contextual alignment (pre-training relevance). Together, these provide a robust framework for evaluating scFMs that model transcriptomic perturbation outcomes. The comprehensiveness of our evaluation framework has also been recognised as one of the strengths of our paper by other reviewers.\n\nWe would also like to note that GEARS is not considered a simple baseline, rather a SOTA model which takes biological priors into account and is developed specifically for perturbation effect prediction. The fact GEARS outperforms the zero-shot scFMs supports our conclusions that a biologically grounded architecture, which incorporates strong inductive biases, is more useful than pretraining for this task. \n\nWe hope these clarifications demonstrate the careful thought that has gone into our experimental design and underscores the broader significance of our findings. By evaluating zero-shot capabilities across scFMs using a rigorous and biologically motivated framework, we provide a valuable benchmark and identify key limitations in current approaches. We believe this work will help guide future research toward more robust, generalizable, and biologically meaningful models.\n\n\\[1\\] [https://arxiv.org/abs/2103.00020](https://arxiv.org/abs/2103.00020)\n\n\\[2\\] [https://arxiv.org/abs/1905.06316](https://arxiv.org/abs/1905.06316) \n\n\\[3\\] [https://doi.org/10.1038/s42256-024-00949-w](https://doi.org/10.1038/s42256-024-00949-w) \n\n\\[4\\]  [https://www.biorxiv.org/content/10.1101/2024.09.16.613342v4](https://www.biorxiv.org/content/10.1101/2024.09.16.613342v4)\n\n\\[5\\] [https://www.biorxiv.org/content/10.1101/2023.12.26.572833v1](https://www.biorxiv.org/content/10.1101/2023.12.26.572833v1)", "claims_and_evidence": "The claims made in the submission regarding the performance of scFM models in perturbation effect prediction are not fully supported by clear and convincing evidence. Specifically, the assertion that scFM models underperform compared to simpler baselines like Gears, and that they are not well-suited for perturbation tasks, is problematic for the following reasons:\n1.\tData Usage and Understanding Issues:\no\tThe submission uses the Norman dataset, which is a CRISPRa dataset designed for activation perturbations of specific genes. However, the authors assume a knockout perturbation scenario for all scFM models. This discrepancy between the actual perturbation type (activation vs. knockout) may lead to inaccurate model evaluations. The scFM models are not adequately simulating the true perturbation conditions, which could skew the results and undermine the validity of the conclusions.\n2.\tUnfair Model Comparisons:\no\tThe comparison between scFM models and baselines appears to be unfair due to the lack of proper fine-tuning for scFM models. scFM models are designed to learn embeddings from single-cell data, but perturbation tasks have unique characteristics, such as the specific changes in gene expression following targeted gene editing. To fairly compare these models, fine-tuning on relevant perturbation datasets is essential. The submission fails to follow this approach, unlike the SCGPT paper, which fine-tunes on a portion of the data and evaluates on the remaining portion. Without this fine-tuning, the scFM models may not be able to leverage their full potential for perturbation prediction, leading to misleading performance metrics.\n3.\tInconsistency with Previous Work:\no\tThe results presented in this submission conflict with those reported in the SCGPT paper, which demonstrated superior performance on the Norman and Replogle datasets. This inconsistency raises questions about the reliability of the current findings and suggests that the methodology or assumptions used in this study may be flawed.\nRecommendations for Improvement:\nTo strengthen the claims and provide more convincing evidence, the authors should consider the following adjustments:\n1.\tCorrect Data Interpretation:\no\tRe-evaluate the perturbation scenarios used in the experiments. For datasets like Norman, ensure that the perturbation type (e.g., activation vs. knockout) is accurately reflected in the model setup. This alignment will provide a more realistic assessment of the models' capabilities.\n2.\tFair Model Evaluation:\no\tImplement a fine-tuning step for scFM models using a portion of the perturbation datasets. This approach will allow the models to adapt to the unique characteristics of perturbation tasks and provide a more accurate comparison with baselines. The evaluation should then be conducted on the remaining data to assess the models' performance fairly.\n3.\tReconciliation with Existing Literature:\no\tAddress the discrepancies between this study's results and those from the SCGPT paper. A detailed discussion of the differences in methodology, assumptions, and data usage will help clarify the reasons for the conflicting findings and enhance the credibility of the conclusions.\nIn conclusion, while the submission provides a valuable attempt to benchmark scFM models for perturbation effect prediction, the current evidence is not sufficiently robust to support the claims. Addressing the data usage issues and ensuring fair model comparisons are critical steps to validate the findings and contribute meaningfully to the field.", "methods_and_evaluation_criteria": "The proposed framework, PertEval-scFM, aims to standardize the evaluation of single-cell foundation models (scFMs) for perturbation effect prediction. While the framework addresses an important problem, it has significant limitations:\n1.\tLack of Fine-Tuning: The framework does not properly fine-tune scFMs on perturbation tasks, likely underestimating their potential. Fine-tuning is essential for adapting models to the unique characteristics of perturbation datasets.\n2.\tLimited Evaluation Metrics: The framework relies on a limited set of evaluation metrics, which may not fully capture the models' performance across different aspects of perturbation prediction.\nRecommendations:\n•\tIncorporate Fine-Tuning: Include a fine-tuning step using a portion of the perturbation datasets to allow scFMs to adapt and showcase their full potential.\n•\tUse More Evaluation Metrics: Expand the set of evaluation metrics to provide a more comprehensive assessment of model performance.\nIn summary, the framework needs to incorporate fine-tuning and use a broader range of evaluation metrics to provide a fair and accurate assessment of scFMs for perturbation tasks.", "theoretical_claims": "Upon reviewing the manuscript, I did not encounter any explicit theoretical claims that required verification through proofs. The paper primarily focuses on empirical evaluations and the development of a benchmarking framework for assessing the performance of single-cell foundation models in perturbation effect prediction tasks. It does not present formal mathematical theorems or proofs that would necessitate validation in the traditional sense.", "experimental_designs_or_analyses": "I have carefully examined the experimental designs and analyses presented in the manuscript, particularly focusing on the use of the Norman dataset. This dataset is a CRISPRa dataset designed for activation perturbations of specific genes using CRISPR technology. However, the authors have incorrectly assumed a knockout perturbation scenario for all single-cell foundation models (scFMs) in their analyses. This discrepancy between the actual activation perturbations in the dataset and the assumed knockout scenario means that the scFMs were not accurately simulating the intended perturbation conditions. As a result, the experimental outcomes are not reliable, and the conclusions drawn from this dataset are questionable.", "relation_to_broader_scientific_literature": "The paper introduces a benchmarking framework aimed at evaluating the performance of several single-cell foundation models (scFMs) on perturbation effect prediction tasks. The authors have chosen to compare models such as Geneformer, scBERT, scFoundation, scGPT, and UCE with baseline models like Gears and a mean baseline across datasets including Norman, Replogle K562, and Replogle RPE1.", "essential_references_not_discussed": "After a thorough review of the paper and the relevant literature, I did not identify any essential works that are missing from the citations or discussions in the paper.", "other_strengths_and_weaknesses": "A significant strength of the paper is the introduction of a novel framework aimed at systematically evaluating single-cell foundation models (scFMs) on perturbation tasks within single-cell data. This is particularly noteworthy as it addresses the challenge of data distribution shifts, which is a common issue in real-world applications and has been less explored in previous studies.", "other_comments_or_suggestions": "I recommend that the authors consider incorporating additional standard evaluation metrics for a more comprehensive assessment. Specifically, the inclusion of Pearson correlation coefficient, which is commonly used in perturbation tasks to measure the linear relationship between predicted and actual perturbation effects, could enhance the robustness of the evaluation. This would provide further insight into the models' performance and align the assessment more closely with established practices in the field.", "questions_for_authors": "1.\tPerturbation Assumption\nYou have modeled all perturbations as knockouts, even though the datasets, such as Norman, actually involve activation perturbations using CRISPRa technology. Could you explain the rationale behind this modeling choice? How might this assumption affect the accuracy of the model evaluations?\n2.\tLack of Fine-tuning\nThe paper does not include a fine-tuning step for the pre-trained models, which is commonly used to adapt models to specific tasks like perturbation prediction. What are the reasons for not incorporating fine-tuning, and could you provide results with fine-tuning to compare?\n3.\tDiscrepancy with SCGPT Results\nThere is a significant discrepancy between your results and those reported in the SCGPT paper for perturbation tasks on the same datasets. Can you provide an explanation for these differences?\nHow Possible Responses Would Change My Evaluation:\n•\tFor the first question, if the authors can provide a compelling justification for the knockout assumption or demonstrate that the results hold even with activation perturbations, it could strengthen the validity of their experimental design.\n•\tFor the second question, including fine-tuning results could potentially show improved performance of the pre-trained models, which might alter the perception of their capabilities in perturbation tasks and could influence my evaluation positively if it addresses a critical limitation.\n•\tFor the third question, an explanation that accounts for the differences without undermining the credibility of either study would be necessary. Understanding the reasons behind these discrepancies is crucial for assessing the reliability of the findings presented in this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ARH6fgLffk", "reviewer": "Reviewer_mxRp", "summary": "This paper establishes a protocol to evaluate, in a standardized way, the performance of single cell foundation models at predicting the effect of perturbations. The authors evaluate using two data sets of CRISPR perturbations, combined with an approach to explicitly evaluate the effect of out-of-distribution learning. The results are very clear: leading foundation models do not fare better than simple baselines, and are outperformed by models that make particular effort to incorporate prior knowledge about gene networks and interactions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback on the work presented. We are committed to maintaining the benchmark as a live and extensible resource. Because we have prioritized reproducibility and modularity in our design, incorporating new models and datasets is straightforward. We are actively monitoring developments in the field, and one exciting recent advancement is Tahoe-100M \\[1\\], currently the largest perturbational single-cell dataset. We intend to extend our benchmark to include such resources in future iterations. As scFMs continue to evolve, especially with respect to how they represent perturbations, we plan to integrate these innovations to support broader perturbation types beyond knockouts.\n\nBelow we address the reviewer’s concerns:\n\n**Perturbation representation**\n\nWe acknowledge that nullifying gene expression may not align with biological reality, especially in gain-of-function settings such as CRISPRa. We refer you to our detailed answer on this point in the rebuttal to Reviewer **pVrS.** You can also see the results of a preliminary experiment with upregulated embeddings at this [link](https://drive.google.com/file/d/1jju-RECJcVANDfUj9s5-oobKPFSNQ6_9/view?usp=drive_link). \n\nWe thank you for the suggestion of incorporating attribution-based methods. If we understand correctly, calculating the effects of infinitesimal perturbations is a post-hoc explainability tool, which we would be interested in incorporating in our framework. However, it is unclear to us how we would use it for the initial representation of perturbations. We would be happy to discuss this point further. \n\n**Limitations of averaging and unpaired observations in scRNA-seq** \n\nCurrent methods do not allow for true paired observation of pre- and post-perturbation states for the same cell, due to the destructive nature of measurement. As the reviewer has noted, this introduces ambiguity into how perturbation effects are estimated.\n\nIndeed, due to the context-dependent nature of biology, it is very likely that there are other factors that affect gene expression. This trade-off is an inherent limitation of current experimental techniques and affects all existing perturbation modeling studies. It is unclear to what extent this distorts the results without access to paired samples, or a robust estimate of paired samples. We believe the development of experimental protocols enabling paired measurements would significantly advance this line of work. \n\nTo attempt to mitigate this, we adopt a pseudo-bulking strategy that averages across cells to reduce noise and obtain more robust perturbation signatures. Specifically, for each condition, we randomly sample 500 control cells and average their expression profiles, pairing them with the mean expression of all perturbed cells within the same perturbation. This approach helps suppress cell-to-cell variability in the perturbed population, thereby making the overall perturbation effect more apparent. However, as the reviewer rightly points out, the unpaired nature of the control introduces uncertainty about whether the starting state truly mirrors that of the perturbed cells pre-perturbation.\n\nThank you for pointing out the typo in the appendix, we have now corrected it. \n\nWe thank the reviewer for recognizing the clarity and relevance of our work. We appreciate the suggestions regarding perturbation representation and data limitations, both of which highlight important areas for continued exploration in this field. We hope our responses address the reviewer’s concerns and demonstrate the care with which we have designed our framework to support the community in developing and evaluating biologically meaningful single-cell foundation models.\n\n\\[1\\] [https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1](https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1)", "claims_and_evidence": "The evaluations and discussion in this manuscript are clear and well supported. The central claim is supported by multiple overlapping analyses, suggesting that small changes are unlikely to explain the main claim. Notably, this paper, explicitly simulates the effect of distribution shift, as well as a wider range of genes. This should conclusively address a main source of controversy surrounding single cell foundation models.", "methods_and_evaluation_criteria": "There are several evaluation criteria, all of which give consistent results. However, all of these derive from perturbseq or other CRISPR-based technologies. As more technologies and datasets come online, I encourage the authors to maintain their online platform to incorporate a wider range of data sets, not all of which are likely to be knockouts.", "theoretical_claims": "No proofs are associated with this paper.", "experimental_designs_or_analyses": "The experimental designs are thorough.", "relation_to_broader_scientific_literature": "Although this is not the first broad evaluation of the ability of single cell foundation models to predict the effect of perturbations, it is notable for its thoroughness as well as exploration of sensitivity to domain shift. If I were going to develop a new set of embedding, I would look to this framework to evaluate the model.", "essential_references_not_discussed": "No essential references missed.", "other_strengths_and_weaknesses": "The strength of this paper lie in its significance clarity and relevance to the field. It is not particularly novel, but that is not of concern.", "other_comments_or_suggestions": "In the appendix, there is a typo in the title of section D.1.", "questions_for_authors": "The one area in which I see room for further analysis and discussion is the form of how perturbations are communicated to the model. Setting gene expression to zero may not be the ideal strategy. There are many other possibilities, furthermore, such as attribution analysis (i.e. calculating the linearized effect of infinitesimal perturbations). \n\nI also have continuing concerns about the data available for this class of question. We are never able to observe the original cell before and after a perturbation. Thus, we often look at the effect of one gene upon other genes, averaged across cells. To what extent does this averaging distort the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ir7cFxNOQ", "reviewer": "Reviewer_pVrS", "summary": "This paper introduces PertEval-SCFM a benchmark for zero-shot single-cell foundational model embeddings to capture transcriptional perturbation effects. It is claimed that scum embeddings do not provide consistent improvements over baseline models.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and positive evaluation of our work. We are glad that the reviewer found our contribution timely and the benchmark, metrics, and manuscript clear and useful for the field. Below, we address the specific concerns raised.\n\n**Causal intervention**   \n\nWe acknowledge that setting gene expression to zero may prevent the model from distinguishing between biological zeros and knockout interventions. However, there will be differences in coexpression accompanying a natural zero, which will not occur for induced zeros. We acknowledge this limitation in our paper and agree that future foundation models would benefit from incorporating representations that explicitly encode the nature of the intervention. A potential avenue of exploration would be to include perturbation tokens during pretraining and not only during fine-tuning. \n\n**Perturbation representation**   \n\nWhile simulating perturbations via gene knockout may seem counterintuitive for CRISPRa datasets like Norman, we chose this strategy to address the challenges inherent in standardizing perturbation presentation across diverse scFMs. Different scFMs represent perturbations in different ways \\- for instance, scGPT encodes gene expression as numerical values, whereas Geneformer uses rank-order representations.\n\nTherefore, upregulation is difficult to simulate consistently: parametric models like scGPT require arbitrary magnitude choices, with no biologically grounded way to determine appropriate magnitude, while rank-based models like Geneformer lack a principled way to re-rank genes. On the other hand, knockout offers a model-agnostic, unbiased representation that avoids model-specific inconsistencies, enabling fairer comparisons across scFMs. \n\nNonetheless, we performed an experiment using an alternative perturbation representation, which simulates gain-of-function by doubling the perturbed gene’s expression in the Norman control expression data. We used scBERT to generate embeddings for the *in silico* upregulated input representation (scBERT+). We observed minimal performance differences (see Table 1 and the Figure in this [link](https://drive.google.com/file/d/1jju-RECJcVANDfUj9s5-oobKPFSNQ6_9/view?usp=drive_link)) compared to the knockout representation (scBERT-), supporting our approach. The difficulty of simulating realistic perturbations reflects a broader challenge in scFM methodology and while a full exploration of perturbation strategies is beyond the scope of this work, we hope our findings motivate further research in this area. \n\nTable 1: MSE ± standard deviation for Norman single gene embeddings generated with scBERT using an in silico knockout (scBERT \\-) vs. an upregulation strategy (scBERT+)\n\n|  | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| scBERT - | 0.0630 ± 0.0031 | 0.0634 ± 0.0062 | 0.0676 ± 0.0051 | 0.0636 ± 0.0031 | 0.0592 ± 0.0015 | 0.0645 ± 0.0107 | 0.0849 ± 0.0060 |\n| scBERT+ | 0.0640 ± 0.0038 | 0.0620 ± 0.0038 | 0.0658 ± 0.0077 | 0.0610 ± 0.0009 | 0.0659 ± 0.0046 | 0.0744 ± 0.0122 | 0.0853 ± 0.0064 |\n\n**Dataset and pre-processing choices**  \n\nWe agree that dataset diversity is critical for generalizability. We selected Norman and Replogle as they represent two of the most widely used, high-quality single-cell perturbation datasets currently available. Moreover, both Norman and Replogle contain two separate datasets each, so we are effectively exploring four datasets. Importantly, these all allow us to benchmark under controlled conditions and introduce distribution shifts via SPECTRA. Our platform is designed to be extensible, and we are actively exploring incorporation of additional datasets, including large-scale resources such as the recently published Tahoe-100M \\[1\\].\n\nRegarding preprocessing, we applied a consistent pipeline that follows standard best practices in single-cell analysis across all models, including highly variable gene (HVG) selection, transcript count normalization, and log transformation. These steps are aligned with those used in the original scFM papers, differing only in hyperparameter value choices. We ensure internal consistency in preprocessing across all model evaluations to enable fair comparisons. While we acknowledge that preprocessing choices can affect results, this is a well-recognized, broader limitation of single-cell data analysis, which we explicitly discuss in our manuscript.\n\nWe hope our responses clarify the rationale behind our design choices and reinforce the value of our benchmark as a foundation for future work. We believe that making the limitations explicit and quantifiable is a critical step for advancing the development of biologically meaningful scFMs.\n\n\\[1\\] [https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1](https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1)", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. \n\nto construct embeddings for perturbed cells, expression counts of perturbed genes are set to zero in cells exposed to that perturbation \"effectively simulating the perturbation in silico\"\n\nWhile I agree that this is a valid design, it also has some limitations. Namely there is no way for the model to distinguish between an intervention and a zero gene expression observation. From the causality literature, this is theoretically an issue. It seems somewhat difficult to claim that these models don't capture perturbation effects if this is the only type of embedding extracted.\n\nHowever, the authors acknowledge this problem in the limitations.\n\nOne other concern is that the Norman dataset uses CrisperA (i.e. activation) so this zeroing of gene expression seems even more wrong in this context. Have the authors tried an activation type intervention?", "relation_to_broader_scientific_literature": "There are scFM models that claim to model perturbations zero-shot. This work disputes those claims. This is a useful contribution.", "essential_references_not_discussed": "None to my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n* Originality: I have not seen a paper that benchmarks the zero-shot effectiveness of scFMs on perturbations. This is a timely work.\n* Significance: While the conclusions of this work are not overly surprising, they set a useful benchmark and set of metrics for future work.\n* Clarity: I found the writing clear given the complexity and domain knowledge of this particular benchmark.\n\nWeaknesses:\n* This work only explores two datasets, both of which have their own issues. It's not clear if these findings are generalizable across datasets.\n* Fairly standard preprocessing is used. It's unclear if this is the same preprocessing used to train various scFMs thus it is unclear how fair this comparison is in this context to me.", "other_comments_or_suggestions": "No", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LnGKLkL4aj", "reviewer": "Reviewer_KAX4", "summary": "The paper introduces **PertEval-scFM**, a standardized framework to evaluate single-cell foundation models (scFMs) for predicting perturbation effects. \nKey contributions include:\n1. **Framework**: A modular toolkit for zero-shot evaluation of scFM embeddings.\n2. **Metrics**: Introduces **AUSPC**, **E-distance**, and **contextual alignment** to assess model robustness and generalization under distribution shifts.\n3. **Findings**: scFM embeddings do not consistently outperform baseline models, especially under distribution shifts. **GEARS** performs best, highlighting the need for task-specific architectures.\n\nPertEval-scFM provides a comprehensive benchmark for scFMs, emphasizing challenges in perturbation effect prediction and guiding future research.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive feedback, as well as for recognizing the strengths of our work, including the comprehensiveness of our benchmark and its potential to guide future research. Below, we respond to the specific concerns raised.\n\n**GEARS**\n\nWe agree that including GEARS in Fig. 2 provides a more complete picture of model performance and have updated the figure accordingly (see [link](https://drive.google.com/file/d/1dJBOtZ2ytpMee1rK5qHWVIqJ_NVx1rnZ/view?usp=sharing)).  \n\nGEARS uses a **gene coexpression** graph and a **gene ontology** graph to model gene relationships and perturbations, respectively, thus injecting biological priors into the model architecture. On the other hand, the SPECTRA graph is built on **cell-to-cell** similarity, which enables us to measure model robustness to distribution shift by simulating increasingly different train-test sets. GEARS is therefore not aware of the relationship between cells, but aware of the relationship between genes, which therefore doesn’t interfere with the way we assess robustness. We have clarified this in the manuscript to avoid any ambiguity.\n\nRegarding the implementation, GEARS was faithfully reproduced following its original design \\[1\\]. Briefly, it considers a perturbation dataset of $N$ cells, where each cell is described by a gene expression vector $g \\\\in \\\\mathbb{R}^K$ and an associated perturbation set $P \\= (P\\_1, \\\\ldots, P\\_M)$. The model learns a function $f$ that maps a novel perturbation set to its post-perturbation gene expression profile. Specifically, GEARS:\n\n* Uses a GNN encoder $f\\_{\\\\text{pert}}: \\\\mathcal{Z} \\\\rightarrow \\\\mathbb{R}^d$ to embed perturbations,\n\n* Uses a second GNN encoder $f\\_{\\\\text{gene}}: \\\\mathcal{Z} \\\\rightarrow \\\\mathbb{R}^d$ to embed genes,\n\n* Combines these embeddings via a compositional module,\n\n* Applies a cross-gene decoder $f\\_{\\\\text{dec}}: {\\\\mathbb{R}^d}^K \\\\rightarrow \\\\mathbb{R}^K$ to predict the post-perturbation gene expression vector.\n\nTraining is conducted end-to-end using the autofocus direction-aware loss, with default hyperparameters from the GEARS paper (e.g., hidden size 64, one GNN layer for both GO and co-expression graphs). The only modification we made was to the GEARS dataloader: we adapted the \\`prepare\\_splits\\` function to use SPECTRA-defined training and test splits via a custom mapping (in \\`set2conditions\\`). \n\n**Figures 3b and 4**\n\nIn Fig. 4 we further analyse the outliers seen in Fig. 3b. There, we explore the distribution of the perturbation effect on the top 20 DEGs (dashed line), and how such perturbation effect is predicted by the models (data points). Dense clustering near the dashed line (e.g., Fig. 4a) indicates high agreement with ground truth, while more dispersed predictions (e.g., Fig. 4c) suggest low reliability. This analysis suggests that the magnitude of the perturbation effect is not the only factor affecting performance, but that the distribution of such effect also matters. Perturbations with lower overall effect, but with an atypical distribution will also challenge the models. These plots emphasize how the structure of the ground truth perturbation distribution affects model accuracy, a point we highlight in the revised figure caption. We have revised the Figure to improve clarity, using smaller, non-overlapping data points to better distinguish their values. The revised figure is available at this [link](https://drive.google.com/file/d/1VymFV4tVhO7Xvp75AuSJZToWEtLC4zSu/view?usp=drive_link).\n\nWe hope these clarifications and updates address the reviewer’s concerns. We thank the reviewer again for their thoughtful comments, which helped us strengthen the clarity and completeness of our work.\n\n\\[1\\] [https://doi.org/10.1038/s41587-023-01905-6](https://doi.org/10.1038/s41587-023-01905-6)", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. This is primarily a benchmarking experiment, with limited theoretical proof.", "experimental_designs_or_analyses": "Yes. Although GEARS achieves the best performance, it requires constructing a cell adjacency matrix based on cell similarity first. This results in GEARS still being aware of the relationships between cells even after SPECTRA divides the training and test sets. Since the paper does not provide detailed information on the implementation of GEARS, I hope this can be clarified further.", "relation_to_broader_scientific_literature": "1. Fair Comparison in Single-Cell Foundation Models\nWhile Geneformer and scGPT have demonstrated the promising potential of foundation models in single-cell biology, this study systematically evaluates their true practical utility under zero-shot settings with distribution shifts.\n\n2. Perturbation Effect Prediction\n\n\n3. Evaluation Frameworks for Biological Models\nComparison to scEval (Wang et al., 2024) evaluated 8 general tasks (e.g., clustering, batch correction), PertEval-scFM introduced perturbation-specific innovations.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weaknesses:\n1.Figure 2: Including GEARS results would provide a more complete comparison of model performance\n2.Figure 3b: The proportion of high E-distance samples under high probability conditions is relatively low. Further analysis of outliers could reveal underlying patterns.\n3.Figure 4: The dense data points make it difficult to draw clear conclusions.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces PertEval-scFM, a standardized evaluation framework for benchmarking the zero-shot performance of single-cell foundation models (scFMs) in predicting perturbation effects. The authors apply their framework to multiple scFMs and compare them against baseline models, concluding that zero-shot embeddings from current scFMs do not offer performance gains over simple baselines.

This work is timely and relevant as scFMs gain traction in single-cell analysis. Systematic evaluations are necessary. This benchmark is a welcome effort in this direction. The writing is clear, the methodology is well-described, and the source code is made available. The framework includes a few datasets, metrics, and domain shift modeling, which is especially important in biological applications.

However, as multiple reviewers point out, the conclusions in the abstract and conclusion sections overgeneralise beyond the experimental design. The claim that “scFM embeddings offer no improvement” is not fully justified, as the study examines only zero-shot usage in a limited perturbation setting (mostly simulating knockout) and even seemingly misrepresents activation-based perturbations. Next, despite the authors’ clarifications, concerns remain about the comparability of GEARS (which is fine-tuned) to non-finetuned scFM embeddings. Lastly, the distinction between “zero-shot” and “fine-tuning” setups is not well-defined, contributing to confusion and likely misinterpretation of the scope of the findings.

Despite its flaws, this work presents a useful framework that can serve as a starting point for future, more comprehensive evaluations of scFM embeddings. The core idea of this work, namely benchmarking perturbation prediction in a zero-shot setting, is important and original. However, the execution and framing need more work. In particular, the conclusions must be narrowed to what is truly supported by the experiments, and comparisons across modeling paradigms must be more carefully delineated to avoid misleading readers. While this paper makes a valuable contribution, its claims are currently too strong for the evidence presented, and the comparative evaluation with GEARS remains insufficiently controlled. I believe this paper would be stronger after substantial revision, with toned-down conclusions, a clarified methodology, and potentially a broader experimental design that includes fine-tuning settings. 

With the authors addressing at least some of the aforementioned concerns in the camera-ready version, my suggestion would be a weak acceptance.
Decision: Accept (poster)