scDrugMap: benchmarking large foundation models for drug response prediction

Domain: neuroscience
Published: 2025-12-11
Editorial outcome: major_revision

Abstract
Drug resistance remains a major challenge in cancer treatment. While single-cell profiling offers unprecedented resolution for uncovering resistance mechanisms, the potential of emerging foundation models for drug response prediction at the single-cell level is still largely unknown. Here, we introduce scDrugMap, a unified framework featuring both Python toolkits and an interactive web server for benchmarking and predicting drug responses with state-of-the-art foundation models. scDrugMap evaluates eight single-cell foundation models and two large language models across 495,000 cells from 60 datasets, spanning diverse tissues, drugs, cancer types, and treatment conditions. In pooled-data evaluation, scFoundation delivered the strongest performance, particularly in tumor tissue. In cross-data analysis, UCE performed best after fine-tuning, while in zero-shot settings, scGPT achieved the highest accuracy. Together, scDrugMap provides the first systematic benchmark of foundation models for single-cell drug response prediction and offers a powerful, user-friendly platform to accelerate drug discovery and translational precision oncology.

Introduction
Drug resistance continues to be a significant challenge in disease treatment, particularly in cancer therapy1. An analysis of 85 cancer drugs approved by the U.S. Food and Drug Administration (FDA) revealed a median response rate (RR) of 41%, with nearly half exhibiting an RR below 40%, and two-thirds showing a complete RR of less than 10%2. Even in biomarker-based personalized cancer treatments, response rates remain modest—30.6% compared to just 4.9% for non-biomarker-based therapies3. Moreover, drug resistance is closely associated with patient survival rates. For example, resistance to KRAS inhibitors, a standard treatment for non-small cell lung cancer (NSCLC), is associated with a median survival of only 6.3 months4. Similarly, in glioblastoma, resistance to chemotherapy and radiotherapy limits median survival post-diagnosis to just 12–15 months, with less than 10% of patients surviving longer than five years5. These findings underline the pressing need for further investigation and advanced approaches to better elucidate drug resistance mechanisms and improve treatment efficacy.

Over the past decade, single-cell RNA sequencing (scRNA-seq) technology has revolutionized our understanding of cellular and molecular heterogeneity, enabling high-resolution analysis of drug responses across diverse cell types and subtypes6–13. For instance, scRNA-seq analysis of NSCLC tumor tissue identified elevated expression of estrogen metabolism enzymes and serum estradiol in cancer cells exhibiting poor pathological responses to PD-1 blockade immunotherapy. Additionally, aged CCL3+ neutrophils interacting with tumor-associated macrophages were identified as potential biomarkers of poor therapy response13. In basal cell carcinoma (BCC), scRNA-seq revealed that novel T cell recruitment, rather than the reinvigoration of pre-existing tumor-infiltrating lymphocytes, was pivotal for effective PD-1 blockade therapy11. Similarly, in inflammatory breast cancer (IBC), single-cell profiling uncovered a shift in cell state from luminal to basal/mesenchymal, implicating the JAK2/STAT3 signaling axis as a key mediator of chemotherapy resistance9. These findings underscore the power of scRNA-seq in uncovering distinct cell states and gene expression profiles driving drug response heterogeneity, enabling the discovery of novel biomarkers and informing personalized therapeutic strategies.

While scRNA-seq offers invaluable insights at single-cell resolution, its high dimensionality, noise, variability, and data sparsity pose significant challenges for interpretation and the extraction of meaningful biological signals. Additionally, differences in sample preparation, library construction, and sequencing platforms across laboratories introduce substantial batch effects and technical variability. To address these issues, recent advancements have introduced large-scale foundation models tailored for single-cell data analysis14–16. Prominent examples include scFoundation15, scBERT14, scGPT16, and Geneformer17. These models were pre-trained on large-scale scRNA-seq datasets without being specifically designed for downstream tasks18. Through transfer learning and fine-tuning, single-cell foundation models have demonstrated strong performance across diverse applications, including cell type annotation and batch effect correction18. Notably, certain models offer specialized capabilities—scFoundation for drug response prediction15, scGPT for multi-omics integration (e.g., gene expression and chromatin accessibility)16, and Geneformer for predicting gene dosage sensitivity and chromatin dynamics17. On the other hand, general domain large language models have also been adapted to analyze sequencing data19,20. Despite these advancements, no comprehensive benchmarking studies have rigorously evaluated these foundation models for drug response prediction across diverse single-cell datasets. This gap motivates our study, which aims to develop a unified platform and systematically evaluate existing large models for drug response prediction across a wide range of cancer types, tissues, and therapeutic regimens.

To achieve this, we developed scDrugMap, a unified framework for drug response prediction, and conducted the first comprehensive benchmarking of ten foundation models, comprising eight domain-specific models for single-cell data and two general-purpose natural language models. The curated dataset of scDrugMap includes a primary collection of 326,751 cells derived from 36 single-cell datasets across 23 studies, along with an external validation dataset, which includes 168,486 cells from 24 single-cell datasets across six drug-related studies. By covering a total of 495,237 single cells, we performed two evaluation strategies: pooled-data evaluation and cross-data evaluation. Our results indicate that different foundation models exhibit varying capabilities in drug response prediction, with pooled-data evaluation demonstrating better performance than cross-data evaluation. To facilitate broader application, scDrugMap is developed with a command-line tool and an interactive web server (https://scdrugmap.com/) for seamless drug response prediction using a variety of foundation models. Collectively, our study makes several key contributions: (1) offering model selection guidance based on dataset-specific characteristics and evaluation scenarios; (2) providing an integrated, practical, and interactive tool for drug response prediction, biomarker discovery, and resistance mechanism analysis; (3) establishing a benchmark framework to inform and accelerate future research in drug discovery.

Results
Overview of scDrugMap
As an integrated framework for drug response prediction using single-cell data (Fig. 1a), scDrugMap consists of three core components: a comprehensive computational pipeline, an interactive web server, and a large-scale curated drug-related dataset—all designed to enable streamlined user access and efficient model evaluation. At the core of scDrugMap are foundation models (FMs), including eight single-cell-specific foundation models (tGPT21, scBERT14, Geneformer17, CellLM22, scFoundation15, scGPT16, CellPLM23, and UCE24) and two general natural language models (LLaMa3-8B25 and GPT4o-mini26). To adapt these models for drug response prediction, scDrugMap supports both zero-shot inference and fine-tuning using Low-Rank Adaptation (LoRA). Users can interact with scDrugMap through either a command-line interface or an intuitive web-based platform, allowing flexible model experimentation and analysis. For benchmarking and validation, we manually curated two distinct single-cell datasets: a primary collection and a validation collection (Data Collection and Preprocessing in Methods). By integrating data curation, model adaptation, and predictive analysis, scDrugMap serves as a comprehensive and accessible resource for advancing single-cell-based drug response research.

As shown in Fig. 1b, the curated datasets in scDrugMap span 14 cancer types, 3 therapy types, 5 tissue types, and 21 treatment regimens. The primary data collection includes 36 scRNA-seq datasets manually curated from 23 published studies (Fig. 1c), covering 11 major cancer types such as lung cancer, multiple myeloma, and melanoma, along with three therapy categories: targeted therapy, chemotherapy, and immunotherapy. The dataset encompasses four major tissue types: cell line, bone marrow aspirates, tumor tissue, and peripheral blood mononuclear cells (PBMCs) (see Supplementary Data 1). In addition, six distinct scRNA-seq technologies are included in our primary collection: 10x Genomics (n=27), BD Rhapsody (n=3), MARS-seqv2.0 (n=2), DropSeq (n=2), inDropv3 (n=1), and SMART-seqv4 (n=1). Following quality control, the primary dataset comprised 326,751 single tumor cells annotated with drug response information. Among these, cell lines, targeted therapy, and lung cancer represented the largest cell counts within their respective categories. Importantly, most subgroups (e.g., those for targeted therapy, bone marrow aspirates, and multiple myeloma) exhibited balanced distributions between drug-sensitive and drug-resistant cells (Supplementary Data 2).

For the validation data collection, we manually collected 17 scRNA-seq datasets from six other drug-related studies on solid tumors available in the GEO database7,8,10–12,27 (Fig. 1d). This validation dataset includes five cancer types: ovarian cancer, NSCLC, pancreatic cancer, colon cancer, and basal cell cancer, along with three therapy types: targeted therapy, chemotherapy, and immunotherapy. The represented tissue types include cell lines, tumor tissue, and organoids (Supplementary Data 1). After quality control, the validation set comprised 18,856 single-cell transcriptomes. Similar to the primary dataset, the largest proportions were again observed in cell line (tissue), targeted therapy (treatment), and NSCLC (cancer type) categories. Most validation subgroups also maintained a balanced representation of drug-sensitive and drug-resistant cells (Supplementary Data 2). Using these curated data, we implemented two distinct evaluations within scDrugMap: (1) pooled-data evaluation, where models were trained and tested on aggregated data from multiple studies; (2) cross-data evaluation, where models were tested independently on datasets from individual studies.

Pooled-data evaluation in primary data collection
First, we evaluated different FMs in scDrugMap using frozen layer training for predicting drug responses in the primary data collection. The results across various tissue types, cancer types, drug classes, and regimen are shown in Fig. 2a and Supplementary Fig. 1a. Specifically, for the cell line data, which contained the highest number of cells, scFoundation demonstrated the best performance (mean F1 score: 0.971) while scBERT performed the worst (mean F1 score: 0.630). In prostate and pancreatic cancer, LLaMa3 exhibited comparable performance with scFoundation (mean F1 scores: scFoundation-0.990, LLaMa3-0.913 for prostate; scFoundation–1.000, LLaMa3–0.963 for pancreatic cancer). Notably, this high performance may be due to the relatively small sample sizes (10,982 cells in prostate cancer and 6136 in pancreatic cancer) and pronounced imbalance (prostate cancer: 20.85% sensitive cells; pancreatic cancer: 11.38% sensitive cells) (Supplementary Data 2). Additionally, LLaMa3 was the best method for predicting drug response in the CAR-T regimen (mean F1 score: 0.875), and scGPT was the best for the carboplatin regimen (mean F1 score: 0.882). Other metrics-including AUROC, accuracy, precision, recall (Supplementary Fig. 1a), as well as F1 score comparison by Student’s t-test (Supplementary Note 1) further demonstrate that scFoundation achieves superior drug response predictions in most of the datasets among the primary data collection. In contrast, some models demonstrated poor performance in specific contexts. For instance, CellLM and scBERT underperformed in on PBMCs data (mean F1 score: 0.461 and 0.483, respectively) while Geneformer presented poor predictions on immunotherapy datasets (mean F1 score: 0.442). Additionally, the optimal performances of scFoundation, scGPT, and CellPLM remained stable or showed minor improvement before and after batch effect correction (Supplementary Note 2). Furthermore, to evaluate whether the scFMs outperform traditional methods in drug response prediction, we compared their performances with that of the multi-layer perceptron (MLP), scDEAL—a deep-learning model for cancer drug response prediction at single-cell level28, and DREEP, a statistical method for drug response estimation from single-cell transcriptional profiles of tumors29. Our results showed that in liver cancer data, scDEAL, DREEP, and MLP achieved approximately equivalent performances to that of scGPT (0.924), CellLM (0.924), LLaMa3 (0.924), Geneformer (0.924), and scBERT (0.924) (Supplementary Note 3).

We next evaluated different models using fine-tuning training for predicting drug responses (Fig. 2b; Supplementary Fig. 1b). The results from fine-tuning were overall consistent with those from embedding-based approaches, with scFoundation remaining the best and scBERT remaining the worst across most of the datasets. An exception was observed with melanoma cancer data, where scGPT outperformed scFoundation (mean F1 scores: 0.993 vs 0.978). In liver cancer, CellPLM performed comparably to scFoundation (mean F1 scores, CellPLM: 0.992; scFoundation: 0.997), both outperforming other models. Across drug regimens, scGPT performed similarly to scFoundation for ceritinib (mean F1 scores, scGPT: 0.985; scFoundation: 0.997) and outperformed scFoundation on vemurafenib (mean F1 scores, scGPT: 1.000; scFoundation: 0.990). Similarly, CellPLM demonstrated comparable performance to scFoundation for ceritinib (mean F1 scores, CellPLM: 0.990; scFoundation: 0.997), sorafenib (mean F1 scores, CellPLM: 0.996; scFoundation: 0.999), and vemurafenib (mean F1 scores, CellPLM: 0.977; scFoundation: 0.990). Other metrics, including AUROC, accuracy, precision, recall (Supplementary Fig. 1b), F1 score comparison by Student’s t-test (Supplementary Note 1), as well as correlation between ground truth and predicted labels (Supplementary Note 4) showed that scFoundation achieves accurate response predictions in diverse datasets.

Unlike the results under the layer-freezing strategy, all models in the fine-tuning setting outperformed the baseline models across various tissue types. Notably, scFoundation achieved the highest mean F1 scores across all tissue categories: 0.940 for peripheral blood mononuclear cells, 0.990 for tumor tissue, 0.962 for bone marrow aspirates, and 0.947 for cell lines (Fig. 2b). Additionally, UCE showed strong performance specifically in tumor tissue analysis, achieving a mean F1 score of 0.869. When evaluating performance across different cancer types, scFoundation remained the top-performing model in all cases except for melanoma, where scGPT outperformed it with a mean F1 score of 0.992, compared to 0.978 for scFoundation. In terms of drug type categories, scFoundation consistently demonstrated the best performance, with mean F1 scores of 0.953 for targeted therapy, 0.996 for chemotherapy, and 0.832 for immunotherapy. Batch effects removal showed minial impact on the model performances (Supplementary Note 2).

Taken together, fine-tuning the FMs generally outperformed layer-freezing based prediction. Among the evaluated models in scDrugMap, scFoundation, when fine-tuned, consistently demonstrated superior performance in drug response prediction across diverse datasets, including various tissues, drugs, cancer types, and regimens. scGPT, also fine-tuned, excelled in predicting responses for melanoma cancer and the vemurafenib regimen. In contrast, CellLM particularly underperformed in immunotherapy and neuroblastoma, under both training strategies.

Cross-data evaluation in primary data collection
To assess the generalizability of FMs in scDrugMap, we performed cross-data evaluation, in which models were trained on certain studies and tested on independent, unseen datasets. This evaluation simulates de novo drug response prediction scenarios in real-world settings. We first examined model performance using layer-freezing training. In general, most models performed less effectively across various tissue types, drug classes, cancer types, and regimens, compared to the pooled-data evaluation results (Fig. 3a; Supplementary Fig. 2). Nevertheless, a few models showed promising results. Specifically, regarding tissue types, scGPT achieved a mean F1 score of 0.858 in tumor tissue and the performance was statistically significantly higher than that of tGPT (mean F1 score: 0.603, Supplementary Note 1). UCE scored 0.723 in tumor tissue, and CellPLM reached 0.676 in bone marrow aspirate. For drug type, chemotherapy showed the best performance, with UCE and CellLM achieving mean F1 scores of 0.577 and 0.617, respectively. Regarding regimens, the top three performances were observed for UCE and scBERT with ibrutinib (mean F1 scores, UCE: 0.645; scBERT: 0.570), and scGPT with paclitaxel (mean F1 score: 0.566). Across cancer types, LLaMa3 generally showed the lowest variance in F1 scores. scFoundation achieved the highest performance in multiple myeloma, with a mean F1 score of 0.734. CellLM performed best specifically in acute lymphoblastic leukemia (mean F1 score: 0.635) and prostate cancer (mean F1 score: 0.636). We also observed that batch effects removal did not notably alter the performance patterns compared to the uncorrected data (Supplementary Note 2). Given that cross-data evaluations better reflect real-world clinical applications, foundation models typically show lower absolute F1 scores (generally below 0.8) under these more stringent conditions, highlighting the inherent challenge of generalizing across heterogeneous datasets. Nevertheless, they consistently outperform traditional architectures such as scDEAL, demonstrating greater robustness and transferability (Supplementary Note 3). This indicates that cross-data evaluations provide a more realistic measure of clinical utility: while foundation models retain a clear advantage over traditional methods, they still face important limitations, emphasizing the need for future work to further enhance their generalizability and performance.

Next, we evaluated model performance under the fine-tuning strategy, where UCE demonstrated relatively stronger performance compared to other models across various categories, despite the overall modest performance observed across all models (Fig. 3b; Supplementary Fig. 3). For example, UCE was the best model across tissue types (highest mean F1 score of 0.774 in tumor tissue), drug types (highest mean F1 score of 0.549 in targeted therapy), and regimens (highest mean F1 score of 0.677 in paclitaxel). In contrast, scFoundation, which had performed best under pooled-data evaluation, showed reduced generalizability in the cross-data setting. It was notably the least effective model for the ibrutinib regimen. Interestingly, CellPLM ranked second across drug types (mean F1 score of 0.681 in immunotherapy). Across cancer types, UCE showed top performance in acute lymphoblastic leukemia (ALL, mean F1 score of 0.752), while scFoundation maintained strong results in multiple myeloma (mean F1 score of 0.743). Additional relatively-strong performers included tGPT and CellLM for ALL (mean F1 scores: tGPT-0.728; CellLM-0.718). We also found that batch effects removal did not notably alter the performance patterns compared to the uncorrected data (Supplementary Note 2). Moreover, we conducted a case study on a multiple myeloma dataset using both ground truth and predicted drug response labels from fine-tuned scGPT (F1 score: 0.2645). Downstream bioinformatics analyses revealed that key biological insights that can be missed when relying on less accurate predicted labels. For example, top up-regulated genes COL1A2 and SEZ6L2 in resistant tumor cells and enriched biological processes such as cell differentiation, cell–cell adhesion, and the p53 pathway in the up-regulated genes were only observed in the ground truth–based analysis (Supplementary Note 5).

To further interpret model behavior, we visualized model-specific embeddings using Uniform Manifold Approximation and Projection (UMAP). These visualizations revealed that scFoundation and scGPT effectively distinguished drug-resistant and drug-resistant and drug-sensitive cells (Fig. 4), while other models showed limited separation (Supplementary Fig. 18). In addition, when stratified by scRNA-seq protocols, we observed consistent patterns where scFoundation achieved the best overall performance, while certain models stood out for specific sequencing technologies—for instance, the fine-tuned Geneformer performed notably well on data from the BD Rhapsody system (F1 score of 0.981) (Supplementary Note 6). Collectively, under the cross-data evaluation scenario, most models yielded comparable performance by layer-freezing and fine-tuning, although both showed generally poorer performance than the pooled-data evaluation scenario. When using layer-freezing, scGPT outperformed others in tumor tissue data, while under fine-tuning, UCE consistently delivered the best performance across various tissue types, drug categories, cancer types, and regimens. A model recommendation guide (Supplementary Data 5) and a detailed analysis of model characteristics (Supplementary Data 6) are provided to support model selection for user applications.

Few-shot learning for drug response prediction
While single-cell FMs such as scFoundation15, UCE24, and scGPT16 demonstrated strong performance in predicting drug responses from single-cell transcriptomic data, we also included a general-purpose foundation model, GPT4o-mini, in the scDrugMap and investigated its applicability in this context. Unlike scFMs specifically pre-trained on single-cell or biomedical data, GPT4o-mini was developed for general natural language tasks. We aimed to assess its zero-shot reasoning capabilities when prompted with single-cell gene expression features for binary drug sensitivity classification (Fig. 5a).

Overall, GPT4o-mini exhibited limited predictive performance, performing near or below baseline levels in most evaluation settings. The model’s highest performance was observed in liver cancer, achieving a mean F1 score of 0.690, which suggests some potential context-specific utility. However, across most other categories, its predictive accuracy was substantially lower. In terms of tissue types, GPT4o-mini achieved mean F1 scores of 0.583 in peripheral blood mononuclear cells, 0.507 in tumor tissue, 0.479 in cell lines, and 0.459 in bone marrow aspirates, highlighting a clear performance gap compared to specialized scFMs. Across drug classes, the model performed best on targeted therapy (mean F1: 0.538), followed by chemotherapy (0.517) and immunotherapy, where performance was notably poor (0.391). When stratified by cancer type, GPT4o-mini’s performance varied widely. Besides liver cancer, the model showed modest performance for chronic lymphocytic leukemia (0.622) and lung cancer (0.594), but struggled on others such as pancreatic cancer (0.190), melanoma (0.320), and breast cancer (0.387) (Fig. 5b).

Additionally, we compared the results using the names of the top 10 highly expressed genes in each cell along with the data source (e.g., GEO accession number) with another two prompting strategies: (1) using the top 100 genes and data source and (2) a chain of thoughts (CoT) prompt, which builds on the original task by adopting a CoT approach, clearly defining the nested data structure (the outer list contains 10 cells, and the inner list contains the 10 highly expressed genes for each cell). This enables the model to follow a predetermined order when assessing drug sensitivity for each cell, helping to reduce ambiguity, improve the clarity and accuracy of reasoning, enhance the consistency and interpretability of results, and provide flexibility for scaling up the task in the future. Across various tissue, drug, and cancer-type datasets, our results showed a consistent pattern: both the top 10-gene and top 100-gene prompts yielded comparable performance based on F1 scores, while the COT prompt achieved relatively higher accuracy in multiple datasets (Supplementary Note 7). These results suggest that, in the absence of domain-specific pretraining or adaptation, general-purpose language models may have limited precision and robustness for drug response prediction in single-cell contexts, highlighting the need for further optimization and domain adaptation.

Modelevaluation in the validation data collection
To further assess model generalizability, we evaluated the performance of foundation models in scDrugMap using a separate validation dataset collection (Fig. 6; Supplementary Fig. 4). For this analysis, we applied layer-freezing training method, where each dataset used 10-fold for model training and testing. Among all models, scFoundation demonstrated the best performance across all categories, including tissue types (mean F1 scores, cell line: 0.901; tumor tissue: 0.932; organoid: 0.973), drug types (mean F1 scores, targeted therapy: 0.946; chemotherapy: 0.899; immunotherapy: 0.915), and cancer types (mean F1 scores, NSCLC: 0.997; ovarian cancer: 0.920; basal cell cancer: 0.918; pancreatic cancer: 0.951; colon cancer: 0.847; hepatocellular cancer: 0.980; melanoma: 0.867; breast cancer: 0.985) (Fig. 6a). When categorized by data source, including cell line, fresh biopsy, organoid, and PDX, scFoundation remains achieved the best performance (Supplementary Note 8). Additionally, the strong performance was further supported by AUROC metrics, where scFoundation again led across tissue types, drug types, and cancer types (Fig. 6b). Other metrics including accuracy, precision, and recall also showed that scFoundation provides accurate predictions in this validation data collection (Supplementary Fig. 4).

In addition to scFoundation, both scGPT and tGPT demonstrated competitive performance in selected categories. For example, across tissue types, they performed well in tumor tissue (mean F1 scores, scGPT: 0.800; tGPT: 0.863) and organoid (mean F1 scores, scGPT: 0.928; tGPT: 0.946) but not in cell line (mean F1 scores, scGPT: 0.559; tGPT: 0.740) (Fig. 6a). Across drug types, the two models performed similarly, with mean F1 scores ranging from 0.714 of scGPT for targeted therapy to 0.874 of tGPT for chemotherapy. In terms of cancer types, CellPLM, scGPT, and tGPT displayed similar performance, performing well across most cancers (mean F1 scores ranged from 0.755 of CellPLM for basal cell cancer to 0.986 of tGPT for NSCLC). However, in ovarian cancer and melanoma, scGPT showed lower performance compared to the other two models. Specifically, in ovarian cancer, scGPT achieved a mean F1 score of 0.578, while CellPLM and tGPT reached 0.700 and 0.753, respectively. For melanoma, scGPT’s score was 0.589, compared to 0.774 for CellPLM and 0.766 for tGPT.

Computational scalability
In addition to prediction accuracy, scDrugMap also assesses the computational scalability of each foundation model by comparing their architectures, parameter counts, output dimensions, and runtime efficiency (Fig. 7). These factors are critical for selecting models that balance performance with practical deployment needs in large-scale single-cell analysis.

For example, scFoundation, using an encoder-decoder architecture, stood out with its large parameter count (121.2 million), high output dimension of 3072, and efficient training and inference speeds (23.26 it/s and 69.98 it/s, respectively), making it one of the most efficient and powerful models in our benchmark. In comparison, scGPT, which also adopts a dual-architecture approach, had fewer parameters (52.5 million) and a smaller output dimension (512), with lower efficiency (1.43 it/s for training; 1.44 it/s for inference) than scFoundation. Another encoder-decoder model, UCE, offers competitive inference and training speeds, though it has a smaller output dimension of 1280. On the other hand, models like tGPT (decoder-only), scBERT (encoder-only), and Geneformer (encoder-only) used a single architecture and are relatively less efficient, with lower output dimensions (1024 for tGPT, 200 for scBERT, 256 for Geneformer) and slower training and inference speeds. Taken together, these results highlight important trade-offs between architectural complexity, model size, and computational performance. By incorporating these metrics, scDrugMap enables users to make informed decisions about model selection based not only on accuracy but also on runtime efficiency and resource constraints, tailoring deployment to the specific demands of different single-cell datasets and drug response tasks.

Methods
Overview of scDrugMap framework for drug response prediction
In the scDrugMap framework, we identified recently developed single-cell-related foundation models (scFMs) for drug response prediction. In total, we selected and evaluated ten FMs, comprising eight scFMs and two general-purpose natural language models. The scFMs evaluated in this study include transcriptome-GPT34 (tGPT), single-cell Bidirectional Encoder Representations from Transformers14 (scBERT), Geneformer17, Single-Cell Language Model22 (CellLM), single-cell Foundation Model15 (scFoundation), single-cell Generative Pre-Trained Transformer16 (scGPT), Cell Pre-training Language Model23 (CellPLM), and Universal Cell Embeddings24 (UCE). The two natural language models are Llama3-8B25 and GPT4o-mini26.

Model evaluation settings
In scDrugMap, we evaluated the FMs in two scenarios: the pooled-data scenario and the cross-data scenario. In the pooled-data scenario, datasets from multiple studies within the same category (e.g., tissue type, drug, regimen, or cancer type) were combined into a single dataset. This pooled dataset was then randomly split into training and test sets, allowing the model to learn and be evaluated on data drawn from the same mixed source. In contrast, the cross-data scenario tested the model’s ability to generalize across studies. The model was trained on data from one or more studies and evaluated on data from a completely separate study. In this setting, training and testing were performed using data from different studies, while still remaining within the same category (e.g., tissue type, drug, regimen, or cancer type). This setting can be considered a zero-shot capability evaluation of the model, which better reflects real-world conditions where models are often applied to new, unseen datasets.

Model training strategies
For both evaluation scenarios, we applied two training strategies: layer-freezing and LoRA35 (Low-Rank Adaptation). These strategies were used to train the foundation models (FMs) on the training sets and evaluate their performance on corresponding test sets. For LLaMa3-8B, only the layer-freezing strategy was applied. For GPT4o-mini, we employed a prompt engineering approach similar to that used in GPTCelltype19, without additional fine-tuning.

In the layer-freezing strategy, pretrained scFMs treat each gene (by ENSEMBL ID or gene symbol) as an individual token within a fixed vocabulary (dictionary). To extract feature embeddings for downstream classification, we averaged all token embeddings generated by the scFM. These aggregated embeddings were then passed to a multi-layer perceptron (MLP) for classification. The MLP structure used for classification was defined as follows:

h1 = ReLU(W1x + b1) (1)

h2 = ReLU(W2h1 + b2) (2)

y = Sigmoid(W3h2 + b3) (3)

where x ∈ Rd is the embedding representation produced by the scFM, d is the model-specific output dimension, and y is the predicted probability of drug sensitivity. ReLU and Sigmoid are the activation functions, and Wi, bi represent weights and biases of fully connected layers. We used 10-fold cross-validation, randomly partitioning the data within each biological category (e.g., tissue type, drug class, cancer type, or treatment regimen). The same data partitions were used consistently across models. Only the MLP parameters were trained, while the foundation model weights remained frozen.

In the fine-tuning strategy, we fine-tuned a subset of the model parameters using LoRA, implemented via the peft36 package, while keeping the rest of the model frozen. LoRA enables efficient adaptation of large models by injecting low-rank matrices into trainable layers. We set the LoRA rank to 8, alpha (scaling) to 8, dropout to 0.05, and defined the task type as “SEQ_CLS” (sequence classification). We also conducted a ablation study to evaluate of different LoRA ranks (4, 8, and 16) on the fine-tuning performance of scLLMs for drug response prediction (Supplementary Note 10). The output embeddings from the partially fine-tuned model were again passed through the MLP classifier, and both the MLP parameters and LoRA-injected parameters were trained using backpropagation. Let a pretrained single cell foundation model be denoted as scFM(D, W), where W = {Wc, Wft}, with Wc representing frozen weights and Wft representing the trainable layers. Given a fine-tuning dataset Dft = {xi, yi}ki=1, the objective is to minimize the binary cross-entropy loss:

W*ft = arg min θft E(x,y)∈Dft L(y, y) (4)

y = scFM(x; Wc, Φft, Wft) (5)

where Φft are the parameters of the MLP classifier, and L is the binary cross-entropy loss.

In LoRA, weight updates are expressed as a low-rank approximation:

W'ft = Wft + ΔW = Wft + AB (6)

where A ∈ Rd×r and B ∈ Rr×k are trainable low-rank matrices, and r is the rank (set to 8). Here Wft remains frozen and ΔW is learned during fine-tuning.

tGPT
tGPT takes gene expression rankings for generative pretraining on 22.3 million single-cell transcriptomic data. Gene sequence are sorted in descending order of expression, enclosed with special tokens <start> and <end>, and padded with <pad>. We followed instructions from the official tGPT GitHub repository (https://github.com/deeplearningplus/tGPT). For benchmarking, we used the pretrained model ‘transcriptome-gpt-1024-8-16-64’, which is stored in https://huggingface.co/lixiangchun/transcriptome-gpt-1024-8-16-64. The input sequence length of tGPT is 64 and the output embedding dimension is 1024. For layer-freezing training, we average the output of the last layer in the sequence dimension as the input of the downstream classifier. The target module Wft for LoRA fine-tuning is the attention layer of Transformer block called ‘c_attn’.

scBERT
scBERT uses gene2vec technology to embed genes into a pre-defined vector space, reflecting the similarity between genes and simplifying model training14. To effectively utilize the transcription levels of each gene, the model discretizes gene expression and converts it into a 200-dimensional vector, similar to the word frequency analysis method in NLP, which serves as the token input for the model. Moreover, scBERT adopts the Performer architecture to mitigate limitation of input length and model efficiency. Though scBERT can handle very long input sequences, we chose to use the first 8000 tokens to maintain uniformity in batch operations. For drug response prediction, we used the pre-trained model provided at https://github.com/TencentAILabHealthcare/scBERT. For the choice of layer-freezing embedding, we keep it consistent with the original operation of scBERT and average the output of the last layer in the sequence dimension. LoRA fine-tuning Wft includes the key (‘to_k’), value (‘to_v’) and query (‘to_q’) matrices of the model.

Geneformer
Geneformer consists of six standard Transformer encoder blocks. The model features an input size of 2048, an embedding dimension of 256, four attention heads per layer, and a feedforward layer of size 512. Geneformer is pre-trained on 30 million cells and released two versions of the pre-trained model. For this drug response prediction task, we used the pre-trained model named as ‘geneformer-6L-30M_CellClassifier_cardiomyopathies_220224’, which is available and can be obtained from https://huggingface.co/ctheodoris/Geneformer/tree/main. For the input, we follow the same approach as Geneformer by first sorting gene expression values in descending order and then retaining the top 2048 genes. For drug response prediction using fixed-layer embeddings, we utilized the representations extracted by the ‘hidden_states’ output of the fine-tuned model, and the target module Wft for LoRA fine-tuning is the attention layer of Transformer block, specifically the key (‘key’), value (‘value’), and query (‘query’) matrices.

CellLM
Traditional methods rely on the BERT architecture, which leads to anisotropy in the embedding space and inefficient semantic representation. CellLM addresses the computational limitations caused by large batches of data through a new divide-and-conquer contrastive learning method, thereby improving the quality of representation. Trained on 2 million scRNA-seq data, CellLM is the first large language model that learns from both normal cells and cancer cells. To ensure batch consistency, we also limit the max input length to 8000. We choose to take the average of the hidden states output by the encoder in the sequence dimension as the layer-freezing embedding. CellLM utilizes a Performer structure similar to scBERT, so the components involved in Wft are consistent with scBERT, specifically the key (‘to_k’), value (‘to_v’) and query (‘to_q’) matrices of the model. The pre-trained model for CellLM can be accessed at https://github.com/PharMolix/OpenBioMed/tree/main.

scFoundation
scFoundation has more than 100 million parameters and is trained on over 50 million human single-cell transcriptome data, covering the complex molecular features of all known cell types15. In addition to gene expression values, the input of scFoundation includes two additional indicators: ‘S’ (source) and ‘T’ (target), which represent the input value and the total gene expression, respectively. scFoundation adopts an encoder-decoder design, using only non-zero genes during the encoding stage, while both expressed and non-expressed genes are considered during the decoding stage. We used the pretrained model available at https://github.com/biomap-research/scFoundation. For fixed-layer drug response prediction, the cell embedding was constructed by concatenating four components, as described in the original paper: the representation of ‘S’ and ‘T’, along with the maximum and average embeddings of all genes. LoRA fine-tuning was applied to the model’s output layer.

scGPT
scGPT is pre-trained on over 33 million single-cell sequencing data and is further optimized through transfer learning16. For its input, scGPT not only uses information about genes and expression values, but also introduces the special Condition Tokens. The Condition Tokens encompass diverse meta information associated with 502 individual genes, such as modality, batch, and perturbation conditions. Additionally, scGPT can handle unknown genes to address the Out of Vocabulary problem. The model can be accessed at https://github.com/bowang-lab/scGPT. The cell embedding used for layer-freezing based drug response prediction is the <CLS> token (the token defined before pre-training, its representation will be trained during the training phase, and is usually added as the first token in the sequence), which can represent the entire sentence. For LoRA fine-tuning, it targeted the output layer (‘out_proj’).

CellPLM
CellPLM addresses the differences between single-cell data and natural language data that are often overlooked by existing models. Unlike traditional methods that treat genes as tokens and cells as sentences, CellPLM treats cells as tokens and tissues as sentences, using spatial transcriptome data during pre-training to better learn the relationships between cells. They consider the embedding at l-th layer of Transformer as a set of N tokens, where N is the total number of cells in a tissue sample. Additionally, CellPLM introduces a Gaussian mixture prior distribution to mitigate issues of insufficient data and noise. The pre-trained model we used is ‘20230926_85M’ from https://github.com/OmicsML/CellPLM. The cell embedding used for layer-freezing drug response prediction is the output of the last layer in CellPLM. LoRA fine-tuning includes the query, key, and value (‘query_projection’, ‘key_projection’, and ‘value_projection’) matrices in the CellPLM model.

UCE
UCE provides a unified biological latent space through fully self-supervised training with single-cell transcriptome data24. The UCE is capable of capturing significant biological changes and can map any new cell to this embedding space without additional data annotation, model training, or fine-tuning. UCE employs an encoder-decoder structure and performs specific processing on the input, such as adding special tokens (e.g., the CLS token is designed to capture cell-level embeddings when training the model). We used the UCE model provided at https://github.com/snap-stanford/UCE. For layer-freezing drug response prediction, the embeddings was taken from the <CLS> token (the first token in the sequence), as provided by UCE. For LoRA fine-tuning, the output layer out_proj was designated as the target module.

LLaMa3-8B
In addition to the above-mentioned scFMs, we also tested the performance of general FMs in this task, specifically using the open-source LLM released by Meta, Llama-3-8B. Llama-3-8B has nearly 8 billion parameters and is designed to provide efficient natural language understanding and generation capabilities. It is an upgraded version of the Llama series, capable of handling more complex language tasks while optimizing inference speed and resource efficiency. For input processing, to align with the methods used by most scFMs, we first arranged the genes in the sequence from high to low based on expression levels. We then limited the input length to 1024 genes, treating these gene symbols as natural language (separated by space) and used Llama-3-8B’s tokenizer for tokenization. Finally, the embedding output by Llama-3-8B was used for layer-freezing drug response prediction. The model can be obtained by applying for authorization from Meta and downloading it from https://huggingface.co/meta-llama/Meta-Llama-3-8B.

GPT4o-mini
Inspired by GPTCelltype19, we also tested the latest generation of FMs, GPT4o-mini, recently released by OpenAI. It contains knowledge up to July 2024. Like GPTCelltype, we use prompt words to guide the FM’s responses. The specific settings for the model input are as follows:

input = prompt_pre + source + prompt_gene + content + prompt_post

The input consists of 4 prompt modules, source, and content. The description of each prompt module is shown in Fig. 5a. The source is the data source of the current single-cell sequence. We hope to help obtain accurate results by prompting the FM with data source. The content is the name of 10 genes with the highest expression in the single cell sequence. To avoid the problem of hallucination caused by too long input in the large language model, we use the mini batch technique to only let the FM answer 10 sequences in each context [questions on Fig. 5a]. We also evaluated another two prompts: top 100 genes in each cell and data source, and (3) a chain of thoughts (CoT) prompt which builds on the original task by adopting a CoT approach, clearly defining the nested data structure (the outer list contains 10 cells, and the inner list contains the 10 highly expressed genes for each cell). This enables the model to follow a predetermined order when assessing drug sensitivity for each cell, helping to reduce ambiguity, improve the clarity and accuracy of reasoning, enhance the consistency and interpretability of results, and provide flexibility for scaling up the task in the future (Supplementary Note 7). Our experiments were conducted by calling the GPT4o-mini’s API provided by OpenAI (https://openai.com/api).

Data collection and preprocessing
To systematically collect scRNA-seq data, we conducted a literature search in PubMed for studies published up until 2024, using the keywords ‘(drug resistance) AND ((single cell) OR (scRNA)’. Only samples from Homo sapiens with annotated drug response information were included. We also included available datasets collected on drug resistance mechanisms37. This yielded a total of 36 scRNA-seq datasets as the primary data collection, encompassing 11 cancer types, three treatment types, and four tissue types (Supplementary Data 1). To construct an external validation collection, we followed the same curation methodology, focusing on studies published since January 2024. This resulted in 24 scRNA-seq datasets for drug response studies on solid tumors covering five cancer types, three treatment types, and three tissue types (Supplementary Data 1). Both the primary and validation collections included experimental annotations of binary drug response status (sensitive/resistant) at cell-level (Supplementary Data 2). The validation collection was incorporated to further evaluate model generalizability. In addition to the standard train–test split approach (Fig. 6), we also assessed the models in a zero-shot learning setting, directly predicting drug response labels without any retraining (Supplementary Note 11), which suggest that the models can generalize to entirely independent datasets. General natural language models such as GPT5, GPTo4-mini-deep-research and GPT4o-mini were assessed for their zero-shot reasoning capabilities when prompted with single-cell gene expression features for binary drug response prediction (Supplementary Note 12).

Specifically, cells of the nonresponsive samples collected from pre- or post-treatment conditions are labeled as resistant cells, while cells of the responsive samples are labeled as sensitive cells. Quality control and preprocessing were conducted using Seurat v4.3.038 with uniform filtering thresholds (nFeature_RNA > 500 and percent.mt < 10%) across all datasets. Although mitochondrial content may vary across tissue types and sequencing technologies, standardized preprocessing was applied to ensure fair and consistent evaluation of model performance. The top 30 principal components (PCs) were then used to cluster the cells at a resolution of 0.5. For datasets exhibiting batch effects, we used Harmony package to correct for them39.

Evaluation metrics
We used four evaluation metrics (F1, AUROC, precision, and recall) for assessing model performance in predicting cell-level drug response. Accuracy is the proportion of correctly predicted samples out of the total number of samples and is most appropriate when the class distribution is balanced. F1 score is the harmonic mean of Precision and Recall, making it suitable for situations where class distribution is imbalanced. F1 can be expressed as: Precision = TP / (TP + FP), Recall = TP / (TP + FN), F1 Score = (2 × Precision × Recall) / (Precision + Recall), where TP (True Positive) represents the number of samples correctly predicted as positive, while TN (True Negative) denotes the number of samples correctly predicted as negative. FP (False Positive) refers to the number of samples incorrectly predicted as positive, indicating a false alarm, and FN (False Negative) is the number of samples incorrectly predicted as negative, signifying a missed detection. AUROC represents the area under the Receiver Operating Characteristic (ROC) curve, which is used to evaluate the discriminatory ability of a classification model. The ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR), where TPR = TP / (TP + FN), FPR = FP / (FP + TN), AUROC = ∫ 0 1 TPR(FPR) dFPR. TPR (True Positive Rate) is also known as Sensitivity or Recall. FPR is False Positive Rate. The closer the AUROC value is to 1, the better the model’s ability in predicting accurate drug response.

Web server implementation
We developed scDrugMap as a web-based system for drug response prediction and benchmarking, built on a robust three-tier architecture consisting of the client, server, and database layers. The backend database was implemented using MySQL version 3.23 with the MyISAM storage engine, selected for its simplicity and fast read performance, which suits the platform’s analytical needs. The server layer was constructed using Perl and its DBI (Database Interface) module, enabling efficient interaction with the MySQL database for dynamic content retrieval and user-driven queries. The client layer provides an intuitive, user-friendly web interface designed to facilitate access to curated datasets, model results, and visualization tools. Through this architecture, scDrugMap allows users to seamlessly explore drug response predictions, compare the performance of foundation models, and interact with large-scale single-cell datasets, supporting translational research in drug discovery and precision oncology.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Cancer drug response prediction is a challenging problem due to cross patient heterogeneity and variability of the\neffectiveness of the drug itself. The recent development of several large foundation models catering to single cell\ntranscriptomics has enabled drug resistance prediction at a cellular level. These models are trained on massive single cell\ndatasets – in the order of several million cells – and are then finetuned on smaller targeted datasets. However, no systematic\nbenchmarking of these models exists.\nThe authors bridge this gap in their work – scDrugMap. The major contributions of their paper are as follows –\n1. They curate a primary dataset consisting of 326,751 cells from 36 single cell datasets spanning 23 separate studies and a\nsecondary validation dataset consisting of 18,856 cells from 17 datasets spanning 6 studies.\n2. They benchmark 10 different foundation models, of which 8 are single-cell-specific models while the other two are general\nnatural language models. They perform the benchmarking under two different scenarios – pooled data and cross-data\nevaluation.\n3. They implement two different model training strategies – layer freezing and fine-tuning using low rank adaptation.\n4. They provide a command line tool and an interactive web server that can be used for dataset specific model selection and\nevaluation.\nMajor comments\n1. The paper would benefit from a more detailed analysis of the results. While the authors mention F1 score for best and\nworst performing models in various scenarios, a deeper discussion is missing. For example, the authors note that in\npancreatic and prostate cancer, LLaMa3 – a general natural language model matches performance with scFoundation\nmodel (F1 scores 0.913 and 0.990, respectively). However, these results are extremely surprising since both prostate and\npancreatic cancer datasets have one of the smallest number of cells reported. My guess is that this anomalous behavior\narises because of limited variability in the data (the data comes from a single study), combined with imbalanced class ratio\nfor the two cancer types – the resistant to sensitive ratio is approximately 80/20 for both types and represents the most\nextreme imbalance in the datasets being analyzed. This leads me to believe that the model is overfitting and will struggle to\ngeneralize to unseen data.\n2. The cross-data evaluation is the more relevant metric here, as recognized by the authors. Cross-data evaluation is a good\nindication of generalizability of the trained models and is understandably significantly worse compared to the pooled-data\nevaluation. One thing to check in the cross-data evaluation would be the generalizability across extraction and data\nprocessing protocols. For example, how does a model trained on data extracted using 10X protocol perform on a test data\nextracted with a different protocol. Further, can such a shift in results be corrected in a systematic manner?\n3. A comparison of the results with the existing non foundation model-based methods would be useful. While some of these\ncontemporary methods that the authors have cited in the discussion section might be trained on domain specific data, it will\nstill be good to put the performance of a foundation model based approach in context of other existing methods. In fact, a\nsimple comparison of the MLP classifier the authors use with the embeddings obtained from a foundation model against an\nMLP classifier operating directly on normalized expression values can give some idea of the improvement attributable to the\nfoundation models themselves.\n4. There is a large variability in the performance of various models applied to different data sources. For example, Fig 3\nshows that scFoundation model performance varies more than any other model. scFoundation model is the second largest\nin terms of parameters with ~120M parameters and the largest in pre-training corpus with ~50M cells. What might be the\ncause of such variability? For instance, is it difficult to finetune such a large model. Such discussions would be useful\nbecause the authors want their work to serve as a tool to make informed decisions about which foundation models to use.\n5. Validation datasets seem to have been used in a non-standard way. My understanding is that usually the validation\ndatasets should be held out and not used in training at all. However, the authors have performed a train-test split on the\nvalidation datasets to train models, independently of models already trained-tested on the primary data. Is there a particular\nreason for such a choice?\nMinor comments\n1. While F1 and AUROC are relevant metrics, the authors should consider reporting Pearson/Sprearman correlations\nbetween known and predicted responses as well.\n2. In figure 3, since two methods are being compared, it would make sense for AUROC plots to either both be violin plots or\nbox plots.\n3. A case study or an example of a drug response predicted by the best performing model would be an interesting addition to\nthe paper.", "reviewer_2": "This manuscript presents scDrugMap, a comprehensive benchmarking framework for evaluating 10 foundation models (8\nsingle-cell-specific, 2 general LLMs) in predicting drug response from single-cell transcriptomics. The study curates 345,607\ncells across 53 datasets spanning 14 cancer types, 3 therapies, and 5 tissues. The authors demonstrated that scFoundation\ndominates pooled-data prediction, while UCE and scGPT lead in cross-data generalization; fine-tuning universally\noutperforms layer-freezing, general LLMs (e.g., GPT4o-mini) fail in zero-shot tasks, and computational efficiency varies\nsharply (scFoundation ≫ scGPT).\nIn the context of growing tumor single-cell data and proliferating LLMs, scDrugMap establishes the first benchmark for\nfoundation models in drug response prediction. It validates specialized single-cell FMs over general LLMs for clinical\nreliability, provides scenario-specific model selection guidelines, and quantifies computational trade-offs—directly enabling\nmore accurate, deployable treatment response prediction in precision oncology. The study's claims of novelty are\nundermined by its exclusive reliance on existing models without algorithmic innovation, and its benchmarking framework\nclosely mirrors prior approaches. Data integration lacks originality, repurposing public GEO datasets without novel\nexperimental validation. Crucially, the work fails to cite key literature (e.g., GPTCelltype for LLM benchmarking; DREEP for\ndrug response prediction), overstating its \"first comprehensive benchmark\" contribution while overlooking established\nmethodologies in adjacent domains.\nMajor Concerns:\n1. Questionable novelty. All models are repurposed (scGPT, Geneformer). Fails to cite comparable benchmarks (e.g.,\nGPTCelltype, Nat Methods 2024). Positions as \"first\" without literature context.\n2. Lack of baseline comparisons. The study does not include comparisons with established non–foundation model (non-FM)\nmethods such as DREEP or scDEAL, which limits the ability to assess the added value of the proposed foundation models.\nWithout such benchmarks, it remains unclear whether the FMs truly offer performance improvements. If the FMs\nunderperform relative to these simpler or more established approaches, the core premise of the study—that foundation\nmodels provide superior predictive power—would be significantly weakened.\n3. Overinterpretation of marginal results. UCE’s \"top\" cross-data performance (F1=0.774) is clinically irrelevant. Claiming it\n\"achieves optimal results\" (Fig 3b) misrepresents data. Similarly, scFoundation’s failure in PBMCs (F1=0.461) is glossed\nover.\n4. Data biases compromise conclusions. Severe class imbalance (e.g., 86% resistant cells in neuroblastoma; Supp Table 2)\ninflates F1 scores. AUPRC should supplement F1 for imbalanced subgroups. Batch effects across 36 datasets are ignored.\nNo correction (e.g., Harmony) was applied, invalidating cross-dataset claims.\n5. Inadequate validation. \"Validation collection\" uses same data sources (GEO) as primary data. Independent clinical\ncohorts (e.g., fresh biopsies, PDX models) are absent.\n6. Methodological opacity. Preprocessing: Gene/cell QC thresholds (e.g., mitochondrial content) undefined; LoRA tuning:\nRank=8 used universally without ablation study; Prompt engineering: GPT4o-mini’s poor performance may stem from\nsuboptimal design (only top 10 genes input).\n7. Authors used four evaluation metrics (F1, AUROC, precision, and recall) for assessing model performance in predicting\ncell-level drug response. However, relying solely on these metrics may not provide a sufficiently rigorous basis for model\ncomparison. It is recommended to add statistical tests (e.g., p-values, confidence intervals) to determine whether the\nperformance differences between models are significant.\n8. In the \"Few-shot learning for drug response prediction\" section, the evaluation of few-shot/zero-shot learning is relatively\nbrief, without a comprehensive analysis, for instance, of whether different prompt templates influence prediction\nperformance. Furthermore, the lack of comparison with similar methods makes it difficult to assess the potential or limitations\nof large language models in drug response prediction.\n9. The manuscript lacks a thorough analysis of biological mechanisms behind model's prediction, whether the model’s\ndecisions depend on key genes, cell subpopulations, drug mechanisms, or other biological features.\nMinor Concerns:\n1. The study omits critical evaluation of technical confounders, including sequencing platform variability, read depth\nheterogeneity, and batch effect correction methodologies. This analytical gap precludes definitive attribution of performance\ndifferences to model capability versus data artifacts, thereby limiting assessment of real-world applicability.\n2. The manuscript contains spelling errors (such as \"cance\", \"componennts\"), inappropriate phrasing (e.g., \"enhance\nlimitation\"). Additionally, some paragraphs contain repetitive sentence patterns, which should be revised for conciseness\nand clarity.\n3. Several figures show missing or unclear information. In Fig.1 and Fig.6, legends are incomplete (e.g., unclear what color\nshades or line colors represent). Fig.2 and Fig.3 lack stylistic consistency and statistical significance markers. In Fig.4c,\nthe UCE group fails to distinguish between resistant and sensitive cells.\n4. Line 102 contains an incorrect attribution to scFoundation.\n5. In Figure 1C-D, the 'Number of cells' bar charts employ graduated color shading within categorical segments (e.g.,\ntissue/drug/cancer types). The manuscript does not define what these intensity variations represent—whether hierarchical\nsubstructures, response status distributions, or data source heterogeneity.\nClarification should be provided in the figure legend or methods to ensure interpretability.\nWeb server Concerns:\n1. The \"README\" part is insufficient and requires clear, step-by-step instructions for using the tool.\n2. In the \"Tool\" section, users can upload all input CSV files. It would be helpful to provide a sample file to clarify the\nexpected forma\n3. Some single-cell drug resistance datasets are available for download in \"Data\" section. To improve interpretability,\nvisual summaries (UMAP/t-SNE) are recommended. Also, the download button beside the dataset is currently not working.", "reviewer_3": "The manuscript presents scDrugMap, a benchmarking framework evaluating large foundation models for single-cell drug\nresponse prediction. Here are the detailed comments：\n1. While the paper appropriately notes that accuracy is suitable for balanced data (lines 502-503) and confirms the drug\nsensitivity/resistance data is balanced (lines 119-121, 129-130), there appears to be an inconsistency as F1 scores (more\nappropriate for imbalanced data, lines 503-504) are predominantly reported. Furthermore, although accuracy is repeatedly\nmentioned as being tested (lines 143-145, 160-161, 240-242), the results figures lack this metric, creating a discrepancy with\nthe Results section.\n2.While the paper provides a comprehensive comparison of model performance across various scenarios using metrics like\nF1, the underlying reasons for the observed performance differences (e.g., pretraining strategy, model architecture, or\ndataset characteristics) remain unexplored; a deeper analysis of these factors would significantly strengthen the mechanistic\nunderstanding of why certain models outperform others in specific contexts, ultimately providing more actionable insights for\nreaders seeking to select or develop appropriate models for their applications.\n3.The statement that \"other models showed limited separation\" in Fig. 4 would be more informative if accompanied by\ncorresponding UMAP visualizations of these underperforming models, as this would enable readers to directly compare the\nseparation quality across all evaluated methods and better understand the performance differences.\n4.The choice of LoRA rank=8 lacks justification. A brief ablation study (e.g., testing ranks 4/8/16) would strengthen the\nmethodology.\n5.Please add a summary table (e.g., \"Model Recommendation Guide\") to highlight optimal models for specific scenarios\n(e.g., UCE for generalizability). This would improve practical utility.\n6.The triplicate repetition of post-prompt in Fig. 5a appears redundant. Please clarify whether this design is necessary.\n7.Several critical aspects of the web interface remain unclear, significantly limiting its usability and reproducibility.\nSpecifically: (1) The required format and structure of uploaded CSV files are not documented. (2) The model selection and\nfine-tuning configuration interface lacks clarity—for example, whether users can choose between freezing layers or enabling\nfull fine-tuning. (3) There is no guidance on how to set key parameters such as LoRA rank. (4) It is unclear how and where\nusers can download the fine-tuned models. (5) The functional distinction between the “LLM TOOLS” and “DATA\nBROWSER” tabs is ambiguous, as both appear to redirect to the same “Data” page."}, "author_response": "Response: Thanks for the comments. We agree with the reviewer that the anomalous results observed for\nLLaMa3 and scFoundation on certain small and imbalanced datasets warrant further clarification. While these\nresults may initially appear to be dataset-specific artifacts, we believe their relatively strong performance is\nnot confined to prostate and pancreatic cancer datasets. Instead, it reflects a broader trend tied to the pooled-\ndata evaluation scenario.\nIn pooled-data evaluations, models consistently performed better because pooling integrates samples\nfrom multiple studies within the same category (e.g., tumor tissue). This integration increases both the\ndiversity and representativeness of the training data, exposing the models to a wider spectrum of biological\nstates, experimental conditions, and patient heterogeneity. Such diversity strengthens generalization and\nreduces sensitivity to dataset-specific noise. Moreover, pooled-data evaluation aligns closely with the\npretraining strategies of single-cell foundation models, which are trained on heterogeneous datasets. When\npresented with similarly diverse inputs, these models are better able to leverage their pretrained embeddings\nand capture robust predictive signals.\nBy contrast, in cross-data evaluations, models are trained on one dataset and tested on another, forcing\nthem to generalize across different experimental protocols, technical platforms, or patient populations. These\ndistributional shifts present significant challenges, as the narrower training context provides fewer cues for\nadaptation. As a result, performance is often substantially lower in cross-data settings compared to pooled-\ndata settings.\nTo further support this interpretation, we extended our analysis to two additional severely imbalanced\ndatasets. In Data 23 (lung cancer), resistant cells dominated the dataset (768 cells; 84.5%) compared to\nsensitive cells (141 cells; 15.5%). Despite this imbalance, pooled-data evaluations consistently outperformed\ncross-data evaluations. For example, top-performing models achieved high AUPRC values: scFoundation\n(0.894), scGPT (0.802), and CellPLM (0.709) (Supplementary Note 9; Supplementary Fig. 22). Similarly, in\nData 32 (prostate cancer), where sensitive cells represented only 20.2% of the dataset, pooled-data models\nagain surpassed cross-data models, with scFoundation, scGPT, and CellPLM ranking among the strongest\nperformers. These results demonstrate that the superior performance of LLaMa3 and scFoundation is not\nrestricted to specific cancer types. Instead, it reflects a more general advantage of pooled-data training in\ncontexts where heterogeneity and imbalance are prominent.\nImportantly, we want to emphasize that these results are unlikely to be explained by overfitting. Our\nevaluation framework was designed to rigorously minimize this risk. Specifically, we employed a 10-fold\ncross-validation scheme in which folds were generated at the sample level, ensuring that no cell-level overlap\noccurred between training and test sets. This design prevents the possibility of information leakage across\nfolds and enforces a strict separation between data used for training and evaluation. As such, the reported\nperformance metrics represent robust model behavior, rather than spurious gains from overfitting.\nTo help the community translate these findings into practice, we developed a Model Recommendation\nGuide (Supplementary Table 5). This guide synthesizes the optimal foundation models identified across four\nevaluation conditions: (1) pooled-data with layer-freezing, (2) pooled-data with fine-tuning, (3) cross-data\nwith layer-freezing, and (4) cross-data with fine-tuning. By mapping model performance to evaluation\nscenario, tissue type, drug class, and cancer type, the guide provides a practical framework for selecting or\ndeveloping models for specific applications. In general, scFoundation and scGPT emerge as strong default\nchoices for pooled-data applications across most tissues, drug classes, and cancer types. In contrast, cross-data\nscenarios require more careful model selection, as performance is more variable. Models such as UCE, tGPT,\nand task-specific architectures often show stronger generalization in these settings. Finally, for rare cancers or\nseverely imbalanced datasets, we recommend researchers pay particular attention to cross-data results, since\nnot all models generalize reliably under these challenging conditions.\n2. The cross-data evaluation is the more relevant metric here, as recognized by the authors. Cross-data\nevaluation is a good indication of generalizability of the trained models and is understandably significantly\nworse compared to the pooled-data evaluation. One thing to check in the cross-data evaluation would be the\ngeneralizability across extraction and data processing protocols. For example, how does a model trained on\ndata extracted using 10X protocol perform on a test data extracted with a different protocol. Further, can such\na shift in results be corrected in a systematic manner?\nResponse: Thanks for highlighting the importance of evaluating model generalizability across protocols and\ndata processing pipelines. In the revised manuscript, we addressed this by explicitly testing model\nperformance across six distinct single-cell RNA-seq protocols and technologies represented in our collection:\n10x Genomics (n=27), BD Rhapsody (n=3), MARS-seq v2.0 (n=2), DropSeq (n=2), inDrop v3 (n=1), and\nSMART-seq v4 (n=1) (Supplementary Table 1). To assess cross-protocol generalizability, we also applied\ntwo training strategies (layer freezing, fine-tuning with LoRA) and compared model performance across\nprotocols.\nOur results showed that when using the layer-freezing training strategy, scFoundation consistently\nachieved the best performance across all six protocols based on F1 scores (BD Rhapsody: 0.989; MARS-seq\nv2.0: 0.963; 10x Genomics: 0.975; DropSeq: 0.952; inDrop v3: 0.937; SMART-seq v4: 0.916)\n(Supplementary Note 6; Supplementary Fig. 19a). Other metrics, including accuracy and AUROC, showed\nconsistent patterns, further supporting scFoundation's overall superiority (Supplementary Note 6;\nSupplementary Fig. 19a). However, when considering precision, scFoundation remained the best in most\nprotocols, except for DropSeq, inDrop v3, and SMART-seq v4, where CellLM achieved perfect precision\n(mean precision = 1.000) (Supplementary Note 6; Supplementary Fig. 19a). In SMART-seq v4, scBERT\nalso achieved perfect precision (mean = 1.000). Similarly, for recall, scFoundation performed best overall, but\nCellLM and scBERT outperformed scFoundation in the BD Rhapsody and MARS-seq v2.0 datasets.\nSpecifically, CellLM and scBERT achieved near-perfect or perfect recall in these protocols (BD Rhapsody:\nCellLM = 1.000, scBERT = 0.999; MARS-seq v2.0: CellLM = 1.000, scBERT = 0.993) (Supplementary\nNote 6; Supplementary Fig. 19a).\nUnder fine-tuning scenario, we observed similar overall patterns, with scFoundation remaining the\ntop-performing model across all six protocols based on F1 scores (BD Rhapsody: 1.000; MARS-seq v2.0: 0.989;\n10x Genomics: 0.958; DropSeq: 0.893; inDrop v3: 0.749; SMART-seq v4: 0.654) (Supplementary\nNote 6; Supplementary Fig. 19b). Interestingly, compared to the layer-freezing method, fine-tuning\nimproved scFoundation's performance in the BD Rhapsody and MARS-seq v2.0 datasets by 1.1% and 2.6%,\nrespectively. However, in the remaining four protocols, fine-tuning led to decreased performance, with\ndeclines of 1.7% in 10x Genomics, 5.9% in DropSeq, 18.8% in inDrop v3, and 26.2% in SMART-seq v4.\nThis may due to overfitting or loss of generalizable features in protocols with higher technical variability.\nOther metrics, including accuracy, AUROC, and recall, showed consistent patterns, further confirming\nscFoundation's overall superiority (Supplementary Note 6; Supplementary Fig. 19b). In terms of precision,\nscFoundation again ranked first in most protocols, except for inDrop v3 and SMART-seq v4, where CellPLM\nachieved perfect precision in inDrop v3 (mean precision = 1.000), and tGPT achieved perfect precision in\nSMART-seq v4 (mean precision = 1.000) (Supplementary Note 6; Supplementary Fig. 19b). We also note\nthat fine-tuned Geneformer consistently ranked the second best model in five out of six protocols, with F1\nscores of 0.981 (BD Rhapsody), 0.919 (MARS-seq v2.0), 0.849 (10x Genomics), 0.850 (DropSeq), and 0.597\n(inDrop v3), except for SMART-seq v4. Despite this, overall model performance remained low in inDrop v3\nand SMART-seq v4, with mean F1 scores ranging from 0 (scBERT) to 0.749 (scFoundation) in inDrop v3,\nand 0 (scBERT) to 0.654 (scFoundation) in SMART-seq v4 (Supplementary Note 6; Supplementary Fig.\n19b).\nIn summary, our results suggest that scFoundation is the top-performing model in this cross-protocol\nevaluation, across all six single-cell RNA-seq protocols, including 10x Genomics, BD Rhapsody, MARS-seq\nv2.0, DropSeq, inDrop v3, and SMART-seq v4, based on F1 scores. Fine-tuning further improved\nscFoundation’s performance in BD Rhapsody and MARS-seq v2.0 datasets but led to decreased performance\nin the remaining protocols. This suggests that fine-tuning can enhance model adaptation to certain protocols\nbut may introduce overfitting or reduce robustness in others. Additionally, Geneformer consistently ranked as\nthe second-best model in five out of six protocols, highlighting some stability in its cross-protocol\ngeneralization, although all models struggled with inDrop v3 and SMART-seq v4.\n3. A comparison of the results with the existing non foundation model-based methods would be useful. While\nsome of these contemporary methods that the authors have cited in the discussion section might be trained on\ndomain specific data, it will still be good to put the performance of a foundation model based approach in\ncontext of other existing methods. In fact, a simple comparison of the MLP classifier the authors use with the\nembeddings obtained from a foundation model against an MLP classifier operating directly on normalized\nexpression values can give some idea of the improvement attributable to the foundation models themselves.\nResponse: Thanks for the comments. To evaluate whether foundation models (FMs) outperform traditional\nmethods in drug response prediction, we compared their performance with three representative non-FM based\nmethods: 1) multilayer perceptron (MLP); 2) scDEAL, a deep-learning model for cancer drug response\nprediction at single-cell level; and 3) DREEP, a statistical method for drug response estimation from single-\ncell transcriptional profiles. All these methods were assessed in cross-data and pooled-data scenarios\n(Supplementary Note 3; Supplementary Fig. 12).\nAmong these three baseline models, the MLP performed better in the pooled-data evaluation setting.\nIt achieved superior performance across different categories, including targeted therapy, cell line data, and\nseveral cancer datasets such as melanoma (0.875 ± 0.0014), breast cancer (0.774 ± 0.0038), lung cancer (0.737\n± 0.0041), chemotherapy (0.711 ± 0.0057), multiple myeloma (0.686 ± 0.0044), bone marrow aspirate (0.682\n± 0.0042), targeted therapy (0.663 ± 0.0036), acute myeloid leukemia (0.656 ± 0.0073), prostate cancer (0.649\n± 0.0188), and cell line data (0.632 ± 0.0045). In liver cancer, we observed that scDEAL, DREEP, and MLP\nachieved nearly identical mean F1 scores (0.924, 0.924, and 0.922, respectively), performing comparably with\nseveral FMs including scGPT, CellLM, LLaMa3, Geneformer, and scBERT in the pooled-data layer-freezing\nscenario, as well as fine-tuned tGPT (0.924) and scBERT (0.925). Interestingly, in melanoma datasets, MLP\nnot only outperformed the other baseline methods but also exceeded the performance of several foundation\nmodels, including both fine-tuned and layer-frozen versions of CellLM, UCE, and scBERT. These findings\nhighlight that in certain contexts, relatively simple architectures trained on well-pooled data can match or even\nsurpass the predictive performance of more complex foundation models.\nIn comparison with the FMs under pooled-data evaluation using layer-freezing (Fig. 2a), the MLP\nmodel performed worse than five of nine FMs in the melanoma dataset: scFoundation (0.993), scGPT (0.946),\nLLaMa3 (0.898), tGPT (0.898), and CellPLM (0.896), but outperformed CellLM (0.804), Geneformer (0.854),\nUCE (0.830), and scBERT (0.825). When evaluated against FMs fine-tuned in the pooled-data scenario, MLP\nwas outperformed by five of nine FMs including scFoundation (0.978), tGPT (0.932), scGPT (0.992),\nGeneformer (0.932), and CellPLM (0.930), while outperforming CellLM (0.745), scBERT (0.788), and UCE\n(0.845). Moreover, across the broader evaluation landscape, particularly in cross-data scenarios where\ngeneralization is critical, foundation models consistently demonstrated superior predictive power compared\nwith non-FMs. This highlights one of the central advantages of pre-trained FMs, i.e., their ability to transfer\nlearned representations and maintain performance across heterogeneous datasets, which is especially\nimportant for drug response prediction in diverse cancer contexts.\nIn summary, while non-FM baselines can occasionally perform well under pooled-data conditions, our\nsystematic comparisons show that foundation models provide more robust and generalizable predictive\nperformance, thereby reinforcing their value for single-cell drug response prediction.\n4. There is a large variability in the performance of various models applied to different data sources. For\nexample, Fig 3 shows that scFoundation model performance varies more than any other model. scFoundation\nmodel is the second largest in terms of parameters with ~120M parameters and the largest in pre-training\ncorpus with ~50M cells. What might be the cause of such variability? For instance, is it difficult to finetune\nsuch a large model. Such discussions would be useful because the authors want their work to serve as a tool\nto make informed decisions about which foundation models to use.\nResponse: Thanks for the comments. We agree that the variability observed in scFoundation’s performance\nacross data sources needs detailed discussion. We anticipate that several factors, related to both model\narchitecture and data characteristics, may help explain this phenomenon.\n1) First, model size and model optimization are likely contributors. With ~120 million parameters,\nscFoundation is the second largest model in our benchmark and was pre-trained on the largest corpus (~50\nmillion cells). While large models generally offer greater representational capacity, they also require extensive\ntask-specific data and careful hyperparameter tuning to achieve stable performance. In our fine-tuning\nexperiments, we used over 320,000 cells stratified by cancer type. For some cancers, however, the number of\ntraining samples was relatively small compared to the model’s size, increasing the risk of underfitting or,\nconversely, overfitting due to domain shifts. 2) Second, heterogeneity between the pre-training corpus and\nevaluated datasets may play a role. scFoundation was pre-trained on a highly diverse set of single-cell profiles\nspanning multiple tissues, protocols, and conditions. This diversity often enhances generalizability but can\nalso mean that representations for specific cancer types in our benchmark are less specialized, requiring\nsignificant adaptation during fine-tuning. When fine-tuning data diverges substantially from the relevant\nportions of the pre-training corpus, performance may become less stable. 3) Third, training strategies may not\nfully leverage scFoundation’s pre-training capacity. Without advanced adaptation methods, such as gradual\nunfreezing, layer-wise learning rate decay, or domain-adaptive pre-training, scFoundation’s large-scale pre-\ntraining may not translate optimally across all tasks, especially in data-limited settings. 4) Finally, differences\nin input gene dictionary may also contribute. scFoundation’s gene dictionary is limited and does not cover all\nhigh variable gene symbols present in our evaluated datasets. This input design may introduce performance\nvariabilies compared to other models.\nMoreover, to support users in making informed decisions about which foundation models to use,\nwe\nprovided a Model Recommendation Guide (Supplementary Table 5). This guide summarizes the best-\nperforming FMs across four evaluation settings: (1) pooled-data with layer-freezing, (2) pooled-data with fine-\ntuning, (3) cross-data with layer-freezing, and (4) cross-data with fine-tuning. By linking model performance\nto evaluation scenario, tissue type, drug class, and cancer type, the guide offers a practical framework for\nselecting or adapting models to specific research contexts. Overall, scFoundation and scGPT emerge as strong\ndefault choices for pooled-data applications across most tissues, drug classes, and cancer types. In contrast,\ncross-data scenarios require more careful consideration, as performance tends to be more variable. In these\ncases, models such as UCE, tGPT, and other task-specific architectures often demonstrate stronger\ngeneralization. Finally, for rare cancers or highly imbalanced datasets, we recommend that researchers place\nparticular emphasis on cross-data performance, since not all models are equally reliable under such\nchallenging conditions.\n5. Validation datasets seem to have been used in a non-standard way. My understanding is that usually the\nvalidation datasets should be held out and not used in training at all. However, the authors have performed a\ntrain-test split on the validation datasets to train models, independently of models already trained-tested on\nthe primary data. Is there a particular reason for such a choice?\nResponse: Thanks for the comments. We would like to clarify the rationale behind our use of the primary and\nvalidation datasets. The primary data collection was assembled at the initial stage of this benchmarking study\nand served as the foundation for our main analyses. As additional datasets became available, we then\nassembled a validation data collection. This independent collection was intended not only to confirm the\nfindings obtained from the primary data, but also to broaden the evaluation to a more diverse range of tissues,\ndiseases, and experimental conditions. In this way, the validation datasets serve as an additional and more\nstringent test of model generalizability beyond the original data pool.\nTo maintain fairness and consistency, we applied the same benchmarking strategy to the validation data as\nwas used for the primary data, performing a train-test split within each dataset and evaluating the foundation\nmodels independently of those trained on the primary collection. This ensured that all models were compared\nunder similar conditions while still being tested on unseen data partitions.\nImportantly, to further evaluate the model generalizability, we also evaluated the models on the validation\ndatasets in a zero-shot learning setting without any additional training. These results, reported in the\nSupplementary Note 11, demonstrate that the models can directly predict drug response labels on entirely\nindependent data. Together, the two analyses, train-test split and zero-shot evaluation, provide a more\ncomprehensive and rigorous assessment of model performance.\nIn summary, the validation datasets were designed to serve a dual role: first, to confirm reproducibility of\nresults obtained from the primary data; and second, to expand the evaluation to newly emerging and more\ndiverse datasets. We believe this approach strengthens the conclusions of the study by giving readers a clearer\npicture of model generalizability across independent data sources.\nMinor comments\n1. While F1 and AUROC are relevant metrics, the authors should consider reporting Pearson/Sprearman\ncorrelations between known and predicted responses as well.\nResponse: Thanks for the comments. In the revised manuscript, we reported the Pearson correlations between\nknown and predicted responses, using results from the pooled-data evaluation in the primary dataset, with\nmodels fine-tuned using LoRA. We examined datasets spanning multiple tissue types, drug classes, cancer\ntypes, and treatment regimens.\nOur results show that scFoundation consistently achieved better correlations between predicted and ground-\ntruth responses. For example, it obtained Pearson correlations of 0.8437 in tumor tissue, 0.8915 in acute\nlymphoblastic leukemia, 0.8652 in targeted therapy, and 0.8984 in the anti-CD19 CAR-T regimen\n(Supplementary Table 3; Supplementary Fig. 13). Other models, such as scGPT, CellPLM, tGPT, and\nGeneformer, also demonstrated competitive performance across several settings. For instance, in tumor tissue,\ntheir mean correlation scores were 0.7289, 0.6156, 0.5925, and 0.5923, respectively. In contrast, UCE,\nscBERT, and CellLM generally did not perform very well (Supplementary Table 3; Supplementary Fig.\n13). These results indicate that while fine-tuned foundation models vary in their ability to preserve predictive\nsignal across datasets, scFoundation shows consistently superior performance, suggesting its robustness in\naligning predictions with true drug response outcomes (Supplementary Note 4; Supplementary Table 3;\nSupplementary Fig. 13).\n2. In figure 3, since two methods are being compared, it would make sense for AUROC plots to either both\nbe violin plots or box plots.\nResponse: Thanks for the comments. In the revised Fig. 3, we have updated the AUROC plot to box plots.\n3. A case study or an example of a drug response predicted by the best performing model would be an\ninteresting addition to the paper.\nResponse: Thanks for the comments. In the revised manuscript, we added a case study of drug response\npredictions generated by the best-performing models. To illustrate the potential impact of inaccurate\npredictions, we analyzed Data 3 (Supplementary Tables 1–2) and compared biological insights derived from\nground truth labels with those obtained from scGPT-predicted labels (Supplementary Note 5;\nSupplementary Figs. 14–17). In Data 3, ground-truth annotations indicated 10,897 resistant cells (67.95%)\nand 5,140 sensitive cells (32.05%), whereas scGPT predicted 62.95% resistant and 37.05% sensitive, showing\nnoticeable shifts in response proportions that may alter downstream biological interpretations.\nFor our analysis, we used Seurat for dimensionality reduction and clustering, followed by manual\nannotation of cell populations based on canonical marker genes. This yielded 16,037 tumor cells, 4,923 CD8⁺\nT cells, 1,157 CD4⁺ T cells, 6,077 monocytes/macrophages, 3,355 natural killer (NK) cells, 519 dendritic cells,\n325 progenitors, 360 erythrocytes, and 3,117 B cells (Supplementary Fig. 14). We then conducted several\ndownstream analyses including differentially expressed genes identification; over-representation enrichment\nanalysis using GO (Gene Ontology), KEGG (Kyoto Encyclopedia of Genes and Genomes), and MSigDB\nHallmark gene sets; cell-cell interaction inference; and pseudotime trajectory analysis. Our results revealed\nthat several key biological findings were only recovered when using ground-truth labels, but were missed\nwhen relying on predicted labels. For example, ground-truth based analysis identified COL1A2 and SEZ6L2\nas top up-regulated genes in resistant tumor cells and highlighted enrichment of biological processes such as\ncell differentiation, cell-cell adhesion, and the p53 pathway (Supplementary Note 5; Supplementary Fig.\n14, Supplementary Fig. 16). In addition, stronger intercellular communication was observed in dendritic cells\nand B cells as receivers in resistant tumor cells, while weaker BTLA and CD160 signalling was detected in\nCD8⁺ T cells, NK cells, dendritic cells, progenitors, and B cells (Supplementary Note 5; Supplementary\nFigs. 15, Supplementary Fig. 17). Furthermore, the ground-truth labels revealed a coherent pseudotime\ntrajectory from sensitive to resistant states (Supplementary Note 5; Supplementary Figs. 15,\nSupplementary Fig. 17). These discrepancies highlight the importance of ensuring high prediction accuracy\nto enable more reliable and meaningful biomechanistic insights for drug discovery.\nReviewer #2 (Remarks to the Author):\nThis manuscript presents scDrugMap, a comprehensive benchmarking framework for evaluating 10\nfoundation models (8 single-cell-specific, 2 general LLMs) in predicting drug response from single-cell\ntranscriptomics. The study curates 345,607 cells across 53 datasets spanning 14 cancer types, 3 therapies, and\n5 tissues. The authors demonstrated that scFoundation dominates pooled-data prediction, while UCE and\nscGPT lead in cross-data generalization; fine-tuning universally outperforms layer-freezing, general LLMs\n(e.g., GPT4o-mini) fail in zero-shot tasks, and computational efficiency varies sharply (scFoundation ≫\nscGPT).\nIn the context of growing tumor single-cell data and proliferating LLMs, scDrugMap establishes the first\nbenchmark for foundation models in drug response prediction. It validates specialized single-cell FMs over\ngeneral LLMs for clinical reliability, provides scenario-specific model selection guidelines, and quantifies\ncomputational trade-offs—directly enabling more accurate, deployable treatment response prediction in\nprecision oncology. The study's claims of novelty are undermined by its exclusive reliance on existing models\nwithout algorithmic innovation, and its benchmarking framework closely mirrors prior approaches. Data\nintegration lacks originality, repurposing public GEO datasets without novel experimental validation.\nCrucially, the work fails to cite key literature (e.g., GPTCelltype for LLM benchmarking; DREEP for drug\nresponse prediction), overstating its \"first comprehensive benchmark\" contribution while overlooking\nestablished methodologies in adjacent domains.\nMajor Concerns:\n1. Questionable novelty. All models are repurposed (scGPT, Geneformer). Fails to cite comparable\nbenchmarks (e.g., GPTCelltype, Nat Methods 2024). Positions as \"first\" without literature context.\nResponse: Thanks for the comments. We agree that many of the foundation models included in our benchmark,\nsuch as scGPT and Geneformer, were originally developed for broader single-cell analysis tasks. Our\ncontribution lies in systematically repurposing and benchmarking these models specifically for drug response\nprediction, a problem of critical importance in cancer treatment but one that has not yet been rigorously\nevaluated across diverse single-cell datasets.\nIn the revised manuscript, we now cite and discuss relevant comparable work, including GPTCelltype (Nat\nMethods, 2024), which highlighted the utility of GPT-4 for cell type annotation in scRNA-seq data. As noted\nin our manuscript (lines 69–76, these foundation models have indeed been applied to a variety of tasks,\nincluding cell type annotation, batch effect removal, multi-omics integration, gene dosage sensitivity\nprediction, and chromatin dynamics modeling. While scFoundation has been tested for drug response\nprediction in a limited context, no prior study has performed a comprehensive, systematic benchmark of\nmultiple large-scale single-cell foundation models for this specific task.\nOur work addresses this gap by providing the first unified evaluation framework, along with a large and\ndiverse collection of drug response-related single-cell datasets, to rigorously compare foundation models\nunder both pooled-data and cross-data scenarios, across multiple cancer types, tissues, and therapeutic\nregimens. By focusing explicitly on drug resistance prediction, we not only highlight the strengths and\nweaknesses of existing models in this domain but also provide practical guidance for selecting models in\nbiomedical and translational research. We believe this represents a meaningful advance beyond simply\nrepurposing existing models, as it delivers a new benchmarking resource tailored to one of the most pressing\nchallenges in precision oncology.\n2. Lack of baseline comparisons. The study does not include comparisons with established non–foundation\nmodel (non-FM) methods such as DREEP or scDEAL, which limits the ability to assess the added value of\nthe proposed foundation models. Without such benchmarks, it remains unclear whether the FMs truly offer\nperformance improvements. If the FMs underperform relative to these simpler or more established approaches,\nthe core premise of the study—that foundation models provide superior predictive power—would be\nsignificantly weakened.\nResponse: Thanks for the comments. In our revision, we included baseline comparisons with three established\nnon–foundation model (non-FM) methods: a multilayer perceptron (MLP), scDEAL (a deep learning\nframework for drug response prediction at the single-cell level), and DREEP (a statistical method for drug\nresponse estimation from single-cell transcriptional profiles of tumors). These non-FM baselines were\nsystematically evaluated under both cross-dataset and pooled-dataset scenarios (Supplementary Note 3;\nSupplementary Figs. 11–12).\nOur results show that, overall, non-FM methods did not outperform foundation models. Interestingly, among\nthe non-FMs, the MLP trained on pooled data achieved strong and consistent results than scDEAL and DREEP\nin several contexts, including targeted therapy, cell line data, and cancer datasets such as melanoma (F1 =\n0.875 ± 0.0014), breast cancer (0.774 ± 0.0038), and lung cancer (0.737 ± 0.0041). In some cases, such as\nliver cancer, scDEAL, DREEP, and MLP achieved performance equivalent to several foundation models (e.g.,\nmean F1 = 0.924). However, across the broader evaluation landscape, particularly in cross-data scenarios\nwhere generalization is critical, foundation models consistently demonstrated superior predictive power\ncompared with non-FMs. This highlights one of the central advantages of pre-trained FMs, i.e., their ability\nto transfer learned representations and maintain performance across heterogeneous datasets, which is\nespecially important for drug response prediction in diverse cancer contexts.\nIn summary, while non-FM baselines can occasionally perform well under pooled-data conditions, our\nsystematic comparisons show that foundation models provide more robust and generalizable predictive\nperformance, thereby reinforcing their value for single-cell drug response prediction.\n3. Overinterpretation of marginal results. UCE’s \"top\" cross-data performance (F1=0.774) is clinically\nirrelevant. Claiming it \"achieves optimal results\" (Fig 3b) misrepresents data. Similarly, scFoundation’s\nfailure in PBMCs (F1=0.461) is glossed over.\nResponse: Thanks for the comments. We have carefully revised the manuscript to clarify and moderate our\ninterpretation.\nSpecifically, we have removed any language that could be construed as overstating UCE’s performance. In\nthe main text, the previous description of UCE as a “top” performer has been replaced with the more accurate:\n“UCE demonstrated relatively stronger performance compared to other models across various categories,\ndespite the overall modest performance observed across all models” (lines 211–213). Moreover, in the abstract\n(lines 30-31), we revised the phrase “achieves optimal results” to the more precise: “UCE for tumor tissue\nshowed relatively better performance (mean F1 score: 0.774)”. Finally, regarding the F1 = 0.461, we would\nlike to clarify that this result corresponds to CellLM, not scFoundation, under the layer-freezing pooled-data\nevaluation scenario. As stated in the manuscript (lines 152–153): “CellLM and scBERT underperformed on\nPBMCs data (mean F1 scores: 0.461 and 0.483, respectively). In contrast, scFoundation performed well on\nPBMCs in the same scenario, achieving a mean F1 score of 0.939, ranking highest among all models (Fig.\n2a).” Moreover, across our benchmarks, CellLM consistently showed limited performance, with its best result\nobserved in acute lymphoblastic leukemia under cross-data evaluation with full fine-tuning (mean F1 = 0.718;\nSupplementary Fig. 3). Taken together, these revisions ensure our interpretations are appropriately cautious,\naccurately represent the results, and avoid overstatement of model performance.\n4. Data biases compromise conclusions. Severe class imbalance (e.g., 86% resistant cells in neuroblastoma;\nSupp Table 2) inflates F1 scores. AUPRC should supplement F1 for imbalanced subgroups. Batch effects\nacross 36 datasets are ignored. No correction (e.g., Harmony) was applied, invalidating cross-dataset claims.\nResponse: Thanks for the comments. In our revised manuscript, we investigated the issue of class imbalance\nand the potential impact of batch effects.\nClass imbalance. According to Supplementary Table 2, the neuroblastoma dataset (Data13) consists of\n5,492 sensitive cells (46.58%) and 6,298 resistant cells (53.42%), which does not show severe imbalance.\nThat said, we agree that class imbalance is a concern in certain datasets. For this study, we defined severe\nimbalance as cases where one class exceeds 70% of the total. Based on this criterion, two datasets in our\nprimary collection are imbalanced: Data 23 (lung cancer; 15.51% sensitive vs. 84.49% resistant) and Data 32\n(prostate cancer; 20.15% sensitive vs. 79.85% resistant). To account for this, we supplemented F1 with\nAUPRC, which is more appropriate for imbalanced data. Using fine-tuned models with LoRA under both\npooled-data and cross-data scenarios, we found that pooled-data evaluations consistently outperformed cross-\ndata ones. For instance, in Data 23, scFoundation (AUPRC = 0.894), scGPT (AUPRC = 0.802), and CellPLM\n(AUPRC = 0.709) all achieved strong predictive performance above 0.7, despite the imbalance. Similarly, in\nData 32, pooled-data training again yielded stronger results, with scFoundation (AUPRC = 0.747), scGPT\n(AUPRC = 0.619), and CellPLM (AUPRC = 0.518) ranking highest (Supplementary Note 9;\nSupplementary Fig. 22). These findings show that for severe imbalanced datasets (e.g. resistant cell\nproportion exceeding 70%), models under pooled-data evaluation consistently outperformed those in cross-\ndata scenarios. Among these, scFoundation, scGPT, and CellPLM generally achieved better predictive\nperformance than other models.\nBatch effects. To address the concern about batch effects across diverse datasets, we applied Harmony on the\ntop 2000 highly variable genes in each dataset for batch correction (Supplementary Note 2; Supplementary\nFig. 7-8). With the batch corrected data, we again conducted comprehensive benchmarking under two\nevaluation scenarios: pooled-data evaluation (Supplementary Fig. 7) and cross-data evaluation\n(Supplementary Fig. 8). In both settings, we also implemented two model training strategies (layer freezing\nand fine-tuning using LoRA). Our results showed that performance with batch correction was largely\nconsistent with the results obtained without correction. Top-performing models such as scFoundation, scGPT,\nand CellPLM remained stable or exhibited only minor improvements after batch correction (Supplementary\nFig. 7a). For example, in tumor tissue, the F1 scores with and without batch correction were: scFoundation =\n0.998 vs. 0.998; scGPT = 0.939 vs. 0.939; and CellPLM = 0.940 vs. 0.936. This phenomynon is also observed\nin other models, such as tGPT, Geneformer, and UCE, exhibited comparable performance before and after\nbatch correction. For instance, in targeted therapy datasets, tGPT had F1 scores of 0.794 (with batch effects)\nvs. 0.756 (after correction), and Geneformer scored 0.751 vs. 0.700. Overall, these results suggest that\nfoundation models tend to capture biologically meaningful features that are relatively robust to batch-specific\nvariation, supporting the validity of our benchmarking results.\n5. Inadequate validation. \"Validation collection\" uses same data sources (GEO) as primary data. Independent\nclinical cohorts (e.g., fresh biopsies, PDX models) are absent.\nResponse: Thanks for the comments. We would like to clarify that our original validation collection already\nincluded independent clinical cohorts, specifically two PDX datasets (Data1 and Data2), three fresh biopsy\ndatasets (Data11, Data12, and Data17), and two organoid datasets (Data13 and Data14), as indicated in the\n“Source” and “Tissue” columns of Supplementary Table 1. In addition, tumor tissues in this collection are\nalso derived from fresh biopsies, as documented in their original publications. Although all datasets were\nobtained from GEO, they are derived from different published studies and generated at different times. These\ndifferences are important, as they ensure broader biological and technical diversity and provide a stronger\nbasis for independent validation.\nTo further strengthen the validation across tissue types and sources, we expanded the validation\ncollection by retrieving seven additional fresh biopsy datasets (Data18–Data24). These represent the most\nrecent and comprehensive fresh biopsy data available in GEO, thereby maximizing the biological and\ntechnical diversity of our validation resource. The external validation collection now comprises 24 datasets,\nincluding 10 fresh biopsies, 2 PDX datasets, 2 organoid datasets, and 10 cell line datasets (Fig. 1;\nSupplementary Table 2).\nAfter quality control, the validation collection contains a total of 168,486 cells, of which 80,237\n(47.62%) are sensitive cells and 88,249 (52.38%) are resistant cells according to the ground truth labels\n(Supplementary Table 2). When stratified by data type, 8 out of 14 categories are approximately balanced\nbetween sensitive and resistant cells. These include organoid, tumor tissue, colon cancer, NSCLC, pancreatic\ncancer, chemotherapy, immunotherapy, and targeted therapy (Fig. 1d; Supplementary Table 2). Using these\ncurated data, we evaluated the performance of foundation models using layer-freezing. Model performance\nwas assessed across tissue types, drug classes, and cancer types. We have updated the abstract (line 22), the\n“Model Evaluation in the Validation Data Collection” in Results section (lines 278-297), and the “Data\ncollection and preprocessing” in Methods (line 551), to include the details of this expanded validation dataset\nand the corresponding evaluation results (Fig. 6; Supplementary Fig. 4).\nSpecifically, by data source (including fresh biopsy, PDX, organoid, and cell line), our results showed that\nscFoundation consistently achieved the best performance across all four tissue types, with mean F1 scores of\n0.932 (fresh biopsy), 0.926 (PDX), 0.973 (organoid), and 0.901 (cell line) (Supplementary Note 8;\nSupplementary Fig. 21). Additionally, CellPLM, scGPT, and tGPT achieved strong performance specifically\non organoid data, with mean F1 scores of 0.900, 0.928, and 0.946, respectively. In addition, based on F1 score\nand recall, we observed that the models generally performed worse in PDX data compared to the remaining\nthree tissue types, while other metrics including accuracy, AUROC, and precision did not vary much across\nthe four tissue types. This could be due to class imbalance, i.e., in the two PDX datasets, resistant cells\naccounted for 72.2% and 57.54% of cells, respectively, which likely skewed the models toward under-\npredicting the minority class (sensitive cells). Beyond imbalance, intrinsic biological features of PDX models\nmay also introduce additional complexity and noise, leading to reduced generalizability.\n6. Methodological opacity. Preprocessing: Gene/cell QC thresholds (e.g., mitochondrial content) undefined;\nLoRA tuning: Rank=8 used universally without ablation study; Prompt engineering: GPT4o-mini’s poor\nperformance may stem from suboptimal design (only top 10 genes input).\nResponse: Thanks for the comments. We have revised the manuscript to improve methodological clarity and\nconducted additional analyses to strengthen our conclusions.\n1) Regarding preprocessing, we have now explicitly described the steps in the “Data Collection and\nPreprocessing” section in Methods (lines 554-558. Briefly, preprocessing of scRNA-seq data was performed\nby Seurat (v4.3.0). Cells were retained if they had at least 500 detected genes (nFeature_RNA > 500) and\nmitochondrial content (percent.mt) < 10%, ensuring removal of low-quality cells and technical artifacts.\nDimensionality reduction was performed using the top 30 principal components (PCs), and clustering was\nconducted at a resolution of 0.5. These clarifications ensure transparency and reproducibility of our\npreprocessing pipeline.\n2) Regarding LoRA tuning, we agree that using rank=8 universally without ablation left room for\nimprovement. To address this, we conducted additional experiments with LoRA ranks of 4 and 16\n(Supplementary Note 10, Supplementary Figs. 23-24). Overall, performance differences across ranks were\nmodest in both pooled-data and cross-data evaluations but some noteworthy exceptions emerged. For example,\nin pooled-data evaluation (Supplementary Fig. 23), scFoundation fine-tuned with a LoRA rank of 8 achieved\nsignificantly better predictive performance in liver and pancreatic cancers compared to the same model fine-\ntuned with rank of 4 and 16. Specifically, in liver cancer, LoRA 8 significantly outperformed both LoRA 4 (p\n= 0.0164) and LoRA 16 (p = 0.0086), while in pancreatic cancer, LoRA 8 outperformed LoRA 4 (p = 0.0104)\nbut not LoRA 16 (p > 0.05). Additionally, several other models fine-tuned with LoRA rank 8 also showed\nimproved performance over ranks 4 and 16 in specific contexts, even though these differences were not\nstatistically significant. Notable examples include UCE and Geneformer in acute myeloid leukemia, UCE in\ntumor tissue, scBERT in immunotherapy, and both CellLM and Geneformer in targeted therapy. These\nfindings suggest that while LoRA rank influences fine-tuning performance in some pooled-data contexts, its\nimpact on generalization across heterogeneous datasets is limited. Importantly, they highlight that LoRA rank\n8 provides a good balance between adaptability and efficiency.\nIn addition, under cross-dataset evaluation (Supplementary Fig. 24), none of the performance\ndifferences across LoRA ranks reached statistical significance. Nonetheless, models fine-tuned with LoRA\nrank 8 often performed better than those with ranks 4 or 16 in certain settings. For example, scFoundation in\ntumor tissue, bone marrow aspirate, scBERT in PBMCs and multiple myeloma, UCE for breast cancer and\nlung cancer, Geneformer and scBERT for prostate cancer, and scGPT for melanoma. This indicates that while\nLoRA rank can influence fine-tuning outcomes in specific contexts, its overall effect on generalization\nperformance appears limited. Together, these results imply that moderate LoRA configurations may be\npreferable in pooled-data scenarios, but LoRA rank selection may be less critical when prioritizing model\nrobustness across heterogeneous datasets.\n3) Regarding prompt engineering for GPT4o-mini, we investigated alternative prompt templates to test\nwhether its suboptimal performance was due to design. In our revision, we compared three prompts: (1) the\ntop 10 highly expressed genes + data source, (2) the top 100 genes + data source, and (3) a chain-of-thought (CoT)\nprompt designed to explicitly structure input and reasoning (Supplementary Note 7; Supplementary\nFig. 20). Across tissues, drugs, and cancer types, both the 10-gene and 100-gene prompts yielded comparable\nF1 performance (e.g., tumor tissue: 0.519 vs. 0.506; PBMCs: 0.583 vs. 0.565). Interestingly, the 10-gene\nprompt slightly outperformed the 100-gene prompt in multiple cancer types (e.g., lung, breast, CLL, prostate).\nIn contrast, the CoT prompt consistently underperformed in F1 scores, though it achieved higher accuracy in\nselect settings (e.g., immunotherapy, lung, MM, prostate). These results suggest that using a simple top-10-\ngene prompt is at least as effective as, and often better than, a more complex 100-gene prompt. While GPT4o-\nmini remains suboptimal overall, often performing close to or below baseline, the results indicate that prompt\ndesign was not the primary factor driving its poor performance.\nIn summary, we have clarified our preprocessing criteria, expanded our LoRA rank evaluation, and tested\nalternative prompt strategies. Together, these efforts address methodological opacity and reinforce the\nrobustness of our conclusions that while LoRA rank and prompt design can influence performance in specific\ncases, the overall trends and comparative benchmarking results across foundation models remain consistent.\n7. Authors used four evaluation metrics (F1, AUROC, precision, and recall) for assessing model performance\nin predicting cell-level drug response. However, relying solely on these metrics may not provide a sufficiently\nrigorous basis for model comparison. It is recommended to add statistical tests (e.g., p-values, confidence\nintervals) to determine whether the performance differences between models are significant.\nResponse: Thanks for the comments. In this revision, we have added comparisons of model performance\nacross tissue types, drug types, and cancer types using Student’s t-test (Supplementary Note 1;\nSupplementary Fig. 5; Supplementary Table 4). In the pooled-data evaluation, we compared the mean F1\nscore of each model to scFoundation, which was the best-performing model in this setting. In the cross-data\nevaluation, we used scGPT as the reference model, as it demonstrated superior performance in tumor tissue.\nOverall, in the pooled-data evaluation, under both layer-freezing and full fine-tuning settings, scFoundation\nsignificantly outperformed all other models across different tissue, drug, and cancer types (Supplementary\nFig. 5; Supplementary Table 4). In contrast, under the cross-dataset evaluation, scGPT achieved significantly\nbetter performance than LLaMa3 and tGPT in tumor tissue under the zero-shot setting. We also observed\nseveral models that significantly outperformed scGPT in certain context (Supplementary Fig. 6;\nSupplementary Table 4). For example, using layer-freezing training method, LLaMa3 was superior than\nscGPT in both cell line and PBMCs (p-values, cell line: 0.0006; PBMCs: 0.0187), and tGPT also surpassed\nscGPT in PBMCs (p-value: 0.0439). In addition, scFoundation and LLaMa3 outperformed scGPT in targeted\ntherapy (p-values, scFoundation: 0.0076; LLaMa3: <0.0001). Using fine-tuning training method, tGPT again\nshowed significantly better performance than scGPT in PBMCs (p = 0.0415). CellPLM and scFoundation\noutperformed scGPT in targeted therapy (p-values, CellPLM: 0.04827; scFoundation: 0.0161). These findings\nsuggest that scFoundation is the most robust choice in pooled-data scenarios, whereas alternative models may\noffer advantages in certain data types or therapeutic contexts under cross-data evaluation.\n8. In the \"Few-shot learning for drug response prediction\" section, the evaluation of few-shot/zero-shot\nlearning is relatively brief, without a comprehensive analysis, for instance, of whether different prompt\ntemplates influence prediction performance. Furthermore, the lack of comparison with similar methods makes\nit difficult to assess the potential or limitations of large language models in drug response prediction.\nResponse: Thanks for the comments. In the revised manuscript, we have expanded both the prompt evaluation\nand baseline comparisons to address these concerns.\n1) As outlined in our response to comment #6, we systematically evaluated the impact of different prompt\ntemplates on GPT4o-mini’s performance. Specifically, we tested three prompt types: (1) the top 10 most\nhighly expressed genes per cell along with the dataset source (e.g., GEO accession); (2) the top 100 genes\nwith dataset source; and (3) a chain-of-thought (CoT) prompt that explicitly defines the nested data structure\nof cells and their expressed genes to guide reasoning. This enables the model to follow a predetermined order\nwhen assessing drug sensitivity for each cell, helping to reduce ambiguity, improve the clarity and accuracy\nof reasoning, and enhance result consistency and interpretability. Our results showed that the top-10 gene\nprompt performed similarly to or slightly better than the 100-gene prompt across diverse tissue, drug, and\ncancer types, while the COT prompt underperformed in all scenarios (Supplementary Note 7;\nSupplementary Fig. 20). Importantly, we note that regardless of prompt design, GPT4o-mini’s overall drug\nresponse prediction performance remained suboptimal, often close to or below baseline, highlighting its\ncurrent limitations for this task (Supplementary Note 7; Supplementary Fig. 20).\n2) To evaluate whether foundation models (FMs) outperform similar traditional (non-FM) methods in drug\nresponse prediction, we compared their performances with that of the multilayer perceptron (MLP), scDEAL\n(a deep-learning model for cancer drug response prediction at single-cell level), and DREEP (a statistical\nmethod for drug response estimation from single-cell transcriptional profiles of tumors). All these three\nmethods were assessed in pooled-data and cross-data scenarios (Supplementary Note 3; Supplementary Fig.\n11-12).\nOur results show that, overall, non-FM methods did not outperform foundation models. Interestingly, among\nthe non-FMs, the MLP trained on pooled data achieved strong and consistent results than scDEAL and DREEP\nin several contexts, including targeted therapy, cell line data, and cancer datasets such as melanoma (F1 =\n0.875 ± 0.0014), breast cancer (0.774 ± 0.0038), and lung cancer (0.737 ± 0.0041). In some cases, such as\nliver cancer, scDEAL, DREEP, and MLP achieved performance equivalent to several foundation models (e.g.,\nmean F1 = 0.924). However, across the broader evaluation landscape, particularly in cross-dataset scenarios\nwhere generalization is critical, foundation models consistently demonstrated superior predictive power\ncompared with non-FMs. This highlights one of the central advantages of pre-trained FMs, i.e., their ability\nto transfer learned representations and maintain performance across heterogeneous datasets, which is\nespecially important for drug response prediction in diverse cancer contexts. In summary, while non-FM\nbaselines can occasionally perform well under pooled-data conditions, our systematic comparisons show that\nfoundation models provide more robust and generalizable predictive performance, thereby reinforcing their\nvalue for single-cell drug response prediction.\n9. The manuscript lacks a thorough analysis of biological mechanisms behind model's prediction, whether the\nmodel’s decisions depend on key genes, cell subpopulations, drug mechanisms, or other biological features.\nResponse: Thanks for the comment. In the revised manuscript, we added analyses to examine the biological\nmechanisms underlying model’s predictions, including contributions from key genes, cell-cell interactions,\nand drug-related pathways. To do this, we used Data 3 from the primary collection (Supplementary Tables\n1–2) and compared downstream analyses based on ground-truth labels versus scGPT-predicted labels\n(Supplementary Note 5; Supplementary Figs. 14–17).\nAccording to ground-truth annotations, Data 3 contained 10,897 resistant tumor cells (67.95%) and 5,140\nsensitive tumor cells (32.05%), whereas scGPT predicted a slightly shifted distribution (62.95% resistant vs.\n37.05% sensitive). After QC, 35,870 cells were retained and analyzed with Seurat, yielding major populations\nincluding 16,037 tumor cells, 4,923 CD8⁺ T cells, 1,157 CD4⁺ T cells, 6,077 monocytes/macrophages, 3,355\nNK cells, 519 dendritic cells, 325 progenitors, 360 erythrocytes, and 3,117 B cells (Supplementary Fig. 14).\nWe then performed a series of downstream analyses, including differential expression, pathway enrichment\n(GO, KEGG, MSigDB Hallmark), cell-cell communication, and pseudotime trajectory inference, to assess\nkey biological insights playing roles in model predictions. Using ground-truth labels, we identified key\nresistance-associated genes (e.g., COL1A2, SEZ6L2), enriched pathways central to tumor biology and drug\nresistance (e.g., cell differentiation, adhesion, p53 signaling), and clear intercellular communication patterns\n(including stronger signaling to DCs and B cells and weakened BTLA/CD160 interactions in CD8⁺ T, NK,\nand DC populations). Ground-truth labels also yielded a coherent pseudotime trajectory from sensitive to\nresistant states (Supplementary Figs. 15–17). In contrast, some of these features were absent or attenuated\nwhen using scGPT-predicted labels, suggesting that these biological signals may indeed inform model\npredictions, but can be obscured when predictions are less accurate.\nTaken together, this case study shows that foundation models can reflect underlying biology, including gene\nexpression, cell states, and drug mechanisms, but only when their predictions are sufficiently accurate. This\nunderscores the importance of ensuring accurate drug response prediction in single-cell studies to derive\nreliable mechanistic insights.\nMinor Concerns:\n1. The study omits critical evaluation of technical confounders, including sequencing platform variability,\nread depth heterogeneity, and batch effect correction methodologies. This analytical gap precludes definitive\nattribution of performance differences to model capability versus data artifacts, thereby limiting assessment\nof real-world applicability.\nResponse: Thanks for the comments. We have added the following analyses to address the comments:\n1) Regarding sequencing platform variability, in our revised manuscript, we explicitly examined model\nperformance across six distinct platforms represented in our primary data collection: 10x Genomics (n=27),\nBD Rhapsody (n=3), MARS-seq v2.0 (n=2), DropSeq (n=2), inDrop v3 (n=1), and SMART-seq v4 (n=1)\n(Supplementary Table 1). Both layer-freezing and fine-tuning with LoRA were evaluated. Our results\nshowed that scFoundation consistently achieved the best performance across all protocols based on F1 scores.\nFor example, under layer-freezing training, scFoundation achieved mean F1 scores of 0.989 in BD Rhapsody,\n0.963 in MARS-seq v2.0, 0.975 in 10x Genomics, 0.952 in DropSeq, 0.937 in inDrop v3, and 0.916 in\nSMART-seq v4. Other metrics such as AUROC and accuracy also supported these findings (Supplementary\nNote 6; Supplementary Fig. 19a). Under fine-tuning, we observed a similar pattern where scFoundation\nremained the best-performing model across all protocols (F1 scores: BD Rhapsody: 1.000; MARS-seq v2.0: 0.989;\n10x Genomics: 0.958; DropSeq: 0.893; inDrop v3: 0.749; SMART-seq v4: 0.654). Notably, fine-tuning\nimproved performance for BD Rhapsody and MARS-seq v2.0 datasets by 1.1% and 2.6%, respectively, but\ndecreased performance for the other four protocols (10x Genomics: -1.7%, DropSeq: -5.9%, inDrop v3: -\n18.8%, SMART-seq v4: -26.2%). This suggests that fine-tuning can enhance model adaptation to certain\nprotocols but may introduce overfitting or reduce robustness in others. Additionally, Geneformer consistently\nranked as the second-best model in five out of six protocols, highlighting some stability in its cross-protocol\ngeneralization, though all models struggled with inDrop v3 and SMART-seq v4 (Supplementary Note 6;\nSupplementary Fig. 19b).\n2) Regarding read depth heterogeneity, our investigation into the effect of read depth on foundation model\nperformance is constrained by data availability. Our curated scDrugMap datasets are predominantly sourced\nfrom NCBI GEO, where raw sequencing files and standardized per-cell QC metrics (e.g., true read depth) are\nrarely available. While some datasets mention sequencing depth in their original publications, their provided\nexpression matrices do not provide measures of read depth. Consequently, in the absence of standardized per-\ncell QC metrics or raw sequencing files, it is not feasible to systematically evaluate read depth effects across\nour diverse dataset collection.\n3) Regarding batch effects, we applied Harmony on the top 2000 highly variable genes in each dataset to\ncorrect batch effects (Supplementary Note 2; Supplementary Figs. 7-8). With the batch corrected data, we\nthen conducted comprehensive benchmarking under two evaluation scenarios: pooled-data evaluation\n(Supplementary Fig. 7) and cross-data evaluation (Supplementary Fig. 8). In both scenarios, we also\nimplemented two model training strategies (layer freezing and fine-tuning using LoRA). Our results showed\nthat performance with batch correction was largely consistent with the results obtained without correction.\nTop-performing models such as scFoundation, scGPT, and CellPLM remained stable or exhibited only minor\nimprovements after batch correction (Supplementary Fig. 7a). For example, in tumor tissue, the F1 scores\nwith and without batch correction were: scFoundation = 0.998 vs. 0.998; scGPT = 0.939 vs. 0.939; and\nCellPLM = 0.940 vs. 0.936. This phenomenon is also observed in other models, such as tGPT, Geneformer,\nand UCE, exhibited comparable performance before and after batch correction. For instance, in targeted\ntherapy datasets, tGPT had F1 scores of 0.794 (with batch effects) vs. 0.756 (after correction), and Geneformer\nscored 0.751 vs. 0.700. Overall, these results suggest that foundation models tend to capture biologically\nmeaningful features that are relatively robust to batch-specific variation, supporting the validity of our\nbenchmarking results.\n2. The manuscript contains spelling errors (such as \"cance\", \"componennts\"), inappropriate phrasing (e.g.,\n\"enhance limitation\"). Additionally, some paragraphs contain repetitive sentence patterns, which should be\nrevised for conciseness and clarity.\nResponse: Thanks for the comments. We have carefully revised the manuscript to correct the identified errors.\nSpecifically, we corrected “cance” (line 121, “componennts” (line 481), and other typographical issues, such\nas the PBMCs acronym in line 152. The phrase “enhance limitation” has been corrected to “mitigate limitation”\n(line 441), and the redundant use of “layer” in line 162 has been removed. We also reviewed the manuscript\nmore broadly to improve conciseness, reduce repetitive sentence patterns, and enhance overall clarity.\n3. Several figures show missing or unclear information. In Fig.1 and Fig.6, legends are incomplete (e.g.,\nunclear what color shades or line colors represent). Fig.2 and Fig.3 lack stylistic consistency and statistical\nsignificance markers. In Fig.4c, the UCE group fails to distinguish between resistant and sensitive cells.\nResponse: Thanks for the comments. We have revised the figures to address these issues:\n1) For Fig. 1c and Fig. 1d, we updated the legend to clarify the meaning of color shades (lines 597-600).\nDarker shades now denote the number of sensitive cells, while lighter shades represent the number of resistant\ncells.\n2) In Fig. 6, we added a complete legend to explain the line colors (lines 631-633), with each color\ncorresponding to a specific category within the data type.\n3) For Fig. 2 and Fig. 3, we improved stylistic consistency and added statistical significance assessments.\nStatistical comparisons are now presented in Supplementary Figs. 5 and 6, with exact p-values provided in\nSupplementary Table 4 and detailed explanations in Supplementary Note 1. Briefly, we used Student’s t-\ntests to evaluate performance across tissue types, drug classes, and cancer types. In pooled-data settings,\nscFoundation consistently outperformed other models under both layer-freezing and fine-tuning strategies. In\ncross-dataset evaluations, results were more context-specific: for example, LLaMa3 outperformed scGPT in\ncell line and PBMC datasets under layer-freezing, while tGPT surpassed scGPT in PBMCs under both layer-\nfreezing and fine-tuning. These findings indicate that scFoundation remains the most robust in pooled-data\ncontexts, whereas alternative models may offer advantages in specific cross-dataset scenarios.\n4) For Fig. 3a, we replaced the violin plot with a box plot to ensure stylistic consistency with Fig. 3b.\n5) For Fig. 4c, we acknowledge that UCE fails to distinguish resistant from sensitive cells. To avoid\nmisinterpretation, we removed the corresponding text and added the UMAP visualization of all other methods\nin Supplementary Fig. 18.\n4. Line 102 contains an incorrect attribution to scFoundation.\nResponse: Thanks for the comments. We have corrected the citation from #16 to #15 (line 102).\n5. In Figure 1C-D, the 'Number of cells' bar charts employ graduated color shading within categorical\nsegments (e.g., tissue/drug/cancer types). The manuscript does not define what these intensity variations\nrepresent—whether hierarchical substructures, response status distributions, or data source heterogeneity.\nClarification should be provided in the figure legend or methods to ensure interpretability.\nResponse: Thanks for the comments. As clarified in our response to Comment #3, we have revised the legends\nfor Fig. 1c and Fig. 1d to explicitly define the meaning of the color shades (lines 597–600). Specifically,\ndarker shades represent the number of sensitive cells, while lighter shades represent the number of resistant\ncells. To provide additional context, the primary data collection comprises 36 scRNA-seq datasets with a total\nof 326,751 single tumor cells, including 162,957 (49.87%) sensitive and 162,794 (50.13%) resistant cells\n(Supplementary Table 2). The validation collection consists of 24 datasets with 168,486 cells, including\n80,237 (47.62%) sensitive and 88,249 (52.38%) resistant cells (Supplementary Table 2). These clarifications\nensure that the meaning of the shading is transparent and that the bar charts in Fig. 1c–d can be accurately\ninterpreted.\nWeb server Concerns:\n1. The \"README\" part is insufficient and requires clear, step-by-step instructions for using the tool.\nResponse: Thanks for the comments. We have substantially revised the README section to provide clear,\nstep-by-step guidance for users. The updated instructions now walk through the full workflow, including\nmodel selection, input file requirements, data upload and preprocessing, prediction modes (fixed embeddings\nand fine-tuning), and downloading results. To improve usability, we also added a downloadable example input\nfile and clarified the expected format for gene expression matrices, including cell barcodes, condition labels,\nand gene counts. The README now explains adjustable training parameters and shows how progress and\nmetrics are displayed in real time. These updates make the README far more comprehensive, ensuring users\ncan easily follow the workflow and successfully run scDrugMap without coding expertise.\n2. In the \"Tool\" section, users can upload all input CSV files. It would be helpful to provide a sample file to\nclarify the expected forma.\nResponse: Thanks for the comments. We agree that providing a sample input file will greatly help users\nunderstand the expected format. In the revised web server, we have added a downloadable sample CSV file\nin the “Tool” section. This example includes the required column names, data types, and example entries,\nclearly demonstrating how input data should be structured. We have also updated the README/user guide to\nexplain the meaning of each column and any formatting requirements (e.g., gene names, cell ID). Together,\nthese additions will make it much easier for users to prepare their own input files and avoid formatting errors.\n3. Some single-cell drug resistance datasets are available for download in \"Data\" section. To improve\ninterpretability, visual summaries (UMAP/t-SNE) are recommended. Also, the download button beside the\ndataset is currently not working.\nResponse: Thanks for the comments. We have addressed both issues in the revised web server. First, to\nimprove interpretability, we added visual summaries (UMAP/t-SNE plots) for each dataset in the “Data”\nsection. These plots provide users with an overview of the cellular heterogeneity and resistance versus\nsensitivity distribution in each dataset, making it easier to understand the data before downloading. Second,\nwe fixed the download functionality so that the download button beside each dataset is now fully operational.\nUsers can now directly access and download the datasets without technical issues. These improvements\nenhance both usability and interpretability of the “Data” section.\nReviewer #3 (Remarks to the Author):\nThe manuscript presents scDrugMap, a benchmarking framework evaluating large foundation models for\nsingle-cell drug response prediction. Here are the detailed comments:\n1. While the paper appropriately notes that accuracy is suitable for balanced data (lines 502-503) and confirms\nthe drug sensitivity/resistance data is balanced (lines 119-121, 129-130), there appears to be an inconsistency\nas F1 scores (more appropriate for imbalanced data, lines 503-504) are predominantly reported. Furthermore,\nalthough accuracy is repeatedly mentioned as being tested (lines 143-145, 160-161, 240-242), the results\nfigures lack this metric, creating a discrepancy with the Results section.\nResponse: Thanks for the comments. To address this, we have now included accuracy plots in\nSupplementary Figures 1–4, alongside the F1 scores. Overall, we find that models with strong F1 scores also\nachieve correspondingly high accuracy, supporting consistency across metrics. Below we provide detailed\nresults:\n1) In the pooled-data evaluation using layer-freezing training method, scFoundation, scGPT, and CellPLM\nachieved the highest mean F1 scores, 0.998, 0.939, and 0.939, respectively, and also ranked as the top three\nin accuracy, with 0.997 for scFoundation, 0.923 for scGPT, and 0.923 for CellPLM. In contrast, the lowest-\nperforming models were scBERT (F1: 0.789, Accuracy: 0.725), LLaMa3 (F1: 0.760, Accuracy: 0.728), and\nCellLM (F1: 0.749, Accuracy: 0.679) (Fig. 2a; Supplementary Fig. 1a).\n2) In the pooled-data evaluation using fine-tuning training method, scFoundation, scGPT, and CellPLM\nachieved the highest mean F1 scores, 0.990, 0.963, and 0.959, respectively, and also ranked as the top three\nin accuracy, with 0.987 for scFoundation, 0.954 for scGPT, and 0.948 for CellPLM. UCE (F1: 0.869,\nAccuracy: 0.838), CellLM (F1: 0.732, Accuracy: 0.676), and scBERT (F1: 0.708, Accuracy: 0.655) (Fig. 2b;\nSupplementary Fig. 1b).\n3) In the cross-data evaluation using layer-freezing training method, scGPT was the only model that performed\nwell, achieving the highest mean F1 score (0.858) and the highest accuracy (0.590), although the accuracy\nwas only slightly above baseline (Fig. 3a; Supplementary Fig. 2a).\n4) In the cross-data evaluation using fine-tuning training method, UCE and Geneformer achieved the highest\nF1 scores (0.774 and 0.701, respectively), indicating relatively good but not outstanding performance. In terms\nof accuracy, however, CellLM ranked first (0.816), while UCE (0.739) and Geneformer (0.749) were in the\nmiddle range. This discrepancy between F1 score and accuracy emphasize the needs to report comprehensive\nmetrics to evaluate model performances, especially when data are unbalanced (Fig. 3b; Supplementary Fig.\n2b).\n5) To complement the five metrics reported, we further assessed area under the precision–recall curve\n(AUPRC) on datasets with resistant-class prevalence >70% (Data 23: 84.49% resistant; Data 32: 79.85%\nresistant; Supplementary Table 2). All models exceeded baseline AUPRC. Pooled-data training consistently\noutperformed cross-data training. For example, in pooled-data scenario, for Data 23, scFoundation, scGPT,\nand CellPLM achieved AUPRC >0.7. For Data 32, scFoundation, scGPT, and CellPLM also outperformed\nothers (Supplementary Note 9; Supplementary Fig. 22).\nIn summary, we have clarified this inconsistency by adding accuracy plots and providing complementary\nAUPRC analyses for unbalanced settings. Together, these results demonstrate that models performing well by\nF1 generally also perform well by accuracy, and that pooled-data evaluation yields consistently stronger\nperformance under class imbalance.\n2. While the paper provides a comprehensive comparison of model performance across various scenarios\nusing metrics like F1, the underlying reasons for the observed performance differences (e.g., pretraining\nstrategy, model architecture, or dataset characteristics) remain unexplored; a deeper analysis of these factors\nwould significantly strengthen the mechanistic understanding of why certain models outperform others in\nspecific contexts, ultimately providing more actionable insights for readers seeking to select or develop\nappropriate models for their applications.\nResponse: Thanks for the comments. We agree that exploring the underlying reasons behind the observed\nperformance differences can provide important mechanistic insights. To address this, we conducted an in-\ndepth analysis across all compared models (tGPT, scBERT, Geneformer, CellLM, scFoundation, scGPT,\nCellPLM, UCE, LLaMA3-7B, and GPT-4), and the results are summarized in Supplementary Table 6.\nOur benchmarking results show that scFoundation achieved the best overall performance, may primarily due\nto its domain-specific, large-scale pretraining on single-cell transcriptomic data, which allows it to capture\nfine-grained gene–gene interactions and cell-state relationships. scGPT and UCE also performed well, likely\nbenefiting from specialized architectures and training corpora that integrate biological priors and task-specific\nobjectives. In contrast, scBERT and CellLM showed lower performance, which may due to their pretraining\ndatasets that were smaller and less diverse, limiting generalization to our test scenarios. GPT-4 ranked lowest,\nconsistent with its general-purpose design, while highly capable in natural language processing, it lacks\nexposure to structured biological expression data and domain-specific vocabulary during pretraining, resulting\nin weaker performance in this specialized task.\nOverall, these results suggest that performance in single-cell drug response prediction tasks is strongly\ninfluenced by three factors: (1) the domain relevance of pretraining data, (2) the scale of pretraining datasets,\nand (3) the architectural design of the models. Models specifically pre-trained on large-scale single-cell\ntranscriptomic data (such as scFoundation, scGPT, and UCE) consistently perform well, offering broad\ncoverage of cell types and states while effectively capturing complex gene-gene and cell-state relationships\nthrough biologically informed vocabularies or multi-omics fusion strategies. By comparison, although\nGeneformer possesses strong embedding capabilities, its lack of specialization for single-cell heterogeneity\nleads to slightly lower performance than these domain-optimized models. Models trained on smaller or less\ndiverse datasets (such as CellPLM, scBERT, and CellLM) face significant generalization challenges,\nespecially when applied to unseen cell types or conditions. For general-purpose large language models (such\nas LLaMA3-7B and GPT-4), the absence of pretraining directly related to gene expression and single-cell\nbiology results in markedly lower performance, underscoring the critical role of domain-specific knowledge\nin bioinformatics prediction tasks.\nIn addition to Supplementary Table 6, we have now included a Model Recommendation Guide\n(Supplementary Table 5), which outlines the optimal scLLMs for each data type across four evaluation\nscenarios: (1) pooled-data with layer-freezing, (2) pooled-data with fine-tuning, (3) cross-data with layer-\nfreezing, and (4) cross-data with fine-tuning. We hope this guide provides readers with actionable insights for\nselecting or developing models most appropriate for their specific tissue, cancer type, or drug-response\napplication.\n3. The statement that \"other models showed limited separation\" in Fig. 4 would be more informative if\naccompanied by corresponding UMAP visualizations of these underperforming models, as this would enable\nreaders to directly compare the separation quality across all evaluated methods and better understand the\nperformance differences.\nResponse: Thanks for the comments. To address this, we have added the UMAP visualizations for all the rest\nmodels in Supplementary Fig. 18. In details, Fig. 4 showed the two models (scFoundation and scGPT) that\nperformed well in most of the datasets and sceniros. These visualizations revealed that scFoundation and\nscGPT effectively distinguished drug-resistant and drug-sensitive cells. However, the UMAP visualizations\nother models in Supplementary Fig. 18 showed limited separation between resistant and sensitive tumor cells.\nThese additional plots allow readers to directly compare separation quality across all methods and better\nappreciate the performance differences.\n4. The choice of LoRA rank=8 lacks justification. A brief ablation study (e.g., testing ranks 4/8/16) would\nstrengthen the methodology.\nResponse: Thanks for the comments. We have conducted additional analyses using LoRA ranks of 4 and 16\nto complement the original rank 8 setting, as detailed in Supplementary Note 10 and mentioned in the\nMethods section. Overall, performance differences across ranks were modest in both pooled-data and cross-\ndataset evaluations. However, some notable exceptions emerged. In the pooled-data evaluation\n(Supplementary Fig. 23), we found that scFoundation fine-tuned with a LoRA rank of 8 achieved\nsignificantly better predictive performance in liver and pancreatic cancers compared to the same model fine-\ntuned with ranks 4 and 16. Specifically, in liver cancer, LoRA 8 significantly outperformed both LoRA 4 (p\n= 0.0164) and LoRA 16 (p = 0.0086), while in pancreatic cancer, LoRA 8 outperformed LoRA 4 (p = 0.0104)\nbut not LoRA 16 (p > 0.05). Additionally, several other models fine-tuned with LoRA rank 8 showed\nimproved performance over ranks 4 and 16 in specific contexts, even though these differences were not\nstatistically significant. Notable examples include UCE and Geneformer in acute myeloid leukemia, UCE in\ntumor tissue, scBERT in immunotherapy, and both CellLM and Geneformer in targeted therapy. These\nfindings suggest that an intermediate LoRA rank (e.g., 8) may offer a favourable balance between model\nadaptability and parameter efficiency in certain cancer types.\nBy contrast, under cross-dataset evaluation (Supplementary Fig. 24), none of the performance\ndifferences across LoRA ranks reached statistical significance. Nonetheless, models fine-tuned with LoRA\nrank 8 often performed better than those with ranks 4 or 16 in certain settings. For example, scFoundation in\ntumor tissue, bone marrow aspirate, scBERT in PBMCs and multiple myeloma, UCE for breast cancer and\nlung cancer, Geneformer and scBERT for prostate cancer, and scGPT for melanoma. This indicates that while\nLoRA rank can influence fine-tuning outcomes in specific contexts, its overall effect on generalization\nperformance appears limited within the tested range. Together, these results imply that moderate LoRA\nconfigurations may be preferable in pooled-data scenarios, but LoRA rank selection may be less critical when\nprioritizing model robustness across heterogeneous datasets.\n5. Please add a summary table (e.g., \"Model Recommendation Guide\") to highlight optimal models for specific\nscenarios (e.g., UCE for generalizability). This would improve practical utility.\nResponse: Thank you for the helpful suggestion. We have added a Model Recommendation Guide as\nSupplementary Table 5, which summarizes the optimal scLLMs for each data type across four evaluation\nscenarios: (1) pooled-data with layer-freezing, (2) pooled-data with fine-tuning, (3) cross-data with layer-\nfreezing, and (4) cross-data with fine-tuning.\nTo enhance interpretability, we defined performance thresholds for inclusion: 1) For pooled-data scenarios,\nmodels with mean F1 ≥ 0.95 were considered optimal. For models with mean F1 between 0.5 and 0.95, we\nadditionally included the top-performing model (top 1) per subtask. 2) For cross-data scenarios, a more\nconservative threshold of mean F1 ≥ 0.70 was used to account for the increased difficulty in generalization.\nFor models with mean F1 between 0.5 and 0.70, we included the top two performers (top 2) per subtask. These\ncutoffs were selected based on the observed distribution of F1 scores across all tasks and aim to provide a\npractical reference for researchers selecting models under different constraints.\nBriefly, based on our benchmarking across tissues, drug types, and cancer types, we provide a Model\nRecommendation Guide for selecting scLLMs for cancer drug resistance prediction task (Supplementary\nTable 5). This guide summarizes the best-performing scLLMs across four evaluation settings: (1) pooled-data\nwith layer-freezing, (2) pooled-data with fine-tuning, (3) cross-data with layer-freezing, and (4) cross-data with\nfine-tuning. By linking model performance to evaluation scenario, tissue type, drug class, and cancer\ntype, the guide offers a practical framework for selecting or adapting models to specific research contexts.\nOverall, scFoundation and scGPT emerge as strong default choices for pooled-data applications across most\ntissues, drug classes, and cancer types. In contrast, cross-data scenarios require more careful consideration, as\nperformance tends to be more variable. In these cases, models such as UCE, tGPT, and other task-specific\narchitectures often demonstrate stronger generalization. Finally, for rare cancers or highly imbalanced datasets,\nwe recommend that researchers place particular emphasis on cross-data performance, since not all models are\nequally reliable under such challenging conditions.\n6. The triplicate repetition of post-prompt in Fig. 5a appears redundant. Please clarify whether this design is\nnecessary.\nResponse: Thanks for the comments. The repeated post-prompts in Fig. 5a are intentional and serve as a\nprompt engineering strategy to encourage the LLM to maintain a consistent output format. Repetition of\ninstructions or contextual cues has been shown in recent studies to improve model adherence to complex tasks\nby reinforcing the desired behavior.\nOur prompt design is inspired by Hou et.al.[1], which evaluated GPT4's performance on cell annotation, and\nwe adapted it to drug sensitivity tasks. Although Hou et.al.[1] did not explicitly employ repeated post-prompts,\nin our experiments we found that repetition improved output stability, minimized formatting errors, and\nreduced hallucinations. This observation is supported by the broader literature. For example, Kojima et al.\n(2022)[2] demonstrated that “chain-of-thought” prompting, which involves repeated or stepwise guidance,\nimproves reasoning and instruction adherence in LLMs. Similarly, Wei et al. (2022)[3] found that providing\nrepeated contextual signals or exemplars can stabilize model outputs and reduce variability. Microsoft Azure\ndocumentation[4] highlights that LLMs display a strong recency bias, they are particularly influenced by text\nappearing at the end of the prompt. As a result, repeating instructions at the end can help the model retain\ncritical guidance that might otherwise be forgotten or overshadowed. Webson et.et.al.[5] compared the effects of\nsingle-task and multi-task prompts, emphasized the impact of repeated prompt structure on the performance\nof different tasks and models, and provided empirical support for the design of repeated prompt\ncomprehension prompts. Shaier et.al.[6] on prompt repetition in reading comprehension tasks found that\nrepeating a question multiple times within a prompt (e.g., 3 or 5 times) can improve model focus and\ncomprehension compared to single-instance presentations[7], suggesting that training a model through repeated\nhuman feedback can enhance its ability to follow instructions. Therefore, the triple repetition of post-prompts\nwas deliberately applied to enhance robustness by reducing ambiguity and ensuring that the model consistently\nproduced outputs in the required format. This design choice aligns with recent advances in prompt engineering\naimed at better controlling large language model behavior.\nReferences:\n[1] Hou, W., & Ji, Z. (2024). Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis. Nature\nmethods, 21(8), 1462-1465.\n[2] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot\nreasoners. Advances in neural information processing systems, 35, 22199-22213.\n[3] Wei, J., Wang, X., Schuurmans, D., Le, Q., & others. (2022). Chain of Thought Prompting Elicits\nReasoning in Large Language Models. Advances in Neural Information Processing Systems (NeurIPS) 2022.\n[4] Microsoft Azure AI. (2024). Prompt engineering techniques: Overcoming recency bias. Microsoft Learn\ndocumentation. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/prompt-engineering\n[5] Webson, A., & Pavlick, E. (2022, July). Do prompt-based models really understand the meaning of their\nprompts?. In Proceedings of the 2022 conference of the north american chapter of the association for\ncomputational linguistics: Human language technologies (pp. 2300-2344).\n[6] Shaier, S., Sanz-Guerrero, M., & von der Wense, K. (2024). Asking again and again: exploring LLM\nrobustness to repeated questions. arXiv preprint arXiv:2412.07923.\n[7] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training\nlanguage models to follow instructions with human feedback. Advances in neural information processing\nsystems, 35, 27730-27744.\n7. Several critical aspects of the web interface remain unclear, significantly limiting its usability and\nreproducibility. Specifically: (1) The required format and structure of uploaded CSV files are not documented.\n(2) The model selection and fine-tuning configuration interface lacks clarity—for example, whether users can\nchoose between freezing layers or enabling full fine-tuning. (3) There is no guidance on how to set key\nparameters such as LoRA rank. (4) It is unclear how and where users can download the fine-tuned models. (5)\nThe functional distinction between the “LLM TOOLS” and “DATA BROWSER” tabs is ambiguous, as both\nappear to redirect to the same “Data” page.\nResponse: Thanks for the comments. We have substantially revised the README and web interface\ndocumentation to address each of the concerns raised:\n1) Input file format: We added a downloadable sample CSV file in the “Tool” section that specifies required\ncolumn names, data types, and example entries. The README now explains how to structure input files (e.g.,\ngene expression matrices with cell barcodes, condition labels, and gene counts) and provides formatting\nrequirements to prevent errors.\n2) Model selection and fine-tuning configuration: The updated user guide now explicitly describes the options\nfor model selection, including whether to freeze layers or enable full fine-tuning. We also explain how to\nconfigure prediction modes (fixed embeddings vs. fine-tuning) so users can make informed choices.\n3) Training parameters (e.g., LoRA rank): We added adjustable parameters such as LoRA rank, with\nrecommended default values and guidance on how parameter choices may influence model performance\n(Supplementary Note 10).\n4) Downloading fine-tuned models: Instructions are now provided on where and how users can download\nfine-tuned models after training. The interface also shows progress and evaluation metrics in real time to\nimprove transparency.\n5) Distinction between “LLM TOOLS” and “DATA BROWSER”: We clarified the functional roles of each\ntab in the user guide. “LLM TOOLS” is now described as the module for model-based prediction, while\n“DATA BROWSER” is designated for uploaded datasets.\nTogether, these revisions ensure that users have step-by-step guidance through the full workflow, from input\nfile preparation to model configuration, training, and result download, making the platform more transparent,\nreproducible, and user-friendly even for those without coding expertise.\nReviewer #3 (Remarks on code availability):\nThe provided code is accompanied by a well-written README, which is helpful for reproduction. To further\nimprove usability and ensure environment consistency, I suggest providing a Docker image or Dockerfile for\neasier setup and deployment.\nResponse: We thank the reviewer for this constructive suggestion. We have created a Dockerfile that allows\nusers to easily set up and deploy the full environment. The Docker configuration specifies all required\ndependencies and versions, ensuring reproducibility across systems. Instructions for building and running the\nDocker image have been added to the GitHub README, alongside the existing installation guide. This\naddition will make it simpler for users to deploy the tool in different computational environments."}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have a done a great job in addressing my concerns.", "reviewer_2": "The authors have revised the manuscript to respond to the reviewers’ initial feedback. These changes have greatly\nenhanced the manuscript’s clarity, methodological rigor, and biological insights. However, several issues still remain,\nincluding aspects such as novelty, clinical applicability, and reliability, as detailed below:\n1．In clinical applications, scenarios are often closer to cross-data rather than pooled-data settings. Moreover, as reported in\nthe manuscript, most models achieve an F1< 0.8 in cross-data evaluations, which may indicate limited performance when\napplied to truly independent datasets. The manuscript devotes substantial discussion to the high scores obtained under\npooled-data conditions, which could potentially mislead readers regarding the practical utility of the proposed models.\n2．The comparison between foundation models and non-foundation models should be a crucial part of the paper, but both\nits design and presentation are extremely weak. The only relevant materials, Supplementary Figs. 11–12, also have obvious\nissues: when presenting the AUROC performance of MLP, scDEAL, and DREEP across different cancer types, the box plots\nin the two figures are inconsistent. Furthermore, the comparison in Supplementary Note 3 relies entirely on the F1 score,\nwhich is insufficient to support the conclusion.\n3．In the Few-shot learning section, only one general-purpose language model, GPT4o-mini, was used. The poor\nperformance of GPT4o-mini may be due to its own characteristics (\"lightweight model\"). It is not possible to conclude that\n\"general-purpose language models lack the precision and robustness\".\nMinor Concerns:\n1.“For datasets exhibiting batch effects, we used Harmony package to correct for them”, the key parameters for Harmony\nwere not specified, and the difference between the datasets before and after batch removal is not shown, it is uncertain\nwhether the model is genuinely immune to batch effects\n2.Although the authors added fresh biopsy and PDX datasets, all the data still come from GEO, lacking completely\nindependent clinical data.\n3.The manuscript reports using Seurat v4 for data preprocessing, retaining cells with nFeature RNA > 500 and percent.mt <\n10%. However, it does not clarify whether preprocessing parameters were adjusted for different datasets (e.g., different\nsequencing technologies or tissue sources). For example, percent.mt vary across different tissue types, the appropriateness\nof applying a uniform filtering threshold requires further clarification.\n4.In Supplementary Fig. 22, the authors show AUPRC results for imbalanced datasets. It looks like for most foundation\nmodels, under both cross-data and pooled-data settings, the prediction performance is pretty poor (AUPRC < 0.7). This\nreally should be pointed out clearly in manuscript. Also, the y-axis scale in Supplementary Fig. 22 seems to have an issue.", "reviewer_3": "The authors have addressed all of my previous comments satisfactorily. I have no further concerns. I recommend\nacceptance of the manuscript.\n(Remarks on code availability)\nI tested the website provided by the authors and found it to be functional. The authors also provide a detailed tutorial, which\nis clear and easy to follow."}, "author_response": "Response: Thank you for your positive feedback and for taking the time to review our revisions. We appreciate\nyour thoughtful comments and are glad that our responses have addressed your concerns.\nResponse: Thanks for the comments. We fully agree that real-world clinical applications are better reflected by\ncross-data rather than pooled-data settings, where models are required to generalize across independent datasets\nand experimental contexts. To address this, we thoroughly reviewed and revised the manuscript to ensure a clear\nand balanced presentation of both pooled-data and cross-data findings. In particular, we strengthened the emphasis\non cross-data results, and explicitly discussed the limitations of cross-data performance to avoid any potential\nmisinterpretation of model utility in practical scenarios.\nSpecifically, we revised the Abstract section (page 1), Results section (page 6), Discussion section (page 10),\nand Supplementary Note 3 to provide deeper quantitative analysis and interpretation of cross-data performance.\nThese revisions together ensure that the manuscript accurately conveys both the current limitations and future\npotentials of foundation models, presenting a transparent and balanced discussion of their performance across\nboth pooled-data and cross-data scenarios.\nThe revised changes are shown as follows:\n(Abstract, page 1): “In the cross-data scenario, UCE for tumor tissue showed relatively better performance (mean\nF1 score of 0.774) after fine-tuning, while scGPT for tumor tissue showed the best performance (mean F1 score\nof 0.858) in a zero-shot learning setting.”\n(Results, page 6): “Given that cross-data evaluations better reflect real-world clinical applications, foundation\nmodels typically show lower absolute F1 scores (generally below 0.8) under these more stringent conditions,\nhighlighting the inherent challenge of generalizing across heterogeneous datasets. Nevertheless, they consistently\noutperform traditional architectures such as scDEAL, demonstrating greater robustness and transferability\n(Supplementary Note 3). This indicates that cross-data evaluations provide a more realistic measure of clinical\nutility: while foundation models retain a clear advantage over traditional methods, they still face important\nlimitations, emphasizing the need for future work to further enhance their generalizability and performance.”\n(Discussion, page 10): “Additionally, our study highlighted that models in pooled-data evaluation outperformed\nthose in cross-data evaluation. This indicates that single-cell foundation models (FMs) benefit from exposure to\na broader range of data within a given category. When datasets from multiple studies within the same category\n(e.g., tumor tissue) are pooled together, the broader diversity may allow the model to learn from heterogeneous\nsources for more accurate prediction. In contrast, when training is performed on each dataset separately, the model\noften struggles to generalize, as it is less able to apply its pre-trained knowledge to a narrower and more specific\ncontext. Importantly, in cross-data evaluations, which more closely reflect real-world clinical conditions, most\nmodels achieved F1 scores below 0.8, indicating that while single-cell FMs show promise, their practical utility\nremains limited when applied to fully independent datasets. This underscores the need for continued\nmethodological innovation to enhance robustness and cross-study generalization.”\n(Supplementary Note 3): “More importantly, when evaluated in cross-data scenarios, which test generalization\nacross unseen datasets, critical for real-world predictive modeling, foundation models demonstrated clear and\nconsistent superiority over all non-FMs. For example, in prostate cancer, MLP achieved F1 = 0.33, Accuracy =\n0.51, AUROC = 0.45, while scDEAL achieved F1 = 0.41, Accuracy = 0.58, AUROC = 0.50. Both models\nexhibited limited transferability and unstable performance across datasets. In contrast, fine-tuned FMs such as\nUCE (AUROC=0.68, F1=0.41), tGPT (AUROC=0.71, F1=0.62), CellLM (AUROC=0.51, F1=0.65) and\nGeneformer (AUROC=0.75, F1=0.47) achieved substantially higher cross-dataset performance, demonstrating\ntheir ability to maintain predictive strength under unseen data. Similarly, in breast cancer and lung cancer datasets,\nnon-FMs showed lower performance, with MLP yielding AUROC = 0.53 and 0.41 respectively, whereas fine-\ntuned FMs such as tGPT (AUROC=0.83 and 0.73), UCE (AUROC=0.79 and 0.77), and CellPLM (AUROC=0.90\nand 0.78) provide more accurate predictions. These findings illustrate the robust generalization of FMs—likely\nattributed to their large-scale pre-training and learned transcriptomic representations—contrasted with the\noverfitting tendencies of traditional architectures. By drug type, similar trends were also observed. In\nimmunotherapy datasets, MLP and scDEAL achieved F1 as 0.15 and 0.17, AUROC as 0.48 and 0.50, whereas\neven the weakest fine-tuned FM (scGPT, F1 = 0.38) outperformed them markedly. The strongest FMs (UCE and\nCellPLM) reached F1= 0.72 and 0.68 and AUROC= 0.73 and 70, highlighting the transfer learning advantage of\nthese pre-trained FM models. Taken together, these cross-data evaluations confirm that, although traditional\nmethods such as MLP or scDEAL can perform competitively in pooled-data conditions where training and testing\ndistributions are similar, they fail to generalize across independent datasets. Foundation models, by contrast,\nconsistently demonstrate stable, transferable performance across diverse cancer types, drug classes, and study\ncohorts.”\nResponse: Thanks for the comments. We corrected the labeling issue in Supplementary Fig. 12, clarifying that\nthe metric shown is ‘Recall’ rather than ‘AUROC’, while Supplementary Fig. 11 correctly reports AUROC\nperformance. In addition, we emphasized the comparison between FM and non-FMs in the Result section, and\nrevised Supplementary Note 3 to include additional evaluation metrics (AUROC, Precision, Recall, and\nAccuracy) alongside the F1 score. These revisions provide a more balanced and comprehensive assessment of\nnon-FM performance in comparison with foundation models.\nThe related revisions are shown below:\n(Result, page 6): “Given that cross-data evaluations better reflect real-world clinical applications, foundation\nmodels typically show lower absolute F1 scores (generally below 0.8) under these more stringent conditions,\nhighlighting the inherent challenge of generalizing across heterogeneous datasets. Nevertheless, they consistently\noutperform traditional architectures such as scDEAL, demonstrating greater robustness and transferability\n(Supplementary Note 3). This indicates that cross-data evaluations provide a more realistic measure of clinical\nutility: while foundation models retain a clear advantage over traditional methods, they still face important\nlimitations, emphasizing the need for future work to further enhance their generalizability and performance.”\n(Supplementary Note 3): “To systematically evaluate whether foundation models (FMs) outperform traditional\nmethods (non-FMs) in drug response prediction, we compared their performance with a multilayer perceptron\n(MLP), scDEAL (a deep learning model for single-cell drug response prediction), and DREEP (a statistical\nframework for inferring drug response from single-cell transcriptomic data). Both MLP and scDEAL were\nevaluated under pooled-data and cross-data scenarios.\nIn the pooled-data evaluation, MLP demonstrated the most consistent and superior performance among traditional\nmethods. Ranked by mean F1 score, MLP outperformed other non-FMs across multiple datasets, including\nmelanoma (0.875), breast cancer (0.774), lung cancer (0.737), chemotherapy (0.711), multiple myeloma (0.686),\nbone marrow aspirate (0.682), targeted therapy (0.663), acute myeloid leukemia (0.656), prostate cancer (0.649),\nand cell line (0.632) (Supplementary Figs. 11–12). In addition to F1, MLP also achieved higher AUROC,\nPrecision, Recall, and Accuracy compared with scDEAL and DREEP. For example, in melanoma, MLP achieved\nAUROC = 0.9121, Precision = 0.8415, Recall = 0.9124, and Accuracy = 0.8737, outperforming both scDEAL\nand DREEP. In contrast, scDEAL showed poor generalization in breast cancer (AUROC = 0.5087, Precision =\n0.3951, Recall = 0.1935, Accuracy = 0.5532). Similarly, DREEP achieved AUROC = 0.6957 and Accuracy =\n0.5524 in chronic lymphocytic leukemia, lower than MLP’s performance (AUROC = 0.7113, Accuracy = 0.6625).\nWhen compared to foundation models in pooled-data evaluations, MLP (F1=0.8752, Accuracy=0.8737,\nAUROC=0.9120) performed worse than five FMs in the melanoma dataset—scFoundation, scGPT, tGPT, and\nCellPLM—but outperformed several others (e.g., CellLM, Geneformer, UCE, and scBERT). Similar patterns\nwere observed in other cancer datasets, confirming that while traditional models can perform competitively under\ncertain pooled conditions, FMs generally offer superior accuracy and robustness.\nMore importantly, when evaluated in cross-data scenarios, which test generalization across unseen datasets,\ncritical for real-world predictive modeling, foundation models demonstrated clear and consistent superiority over\nall non-FMs. For example, in prostate cancer, MLP achieved F1 = 0.33, Accuracy = 0.51, AUROC = 0.45, while\nscDEAL achieved F1 = 0.41, Accuracy = 0.58, AUROC = 0.50. Both models exhibited limited transferability and\nunstable performance across datasets. In contrast, fine-tuned FMs such as UCE (AUROC=0.68, F1=0.41), tGPT\n(AUROC=0.71, F1=0.62), CellLM (AUROC=0.51, F1=0.65) and Geneformer (AUROC=0.75, F1=0.47)\nachieved substantially higher cross-dataset performance, demonstrating their ability to maintain predictive\nstrength under unseen data. Similarly, in breast cancer and lung cancer datasets, non-FMs showed lower\nperformance, with MLP yielding AUROC = 0.53 and 0.41 respectively, whereas fine-tuned FMs such as tGPT\n(AUROC=0.83 and 0.73), UCE (AUROC=0.79 and 0.77), and CellPLM (AUROC=0.90 and 0.78) provide more\naccurate predictions. These findings illustrate the robust generalization of FMs—likely attributed to their large-\nscale pre-training and learned transcriptomic representations—contrasted with the overfitting tendencies of\ntraditional architectures. By drug type, similar trends were also observed. In immunotherapy datasets, MLP and\nscDEAL achieved F1 as 0.15 and 0.17, AUROC as 0.48 and 0.50, whereas even the weakest fine-tuned FM\n(scGPT, F1 = 0.38) outperformed them markedly. The strongest FMs (UCE and CellPLM) reached F1= 0.72 and\n0.68 and AUROC= 0.73 and 70, highlighting the transfer learning advantage of these pre-trained FM models.\nTaken together, these cross-data evaluations confirm that, although traditional methods such as MLP or scDEAL\ncan perform competitively in pooled-data conditions where training and testing distributions are similar, they fail\nto generalize across independent datasets. Foundation models, by contrast, consistently demonstrate stable,\ntransferable performance across diverse cancer types, drug classes, and study cohorts.”\nIn summary, these revisions together strengthen the presentation of our FM vs. non-FM comparison. It clearly\ndemonstrates that while non-FMs can occasionally achieve competitive performance under pooled-data scenarios,\nfoundation models often offer better robustness, generalization, and predictive reliability across heterogeneous\nsingle-cell datasets.\nResponse: Thanks for the comments. We agree that relying solely on GPT-4o-mini may not be sufficient to\ngeneralize conclusions regarding the performance of general-purpose language models. In response, we have\nexpanded our few-shot learning analysis to include two additional general-purpose models from the GPT family,\nGPT-5 and GPTds (o4-mini-deep-research), to ensure a better representative evaluation across different model\nscales and capabilities. We have also revised the Results section (page 8) to interpret the observed performance\nlimitations of general-purpose language models more cautiously.\nThe results of this expanded analysis are presented in Supplementary Fig. 28 and detailed in Supplementary\nNote 12 as follows.\n“To further examine whether the limited performance observed in few-shot drug response prediction arises from\nmodel-specific factors or reflects a broader limitation of general-purpose language models (LLMs), we extended\nour experiments to include three models in the ChatGPT series: GPT5, GPTds, and GPT4o-mini. Across diverse\nevaluation categories (tissue, drug type, and cancer type), these models exhibited highly consistent performance,\nwith marginal F1 score differences. The results are shown in Supplementary Fig. 28.\nSpecifically, within the Tissue category, the F1 scores for Bone marrow aspirate are 0.4863 (GPT-5), 0.4768\n(GPTds), and 0.4585 (GPT-4o-mini); for Peripheral blood mononuclear cells, the scores are 0.5857, 0.5854, and\n0.5831, respectively; and for Tumor tissue, 0.5313, 0.5277, and 0.5186. These results show minimal variation,\ndemonstrating that these three models yield comparable prediction results. A similar pattern is observed in the\nDrug type category. For Chemotherapy, the F1 scores are 0.4741 (GPT-5), 0.4704 (GPTds), and 0.4637 (GPT-4o-\nmini); for Targeted therapy, 0.5556, 0.5528, and 0.5379; and even for the more challenging Immunotherapy class,\ndifferences remain small (0.3973, 0.3936, and 0.3906). The close alignment across these results may indicate that\ncurrent general-purpose models have limited few-shot capability in drug response prediction. Consistent trends\nare also evident in the Cancer type category. For Acute lymphoblastic leukemia, GPT-5, GPTds, and GPT-4o-\nmini achieve F1 scores of 0.5776, 0.5806, and 0.5611, respectively. For the more difficult Pancreatic cancer\ndataset, the values are 0.2182, 0.2155, and 0.1897. Similarly, for Liver cancer and Lung cancer, all three models\ndisplay similar performance levels (e.g., Liver cancer: 0.7182 vs. 0.7177 vs. 0.6893). Across nearly all tissue,\ndrug, and cancer categories, the performance obtained by GPT-5, GPTds, and GPT-4o-mini remain quite\ncomparable with minimal differences.\nIn addition to the F1-based evaluation, we also examined their AUROC scores across the tissue, drug, and cancer\ncategories to assess the few-shot performance of those general-purpose models. Herein, we observe a similar\npattern that GPT-5, GPTds, and GPT-mini achieve aligned AUROC values across most categories. For example,\nwithin the tissue category, the AUROC for Bone marrow aspirate is 0.5188 (GPT-5), 0.5225 (GPTds), and 0.4982\n(GPT-mini); for Peripheral blood mononuclear cells, the AUROC scores are 0.5242 (GPT-5), 0.5214 (GPTds),\nand 0.5093 (GPT-mini), respectively; and for Tumor tissue, the AUROC scores are 0.5212 (GPT-5), 0.5279\n(GPTds), and 0.5111 (GPT-mini). These differences are minimal generally within 1%–2%. Meanwhile, within\nthe drug type category, GPT-5, GPTds, and GPT-mini achieve AUROC values of 0.5101, 0.5085, and 0.5072 in\nChemotherapy; AUROC values of 0.5178, 0.5213, and 0.5083 in Immunotherapy; and AUROC values of 0.5194,\n0.5196, and 0.5018 in Targeted therapy. Likewise, within the cancer type category, performance remains\nconsistent across the three models. For example, GPT-5, GPTds, and GPT-mini yield AUROC values of 0.5198,\n0.5156, and 0.4900 in the acute lymphoblastic leukemia data; and AUROC values of 0.5209, 0.5159, and 0.4978\nin the pancreatic cancer; and AUROC values of 0.5120, 0.5258, and 0.5009 in the liver cancer. Across nearly all\nother cancer categories, GPT-5, GPTds, and GPT-mini present comparable performances in drug response\nprediction.\nOverall, these results suggest that general-purpose GPT models suggest that general-purpose GPT models may\nface certain limitations in few-shot learning tasks. The consistent patterns observed across the three GPT models\nsuggest that, without additional domain adaptation or fine-tuning, general-purpose language models may have\nreduced domain-specific precision for reliable single-cell drug response prediction.”\nResponse: We thank the reviewer for this valuable comment and appreciate the opportunity to clarify the batch\ncorrection procedure, its quantitative evaluation, and the interpretation of our results.\nFor datasets exhibiting batch effects, we applied the Harmony package to align data across batches (datasets).\nHarmony was implemented using its default parameters (theta = 2, lambda = 1, sigma = 0.1, nclust = 50), to\nprovide consistent integration across all evaluated datasets. To quantitatively assess the effects of batch correction,\nwe computed four widely used evaluation metrics: Silhouette Score1, 2, kBET3 (k-nearest Batch Effect Test),\nAdjusted Rand Index4 (ARI), and LISI5, 6 (Local Inverse Simpson’s Index).\nThese complementary metrics jointly characterize both global separation and local mixing across batches,\nproviding a comprehensive assessment of batch-effect removal. Specifically, the Silhouette Score1, 2 measures\ninter-batch separation, where lower or negative values indicate improved mixing between batches. kBET3\nquantifies local batch balance, with lower rejection rates reflecting more uniform distribution of batches within\nlocal neighborhoods. ARI4 evaluates the concordance between clustering results and batch labels; values near\nzero suggest that clustering is no longer driven by batch identity. LISI5, 6 assesses neighborhood diversity, where\nhigher values correspond to stronger inter-batch mixing and better integration. Together, these metrics offer a\nmultifaceted view of how effectively batch effects have been mitigated while preserving biological variability.\nThe results of these four metrics, shown in Supplementary Fig. 9c and Supplementary Note 2, show batch\ncorrection performance before and after applying Harmony to the collected data. Red bars represent their values\nbefore batch correction, and blue bars represent their values after batch correction. Across all evaluation metrics,\nHarmony effectively reduced batch effects and improved data integration. Specifically, LISI increased from\n1.0457 to 1.7502 (+67.4%), indicating enhanced inter-batch mixing and greater neighborhood diversity. kBET\ndecreased from 681.97 to 589.39 (−13.6%), reflecting a lower batch-specific bias and improved local batch\nbalance. ARI declined from 0.5081 to 0.4251 (−16.7%), suggesting that clustering became less driven by batch\nidentity. Similarly, the Silhouette Score decreased from 0.1488 to −0.0403 (−127.1%), demonstrating improved\noverlap and reduced global separation among batches. Collectively, these results confirm that Harmony\neffectively mitigated batch-related variation and achieved well-harmonized integration across multiple datasets.\nIn addition, regarding the reviewer’s comment on whether the FMs are genuinely immune to batch effects,\nwe would like to clarify that those models already incorporate architectural or pretraining strategies to enhance batch\nrobustness. For example, models such as scGPT, scBERT, UCE, and tGPT were pretrained on large-scale, multi-\nbatch single-cell datasets, with objectives explicitly designed for embedding alignment or domain-invariant\nfeature learning. These designs make them inherently resistant to technical batch variation even before applying\nadditional batch correction. That also explains why the FMs perform comparable with/without batch correction,\nas evidenced in Supplementary Fig. 9a-b.\nIn summary, our results show that Harmony-based batch correction effectively reduces batch effects, and the\nfoundation models assessed in this study possess intrinsic batch-robustness mechanisms derived from pretraining\non heterogeneous datasets.\nResponse: We thank the reviewer for this valuable comment. While it is correct that the datasets were accessed\nthrough the GEO repository, we would like to clarify that these GEO datasets were originally generated from\nindependent clinical studies conducted across different labs/institutions. Each dataset represents a distinct\npatient/sample cohort with unique clinical and experimental contexts, ensuring biological and clinical\nindependence rather than repeated sampling from a single source. In addition, we included fresh biopsy and PDX\ndatasets, which provide higher translational relevance and further diversify the data sources beyond conventional\nin vitro models. These collected datasets represent the most diversified and representative collection,\nencompassing a broad spectrum of cancer types, data generation platforms, and clinical origins, effectively\nmitigating concerns about data dependency. Moreover, these datasets represent the most comprehensive and\ndiverse publicly available clinical resources suitable for evaluating the drug response prediction capabilities of\ncurrent foundation models. We further clarified this point in the revised manuscript (page 9) to better reflect the\nindependence and representativeness of the datasets used.\nResponse: Thanks for the comments. We acknowledge that sequencing technologies and tissue sources can lead\nto differences in mitochondrial content and gene detection rates. Given that our goal in this study was to evaluate\nmodel performance across datasets in a standardized and unbiased manner, therefore, we intentionally applied\nuniform preprocessing parameters (nFeature_RNA > 500 and percent.mt < 10%) to all datasets to ensure that\nmodel comparisons were made on an equal basis. Using different thresholds for each dataset could introduce\nadditional variability that reflects preprocessing choices rather than true model performance, thereby reducing\nfairness and comparability across datasets. In addition, considering that scDrugMap integrates a large number of\nsingle-cell datasets generated from multiple technologies and tissues, it is also impractical to fine-tune\npreprocessing parameters for each dataset, as optimal thresholds are often dataset-specific and not clearly defined\nin the original studies.\nTo clarify this, we have revised the Methods section (page 16) as follows:\n“Quality control and preprocessing were conducted using Seurat v4.3.0 with uniform filtering thresholds\n(nFeature_RNA > 500 and percent.mt < 10%) across all datasets. Although mitochondrial content may vary across\ntissue types and sequencing technologies, standardized preprocessing was applied to ensure fair and consistent\nevaluation of model performance.”\nResponse: Thanks for the comments. We carefully revised Supplementary Fig. 22 and corrected the y-axis\nscaling issue. In addition, we have revised the main text (page 10) to explicitly highlight that the performance of\nmost FMs, as measured by AUPRC, remains below 0.7 for imbalanced datasets under both pooled-data and cross-\ndata settings. This indicates that prediction in imbalanced datasets remains a significant challenge for current FMs\nand calls for the development of strategies such as re-weighting, resampling, or task-specific fine-tuning.\nRelated changes are as follows:\n“Furthermore, class imbalance in cross-data scenarios can further deteriorate performance, as measured by area\nunder the precision-recall curve (AUPRC). As shown in Supplementary Fig. 22, the AUPRC values for most FMs\nremain below 0.7 under both pooled- and cross-data settings, underscoring the difficulty of reliable prediction in\nimbalanced datasets. This finding highlights an important limitation of current FMs and suggests that future work\nshould explore strategies such as re-weighting, resampling, or tailored fine-tuning to mitigate the effects of class\nimbalance. Together, these results indicate that while large-scale pooled data can help FMs leverage their broad\npre-trained knowledge more effectively, careful consideration of cross-data generalizability and data imbalance\nis essential for their translation into clinical practice.”\nResponse: We truly appreciate your kind words and thoughtful review. We’re glad the revisions have resolved\nyour earlier concerns and thank you for helping us improve the manuscript.\nResponse: We appreciate your feedback and are pleased that the website and tutorial met your expectations."}, {"round": 3, "reviewer_comments": {"reviewer_2": "I appreciate the authors for their thorough and thoughtful revision. All my comments have been adequately addressed."}, "author_response": "Response: We are grateful to the reviewers for their positive feedback and have ensured that all previously\nrequested revisions remain clearly reflected in the manuscript and supplementary materials."}]

Decision letter