A long range foundation model for zero-shot predictions in single-cell and spatial transcriptomics data

Authors: Ameya Joshi, Raphael Boige, Lee Zamparo, Ugo Tanielian, Juan Jose Garau-Luis, Michail Chatzianastasis, Priyanka Pandey, Janik Sielemann, Alexander Seifert, Martin Brand, Maren Lang, Karim Beguir, Thomas PIERROT
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Foundation models, single-cell RNA-seq, spatial transcriptomics, masked language modelling, computational biology, zero shot inference

Abstract:
Large transformers pre-trained with language model objectives have demonstrated success in multiple fields, and have tremendous potential for modeling single-cell RNA-seq and spatial transcriptomics data. However, these approaches are yet to overcome various challenges, including inductive biases that hinder generalization, artifacts and quality of the underlying data, as well as downstream evaluation pipelines that do not reflect the biological challenges in the field. In this work, we propose a new framework, sCellTransformer (sCT), that relies on a first principles formulation of the problem as well as a validation pipeline designed to evaluate models generalization through zero-shot predictions. sCT leverages a long-range convolutional-transformer architecture that is trained from unprocessed single-cell and spatial transcriptomics data. In contrast to previous works, sCT represents cells with up to 20,000 protein-coding genes, processes sets of multiple cells, and predicts about a million discretized gene expression tokens. We show that representing gene expression as discrete levels allows us to mitigate the high sparsity present in single-cell data both during training and evaluation. We present state-of-the-art empirical results on several zero-shot gene expression imputation, cell-typing, and clustering tasks in both single-cell as well as spatial domains, outperforming current foundation models.

Decision: reject
Reviewer scores: 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730054438766

thread_type:
official_review

<thread_object>
creator: reviewer_ezip

<content>
summary:
Here the authors present sCellTransformer (sCT), a transformer model for analyzing single-cell RNA-seq and spatial transcriptomics data. sCT is distinguished from previously proposed "single-cell foundation models" through its use of a convolutional transformer architecture that scales to handle all ~20,000 protein coding genes per cell. The authors apply their model to a set of imputation/clustering tasks.

strengths:
* **Architectural choices**: The authors incorporate new architectural ideas (e.g. from Linder et al. 2023) to enable the use of all ~20,000 protein coding genes with single-cell transformer methods.
* **Incorporation of spatial information**: To my knowledge, sCT is the first single-cell transformer that specifically attempts to incorporate spatial information via spatial positioning tokens.

weaknesses:
After reading the paper I have numerous major concerns that prevent me from recommending acceptance. In short, the presented results suffer from a number of common issues in recent "single-cell foundation model" papers (e.g. lack of rigorous benchmarking against current best practice methods), and it's not clear to me that sCT represent a major advance for the analysis of single-cell data. Details below:

* **Missing comparisons with non-"foundation model" baselines**: A number of recent benchmarking studies (e,g, [1, 2, 3] among others) have found that recently published transformer-based single-cell foundation models perform no better than significantly simpler (e.g. linear) baseline models at a variety of downstream tasks. However, the authors' manuscript only compares against other transformer models, thus making it impossible to assess whether the authors' method indeed represents a meaningful advancement. For for the batch effect/cell type classification tasks, how do simpler baselines (e.g. logistic regression, scANVI, etc.) compare?
* **Unclear real-world utility**: Based on the provided experimental results, it's not clear to me that the authors' method provides additional utility beyond current standard scRNA-seq analysis workflows. For example, given that (as mentioned by the authors) <6% of protein-coding genes are expressed in an individual cell on average, I'm not sure why the results presented in Table 2 are meaningful, as most of the time all the masked values will have zero measured expression. Similarly, I'm not clear on what real-world need the authors are trying to address in the "whole-cell imputation task". The authors mention that measurements from some individual cells can be missed due to misalignment with spot boundaries, but it's not clear to me why this phenomenon would lead to an entire spot dropping out. The bottom line is: what **new** analyses are enabled by the authors' approach? As-is I'm sure what sCT brings to the table compared to previous approaches, beyond (maybe) better imputation performance.
* **Preprocessing/evaluation details**: For preprocessing the authors mention only applying a log transform to raw expression values and then tokenizing via a uniform binning strategy. How do the authors handle differences in sequencing depth/library size between cells? From the description provided in the manuscript I don't see which preprocessing steps (if any) handle this issue, which could confound the authors' tokenization scheme. On a related note, the performance of some baseline methods is bad enough that I'm concerned the reported results may be an artifact of the authors tokenization/evaluation setup. For example the results for scGPT in Table 2 show mean absolute error values of 260, which seems suspicious to me given that the data has been log-transformed/binned. Are the scGPT-outputted values on a significantly different scale than the bin values calculated for sCT during preprocessing? Or as another sanity check, what do the corresponding errors look like for simple baseline of prediction zero expression or the mean expression for that gene? 

[1]: Kedzierska et al. "Assessing the limits of zero-shot foundation models in single-cell biology"
[2]: Kernfeld et al. "A systematic comparison of computational methods for expression forecasting"
[3]: Ahlmann-Eltze et al. "Deep learning-based predictions of gene perturbation effects do not yet outperform simple linear methods"

questions:
See "Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730269405283

thread_type:
official_review

<thread_object>
creator: reviewer_pjds

<content>
summary:
In this study, the authors pretrained a convolutional-transformer combined model on both single cell and spatial transcriptomics data. Quantitative comparison with previous models on the imputation and clustering tasks showed the advance of the current model. Several ablation experiments motivate specific choices in the model.

strengths:
The authors proposed several architecture-level improvements including convolutional layer and learned gene normalization. The ablation experiments on certain aspects are well-designed and provide insights into the model design. The author also extended the imputation task to the whole cell level.

weaknesses:
First of all, current downstream tasks can not support the claim of a foundation model; it is more like an imputation model. A foundation model should deal with various tasks. Also, some concepts like "a first principles formulation of the problem" are not clearly explained.

While new convolutional architecture is interesting, the authors did not verify its effectiveness by comparing it with simple MLP, and also did not clearly explain the reason behind the design. The authors also did not fully compare the different gene expression discretization methods, making it hard to show its novelty. At least the authors should refer to the previous works on the discretization methods (https://openreview.net/pdf?id=gdwcoBCMVi). Also for the imputation task, some important baseline methods are missing, such as gimVI (https://arxiv.org/abs/1905.02269), scFoundation (https://www.nature.com/articles/s41592-024-02305-7), which are the foundation and also domain specific models claimed to deal with the missing data.

Some critical details are missing for the spatial imputation experiment design. For example, how did the author use other compared methods, did these models fine-tuned the same training data? Also, the authors did not provide detailed information on held-out data in HEST, as some of the fovs are actually replicates of the training data, which may lead to the risk of data leakage.

questions:
1. Could the author further explain the concept of "a first principles formulation of the problem"? or it is better to remove this claim in the abstract.

2. What is the rationale behind the design of the new convolutional architecture? As gene expression data is nonsequential, how can the author define the "local" and "global" patterns in the data? Also, could you compare it with simple MLP to show its effectiveness?

3. Could the author further explain the difference between learned gene normalization and the original layer normalization? What is the meaning of "subsequence"? How does the learned gene normalization help the model?

4. Could the author justify the choice of the discretization method? How did you decide the bing edges? How does it compare with the previous works on discretization methods?

5. Since the expression distribution is quite different between the two data types, how can the author justify the contribution of single cell data to the spatial imputation tasks? I would like to see the performance of sCT pretrained on the spatial data only.

6. For the imputation benchmark, could the author compare with the gimVI and scFoundation, which are models that can deal with the missing data?

7. Still in the imputation tasks, did the author fine-tune the compared models on the same training data? It should be clarified in the paper.

8. Could the author provide the details of the held-out data in HEST? I suggest the author leave out FOV from new studies to avoid the risk of data leakage.

9. Since the cell type information is not used in the training process, how can the author assign cell types based on the training-free cell embeddings? 

10. Some incomplete sentences in the paper, such as "with a decay rate of ...." could be revised.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730341322660

thread_type:
official_review

<thread_object>
creator: reviewer_kxrs

<content>
summary:
The paper introduces sCellTransformer (sCT), a novel framework for modeling single-cell RNA-seq and spatial transcriptomics data using a convolutional-transformer architecture. Unlike previous approaches, sCT is designed to address key challenges such as inductive biases, data quality issues, and biologically relevant downstream evaluation pipelines. It employs a first-principles approach and aims for better generalization through zero-shot predictions. sCT processes sets of multiple cells, representing them with up to 20,000 protein-coding genes and predicting approximately one million discretized gene expression tokens. By discretizing gene expression levels, the model effectively tackles the sparsity problem in single-cell data during both training and evaluation. The framework demonstrates superior performance in zero-shot gene expression imputation, cell-typing, and clustering tasks across single-cell and spatial datasets, surpassing existing foundational models.

strengths:
1. Study the foundation models on single-cell and spatial transcriptomics data.
2. Employ convolution layers for efficiency.

weaknesses:
1. The proposed improvements in the paper are rather straightforward. Learned Gene Normalization refers to the layer normalization, Positional Embeddings refer to standard sinusoidal position embeddings. The authors stack 50 cells into a single sequence, which from my perspective, creates sudo bulks and lowers the granularity instead of modeling intercellular dependencies. It would be great if the authors could elaborate more on how the proposed strategy can help with modeling intercellular dependencies.
2. Some statements in the manuscript are not put correctly. For example, in line 231, the authors state that CellPLM and scGPT are built on MLM objective and predict raw gene expression values. In fact, CellPLM conducts library size normalization and log1p on the counts, and scGPT utilizes next token prediction with data binning. Note that the binning strategy has already been employed in scGPT. The authors should explicitly acknowledge where their approach uses similar techniques (like binning) to prior work.
3. The authors mention that *downstream evaluation pipelines that do not reflect the biological challenges in the field* in the abstract. However, in the experiments, the authors only include gene imputation and cell clustering tasks, which is quite limited. Please consider tasks like gene perturbation prediction, gene regulatory network inference, cell-cell communication, etc.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639866013

thread_type:
official_review

<thread_object>
creator: reviewer_rxfw

<content>
summary:
The authors of the paper develop sCellTransformer, a new foundation model for single-cell and spatial transcriptomics data. The sCT model overcomes existing issues in single-cell transformers, namely the limit to processing one cell at a time due to memory constraints and the lack of a principled method to model properties in single cells such as discreteness and overdispersion. To tackle said problems, sCT utilises a convolutional architecture to process multiple cell representations for better contextual information. The model also discretises the predicted gene expression to better represent the true data domain via masked language modelling. Finally, sCT introduces spatial positional encodings to infuse spatial awareness into the sequence tokens. The authors showcase the model on multiple tasks such as whole cell imputation, which is crucial to infer gene expression from neighbourhoods when array-based spatial assays fail to capture single-cell profiles in the spatial slide. The paper also tackles single-cell RNA-seq reconstruction and clustering using the model’s embeddings.

strengths:
Overall, I find the ideas covered in the paper interesting. What I particularly liked is a certain attention to biological properties of single-cell data, such as discreteness and variability, which, in my opinion, have been severely overlooked in other models. I also liked the reasoning towards contextual modelling by introducing an architecture able to handle multiple cells at the same time. Finally, I appreciate the use of positional encodings for modelling the spatial identities of cells. The performance seems pretty promising.

weaknesses:
* In lines 59-60, I find that the connection between modelling continuity and the lack of accounting for over-dispersion is not very clearly motivated. I think this is a very important point when modelling single-cell RNAseq and I am happy the authors brought it up. In general, though, it could be stated clearly why modelling continuous raw counts is a problem when trying to recover the over-dispersion in scRNA-seq. 
* From a terminology perspective, I would not define individual cell expression vectors as “gene expressions”. I would replace the term with something like “expression values” or so. 
* I find myself disagreeing with the sentence “Unlike scRNA-seq, spatial transcriptomics is a relatively new technology and publicly available data is still scarce.” Nowadays there are a lot of public spatial transcriptomics datasets out there and technologies to produce them, which makes the field so exciting. I would probably tune the claim down a bit. 
* The architecture in 3.2 is understandable and clear. However, I feel it is a bit confusing for a less expert audience what a sequence here actually is (i.e. a stack of cells), especially because this is what makes sCT different from foundation models processing one cell at a time. You describe this in Learning intercellular dependencies but I believe it would make the flow better if you define this formalisation from before.
* I think the structure of the paper could be improved. Especially, Table 1 is referenced multiple times in the methods section, but it contains a metric (MCC) that goes undefined till much later in the paper as a representation of optimality. Probably I would either refer to Table 1 only in the results or give some elements of understanding before. 
* I would not discuss results in the Table caption (e.g. in Table 2) but only in the text. 
* I appreciated the experiment in Table 3. But I think it would add value to it to add how the two models perform on gene predictions in the three sparsity levels separately. Also, Table 3’s caption does not describe the depicted evaluation metric.
* I believe the presentation of results in Figure 3 is suboptimal. I would add titles to the single plots and definitely include a UMAP plot from the real cell gene expression to show how the embedding compares with the gene expression space. As of now, saying that sCT “corrects gene expression” sounds unsupported, as correction implies visualising and comparing with the data when the batch effect is still present. I also think that clustering requires some simpler baseline (like scVI or even the plain PCA embedding of the data).

In general, I believe the paper has potential and I like the ideas at its foundation. However, I think the current presentation is slightly below the acceptance line for ICLR. I am looking forward to hearing the authors’ comments in these regards and engage in discussion.

questions:
* I also think that Table 1 is overall unclear. I think I would add more detail in the legend on how to read it. For example, are the “-“ in the first columns bullet points or “minuses” with a specific meaning? Does removing layer normalisation make the results better (0.39—>0.40)?  From this result, I would not claim it is a very useful add-on. I recommend improving the Table presentation and description since it is very important to add value to the story. 
* I personally do not find the reasons for using discretised expression convincing in section 3.3. Isn’t binning based on the raw data? How does this buffer out the problem with raw count heterogeneity among studies? 
* Could you please elaborate more on this sentence “In order to ensure a fair comparison with models like scGPT and CellPLM, which output continuous values, we bin the output predictions with the bin-edges that we calculate during preprocessing.” How are lower vs higher bin edges chosen here to assign a continuous expression value?
* “We also evaluate the effect of using multiple cells as input, by training and testing variants of sCT with one and ten cells in Fig. 9.”. Here you miss the fact that you tested with 50 as well (which eventually you use as the default value). 
* How many neighbours were selected for the KNN approach for predicting whole cell expression tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732730516320

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by authors

comment:
We sincerely thank the reviewers for their insightful comments and suggestions. We are pleased that you found our approach novel (rxFw, kxRs), interesting (rxFw), and that our evaluations are well-designed (PjDS). We particularly appreciate the recognition of our new architectural ideas (ezip).

We have responded to each reviewer individually, addressing their specific comments and questions. Several reviewers noted that the manuscript's presentation and structure could be improved. In response, we have revised the manuscript to enhance its overall clarity and organization. We also took this opportunity to update Figure 3 and improve the presentation of all tables and captions. We have also added additional UMAPs for cell-embeddings from sCT and other single-cell models in Appendix E.

Additionally, we acknowledge the reviewers' requests for additional results and baselines. We have now included standard deviations for all experiments, except for clustering experiments which do not exhibit stochasticity. We also added additional baselines for gene imputation, and cell clustering, including several bioinformatics pipelines. To enhance clarity, we have annotated the tables to highlight that sCT and other transformer-based single-cell models were evaluated under zero-shot settings. In contrast, unsupervised baselines like MAGIC and scVI were fitted on the evaluation data, while supervised baselines were trained using k-fold cross-validation. We further also show stratified detailed results on the fixed gene masking task. We are also training a version of sCT on only the spatial data as requested by Reviewer PjDS and will be updating the comments as soon as the results are available.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733281704917

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal discussion and results

comment:
We wish to thank all our reviewers again for their efforts during this discussion period.  Your criticism has helped us to improve our manuscript. In response to your critiques, we have made the following changes:

**Manuscript Clarity and Presentation**

- We have comprehensively revised the manuscript's text structure to enhance readability and comprehension.
- Figures and tables have been systematically updated:
   - Figure 3 has been redesigned
   - Tables now consistently present standard errors where statistically meaningful
   - Captions have been restructured for precision

**Experimental Validation and Baseline Expansion**

- We have significantly expanded our experimental evaluation:
    - Added UMAP visualizations for cell embeddings from sCT and comparable methods (detailed in Appendix E)
    - Introduced additional baselines for gene imputation and cell clustering, incorporating standard bioinformatics tools (Tables 1-3)
    - Annotated tables to clarify evaluation contexts:
        - Highlighted zero-shot settings for transformer-based models
        - Specified fitting procedures for unsupervised baselines
        - Noted cross-validation approach for supervised baselines

**Targeted Methodological Investigations**
- Implemented specific investigative analyses requested by reviewers:
   - Developed a sCT model trained exclusively on spatial data (addressing Reviewers kxRs and PjDS)
   - Compared this variant's performance against a model trained solely on scRNA-seq data
   - Provided a detailed comparative table (see below)

**Normalization and Embedding Techniques**
- Clarified descriptions of gene normalization and cell stacking techniques
- Demonstrated the performance advantages of cell stacking over single-cell or single-spot approaches (evidenced in new tables)

**Biological Relevance Emphasis**
- Shifted focus to more biologically meaningful evaluation metrics:
   - Presented detailed results on fixed gene masking task (Table 7)
   - Highlighted fixed gene imputation for spatial transcriptomics, emphasizing real-world biological relevance

We hope these changes address the concerns of the reviewers, and thank them for their support in improving the paper.

Sincerely,

The authors.

========

**The effect of cross-modal pre-training on spatial transcriptomics prediction**

| **Model**                            | **15% MCC (↑)** | **15% MAE (↓)** | **30% MCC (↑)** | **30% MAE (↓)** | **80% MCC (↑)** | **80% MAE (↓)** |
|--------------------------------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| **Spatial Transcriptomics (ST)**     |                 |                 |                 |                 |                 |                 |
| sCT (sc only) (zero-shot)    | 0.05 ± 0.01     | 1.40 ± 0.05     | 0.05 ± 0.01     | 1.41 ± 0.05     | 0.03 ± 0.01     | 1.51 ± 0.05     |
| sCT (sc + ST) (zero-shot)    | **0.35 ± 0.03** | 1.31 ± 0.05 | **0.34 ± 0.02** | **1.32 ± 0.05** | **0.28 ± 0.02** | **1.45 ± 0.06** |
| sCT (ST only) (zero-shot)    | 0.31 ± 0.03     | 1.30 ± 0.03   | 0.30 ± 0.04     | 1.61 ± 0.04     | 0.12 ± 0.00     | 1.78 ± 0.18     |
| CellPLM (zero-shot)                  | 0.23 ± 0.02     | 1.48 ± 0.05     | 0.20 ± 0.02     | 1.52 ± 0.05     | 0.03 ± 0.01     | 2.02 ± 0.07     |
</content>
</thread_object>

</thread>


Metareview:
metareview:
Following the concept of CellPLM, the authors introduced a framework, sCellTransformer (sCT), to scale up the training of a convolutional-transformer based foundation models for single-cell RNA-seq and spatial transcriptomics data. The reported empirical results on zero-shot gene expression imputation, cell-typing, and clustering tasks have shown improved performances over the existing similar models. 

The reviewers have commented that the proposed evaluation practice based on discrete expression at different sparsity levels can be useful for the community to better evaluate and further develop foundation models. Additional experimental results provided during the discussion phases have also demonstrated the effectiveness of sCT in imputation and cell-typing. 

The remaining major concerns is on the limited methodological contributions, and missing "biologically meaningful" experimental results considering many similar foundation models focus on perturbation prediction instead of simply evaluating the performance on imputation, cell-typing, and clustering only as they depend on mostly the learned embedding from the training data. 

The authors may consider these when they revise for future submission. Additionally, the authors need to further improve the presentation. For example, as reviewers pointed out, the claim of the 'first principles' formulation can be misleading. Also, throughout the paper, there are still format/notation/language typos or inconsistency, for example, sCT and CellCT were used in the figures in Appendix. It may cause unnecessary confusions.