GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context

Authors: Minsheng Hao, Haiyang Bian, Nan Yan, Yixin Chen, Lei Wei, Xuegong Zhang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Generative model, Transformer, Spatial Transcriptomics

Abstract:
Learning the spatial context of cells through pre-training may enable us to systematically decipher tissue organization and cellular interactions in multicellular organisms. Yet, existing models often focus on individual cells, neglecting the intricate spatial dynamics between them. We develop GeST, a deep generative transformer model that is pre-trained on the task of using information from neighboring cells to iteratively generate cellular profiles in spatial contexts. In GeST, we propose a novel serialization strategy to convert spatial data into sequences, a robust cell quantization method to tokenize continuous gene expression profiles, and a specialized attention mechanism in the transformer to enable efficient training. We pre-trained GeST on a large-scale spatial transcriptomics dataset from the mouse brain and demonstrated its performance in unseen cell generation. Our results also show that the pre-trained model can extract spatial niche embeddings in a zero-shot way and can be further fine-tuned for spatial annotation tasks. Furthermore, GeST can simulate gene expression changes in response to spatial perturbations, closely matching experimental results. Overall, GeST offers a powerful framework for generative pre-training on spatial transcriptomics.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730536258246

thread_type:
official_review

<thread_object>
creator: reviewer_brab

<content>
summary:
The authors introduce an innovative generative pre-trained model (GeST) designed to learn the spatial context of cells within spatial transcriptomics. This model ingeniously converts two-dimensional spatial data into a serialized one-dimensional sequence to accurately capture and model the intricate spatial relationships between cells. This novel approach facilitates a deeper understanding of complex tissue organizations and provides a promising direction for further research in the field. Preliminary results demonstrate the model’s effectiveness in capturing relevant spatial patterns, although further validation is required to assess its performance across diverse datasets and potential limitations in handling varying spatial resolutions.

strengths:
The authors creatively employ a generative pre-trained transformer for the first time to understand spatial transcriptomics at the single-cell level, introducing innovative methods to the field.

weaknesses:
1. The introduction could be enhanced by comparing the proposed GeST model with other relevant models like GraphGT or SpaGCN, as CellPLM, which is mentioned, differs significantly in context and isn’t directly related to spatial transcriptomics.
2. The model appears to primarily apply the Vision Transformer architecture to spatial transcriptomics with minimal modifications, suggesting a lack of substantial innovation.
3. The resolution variance among different spatial transcriptomics technologies should be more thoroughly addressed, potentially by incorporating datasets from Stereo-seq, Slide-seq v2, STARmap, and 10x Visium to provide a broader validation of the model’s utility. However, it is important to note that the resolution of 10x Visium is based on spots rather than individual cells. Does the model still perform effectively under these conditions?
3. In the ablation studies detailed in Table 3, it is unclear whether changes in the number of layers and heads simultaneously affect the window size. Clarification on how these architectural modifications impact the model’s spatial resolution would be valuable.
4. Consider the possibility of conducting an ablation study where the neighborhood  information is removed, to assess its impact on the model’s performance and spatial understanding.

questions:
1. It is advisable to try data from multiple resolutions, as different technologies offer varying levels of resolution. For example, Stereo-seq can achieve subcellular resolution, which may allow the algorithm to examine the impact of organelles on the structure.
2. When conducting biological experiments involving tissue sections, these sections are often assembled from multiple pieces. This assembly process can introduce inaccuracies that may impact the reliability of neighborhood information. How should this issue be addressed to ensure data integrity?
3. As the authors mentioned, the model is expected to perform well in predicting genes with high spatial variation. It would be beneficial to validate this conclusion using cancer datasets, which are characterized by high variability. Additionally, considering cancer datasets could be crucial for addressing key questions about the tumor microenvironment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673249339

thread_type:
official_review

<thread_object>
creator: reviewer_2o83

<content>
summary:
I'm initially rating this paper as "5: marginally below the acceptance threshold".

Summary: the paper proposes an auto-regressive generative model for spatial transcriptomic data. A notion of "order" is introduced thereby making use of (modified version of) pipelines for sequences with incremental updates. The method is evaluate on niche clustering, niche label annotation, unseen cell generation, and spatial perturbation prediction. To facilitate the generation of the final counts, a hierarchical clustering and meta-cell vocabulary is used.

strengths:
- Clear writing and explanatory figures
-

weaknesses:
- typo (not included in score): In line 236 the sentence shouldn't spot at "$g(x)$. Instead ..."

questions:
- My main question/concern is that there is no inherent order in cells located in spatial positions (as mentioned in the paper). Lines 150-160 explain a procedure to assign a "pseudo-order" to cells. This procedure contains cropping a square from the spatial data, selecting one of the anchors, and repeatedly selecting cells based on their spatial distance to the selected anchor. At least I do not intuitively understand why such a procedure should resemble "an order"?
- For evaluating the generative power of the model, in Figure. 4 metrics like RMSE and correlation are used. Was there a reason for not using the commonly used metrics for this purpose, like Wasserstein distance, MMD, EMD, etc?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703612738

thread_type:
official_review

<thread_object>
creator: reviewer_b5ci

<content>
summary:
The authors present a generative pre-trained transformer model designed for spatial transcriptomics. The authors propose strategies to tackle common challenges in applying transformer models to ST data, including a serialization strategy, cell quantization method, and spatial attention mechanism.

strengths:
- The authors adopted a clever strategy to tokenize continuous gene expression profiles into discrete cell states. In particular, this helps mitigate error accumulation in autoregressive generation, a common issue when dealing with continuous data in transformer models.

- The model demonstrates strong performance across multiple tasks, including unseen cell generation, niche clustering/annotation, and in-silico spatial perturbation analysis. This versatility showcases the model's potential as a foundation for various spatial transcriptomics applications.

weaknesses:
- The cell quantization strategy presented in the paper is not significantly different from previous strategies employed by existing methods for ST. For example, this problem of discretizing spatial data is addressed before by [1] Wen et al., [2] Yarlagadda et al, [3] Schaar et al.

- The evaluation is focused mainly on mouse brain datasets - which are known to have organized spatial structures of various distinct cell types. Evaluating model on more challenging datasets like from cancerous tissues will help solidify the work.  While spatial serialization introduces an ordinal structure, its application might overlook the full potential of irregular spatial patterns within tissues, limiting the model’s adaptability across different spatial configurations.

- The multi-level cell quantization and hierarchical loss approach are suited for well preserved mouse brain tissues. But in practice, ST data have several artifacts due to poorly preserved tissues and not very clean - transformer models for ST tend to perform relatively poorly compared to their CNN counterparts for modeling hierarchical information in the tissues.
 
- The reliance on a vocabulary to tokenize gene expression may lead to loss of subtle gene-level variations, potentially limiting the granularity of predictions, especially for rare cell subtypes.

- The model’s design does not fully account for dynamic gene-gene interactions within perturbed cells during in-silico simulations, which could lead to oversimplified, and often incorrect, biological interpretations.

-  The Spatial Attention mechanism is computationally expensive, and not optimized for long-range dependencies in large tissue sections, which may lead to biased local predictions without sufficient contextual global information. The pre-training is computationally intense and require multiple GPUs - and the authors should report how the model generalizes to non-brain tissues without sufficient available ST data.

- Authors use RMSE and Spearman correlation for evaluation, but lacks biologically relevant validation metrics, such as alignment with known cell types or tissue architectures.

- While the authors mention error accumulation in autoregressive generation, they don't provide a detailed analysis of how this affects long-range predictions or the model's stability over multiple generation steps.

[1] Wen, Hongzhi, et al. "Single cells are spatial tokens: Transformers for spatial transcriptomic data imputation." arXiv preprint arXiv:2302.03038 (2023).

[2] Yarlagadda, Dig Vijay Kumar, Joan Massagué, and Christina Leslie. "Discrete representation learning for modeling imaging-based spatial transcriptomics data." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3]. Schaar et al. "Nicheformer: a foundation model for single-cell and spatial omics." bioRxiv (2024): 2024-04.

questions:
- What is the impact of tissue preparation methods and batch effects on GEST's performance? 

How does the model,
- handle rare cell types or spatially isolated cells that may not have sufficient neighboring context? 
- perform across different tissue types beyond the mouse brain?
- compare to graph-based approaches for ST data, such as spaGCN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732846995859

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Common Questions and Concerns

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. Guided by these suggestions, we have made significant revisions and introduced new experiments. These new findings further solidify our novel and effective design for generative modeling of spatial transcriptomic data, support the generalizability of our model, and also expand the scope of our study. Below, we summarize the key updates:
1. **Extended Evaluation of Model Generalizability to Cancer Dataset**: In response to the reviewers' concern on model generalizability to other tissues, we have extended our model applications to the 10X Visium human primary liver cancer (PLC) dataset (Figures A3). Despite the challenges of irregular spatial architecture and complex cellular heterogeneity, our model outperformed other baselines in terms of generative ability (Table A1). Notably, in the tumor leading-edge region, GeST was the only method capable of recovering unseen spatial patterns of marker genes for malignant cells and fibroblasts (Figures A5).
2. **Broadened Applications of More Technologies with Various Resolutions**: Another new experiment on Stereo-seq mouse sagittal brain dataset also indicated the superior generative performance of our model (Figures A4, Table A1). Taken together, we have validated GeST's broad learning ability for Visium (multi-cell resolution), MERFISH (single-cell resolution) and Stereo-seq (sub-cell resolution), which are detailed in Section 4.1, Appendix A.1.
3. **Expanded Ablation Studies**: We have added more ablation studies to show the effectiveness of different modules.  
    - We trained a version of our model without spatial neighborhood information, which resulted in significantly reduced performance (Table A4).
    - We clarified that our cell quantization approach is an unsupervised method independent of cell-type labels. By building the meta cell vocabulary of a large size, we demonstrated the broad coverage of cell distribution in the dataset (Figure A9). Removing this strategy would cause model failure (Figure A1), highlighting its important role in mitigating error accumulation.
4. **Comparison with Additional Methods:** We conducted a niche clustering experiment using SpaGCN and expanded the comparison to include five additional methods, spanning both graph- and transformer-based approaches (Table 1).

We have carefully revised the manuscript to incorporate these updates. We also appreciate the reviewers’ interest on the GeST model’s strengths in spatial cell generation and in-silico spatial perturbation. We look forward to addressing any additional comments or feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reviewers highlighted issues with the model's novelty, as its design heavily resembles existing architectures with minimal innovations tailored to spatial transcriptomics. Furthermore, evaluations were limited to specific datasets, with insufficient exploration of more complex spatial patterns or diverse tissue types. While the authors' rebuttal addressed some concerns through additional experiments and clarifications, key questions regarding scalability, computational efficiency, and biological validation remained inadequately resolved.