Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images

Authors: Sichen Zhu, Yuchen Zhu, Molei Tao, Peng Qiu
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Gene Expression Prediction, Diffusion Model, Spatial Transcriptomics, H&E

Abstract:
Spatial Transcriptomics (ST) allows a high-resolution measurement of RNA sequence abundance by systematically connecting cell morphology depicted in Hematoxylin and eosin (H\&E) stained histology images to spatially resolved gene expressions. ST is a time-consuming, expensive yet powerful experimental technique that provides new opportunities to understand cancer mechanisms at a fine-grained molecular level, which is critical for uncovering new approaches for disease diagnosis and treatments. Here, we present $\textbf{Stem}$ ($\underline{\textbf{S}}$pa$\underline{\textbf{T}}$ially resolved gene $\underline{\textbf{E}}$xpression inference with diffusion $\underline{\textbf{M}}$odel), a novel computational tool that leverages a conditional diffusion generative model to enable in silico gene expression inference from H&E stained images. Through better capturing the inherent stochasticity and heterogeneity in ST data, $\textbf{Stem}$ achieves state-of-the-art performance on spatial gene expression prediction and generates biologically meaningful gene profiles for new H&E stained images at test time. We evaluate the proposed algorithm on datasets with various tissue sources and sequencing platforms, where it demonstrates clear improvement over existing approaches. $\textbf{Stem}$ generates high-fidelity gene expression predictions that share similar gene variation levels as ground truth data, suggesting that our method preserves the underlying biological heterogeneity. Our proposed pipeline opens up the possibility of analyzing existing, easily accessible H&E stained histology images from a genomics point of view without physically performing gene expression profiling and empowers potential biological discovery from H&E stained histology images. Code is available at: https://github.com/SichenZhu/Stem.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729711571957

thread_type:
official_review

<thread_object>
creator: reviewer_8d3n

<content>
summary:
The authors proposed a novel computational tool that leverages a conditional diffusion generative model to enable in silico gene expression inference from H&E stained images. This tool can capture better heterogeneity and stochasticity of ST data and achieve SOTA on the gene expression prediction across various datasets with different metrics.

strengths:
The authors considered a limitation I fully agreed with: gene expression cannot be gained from cellular morphology only. To solve this, proposed model learns a conditional distribution over the potentially associated gene expression profiles given the histology images, facilitating a one-to-many correspondence between the image and ST data.

weaknesses:
I think the model should be tested with larger gene panel (maybe thousands of genes) to evaluate the performance limits compared with other methods. I also want to see the evaluation other than the datasets with super-resolution spots (for example, how about the performance under Slide-seq? Stereo-seq? or Even in situ technologies?). Does the image patch size change to match the size of the spots?

questions:
1. I suggest testing the model with a larger gene panel (>1000) and compare with other methods.
2. I suggest testing on datasets generated by other platforms (Slide-seq, Stereo-seq, or in situ technologies).
3. For Figure 3, it is not enough to choose one sample with two marker gene. I would like to see more visualization results with more marker genes across different cell types. More importantly, compare these visualization results with existing methods. I would like to see the difference about cell type identification.
4. Since the authors used a transformed version of original histology images, I want to know how they transformed images.
5. How many genes did authors used for calculating MSE and MAE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730421226711

thread_type:
official_review

<thread_object>
creator: reviewer_uqtz

<content>
summary:
**Motivation:** Infer spatial resolved gene expression only from histology images
**Technical contribution:** A diffusion model based approach is proposed to predict spatial gene expression from histology images
**Strengths:** The paper is clearly written and the results support the story. The method is easy to understand and effective.
**Weakness:** The novelty of proposed method (clever application vs. novel technical contribution) is not very clear. Analysis can be expanded to more datasets.  

**Justification of score:** Overall, the paper proposes a simple and effective method, which is above acceptance threshold. I would be happy to revise my score pending clarifications on my questions from the authors.

strengths:
**Clear grounding of literature:** The introduction and related works sections are comprehensive and ground the problem well. In addition to working on better predicting gene expression, the authors also highlight problems with current simplistic evaluation framework based on Pearson correlation. They also propose new measures of correctness. Overall, the paper is clearly written.

**Comparison with recent baselines:** The authors benchmarked their model against recent baselines, such as BLEEP.

weaknesses:
**Additional evaluation**
While the authors do demonstrate their method on two datasets and organs, there is a large amount of spatial transcriptomics data available publicly [1]. I encourage the authors to try it out and see if their method works equally well for different organs, cancer vs. non-cancer, and/ or different species.

**Using pre-trained encoders for gene expression**
The authors seem to train from scratch their gene value encoder. Can they comment on why do they do this when numerous gene expression encoder foundation models are available, such as scGPT [2]? While these encoders have been trained on single cell data, they have shown promising results for bulk expression, hence showing their versatility. 

**Limited novelty**
The proposed method seems like a clever application of conditional diffusion models to the problem. Can the authors further comment on the novelty of their method and how is it different compared to the existing literature?



---
References 
[1] Jaume, Guillaume, et al. "Hest-1k: A dataset for spatial transcriptomics and histology image analysis." arXiv preprint arXiv:2406.16192 (2024).
[2] Cui, Haotian, et al. "scGPT: toward building a foundation model for single-cell multi-omics using generative AI." Nature Methods (2024): 1-11.

questions:
- Was any batch correction applied to the spatial transcriptomics data?
- Since many metrics are presented in the table, it might be easier to interpret them if authors have conventional up and down arrows next to the metric name.
- What was the rationale behind trying only UNI and CONCH, when larger models exist, such as Virchow [1]?

---
References
[1] Vorontsov, Eugene, et al. "A foundation model for clinical-grade computational pathology and rare cancers detection." Nature medicine (2024): 1-12.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668600229

thread_type:
official_review

<thread_object>
creator: reviewer_vznu

<content>
summary:
The authors propose STEM, which is a conditional diffusion generative model-based ST prediction framework. On two public datasets, the authors show that STEM outperforms the state-of-the-art ST prediction methods. The authors also show through a series of ablation experiments what factors are contributing to good performance of their model.

strengths:
I think STEM adds a valid contribution to a large suite of ST-prediction methods, which are predominantly regression-based approaches (except for BLEEP, which relies on the embedding neighborhood in the training set). The application of generative model is novel, the strength of which is clearly shown in the great performance. I also appreciate that the authors invested efforts into several ablation study factors and carefully designed the experiments to mitigate potential bias (which I don't think BLEEP did well at all). Assuming that the codebase is provided, I think this can be integrated into pratictioner's toolbox.

weaknesses:
I think there are several weaknesses that the authors can address to make it an even better contribution to the field.

- For reproducibility, the authors NEED to be more specific about hyperparameters used in STEM - I don't think any information was provided.
- This is a generative model, meaning multiple gene predictions are generated from the given histology image. While the authors say that these are averaged to yield the final expression, I would like to see an ablation or example of what each sample looks like and how they would affect all the performance metrics that we observe.
- While not required, I would like to see the authors also try with different histopatholgy foundation models, such as Virchow or H-Optimus, just for robustness
- I am slightly confused as to why TRIPLEX performance is really low. In their CVPR 2024 paper, they show that TRIPLEX outperforms HisToGene and BLEEP significantly. But it is really bad. Have authors used their model properly?
- To really show that it is indeed the *conditional generative model* part of STEM contributing to the increase PCC performance, the authors need to be careful in comparing to other SOTA baselines. BLEEP uses ResNet50, TRIPLEX uses Resnet18, whereas STEM uses UNI or CONCH. Could the difference simply be coming from difference in patch encoders?
- The authors (in line 78~80) argue that even same cell type might be in different cell types of differ in spatial locations causing different gene expression outcomes and that previous models cannot 'capture them. However, I am not sure if STEM can also address this shortcoming of the previous models, since it is also simply using patch encoder (albeit a powerful one) to summarize the image patch.
- When encoding gene count, the input is a scalar, passed through MLP? The authors need to elaborate more on what this MLP structure is, since it is confusing.
- In encoder ablation study, what does CONCH+UNI mean? Simple concatenation of both features?

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732244476325

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Reply and Updated Manuscript

comment:
We thank all reviewers for their supportive comments and constructive suggestions for this paper. We deeply appreciate reviewers' recognition of the importance of generative modeling in the task of predicting spatially resolved gene expression. We are glad to see that all three reviewers agree that our proposed method has a solid contribution, with a concrete motivation and clear presentation. We have addressed all the questions and comments raised by all three reviewers and further improved the quality of the paper. We are very grateful for all the precious suggestions and efforts made to better this work.

We have revised our paper in response to the comments and updated it in the portal. To make it easier for the reviewers to track the updates, **the changes made in the main text are highlighted in purple**. We do not highlight changes in the Appendix due to the large amount of new materials added. We summarize the major changes in the items below.


**Main text**
- **Line 340**, we clarified the number of genes used in the computation of MAE and MSE, as suggested by reviewer 8D3n.
- **Line 485-486**, we added the explanation for CONCH + UNI, as suggested by reviewer VznU
- **Line 495-498**, we added a detailed explanation for image transformations used in augmentation, as suggested by reviewer 8D3n.

**Appendix**
- **Appendix A**, we updated the plots for gene variation curves on the Kidney Visium and HER2ST dataset to include all methods for comparison
- **Appendix B**, we included details for four new ablation experiments, with the following new paragraphs as jointly suggested by all three reviewers,
    - __Large gene sets__, focus on __Stem__'s performance on a gene panel of size 1000, as suggested by reviewer 8D3n
    - __Generated Samples and Sample Statistics__, focus on how different samples contribute to the algorithm performance, as suggested by reviewer VznU
    - __Large Pathology Foundation Models__, focus on evaluating **Stem** with large-scale pathology foundation models, as suggested by reviewer VznU and uQTZ
    - __Power of Histology Image Patch Encoder__, focus on investigating the contribution of image patch encoder to the overall performance of __Stem__, as suggested by reviewer VznU
- **Appendix C**, we added a detailed explanation for neural network architecture and training procedures to improve paper reproducibility, as suggested by reviewer VznU
- **Appendix D**, we included numerical results on two new datasets with different organs, species, and health conditions from the ones considered in the main text, as suggested by reviewer uQTZ
- **Appendix E**, we added additional visualization for other cell-type-specific marker genes predictions, as suggested by reviewer 8D3n
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces STEM, a conditional diffusion generative model designed to predict spatial gene expression from histology images, achieving state-of-the-art performance across multiple datasets and evaluation metrics. The model effectively captures biological heterogeneity by learning a one-to-many mapping between histology images and spatial transcriptomics data, addressing key limitations of existing approaches. While reviewers acknowledged the paper's strengths, they highlighted areas for further improvement, including broader evaluation on datasets such as Slide-seq and Stereo-seq, testing with larger gene panels, and providing more comprehensive visualizations comparing results with existing methods. Although many of these concerns were adequately addressed by the authors, the evaluation on Slide-seq and Stereo-seq datasets remains unaddressed due to the lack of a super-resolution pathology foundation model . Overall, it is well-motivated, clearly written, and demonstrates improvements over existing baselines with comprehensive experiments and ablation studies, leading to a consensus in acceptance among all reviewers.