Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning
Authors: Ronald Xie
Kuan Pang
Sai W Chung
Catia Perciani
Sonya MacParland
BO WANG
Gary Bader
Venue: NeurIPS 2023 poster (2023)
Keywords: BLEEP
Histology
H&E
Gene Expression Prediction
Spatial Transcriptomics
Contrastive Learning

Abstract: Histology imaging is an important tool in medical diagnosis and research, enabling the examination of tissue structure and composition at the microscopic level. Understanding the underlying molecular mechanisms of tissue architecture is critical in uncovering disease mechanisms and developing effective treatments.Gene expression profiling provides insight into the molecular processes underlying tissue architecture, but the process can be time-consuming and expensive. We present BLEEP (Bi-modaL Embedding for Expression Prediction), a bi-modal embedding framework capable of generating spatially resolved gene expression profiles of whole-slide Hematoxylin and eosin (H&E) stained histology images. BLEEP uses contrastive learning to construct a low-dimensional joint embedding space from a reference dataset using paired image and expression profiles at micrometer resolution. With this approach, the gene expression of any query image patch can be imputed using the expression profiles from the reference dataset. We demonstrate BLEEP’s effectiveness in gene expression prediction by benchmarking its performance on a human liver tissue dataset captured using the 10x Visium platform, where it achieves significant improvements over existing methods. Our results demonstrate the potential of BLEEP to provide insights into the molecular mechanisms underlying tissue architecture, with important implications in diagnosis and research of various diseases. The proposed approach can significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughput analysis of histology images for both research and clinical applications.

Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning
Ronald 2,,3,
Kuan Pangl 1,2,4 Sai W. Chung Catia T. Perciani 1,5
Sonya A. MacParland 1,5 Bo Wan Gary D. 3,,
University of Toronto, Vector Institute, *University Health Network, "The Donnelly Centre, TTornno General Hospital Research Institute,
Canadian Institute for Advanced Research (CIFAR) fronald.xie, kuan. kuan.pang, sainchung, catia. percian,, gary.badery baw.eerrdrrelleeeeeee wetmaillutoronnto.com
Soayy./MaPParladduueeseerd. ca, bovang@vettorinssiitte..t
Abstract
Histology imaging is an important tool in medical diagnosis and research, en- abling the examination of tissue structure and composition at the microscopic level. Understanding the underlying molecular mechanisms of tissue architecture is critical in uncovering disease mechanisms and developing effective treatments. Gene expression profiling provides insight into the molecular processes underlying tissue architecture, but the process can be time-consuming and expensive. We present BLEEP (Bi-modal. Embedding for Expression Prediction), bi-modal embedding framework capable of generating spatially resolved gene expression profiles of whole-slide Hematoxylin and cosin (H&E) stained histology images. BLEEP uses contrastive learning to construct a fow-dimensional joint embedding space from a reference dataset using paired image and expression profiles at mi- crometer resolution. With this approach. the gene expression of any query image patch can imputed using the expression profiles from the reference dataset. We demonstrate BLEEP's effectiveness in gene expression prediction by benchmarking performance on a human liver tissue dataset captured using the 10x Visium platform, where it achieves significant improvements over existing methods. Our results demonstrate the potential of BLEEP to provide insights into the molecular mechanisms underlying tissue architecture, with important implications in diag- nosis and research of various diseases. The proposed approach can significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughput analysis of histology images for both research and clinical applications.
Code available at littttp:/// /tte//ggttttttteeee www..oooobwwwe...c 1ab/ BLEEP
Introduction
Histology imaging of whole-slide Hematoxylin and cosin (H&E) stained tissues has long been used in academic and clinical settings for research and diagnosis. It provides useful information pertaining to tissue architecture and composition at the microscopic level, which are critical for understanding disease mechanisms and developing effective treatments. Gene expression profiling is powerful tool that offers deeper insights into the molecular processes underlying tissue architecture. However,
"Co-senior author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

bulk RNA sequencing does not capture heterogeneity within one sample whereas single-cell RNA sequencing (scRNA-seq) or single-nucleus RÑA sequencing (snRNA-seq) captures heterogeneity without spatial context.
In recent years, various spatial transcriptomics methods such as Visium 22 MERFISH 4.
seqFISH+(8), STARmap(23) smFISH[6), and Targeted ExSeq[l] have emerged as a promising direction to bridge the gap between histology imaging and gene expression profiling, However, these methods are often low throughput or low content. They also tend to be time-consuming, expensive and require specialized equipment and extensive domain expertise to optimize.
With the availability of these bi-modal datasets, a unique opportumity arises to explore the possibility of predicting spatially resolved expression profiles of whole tissues solely from their histology image. ST-Net and HisToGiene are two methods developed for this purpose [1 but so far have achieved limited success. Current progress has been limited due three significant challenges. Firstly, the problem is ill-posed. While the histology images share some information with their paired spatial transcriptomics, it is likely that the image features can not be used to predict the expression of all genes and vice versa. However, expression of marker genes for cell types or subtypes (MG), highly expressed genes (HEG), and highly variable genes (HVG) should be prioritized as they tend to be the most biologically relevant candidates for disease diagnosis and drug development. Secondly, the problem is dimensionally cursed. It is estimated that typical mammalian cells express around 5,000 to 15,000 genes. Existing solutions often only predict the expression of limited panel of genes (-200) and often fail to preserve the variance and heterogeneity in the original dataset. This obfuscates the biological signals intrinsic in the original data, rendering the predictions ineffective for practical use. Thirdly, due to the developing landscape of spatial transcriptomic methods, existing datasets are prone to experimental artifacts both within one sample and across different samples, which complicates the training of expression prediction models. Furthermore, the measured gene expressions often show poor agreement with their protein profiles, which could be reflected during H&E staining.
We present BLEEP (Bi-modaL Embedding for Expression Prediction), novel bi-modal embedding framework designed to address the aforementioned challenges associated with predicting gene expression from histology images. BLEEP uses contrastive learning. which effectively aligns paired image and expression representations from reference dataset in a low-dimensional joint embedding space. Using this joint embedding space, BLEEP accurately imputes the gene expression of any query image patch by leveraging the expression profiles from the reference dataset.
We demonstrate the effectiveness of BLEEP by benchmarking its performance using a challenging human liver tissue dataset captured via the 10x Visium platform, where it significantly outperforms existing methods such as HisToGene and ST-Net in terms of the average correlation to original expressions across marker genes (MG), highly expressed genes (HEG) and highly variable genes (HVG). BLEEP also preserves heterogeneity in the predicted expression profiles and recaptures and denoises the gene-gene correlations present the original dataset.
The proposed approach alleviates the ill posed nature of the expression prediction problem by implicitly encouraging shared features be learned between image and expression modalities via a contrastive objective which could prevent modality specific information from disorienting the joint embedding space. The novel query-reference imputation process from the learned joint embedding serve to mitigate the curse of dimensionality of the expression prediction problem as the gene expression profiles of query image patches are no longer predicted independently, but rather calculated from & closest reference expression profiles in the joint space. Lastly, we demonstrate that the resulting expression prediction is resilient to experimental artifacts both within one sample and across different samples, in addition to outperforming existing solutions on the aforementioned metries.
To our knowledge, this is the first bi-modal embedding- baasdd based framework proposed for the task of expression prediction from histology images. The proposed approach has the potential to significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughht analysis of histology images for both research and clinical applications.
2

---

2
Related Works
2.1
Existing histology expression prediction approaches
Several existing approaches have shown promising results in predicting expression from histology
images including HE2RNA(20) [20] ST-Net(9), HisToGene 14, hist2mal 13]. Hist2ST and others
24
ST-Net and HisToGene are two of the most popular methods for predicting spatially resolved expression from H&E images. Both of these approaches frame the task of expression prediction as regression tasks trained in a werrrd fashion. ST-Net uses a resnet50 image encoder followed by fully connected layer where HisToGene leverages vision transformer backhone and an increased field of view.
Methods that predict tissue-level expression generally achieve good correlation but lack the ability to generate spatially resolved expression profiles (HE2RNA). Existing methods that are capable of generating spatially resolved expression predictions either not quantitatively evaluated (hist2RNA), limited in terms the predicted panel (ST-Net, Hist2ST, HisToGene), or prone to overfitting 24
Both HisToGene and Hist2ST utilize spot-spatial relations to improve performance. However, our work challenges the necessity of this information, particularly in tissues with distinct and repetitive spatial patterns like human liver tissue. The implicit assumption that spatially adjacent regions should have similar representations compared to spatially distant regions may not be beneficial for performance in such cases. Hard coding position information could also lead to overfitting in data-scarce scenarios.
2.2 Contrastive representation learning
Contrastive learning plays pivotal role in advancing the capabilities of deep learning models, particularly in recent visual language models [5] [8] 17 [5]. One notable application that has emerged from contrastive learning is the Contrastive Language-Iman Image Pretraining (CLIP) framework 15]. CLIP bridges the gap between language and vision domains by learning joint representations of paired images and textual descriptions, enabling cross-mout moosll understanding and reasoning.
BLEEP draws inspiration from CLIP with modifications to learn a similar joint embedding between spot expression profiles captured by the 10x Visium platform and their spatially paired image patch spanning roughly 55ym. However, unlike CLIP's usage setting, BLEEP directly interpolates in the joint embedding space produce expression predictions, which is unfeasible for image and text domains where domain-specific decoder required to produce the final prediction.
2.3 Query-reference imputation
The query-reference imputation of BLEEP is partly inspired by SeuratV3's [19 integration process, where the expression profiles are calculated from linear combination of the closest anchors in the reference dataset given a query. However, Seurat requires a shared expression panel across modalities to integrate, whereas BLEEP is able make spatially resolved expression predictions based on the morphological features present in the histology image alone.
3 Methods
3.1 Data and preprocessing
The dataset
3 used to train and benchmark BLEEP, HisToGene, and ST-Net consists of four consecutive micrometer thick slices of human liver tissue from neurologically deceased donor livers suitable for transplantation that were OCT embedded, frozen, sliced with cryostat and imaged using the 10x Genomics Visium platform approval from the University Health Network (REB# 14-7425-AF). After quality control, the slices contain 2378, 2349, 2277, and 2265 spots respectively. A 224 224 image patch centered around each spot roughly 55pm each side is extracted from the whole slide H&E image and paired with the
3 Samples were collected with institutional ethics

---

H&E images
H&E Image (for prediction)
Image Space
Image Encodar
Queries
Image Encoder
Contrastive Loss
Expr Encoder
Quaries the Latent Space
Expr. Latent Space
Expression Profile
Reference
Expr.
Imputed Expr.
Figure I: BLEEP achieves gene expression prediction from H&E image through (a) BLEEP learns bimodal embedding from expression profiles and H&E image patches, (b) images patch queries are projected into the joint embedding space to index the k nearest reference expression profiles, and (c) the indexed reference expression profiles are linearly combined to produce the imputed gene expressions for queries.
corresponding gene expression profile of each spot. The data is publicly available for download https://www.ncbs wttp//iwww.cii....momm.......... nlm. nih, gow
Each spot is normalized to the total count and log normalized before highly variable genes computed using the Scanpy 25 package. The union of the top 1000 most highly variable genes from each of the 4 slices the expression data of these four samples are batch corrected using Harmony [11] before one of the slices (slice #3) is randomly selected to be held out for testing.
used for training and prediction, amounting 3467 genes in total. Finally,
3.2 Learning bimodal embedding for expression prediction
As illustrated in Figure Th. the first step towards learning bimodal embedding for expression prediction to extract features from the two modalities respectively using the image encoder and the expression encoder. Given batch of B paired image patches (V E NBBI.x and normalized expression profiles (X E NBxC), where L is the image patch size and is the gene set size, image encoder Simg and expression encoder feapr project the inputs into h-dim image embeddings and expression embeddings through H, fimg (H) NBxh and H. fexpr(X) (x), We use contrastive learning to further align the latent space for H. and H
Our contrastive learning approach is inspired by CLIP [66 21,, but adjusted loss function better smooths out the loss landscape for our specific use case. Unlike natural images, multiple spots with similar expression profiles or image morphology are often expected be sampled in the same batch. The modification will prevent the model from pulling apart spots with similar expression profiles

---

Algorithm Bimodal Embedding for Expression Prediction
Input: Paired image patches (V E NBxLxx)) normalized expression profiles (X' E NBxC), image encoder fang. expression encoder feepr, temperature
Output: Joint embedding space (H,, H,,)
I: function AMOODALEBEEDDNNGUO X, fing: fexpr.:)
2: Hy: H₂ + (Vmgg(V), (X)
Image and Expression embeddings
3: sim Hy, H2) Hz H²
Paired similarity
4:
sim sim(H,, Ho) <- H. H
> Internal similarities (image) b Internal similarities (expression) & Similarity-adjusee target
sim(H,, Hz) t- H. H
5:
6: target &- softmax (ssnnax((mmm((aa)) Ha, Ha)
TT) 7: C mean(ce H, H,), target) ce(sim Ho, H target")) )) Cross entropy loss
8:
return Ho, Ha.
9:
function QUEEYRREFRRENCEAPPTTTTTNNNMM
fang- H k)
10:
V < split((maage)
> Split into image patches Project patches into embedding space > Compute distance for each query Get indices of top K matched expression from reference enference_profile + [indices]
11:
Q 4 frmo
12:
distances + dist(g'. H, .12) for q' E EQ'
13:
indices topk(distances)
14:
15:
return weighted_avggrreeeeeee cectrferrenceprrfflle
> Return prediction
within the same batch, thereby increasing the coherence of the resulting joint embedding. We use an existing implementation of this loss variant[2]] In detail, we first generate the paired similarity through sim (,, H₂) = H2 H, H. To take into account the pairs with similar morphological features or expression landscapes, we begin with calculating the internal similarities, sim HH,, H,) = H, H and sim Hz. H2) = H.H H. then a similarity adjusted target matrix is denoted as:
target (softmaz(siml (iimmnzsssimi....ee.. H2) + sim(H,.. H,))/2-T)
where is a temperature hy byperparameter. Cross entropy(ce) loss is applied to align the image features and expression features produce the final loss C:
mean(ce(sim()))) He, target) + ce(sim H²)² target"))
For BLEEP, the pretrained ResNet50 10 the image encoder and fully connected network (FCN) with an output dimension of 256 as the expression encoder, which doubles as projection head. The image features from the image encoder are passed through separate projection head to bring the two modalities to the same dimension before applying the contrastive loss similar to CLIP where the model learns pull the paired representations together while pushing other representations apart. We find that the ResNet50 image encoder with fewer trainable parameters obtained more favorable results compared various pretrained daiion-ransformes (ViT) encoders (Supplementary Table 1). Larger models in conjunction with relatively small training dataset may encourage information to be memorized in the weights of the network rather than being encoded in the projections, therefore rendering the learned joint embedding ineffective for downstream imputation for our use case.
BLEEP is trained using 4 NVIDIA V100 GPUs with the AdamW optimizer 12), batch size of 512 and a learning rate 0.001 for 1 150 epochs.
3.3 Eneer-Referennee imputation
As illustrated in Figure Пp. the process starts by first splitting the H&E image into N small image patches to be encoded by the trained image encoder. Once the image patches are represented in the joint embedding space, the à nearest expression profiles from the reference are selected based on their proximity (by Euclidean distance) in the joint embedding space to each patch. Finally, the expression profiles of the query patches are imputed as a linear combination of the selected expression profiles in the reference C. Refer to supplementary materials for implementation details.
5

---

Table 1: Average correlation of predicted expression for 8 marker genes derived from Andrews et al. 2 (MG), top 50 most highly expressed genes (HEG) and top 50 most highly variable genes (HVG) compared to ground truth expressions on held out dataset.
Method
MG
HEG
HVG
HisToGene
0.097=0.015
0.072:10.018
0.071
ST-Net
0.099=0.020
0.126:t0.005
0.091=0.007
BLEEP
0.217=0.002
0.175:0.016
0.173=0.011
Table 2: Predicted gene expression values with top 5 correlations with original profile for each method from one representative replicate.
HisToGene
ST-Net
BLEEP
Gene Name
Gene Name
r
Gene Name
r
CYP3A4
0.549
CYP3A4
0.549
CYP3A4
0.741
CYP1A2
0.542
CYP1A2
0.532
CYP1A2
0.681
GLUL
0.488
CYP2E1
0.530
CYP2E1
0.675
CYP2E1
0.330
GLUI
0.463
GLUI
0.656
FABPI
0.328
SLCO1B3
0.375
FABPI
0.503
4 Experiments
4.1 BLEEP predicts spatially resolved gene expression profiles that correlate well with original expression
Table D shows the performance of HisToGene, ST-Net and BLEEP for predicting marker gene set (MG) derived from literature by Andrews et al. (2). the top 50 most highly expressed genes (HEG) and the top 50 most highly variable genes (HVG). The BLEEP predicted expression profiles show the highest correlation with ground truth across all three gene sets, achieving an increase of 120%, 39% and 90% in value across the three gene sets, respectively, compared to the second scoring method.
Furthermore, Table 2 shows the top 5 predicted genes from each method. We observed that the most well-predicted genes relatively consistent across methods. These genes known to be spatially zonated, with CYP3A4, CYPIA2, CYP2E1, GLUL being pericentrally zonated and FABP1 being periportally zonated. We also observe that BLEEP consistently achieved higher correlation values for these genes compared to HisToGene and ST-Net, further demonstrating the effectiveness BLEEP in expression prediction.
Despite the good prediction select genes in Table we note that the overall absolute correlation in Table [ remains low, highlighting the difficulty of the prediction task for most genes. The low scores could be attributed to several causes, including the expression of certain genes being poorly correlated with morphological features; the poor detection of certain genes by the Visium platform causing their expression to be less predictable; and experimental artifacts that could introduce non-biological variance to the data independent of the image.
4.2 BLEEP retains biological heterogeneity
Figure 2 Thighlihts the key advantage of BLEEP compared to supervised regression-based approaches such as HoGneGene and ST-Net. We observe that while BLEEP only narrowly outperforms HisToGene and ST-Net in correctly predicting the mean expression of genes within one sample, both HisToGene and ST-Nee fail to recapitulate the variance of the genes being predicted.
While the variance of BLEEP expression predictions is in general underestimated, they still maintain sufficient biological heterogeneity particularly when the predicted expressions are plotted in fixed scale with the original expression are available (Supplementary Figure 1). Both HisToGene and ST-Net fail resemble the original expression, likely due to the curse of dimensionality of the prediction task, resulting in the model
3 Additional figures depicting the spatial expression of other genes

---

BLEEP
HisToGene
ST-Net
Gene counts proportion
Reference
Inferred
Cene variation proportion
35.00
Ranked gene
Figure 2: Predicted expression profiles compare with reference expression profiles normalized by gene count means (Upper) gene count variance (Lower).
Original
BLEEP
HisToGene
ST-Net
Variable Scale
Fixeid Scale
Figure 3: Original and predicted spatially resolved expression levels for CYP3A4 overlaying the H&E image, visualized with variable (Top) and fixed (Bottom) color scale.
learning mean expressions shorteut, with each gene having roughly the same variance regardless of their mean expression, which undesirable.
Figure T demonstrates the effectiveness of BLEEP in preserving gene-gene correlations (GGCs) as further testament to its ability to preserve relevant biological heterogeneity.
4.3 BLEEP inferred expression robust to experimental artifacts and batch effects
Supplementary Table 2 illustrates the clustering statistics of the unsupervised clusters resulting from predicted expression profiles. When resulting unsupervised clusters are projected onto histology image in Figure 5 we make two observations. Firstly, perhaps unsurprisingly, all three methods are robust to experimental artifacts within one slice (The red regions) as the histology image surrounding the region was not affected. The low-quality regions are characterized by lower-than-
7

---

Original
BLEEP
HisToGene
ST-Net
Figure 4: Gene-gene correlation heatmap calculated using the predicted expressions for each method.
normal number of counts detected and higher-than normall fraction of mitochondrial genes. Their expression profiles can be rescued by any method that predicts expression from the image. Secondly, all three methods are capable of capturing the zonation patterns across the tissue slice reasonable extent. While HisToGene has the best clustering agreement out of the three methods according to Table (Supplementary Table 2) with BLEEP and ST-Net following close behind, for this particular dataset, the clustering metrics such as NMI and ARI are likely not accurate measures of prediction quality because the hepatocytes already closely resemble each other and lie along a continuous gradient. Hence, the resulting clusters will be very sensitive to the choice of the clustering method or the hyperparameters of the method.
However, unlike HisToGene and ST-Net, BLEEP the least prone to potentially introduce batch effects between samples during the prediction process owing to the imputation strategy. We see from Supplementary Figure that both ST-Neet and especially HisToGene predictions are a bit out of distribution when their predicted expression profiles are plotted alongside that of all the reference expression profiles.
4.4 BLEEP ablation experiments
Here we present the ablation experiments conducted in Table From these experiments we make a few important observations:
The choice K during the query-reference imputation process influences the prediction quality quite negatively when low value is selected (K 10). Values of K above our default value could provide some small improvements to correlation of the resulting predicted expression values for the HVG and HEG gene sets, but the differences were not pronounced. This is inline with what might expect from taking the pscudo-bulk of the top K most likely expression profiles given any query image. However, the MG gene set did not show much improvement with increasing K. Furthermore, doing so may carry a trade off of further systematically deviating from the original variance of the dataset due to the increased averaging effect. With this in mind, we feel our default value of K 50 remains adequate.
We also observe that the most similar match between query and reference is usually not the best prediction (as seen from the 3rd row the ablation table and indirectly 4th row when predictions are weighted by their similarity). We suspect the gap may close to some degree as the reference grows further size, but in general some amount of averaging is desired for query-reference imputation to remove some noise intrinsic in the Visium platform. In addition, profile will perfectly match that of the query image, averaging multiple profiles is required to more appropriately describe the query expression profile. Nevertheless, we also highlight the possibility of genuine biological signals being averaged out, which is an important consideration to be further investigated.
reference expression
Smoothing the contrastive loss objective to take into account patch similarity showed modest increase in performance. The gain in performance may be due to the fact that relaxing the contrastive objective more compatible with the similarity based inference strategy. The smoothing may help lessen the extent similar references pushed apart in embedding space during training, resulting in improved

---

Original
BLEEP
HisToGene
ST.Net
Figure 5: Leiden clusterings for original and predicted expressions overlaying the H&E image. Image-only expression predictions are invariant low quality regions (red) during actual experiment.
Table 3: BLEEP ablation experiments. Variables tested include: smoothed objective versus the original CLIP objective, choice of top K and the method of aggregation. Pearson correlation values reported from the marker gene set (MG), the top 50 highly variable genes (HVG) and the top 50 highly expressed genes (HEG). The correlation values show the average of 3 replicates. Accompanied uncertainty values denote the maximum difference from the mean of the 3 replicates.
Average Pearson Correlation
Smoothed Obj.
K
Aggregation
MG
HVG
HEG
Yes
10
average
0.179 + 0.020
0.146 = 0.008
0,148 0.022
Yes
100
average
0.215 + 0.011
0.180 0.011
0.181 0.015
Yes
simple
0.079 + 0.032
0.075
0.084 0.022
Yes
50
weighted
0.186 + 0.018
0.161
0.157 0.026
No
50
average
0.209 0.017
0.165
0.170 +
Yes
50
average
0.217 + 0.002
0.175 0.016
0.173 + 0.011
querying of the top K most likely expression profiles given query image patch during the inference stage.
Discussion and Conclusion
In this study, we introduced BLEEP (Bi-modaL Embedding for Expression Prediction), novel frame- work for predicting gene expression from histology images. BLEEP constructs joint embedding space from paired image and expression features reference dataset and subsequently utilizes k most similar expression profiles in the joint space to impute the expression for any given image query. To the best of our knowledge, this the first bi-modal embedding- based framework proposed for the task of expression prediction from histology images. We also show that the query-referee

---

imputation process is effective and well suited for the task of expression prediction from the bi-modal joint embedding.
We demonstrated that BLEEP effectively addresses three major challenges in the H&F image to expression prediction task. Firstly, the ill-posed nature of the problem is alleviated by the proposed joint image and expression embedding space, optimized using contrastive learning objective inspired by CLIP. This encourages the shared features between the two modalities be preferentially encoded in the joint space. Secondly, the curse of dimensionality is tackled by the query-reference imputation process, which predicts the entire expression profile joindy through linear combination rather than predicting each individual gene separately using supervised regression tasks. Lastly, we showed that BLEEP's expression prediction is resilient experimental artifacts, both within a single sample and across different samples.
We further observed that the correlation matrix of BLEEP's predicted expressions only captures existing patterns in gene-gent correlations (GGCs) but also accentuates more subtle positive and negative GGCs. This could be attributed to the ability of BLEEP's imputation process to average out the noise intrinsic in the 10x Visium platform, thereby increasing the absolute values of these correlations. This consistent with the results presented in Figure 2 where the predicted gene expressions by BLEEP exhibit lower variance compared to the original dataset. This highlights BLEEP's capability combine expression profiles of similar-looking spots in the dataset and generate expression profiles with reduced noise and enhanced biological signal compared to the original dataset. Ongoing work is evaluating BLEEP's ability to detect new spatially resolved gene modules in response to this observation.
However, an alternate explanation for the observed results could be that averaging during imputation removes genuine, abrupt biological signals, resulting in artificially smoothed expression patterns. It possible that this may be true for certain genes, especially ones that correlate poorly with image features. Increasing the size of the reference dataset may mitigate this issue by reducing the distance between any query patch and the reference expression profiles. In practice, once the expression encoder is trained, it can be quickly used to integrate new datasets to the shared embedding space as reference. Furthermore, imputed expressions from query datasets could be subsequently integrated, making the process of improving BLEEP prediction natural.
Nevertheless, the improvement offered by BLEEP over existing methods is substantial. We achieved significantly higher correlation with actual expression profiles across the marker gene set (MG), top 50 highly expressed genes (HEG), and top 50 highly variable genes (HVG) (Figure T]. with improvements ranging between 39% to 120% compared to the second highest scoring method. The top predicted genes from BLEEP
consistent with those from ST-Net and HisToGenee but the correlation values (r) are higher by up to 35% compared to the second-highest scoring method. BLEEP also excel in retaining the biological heterogeneity of the original sample as shown in Figures and 5
Overall, our proposed framework, BLEEP, has the potential to significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughput analysis of histology images for both research and clinical applications.
6 Acknowledgements
All authors thank the Vector Institute, Calcul Québec, and the Digital Research Alliance of Canada for their support. This research has been made possible in part by a grant G.D.B. from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. B.W. is supported by the NSERC discovery grant and CIFAR chair programs (RGP1N-2020-06189, DGECR- 2020-00294]. R.X. is supported by the Ontario Graduate Scholarship.
10

---

References
[1] Shahar Alon, Daniel R Goodwin, Anubhav Sinha, Asmamaw Wassie, Fei Chen, Evan R Daugharthy, Yosuke Bando, Atsushi Kajita, Andrew G Xue, Karl Marrett, et al. Expansion sequencing: Spatially precise in situ transcriptomies in intact biological systems. Science, x7166658))aaxx2666 2021.
[2] Tallulah S Andrews, Jawairia Atif, Jeff C Liu, Catia T Perciani, Xue-Zhong Ma, Comnelia Thoeni, Michal Slyper, Gökcen Eraslan, Asa Segerstolpe, Justin Manuel, et al. Single-cell, single-nucleus, and spatial ma sequeneing of the human liver identifies cholangiocyte and mesenchymal heterogeneity. Hepatology Communications 6(4):821 2022.
[3] Tallulah S Andrews, Diana Nakib, Catia Perciani, Xue Zhong Ma, Lewis Liu, Erin Winter, Damra Camat, Sai Chung, Justin Manuel, Shantel Mangroo, et al. Single-cell and spatial transcriptomics reveals the human liver immunological landscape and myeloid dysfunction in psc. bioRxiv, pages 2023-07, 2023.
[4] Kok Hao Chen, Alistair N Boettiger, Jeffrey R Moffitt, Siyuan Wang, and Xiaowei Zhuang. Spatially resolved, highly multiplexed ma profiling in single cells. Science, 348(6233);aa6600)) 2015.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[6] Simone Codeluppi, Lars E Borm, Amit Zeisel, Gioele La Manno, Josina A van Lunteren, Camilla Svensson, and Sten Linnarsson. Spatial organization of the somatosensory cortex revealed by osmfish. Nature methods, 15(1 (5(11):9329935, 2018.
[7] Charles Comiter, Eeshit Dhaval Vaishnav, Metamia Ciapmricotti, Bo Li, Yiming Yang. Scott J Rodig, Madison Turner, Kathleen I. Pfaff, Judit Jané- Valbuena, Michal Slyper, et al. Inference of single cell profiles from histology stains with the single-cell omics from histology analysis framework (schaf). bioRxiv, pages 2023-03, 2023.
[8] Chee-Huat Linus Eng, Michael Lawson, Qian Zhu, Ruben Dries, Noushin Koulena, Yodai Takei, Jina Yun, Christopher Cronin, Christoph Karp. Guo-Cheng Yuan, et al. Transcriptome super-resolved imaging in tissues by seqfish+. Nature, 568(7751):235 239, 2019.
[9] Bryan He, Ludvig Bergensträhle, Linnea Stenbeck, Abubakar Abid, Alma Andersson, Åke Borg, Jonas Maaskola, Joakim Lundeberg, and James Zou. Integrating spatial gene expression and breast tumour morphology via deep learning. Nature biomedical engineering, 4(8):827-834, 2020.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods, 16812):1289-1226 2019.
[12] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[13] Raktim Kumar Mondol, Ewan KA Millar, Peter H Graham, Lois Browne, Arcot Sowmya, and Erik Meijering. hist2rna: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images. Cancers, 15(9):2569, 2023.
[14] Minxing Pang, Kenong Su, and Mingyao Li. Leveraging information spatial transcriptomies to predict super-resolution gene expression from histology images in tumors. bioRxiv, pages 2021-11, 2021.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.

---

[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
[17] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[18]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831. PMLR, 2021.
[19] Rahul Satija, Jeffrey A Farrell, David Gennert, Alexander F Schier, and Aviv Regev. Spatial reconstruction of single-cell gene expression data. Nature biotechnology. 33(5))495-500 2015.
[20] Benoît Schmauch, Alberto Romagnoni, Elodie Pronier, Charlic Saillard, Pascale Maillé, Julien Calderaro, Aurélie Kamoun, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, al. A deep learning model to predict ma-seq expression of tumours from whole slide images. Nature communications, 11(1):3877, 2020.
[21] M. Moein Shariatnia. Simple CLIP, 4 2021.
[22] Patrik I. Ståhl, Fredrik Salmén, Sanja Vickovic, Anna Lundmark, José Fernández. Navarro, Jens Magnusson, Stefania Giacomello, Michaela Asp, Jakub O Westholm, Mikael Huss, et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science, 353((294):778883 2016.
[23] Xiao Wang, William Allen, Matthew Wright, Emily Sylwestrak, Nikolay Samusik, Sam Vesuma, Kathryn Evans, Cindy Liu, Charu Ramakrishnan, Jia Liu, et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science, (6400))6),, 2018.
[24] Yinxi Wang, Kimmo Kartasalo, Philippe Weitz, Balazs Acs, Masi Valkonen, Christer Larsson, Pekka Ruusuvuori, Johan Hartman, and Mattias Rantalainen. Predicting molecular phenotypes from histopathology images: transcriptome-wide expression-mophhlogy analysis in breast cancer. Cancer Research, 81(19):5115 5126, 2021.
[25] Alexander Wolf, Philipp Angerer, and Fabian Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.
[26] Yuansong Zeng, Zhuoyi Wei, Weijiang Yu, Rui Yin, Yuchen Yuan, Bingling Li, Zhonghui Tang, Yutong Lu, and Yuedong Yang. Spatial transcriptomies prediction from histology jointly through transformer and graph neural networks. Briefings in Bioinformatics, 23(5), 2022.
12


Reviews:
[{"review_id": "m9WwZIxwj6", "reviewer": "Reviewer_XnkF", "summary": "This paper studies the problem of gene expression profiling using histology images. They propose a bi-modal embedding framework BLEEP (Bi-modaL Embedding for Expression Prediction), which is capable of generating spatially resolved gene expression profiles of whole-slide Hematoxylin and eosin (H&E) stained histology images. The proposed method can significantly reduce the time and cost associated with gene expression profiling.", "questions": "See weaknesses.", "limitations": "The proposed method can effectively address the problem of gene expression prediction. However, it is not clear whether we can apply this method to more applications.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper utilizes a deep learning method to tackle the problem of gene expression prediction, reducing the time and cost.\n2. The paper is well-written and easy to follow. The motivation is clearly illustrated.\n3. The experiment design looks reasonable and interesting. Supplementary materials illustrate details of experimental settings.", "weaknesses": "1. Weak comparisons. This work only compares with two deep learning methods HisToGene and ST-Net. There should be other deep learning methods about image processing that can be applied to this area and the authors should compare with more baselines to confirm the efficacy of the proposed method.\n2. Limited impact of the proposed method. Although the problem in this paper is interesting, this work should discuss broad impact of the proposed method. For instance, how can we apply this method to other biomedical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for these comments. We summarize the major points below along with our rebuttal to each point.\n\nWeak comparisons. This work only compares with two deep learning methods HisToGene and ST-Net:\n- HisToGene and ST-Net are state of the art methods for the task of expression prediction from images in this field and are not weak comparisons. The performances of these methods highlight the difficulty of the image to expression task and are competitive reflections of the current state of the field to our best knowledge. \n- We surveyed many other related methods such as HE2RNA[1], hist2RNA[2], HIST2ST[3], and more recently, CeLEry[4] and SCHAF[5]. However they are not 1 to 1 comparisons that either take different information as input or tackle different settings with different use cases. For example, SCHAF requires both H&E image and the paired single cell RNA sequencing data rather than spatial transcriptomics data, the former containing no spatial information but higher expression resolution. In short all these surveyed methods were carefully evaluated and deemed not suitable to include as comparisons. \n\n[1] Schmauch, Benoît, et al. \"A deep learning model to predict RNA-Seq expression of tumours from whole slide images.\" Nature communications 11.1 (2020): 3877.\n\n[2] Mondol, Raktim Kumar, et al. \"hist2RNA: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images.\" Cancers 15.9 (2023): 2569.\n\n[3] Zeng, Yuansong, et al. \"Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks.\" Briefings in Bioinformatics 23.5 (2022): bbac297.\n\n[4] Zhang, Qihuang, et al. \"Leveraging spatial transcriptomics data to recover cell locations in single-cell RNA-seq with CeLEry.\" Nature Communications 14.1 (2023): 4050.\n\n[5] Comiter, Charles, et al. \"Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF).\" BioRxiv (2023): 2023-03.\n\nThis work should discuss broader impact of the proposed method. For instance, how can we apply this method to other biomedical applications:\n- We currently have no plans of applying this method to other biomedical applications. However, we are dedicating our efforts to extend this work to cover multiple organs and modalities, which we anticipate will help further improve performance and generalizability of BLEEP for prediction expression based on H&E image. \n- In terms of broader impact, we anticipate that BLEEP may improve biological understanding of h&e images by drawing connections between image features and gene expression. H&E staining is a ubiquitously used experimental technique in biology, our work may allow further understanding of H&E stained samples through learning the joint embedding.  Furthermore, clinical classification in the field of pathology could benefit from better understanding of image features in terms of molecular features, which underlie disease. Lastly, understanding the relationship of images and gene expression will help projects like the human cell atlas [6] to create coordinate frameworks that position genes spatially in terms of tissue anatomy, and ultimately in terms of the whole body.\n\n[6] Rozenblatt-Rosen, Orit, et al. \"The Human Cell Atlas: from vision to reality.\" Nature 550.7677 (2017): 451-453."}, {"review_id": "5Qjr2fnisS", "reviewer": "Reviewer_VLr1", "summary": "The authors have developed the model titled BLEEP, which is a contrastive learning implementation, trained on data from the 10x Genomics Visium platform, a common spacial genomics platform.  Spacial genomics generates high dimensional data that includes both images and RNA expression on small patches on tissue slides.   BLEEP uses two destinct encoder for the image and expression data.  These embeddings are then used as inputs for contrastive learning by a modified CLIP method. The approach of contrastive learning is sound given the high feature status in both the data which is being learned from and the images that are to be predicted.  \n\nThe model is tested vs. benchmark models on marker gene, highly variable genes, and highly expressed genes and model shows modest increase in correlation to other models.  BLEEP is also compared to how well gene expression is predicted for several genes. Figure 2 illustrates that the BLEEP model does show a better correlation to gene expression variation than other models. Additional figures show the heatmaps of actual relative to model predictions for fixed and variable scaled outputs.  Section 4.3 discusses how the model is perhaps more robust to artifactual portions of images.  \n\n\n\n", "questions": "Can you foresee any circumstance where the model could be used on image data alone to provide information at a low cost relative to running the spacial genomics platform. \n\n", "limitations": "The main limitation is that this may not be a useful task to pursue.  However, it does seem authors are aware of this limitation and if we don't try to tackle what seems like an impossible task then it is hard to make progress. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "I appreciate the color expressed around how \"ill-posed\" the problem is that the authors are trying to address.  Likewise, there is a huge dimensionality challenge and the sequencing methods are rather flimsy as stated.  Going into this problem realizing the challenge is very good. \n\nThe contrastive learning approach is likely the best approach the the task at hand. \n\nThe presentation of the data is not to express absolute confidence in the results and the authors only suggest that perhaps the methods are more sound given that the task itself may not be a solvable task. ", "weaknesses": "There is no guarantee that the signal that is being derived from given methods is in learning from the data presented. While the presentation of improved variance prediction is probably the strongest evidence in this regard. \n\n\nEven though the model appears to outperform benchmarks, the correlations remain very low and likely the predictions are not particularly useful  task. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for these comments. We summarize the major points below along with our rebuttal to each point.\n\nThere is no guarantee that the signal that is being derived from given methods is in learning from the data presented. While the presentation of improved variance prediction is probably the strongest evidence in this regard:\n- Our proposed method aims to learn a joint representation between H&E image and expression from the data presented. The improved variance prediction relative to other methods may be attributed to a well-learned joint embedding in conjunction with our choice of the query-reference inference strategy\n\nEven though the model appears to outperform benchmarks, the correlations remain very low and likely the predictions are not particularly useful task:\n- We thank the reviewer for this important question. We are aware that full expression prediction from image features is likely ill-posed. However we do believe there is substantial mutual information between image features and a subset of gene expressions. Our proposed contrastive learning objective helps prioritize these genes without any injection of prior knowledge. As seen in Table 2. The most well predicted genes for this dataset are also functional and well documented by prior biological research. For example, CYP1A2 is an oxidizing enzyme that is also widely cited as a liver zonation marker. Furthermore, the top predictions from BLEEP are rather consistent with top predictions by His2Gene and ST-Net, but with higher correlations to original expression, highlighting a step in a promising direction. \n\nCan you foresee any circumstance where the model could be used on image data alone to provide information at a low cost relative to running the spatial genomics platform:\n- Yes we believe our work may stimulate others to iterate upon our efforts and ultimately make advances towards this goal. \n- As it stands currently, we chose to benchmark our method on 3 sets of genes (Marker genes, highly expressed genes and highly variable genes). We believe these 3 sets of genes are likely more enriched for genes useful for diagnosis and treatment (drug targets). Within these categories while average correlation hovers between 0.17-0.21, some subset of these genes are quite reasonably predicted and potentially useful for aforementioned applications. However we agree more work is needed to further examine the potential of BLEEP in this use case. We are undergoing followup experiments to examine the effect of increased reference size and working on expanding the reference to cover multiple tissue types\n- Furthermore, this method could pave the way for the construction of more comprehensive foundation models for spatial transcriptomics analysis [1]\n\n[1] Cui, Haotian, et al. \"scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI.\" bioRxiv (2023): 2023-04."}, {"review_id": "ivLqTMO07d", "reviewer": "Reviewer_QFTu", "summary": "The authors introduce a method, called BLEEP, of imputing the (aggregate) gene expression profile of cells in patches of histology images. Inspired by CLIP, BLEEP trains image and profile encoders to jointly embed paired images and expression profiles, except in replaces the typical CLIP loss with a novel loss that matches the matrix of similarities between embeddings of image and expression profiles to a matrix of target similarities. Gene expression profiles are imputed for a patch by averaging the expression profiles of reference samples closest in the embedding space. The authors demonstrate SOTA predictive performance over related methods (e.g., HisToGene and ST-Net), while maintaining greater biological and spatial variability, and being less susceptible to batch effects.\n\n", "questions": "- It is not clear if any hyperparameter tuning was done when choosing the default configurations of all models used in this paper. The huge differences in performance make this reviewer wonder if suboptimal parameters were chosen, even if published configurations were used for HisToGene and ST-Net. Please comment.\n\n- It is not clear why HisToGene and ST-Net were chosen for comparison. Please comment.", "limitations": "The Discussion section adequately addresses some potential limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n\nTo the best of this reviewer's knowledge, this is the first use of a CLIP-like joint embedding objective for learning to predict gene expression profiles from H&E histology images. Also, the loss introduced by the authors in place of the typical CLIP loss appears to be novel.\n\n### Quality\n\nThe work appears to be of sufficient quality to be credible.\n\n### Clarity\n\nThe paper is well-written and easy to understand. The discussion is particularly detailed and interesting.\n\n### Significance\n\nGene expression profile prediction from unstructured biological readouts like H&E images has significant value, since gene expression is interpretable and causal.", "weaknesses": "- It is not clear that the loss introduced by the authors, replacing the typical CLIP loss, is required to obtain the reported performance as the the authors claim. A comparison with results obtained using the typical CLIP loss would be useful and should be added.\n\n- The prediction procedure aggregates profiles from a reference dataset, making BLEEP limited in its potential application. It would be interesting to train an image-to-profile decoder and compare the results, especially wrt improved generalization. This is a heavy lift, though, and not required for the revisions.\n\n- As the authors already point out, the aggregation used for query-reference imputation could be removing useful biological signal or have a number of smoothing effects. A discussion of the choices made for imputation and their effects on prediction performance should be added.\n\n- Regarding the previous point, however, there is a lack of details regarding the query-reference imputation step, like number of samples used, method of aggregation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for these comments. We summarize the major points below along with our rebuttal to each point.\n\nCLIP loss w/ smooth vs. w/o smooth comparison: \n- Please see rebuttal addressed to all reviewers for ablation studies and discussion\n\nChoices made for imputation and their effects on prediction\n- Please see rebuttal addressed to all reviewers for ablation studies and discussion\n\nDetails on query-reference steps, implementation details:\n- During query-reference imputation, we agree more details will be useful and plan to include them in the camera-ready version of our manuscript if we are successful during rebuttal. We will expand the discussion on implementation details in the revised version. In BLEEP’s default setting, we use the top 50 most similar samples for query reference imputation, and take their average expression for aggregation. \n\nSettings for the comparison experiments: \n- For HisToGene and ST-Net, we adopted the published configurations, as these parameters had been determined optimal in the original studies. Further, we conducted 3-trial experiments with different random initiations for more robust results. For BLEEP, extensive hyperparameter tuning was done during the development process, e.g. ViT backbone vs. Conv. Backbone for image encoder. Therefore, the performance differences are not due to manually chosen suboptimal parameters but reflect the inherent differences in these approaches. We will include evaluation scripts in the code release for community replication.\n\nComparison with HisToGene and ST-Net:\n- HisToGene and ST-Net are state of the art methods for the task of expression prediction from images in this field and are not weak comparisons. The performances of these methods highlight the difficulty of the image to expression task and are competitive reflections of the current state of the field to our best knowledge. \n- We surveyed many other related methods such as HE2RNA[1], hist2RNA[2], HIST2ST[3], and more recently, CeLEry[4] and SCHAF[5]. However they are not 1 to 1 comparisons that either take different information as input or tackle different settings with different use cases. For example, SCHAF requires both H&E image and the paired single cell RNA sequencing data rather than spatial transcriptomics data, the former containing no spatial information but higher expression resolution. In short all these surveyed methods were carefully evaluated and deemed not suitable to include as comparisons. \n\n[1] Schmauch, Benoît, et al. \"A deep learning model to predict RNA-Seq expression of tumours from whole slide images.\" Nature communications 11.1 (2020): 3877.\n\n[2] Mondol, Raktim Kumar, et al. \"hist2RNA: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images.\" Cancers 15.9 (2023): 2569.\n\n[3] Zeng, Yuansong, et al. \"Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks.\" Briefings in Bioinformatics 23.5 (2022): bbac297.\n\n[4] Zhang, Qihuang, et al. \"Leveraging spatial transcriptomics data to recover cell locations in single-cell RNA-seq with CeLEry.\" Nature Communications 14.1 (2023): 4050.\n\n[5] Comiter, Charles, et al. \"Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF).\" BioRxiv (2023): 2023-03.\n\n"}, {"review_id": "9IjOI22Jj4", "reviewer": "Reviewer_WHAP", "summary": "The authors present, BLEEP, a bi-modal embedding framework capable of generating spatially resolved gene expression profiles of whole-slide H&E stained histology images. This work stems directly from spatial transcriptomics, where we have spatially map gene expression profiles with H&E images. A well descripted tasks is predicting these gene expression profiles from the H&E images alone. This is the task the author set out to address and compare their results with several related methods. Their technical contribution was to apply CLIP-style training plus 'query-reference imputation' to predict expression profiles, rather than relying on disciminative learning alone. They were able to show results that outperform previously reported methods. ", "questions": "See above. ", "limitations": "No, there was not a clear description of the limitations or societal impact of the work. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strength: \n1) The authors introduce the topic well. They do a good job discussing previous work. \n2) The paper is well written. \n3) The figures are clear and instructive.\n4) This is a potentially important biomedical computer vision task.\n5) The smoothed target and the query look-up strategy is a novel and intuitive contribution to this task.", "weaknesses": "Weaknesses: \n1) The paper is, overall, underdeveloped. They have made a small technical contribution and applied this to a very small and narrow domain dataset (normal hepatic tissue). I believe that small technical contributions have an essential place in biomedical AI research, but they must be accompanied by a very strong validation across organ systems, tumor types, platforms, etc. I understand that the availability of these datasets is limited, but the authors have limited their contribution to a single dataset for no clear reason. I would recommend additional experiments in at least two other organ systems, preferably a disease dataset, to demonstrate the generalizability of this method. \n2) Minor comment: how are the 'pairs with similar morphological features or expression landscapes' selected? The authors do not specify how these pairs are selected in order to generate the smoothed targets. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for these comments. We summarize the major points below along with our rebuttal to each point.\n\nAdditional experiments:\n- While our paper focuses on a single organ system, this choice was made to deeply explore and validate our approach to demonstrate the effectiveness and applicability of the proposed method. There are no technical barriers to extend BLEEP to other organs. Ongoing internal experiments on brain tissue has shown promising results of BLEEPs generalization capacity. We are open to including the results of our experiments in the camera ready version if deemed necessary by the reviewer.  In line with your recommendation, we are currently building foundation models that span multiple organ systems, and it will be released as future work. \n\nPairs with similar morphological features or expression landscapes:\n- In our approach, we don't explicitly \"select\" these pairs but rather calculate an internal similarity matrix. This is done for both the morphological features ($H_v$) and the expression landscapes ($H_x$). The internal similarities, $sim(H_v, H_v)$ and $sim(H_x, H_x)$, are computed as the dot product of the feature matrix with its transpose, representing the similarity between every pair of samples. The resultant similarity-adjusted target matrix ($target=(softmax(sim(H_x, H_x) + sim(H_v, H_v) )/2 \\cdot \\tau)$) allows us to account for the inherent similarities in the data, smoothing the targets based on these similarities.\n"}, {"review_id": "XyACdMXy3w", "reviewer": "Reviewer_J9t4", "summary": "The paper proposes learning a contrastive learning based joint embedding space to align gene expressions and histology. This expression space is used to generate expression predictions for queried patches from the histology modality. The paper shows improved correlations in predicting gene expressions as well as better measure of expression heterogeneity when compared to HisToGene and ST-Net, two popular supervised regression-based methods for gene expression prediction.", "questions": "1. Which correlation measure is used? Please specify.\n2. It's not clear how the plot in Figure 2 was obtained. Is it the mean and variance for each gene across all slides in the test dataset? Why are there multiple points for the predicted expression profile? Some details on this section would be helpful for readers.\n3. Can you please share your thoughts on points 1, 3, 4, 5 in the previous section.\n\n", "limitations": "I believe the authors have shared some limitations in various sections including some seemingly negative results (which is applaudable!). Would really appreciate a commentary on when an alignment based method is expected to do better than a direct regression method and vice versa and explicitly mention some limitations and potential future directions of exploration in this area.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper propose a new approach for the problem of predicting gene expression through histology. The problem space is important since sequencing is still expensive, not always available, and its relationship with histology is ripe ground for research in multimodal analysis.\n2. The paper seems to be the first one to predict gene expression using bi-modal alignment (although CCA based alignment has been done previously). The reasoning for choosing an alignment-based method is well motivated. The contrastive learning method is simple and effective, and probably overfits less than a direct regression-based method.\n3. The correlation results show significantly improved performance on the task of gene expression prediction. Moreover, the results in Figure 2 highlight the very important characteristic of the method not losing out on the heterogeneity information of the genes.\n\n\n[1] Ash, Jordan T., et al. \"Joint analysis of expression levels and histological images identifies genes associated with tissue morphology.\" Nature communications 12.1 (2021): 1609.", "weaknesses": "1. The results show correlations of the reference and predicted gene expressions. However, there's no metrics to show how well the method does for spatially resolving these predictions. The results are also shown on a single dataset.\n2. The contrastive learning framework for multimodal problems isn't novel in itself. The smooth loss too has been formulated in different ways in previous work [1, 2, 3]. Thus the main contribution of the paper is its formulation and application to the task of gene expression prediction. The paper is predominantly an application paper, therefore more extensive ablations or more datasets would have been more convincing. The results shown however do show improvements over previous baselines.\n3. The paper doesn't motivate choices like smooth contrastive loss, k-nearest neighbor based selection, linear combination to get the imputed genes etc. through empirical ablation experiments. \n4. The paper shows the differences between using a fixed scale vs variable scale for spatially resolved predictions. It's useful to motivate why the absolute values for these predictions matter and looking at fixed scale is the way to go. \n5. The gene-gene correlation is interesting because HisToGene seems to be closer to the original expression data than the proposed method. The authors note that the method appears to accentuate certain positive and negative correlations, but it's not clear if this is desirable. Similar arguments about the clustering experiment can be made as well where HisToGene seems to do better (although I agree that this isn't a great measure of prediction quality).\n\n[1] Denize, Julien, et al. \"Similarity contrastive estimation for self-supervised soft contrastive learning.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n[2] Zheng, Mingkai, et al. \"Ressl: Relational self-supervised learning with weak augmentation.\" Advances in Neural Information Processing Systems 34 (2021): 2543-2555.\n[3] Wei, Chen, et al. \"Co2: Consistent contrast for unsupervised visual representation learning.\" arXiv preprint arXiv:2010.02217 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for these comments. We summarize the major points below along with our rebuttal to each point.\n\nCorrelation Measure:\n- We used the Pearson correlation coefficient for measuring both prediction-GT correlation and gene-gene correlation.\n\nClarification on Figure 2:\n- Each point on the plot refers to a single gene, whose value reflects its normalized mean or variance across a test sample slice. Blue dots correspond to the ground truth values, while the orange dots represent the inferred values. The gene rankings are performed based on ground truth values in increasing order. \n\nMetrics for spatially resolving the predictions:\n- We provide the qualitative outcomes in Figure 3 and Supplementary Figure 1 to demonstrate BLEEP's proficiency in spatially resolving these predictions. BLEEP generates meaningful unsupervised clusters that align with the periportal and pericentral regions of the sampled liver tissue.\n- For a quantitative evaluation of BLEEP's clustering, we reference the Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI) metrics in Supplementary Table 2. However, due to the continuous gradient of biological variation, the selection of the exact clustering method and its parameters significantly influences the definition of discrete clusters. Therefore, involvement of domain experts to annotate ground truth regions on the H&E image is necessary for a more comprehensive and robust assessment of spatial resolution and planned for future work\n\nMotivation and empirical experiment for using smooth contrastive loss, knn, linear combination for imputation:\n- Please see rebuttal addressed to all reviewers for ablation studies and discussion\n\nFixed scale vs. variable scale:\n- The importance of fixed scale vs variable scale could be split into two components. Fundamentally, Visium expression data measure the discrete number of transcripts captured that map to each gene. The absolute values in the ground truths provided are therefore meaningful and should be reproduced by any predictive model in terms of both mean and variance. Practically, one may argue that relative expression irrespective of variance is sufficient information for downstream use cases particularly when the counts measured via Visium are themselves subject to a stochastic sampling process during the experiment. However, predicting absolute expression allows meaningful comparisons in magnitude across different genes, which contain richer information than z scores (as each gene has different mean and variance). Furthermore, a fixed scale enables direct comparisons of expression levels across different H&E slides. This approach gives an absolute frame of reference. Visualizing predictions in fixed scale benchmarks the methods’ ability to retain these types of information. \nLastly, both His2Gene and ST-Net were trained on the same data as BLEEP. The former two approaches’ inability to learn the variance of the held out dataset is a bug not a feature, and reflect shortcomings in the method. \n\nConcern regarding the gene-gene correlation heatmap:\n- The gene-gene correlation heatmap from HisToGene does seem more similar to the original expression. This is likely attributed to BLEEP’s averaging effect during the imputation process. As discussed in the paper, the Visium experimental platform is susceptible to sampling noise. Averaging the expression levels across the top K most similar spots to any query image patch may serve to average out some of this noise and uncover additional biological meaningful signals akin to pseudo-bulking the top K most likely expression profiles for any given query image patch. Supporting this observation, we draw attention to the high and low values in the original gene-gene correlation heatmap. The same correlation patterns are well represented in BLEEP’s heatmap, but just more accentuated, similar to what one would expect from pseudo-bulking. By learning a joint embedding between image and expression, BLEEP is capable of identifying suitable spots to perform pseudo-bulking in a non-biased way and we argue this is desirable for downstream biological inquisition. Nevertheless, such averaging effects may also mask out some genuine biological signals. This limitation is included in the manuscript and requires further investigation. \n"}]

Author rebuttal:
We sincerely thank the reviewers for their comments and their acknowledgment to the strengths of our paper including: 
- “significantly improved performance” and “not losing out on the heterogeneity information.” (J9t4).
- “first use of a CLIP-like joint embedding objective for learning to predict gene expression profiles from H&E histology images” (QFTu)
- “The smoothed target and the query look-up strategy is a novel and intuitive contribution” (WHAP)
- “Contrastive learning approach is likely the best approach to the task at hand” (VLr1)
- “Experiment looks reasonable and interesting” (XnkF)

We have performed the requested ablation experiments from some of the reviewers. We share the resulting ablation table in the attached PDF below. From this experiment we make a few important observations:

- The choice of K during the query-reference imputation process influences the prediction quality quite negatively when a low value is selected (K = 10). Values of K above our default value could provide some small improvements to correlation of the resulting predicted expression values for the HVG and HEG gene sets, but the differences were not pronounced. This is interesting but inline with what one might expect from taking the pseudo-bulk of the top K most likely expression profiles given any query image. However, the MG gene set did not show much improvement with increasing K. Furthermore, doing so may carry a trade off of further systematically deviating from the original variance of the dataset due to the increased averaging effect. With this in mind, we feel our default value of K = 50 remains adequate. 

- The most similar match between query and reference is usually not the best prediction (as seen from the 3rd row of the ablation table and indirectly the 4th row when predictions are weighted by their similarity). However we suspect the gap may close to some degree as the reference grows further in size, but in general some amount of averaging is desired for query-reference imputation to remove some noise intrinsic in the Visium platform. However, in the discussion section of the manuscript, we further highlight the possibility of genuine biological signals being averaged out, which is an important consideration to be further investigated. 

- Smoothing the contrastive loss objective to take into account patch similarity showed modest increase in performance. The gain in performance may be due to the fact that relaxing the contrastive objective is more compatible with the similarity based inference strategy. The smoothing may help lessen the extent similar references are pushed apart in embedding space during training, resulting in improved querying of the top K most likely expression profiles given a query image patch during the inference stage. 

Lastly, for all reviewers, we want to further motivate BLEEP and clarify our vision regarding future extensions of BLEEP and the project’s potential impact:

- The task of spatially resolved, transcriptome-wide expression prediction given an H&E stained image is likely extremely difficult. BLEEP offers an unbiased way to prioritize genes that are more likely to be well predicted from image features via the learning of a joint embedding between image and expression features. It also tackles the curse of dimensionality through the use of query-reference imputation while simultaneously alleviating some technical noise intrinsic to the Visium platform. These design choices allowed us to see significant improvements over state-of-the-art methods such as ST-Net and His2Gene (upwards of 120% increase in correlation with ground truth expression).
- BLEEP is currently restricted for research purposes only and still has a lot of room for improvement. Nevertheless, BLEEP may already be immediately useful for gaining biological understanding of H&E images due to the connections drawn between image and expression features. H&E staining is a ubiquitously used experimental technique in biology, our work may allow further understanding of H&E stained samples through learning the joint embedding.  Similarly, clinical classification in the field of pathology could benefit from better understanding of image features in terms of molecular features, which underlie disease. Lastly, understanding the relationship of images and gene expression will help projects like the human cell atlas [1] to create coordinate frameworks that position genes spatially in terms of tissue anatomy, and ultimately in terms of the whole body.
- While our paper focuses on a single organ system, this choice was made to deeply explore and validate our approach to demonstrate the effectiveness and applicability of the proposed method. There are no technical barriers to extend BLEEP to other organs. Ongoing internal experiments on brain tissue have shown promising results of BLEEP’s generalization capacity. We are open to including the results of our experiments in the camera ready version if deemed necessary by the reviewer. In line with the recommendation of some reviewers, we have already begun efforts towards building foundation models that span multiple organ systems, and it will be released as future work. 
- Finally, we believe expert annotation of the different regions of the H&E image will provide us with better ground truths for tissue-wide expression pattern benchmarking. This will be planned for future work. 

[1] Rozenblatt-Rosen, Orit, et al. "The Human Cell Atlas: from vision to reality." Nature 550.7677 (2017): 451-453.

Meta-review: The paper addresses an interesting problem of predicting gene expression profiling from histological images, with significant impact to various biomedical applications. The initial review ratings were mixed. The reviewers raised concerns on the interpretation of results (low correlation with transcriptomics data), and the experimental evaluation (results on a single dataset with one tissue type). They also questioned many technical details, and pointed out potential issues in ethics. The author provided a detailed response. Most of the concerns have been addressed, resulting in favorable final ratings from the reviewers. One remaining concern lies in the consideration of a single dataset, which the authors also admitted yet explained the difficulty of acquiring spatial transcriptomics with high resolution imaging data.

The AC considers that the paper introduced a challenging biomedical problem, presented a promising solution, and is thus worth accepting to NeurIPS. The AC encourages the authors to further address the review comments and strengthen the paper.
Decision: Accept (poster)