SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The choice of K during the query-reference imputation process influences the prediction quality quite negatively when a low value is selected ( $\mathrm{K} = 10$ ). Values of K above our default value could provide some small improvements to correlation of the resulting predicted expression values for the HVG and HEG gene sets, but the differences were not pronounced. This is inline with what one might expect from taking the pseudo-bulk of the top K most likely expression profiles given any query image. However, the MG gene set did not show much improvement with increasing K. Furthermore, doing so may carry a trade off of further systematically deviating from the original variance of the dataset due to the increased averaging effect. With this in mind, we feel our default value of  $\mathrm{K} = 50$  remains adequate.
We also observe that the most similar match between query and reference is usually not the best prediction (as seen from the 3rd row of the ablation table and indirectly the 4th row when predictions are weighted by their similarity). We suspect the gap may close to some degree as the reference grows further in size, but in general some amount of averaging is desired for query-reference imputation to remove some noise intrinsic in the Visium platform. In addition, as no reference expression profile will perfectly match that of the query image, averaging multiple profiles is required to more appropriately describe the query expression profile. Nevertheless, we also highlight the possibility of genuine biological signals being averaged out, which is an important consideration to be further investigated.
Smoothing the contrastive loss objective to take into account patch similarity showed modest increase in performance. The gain in performance may be due to the fact that relaxing the contrastive objective is more compatible with the similarity based inference strategy. The smoothing may help lessen the extent similar references are pushed apart in embedding space during training, resulting in improved
Figure 5: Leiden clusterings for original and predicted expressions overlaying the H&E image. Image-only expression predictions are invariant to low quality regions (red) during actual experiment.
Table 3: BLEEP ablation experiments. Variables tested include: smoothed objective versus the original CLIP objective, choice of top K and the method of aggregation. Pearson correlation values are reported from the marker gene set (MG), the top 50 highly variable genes (HVG) and the top 50 highly expressed genes (HEG). The correlation values show the average of 3 replicates. Accompanied uncertainty values denote the maximum difference from the mean of the 3 replicates.
Smoothed Obj.	K	Aggregation	Average Pearson Correlation
MG	HVG	HEG
Yes	10	average	0.179 ± 0.020	0.146 ± 0.008	0.148 ± 0.022
Yes	100	average	0.215 ± 0.011	0.180 ± 0.012	0.181 ± 0.015
Yes	-	simple	0.079 ± 0.032	0.075 ± 0.016	0.084 ± 0.023
Yes	50	weighted	0.186 ± 0.018	0.161 ± 0.015	0.157 ± 0.026
No	50	average	0.209 ± 0.017	0.165 ± 0.007	0.170 ± 0.005
Yes	50	average	0.217 ± 0.002	0.175 ± 0.016	0.173 ± 0.011
querying of the top K most likely expression profiles given a query image patch during the inference stage.
In this study, we introduced BLEEP (Bi-modal Embedding for Expression Prediction), a novel framework for predicting gene expression from histology images. BLEEP constructs a joint embedding space from paired image and expression features in a reference dataset and subsequently utilizes the k most similar expression profiles in the joint space to impute the expression for any given image query. To the best of our knowledge, this is the first bi-modal embedding-based framework proposed for the task of expression prediction from histology images. We also show that the query-reference
imputation process is effective and well suited for the task of expression prediction from the bi-modal joint embedding.
We demonstrated that BLEEP effectively addresses three major challenges in the H&E image to expression prediction task. Firstly, the ill-posed nature of the problem is alleviated by the proposed joint image and expression embedding space, optimized using a contrastive learning objective inspired by CLIP. This encourages the shared features between the two modalities to be preferentially encoded in the joint space. Secondly, the curse of dimensionality is tackled by the query-reference imputation process, which predicts the entire expression profile jointly through linear combination rather than predicting each individual gene separately using supervised regression tasks. Lastly, we showed that BLEEP's expression prediction is resilient to experimental artifacts, both within a single sample and across different samples.
We further observed that the correlation matrix of BLEEP's predicted expressions not only captures existing patterns in gene-gene correlations (GGCs) but also accentuates more subtle positive and negative GGCs. This could be attributed to the ability of BLEEP's imputation process to average out the noise intrinsic in the 10x Visium platform, thereby increasing the absolute values of these correlations. This is consistent with the results presented in Figure 2, where the predicted gene expressions by BLEEP exhibit lower variance compared to the original dataset. This highlights BLEEP's capability to combine expression profiles of similar-looking spots in the dataset and generate expression profiles with reduced noise and enhanced biological signal compared to the original dataset. Ongoing work is evaluating BLEEP's ability to detect new spatially resolved gene modules in response to this observation.
However, an alternate explanation for the observed results could be that averaging during imputation removes genuine, abrupt biological signals, resulting in artificially smoothed expression patterns. It is possible that this may be true for certain genes, especially ones that correlate poorly with image features. Increasing the size of the reference dataset may mitigate this issue by reducing the distance between any query patch and the reference expression profiles. In practice, once the expression encoder is trained, it can be quickly used to integrate new datasets to the shared embedding space as reference. Furthermore, imputed expressions from query datasets could be subsequently integrated, making the process of improving BLEEP prediction natural.
Nevertheless, the improvement offered by BLEEP over existing methods is substantial. We achieved significantly higher correlation with actual expression profiles across the marker gene set (MG), top 50 highly expressed genes (HEG), and top 50 highly variable genes (HVG) (Figure 1), with improvements ranging between  $39\%$  to  $120\%$  compared to the second highest scoring method. The top predicted genes from BLEEP are consistent with those from ST-Net and HisToGene, but the correlation values (r) are higher by up to  $35\%$  compared to the second-highest scoring method. BLEEP also excel in retaining the biological heterogeneity of the original sample as shown in Figures 2, 3, 4 and 5.
Overall, our proposed framework, BLEEP, has the potential to significantly reduce the time and cost associated with gene expression profiling, opening up new avenues for high-throughput analysis of histology images for both research and clinical applications.
All authors thank the Vector Institute, Calcul Quebec, and the Digital Research Alliance of Canada for their support. This research has been made possible in part by a grant to G.D.B. from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. B.W. is supported by the NSERC discovery grant and CIFAR chair programs [RGPIN-2020-06189, DGECR-2020-00294]. R.X. is supported by the Ontario Graduate Scholarship.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[17] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[18] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[19] Rahul Satija, Jeffrey A Farrell, David Gennert, Alexander F Schier, and Aviv Regev. Spatial reconstruction of single-cell gene expression data. Nature biotechnology, 33(5):495-502, 2015.
[20] Benoit Schmauch, Alberto Romagnoni, Elodie Pronier, Charlie Saillard, Pascale Maillé, Julien Calderaro, Aurélie Kamoun, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, et al. A deep learning model to predictrna-seq expression of tumours from whole slide images. Nature communications, 11(1):3877, 2020.
[21] M. Moein Shariatnia. Simple CLIP, 4 2021.
[22] Patrik L Stahl, Fredrik Salmen, Sanja Vickovic, Anna Lundmark, José Fernández Navarro, Jens Magnusson, Stefania Giacomello, Michaela Asp, Jakub O Westholm, Mikael Huss, et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science, 353(6294):78-82, 2016.
[23] Xiao Wang, William E Allen, Matthew A Wright, Emily L Sylwestrak, Nikolay Samusik, Sam Vesuna, Kathryn Evans, Cindy Liu, Charu Ramakrishnan, Jia Liu, et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science, 361(6400):eaat5691, 2018.
[24] Yinxi Wang, Kimmo Kartasalo, Philippe Weitz, Balazs Acs, Masi Valkonen, Christer Larsson, Pekka Ruusuvuori, Johan Hartman, and Mattias Rantalainen. Predicting molecular phenotypes from histopathology images: a transcriptome-wide expression-morphology analysis in breast cancer. Cancer Research, 81(19):5115-5126, 2021.
[25] F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scany: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.
[26] Yuansong Zeng, Zhuoyi Wei, Weijiang Yu, Rui Yin, Yuchen Yuan, Bingling Li, Zhonghui Tang, Yutong Lu, and Yuedong Yang. Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks. Briefings in Bioinformatics, 23(5), 2022.