SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\text{Common}_{ij}$  denotes the index set for common HVGs,  $\text{Diff}_i$  represents the index set for different HVGs in graph  $i$ , and  $\text{Diff}_{\mathcal{N}(i)}$  indicates the index set for neighbors of  $\text{Diff}_i$  in graph  $i$ . To expedite the training process and conserve memory usage, we sample graph  $j$  for each graph  $i$  during model training. We also employ multi-thread programming to accelerate the index set extraction process.  $\lambda_c$  is the weight for InfoNCE loss. All of the components in MuSe-GNN are supported by ablation experiments in Appendix E.1. Details of hyper-parameter tuning can be found in Appendix E.2.
Datasets & Embeddings generation. Information on the different datasets used for different experiments is included at the beginning of each paragraph. The training algorithm of our model
Algorithm 1 Multimodal Similarity Learning Graph Neural Network (MuSe-GNN)
Input: Model ENCODERGNN, DECODERMLP; Dataset  $\mathcal{D} = (\{V_i,E_i\})_{i = 1}^T$  ; Number of epochs  $K$  ; Neighbor overlap list  $\lambda$  Weight coefficient for contrastive learning  $\lambda_{c}$    
Helper Functions: A function to find the genes with the same name FINDCOMMONGENES; A function to find the genes with different name FINDDIFFGENES; A function to calculate cosine similarity COSSIM; A function to sample data from a graph collection SAMPLE; A optimizer function to update weights ADAM. Output: Model ENCODERGNN   
1: INIT: initialize all parameters.   
2: for s in  $K$  steps do   
3: for id,  $\{V,E\}$  in enumerate(D) do   
4:  $e\gets$  ENCODERGNN(  $\{V,E\} ;\theta_{enc}$  );  $E_{rec}\gets$  DECODERMLP(eeT;  $\theta_{dec}$  5:  $\mathcal{L}\gets \mathcal{L}_{\mathrm{BCE}}(E_{\mathrm{rec}},E)$    
6:  $\mathrm{id}_{\mathrm{new}},\{V_{\mathrm{new}},E_{\mathrm{new}}\} \leftarrow \mathrm{SAMPLE}(G\backslash \{V,E\})$    
7:  $e_{\mathrm{new}}\gets \mathrm{ENCODERGNN}(\{V_{\mathrm{new}},E_{\mathrm{new}}\} ;\theta_{\mathrm{enc}})$    
8:  $e_{\mathrm{Common}},e_{\mathrm{Common}}^{\prime}\gets \mathrm{FINDCOMMONGENES}(e,e_{\mathrm{new}})$    
9:  $\mathcal{L}\gets \mathcal{L} - \mathrm{COSSIM}(e_{\mathrm{Common}},e_{\mathrm{Common}}^{\prime})\lambda_{id,id_{\mathrm{new}}}^{T}$    
10:  $e_{\mathrm{Diff}},e_{\mathrm{Diff}}^{\prime},e_{\mathrm{Diff}_N},e_{\mathrm{Diff}_N}^{\prime}\gets \mathrm{FINDDIFFGENES}(e,e_{\mathrm{new}})$    
11:  $\mathcal{L}\gets \mathcal{L} + \lambda_c\mathcal{L}_{\mathrm{InfoNCE}}(e_{\mathrm{Diff}}\oplus e_{\mathrm{Diff}_N},e_{\mathrm{Diff}}^{\prime}\oplus e_{\mathrm{Diff}_N}^{\prime})$    
12:  $\theta_{enc}\gets \mathrm{ADAM}(\mathcal{L},\theta_{enc});\theta_{dec}\gets \mathrm{ADAM}(\mathcal{L},\theta_{dec})$    
13: Return ENCODERGNN
is outlined in Algorithm 1. We stored the gene embeddings in the AnnData structure provided by Scanpy [104]. To identify groups of genes with similar functions, we applied the Leiden clustering algorithm [91] to the obtained gene embeddings. Details can be found in Appendix E.4.
Evaluation metrics. For the benchmarking process, we used six metrics: edge AUC (AUC) [74], common genes ASW (ASW) [59], common genes graph connectivity (GC) [59], common genes iLISI (iLISI) [59], common genes ratio (CGR), and neighbors overlap (NO) to provide a comprehensive comparison. Detailed descriptions of these metrics can be found in Appendix H. We computed the metrics for the different methods and calculated the average rank (Avg Rank  $\in$  [1, 9]). Moreover, to evaluate the model performance improvement, we applied min-max scaling to every metric across different models and computed the average score (Avg Score  $\in$  [0, 1]).
Baselines. We selected eight models as competitors for MuSe-GNN. The first group of methods stems from previous work on learning embeddings for biomedical data, including Principal Component Analysis (PCA) [74] used by GSS, Gene2vec, GIANT (the SOTA model for gene representation learning), Weight-sharing Multi-layer Auto-encoder (WSMAE) used by [109] and scBERT (the SOTA model with pre-training for cell type annotation). The second group of methods comprises common unsupervised learning baseline models, including GAE, VGAE [48] and Masked Autoencoder (MAE) (the SOTA model for self-supervised learning) [42]. MuSe-GNN, GIANT, GAE, VGAE, WSMAE and MAE have training parameter sizes between 52.5 and  $349\mathrm{M}$ , and all models are tuned to their best performance. Details are shown in Appendix E.2.
Biological applications. For the pathway analysis, we used Gene Ontology Enrichment Analysis (GOEA) [1] to identify specific biological pathways enriched in distinct gene clusters with common functions. Moreover, we used Ingenuity Pathway Analysis (IPA) [2] to extract biological information from the genes within various clusters, including causal networks [50] and sets of diseases and biological functions. Biological pathways refer to processes identified based on the co-occurrence of genes within a particular cluster. The causal network depicts the relationships between regulatory genes and their target genes. Disease and biological function sets facilitate the discovery of key processes and complications associated with specific diseases. Using gene embeddings also improves the performance of models for gene function prediction. To visualize gene embeddings in a low-dimensional space, we utilized Uniform Manifold Approximation and Projection (UMAP) [63].
Table 1: Avg Score across different tissues. Standard deviations are reported in Appendix E.3.
Methods	Heart	Lung	Liver	Kidney	Thymus	Spleen	Pancreas	Cerebrum	Cerebellum	PBMC
PCA	0.52	0.48	0.56	0.47	0.56	0.60	0.51	0.62	0.53	0.51
Gene2vec	0.40	0.37	0.33	0.29	0.21	0.31	0.24	0.27	0.31	0.19
GIANT	0.50	0.40	0.33	0.38	0.58	0.33	0.56	0.29	0.28	0.28
WSMAE	0.50	0.47	0.54	0.46	0.57	0.53	0.52	0.55	0.59	0.50
GAE	0.61	0.45	0.58	0.40	0.56	0.58	0.52	0.56	0.60	0.54
VGAE	0.64	0.32	0.33	0.38	0.56	0.31	0.33	0.41	0.33	0.47
MAE	0.36	0.47	0.50	0.45	0.41	0.52	0.39	0.50	0.49	0.50
scBERT	0.41	0.49	0.55	0.62	0.17	0.58	0.46	0.60	0.61	0.58
MuSeGNN	0.77	0.96	0.92	0.89	0.89	0.94	0.80	0.95	0.90	0.92
4.1 Benchmarking Analysis
We executed each method 10 times by using the same setting of seeds to show the statistical significance based on datasets across different tissues. The performance comparison of nine gene embedding methods is presented in Tables 1 and 14. Based on these two tables, MuSe-GNN outperformed its competitors in terms of both average ranks and average scores across all the tissues. Based on Table 1, for major tissues, such as heart and lung, MuSe-GNN's performance was  $20.1\%$  higher than the second-best method and  $97.5\%$  higher than the second-best method in heart and lung tissue, respectively. According to Appendix E.3, MuSe-GNN's stability was also demonstrated through various metrics by comparing standard deviations, including AUC, GC, and NO. In contrast, methods such as Gene2vec, GAE, VGAE, MAE and scBERT exhibited significant instability in their evaluation results for kidney or thymus. Consequently, we concluded that MuSe-GNN is the best performing model for learning gene representation based on datasets from different tissues, making it applicable to learn gene embeddings from diverse multimodal biological data.
4.2 Analysis of Gene Embeddings from Multimodal Biological Data
In Figure 3, we displayed the integration results for multimodal biological data from Humans. Figure 3 (a) and (b) demonstrated that MuSe-GNN could successfully integrate genes from different modalities into a co-embedded space, allowing us to identify functional groups using the Leiden algorithm shown in Figure 3 (c). Furthermore, Figure 3 (d) revealed that most of the clusters in (c) were shared across different modalities. We also identified three significant functional groups in Figure 3 (a): the nervous system (predominantly composed of the cerebrum and cerebellum [26]), the cardiovascular system (mainly composed of heart, lung, and kidney [29]), and the immunology system (primarily consisting of spleen, liver, and peripheral blood mononuclear cells (PBMC) [70]). All systems are important in regulating the life activities of the body. We also uncovered a pre-epigenetics group (mainly consisting of scATAC-seq data without imprinting, modification, or editing), emphasizing the biological gap existing in multi-omics and the importance of post-transcriptional regulation.
Using GOEA, we could identify significant pathways enriched by different co-embedded gene clusters. For instance, Figure 3 (d) displayed the top 5 pathways in an immunology system cluster. The rank was calculated based on the negative logarithm of the false discovery rate. Since all top pathways were related to immunological defense and response, it further supported the accuracy of our embeddings in representing gene functions. For our analysis of shared transcription factors and major pathways across all the tissues, please refer to Appendix I. For our analysis of multi-species gene embeddings, please refer to Appendix J.
4.3 Analysis of Gene Embeddings for Diseases
We generated gene embeddings for human pancreas cells from samples with and without SARS-CoV-2 infection, as depicted in Figure 4 (a) and (b). We identified specific genes from COVID samples that did not align with control samples, which piqued our interest. These genes, highlighted by a red circle in Figure 4 (c), could be interpreted as differentially functional genes in diseased cells.
We conducted GOEA for the genes of interest and discovered a close relationship among these gene enrichment results and the top 5 pathways associated with immune activity. These results are displayed in Figure 4 (d). For the genes within our target cluster, we utilized IPA to identify the Entrez name of these genes, and  $90.3\%$  (122/135) genes in our cluster are related to immunoglobulin.
Figure 3: Gene representation learning results for multimodal biological data. (a) represents the UMAPs of gene embeddings colored by tissue type and highlighted by biological system. (b) represents the UMAPs of gene embeddings colored by omics type. (c) represents the UMAPs of gene embeddings colored by common function groups. (d) is a Sankey plot [80] to show the overlap of different modalities in the same clusters. (e) shows the top5 pathways related to the genes in the special cluster discovered by GOEA. The bubble plots in this paper were created based on ggplot2 [102].
Figure 4: Gene embeddings from COVID samples and healthy samples. (a) represents the UMAPs of gene embeddings colored by functional groups. (b) represents the UMAPs of gene embeddings colored by datasets. (c) represents the gene embeddings colored by the conditions, and the red circle reflects the differential co-expression genes. (d) shows the top6 pathways related to the genes in the special cluster discovered by GOEA. (e) represents the causal network existing in the special cluster discovered by IPA. (f) represents the top diseases & biological functions discovered by IPA.
We could also infer the causal relationship existing in the gene regulatory activity of the immune system. For example, Figure 4 (e) showed a causal network inferred by IPA based on our genes cluster. PARP16, as an enzyme, can regulate ERN1 and EIF2AK3, and certain pathways are also related to this causal network. Moreover, we also showed the relation between the set of genes and Disease & Bio functions in Figure 4 (f). We identified top related Diseases & Bio functions ranked by negative logarithm of p-value, and all of these diseases could be interpreted as complications that may arise from new coronavirus infection [6, 64, 79, 69, 33, 7, 83, 84]. Our extra analyses for lung cancer data can be found in Appendix K.
4.4 Analysis of Gene Embeddings for Gene Function Prediction.
Here we intend to predict the dosage-sensitivity of genes related to genetic diagnosis (as dosage-sensitive or not) [90]. We used MuSe-GNN to generate gene embeddings for different datasets based on an unsupervised learning framework and utilized the gene embeddings as training dataset to predict the function of genes based on k-NN classifier. k-NN classifier is a very naive model and can reflect the contribution of gene embeddings in the prediction task.
In this task, we evaluated the performance of MuSe-GNN based on the dataset used in Geneformer [90, 31], comparing it to the prediction results based on raw data or Geneformer (total supervised learning). As shown in Table 2, the prediction accuracy based on gene embeddings from MuSe-GNN is the highest one. Moreover, the performance of gene embeddings from MuSe-GNN is better than Geneformer, which is a totally supervised learning model. Such finding proves the advantages of MuSe-GNN in the application of gene function prediction task. Further application analysis can be found in Appendix L.
Table 2: Accuracy for dosage-sensitivity prediction
MuSe-GNN (unsup)	Geneformer (sup)	Raw
Accuracy	0.77±0.01	0.74±0.06	0.75±0.01
In this paper, we introduce MuSe-GNN, a model based on Multimodal Machine Learning and Deep Graph Neural Networks, for learning gene embeddings from multi-context sequencing profiles. Through experiments on various multimodal biological datasets, we demonstrate that MuSe-GNN outperforms current gene embedding learning models across different metrics and can effectively learn the functional similarity of genes across tissues and techniques. Moreover, we performed various biological analyses using the learned gene embeddings, leveraging the capabilities of GOEA and IPA, such as identifying significant pathways, detecting diseases, and inferring causal networks. Our model can also contribute to the study of the pathogenic mechanisms of diseases like COVID and lung cancer, and improve the prediction performance for gene functions. Overall, the gene representations learned by MuSe-GNN are highly versatile and can be applied to different analysis frameworks.
At present, MuSe-GNN does not accept graphs with nodes other than genes as input. In the future, we plan to explore more efficient approaches for training large models related to Multimodal Machine Learning and extend MuSe-GNN to a more general version capable of handling a broader range of multimodal biological data.
This research was supported in part by NIH R01 GM134005, R56 AG074015, and NSF grant DMS 1902903 to H.Z. We appreciate the comments, feedback, and suggestions from Chang Su, Zichun Xu, Xinning Shan, Yuhan Xie, Mingze Dong, and Maria Brbic.
[1] Gene ontology enrichment analysis. URL http://geneontology.org.
[2] Ingenuity pathway analysis. URL https://www.qiagenbioinformatics.com/products/ingenuitypathway-analysis.
[3] 10x Genomics. 10x genomics acquires spatial transcriptomics. Science, 2018. URL https://www.10xgenomics.com/news/10x-genomics-acquires-spatial-transcriptomics.
[4] Adrian M Altenhoff, Clément-Marie Train, Kimberly J Gilbert, Ishita Mediratta, Tarcisio Mendes de Farias, David Moi, Yannis Nevers, Hale-Seda Radoykova, Victor Rossier, Alex