MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Data
Authors: Tianyu Liu
Yuge Wang
Zhitao Ying
Hongyu Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal Learning; Representation Learning; Graph Neural Network; Similarity Learning; Contrastive Learning; Computational Biology and Bioinformatics; Single-cell genomics
TL;DR: We propose MuSe-GNN to effectively generate meaningful gene embeddings.

Abstract: Discovering genes with similar functions across diverse biomedical contexts poses a significant challenge in gene representation learning due to data heterogeneity. In this study, we resolve this problem by introducing a novel model called Multimodal Similarity Learning Graph Neural Network, which combines Multimodal Machine Learning and Deep Graph Neural Networks to learn gene representations from single-cell sequencing and spatial transcriptomic data. Leveraging 82 training datasets from 10 tissues, three sequencing techniques, and three species, we create informative graph structures for model training and gene representations generation, while incorporating regularization with weighted similarity learning and contrastive learning to learn cross-data gene-gene relationships. This novel design ensures that we can offer gene representations containing functional similarity across different contexts in a joint space. Comprehensive benchmarking analysis shows our model's capacity to effectively capture gene function similarity across multiple modalities, outperforming state-of-the-art methods in gene representation learning by up to $\textbf{100.4}$%. Moreover, we employ bioinformatics tools in conjunction with gene representations to uncover pathway enrichment, regulation causal networks, and functions of disease-associated genes. Therefore, our model efficiently produces unified gene representations for the analysis of gene functions, tissue functions, diseases, and species evolution.

MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Data
Tianyu Liu Yale University tianyu. liu@yale...... edu
Yuge Wang Yale University yuge. wwng@yale
Rex Ying Yale University rex. rex.yingdyale.edd
Hongyu Zhao
Yale University hongyu. zhao@yale edu #; Corresponding author
Abstract
Discovering genes with similar functions across diverse biomedical contexts poses significant challenge in gene representation learning due data heterogeneity. this study, we resolve this problem by introducing a novel model called Multimodal Similarity Learning Graph Neural Network, which combines Multimodal Machine Learning and Deep Graph Neural Networks to learn gene representations from single-cell sequencing and spatial transcriptomic data. Leveraging 82 training datasets from 10 tissues, three sequencing techniques, and three species, create informative graph structures for model training and gene representations generation, while incorporating regularization with weighted similarity learning and contrastive learning to learn cross-data gene-gene relationships. This novel design ensures that offer gene representations containing functional similarity across different contexts in joint space. Comprehensive benchmarking analysis shows model's capacity to effectively capture gene function similarity across multiple modalities, outperforming state-of-the-arr methods in gene representation learning by up 97.5%. Moreover, employ bioinformatics tools in conjunction with gene representations to uncover pathway enrichment, regulation causal networks, and functions of disease-associated or dosage-sensitive genes. Therefore, our model efficiently produces unified gene representations for the analysis of gene functions, tissue functions, diseases, and species evolution.
Introduction
Progress in biological technology has broadened the variety of biological data, facilitating the exami- nation intricate biological systems. A prime example of such technology is single-cell sequencing, which allows for the comprehensive characterization of genetic information within individual cells 38 78). This technology provides access to the full range of a cell's transcriptomics. epigenomics and proteomics information, including gene expression (scRNA-seq) [46] 114), chromatin accessibil- ity (scATAC-seq) [20] [7]. methylation [60] (68) and anti-bodies [85 By sequencing cells from the same tissue at various time points, we can gain insight into patterns of cellular activity over time [30]. Moreover, spatial information of cellular activity represents an equally vital, additional dimension.
Such data defined spatial data. All of these data are known multi-omics data, and integrating multi-omics data for combined analysis poses a significant challenge.
However, the traditional idea of multi-omics data integration known using cells anchors [56] 87 39 12 26 76 is only partially suitable because of the following two challenges. 1. Different omics
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

data pose their own challenges. For example, the unit of observation of the spatial transcriptomic data different from other single-cell data, as a single spatial location contains mixed information (see the left part of Figure | a) from various cells (23] and it is not appropriate to generate spatial clusters based on gene expression similarity []6]. Current research [66] also indicates that chromosome accessibility feature is not powerful predictor for gene expression at the cell level. 2. The vast data volume in atlas-level studies challenges perforrmac computing, risking out-of-memory or time-out errors (58 [OI]. With nearly 37.2 trillion cells in the human body (77]. comprehensive
analysis is computationally infeasible. 3. Batch effects may adversely impact analysis results [54] by introducing noise. Consequently, efficient and powerful method focusing on multi-omics and multi-tissue data (referred to as multimodal biological data) analysis is urgently needed to address these challenges.
Acknowledging the difficulties arising from the cell-oriented viewpoint, prior work shifted focus to the gene perspective. Using gene sets as summary of expression profiles, based on natural selection during the species evolution, may provide a more robust anchor [8] Protein-coding genes are also thought to interact with drugs [19]. which are more relevant to diseases and drug discovery. Gene2vec [24] is a method inspired by Word2vec [8]. which learns gene representations by generating skip- gram pairs from the co-expression network. Recently, the Gene-based data Integration and ANalysis Technique (GIANT) [16] has been developed, based on Node2vee [34] and OhmNet [19]. to learn gene representations from both single-cell and spatial datasets. However, shown in Figure [I (b). sigmificant functional clustering for genes from different datasets but the same tissue was not observed based the gene embeddings from these two models, because these methods do not infer the similarity of associated genes from different multimodal data. Additionally, they did not offer metrics to quantitatively evaluate the performance of gene embeddings compared to baseline models.
Here we introduce a novel model called Multimodal Similarity Learning Graph Neural Network (MuSe-GNN) | for multimodal biological data integration from a gene-centric perspective. The overall workflow of MuSe-GNN is depicted in Figure ( Figure (b) shows MuSe-GNN's superior ability to learn functional similarity among genes across datasets by suitable model structure and novel loss function, comparing to GIANT and Gene2vec. MuSe-GNN GNN utilizes weight-sharing Graph Neural Networks (GNNs) to encode genes from different modalities into shared space regularized by the similarity learning strategy and the contrastive lcarning strategy. At the single-graph level, the design of graph neural networks ensures that MuSe-GNN can learn the neighbor information in each co-expression network, thus preserving the gene function similarity. At the cross-data level, integrate genes with similar functions
the similarity learning strategy ensures that MuSe-GNN into a common group, while the contrastive learning strategy helps distinguish genes with different functions. Furthermore, MuSe-GNN utilizes more robust co-expression networks for training and applies dimensionality reduction 89 to high-dimensional data (73.
To the best of our knowledge, this is the first paper in gene representation learning that combines the Multimodal Machine Learning (MMML) concept (9, (2] with deep GNNs [49] 81] design. Both approaches are prevalent in state-of- the-art (SOTA) machine learning research, inspiring us to apply them to the joint analysis of large-scale multimodal biological datasets []] As application examples, we first used the gene embeddings generated by MuSe-GNN cesses and causal networks for gene regulation in the human body. We also applied our model to analyze COVID and cancer datasets, aiming to unveil potential disease resistance mechanisms or complications based on specific differentially co-expressed genes. Lastly, we used gene embeddings from MuSe-GNN to improve the prediction accuracy for gene functions. Here genes are co-expressed means genes are connected with the same edge.
investigate crucial biological pro-
Given the lack of explicit metrics to evaluate gene embeddings, we proposed six metrics inspired by the batch effect correction problem (54) in single-cell data analysis. We evaluated our model using real-world biological datasets from one technique, and the benchmarking results demonstrated MuSe-GNN's significant improvement from 20.1 % to 97.5 % in comprehensive assessment. To summarize the advantages of our model, MuSe-GNN addresses the outlined problem about cross-data gene similarity learning and offers four major contributions: 1. Providing an effective representation learning approach for multi-structure biological data. 2. Integrating genes from different omics and tissue data into a joint space while preserving biological information. 3. Identifying co-located
I 'Codes of MuSe-GNN: httCNN/nt/////gthaaaaaaaaaaaa http://ggttthhcc/e/e http://hhhhaccac////aaaaaaaaaaaa
"Download links: Appendix M

---

Co-orgrss on grapts
Dann artleddngg
Tashwayy pooD
VON DDA
Spata
INPARRU EEROO
soATAC-ooa
Glis fancion pieditiio
Griphi hurofannatinn
MiSe-GNN
P-A8
b
MuSe-GNN
Gene2vec
GIANT
Sorea Datasat
somo, hewrttetttttto
scrrahhoor
soma_heeatt_
UMAP 2
Scita hoart
scma
SCITAL Hourt
amma, heart
sorma, heart DS
orma, heart H5
BOrT heart
srma heart
somaa Do
somna heart_47
SCTTa www.t__tttor. D11
UMAP1 1
Figure 1: The workflow of MuSe-GNN, the visualization of gene embeddings, and the problems of two exitsing methods, GIANT and Gene2vec. (a) The process of learning gene embeddings by MuSe-GNN. Here we highlight the difference between single-cell data and spatial data, and the major applications of gene embeddings. Each spot from single-cell data represents one cell, while each spot from spatial data represents mixture of cells. (b) UMAPs [63] for the gene embeddings of MuSe-GNN, Gene2vec and GIANT. They are colored by datasets. Based on UMAPs we can conclude that both Gene2vec and GIANT failed to learn the gene similarity based on the datasets from the same tissue, while MuSe-GNN produces meaningful embeddings agnostic to datasets.
genes with similar functions. 4. Inferring specialized causal networks of genes and the relationships between genes and biological pathways or between genes and diseases.
2 Related work
Co-expression Network Analysis. While direct attempts at joint analysis of gene functions across modalities are limited, there is relevant research in identifying correlation networks based on a single dataset. WGCNA [52] is a representative method that employs hierarchical clustering to identify gene modules with shared functions. However, as an early tool, WGCNA has restricted functionality. Its inference of co-expression networks necessitates more rigorous methods, and it cannot be directly applied to the milysss of multimodal biological data.
Network Based Biological Representation Learning. Apart from directly generating gene co- expression networks, learning quantitative gene representations in a lower dimensional space may better describe gene-gene relationships and facilitate downstream analyses. Gene2vec generates gene embeddings based on the co-cxpression network from a given database. However, it disregards the expression profile information and is based on the old Gene Expression Omnibus (GEO) up until 2017 (24). GIANT leverages Node2vec [34] and OhmNet [119] to learn gene representations from both single-cell and spatial datasets by constructing graphs and hypergraphs for training. However, this approach still over-compresses multimodal biological datasets by removing expression profiles. Moreover, their co-expression networks created by Pearson correlation have high false positive rate 82). Additionally, some of the datasets used by GIANT are of low quality (see Appendix AA. There are also methods sharing a similar objective of learning embeddings from other dataastss. GSS (71) aims to learn set of general representations for all genes from bulk-seq datasets [62] using Principal Component Analysis (PCA) and clustering analysis. However, it is for bulk-seg data and cannot be directly applied to single-cell datasets from different tissues. Gemini [103] focuses integrating
different protein function networks, with graph nodes representing proteins rather than genes.
Graph Transformer. In the deep learning domain, Transformer 95
100 is one of the most
successful models, leveraging seq2seq structure design, multi-head self aelf-attention design, and position encoding design. Many researchers have sought to incorporate the advantages of Transformer to graph structure learning. TransformConv [8] introduces multi-head attention mechanism to the graph version of supervised classification tasks and achieved significant improvements. MAGNA [98] considers legeer-level hop relationships in graph data to enhance node classification capabilities. Graphomer [[10] demonstrates the positive impact of the Transformer structure various tasks
3

---

using data from the Open Graph Benchmark Large-Scale Challenge 44]. which is further extended by GraphomerGD [TT. Recently, GPS [75 proposes a general Graph Transformer (GT) model by considering additional encoding types. Transformer architecture also contributes solutions to several biological questions [[13]. scBERT [008] generates gene and cell embeddings using pre-training to improve the accuracy of cell type annotations. The substantial impact of these efforts highlights the crucial contribution of the Transformer architecture to graph data learning.
3 Methods
In the following sections of introducing MuSe-GNN, we will elaborate on our distinct approaches for graph construction that utilize multimodal biological data, followed by explanation of weight-sharing network architecture and the elements of the final loss function.
3.1
Preliminaries
GNN. GNNs aim to learn the graph representation of nodes (features) for data with graph structure. Modern GNNs iteratively update the representation of a node by aggregating the representations its k-order neighbors (k 2 1) and combining them with the current representation. As described in [100, considering graph = (V, F} with nodes V {fi, U2, J, the AGGREGATE-COMMIE update for node v is defined as:
(L+1) AGGREGATE
:E : ((+)) COMBINE (1 ((+1)
where A (us) represents the neighbors of node v; in the given graph, and h and h ((+1) represent the node representation before and after updating, respectively.
Problem Definition. We address the gene embeddings generation task by handling multimodal
biological datasets, denoted as D = ((V,, E,)) Our goal is to construct a model M(. 0). designed to yield gene embeddings set & = {e,, ET} M(D,0) In this context, D represents the input, 0 represents the parameters, and & represents the output. In other words, we aim to harmonize gene information from diverse modalities within unified projection space, thereby generating consolidated gene representations.
3.2 Graph construction
Before constructing gene graphs, our first contribution involves the selection of highly variable genes (HVGs) for each dataset. These HVGs constitute a group of genes with high variance that can represent the biological functions of given expression profiles. Moreover, considering co-expression networks is important for gene representation learning because allows us to characterize gene-gene relationships. As sequencing depth, or the total counts of each cell, often serves as a confounding fac- tor in the co-cxpression networks inference [ we employ two unique methodologies, scTransform (35) and CS-CORE [88) process scRNA-seq and scATAC-seq data, thus creating gene expression profiles and co-expression networks unaffected by sequencing depth. For spatial transcriptomic data, focus is genes displaying spatial expression patterns. We use SPARK-X [116] to identify such genes and then apply scTransform and CS- ES-CORE. For detailed algorithmic information regarding these methods, please see Appendix B Additionally, to batch effects when estimating gene-gene correlations in Appendix [ In all our generated graphs (equivalent co-expression datasets), nodes represent genes and edges represent co-expression relation of genes.
demonstrate the immunity of CS-CORE
3.3 Cross-Graph Transformer
To capitalize on the strengths of the Transformer model during our training process, we integrate graph neural network featuring a heal self-attention design [8I) called TransformerConv, to incorporate co-expression information and generate gene embeddings. Details of TransformerCom can be found Appendix [3.4] Incorporating multimodal information can estimate more accurate gene embeddings, supported by Appendix D The cross-graph transformer can efficiently learn gene embeddings containing gene functions across different graphs, advocated by the comparison of different network structure choices in Appendix E.

---

Weight sharing, Given the variability among multimodal biological datasets, we employ weight- sharing mechanism ensure that our model learns shared information across different graphs. representing novel approach for learning cross-graph relationships. We also highlight the importance of weight-sharing design Appendix E.T
Embeedd common genes
Embeddiings of co-expressed genes
No gregroorooo overtap weisht
Emboddings different genes
Coufine
Lin
Datasets Deccder
Au
D2
Graph Transforeer
Dvc
L BCB
Hightere
Bample
Datasets
Graph Transfrrer
Anchors
LafaNCE
Graph datasets
Model Architecture
Loss Functions
Figure 2:
The overall model architecture and the design of loss functions for MuSe-GNN The color of nodes in the green block represents common/different genes across datasets. The brown block represents the network architecture of MuSe-GNN, and the blue block represents different loss function components of MuSe-GNN. The color gradients of the left two matrices represent different gene expression levels.
Datasets & Modalities Graph Transformer. Drawing inspiration from the hard weight-sharing pro- cedure [S] [T5], we not only employ datasct-specific Graph Transformer (GT) layers L1 L2. Ln for each graph (G₁, G2 .... G.,) from the same modality m. but also connect all these datasct-specific layers to set of shared GT layers, denoted as Dm This design showcases our novel approach to incorporating shahing into the GT framework. The forward process of MuSe-GNN, given dataset i with network parameter 0., is defined as follows:
X Dm (G;; (1,);
Datasets Decoder. Here we propose a dataset sppecicc decoder structure based on Multi-layer Perceptrons (MLP). This decoder model is crucial in reconstructing the co-expression relationships among different genes, showcasing our inventive use of MIP for this purpose. Given a graph G, and its corresponding gene embedding é;, the decoding process of MuSe-GNN, with network parameter dec,, is defined follows:
Erec DECODERMLP(EIC) Odec,))s
where Erec represents the reconstructed co-expression network.
3.4 Graph Reconstruction Loss (LECr)
Within single graph, we implement loss function inspired by the Graph Auto-encoder (GAE) 48 This function is designed to preserve two key aspects: 1. the similarity among genes sharing common functions, and 2. the distinctions among genes with differing functions. This innovative use of GAE-inspired loss function constitutes significant contribution the methodological design. For graph G; = (V,, E, ). the loss function for edge reconstruction is defined as:
e, ENNODER F.); Dene) = Dm (G, Oz, 0...)
Freec DECODER Mr.PDDE (dce,i); Fy FLATTEN Free); F3J FLATTEN
1 1
LBCE
Σ
E log Ey (1- log (1-'
EE
1=1
where ENCODERGNN respectively. LECF. denotes the computation of binary cross entropy loss (BCEL Loss) for the given input data. Free is the reconstructed adjacency matrix and /F/eel the length of its flatten version. Further justification of the model design can be found in Appendix F
DECODERMLP represent the encoder and decoder parts of our model,
5

---

3.5 Weighted-similarityy Learning (Csim)
To integrate shared biological information across graphs from disparate datasets, we fuse the re- construction loss of the input graph structure with a cosine similarity learning loss. In this process, we treat common HVGs between each pair of datasets as anchors. Our aim to maximize the a a for each common HVG across two arbitrary datasets cosine similarity (CosSIM(a, b) |ai 2- 2-]
(represented by the yellow blocks in Figure 2). utilizing their gene embeddings. However, in practice, different common HVGs may have different levels of functional similarity in two datasets, which is hard to be directly quantified. Consequently, we employ the shared community score an indirect measurement, which is incorporated as a weight for the cosine similarity of different common HVG pairs within the final loss function. Considering two graphs G. = (V,, Pij and Gj- {Vz E3 J. and one of their shared genes we identify the co-expressed genes of the given gene in both datasets, denoted as Nig and Njg- Thus, the weight Aijo for gene g can be expressed as follows:
[Nag 0 NNj
Xij9
(5)
NN U Njal
We can iterate over all shared genes from 1 to n between these two graphs, ultimately yielding a vector Xij = [Ae Aijnn. This vector encapsulates the community similarity between the two graphs across all common HVGs. We can then modify the cosine similarity of various gene pairs by first multiplying this vector with cosine similarities and then summing the resultant values across all genes. The negation of the outcome is our final weighted similarity loss, denoted as Lsim- The ablation test for weighed similarity loss can be found in Appendix E.1 More detailed explanations and evidence supporting this design in multimodal conditions can be found in Appendix
3.6 Self-supervised Graph Contrastive Learning (CinFONCE)
Specifically, when integrating multimodal biological data, employ the contrastive learning strategy 772 to ensure that functionally similar genes are clustered together as closely as possible, while functionally different genes are separated apart from each other. We utilize Information Noise Contrastive Estimation (InfoNCE) as a part of our loss function to maximize the mutual information between the anchor genes and genes with the same functions. This loss is applicable to different genes in two arbitrary graphs during the training process. In general. if we represent the embeddings of N genes as GeneN (e1, CN InfoNCE is designed minimize:
exp (e (T)
LInfaNCE = E log E exp (e kx/T)
(6)
where samples (ko, kr, }2...} compose a set of gene embeddings known as keys of one dictionary
and e query gene embedding. h4 represents a positive sample of and k, denotes a negative
sample of e. Equation 6 can be interpreted as a log loss of a (K + 1)-way Softmax classifier, which attempts classify e as k4. is temperature parameter. 7 set to 0.07 referenced in MoCo 41]
3.7 Final Loss Function
In summary, the training objective of MuSe-GNN for graph comprises three components:
min CBCE (DCE(DEEEDEEM E.) E (rossiM(e(CCommnn... ey (lommon,)))?! Rirty
AchintaNcr(e [Difr.] @ e, [DifT]]] e [ool @ e, (Diiii
where Common,, denotes the index set for common HVGs, Diff, represents the index set for different HVGs in graph i, and Difl indicates the index set for neighbors of DifT, in graph To expedite the training process and conserve memory usage, we sample graph j for each graph i during model training. We also employ multi-thread programming to accelerate the index set extraction process. Ao is the weight for InfoNCE loss. All of the components in MuSe-GNN are supported by ablation experiments in Appendix E.1 Details of parameee tuning can be found in Appendix E.2
4 Experiments
Datasets & Embeddings generation. Information on the different datasets used for different experiments is included at the beginning of each paragraph. The training algorithm of our model

---

Algorithm 1 Multimodal Similarity Learning Graph Neural Network (MuSe-GNN)
Input:
Model ENCODERGNN, DECODERMLP; Dataset D (1V, E.)) Number of epochs K; Neighbor overlap list X; Weight coefficient for contrastive learning Ac:
Helper Functions: A function to find the genes with the same name FINDCOMMON GENES; A function to find the genes with different name FINDDIFFGENES; A function to calculate cosine similarity COSSIM; A function to sample data from graph collection SAMPLE; A optimizer function to update weights ADAM.
Output: Model ENCODERGNN
1: INIT: initialize all parameters.
2: for in K steps do
3: for id, {V, E} in enumerate (D) do ENN.DERGNN((V,
4:
Bene); Erec + DECODERMML((E"" dde))
5: C + LBCE Erec, E)
6:
idnew: Venw, Enow J SAMPLE(GA E))
7:
Gnew + ENNODER ( (V new Enew 1 .enn)
8:
CCommon: Common + FINDCOMMONGENESCC Cnew)
9:
L C COS.COSSMMMMM. Coommmon Commoon > iid
CDIT: DiR: CDIIN Diftyy & FINDDIPFGENES(E, Cvew)
L L + >cCInSONCE (Dpir + eparx SDi eDiitty
12:
Oena ADAM(L, C,, Oenc); Odee + ADAM(C, Odoc)
13:
Return ENCODERGNN
outlined Algorithm We stored the gene embeddings the AnnData structure provided by Scanpy [104). To identify groups of genes with similar functions, we applied the Leiden clustering algorithm [91] to the obtained gene embeddings. Details can be found in Appendix E.4
Evaluation metries. For the benchmarking process, we used six metrics: edge AUC (AUC) (74). common genes ASW (ASW) [59], common genes graph connectivity (GC) [59] common genes iLISI (iLISI) 59, common genes ratio (CGR), and neighbors overlap (NO) to provide a comprehensive comparison. Detailed descriptions of these metrics can be found in Appendix H. We computed the metrics for the different methods and calculated the average rank (Avg Rank E [1. Moreover, to evaluate the model performance improvement, we applied min-max scaling to every metric across different models and computed the average score (Avg Score E [0. 1)).
Baselines. We selected eight models as competitors for MuSe-GNN, The first group of methods stems from previous work on learning embeddings for biomedical data, including Principal Component Analysis (PCA) (74) used by GSS, Gene2vec, GIANT (the SOTA model for gene representation learning), Weight-sharing Multi-layer Auto-encoder (WSMAE) used by [109] and scBERT (the SOTA model with pre-training for cell type annotation). The second group of methods comprises common unsupervised learning baseline models, including GAE, VGAE [88 and Masked Auto- encoder (MAE) (the SOTA model for self-supervised learning) VGAE, WSMAE and MAE have training parameter sizes between 52.5 and 349 M, and all models are tuned to their best performance. Details are shown in Appendix F.2
42.. MuSe-GNN GIANT, GAE,
Biological applications. For the pathway analysis, we used Gene Ontology Enrichment Analysis (GOEA) ( to identify specific biological pathways enriched in distinct gene clusters with common functions. Moreover, we used Ingenuity Pathway Analysis (IPA) (2 to extract biological information from the genes within various clusters, including causal networks [50] and sets of diseases and biological functions. Biological pathways refer to processes identified based on the co-occurrence of genes within a particular cluster. The causal network depicts the relationships between regulatory genes and their target genes. Disease and biological function sets facilitate the discovery of key processes and complications associated with specific diseases. Using gene embeddings also improves the performance of models for gene function prediction. To visualize gene embeddings in a low- dimensional space, we utilized Uniform Manifold Approximation and Projection (UMAP) 63
7

---

Table 1: Avg Score across different tissues, Standard deviations are reported in Appendix E.3
Methods
Lung
Kidney
Thymus
Spleen
Cerebrum
Cerebellum
PBMC
PCA
0.52
0.48
0.56
0.47
0.56
0.60
0.51
0.62
0.53
0.51
Gene2vec
0.40
0.37
0.33
0.29
0.21
0.31
0.24
0.27
0.31
0.19
GIANT
0.50
0.40
0.33
0.38
0.58
0.33
0.56
0.29
0.28
0.28
WSMAE
0.50
0.54
0.57
0.50
GAE
0.61
0.45
0.58
0.40
0.56
0.58
0,52
0.56
0.60
0.54
VGAR
0.64
0.33
MAE
0.36
0.47
0.50
0.45
0.41
0.52
0.39
0.50
0.49
0.50
scBERT
0.41
0.49
0.55
0.62
0.17
0.58
0.46
0.60
0.61
0.58
MuSeGNN
0.77
0.92
0.89
0.89
0.94
0.80
0.95
0.90
0.92
4.1
Benchmarking Analysis
We executed each method 10 times by using the same setting of seeds to show the statistical significance based on datasets across different tissues. The performance comparison of nine gene embedding methods is presented in Tables [ and 14 Based on these two tables, MuSe-GNN outperformed its competitors in terms of both average ranks and average scores across all the tissues. Based on Table [ for major tissues, such as heart and lung, MuSe-GNN's performance was 20. % higher than the second-best method and 97.5% higher than the second-best method in heart and lung tissue, respectively. According to Appendix E.3 MuSe-GNN's stability was also demonstrated through various metries by comparing standard deviations, including AUC, GC, and NO. In contrast, methods such as Gene2vec, GAE, VGAE, MAE and scBERT exhibited significant instability their evaluation results for kidney or thymus. Consequently, performing model for learning gene representation based on datasets from different tissues, making it applicable learn gene embeddings from diverse multimodal biological data.
concluded that MuSe-GNN is the best
4.2 Analysis of Gene Embeddings from Multimodal Biological Data
In Figure 3 we displayed the integration results for multimodal biological data from Humans. Figure (a) and (b) demonstrated that MuSe-GNN could successfully integrate genes from different modalities into a co-embedded space, allowing us to identify functional groups using the Leiden algorithm shown in Tigure((c) Furthermore, Figure 3 (d) revealed that most of the clusters in (c) were shared across different modalities. We also identified three significant functional groups in Figure [ the nervous system (predominantly composed of the cerebrum and cerebellum (26)), the cardiovascular system (mainly composed of heart, lung, and kidney (29)). ) and the immunology system (primarily consisting of spleen, liver, and peripheral blood mononuclear cells (PBMC) (70). All systems important in regulating the life activities of the body. We also uncovered a pre-epigenstics group (mainly consisting of ScATAC-seg data without imprinting, modification, or editing), emphasizing the biological gap existing in multi-omics and the importance of post-transcriptional regulation.
Using GOEA, could identify significant pathways enriched by different co-embedded gene clusters. For instance, Figure (d) displayed the top 5 pathways in an immunology system cluster. The rank caleulated based on the negative logarithm of the false discovery rate. Since all top pathways were related to immunological defense and response, it further supported the accuracy of our embeddings in representing gene functions. For our analysis of shared transcription factors and major pathways across all the tissues, please refer to Appendix [] For our analysis of multi-species gene embeddings, please refer Appendix
4.3 Analysis of Gene Embeddings for Diseases
We generated gene embeddings for human pancreas cells from samples with and without SARS-CoW 2 infection, depicted in Figure 4 (a) and (b). We identified specific genes from COVID samples that did not align with control samples, which piqued our interest. These genes, highlighted by a red circle in Figure 4 (c), could be interpreted differentially functional genes in diseased cells.
We conducted GOEA for the genes of interest and discovered close relationship among these gene enrichment results and the top 5 pathways associated with immune activity. These results are displayed in Figure 4 (d). For the genes within our target cluster, we utilized IPA to identify the Entrez name of these genes, and 90.3% (122/135) genes cluster related to immunoglobulin.

---

Farcsol grogle
cenebellum
corebrum
kinney
liver
scelac
king
scma spatiel
pancreas
jemetteahtly
soleen
thymus
seferse response
COUNT
Inmure system piocees
repprrse to strava
miuute response
interndary rasporou
Ennchment
Figure 3:
Gene representation learning results for multimodal biological data. (a) represents the UMAPs of gene embeddings colored by tissue type and highlighted by biological system. (b) represents the UMAPs of gene embeddings colored by omics type. (c) represents the UMAPs of gene embeddings colored by common function groups. (d) is Sankey plot [80] to show the overlap of different modalities the same clusters. (e) shows the top5 pathways related to the genes in the special cluster discovered by GOEA. The bubble plots in this paper were created based on ggplor2 102]
Cow Daooder
sewdaay Daocle
Inmangeebuin
Hesocogia Ciense
Resnduuiie Diesse
Pors consiiiion tnpficaaion
Abcttryy
-2rip-oolue)
Figure 4: Gene embeddings from COVID samples and healthy samples. (a) represents the UMAPs of gene embeddings colored by functional groups. (b) represents the UMAPs of gene embeddings colored by datasets. (c) represents the gene embeddings colored by the conditions, and the red circle reflects the differential co-expression genes.
(d) shows the top6 pathways related to the genes in the special cluster discovered by GOEA. (e) represents the causal network existing in the special cluster discovered by IPA. (f) represents the top diseases & biological functions discovered by IPA.
We could also infer the causal relationship existing in the gene regulatory activity of the immune system. For example, Figure 4 (e) showed causal network inferred by IPA based on our genes cluster. PARP16, as an enzyme, can regulate ERNI and EIF2AK3, and certain pathways are also related to this causal network. Moreover, we also showed the relation between the set of genes and Disease Bio functions in Figure (f). We identified top related Diseases & Bio functions ranked by negative logarithm of p-value, and all of these diseases could be interpreted as complications that Our extra analyses for lung
may arise from new coronavirus infection 6
cancer data can be found in Appendix K

---

4.4 Analysis of Gene Embeddings for Gene Function Prediction.
Here we intend to predict the losage-sensitivity of genes related to genetic diagnosis (as dosage- sensitive not) [90]. We used MuSe-GNN to generate gene embeddings for different datasets based on an unsupervised learning framework and utilized the gene embeddings as training dataset to predict the function of genes based on k-NN classifier. k-NN classifier is very naive model and can reflect the contribution of gene embeddings the prediction task.
In this task, we evaluated the performance of MuSe-GNN based on the dataset used in Geneformer 190, 31], comparing it to the prediction results based on raw data Geneformer (total supervised learning). As shown in Table 2
the prediction accuracy based on gene emebddings from MuSe-GNN the highest one. Moreover, the performance of gene embeddings from MuSe-GNN is better than Geneformer, which is totally supervised learning model. Such finding proves the advantages of MuSe-GNN in the application of gene function prediction task. Further application analysis can be found in Appendix
Table Accuracy for dosage-sensitivity prediction
MuSe-GNN (unsup)
Geneformer (sup)
Raw
Accuracy
0.77+0.01
0.74+0000
0.75+:0.01
Conclusion
In this paper, we introduce MuSe-GNN, model based Multimodal Machine Learning and Deep Graph Neural Networks, for learning gene embeddings from multi-context sequencing profiles. Through experiments various multimodal biological datasets, demonstrate that MuSe-GNN outperforms current gene embedding learning models across different metrics and can effectively learn the functional similarity of genes across tissues and techniques. Moreover, we performed various biological analyses using the learned gene embeddings, leveraging the capabilities GOEA and IPA, such identifying significant pathways, detecting diseases, and inferring causal networks. Our model can also contribute to the study of the pathogenic mechanisms of diseases like COVID and lung cancer, and improve the prediction performance for gene functions. Overall, the gene representations learned by MuSe-GNN are highly versatile and can be applied to different analysis frameworks.
At present, MuSe-GNN does not accept graphs with nodes other than genes as input. In the future, we plan to explore more efficient approaches for training large models related to Multimodal Machine Learning and extend MuSe-GNN to a more general version capable of handling a broader range of multimodal biological data.
6
Acknowledgements
This research was supported in part by NIH ROI GM134005, R56 AG074015, and NSF grant DMS 1902903 to H.Z. We appreciate the comments, feedback, and suggestions from Chang Su, Zichun Xu, Xinning Shan, Yuhan Xie, Mingze Dong, and Maria Brbic.
References
[1] Gene ontology enrichment analysis. URL MtLLattp.//gneenttoogy.oog
[2] Ingenuity pathway analysis. products/ingenie httplicss/iignuutttttttaeeeeeeee anasiis
URL
http://ww..ggaeenboooffoorrttts com/
[3] 10x Genomics. Science, 2018.
10x
genomics acquires spatial transcriptomics.
URL
https://www. 10xgenomics...... seggenmmmiss.ccm//esssss com/news/
10x sap.accssaaiiirssspsaaaaaaaaaaaa twwnscriptomicc........
[4] Adrian Altenhoff, Clément-Marie Train, Kimberly J Gilbert, Ishita Mediratta, Tarcisio Mendes de Farias, David Moi, Yannis Nevers, Hale-Seda Radoykova, Victor Rossier, Alex
10

---

Warwick Vesztrocy, et al. Oma orthology in 2021: website overhaul, conserved isoforms, ancestral gene order and more. Nucleic acids research, 49(D1):D373-DD D379, 2021.
[5] Tallulah Andrews and Martin Hemberg. False signals induced by single-cell imputation. F1000Research, 7, 2018.
[6] Khashayar Aram, Anant Patil, Mohamad Goldust, and Fateme Rajabi. Covid-19 19 and exacerba- tion of dermatological diseases: a review of the available literature. Dermatologic Therapy, 34 (6):e15113, 2021.
[7] Arash Ardestani Zadch and Davood Arab. Covid-19 19 and male reproductive system: pathogenic features and possible mechanisms, Journal of molecular histology, 52:869-878, 2021.
[8] Ricard Argelaguet, Anna SE Cuomo, Oliver Stegle, and John C Marioni. Computational principles and challenges single-cell data integration. Nature biotechnology. 39(10):1202- 1215, 2021.
[9] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-444 443, 2018.
[10] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[11] A Sina Booeshaghi, Ingileif B Hallgrímsdóttir, Ángel Gälvez-Merchán, and Lior Pachter. Depth normalization for single-cell genomics count data. bioRxiv, pages 2022-05, 2022.
[12] Andrew Butler, Paul Hoffman, Peter Smibert, Efthymia Papalexi, and Rahul Satija. Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nature biotechnology, 36(5):4 1--20, 2018.
[13] Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. International Conference on Machine Learning, pages 1204-1215 PMLR, 2021.
[14] Junyue Cao. Diana R O' O'Day, Hannah A Pliner, Paul D Kingsley, Mei Deng, Riza M Daza, Michael A Zager, Kimberly A Aldinger, Ronnie Elecher-Gonent Fan Zhang, et al. human cell atlas of fetal gene expression. Science, 370(65 (7066518)cabb77221 2020.
[15) R Caruana. Multitask learning: A knowledge-basee source of inductive bias In Proceedings of the Tenth International Conference on Machine Learning, pages 41-48. Citeseer, 1993.
[16] Hao Chen, Nam D Nguyen, Matthew Ruffalo, and Ziv Bar-Joseph. A unified analysis of atlas single cell data. bioRxiv, 2022.
[17] Xi Chen, Ricardo Miragaia, Kedar Nath Natarajan, and Sarah A Teichmann. A rapid and robust method for single cell chromatin accessibility profiling. Nature communications, 9(): 1-9, 2018
[18] Kenneth Ward Church. Word2vec. Natural Language Engineering, 23(1):155-162, 2017.
[19] Kelsy C Cotto, Alex Wagner, Yang- Yang Feng, Susanna Kiwala, Adam Coffman, Gregory Spies, Alex Wollam, Nicholas C Spies, Obi I. Griffith, and Malachi Griffith. Dgidh 3.0: redesign and expansion of the drug-gene interaction database. Nucleic acids research, 46(D1): 2018.
[20] Darren Cusanovich, Riza Daza, Andrew Adey, Hannah A Pliner, Lena Christiansen, Kevin Gunderson, Frank J Steemers, Cole Trapnell, and Jay Shendure. Multiplex single-cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348(6237):110-144 2015.
[21] C Domínguez Conde, C Xu, LB Jarvis, DB Rainbow, SB Wells, T Gomes, SK Howlett, O Suchanek, K Polanski, HW King, al. Cross-tissue immune cell analysis reveals tissue- specific features in humans. Science, 17166977 2022.

---

[22] Mingze Dong and Yuval Kluger. Towards understanding and reducing graph structural noise for gnns. 2023.
[23] Rui Dong and Guo- Cho-Cheng Yuan. Spatialdwls: accurate deconvolution of spatial transcriptomic data. Genome biology, 22(1):145, 2021.
[24] Jingcheng Du Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vee: distributed representation genes based on co-expression. BMC genomics, 20(1):7-15, 2019.
[25] Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. 2021. URL httppps://ppeereviieee.. id=pW2Q2xI.IMM
[26] Ramazan Durmaz, Metin Ant Atasoy, Gül Durmaz, Baki Adapinar, Ali Arslantas, Aydin Aydinli, and Esref Tel. Multiple nocardial abscesses of cerebrum, cerebellum and spinal cord, causing quadriplegia. Clinical neurology and neurosurgery, 103(1):59-62 -2:::299-22220 2001.
[27] Yasha Ektefaic, George Dasoulas, Ayush Noori, Maha Farhat, and Marinka Zitikk Multimodal learning with graphs. Nature Machine Intelligence, pages 1-11, 2023.
[28] HuBMAP Consortium al. The human body at cellular resolution: the nih human biomolecu- lar atlas program. Nature, 574(7777:187-192, 2019.
[29] HE Fessler. Heart-lung interactions: applications in the critically ill. European Respiratory Journal, 10(1):226-237, 1997.
[30] David S Fischer, Anna K Fiedler, Eric M Kernfeld, Ryan MJ Genga, Aimée Bastidas-Pone Mostafa Bakhti, Heiko Lickert, Jan Hasenauer, Rene Machr, and Fabian Theis. Inferring population dynamics from single-cell ma-sequencing time series data. Nature biotechnology. 37(4):461-468 2019.
[31] Oscar Franzén, Li-Ming Gan, and Johan LM Björkegren. Panglaodh: web server for exploration of mouse and human single-cell rma sequencing data. Database, 2019:baz046,, 2019.
[32] Toni Gabaldón and Eugene V Koonin. Functional and evolutionary implications of gene orthology. Nature Reviews Genetics, 14(5):360- 0666000-6666000 2013.
[33] Suhita Gayen Nee Betal, Pedro Urday, Huda B Al-Kouatly, Kolawole Solarin, Joanna SY Chan, Sankar Addya, Rupsa Boelig, and Zubair H Aghai. Covid-19 19 infection during pregnancy induces differential gene expression in human cord blood cells from term neonates. Frontiers in Pediatrics, 10:547, 2022.
[34] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855 864, 2016.
[35] Christoph Hafemeister and Rahul Satija. Normalization and variance stabilization of single-cell ma-seq data using regularized negative binomial regression. Genome biology, 20(1): 1011) 2019.
[36] Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[37] Heonjong Han, Jae- Won Cho, Sangyoung Lee, Ayoung Yun, Hyojin Kim, Dasom Bae, Sunmo Yang, Chan Yeong Kim, Muyoung Lee, Eunbeen Kim, al. Tirust v2: an expanded reference database of human and mouse transcriptional regulatory interactions. Nucleic acids research, 46(D1):D380-D336 2018
[38] Xiaoping Han, Ziming Zhou, Lijiang Fei, Huiyu Sun, Renying Wang, Yao Chen, Haide Chen, Jingjing Wang, Huanna Tang, Wenhao Ge, et al. Construction of human cell landscape at single-cell level. Nature, 58 3300730)) 2020.
12

---

[39] Yuhan Hao, Stephanie Hao, Erica Mandersen-Niseey William M Mauck, Shiwei Zheng, Andrew Butler. Maddie Lee, Aaron Wilk, Charlotte Darby, Michael Zager, al. Integrated analysis of multimodal single-cell data. Cell, 184(13):3573-3587 2021.
[40] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[41] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729 9738, 2020.
[42] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[43] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang. Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594-604, 2022.
[44] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:221 18-22133, 2020.
[45] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944 199444499-996, 2021.
[46] Byungjin Hwang. Ji Hyun Lee, and Duhee Bang. Single-cell ma sequencing technologies and bioinformatics pipelines. Experimental & molecular medicine, 50(8): 2018.
[47] Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity neural network functions. arXiv preprint urXiv:2302. 10886, 2023.
[48] Thomas N Kipf and Max Welling. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
[49] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https //openrvview. www.:eee/ffooeooee net//oorumid--SJU4yyy
[50] Andreas Krämer, Jeff Green, Jack Pollard Jr, and Stuart Tugendreich. Causal analysis ap- proaches ingenuity pathway analysis. Bioinformaties, 30(4):523-530, 2014.
[51] M Krassowski. Complexupset: Create complex upset plots using ggplot2 components. package version 0.5, 18, 2020.
[52] Peter Langfelder and Steve Horvath, Wgena: an r package for weighted correlation network analysis. BMC bioinformatics, 9(1):1113, 2008
[53] Patty J Lee, Philip Blood, Katy Börner, Judith Campisi, Feng Chen, Heike Daldrup-Link Phil De Jager, Li Ding, Francesca E Duncan, Oliver Eickelberg, et al. Nih sennet consortium: Mapping senescent cells in the human body to understand health and disease, 2022.
[54] Jeffrey T Leek, Robert B Scharpf, Héctor Corrada Bravo, David Simcha, Benjamin Langmead, Evan Johnson, Donald Geman, Keith Baggerly, and Rafael A Irizarry. Tackling the widespread and critical impact of batch effects in thghthhroughhut data. Nature Reviews Generics, 1 11100)733-773 2010.
[55] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint 20X.:22009033430, 2022.
[56] Xiang Lin, Tian Tian, Zhi Wei, and Hakon Hakonarson. Clustering of single-cell multi-omics data with multimodal deep learning method. Nature communications, 13(1):7705, 2022.
13

---

[57] Yinyin Liu, Janusz A Starzyk, and Zhen Zhu. Optimized approximation algorithm in neural networks without overfitting. IEEE transactions on neural networks, 19(6):983-999) 2008.
[58] Romain Lopez, Jeffrey Regier, Michael B Cole, Michael I Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, 15(12):1033-1058 2018.
[59] Malte D Luecken, Maren Büttner, Kridsadakorn Chaichoompu, Anna Danese, Marta Interlandi, Michaela Müller, Daniel C Strobl, Luke Zappia, Martin Dugas, Maria Colomé et al. Benchmarking atlas-level data integration in single-cell genomics. Nature methods, 19(1):
41 41-50, 2022.
[60] Chongyuan Luo, Christopher L Keown, Laurie Kurihara, Jingtian Zhou, Yupeng He, Junhao Li, Rosa Castanon, Jacinta I Lucero, Joseph R Nery, Justin P Sandoval, al. Single-cell methylomes identify neuronal subtypes and regulatory elements in mammalian cortex. Science, (57(6351))6000-66) 2017.
[61] Kaili Ma, Haochen Yang, Han Yang, Tatiana Jin, Pengfei Chen, Yongqiang Chen, Bara- keel Fanseu Kamhoua, and James Cheng. Improving graph representation learning by con- trastive regularization. arXiv preprint 1rXiv:2101.11525 2021.
[62] Samuel Marguerat and Jürg Bähler. Rna-seq: from technology to biology. Cellular and molecular life sciences, 67:569-579, 2010.
[63] Leland Melnnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
doi: 10.21 22110005sss.O0S 00061. URL https://t://///// org/ 21s05/joss 00861
[64] Bekir Fatih Meral. Parental views of families of children with autism spectrum disorder and developmental disorders during the covid-19 pandemic. Journal of Autism and Developmental Disorders, 52(4):1712-13) 17244 2022.
[65] Diganta Misra. Mish: self regularized non-monotonic activation function. arXiv preprint arXiv:/908.0800 2019.
[66] Yujie Mo, Liang Peng. Jie Xu, Xiaoshuang Shi, and Xiaofeng Zhu. Simple unsupervised graph representation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36 (7):7797-7805, Jun. 2022. doi: 10. (609///ai..36667. 20748. Uttpttss://oojs aaai..iss..a.................... org/ index
[67] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
[68] Ryan M Mulqueen, Dmitry Pokholok, Steven J Norberg, Kristof A Torkenezy, Andrew J Fields, Duanchen Sun, John R Sinnamon, Jay Shendure, Cole Trapnell, Brian J O' OoRa,,, et al. Highly scalable generation of dna methylation profiles single cells. Nature biotechnology. 36(5):428-455 431, 2018
[69] Edward Needham, Sherry H-Y Chou, Alasdair J Coles, and David Menon. Neurological implications of 1ovid-19 infections. Newrocritical care, 32:667-671, 2020.
[70] Rana Nikzad, Laura Angelo, Kevin Aviles-Padilla, Duy T Le, Vipul K Singh, Lynn Bimler, Milica Vukmanovic-Steçic. Elena Vendrame, Thanmayi Ranganath, Laura Simpson, et al. Human natural killer cells mediate adaptive immunity to viral antigens. Science immunology, 4(35)ceaa88 16, 2019.
[71] Sehyun Oh, Ludwig Geistlinger, Marcel Ramos, Daniel Blankenberg, Marius van den Beek, Jaclyn N Taroni, Vincent J Carey, Casey S Greene, Levi Waldron, and Sean Davis. Ge- nomissupersignature facilitates interpretation of ma-seq experiments through robust, efficient comparison to public databases. Nature Communications, 13(1):3695, 2022.
[72] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018.
14

---

[73] Subarna Palit, Christoph Heuser, Gustavo P De Almeida, Fabian J Theis, and Christina E Zielinski. Meeting the challenges of nigh-dimensional single-cell data analysis in immunology. Frontiers in immunoloyy, 10:1510: 1515, 2019.
[74] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2883 2011.
[75] Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35: 14501-14515, 2022.
[76] Yanay Rosen, Maria Brbić, Yusuf Roohani, Kyle Swanson, Ziang Li, and Jure Leskovec. Towards universal cell embeddings: Integrating single-cell ma-seq datasets across species with saturn. Biorxiv: the Preprint Server for Biology, 2023.
[77] Orit Rozenblatt- Michael JT Stubbington, Aviv Regev, and Sarah A Teichmann. The human cell atlas: from vision to reality. Nature, 550(7677)451-453, 2017.
[78] Eaaonne- Saliba, Alexander J Westermann, Stanislaw A Gorski, and Jörg Vogel. Single-cell rea advances and future challenges. Nucleic acids research, 42(11))8845-888) 2014.
[79] Amanda Salih, Aaron Chin, Manisha Gandhi, Amir Shamshirsaz, Hennie Lombaard, and Joud Hajjar. Hereditary angioedema and covid-19 19 during pregnancy: Two case reports. The Journal of Allergy and Clinical Immunology: In Practice, 11(3):611-96 962, 2023.
[80] Pierre Sassoulas, Anazalea, and Marcomanz. pySankey, 2018. URL https: htt.gggtttuucccooo com/ anazalea/pySankey
[81] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classifica- tion. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, IJCAI-21, pages 1548-1554. International Joint Confer- ences on Artificial Intelligence Organization, 8 2021. doi: 00224963/jccii222212444 URL
bettp::/ddoi.or.. 24963/1jcaaaa 2001/21422221111 Main Track.
[82] Michael A Skinnider, Jordan W Squair, and Leonard Foster. Evaluating measures of association for single-cell transcriptomics. Nature methods, 16(5):381-386 2019.
[83] Rajneesh Srivastava, Swapna Vidhur Daulatabad, Mansi Srivastava, and Sarath Chandra Janga.
Role of sars-cov-2 in altering the ma-binding protein and mirna-directed post-transeriptional regulatory networks in humans. International journal of molecular sciences, 21(19):7090, 2020.
[84] Won Sriwijitalai and Viroj Wiwanitkit. Hearing loss and covid- 19: note. American journal of otolaryngology. 2020.
2020.
[85] Marlon Stoeckius, Christoph Hafemeister, William Stephenson, Brian fouck-Loomin Pratip K Chattopadhyay, Harold Swerdlow, Rahul Satija, and Peter Smibert. Simultaneous epitope and transcriptome measurement in single cells. Nature methods, (4(9))865-866 2017.
[86] BJ Strober, Reem Elorbany, K Rhodes, Nirmal Krishnan, Karl Tayeb, Alexis Battle, and Yoav Gilad. Dynamic genetic regulation of gene expression during cellular differentiation. Science, 12877747) 2019.
[87] Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija. Compre- hensive integration of single-cell data. Cell, 177(7):1888-190)) 2019.
[88] Chang Su, Zichun Xu, Xinning Shan, Biao Cai, Hongyu Zhao, and Jingfei Zhang. Cell-type- specific co-expression inference from single cell rna-sequencing data. Nature Communications, 14(1):4846, 2023.
15

---

[89] Masashi Sugiyama. Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. Journal of machine learning research, 8(5), 2007.
[90] Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, al. Transfer learning enables predictions in network biology. Nature, pages 1-9, 2023.
[91] Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. From louvain leiden: guaranteeing well-connected communities. Scientific reports, (():11-12,0013
[92] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852- 7862, 2020.
[93] Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear repre- sentations. In International Conference on Machine Learning, pages 10434-10443, PMLR, 2021.
[94] David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine I. Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell, 174(3):716-729, 2018.
[95] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[96] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Represen- tations, 2018.
[97] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3): 261-272, 2020.
[98] Guangtao Wang. Rex Ying, Jing Huang. and Jure Leskovec. Multi-hop attention graph neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pages 3089-3096. International Joint Conferences Artificial Intelligence Organization, 2021. doi: 10.249633iccai.2222/242 URL https
//doi.orr/// 10. 24963/129111
1j cai 2021/4222222
Main Track.
[99] Juexin Wang, Anjun Ma, Yuzhou Chang, Jianting Gong, Yuexu Jiang, Ren Qi, Cankun Wang, Hongjun Fu, Qin Ma, and Dong Xu. segnn is novel graph neural network framework for single-cell rna-seq analyses. Nature communications, 12(1):1-11, 2021.
[100] Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, and Steven C.H. Hoi. VD-BERT: A Unified Vision and Dialog Transformer with BERT. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 3325- 3338, Online, November 2020. Association for Computational Linguistics. doi: 10. 18.38853/v1/ 2020. 1.:00.mmll-maan .69. URL https: http::::////aaaaaahhlaoa .rg 20rg/2020..r emnlp main 269
[101] Yuge Wang, Tianyu Liu, and Hongyu Zhao. Respan: a powerful batch correction model for serna-seq data through residual adversarial networks. Bioinformatics, 38(16):3942-3999, 2022.
[102] Hadley Wickham. ggplot2. Wiley interdisciplinary reviews: computational statistics, 3(2): 180-185, 2011.
[103] Addie Woicik, Mingxin Zhang, Hanwen Xu, Sara Mostafavi, and Sheng Wang. Gemini: memory-efficient integration of hundreds of gene networks with high-order pooling. Bioinfor- matics, 39, 06 2023. doi: 10. 10.0999biomfommaics/haaa444
16

---

[104] F Alexander Wolf, Philipp Angerer, and Fabian Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19(1):1-5, 2018.
[105] Renjie Xie, Wei Xu, Yanzhi Chen, Jiabao Yu, Aiqun Hu, Derrick Wing Kwan Ng, and A. Lee Swindlehurst. A generalizable model-and-data driven approach for open-set rff authentication. IEEE Transactions on Information Forensics and Security, 16:4435-4450, 2021. doi: 10.1109/TIFS.2221 2021.11666...
[106] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representatins, 2019. URL
https://openreview nwt http:/ooonnnnn-ddaaiiseeee
[107] Mingcong Xu, Xuefeng Bai, Bo Ai, Guorui Zhang, Chao Song, Jun Zhao, Yuezhu Wang, Ling Wei, Fengcui Qian, Yanyu Li, et al. Tf-marker: a comprehensive manually curated database for transcription factors and related markers in specific cell and tissue types in human. Nucleic acids research, 50(D1):440)) D412, 2022.
[108] Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang. Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. schert as large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4(10):852-8)) 856-866, 2022.
[109] Karren Dai Yang, Anastasiya Belyaeva, Saradha Venkatachalapathy, Karthik Damodaran, Abigail Katcoff, Adityanarayanan Radhakrishnan, GV Shivashankar, and Caroline Uhler. Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nature commumications, 12(1):1-10, 2021.
[110] Chengxuan Ying. Tianle Cai, Shengjie Luo. Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 11-28877-28888 2021.
[111] Bohang Zhang, Shengjie Luo, Liwei Wang. and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations, 2023. URL
[112] Di Zhang, Yanxiang Deng, Petra Kukanja, Eneritz Agirre, Marek Bartosovic, Mingze Dong, Cong Ma. Sai Ma, Graham Su, Shuozhen Bao, et al. Spatial epigenome-transcrrioonme co-profiling of mammalian tissues. Nature, pages 1-10, 2023.
[113] Shuang Zhang, Rui Fan, Yuti Liu, Shuang Chen, Qiao Liu, and Wanwen Zeng. Applications transformer- based language models in bioinformatics: A survey. Bioinformatics Advances,
2023.
[114] Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications, 8(1):1-12, 2017.
[115] Kaixiong Zhou, Xiao Huang, Qingquan Song, Rui Chen, and Xia Hu. Auto-gnn: Neural architecture search of graph neural networks. Frontiers in Big Data, 5, 2022. ISSN 2624-909X. doi: 10.3389/fdata 1 1029307. URL https:tts::/// h/ www.tiiiniiiiinnnn.ommmmm org/ articles/110 3389/Edata. 2022. 1029307
[116] Jiaqiang Zhu, Shiquan Sun, and Xiang Zhou. Spark-x: non-parametric modeling enables scalable and robust detection of spatial expression patterns for large spatial transcriptomic studies. Genome Biology, 22(1):1-25, 2021.
[117] Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, and Danai Koutra. Graph neural networks with heterophily. Proceedings of the AAAI Conference on Artificial Intelligence, 35( 12::1 161-11176, May 2021. doi: 10. 1609/aaai 10.1100/aaii.v35112...77.
URL
https://ojs aaai. org/ index. php/ happ/AAA//aaa httal//riill///eeooaoo wwaccceevviitie /it///iiwww//ieee/eee/ 17332
[118] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.0413 04131, 2020.
[119] Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14)i190-i198, 2017.
17


Reviews:
[{"review_id": "5KSmddazL0", "reviewer": "Reviewer_cB5G", "summary": "The paper describes an approach that combines Multimodal Machine Learning and Deep Graph Neural Networks to learn gene representations from multi-omics and multi-tissue data. The main issue that this paper tries to address is that existing approaches fail to obtain gene representations that are consistent across modalities (esp. from different data sources but same tissue). To accomplish this goal, the authors propose a model called MuSE-GNN that takes as input graph co-expression networks extracted from different modalities and projects them into a unified and consistent latent space to extract gene embeddings. These gene embeddings are then used for several downstream tasks such as pathway analysis, causal network analysis, and disease analysis. In particular, the model is learned through a loss function composed of three terms: a co-expression graph reconstruction term, a weighted similarity term (where the pairwise similarity of gene representations across modalities is maximized), and a self-supervised contrastive term (to cluster together functionally similar genes and pull apart functionally different genes). \n\nThe model is evaluated on a benchmark consisting of different tissues against 9 other methods, where it shows excellent performance on different metrics. In a subsequent experiment, the authors show that MuSE-GNN embeddings successfully relate to functional groups in human data. Lastly, they present a case study on COVID-19 where they run the model on pancreatic human cells of healty vs diseased COVID patients to identify differentially expressed genes, showing (with gene enrichment) that these genes are related to immune system activity and that some of them are part of known causal networks of the regulatory activity of the immune system.", "questions": "- from the code, I can see that the competitors are trained with random seeds in the range [0, 10) while I can't see the same for MuSE-GNN. Can you confirm that MuSE-GNN has been trained with the same random seeds and thus the results are comparable?\n- can you explain better this sentence in the appendix: \"Methods such as WSMAE, GAE, VGAE, and MAE, which are based on MLP/GNNs and inspired by other methods, are configured with the best hyperparameters based on parameter searching to ensure fairness.\"?\n- Your model uses 349M parameters. How many parameters GIANT uses? If they don't match, can you train a version of MuSE-GNN with a comparable number of parameters to GIANT (or alternatively, a version of GIANT with a number of parameters comparable to MuSE-GNN)?", "limitations": "Not discussed. I don't see many limitations of this approach besides getting data of proper quality (which is independent of the contribution). Perhaps, as mentioned by the authors, the model is quite \"heavy\" in terms of number of parameter and one needs to bring down the embedding dimension to avoid OOM errors.\n\nNo negative societal impact of concern.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow. It studies a relevant problem (how to obtain consistent gene embeddings across modalities) and proposes a quite elegant solution (make them consistent mainly through a tailored loss function). The experimental results are good and the ablations (in the appendix) are exhaustive and very informative.", "weaknesses": "It seems that the competitors have been evaluated with default hyper-parameters, while the hyper-parameters of MuSE-GNN have been tuned. This is unfair, since the results of MuSE-GNN are optimized to the experimental settings of this paper, while those of the competitors are not. This might be addressed by tuning (some of) the hyper-parameters of the competitors, although I understand it might be difficult given the time constraints of the review process. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive comments. The detailed response to each point is as follows.\n\n**1. It seems that only the hyper-parameters of MuSe-GNN were tuned, while other competitors were not.**\n\nWe appreciate your question. We have mentioned the details of parameter tuning for other methods in Appendix D.2 of the Supplementary Materials. We search for the best hyper-parameter combination for methods based on deep learning without pre-training.\n\nWhen it comes to methods not grounded in deep learning but specifically tailored for gene representation learning - PCA, Gene2vec, and GIANT - we did not adjust their parameters. However, in light of your feedback, we agree that it is important to consider the hyper-parameters for these methods. We have performed experiments by adjusting the dimensions of the resultant gene embeddings, thereby testing the performance across different models. The search range for these dimensions is set between 32 and 256. For this analysis, we utilized the datasets derived from the heart tissue. Due to the word limit, we recorded the raw score with standard deviation in the pdf file attached to the Author Rebuttal.\n\nTable 1: Min-max scaled benchmarking score table for different optimized models, where the star represents the optimized version.\n\n| Methods | ASW | AUC | iLISI | GC | CGR | NO | Average |\n|-----------------------------|-------------------------------------|--------------------------------------|---------------------------|----------------------------------------|---------------------------------------|--------------------------------------|-----------------------------|\n|PCA*      | 0.17                                | 0.37                                 | 0.76                      | 0.08                                   | 0.47                                  | 0.90                                 | 0.46                        |\n|Gene2vec* | 0.70                                | 0.91                                 | 0.00                      | 0.68                                   | 0.00                                  | 0.32                                 | 0.44                        |\n|GIANT*    | 1.00                                | 0.00                                 | 0.40                      | 0.00                                   | 0.16                                  | 0.00                                 | 0.26                        |\n|MuSe-GNN  | 0.00                                | 1.00                                 | 1.00                      | 1.00                                   | 1.00                                  | 1.00                                 | **0.83**                       |\n\nBased on the results presented in Table 1, we can conclude that even when compared with the other three competitors, which have been optimized for gene representation learning tasks, MuSe-GNN consistently outperformed the other methods.\n\n**2.   Can you confirm that MuSE-GNN has been trained with the same random seeds and thus the results are comparable?**\n\nWe appreciate your observation and question. We confirm that the same random seeds were consistently utilized during the training phase for all benchmarking methods and MuSe-GNN. We intend to provide more comprehensive code in the subsequent version of our work.\n\n**3. can you explain better this sentence in the appendix: \"Methods such as WSMAE, GAE, VGAE, and MAE, which are based on MLP/GNNs and inspired by other methods, are configured with the best hyperparameters based on parameter searching to ensure fairness.\"?**\n\nWe appreciate your question. We now elaborate our statement below:\n\nThe methods under discussion, namely WSMAE, GAE, VGAE, and MAE, are primarily predicated on Multi-Layer Perceptron (MLP) or Graph Neural Networks (GNNs) frameworks and draw inspiration from various other methodologies. To uphold fairness in our study, these methods were fine-tuned and configured with their most conducive hyper-parameters.\n\nThe hyper-parameter tuning process entails carrying out multiple experiments with various hyper-parameter configurations to identify the one that delivers the most superior model performance.\n\nHowever, given that we have now incorporated hyper-parameter tuning even for methods that do not rely on deep learning, we will strive to improve the clarity of this sentence in the forthcoming revision of our manuscript.\n\n**4. Your model uses 349M parameters. How many parameters GIANT uses? If they don't match, can you train a version of MuSE-GNN with a comparable number of parameters to GIANT?**\n\nWe appreciate your question. In our experiments, we found that GIANT consumes approximately 260 MB of memory. This result is relatively close to the 349 MB memory usage of our method, suggesting that the memory footprint of both approaches is comparably sized.\n\nUpon further examination of the GIANT model and GIANT paper, we have found that its architecture cannot be modified due to the absence of a user-accessible API for changing the number of layers. We do modify the number of latent dimensions and choose the largest model now. Moreover, we remain open to exploring further possibilities in future research.\n\n**5. Perhaps, as mentioned by the authors, the model is quite \"heavy\" in terms of number of parameter and one needs to bring down the embedding dimension to avoid OOM errors.**\n\nWe appreciate your question and concern. Our ablation tests have indeed shown that excluding raw gene expression from the input significantly impacts MuSe-GNN's performance. Moreover, we have optimized our model to require only a single GPU for training and inference as mentioned in Appendix D.4 of the Supplementary Materials. Nonetheless, we will keep on exploring the optimization of memory efficiency in the future.\n\nPlease let us know if you have further questions, and we will do our best to address them. Once again, we thank you for your thoughtful review.\n"}, {"review_id": "fQJaxEqYTM", "reviewer": "Reviewer_anSh", "summary": "This paper addresses the heterogeneity problem in the gene representation learning from multi-context biomedical sequencing profiles. Specifically, gene connections are formulated through co-expression network, then the proposed MuSe-GNN utilizes cross-graph Transformer to generate gene embeddings, while designing a contrastive-learning based regularization term to integrate multi-modal data.", "questions": "Minors:\nThe application area is rather specialized, so this paper may be of interest to a smaller subset of the NeurIPS community.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-motivated. Effective gene representation learning from multi-context profile datasets is critical real-world biomedical applications.\n2. The experimental results demonstrate that MuSe-GNN achieves the state-of-art performance and brought a significant improvement.\n3. Bioinformatical analysis on identifying significant pathways, diseases, and causal networks provides biologically-informed insights .", "weaknesses": "1. The novelty of the proposed method is limited. \n2. The methodology part is not adequately described. The cross-graph Transformer section is hard to follow and Figure 2 can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive comments. The detailed response to each point is as follows.\n\n\n**1.   The novelty of the proposed method is limited.**\n\nWe appreciate your inquiry concerning the novelty of our work, which we address in the following from two aspects.\n\nThe first aspect pertains to the novelty of our model objective. As outlined in the Introduction section, the conventional method of multi-omics data integration hinges on using cells as dimensionality reduction anchors. Such approach, however, faces significant computational hurdles including data-specific challenges, substantial data volumes, and batch effects.\n\nTherefore, we propose to shift our focus to gene perspective. The existing models fail in integrating gene information across different multimodal biological data. By designing MuSe-GNN, we aim to address a fundamental constraint of a representation learning problem in the computational biology area. We provide more details in the Introduction section.\n\nThe second aspect revolves around the novelty of our methodology framework. To our knowledge, our work is the first instance in gene representation learning where the Multimodal Machine Learning (MMML) paradigm is synergistically paired with deep Graph Neural Network (GNN) designs. This combination has enabled us to achieve state-of-the-art performance according to our experimental results.\n\nFurthermore, our core model comprises three unique components, each validated through an ablation study presented in Appendix D.1 of the Supplementary Materials. Finally, we consider constructing our graph data based on **multimodal data context** and **statistical test**, which is also novel and reliable.\n\n**2.   The methodology part is not adequately described. The cross-graph Transformer section is hard to follow and Figure 2 can be improved.**\n\nWe appreciate your comment on the lack of clarity of the description of our method. For the explanation of graph transformer design, we can add more details as follows:\n\nRegarding the description of our methodology, we have summarized all the essential components in the Methods section of our main text. Due to the page limit, we extended our method explanation in Appendix B of the Supplementary Materials.\n\nRegarding our cross-graph transformer design, we have addressed it in the main text and appendix. We offered comprehensive explanation in Section 3.3 of the Methods in our manuscript. Additionally, we have provided more details about our graph transformer architecture in Appendix B.4 of the Supplementary Materials.\n\nWe also improved Figure 2 with more annotations and attached it to the Author Rebuttal according to the rebuttal requirement. Please let us know if you have further questions or comments.\n\nThe updated figure description is: The overall model architecture and the design of loss functions for MuSe-GNN. The brown block represents the network architecture of MuSe-GNN. The blue block represents different loss function components of MuSe-GNN. The green block represents the input datasets. The color gradients of the left two matrices represent different gene expression levels. All the components of MuSe-GNN and gene embeddings are annotated.\n\n**3. This paper may be of interest to a smaller subset of the NeurIPS community.**\n\nWe appreciate your comment. As for the concerns regarding the specialized nature of our topic, we respectfully offer a different perspective. The intersection of AI and science [1], particularly within the realm of biology, has been an integral part of top-tier machine learning conferences in the past decades. The application of AI technology to address critical biological questions holds immense excitement and significance. This is evidenced by groundbreaking works such as Alphafold2 [2], which revolutionized protein design.\n\nWe note that there are many papers and activities about computational biology in NeurIPS, including:\n\n**Papers:**\n\nHetzel, Leon, et al. \"Predicting cellular responses to novel drug perturbations at a single-cell resolution.\" Advances in Neural Information Processing Systems 35 (2022): 26711-26722.\n\nXiao, Feiyi, et al. \"Estimating graphical models for count data with applications to single-cell gene network.\" Advances in Neural Information Processing Systems 35 (2022): 29038-29050.\n\nTu, Xinming, et al. \"Cross-linked unified embedding for cross-modality representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 15942-15955.\n\n**Workshops:**\n\nLMRL - Learning Meaningful Representations of Life (https://www.lmrl.org/).\n\nAI for Science (https://ai4sciencecommunity.github.io/neurips23/).\n\nMachine Learning in Structural Biology Workshop (https://neurips.cc/virtual/2023/workshop/66513).\n\n**Competitions:**\n\nOpen Problems in Single Cell Analysis - NeurIPS 2022: Multimodal Single-Cell Integration Across Time, Individuals, and Batches (https://openproblems.bio/events/2022-08_neurips/) (More than 1000 teams).\n\nOpen Problems in Single Cell Analysis - Coming August 2023 - Single-Cell Perturbation Prediction (https://openproblems.bio/events/2023-08_neurips/).\n\nWe think these references underscore the impact and relevance of our work within the realm of AI for science and biology. As commented by reviewer vvZM, our manuscript has excellent impact on at least one area, or high-to-excellent impact on multiple areas. Therefore, we think our research will capture the interest of scholars beyond these disciplines. There is also likely knowledge transfer, as a key property of machine learning research, among different areas.\n\nPlease let us know if you have further questions, and we will do our best to address them. Once again, we thank you for your thoughtful review.\n\nReferences:\n\n[1] Wang, Hanchen, et al. \"Scientific discovery in the age of artificial intelligence.\" Nature 620.7972 (2023): 47-60.\n\n[2] Jumper, John, et al. \"Highly accurate protein structure prediction with AlphaFold.\" Nature 596.7873 (2021): 583-589.\n"}, {"review_id": "jEQfbfzhZq", "reviewer": "Reviewer_mFBG", "summary": "The authors proposed a novel deep graph neural network, named MuSe-GNN, to learn gene representations from single-cell sequencing and spatial transcriptomic data. \nThe idea is to construct gene graphs from single-cell sequencing and spatial transcriptomic data, and then learn gene embeddings via cross-graph transformer.", "questions": "One additional ablation experiment would be of interest: how much would the performance be impacted if you induce the embedding using only part of the modules.\nThey can compare their method with other deep learning-based multimodal representation learning models, such as [1].\n[1] Lin X, Tian T, Wei Z, et al. Clustering of single-cell multi-omics data with a multimodal deep learning method[J]. Nature communications, 2022, 13(1): 7705.", "limitations": "N/A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The tasks of learning gene embeddings from multi-context sequencing profiles is of interest especially in the bioinformatic domain.\nThe empirical results seem to indicate that the approach can be applied to different analysis frameworks and the gene representations learned by their approach are highly versatile.\nWriting and presentation skill is well.", "weaknesses": "This idea is not new, and the models they used are all well established. \nLack of comparison with related works.\nLack of ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive comments. The detailed response to each point is as follows.\n\n**1.   This idea is not new, and the models they used are all well established. Lack of comparison with related works. Lack of ablation studies.**\n\nWe appreciate your comments concerning the novelty of our work, which we address in two distinct aspects.\n\nThe first aspect pertains to the novelty of our model objective. As outlined in the Introduction section, the conventional method of multi-omics data integration hinges on using cells as dimensionality reduction anchors. Such approach, however, faces significant computational hurdles including data-specific challenges, substantial data volumes, and batch effects.\n\nTherefore, we propose to shift our focus to gene perspective. The existing models fail in integrating gene information across different multimodal biological data. By designing MuSe-GNN, we aim to address a fundamental constraint of a representation learning problem in the computational biology area. We provide more details in the Introduction section.\n\nThe second aspect revolves around the novelty of our methodology framework. To our knowledge, our work is the first instance in gene representation learning where the Multimodal Machine Learning (MMML) paradigm is synergistically paired with deep Graph Neural Network (GNN) designs. This combination has enabled us to achieve state-of-the-art performance according to our experimental results.\n\nFurthermore, our core model comprises three unique components, each validated through an ablation study presented in Appendix D.1 of the Supplementary Materials. Finally, we consider constructing our graph data based on **multimodal data context** and **statistical test**, which is also novel and reliable.\n\nRegarding the question about the model established level, we think that the **Cross-graph Transformer** design remains an active area of exploration, especially in the computational biology track. Furthermore, we introduced the **Weighted-Similarity Learning** design, which represents a novel attempt to enhance the performance of Graph Neural Networks (GNN). Please find more details about these design elements in the Methods section of our manuscript.\n\nRegarding the question about model comparisons, our analysis involves a range of competitors that span gene representation learning methods (Gene2vec, GIANT), representative Graph Neural Networks (GNNs) (GAE, VGAE), and methods rooted in pre-training (scBERT) or other machine learning frameworks (PCA, WSMAE, MAE). We also intend to include Geneformer [1], a relatively recent model, in our comparisons. As indicated in Table 1 of our main text, our model outperformed all these competitor methods. We think our benchmarking is comprehensive, as noted by Reviewers cB5G and vvZM.\n\nRegarding the question about ablation studies, please see the information from Appendix D.1 of the Supplementary Materials, where we tried the ablation tests with different loss functions, different input settings, the contribution of cross-graph design, and the performance of different GNN structures. As noted by reviewer cB5G, our ablation tests are exhaustive and informative.   \n\n**2.   One additional ablation experiment would be of interest: how much would the performance be impacted if you induce the embedding using only part of the modules. They can compare their method with other deep learning-based multimodal representation learning models, such as [2].**\n\nWe appreciate your suggestions. Regarding the possible ablation test about dropping modalities (We assume 'modules' was meant to refer to 'modalities'; please let us know if this is not the case), we would like to emphasize the crucial role that data from diverse contexts play in our model's design. Comparisons between data with a greater number of modalities and those with fewer might be deemed unfair due to the variations in the sizes of the training datasets.\n\nAdditionally, theoretical research, as exemplified by [3], has demonstrated that the introduction of more modalities can enhance a model's performance in addressing related downstream tasks. We extrapolate this principle to the domain of gene representation learning. The central premise here is that the inclusion of data from a wider range of modalities can generate a mapping function that features a reduced upper bound of loss. Such information will be included in our updated version.\n\nRegarding the comparison with [2], we posit that such a comparison might not be entirely relevant due to two significant differences between the methods. Firstly, scMDC is designed for cell embeddings, while our model centers around gene embeddings. Secondly, scMDC necessitates 'paired' scRNA-ADT or scRNA-scATAC data, which are not only limited in availability but also differ considerably from the input our model requires. Instead of a direct comparison with scMDC, we have expanded on the comparison between MuSe-GNN and Geneformer in our rebuttal suggested by Reviewer vvZM.\n\nNevertheless, we will discuss this paper [2] as an example of learning cell embeddings in our Related work section.\n\nPlease let us know if you have any further questions, and we will try our best to address them. Once again, we thank you for your thoughtful reviews.\n\nReferences:\n\n[1] Theodoris, Christina V., et al. \"Transfer learning enables predictions in network biology.\" Nature (2023): 1-9.\n\n[2] Lin X, Tian T, Wei Z, et al. Clustering of single-cell multi-omics data with a multimodal deep learning method[J]. Nature communications, 2022, 13(1): 7705.\n\n[3] Huang, Yu, et al. \"What makes multi-modal learning better than single (provably).\" Advances in Neural Information Processing Systems 34 (2021): 10944-10956."}, {"review_id": "srxV430cao", "reviewer": "Reviewer_vvZM", "summary": "This paper introduces MuSe-GNN, a model for learning gene embeddings from single-cell sequencing and spatial transcriptomic data that is based on multimodal machine learning and deep graph neural networks.  While incorporating regularization with weighted similarity learning and contrastive learning to learn cross-data gene-gene relationships, the proposed method creates informative graph structures for model training and the generation of gene representations, which can make the learned gene representations contain functional similarity across various contexts in a joint space. The results of tests on diverse multimodal biological datasets show that MuSe-GNN can efficiently learn the functional similarity of genes across tissues and methods and surpasses current gene embedding learning models across several metrics. This model can aid in the investigation of the pathogenic processes underlying conditions like COVID and lung cancer.", "questions": "See above.", "limitations": "Yes, the authors have discussed their limitations and potential impact of gene embeddings for diseases, COVID.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed model learns the gene expression, which incorporates the regularization with weighted similarity learning and contrastive learning to learn cross-data gene-gene relationships. The proposed model can analyze the large-scale multimodal biological datasets. \n2. The model is applied to analyze COVID and cancer datasets, which helps to unveil potential disease resistance mechanisms or complications based on specific differentially co-expressed genes.\n3. MuSe-GNN has obtained significant improvement from 24.2% to 100.4% in comprehensive assessment.\n4. The overall pipeline and the mechanisms are presented in detail.", "weaknesses": "1. What is the difference between this paper and the Geneformer [1]. What about the comparisons of the performance of the experiments if they can be compared together?\n2. In Figure 1, why Gene2vec and GIANT failed to learn the gene similarity? The embeddings are clustered by datasets.\n\n[1] Theodoris, Christina V., Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo et al. \"Transfer learning enables predictions in network biology.\" Nature (2023): 1-9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive comments. The detailed response to each point is as follows.\n\n**1. What is the difference between this paper and the Geneformer [1]. What about the comparisons of the performance of the experiments if they can be compared together?**\n\nWe appreciate your questions.  The distinguishing features between the work presented in this paper, namely MuSe-GNN, and Geneformer can be summarized as follows:\n\n1. Geneformer is a Large Language Model (LLM) for single-cell data, which leverages a large-scale single-cell RNA-seq (scRNA-seq) dataset (a single modality). It employs a self-supervised pre-training approach, similar to BERT [2], to predict cell types or gene functions through fine-tuning on specific datasets.\n\n2. MuSe-GNN is specifically designed for gene representation learning, leveraging large-scale single-cell multi-omics data to create meaningful gene embeddings through an unsupervised learning framework.\n\nThe main function of Geneformer is to perform annotation and prediction, while it does not guarantee the generation of high-quality embeddings. The training framework of Geneformer is supervised and needs labels, in contrast to our unsupervised approach. Geneformer also has more limited application scenarios based on our following experiments. Moreover, we have already benchmarked a similar method, known as scBERT [3]. Therefore, MuSe-GNN has more functions than Geneformer, and can handle multimodal datasets. Moreover, MuSe-GNN is not as resource-intensive and time-consuming (the limitation of LLM) as Geneformer.\n\nRegarding possible shared experiments between Geneformer and MuSe-GNN, we considered both the gene function prediction task, a key feature of Geneformer, and the gene representation learning task, central to MuSe-GNN.\n\nFor the first task, we produced gene embeddings from the dataset utilized in [4] via MuSe-GNN and employed Support Vector Machine [5] as the classifier. This dataset was previously used by Geneformer for the gene function prediction task. The resulting accuracy scores are presented in Table 1.\n\nTable 1: Accuracy of gene function prediction task. Here \"unsup\" means unsupervised learning and \"sup\" means supervised learning.\n\n|          | MuSe-GNN (unsup) | Geneformer (sup) | Raw          |\n|----------|------------------|------------------|--------------|\n| Accuracy | 0.77 $\\pm$ 0.01     | 0.74 $\\pm$ 0.06     | 0.75 $\\pm$ 0.01 |\n\nWe present both the average accuracy and standard deviation for three different configurations: one based on gene embeddings derived from MuSe-GNN, another based on Geneformer, and the last one based on raw gene expression data (Raw). The classification process for raw data also utilizes the Support Vector Machine.\n\nOur results indicate that MuSe-GNN's performance is comparable to that of Geneformer, yet with less variability, suggesting that MuSe-GNN is more robust. Importantly, while Geneformer operates on the principles of supervised learning, MuSe-GNN employs an unsupervised approach. As a result, the gene embeddings produced by MuSe-GNN are capable of being utilized across a broader range of downstream tasks, as they encapsulate more meaningful biological contexts.\n\nFor the second task, we attempted to generate gene embeddings using Geneformer and record its performance. However, due to the absence of gene names in Geneformer's input and output, the benchmarking process could not be executed. Moreover, the absence of gene names greatly affects downstream applications; for example, analyses like pathway enrichment or disease analysis become infeasible. Hence, MuSe-GNN has more functions than Geneformer.\n\n**2. In Figure 1, why Gene2vec and GIANT failed to learn the gene similarity? The embeddings are clustered by datasets.**\n\nWe appreciate your insightful observation. We have the following explanation:\n\nThis observed phenomenon arises due to these methods failing to accurately capture similar associated genes across various multimodal data. More details can be found in the Introduction section of our manuscript.\n\nThe UMAP visualizations of GIANT's gene embeddings reveal that it only learned such information for parts of datasets. Therefore, GIANT does not effectively capture the common gene information that spans different datasets and contexts. The UMAP visualizations of Gene2vec reveal that it failed in learning such information across all the datasets.\n\nIn contrast, our proposed model, MuSe-GNN, addresses this issue through the integration of a cross-graph Transformer model along with various loss functions. This unique approach enables our model to efficiently capture and assimilate information from multiple datasets. This results in improved gene similarity learning and effectively overcomes the dataset bias that both Gene2vec and GIANT encounter. Our design for such a function was explained in the Methods section and Appendix D.1 of the Supplementary Materials.\n\nPlease let us know if you have further questions, and we will do our best to address them.  Once again, we thank you for your thoughtful review.\n\nReferences:\n\n[1] Theodoris, Christina V., Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo et al. \"Transfer learning enables predictions in network biology.\" Nature (2023): 1-9.\n\n[2] Devlin, Jacob, et al. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" arXiv preprint arXiv:1810.04805 (2018).\n\n[3] Yang, Fan, et al. \"scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data.\" Nature Machine Intelligence 4.10 (2022): 852-866.\n\n[4] Franzén, Oscar, Li-Ming Gan, and Johan LM Björkegren. \"PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data.\" Database 2019 (2019): baz046.\n\n[5] Pedregosa, Fabian, et al. \"Scikit-learn: Machine learning in Python.\" the Journal of machine Learning research 12 (2011): 2825-2830.\n"}]

Author rebuttal:
# General response

We would like to thank the reviewers for their overall positive comments on the aims of our manuscript, as well as their insightful comments and inquiries.

We appreciate all the reviewers for their comments about the strengths of MuSe-GNN, including the importantance of the topic (supported by all four reviewers), elegant solutions with informative ablation tests (supported by reviewers cB5G and vvZM), significant improvement over other methods (supported by reviewers anSh and vvZM), meaningful downstream applications (supported by reviewers anSh and vvZM), and a well-written manuscript (supported by reviewers cB5G, mFBG, and vvZM).

We have diligently addressed all the comments, which include clarifying the model's novelty (as raised by reviewers anSh and mFBG), improving the presentation of the model (as suggested by reviewer mFBG), refining our experimental design (as per the feedback from reviewers cB5G and mFBG), and expanding the discussion on related work (as noted by reviewers mFBG and vvZM).

Furthermore, we underscore the importance of our work within the realm of computational biology and provide supplementary materials to advocate for the wider application of AI in scientific research. Once again, we are immensely grateful for the invaluable comments and suggestions from all reviewers.

Meta-review: The paper proposes to integrate multimodal machine learning and deep graph neural networks for learning unified gene representations from multimodal biological graph data. The proposed model, referred to as Multimodal Similarity Learning Graph Neural Network, or MuSe-GNN, aims to learn gene representations containing functional similarity across different contexts in a joint latent space. Experiments are conducted on real-world biological datasets and results show significant improvements over SOTA models. 

Reviewers expressed some concerns in their original reviews, regarding (1) potential unfair comparison with baselines due to lack of fine-tuning of their hyperparameters, (2) limited novelty in the proposed method as it combines well-known techniques, (3) inadequate description of the methodology, as well as (4) missing comparison with highly relevant baselines. The authors’ rebuttal adequately addressed all these concerns and clarified some key issues. As a result,  multiple reviewers raised their scores, which reached a consensus in support of the acceptance of the paper.  The authors are encouraged to incorporate all these changes to the final version of the paper.
Decision: Accept (poster)