Biological Factor Regulatory Neural Network
Authors: Xinnan Dai
Caihua Shan
Jie Zheng
Xiaoxiao Li
Dongsheng Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: biological factor
gene regulatory network

Abstract: Genes are fundamental for analyzing biological systems and many recent works proposed to utilize gene expression for various biological tasks by deep learning models. Despite their promising performance, it is hard for deep neural networks to provide biological insights for humans due to their black-box nature. Recently, some works integrated biological knowledge with neural networks to improve the transparency and performance of their models. However, these methods can only incorporate partial biological knowledge, leading to suboptimal performance. In this paper, we propose the Biological Factor Regulatory Neural Network (BFReg-NN), a generic framework to model relations among biological factors in cell systems. BFReg-NN starts from gene expression data and is capable of merging most existing biological knowledge into the model, including the regulatory relations among genes or proteins (e.g., gene regulatory networks (GRN), protein-protein interaction networks (PPI)) and the hierarchical relations among genes, proteins and pathways (e.g., several genes/proteins are contained in a pathway). Moreover, BFReg-NN also has the ability to provide new biologically meaningful insights because of its white-box characteristics. Experimental results on different gene expression-based tasks verify the superiority of BFReg-NN compared with baselines. Our case studies also show that the key insights found by BFReg-NN are consistent with the biological literature.

Under review as a conference paper at ICLR 2023
BIOLOGICAL FACTOR REGULATORY NEURAL NET- WORK
Anonymous authors Paper under double-bling review
ABSTRACT
Genes are fundamental for analyzing biological systems and many recent works proposed to utilize gene expression for various biological tasks by deep learning models. Despite their promising performance, it is hard for deep neural networks to provide biological insights for humans due their black-box nature. Recently, some works integrated biological knowledge with neural networks to improve the transparency and performance of their models. However, these methods can only incorporate partial biological knowledge, leading to suboptimal performance. In this paper, we propose the Biological Factor Regulatory Neural Network (BFReg- NN), a generic framework to model relations among biological factors in cell sys- tems. BFReg-NN starts from gene expression data and capable of merging most existing biological knowledge into the model, including the regulatory relations among genes or proteins (e.g., gene regulatory networks (GRN), protein-protein interaction networks (PPI)) and the hierarchical relations among genes, proteins and pathways (c.g., several genes/proteins are contained in pathway). Moreover, BFReg-NN also has the ability to provide new biologically meaningful insights because of its white-box box characteristics. Experimental results on different gene expression- based tasks verify the superiority of BFReg-NN compared with base- lines. Our case studies also show that the key insights found by BFReg-NN are consistent with the biological literature.
INTRODUCTION
Understanding how cells work an essential problem in biology, and also very important in biomedical areas because of disease phenotype and precision medicine. From genome-scale view, the whole cell system is modeled by level, starting from DNA, mRNA, and protein to metabolomics, and finally, inferring the phenotype. We define these molecules and molecule sets as biological factors. At each level, the same type of biological factors interact or regulate each other, which determines cell fate, driving the cells to develop, differentiate, and do other activities (Angione) 2019). Thanks to single-cell sequencing technologies, we can obtain gene expression data from the mRNA level, which is fundamental analyzing the whole cell system. Currently, gene expression data is widely used to identify cell states during cell development, characterize specific tissues or organs, and analyze patient-speeific drug responses (Paik et 2020).
Many deep learning methods are proposed to utilize gene expression data for predictions, and achieve extraordinary performance different biological tasks. For instance, gene expression could be treated as a type of input feature to classify cell types, cluster cells and even calculate patient sur- vival time (Erfanian et al. 202T Huang al. 2020). Although most deep neural networks (DNNs) model could diagnose cancers with high precision, the original DNNs cannot tell detailed bio- logical factors/processes which cause cancers. For instance, the regulation between gene PFKI. and HIFIA under HEPG2 pathway has high probability of causing liver cancer (Shoemaker 2006 Garcia-Alonas et al. 2019).
Recently, some works leverage existing biological knowledge as graphs represent the relations of biological factors into the prediction models, and significantly improve the prediction accuracy of specific tasks. For example, Rhee et al. (2018) and Chereda et al. (2021) mapped gene expression data into the protein-protein protein interaction network, and used graph neural networks to predict cancer. Elmarakeby et al. (2021) modeled the relations of gene-pathway and pathway-biologiaa process as

---

Under review as a conference paper at ICLR 2023
network, and used deep neural network diagnose prostate cancer. Yu et al. 2018) used the Gene Ontology (GO) knowledgebase to build the neural network architecture, but they are too sketchy to simulate the gene protein reactions in the cells, and may lead to subop- timal performance. Although they mitigate the black-box issues, they only use partial biological knowledge, and they cannot explore new knowledge from gene expression data.
2016 Ma et al.
In this paper, we propose a generie framework, named biological factor regulatory neural network (BFReg-NN), whose goal is simulate the complex biological processes in a cell system, under- stand the functions of genes or proteins, and ultimately give insights into the mechanism of larger living systems. Particularly, BFReg-NN is a neural network with the following characteristics. First of all, each neuron is mapped into a biological factor (e.g., specific gene or protein), and arranged level by level based on the hierarchy of biological concepts, such as genes, proteins, pathways, bio- logical processes, and so on. Secondly, since biological factors regulate each other, edges between neurons (and hyperedges among neurons) are set to reflect the existence of these regulations. such manner, edges also model biological meanings. Moreover, two different operations are utilized to simulate the reactions inside/across the layer. Since genes regulate each other and create feedback loops to form cyclic chains of dependencies in gene regulatory network, graph neural network styled operations are suitable to model the "steady state" of genes. It is the same for proteins in PPI. In the layer of pathways, it is hypergraph where each hyperedge is pathway including multiple pro- teins. Accordingly, a hypergraph neural network is used to aggregate and balance the information of each protein. The operations across the layers imitate the material transformation (c.g., genes translate protein), so we adopt deep neural network styled operations to map the relations. Finally, BFReg-NN flexible explore new biological knowledge by adding important but not existing edges in the current biological neural network. We illustrate BFReg-NN simulates on genome-scale cell system as an example in Figure
The advantages of our proposed model include: (1) Compared with previous works, BFReg-NN merges with the structural biological knowledge in cell systems, including hierarchical relations (e.g., genes-proteins, proteins-pathways mappings), and regulatory relations among certain factors, such as GRN and PPI. Therefore, it could imitate how different biological factors work inside a cell. (2) The model of BFReg-NN is transparent and interpretable, as each neuron and edge has its corresponding biological meaning. Thus, the learned model weights give evidence of which biolog- ical parts are activated and which biological products generated, leading to the final prediction. (3) By adding new edges between neurons, BFReg-NN not only achieves better performance in downstream tasks, but also has the potential to complete undiscovered biological knowledge. Tra- ditional knowledge completion methods for biological domains (e.g.. link prediction by knowledge bases/graphs) suffer from imbalanced data problems (Bonner et 2022). BFReg-NN utilizes the gene expression data, which reflects the real cell states, and thus obtains more reliable results.
In the experiment, we show the effectiveness of BFReg-NN on several biological tasks which have different output formats, including missing gene expression value prediction, cell classification and future gene expression value forecasting. We also test the knowledge completing ability of BFReg- NN by the recall of the existing biological knowledge. Further, we do case studies for newly discov- ered knowledge. The results demonstrate that BFReg-NN provides biologically meaningful insights.
RELATED WORK
Gene expression and its applications: By RNA sequencing, it is easy to obtain gene expression which is a value to represent the amount of gene transcripts from DNA fragment (Eberwine al., 2014). It has been used in variety of biological applications, including single-cell analysis Yuu et al. 2022, Zhou et al.. 2002), disease diagnosis (Xing et al. (022) and drug discovery (Pham et al., 202
But most of these models lack transparency and ignore the existing biological knowledge.
The emergence of knowledge bases/graphs has led to enhancing the performance in many fields of computer science, such computer vision and natural language processing (Ren et al.) 2021 Hao et al. 2021 Liu et al. 2021). Similarly, knowledge graphs also have been widely used for specific biological tasks such cancer diagno- sis in recent years (Elmarakeby et al. 2021 Rhee et al., 2018)). OntoProtein (Zhang et al., (222)) embedded the gene ontology knowledge in pre-training to improve the performance of several pro-
Knowledge graph enhanced downstream fasks:
2

---

Under review as a conference paper at ICLR 2023
Trandderion
Integration
MiNA
Protein
Biclagial nourons (genne, proteiss
Sweae
peprossion
GRN
PPI
P Pallwaye
Dowunstrea task module
Interdawell in Incerattio M
PLL ESR
single ovrput
REFICE
PFRL
ier lovel resulations defined by of kowlllgge
AURKA
Hi
QLE
25
HIF1A
intra-level by hyporedaee detined by universsa biolog iia knowledge
COKNIA
CF
OND
TF
FSU
TP53
S3
42
discovered knowledge
Figure 1: The pipeline of BFReg-NN. We build the hierarchical biological network inspired by the cell system, modeling several levels to separate mRNA, protein, pathway and phenotype, and simulating biological factor interactions both at the intra-level and inter-level. We obtain the gene expression value from RNA sequencing input, predict the property of the phenotype output. Taking the cancer dataset NCI-60 as an example. We know that the regulatory activations start from gene HIFI A and TP53 to gene AURKA and CDKNIA at the gene level (Garcia-Alone Aleiso al. 2019). And at the protein level, the translated proteins Q16665 and P04637 stimulate P17858 and 014965 respectively (Türei et al., 2016). We mark those relations as existing biological knowledge by the black solid lines. Multiple genes with their products consist of pathways to drive cells to different types. For example, PFKL and HIFIA are activated in the HEPG2 cell line pathway, which leads to type of liver cancer (Shoemaker (0066). Besides, new regulatory relations would be identified at each level shown as dotted lines. After training, we learn the good representations of biological factors, and employ them for different downstream tasks with various output formats.
tein downstream tasks. But it limited the interpretation ability because of transforming knowledge into embeddings. Elmarakeby et al.] (2021) modeled the relations of cene-pathway and pathway- biological process neural network to diagnose prostate cancer. Chereda al. (202I) assigned genes to the PPI network to predict breast cancer, and then explained the important genes by Layer- wise Relevance Propagation. Although they mitigate the black-box issues, they all use a small part of biological knowledge, and they cannot explore new knowledge from gene expression data.
Knowledge complement: Our work is related to knowledge complement Yao et al. 2020 Goel et al. 2020). In particular, lots of work on biological knowledge complement has been done cently (Yu et al. 2021; Zitnik et al. 2018). Much attention is paid to predicting the relation between specific biological factors, which can be regarded as link prediction problem. For example. Mo- hamedd et al. (2020) predicted drug-target interaction (DTI) by learning representations of drugs and targets from the KEGG database. Hamilton et al. (2018) discovered new drugs for discases by embedding the drug-gene-diseams database in multiple relations. However, existing biological knowledge is collected in imbalance, which causes little biological meaningful predictions (Bonner et al., 2022). In our work, we expect discovered knowledge from gene expression data, which can interpret the cell state by biological factor interactions and avoid imbalanced situations.
3 BIOLOGICAL BACKGROUND
The biological system is modeled by complex network that connects many biologically relevant entities to work together to perform one or more particular functions. could be at the organ/tissue scale, such as the nervous system, the integumentary system. On the micro/nanoscopic scale, examples include cells, organelles, and so on. In this work, we focus on the simulation of the biological system in cell at genome-scale.
Thanks to the development of sequencing technologies, it is easy and cheap to obtain an amount of gene expression data to build genome-scale analysis. Starting from genes, model the cell genome-scale system by single-omic level (intra-level) regulations and different omic level (inter- level) mappings (Angione) 2019). Among different omic levels, defined as inter-level in BFReg-
3

---

Under review as a conference paper at ICLR 2023
NN, genes are first transcribed from DNA and then translate to proteins. Then, pathways integrate individual genes or protein products to perform certain cell functions. To model single-omic level regulations, we define the intra-level interactions. Genes regulated by each other, known as GRN, which means the gene expression values are governed to be inhabited or activated by association molecule products, such as RNA. Proteins interact with each other in similar way, inhabiting, activating, or combining with others to influence the expression values (or protein abundances) in cells, called PPI. Notice that in general, the relations among factors are universal for all the cells, thus the biological knowledge could adapt to different types of cells. But sometimes the biological factor interaction would rewire in specific cells, and thus the extra knowledge discovered and verified is also important (Lynch et al. 2011)
4 METHOD
Assume we have gene expression data R show the state of a cell, where n is the number of genes. Besides, we also define the neural network architecture of BFReg-NN according to structural biological knowledge with the intra-level regulations A and the inter-level mappings M. After training BFReg-NN, obtain the hidden embedding H. BFReg-NN(x, A, for each gene and do predictions for downstream tasks.
The output format of predictions could be single value y f(H), or multiple values x = f(H). Further, we also handle the time-series output. Given the gene expressions of cells at the to time, we aim to predict the gene expressions X E R = f(H) in the following t1...., time steps. Finally, BFReg-NN could explore new insights by adding more edges A'.
In the following sections, we first extract AA and M from the existing biological databases/graphs. Then, we propose versions of BFReg-NN. Finally, introduce how apply BFReg-NN to different downstream tasks.
4.1 BIOLOGICAL KNOWLEDGE DATABASES/GRAPHE
Based on the existing biological knowledge, we first divide biological factors into different levels, f = [Gene, Protein, Pathway, At each level, regulatory relations between factors in the intra- level are formulated as a matrix A {AGene A A Pathway .... A.), where AGene could be gene relations and defined by GRN, and Aprotein is decided by PPI. Pathway is a little special because is a hypergraph, where each edge could connect more than two nodes, called the hyperedge. The proteins in a hyperedge propagate the information to influence each other directly. The values in A, are binary to represent the existence of relations. We also identify the binary mapping matrixes M (M,, M2.... M from level to its upper level 1 + 1 as the inter-level interaction. The dimension of M, dependent on the factor numbers of the two levels. We set M, to map between genes and proteins, and M2 as direct mapping between protein-level and pathway-level. A and M both decide the architecture of the neural network, including neurons and links between neurons.
4.2 BASIC BFREG-NN MODEL
Given the gene expression of a cell R", we first utilize embedding layer to encode each gene independently, where HOD0 Emb(x:). The embedding layer let genes share a same parameter MLP layer to obtain the gene expression representation. As the gene expression in a float form with little measurement error, the embedding layer is utilized to reduce this error and thus enhance the performance of the model. Then we obtain 00.0 E Rnxd as the input to neurons at the first level, i.e., gene neurons.
We have A, as intra-level relations between neurons at level I. Inspired by graph neural networks, the message passing mechanism to update H..k, which is the hidden representation of the neuron in the k-hop at level l. The formulation is:
H = update
Σ jEA(())
message (H!* ,1,") ..L
(I)
The message function is to generate the message from neuron j to neuron i, where Ai(i) decides which neurons neighbors for neuron Then the update function is to update the embedding
4

---

Under review as a conference paper at ICLR 2023
of neuron using the obtained messages and the previously hidden embedding. A [0. K 1] is the number of hops to determine that neuron is influenced by other neurons in the Khhop neigh- borhood. In detail, attention weight for each message, and weighted sum up all the messages to update. As for op- erations in hypergraphs, we apply HGNN-styled (hypergraph neural network) (Bai al., 2021) message and update functions. Batch normalization is conducted at the end of each level.
utilize GAT-styled (graph attention) (Vaswani et al. 2017) to compute the
Due to multiple level interactions in biological systems, the embedding is also learned level by level. Since M, is inter-level relations between level and I + 1, we utilize the masked neural network to update the initial representation H at level +
H H activation (M O Wi) H + bi)
(2)
H'K is the output after batch normalization. The element-wise multiplication M O W ensures that non-existing relations are not used for updating. In fact, the important biological factors could be ranked by the weighted matrix W calculated between levels. W, and by are learnable parameters.
4.3 ENHANCED BFREG-NN MODEL
Here the enhanced version is introduced by adding new edges in A {AGene A A ret It can explore new biological insights and improve the performance simultaneously.
Existing biological knowledge is detected by biological technology reflect the implicit relations among factors. However, some knowledge is still hard to be discovered due to technological limi- tations and rewiring phenomenons in individual cells. Therefore, we spilt the interaction into two types. One is the universal regulation, supported by existing knowledge A. The other is local inter- action, inferring the new biological knowledge or rewiring individual cells, but now hidden in the non-existent edges of A.
Instead of a binary matrix A used the basic model, we use Ar to constrain the learnable matrix A': to discover new knowledge. As universal regulations are verified by biological methods, we use them the same as the basic model. For non-exister con-exxitent edges, we reweigh by a small value S a < due to it being less convincing. Thus, the edge intensity based on two types of knowledge is modified as:
why awly
universal regulation between and j and already exists A local interaction between and j but it is ignored in A:
A'
where w/,
H))). In the enhanced model, we not only learn the neuron em- beddings, but also utilize these embeddings with an MLP transformation to infer the intensity of the hidden interaction between neuron and j. Then we update the representations by
H I a (MLP A' (A)
After the model is trained to converge, we obtain the learned weights for non-existence edges, which provides insights for new knowledge and the rewiring phenomenon. We sort the weights w'y to iden- tify the candidates which deserve verification by biological experiments. Here we use simple edge weight modification method instead of gated edges implemented by the gumble-softmax function or advanced graph structure learning algorithms (Zheng et al.) 2020 Jin et al., (020)). The reason is that the biological knowledge is not sparse, and even dense in some core genes. Thus, it does not satisfy the sparse and low-rank requirements. Our simple method does not add much extra computation cost while achieving great effectiveness biological tasks.
4.4 DOWNSTREAM TASKS
After we obtain the final embedding L..K we could conduct different types of downstream tasks, whose output format could be onediimensional, maltiple-dimensional. time-series. Here we illus- trate them by three specific tasks, missing gene expression value prediction, future gene expression value forecasting, and cell classification.
Missing gene expression value prediction: Suppose we have a gene expression vector x E R" for n genes, measured by the single-cell sequencing technology. Some parts in the vector are zeros
5

---

Under review as a conference paper at ICLR 2023
or in low value because of dropout events (Gong et al., 2018 causing biases among cells. So we aim to accurately recover all the missing values and obtain a new vector X. Since the gene expression is dependent on certain cell, we merge gene representations. The new gene value vector is generated directly by x = MLP(HL. ) = ML. (B)( We minimize the mean squared loss between predicted values and ground truth for all the non-zero elements in x. Finally, the missing values are imputed with the predicted values.
This task is to classify the cell types using gene expression values x, which is important to determine the situation of tissues or patients. Because cell type is the observable result of a biological system, we simulate each gene to pass the multiple levels of transformation to infer the property of the cell. The prediction is computed by ÿ = MLP(H MLP(BFRcg-NNFF (x)x). is a multi-class classification task and we employ a cross-entropy loss.
Cell classification:
Future gene expression value forecasting: The single-cell data is efficient to analyze the cell response under different drugs. However, it is expensive to collect the data at different time steps and draw the development trend of cells. Thus, we model the cell response by future gene expression value prediction. In other words, given the gene expression data x E R n at lg. we aim to forecast the gene expression data & E RaxT in the following f ... time steps. We use two backbones, MLP and LSTM. MLP predicts the gene expression in the future time steps simultancously, and the equation is X MLP(BFReg (xx)) LSTM models the dynamic values step by step, where it takes the last time output as the next step input, and the equation is X = LSTM(BFReg-NNT) We employ MSE the loss function.
EXPERIMENTS
5.1
SETUP
Here we introduce the experimental setting for downstream tasks, including datasets, neural network architecture, and baselines. Training details are presented in Appendix. The code and toy datasets are attached in the supplementary material.
Missing gene expression value prediction: In this task, we collect 10 cell lines (BT20, HS578T, LNCAP, A549, MCF7, MCFIOA, MDAMB231, PC3, SKBR3 and A375) from the L1000 dataset. Because of the limitation of sequencing technology, the L1000 dataset does not have the golden standard values for non-landmark genes. Thus, we employ the pre-processed data based on Quu al. 2020 Besides, we also ignore the time point information in cell lines. each cell line, we have 1482 cell samples and 714 genes. We also collect 541 and 2305 existing knowledge edges for GRN and PPI respectively. We randomly mask the gene expression value with a 60% prob- ability, and predict these missing values. The cell samples are randomly split by 60/20/20% as training/validationneeet data. The evaluation metric is mean square error (MSE), which means the smaller the better.
Cell classification: We gather 5 datasets to predict cell types. The first one is GSE756888 (breast cancer) collected from Chung et (201)). The following four datasets about organs, including muscle, diaphragm, lung and trachea, obtained from Consortium et al.] (2018). We pre-process the data by deleting the cells and genes which have larger ratio of zeros. The remaining data are summarized in Table 6 We also split the data same as the missing value task, and evaluate the results by macro AUC.
Future gene expression value forecasting: For future value prediction, we choose the breast cancer dataset from 2019 Dream Challenge (Gabor al. 202T). There are 44 cell lines and treatments. Different treatments mean using various drugs on cells to test the drug effects. Since the data are quantified the proteomic level, we have 37 biomarkers as genes and 38 related PPIs as existing knowledge, which is a small protein-protein interactions graph where nodes are phosphorylated genes. The detailed process to obtain dynamic gene expression data is: all the cells are first treated and then divided into several groups. At regular intervals, a group of cells is killed and measured their gene expression. To avoid noise caused by this measurement on each individual cell, we calculate the median for each gene at every time step. Different from the previous two tasks, 5 treatments as training data, and the remaining one as test data. The 5-fold cross validation is done. We use MSE and Pearson correlation coefficient (PCC) to evaluate average performance.

---

Under review as a conference paper at ICLR 2023
Table Results on missing gene expression value prediction (evaluated by MSE L).
Knowledge
Models
BT20 HS578T LNCAP
A549
MCF7
MCFI0A
MDAMB231
PC3
SKBR3
A375
MLP
0.314 0.344
0.344
0.320
0.326
0.337
0.325
0.378 0.317
0.336
knowledge
transformer
0.396 0.377
0.406
0.383
0.396
0.399
0.418
0.402 0.387
0.407
Gated NN
0.709 0.681
0.644
0.533
0.647
0.700
0.667
0.722 0.594
0.729
GCN
0.715 0.714
0.722
0.653
0.688
0.675
0.707
0,731 0.677
0.702
co expression
GAT
0.719 0.715
0.723
0.655
0.693
0.669
0.720
0.721 0.676
0.716
MLP4N2V
0.351 0.378
0,380
0.349
0,368
0.350
0.358
0,392 0.343
0.354
GCN
0.331 0.329
0.330
0,304
0.340
0.313
0.319
0,338 0.331
0.317
existing
GAT
0.324 0.331
0.332
0,308
0.321
0.314
0.317
0.347 0.317
0.315
knowledge
P-NET
0.335
0.340
0.336
0.331
(Re--NN(Basiic
0.318
0.336
0.332
0.329
0.308
0.328
0.328
0,344
0.318
0.328
BFReg- NN(Enhanced)
0.318 0.317
0.320
0.297
0.308
0.306
0.320
0.335
0.307
0.308
Table 2: Results on cell classification (evaluated by macro AUC f).
Knowledge
Models
GSE
muscle
diaphragm
lung
trachea
prior
MLP
0,9122
0.8586
0.7881
0.8545
0,9387
transformer
0.9476
0.8785
0.8717
0.8900
0,9242
knowledge
Gated NN
0.7910
0.7784
0,7110
0.8240
0.9120
GCN
0.6916
0.6424
0.5945
0.5822
0.6612
co-expression
GAT
0.7790
0.6882
0.6305
0.6312
0.6294
NLP
0,9064
0.8772
0.7501
0.8180
0,9404
GCN
0,9285
0.8449
0.7896
0.8581
0,9421
existing
GAT
0.9255
0.8470
0.8039
0.8246
0,9409
knowledge
P-NET
0.9052
0.8654
0.7973
0.8425
0.9332
BeReg-Ne NN,(aasic)
0,9476
0.8808
0,8420
0.8808
0,9376
BFReg-NN (Enhaned())a
0.9693
0,8884
0.8509
0.8903
0.9446
Neural network architecture: The architecture is totally defined by existing knowledge. From open-source databases, we select Dorothea (Garcia-Alonso et al.) 2019) as GRN construction, and Omnipath (Türei et al. 2016) as PPI. As the pathway is specific in different cells, we collect path- ways for each dataset by Enrich (Chen et al. 2013), which provides knowledge of the hyperedge connection among genes and proteins, including LINCS L1000 L Perturbations for missing value task and WikiPathways 2019 MOUSSEHUM for cell classification task. For future value forecasting, we utilize its corresponding relations in the dataset.
Baselines: We divide baselines into three types: (1) There is no prior knowledge integrated into the model, and the input is only gene expression data. MLP is the simplest model for static prediction. LSTM is the typical RNN-based model for dynamic value prediction. We also select more com- plex models, Transformer (Vaswani et al., 2017) and GatedNN as competitors. GatedNN is an improved version of deep neural networks, where edges between neurons are gated by the gumble- softmax function so the network could be sparse. (2) The second type of baselines is to first learn a gene co-expression matrix as knowledge and then do the prediction. The co-expression matrix identifies which genes have tendency to show coordinated expression pattern, so it can be built by the similarity between gene expressions. We follow the process described in MLA-GNI (Xing et al. 2022) to learn the knowledge and then apply two classical graph neural networks, GCN (Kipf & Welling 2017) and GAT (Veličković et al. 2018). to aggregate the information and do the pre- diction. (3) We also compare the models using the existing biological knowledge. This knowledge could be graphs, such as GRN or PPI. MLP+N2V used node2vec (Grover & Leskoved 2016) to learn node embeddings of the prior graph, combine them with gene expression data and employ MLP to predict. GCN and GAT directly utilized the prior graph. As there are different prior graphs (GRN or PPI), we report the best results for GCN and GAT with the most suitable graph. P-NET (Elmarakeby et al., (021) utilized protein-patwway relations as prior knowledge and apply DNNs to predict. For our proposed model BFreg-NN, we test basic and enhanced versions.
5.2 EXPERIMENTAL RESULTS
Static task results:
We report the main results for missing value prediction and cell classification, Firstly, in general, compared to no prior knowledge
summarized in Table I and T methods, biological knowledge improves the performance of models effectively, which means that modeling specific meanings for neural networks could improve the performance in biological tasks. Although Gated NN includes the gumble-softmax function which also is able to predict edges in
7

---

Under review as a conference paper at ICLR 2023
Table 3: Results on future gene expression value forecasting (evaluated by MSE, and PCCT).
Knowledge
Models
MSE
PCC
0.0761 0.0038
0.9704 0.0018
no prior
LSTM
0.0822 0.0028
0.8888 0.0034
knowledge
transformer
0,0743 + 0.0013
0.9717 - 0.0007
Gated NN
0.0743 + 0.0026
0,9717 + 0.0007
GCN
0.3838 0.0090
0.8293 = 0.0036
co-expression
GAT
0.3447 0.0236
0.8455 0.0114
MLP+N2V
0,0760 0.0043
0,9706 0:0018
GCN
0.0920 0.0053
0.9646 = 0.0020
existing
GAT
0.0869 + 0.0044
0.9661 + 0.0017
BFReg-NN (MLP, Basic)
0.0732 0.0026
0.9719 0.0009
knowledge
BiReg-NN (MLP. Enhanced)
0.0724 0.0023
0.9724 0.0007
BFReg-NN (LSTM, Basic)
0.0825 0.0029
0.8906 0.0041
BFReg-NN (LSTM, Enhanced)
0.0819 0.0027
0.8907 = 0.0049
Table 4: Ablation study results on cell classification (evaluated by macro AUC f).
Models
Knowledge
muscle
diaphragm
lung
trachea
GRN
0.8165
0.7564
0.8230
0,9376
Basic
GRN&PPT
0.8798
0.8219
0.8784
0.9241
GRtthaal&Patwaay
0,8808
0.8420
0.8808
0,9321
GRN
0.8165
0.7873
0.8373
0.9446
GRN&PPI
0.8807
0.8417
0.8892
0,9297
Enhanced
0.8884
0.8509
0.8903
0.9360
No Intra-level
0.8799
0.8382
0.8809
0,9235
No Inter-level
0,8790
0.8058
0.8268
0.9391
probability, it is hard generate the existing biological knowledge. Besides, co-expression knowl- edge does not show outstanding performance because it is extracted from a small size of data and thus brings the noise. Some complex model also achieves promising results, such as Transformer, which indicates that building dense biological relations may be needed for the cell samples. Sec- ondly, BFReg-NN also outperforms the baselines using the existing knowledge. MLP+N2V fails in prediction because it uses the knowledge by embedding nodes implicitly, rather than in explicit architecture. Compared with GCN, GAT and P-NET, the basic BFReg-Ni could merge hicrarchical knowledge from different levels of the biological system. Further, the enhanced version learns im- portant undiscovered knowledge from the gene expression data. Overall, the enhanced BFReg-NN achieves the best results in most cases.
Dynamic task results: Results for future value prediction are presented in Table Compared with directly employing MLP/LSTM forecast future values, BFReg-NN with MLP/LSTM could im- prove the quality of embeddings and achieve better performance. GCN and GAT with co-expression matrix perform badly due to the small dataset. When using existing knowledge, MLP+N2V, GCN and GAT still cannot obtain good result because the knowledge is incomplete. For example, there are several isolated nodes in the dataset, and these methods fail in updating the embeddings of these nodes. Since transformer and Gated NN could merge the gene information densely, they obtain bet- ter results. Finally, the enhanced BeReg-NN could add the extra discovered knowledge to mitigate the problem of incomplete prior knowledge, and learn the embeddings with suitable architecture. Therefore, it can reach the best performance.
5.3 ABLATION STUDY
In this section, we provide a brief description of the effectiveness of each part in BFReg-NN, The ablation analysis for the cell classification task shown in Table 4 The accuracy is gradually improved when adding a higher level of knowledge to the model in most cases. In trachea dataset, GRN shows the best performance which indicates PPI may lead to an overfitting problem in this organ. In addition, we present the results of no intra-level and no inter-level situations. No intra- level version removes PPI and GRN knowledge and maintains the hierarchical network; and no inter-level merges the PPI and GRN into large graph and deletes the hierarchical structure. The results show that hicrarchical structure is more important than simply linking the factors together, which obeys the phenomenon that the biological factors in cell are produced step by step.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study results on missing value prediction (evaluated by MSE ().
Knowledge
BT20 I1S578T LNCAP A549 MCF7 MCFI0A MDAMB231
PC3 SKBR3
A375
GRN
0.318 0.317
0.320
0.297
0.308
0.306
0.320
0.335 0.307
0.308
GRN&PPI
0.343 0.342
0.338
0.326
0.339
0,333
0,343
0.357
0.338
0,344
GRN&Pathway
0.335 0.335
0.326
0.314
0.324
0.319
0,331
0.332
0.327
0.325
0.9
Figure Recall score for all the modes breast cunsec. The everage score 0.4175.
Figure Proquency of newly discovered intrrctionn inclading p53.
In addition, we could choose different levels from the biological system as layers for BFReg-NN to adapt different tasks. For example, the cell classification task aims capture the cell state involving variety of factors in a cell, and thus it needs multiple levels of biological knowledge to build the cell phenotype. In contrast, shown in Table [5] the missing value task focuses on gene expression value, which means single gene level knowledge is enough while protein or pathway level knowledge may bring the noise to predictions.
5.4 KNOWLEDGE COMPLETION BY BFREG-NN
To demonstrate the ability of BFReg-NN NN to complete the knowledge, we test it on the future value forecasting task with breast cancer to evaluate the knowledge recall score. For each gene in the dataset, we remove its related edges to make it an isolated node, and then conduct the model to recover the knowledge. We run BFReg-NN several times to avoid randomness. In each run, the discovered edges come out with weights, and are ranked to select the top-k list. The frequency of an edge is computed by (the times that the edge is in the top-k list)/(the total runs). We select the top-20 interaction pairs with the highest frequency as discovered knowledge. The results are shown in Figure 2 where BFReg-NN achieves an average 0.4172 recall score for all the nodes. We also take p.p53 gene as example. The detailed recalled edges are shown in Figure [..]] We found all the existing edges for p.p53 the top-20 list. Overall, we notice that most of the edges recalled by our model, while BFReg-NN also adds knowledge that does not appear in the existing database.
In recent breast cancer research, some new knowledge is discovered by biological methods, which is also found in our proposed BFReg-NN. We take the gene p53, well-known tumor suppressor, as an example to verify the biological meaning of discovered knowledge. We show the top-10 frequent interaction pairs including p53 in Figure 3 And then we find the cues in recent biological literature. Iododeoxyuridine (Idu) is the most frequent marker, which identifies cell phases, indicating whether the cell division is continued in breast cancer (Behbehani et al. 2012 Gabor et al., 2021). S6 takes the second place, which is regulated in mTOO signaling by p53, and the level S6 increases when p53 is insufficiency (Luo et al. 2021). Besides, p53 also regulates the expression level of Her2, where the interaction frequency is about 0.15 in our prediction. As a kind of Her2-positive cancer, breast cancer could be treated by inhibiting p53 (Fedorova et al. 2020). After that, the regulatory relation between p53 and RB is in the existing biological knowledge dataset.
CONCLUSION
In this paper, we propose generic framework BFReg-NN, which offers biological meanings to neurons and links between neurons, and imitates the whole cell system as a neural network at both intra-level and inter-level. We apply BFReg-NN to different downstream tasks and our experimental results demonstrate that BFReg-Ni NNN consistently outperforms baselines and discovers new biological meaningful insights. BFReg-NN provides novel paradigm to merge cell sequencing data and biological knowledge, and it could be extended to more types of genomics data to simulate and understand complex biological systems in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Claudio Angione. Human systems biology and metabolic modelling: review-froe disease metabolism to precision medicine. BioMed research international, 2019, 2019.
Song Bai, Feihu Zhang. and Philip HS Torr. Hypergraph convolution and hypergraph attention.
Pattern Recognition, 1 10: 107637, 2021.
Gregory K Behbehani, Sean C Bendall, Matthew R Clutter, Wendy J Fantl, and Garry P Nolan. Single-cell mass eytometry adapted to measurements of the cell cycle. Cytometry Part A, 81(7); 552-566, 2012.
Stephen Bonner, Ufuk Kirik, Ola Engkvist, Jian Tang, and Ian P Barrett. Implications of topological imbalance for representation learning on biomedical knowledge graphs. Briefings in Bioinformat- ics, 2022.
Edward Y Chen, Christopher M. Tan, Yan Kou, Qiaonan Duan, Zichen Wang, Gabriela Vaz Meirelles, Neil R Clark, and Avi Ma Maya.... Enrichr: interactive and collaborative html5 gene list enrichment analysis tool. BMC bioinformatics, 14(1):1-14, 2013.
Hryhorii Chereda, Annalen Bleckmann, Kerstin Menck. Júlia Perera-Bel, Philip Stegmaier, Florian Auer, Frank Kramer, Andreas Leha, and Tim Beißbarth. Explaining decisions of graph con- volutional neural networks: patient-specific molecular subnetworks responsible for metastasis prediction in breast cancer. Genome medicine, 13(1):1-16, 2021.
Woosung Chung, Hye Hyeon Fum, Hae-Ock Lee, Kyung-Min Lee, Han-Byoel Lee, Kyu-Tae Kim, Han Suk Ryu, Sangmin Kim, Jeong Eon Lec, Yeon Hee Park, et al. Single-cell rna-seq enables comprehensive tumour and immune cell profiling in primary breast cancer. Nature communica- tions, 8(1):1-12, 2017.
Tabula Muris Consortium et al. Single-cell transcriptomics of 20 mouse organs creates a tabula muris. Nature, 562(7727)3367-377 2018.
James Eberwine, Jai-Yoon Sul, Tamas Bartfai, and Junhyong Kim. The promise of single-cell sequencing. Nature methods, 11(1):25-27,2 2014.
Haitham A Elmarakeby, Justin Hwang, Rand Arafeh, Jett Crowdis, Sydney Gang, David Liu, Saud H AlDubayan, Keyan Salari, Steven Kregel, Camden Richter, al. Biologically informed deep neural network for prostate cancer discovery. Nature, 598(7880):348 3422322),44832,,,.,. 2021.
Nafiseh Erfanian, A Ali Heydari, Pablo lanez, Afshin Derakhshani, Mohammad Ghasemigol,
Mohsen Farahpour, Saeed Nasseri, Hossein Safarpour, and Amirhossein Sahebkar. Deep learning
applications in single-cell omics data analysis. bioRxiv, 2021.
Olga Fedorova, Alexandra Daks, Oleg Shuvalov, Alena Kizenko, Alexey Petukhov, Yulia Gnennaya, and Nikolai Barlev. Attenuation of p53 mutant as an approach for treatment her2-positive cancer. Cell death discovery, 6(1):1-8, 2020.
Attila Gabor, Marco Tognetti, Alice Driessen, Jovan Tanevski, Baosen Guo, Wencai Cao, He Shen, Thomas Yu, Verena Chung, Single Cell Signaling in Breast Cancer DREAM Consortium mem- bers, et al. Cell-to-cell and type-to-type heterogeneity of signaling networks: insights from the crowd. Molecular systems biology. 17(10)e 2021.
Luz Glrcia-Alonso, Christian H Holland, Mahmoud M Ibrahim, Denes Turei, and Julio Saez- Rodriguez. Benchmark and integration of resources for the estimation of human transcription factor activities. Genome research, 1963)113331137, 2019.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3988-3993 2020.
Wuming Gong, Il- Youp Kwak, Pruthvi Pota, Naoko pute: imputing dropout events in single cell ma sequencing data. BMC bioinformatics, 19(1): 1-10, 2018.
and Daniel J Garry. Drim-
10

---

Under review as a conference paper at ICLR 2023
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855-864, 2016.
Will Hamilton, Payal Bajaj, Marinka Zitiik, Dan Jurafsky, and Jure Leskovec. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31, 2018 88
Yu Hao, Xin Cao, Yufan Sheng, Yixiang Fang, and Wei Wang. Ks-gnn: Keywords search over in- complete graphs via graphs neural network. Advances in Neural Information Processing Systems, 34:1700-1712, 2021.
Zhi Huang, Travis Johnson, Zhi Han, Bryan Helm, Sha Cao, Chi Zhang, Paul Salama, Maher Rizkalla, Christina Y Yu, Jun Cheng, et al. Deep tearning-based cancer survival prognosis from ma-seq data: approaches and evaluations. BMC medical genomics, 13(5):1-12, 2020.
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 66-74, 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Xu Sun, et al. Auto-encoding knowledge graph for unsupervised medical report generation. Advances in Neural Information Processing Systems, 34: 16266-16279, 2021.
Yuan-Deng Luo, Lei Fang, Hong-Qiang Yu, Jie Zhang, Xiao-Tong Lin, Xiao- Yu Liu, Di Wu, Gui-Xi Li, Deng Huang, Yu-Jun Zhang, et al. p53 haploinsufficiency and increased mtor signalling define subset of aggressive hepatocellular carcinoma. Journal of hepatology. (4(1):96-11 108, 2021.
Vincent J Lynch, Robert Leclere, Gemma May, and Günter P Wagner. Transposon-meddin mediated rewiring of gene regulatory networks contributed to the evolution of pregnancy in mammals. Nature genetics, 43(11):1114-11)) 1159, 2011.
Jianzhu Ma, Michael Ku Yu, Samson Fong, Keiichiro Ono, Eric Sage, Barry Demchak, Roded Sharan, and Trey Ideker. Using deep learning to model the hierarchical structure and function of cell. Nature methods, 15(4):290-298, 2018.
Sameh K Mohamed, Vít Nováček, and Aayah Nounu. Discovering protein drug targets using knowl- edge graph embeddings. Bioinformatics, 36(2):603-610, 2020.
David T Paik, Sangkyun Cho, Lei Tian, Howard Y Chang. and Joseph C Wu. Single-cell ma sequencing in cardiovascular development, disease and medicine. Nature Reviews Cardiology, 17 (8):457-473, 2020.
Thai- Thai-Hoang Pham. Yue Qiu, Jucheng Zeng, Lei Xie, and Ping Zhang. deep learning framework for high-throughput mechanism-driven phenotype compound screening and its application to covid- 19 drug repurposing. Nature machine intelligence, 3(3):247-257, 2021.
Yue Qiu, Tianhuan Lu, Hansaim Lim, and Lei Xie. A bayesian approach to accurate and robust signature detection lines 11000 data. Bioinformaties, 36(9):2787-2795 2020.
Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michihiro Yasunaga, Haitian Sun, Dale Schuur- mans, Jure Leskovec, and Denny Zhou. Lego: Latent execution execution-guided reasoning for multi-hop question answering on knowledge graphs. In International Conference on Machine Learning, pp- 8959-8970. PMLR. 2021.
Sungmin Rhee, Scokjun Seo, and Sun Kim. Hybrid approach of relation network and localized graph convolutional filtering for breast cancer subtype classification. The International Joint Conference on Artificial Intelligence, 2018
Robert H Shoemaker. The nci60 human tumour cell line anticancer drug screen. Nature Reviews Cancer, 6(10):81-8833 2006.
11

---

Under review as a conference paper at ICLR 2023
Dénes Türei, Tamás Koresmáros, and Julio Saez-Rodriguez Omnipath: guidelines and gateway for literature-curatee signaling pathway resources. Nature methods, 13(12):966 (11)))9666777..0. 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
accepted as poster.
Xiaohan Xing, Fan Yang, Hang Li, Jun Zhang, Yu Zhao, Mingxuan Gao, Junzhou Huang, and Jianhua Yao. Multi-level attention graph neural network based on co-expression gene modules for disease diagnosis and prognosis. Bioinformatics, 38(8):2178 (118))217-2186 2022.
Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. 2020.
Michael Ku Yu, Michael Kramer, Janusz Dutkowski, Rohith Srivas, Katherine Licon, Jason F Kreis- berg, Cherie T Ng. Nevan Krogan, Roded Sharan, and Trey Ideker. Translation of genotype to phenotype by hierarchy cell subsystems. Cell systems, 2(2):77-88, 2016.
Yue Yu, Kexin Huang, Chao Zhang, Lucas M Glass, Jimeng Sun, and Cao Xiao. Sumgnn: multi- typed drug interaction prediction via efficient knowledge graph summarization. Bioinformatics, 37(18):2988-9999 2021.
Zhuohan Yu, Yifu Lu, Yunhe Wang, Fan Tang, Ka-Chun Wong, and Xiangtao Li. Zinb-based graph embedding autoencoder for single-cell ma-seq interpretations. In Proceedings of the AAAI con- ference on artificial intelligence, 2022.
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. International Conference on Learning Representations, 2022.
Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning, pp. 11458-11468. PMLR, 2020.
Zihan Zhou, Zijia Du, and Somali Chaterji. Kratos: Context-aware cell type classification and interpretation using joint dimensionality reduction and clustering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2616-2625, 2022.
Marinka Zitik, Monica Agrawal, and Jure Leskovec, Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):1457-44 2018.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX EXPERIMENTAL SETUP
Table 6: statistics for cell classification datasets
Dataset
#Sample
#Gene GRN knowledge PPI knowledge Pathways #Label
GSE
4658
418
172
136
89
lung
1642
148
332
muscle
1051
401
317
401
119
trachea
3587
326
109
207
110
diaphragm
853
355
239
299
115
5
Table 7: Hyperparameter setting
Dimension of hidden embeddings
Task
Learning rate
Max epoch
BeReg-NN module
Prediction module
missing value prediction
le-3
200
1024
cell classification
5e-4
200
256
future value forecasting
le-4
2000
16
512
Table 8: The setting of a in different tasks
Missing value prediction
Knowledge
set
HS578T
LNCAP
A375
MCFI0A
MDAMB231
SKBR3
GRN
le-2
1o.4
le-5
le44
le-3
0
le-3
Se.4
le-3
5e-5
GRN&PPT
01
1e.4
5e.5
0
0
0
0
D
Cell chasification
Future value forecasting
Knowledge set
GSE
musele
diaphragm
lung
trachea
Backbone
a set
41
le:5
le-5
GRN&PPI
Sc-4
1o-5
lo-5
lo-4
1c-4
LSTM
5e-3
0g
1o.5
50-5
A.1 TRAINING DETAILS
Data statistics are summarized in Table for the cell classification task. For optimizers, we use Adam for all the models. The hyperparameters are summarized in Table We set a small dimension of hidden embeddings to avoid overfitting. For the prediction module, we set the number of hidden units with the best value obtained from the validation data. In addition, the update function could be implemented by the sum or concatenation operation. We set K for all the experiments.
The performance of BFReg-NN is influenced by the value of hyperparameter the acceptance of newly discovered knowledge, knowledge is suitable for the current dataset, shown in Table g For example, in the future value forecasting task, a is larger than in other tasks to get the best result because existing knowledge is incomplete in the dataset. Besides, in the missing value task, our model performs better when a = 0 in the protein level, which means BFReg-NN ignores the irrelevant level when discovering knowledge. We vary from (0, 0.00001, 0.00005, 0.0001, 0,0005, 0.001, 0.005) and select the best value when the loss is the smallest in the validation dataset.
While a controls dependent on whether the existing biological
A.2 MODEL COMPLEXITY
The complexity of the BFREg-NN model depends on the number of levels L and the number of biological factors at each level nı. Therefore, the model complexity O(Σ\_ n²²). In the real application, we choose the potentially important genes. If the computation resource is adequate, we could set as many genes as possible. this paper, we use a single 1000 GPU to run experiments.
B APPENDIX - OTHER BASELINES
We complement three baselines, Random Forest, XGboost and DCell, in this section.
13

---

Under review as a conference paper at ICLR 2023
B.1
CELL CLASSIFICATION TASK
Table 9: other baselines for the cell classification task.
Models
GSE
muscle
diaphragm
lung
trachea
Random Forest
0,9688
0.8771
0.7161
0.8834
0.9273
XGBoost
0.8867
0.8534
0.7549
0.8625
0,9193
DCell
0,9482
0.7158
0.6731
0.7545
0.9322
BFReg-NN(basic)
0.9476
0.8798
0.8420
0.8808
0,9376
(eReh-NN)(nnhanee)
0.9693
0.8884
0.8509
0.8903
0.9446
Although in single-cell tasks, Random Forest and XGBoost models could have comparable perfor- mance, they are still worse than BFReg-NN most of the datasets in Table
DCell (Ma et al. 2018 used the Gene Ontology (GO) knowledgehase to build the neural network architecture, which limits the performance. GO defines GO terms (c.g., molecular function, cellu- lar component, biological process) and builds the architecture based on term relations. Since each GO term includes several genes, the authors leverage the genes as the input layer and GO terms as the following layers in the neural network. We think the network is too sketchy to simulate the gene/protein reactions in the cell, which may lead to suboptimal performance. Instead, BFReg-NN uses more specific knowledge (gene regulatory network, protein-protein interactions and pathways) to mimic the cell system. Therefore, -FFRe--NN(bsiic much better than DCell on the mus- cle/diaphragm/lung dataset. Further, BFReg-NN has the potential ability to discover new biological relations from inputs (i.c., gene expression data) to enhance performance. As shown in Table 9 the enhanced BFReg-NN achieved the best results.
B.2 FUTURE GENE EXPRESSION PREDICTION
Table 10: other baselines for future gene expression task.
Model
MSE
PCC
RF
0.1115 + 0.0021
0.9685 + 0.0006
XGBoost
0.0837 + 0.0033
0.9753# 0.0008
BFReg-NN (MLPenhanced)
0.0724 0.0023
0.9724 + 0.0007
The best-performing models in the 2019 Dream Challenge are ensemble methods, which include several sub-models of Random Forest and XGBoost. Here verify the performance of single model shown in Table 10 In the future gene expression prediction task, Random Forest has pool performance both MSE and PCC metrics. XGboost has little higher value PCC value but also a higher MSE value, which means XGboost could predict the trend of the cells but fail on the specific values. However, they lack the interpretation of the gene regulatory level from the biological view, and they can not discover new biological knowledge.
C APPENDIX PRE-TRAINING
Table 11: performances pre-trained BFReg-NN.
Model
MSE
PCC
BFReg-NN
0.0724 + 0.0023
0.9724 0.0077 0.0007
Pre-trained BFReg-NN
0.0719+ 0.0042
0.9723+ 0.0018
BFReg-NN could benefit from the pre-training and fine-tuning framework. The detailed experimen- tal steps are: (I) Utilize the missing gene expression prediction task to pre-train a BFReg-NN model on the breast cancer dataset; (2) Freeze the parameters of the BFReg-NN model except for the last
14

---

Under review as a conference paper at ICLR 2023
MLP layers, and (3) Fine-tune the last MLP layers to do the future gene expression prediction. The results are shown in Table
With the pre-training on the missing value task, the performance increases significantly the MSE metric. Because biological knowledge is universal and similar in the cell across different tasks, is reasonable that pre-trained BFReg-NN improves the other tasks' performance with fine-tuning. However, the standard deviation slightly rises, which means a more stable method to fine-tune the model should be discussed in the future.
D APPENDIX ABLATION STUDIES
D.1 TOP-20 GENE PAIRS FOR P.P53 GENE
0.7
0/6
frequency
0.84
0.03
000
IdU
2
P.Ak.Smd73
K167
p.S6
RBB
P.MK4
P.A PAMPK
D.BTK
P44: P.SK336 3b
APR0KSK
P.CREB
CAATENIN
P.MAP2K
JNK
ESTATI
GAPDH
P.RKK
2.133
aFAAK
Cyclint
cleaved
PL.CE2
D-STATS
PPDPKI
P.MAPA P.MEK
R.SMAD2S
p.Sck
PS
P.RKEMIKK M&
factor names
Figure 4: Top-20 frequent interactions for gene p.p53
Figure shows the top-20 frequent interactions for gene p.p53 in breast cancer. The purple color marks the relations that have been identified the existing knowledge. All the related genes of p.p53 are discovered by BFReg-NN. The recall of p.p38 is 0.4933 averaged by the total runs.
D.2 TRAINING SAMPLE INFLUENCE
0.48
0.35
MSE score
0.62
MSE
0.00
0.34
0.5
0.54
0.4
H15578T
0.41
0.42
E2 osal)
Effling bwad
0.38
MSE
MSE
0.35
0.35
0.36
0.5
0.34
MCF7
Figure 5: Influence on missing value task when reducing cell samples
15

---

Under review as a conference paper at ICLR 2023
We test the model performance with the number of training samples. Take the missing value task as an example, shown in Figure 5 We randomly leave the cell samples in {0.8, 0.6, 0.4, 0.2] proportions of the dataset, then the models 5 times to compute the average MSE score. Most models perform badly when removing more samples. GCN is not stable with a steep curve. MLP performs worse than others because only relies on gene expression data while others are also supported by existing biological knowledge. Compared to GAT, BFReg-NN (basic) and BFReg-NN (enhanced) have better results. Also, BFReg-NN (enhanced) has the lowest MSE scores and less influenced by small cell samples. The reason is that BFReg-NN (enhanced) has the ability to build new knowledge as the additional features predict the gene expression.
D.3 STABILITY ANALYSIS
6:0
3 915
AUC 500res
AUC spores
1:06
6.2
8.22
6/7
8.13
31.2
MLP-NY
out
NN (saso Syieg NR (nhbnned))
MEP
MLPPNN
oAT
DES
tor Not dauis) W Beg (naunned))
lung
trachea
AUC scoress
00:22
AUC
0,50
0,72
0770
0722
MLDHiCY GAT
Weiteg Rit Desic) )
Wiffle cacherco)
MLLF
Wikkyy Nis (tesid) Wiffee codloredd
musele
diaphragm
Figure 6: Distribution of AUC score on the cell classification task
In this section, we report the distribution of AUC score on the cell classification task, shown in Figure We ignore the GatedNN and GCN/GAT with co-expression knowledge due their poor performance. The results of MIP are in large range or have outliers, indicating it is severely influenced by random seeds. GCN, transformer and BFReg-NN (enhanced) have fewer outliers over four datasets. GCN has low AUC score, while transformer and BFReg-NN (enhanced) perform well both stability and accuracy, which means learning complex relations is beneficial in the cell classification task. However, the average transformer in most cases, because it uses confident knowledge to define important regulations. GAT and BFReg-NN (basic) less stable than the above three, but BFReg-NN (basic) achieves better results than GAT. In conclusion, BFReg-NN (enhanced) consistently obtains stable and accurate predictions.
of BFReg-NN (enhanced) is still higher than the
Table 12: Stablity analysis of BFReg-NN.
Metrics
GSE
muscle
diaphragm
lung
trachea
AUC
0.9399
0.9046
0.8201
0.9065
0.9540
STD
0.0583
0.0096
0.0213
0.0193
0.0059
More specifically, randomly split the whole dataset into training/validationnesst sets and verify
BFReg-NN model stability in Table [2] We observe that the standard deviations are a small value, which means BFReg-NN is stable.
16


Reviews:
[{"review_id": "TpIKcrND5H", "reviewer": "Reviewer_J4we", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors proposed BFReg-NN as a general deep learning model and designed its architecture based on the regulatory relations and hierarchical relations among genes, proteins and pathways. Incorporating the biological knowledge into the network architecture design, authors tried to break the “black-box” nature of neural networks and learn new biologically insights from the data. BFReg-NN achieved superior performance in three gene expression-based tasks compared with the baseline. Authors also conducted ablation tests to highlight the contribution of each module in BFReg-NN.\u0000", "summary_of_the_review": "The manuscript is not ready yet to be presented in its current form. Extra work needs to be done to polish the manuscript.\u0000", "strength_and_weaknesses": "It’s an important question to understand the black box characteristics of neural networks given its superior performance. Authors addressed this question leveraging the existing knowledge to design biologically meaningful architectures for BFReg-NN. The manuscript is well written with a comprehensive review of the field. There are some experiment details that need to be clarified as follows.\n\nMajor Questions\n\n1.\t(Section 4.2) Could you elaborate the embedding layer that was utilized to embed each gene?\n\n2.\t(Section 5.2, Table 2-4) Could you repeat the experiments multiple times varying the training/validation/test split and report the standard deviations to justify the performance? This information is likely to be available because of the discussion in the section A.2 stability analysis.\n\n3.\t(Section 5.3) As Table 3 and Table 5 are both related to the task of cell classification. The performance of predicting muscle (Basic AUC = 0.8798) in Table 3 corresponds to the GRN&PPI according to Table 5. The performance of predicting diaphragm (Basic AUC = 0.8420) corresponds to GRN&PPI&Pathway based on Table 5. It seems the performances reported in Table 3 are corresponding to different combinations of knowledge. Could you elaborate on this? \n\n4.\t(Table 8) alpha was tuned via grid search based on the appendix. It’s not clear if it was tuned based on the performance on the validation set. Also did alpha vary in the ablation test?\n\n5.\t(Discussion) For each of three tasks presented in the paper, authors trained BFReg-NN separately. If BFReg-NN learns the underlying biological parameters through training, it may have a good performance in a related task even after being trained with respect to a different task. For example, did the parameters learned with respect to the task of missing gene expression value prediction also perform well in the experiment of future gene expression value forecasting with limited fine tuning? If there are cell lines shared across these two tasks, this may be a valid experiment to pursue.\n\nMinor Questions\n\n1.\tBFReg-NN is designed to mimic the hierarchical biological network in nature. Is there a way to incorporate the feedback loop across hierarchies to make RFReg-NN architecture more similar to the real biological regulatory hierarchy. This is definitely beyond the scope of this manuscript, but a natural extension to consider.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MAVlEh6nwE", "reviewer": "Reviewer_PD4L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a neural network model whose weights and connections between nodes are derived from prior biological knowledge. Every node in the neural network represents a biological entity such as a gene, a protein, a pathway, or a phenotype. Besides injecting prior knowledge into neural network architecture, the paper also presents a mechanism to learn weights for non-existence edges to complete the prior knowledge. The model's performance is evaluated by three tasks: imputing missing genes, classifying cell types, and forecasting gene expression over time.", "summary_of_the_review": "I wouldn’t recommend this paper for ICLR. It is a decent paper, but given the lack of technical depth and novelty, it seems other forums focusing on biology or bioinformatics would fit this paper better.", "strength_and_weaknesses": "Regarding novelty\nIngesting prior knowledge into neural network architecture is not a new idea. A few studies have studied the problem of Ingesting prior knowledge into neural network architecture for explainability, but those studies are not discussed in the paper.\nYu MK, Kramer M, Dutkowski J, Srivas R, Licon K, Kreisberg JF, Ng CT, Krogan N, Sharan R, Ideker T. Translation of genotype to phenotype by a hierarchy of cell subsystems. Cell systems. 2016 Feb 24;2(2):77-88.\nMa J, Yu MK, Fong S, Ono K, Sage E, Demchak B, Sharan R, Ideker T. Using deep learning to model the hierarchical structure and function of a cell. Nature methods. 2018 Apr;15(4):290-8.\n\nRegarding evaluation:\nIt is confusing since the L1000 dataset used in the missing gene expression prediction task is not collected by single-cell sequencing, while section 4.4 motivates the task by mentioning the problem of dropout events in single-cell sequencing. For L1000, predicting the expression of non-landmark genes might be a useful task for this model.  \nIt may be worth discussing the best-performing models from the 2019 Dream Challenge for performance comparison. \n\nThe technical depth of this paper seems to be limited, in my opinion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "5AA_7jCPoUa", "reviewer": "Reviewer_1ymJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a neural network method BFReg-NN for several tasks involving RNA-seq data. This method defines its architecture using biological databases that annotate gene regulatory networks, protein-protein interactions and pathways, represented as graphs/hypergraphs. The neural network uses graph attention to perform computations on these graphs. The authors apply BFReg-NN to several tasks: imputing held-out gene expression values, predicting cell type, and predicting future gene expression values. ", "summary_of_the_review": "The problem of using prior knowledge to design transparent NN architectures is important and the authors' approach seems reasonable. However, there is a great deal of literature on this topic already and I could not tell which parts of the proposed approach are novel. ", "strength_and_weaknesses": "Many approaches have been developed that aim to incorporate biological databases into neural network architecture. For example, see section \"Using prior knowledge for transparent models\" of the review below, which includes some missing citations. I couldn't tell which parts of the proposed strategy are novel. Also, how did the authors choose which existing methods to compare to? \nhttps://www.nature.com/articles/s41576-022-00532-2\n\nWhat are the state-of-the-art methods for each of the three target tasks? How does BFReg-NN's compare to these methods? It's okay if BFReg-NN has worse performance in the name of interpretability, but it is important at least to know what the loss is. \n\n5.4: It is impossible to judge whether 0.4172 is a good recall without a measure of precision. Is BFREG-NN predicting many non-edges. One way to evaluate this would be to rank potential edges by how highly they were predicted, then show how high the known edges are in this ranking. Even if there are some previously-unknown true edges, a good predictor would rank known edges highly. \n\nMinor notes: \n\nBoth parts of this statement are too strong: \"It is easy for DNNs to diagnose cancer for patients, but DNNs cannot tell us how biological processes cause cancers\"\n\nI don't understand this: \"Moreover, regulations and reactions happen at both intra-level and inter-level modeled in the cell system view. Graph neural network styled operations for intra-level (hyper)edges and deep neural network styled operations for inter-level mappings, are applied respectively.\" \n\nRef \"Yao et al., 2030\"\nIncorrectly cited: \"Consortium et al. (2018)\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0kk43y2xTaj", "reviewer": "Reviewer_GToB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework based on some existing works to incorporate gene regulatory networks (Garcia-Alonso et al., 2019), protein-protein interaction (Chereda et al., 2021) networks, protein-pathway networks (Elmarakeby et al., 2021), and relations among genes, proteins, and pathways for biological discoveries. The idea is very interesting, and the empirical results showed its superiority. ", "summary_of_the_review": "Overall, I like the paper's idea, and it is new to me to have so many biological layers of information in one model for downstream analysis. However, I have some concerns regarding the sparse architecture and the interpretation of this model.\n\n1:  PNET builds the sparse network by the sparse connections between genes and pathways from the Reactome database. How did this happen for your sparse connection between gene-mRNA, mRNA-Protein, and protein-Pathway? \n\n2:  Compare to PNET, they have the ranking of top features in different layers which is meaningful for biomarker discovery, can BFReg-NN do this?\n\n3:  Why does the paper put an embedding layer to encode each gene rather than set each node as one gene, mRNA, or protein? What are the intuitions and benefits of the embedding layer?\n\n4:  The enhanced BFREG-NN model is not clear to me. How did it transform $A_l$ to $A_l^`$? ", "strength_and_weaknesses": "Strengths: 1. This paper which takes different levels of biological prior information and formulates them in a hierarchical structure for potential discoveries is novel.\n2. The experiments are comprehensive.\n\nWeaknesses: 1. There is no theoretical guarantee that the discoveries resulting from the sparse network architecture are unique.\n                      2. The paper missed the description of how to quantify the top-k frequent interaction pairs, which is discussed in section 5.4. As the paper claims, one advantage of this framework is novel discovery. However,  very limited results and discussions are presented here.\n                      3. It lacks model complexity analysis and comparison. Given different levels of biological entities' intra and inter interactions, I am worried about the model's real applicability. \n\n                     \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject