Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors proposed BFReg-NN as a general deep learning model and designed its architecture based on the regulatory relations and hierarchical relations among genes, proteins and pathways. Incorporating the biological knowledge into the network architecture design, authors tried to break the “black-box” nature of neural networks and learn new biologically insights from the data. BFReg-NN achieved superior performance in three gene expression-based tasks compared with the baseline. Authors also conducted ablation tests to highlight the contribution of each module in BFReg-NN.
strength_and_weaknesses: It’s an important question to understand the black box characteristics of neural networks given its superior performance. Authors addressed this question leveraging the existing knowledge to design biologically meaningful architectures for BFReg-NN. The manuscript is well written with a comprehensive review of the field. There are some experiment details that need to be clarified as follows.

Major Questions

1.	(Section 4.2) Could you elaborate the embedding layer that was utilized to embed each gene?

2.	(Section 5.2, Table 2-4) Could you repeat the experiments multiple times varying the training/validation/test split and report the standard deviations to justify the performance? This information is likely to be available because of the discussion in the section A.2 stability analysis.

3.	(Section 5.3) As Table 3 and Table 5 are both related to the task of cell classification. The performance of predicting muscle (Basic AUC = 0.8798) in Table 3 corresponds to the GRN&PPI according to Table 5. The performance of predicting diaphragm (Basic AUC = 0.8420) corresponds to GRN&PPI&Pathway based on Table 5. It seems the performances reported in Table 3 are corresponding to different combinations of knowledge. Could you elaborate on this? 

4.	(Table 8) alpha was tuned via grid search based on the appendix. It’s not clear if it was tuned based on the performance on the validation set. Also did alpha vary in the ablation test?

5.	(Discussion) For each of three tasks presented in the paper, authors trained BFReg-NN separately. If BFReg-NN learns the underlying biological parameters through training, it may have a good performance in a related task even after being trained with respect to a different task. For example, did the parameters learned with respect to the task of missing gene expression value prediction also perform well in the experiment of future gene expression value forecasting with limited fine tuning? If there are cell lines shared across these two tasks, this may be a valid experiment to pursue.

Minor Questions

1.	BFReg-NN is designed to mimic the hierarchical biological network in nature. Is there a way to incorporate the feedback loop across hierarchies to make RFReg-NN architecture more similar to the real biological regulatory hierarchy. This is definitely beyond the scope of this manuscript, but a natural extension to consider.


clarity,_quality,_novelty_and_reproducibility: Authors need to clarify some details on the experiments (see comments above).
summary_of_the_review: The manuscript is not ready yet to be presented in its current form. Extra work needs to be done to polish the manuscript.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a neural network model whose weights and connections between nodes are derived from prior biological knowledge. Every node in the neural network represents a biological entity such as a gene, a protein, a pathway, or a phenotype. Besides injecting prior knowledge into neural network architecture, the paper also presents a mechanism to learn weights for non-existence edges to complete the prior knowledge. The model's performance is evaluated by three tasks: imputing missing genes, classifying cell types, and forecasting gene expression over time.
strength_and_weaknesses: Regarding novelty
Ingesting prior knowledge into neural network architecture is not a new idea. A few studies have studied the problem of Ingesting prior knowledge into neural network architecture for explainability, but those studies are not discussed in the paper.
Yu MK, Kramer M, Dutkowski J, Srivas R, Licon K, Kreisberg JF, Ng CT, Krogan N, Sharan R, Ideker T. Translation of genotype to phenotype by a hierarchy of cell subsystems. Cell systems. 2016 Feb 24;2(2):77-88.
Ma J, Yu MK, Fong S, Ono K, Sage E, Demchak B, Sharan R, Ideker T. Using deep learning to model the hierarchical structure and function of a cell. Nature methods. 2018 Apr;15(4):290-8.

Regarding evaluation:
It is confusing since the L1000 dataset used in the missing gene expression prediction task is not collected by single-cell sequencing, while section 4.4 motivates the task by mentioning the problem of dropout events in single-cell sequencing. For L1000, predicting the expression of non-landmark genes might be a useful task for this model.  
It may be worth discussing the best-performing models from the 2019 Dream Challenge for performance comparison. 

The technical depth of this paper seems to be limited, in my opinion.

clarity,_quality,_novelty_and_reproducibility: The presentation of the paper can be improved. 
* problem definition in sec 3.2 is unclear; It reads more like the high-level description of the model.
* It would be helpful to elaborate on how “pathways” is designed. Some specific questions I have when reading the paper:
1. What are the edges between pathways?
 2. how can Enrichr represent pathways? It is a collection of genesets. 
 3. What’s the reason for using LINCS genesets for predicting L1000 gene expression? The ligand perturbation genesets are not “pathways.” And would it introduce any leak by using such genesets?
 4. I am unsure what graphs are used when forecasting gene expression time series.

summary_of_the_review: I wouldn’t recommend this paper for ICLR. It is a decent paper, but given the lack of technical depth and novelty, it seems other forums focusing on biology or bioinformatics would fit this paper better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework based on some existing works to incorporate gene regulatory networks (Garcia-Alonso et al., 2019), protein-protein interaction (Chereda et al., 2021) networks, protein-pathway networks (Elmarakeby et al., 2021), and relations among genes, proteins, and pathways for biological discoveries. The idea is very interesting, and the empirical results showed its superiority. 
strength_and_weaknesses: Strengths: 1. This paper which takes different levels of biological prior information and formulates them in a hierarchical structure for potential discoveries is novel.
2. The experiments are comprehensive.

Weaknesses: 1. There is no theoretical guarantee that the discoveries resulting from the sparse network architecture are unique.
                      2. The paper missed the description of how to quantify the top-k frequent interaction pairs, which is discussed in section 5.4. As the paper claims, one advantage of this framework is novel discovery. However,  very limited results and discussions are presented here.
                      3. It lacks model complexity analysis and comparison. Given different levels of biological entities' intra and inter interactions, I am worried about the model's real applicability. 

                     

clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Overall, I like the paper's idea, and it is new to me to have so many biological layers of information in one model for downstream analysis. However, I have some concerns regarding the sparse architecture and the interpretation of this model.

1:  PNET builds the sparse network by the sparse connections between genes and pathways from the Reactome database. How did this happen for your sparse connection between gene-mRNA, mRNA-Protein, and protein-Pathway? 

2:  Compare to PNET, they have the ranking of top features in different layers which is meaningful for biomarker discovery, can BFReg-NN do this?

3:  Why does the paper put an embedding layer to encode each gene rather than set each node as one gene, mRNA, or protein? What are the intuitions and benefits of the embedding layer?

4:  The enhanced BFREG-NN model is not clear to me. How did it transform $A_l$ to $A_l^`$?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a neural network method BFReg-NN for several tasks involving RNA-seq data. This method defines its architecture using biological databases that annotate gene regulatory networks, protein-protein interactions and pathways, represented as graphs/hypergraphs. The neural network uses graph attention to perform computations on these graphs. The authors apply BFReg-NN to several tasks: imputing held-out gene expression values, predicting cell type, and predicting future gene expression values. 
strength_and_weaknesses: Many approaches have been developed that aim to incorporate biological databases into neural network architecture. For example, see section "Using prior knowledge for transparent models" of the review below, which includes some missing citations. I couldn't tell which parts of the proposed strategy are novel. Also, how did the authors choose which existing methods to compare to? 
https://www.nature.com/articles/s41576-022-00532-2

What are the state-of-the-art methods for each of the three target tasks? How does BFReg-NN's compare to these methods? It's okay if BFReg-NN has worse performance in the name of interpretability, but it is important at least to know what the loss is. 

5.4: It is impossible to judge whether 0.4172 is a good recall without a measure of precision. Is BFREG-NN predicting many non-edges. One way to evaluate this would be to rank potential edges by how highly they were predicted, then show how high the known edges are in this ranking. Even if there are some previously-unknown true edges, a good predictor would rank known edges highly. 

Minor notes: 

Both parts of this statement are too strong: "It is easy for DNNs to diagnose cancer for patients, but DNNs cannot tell us how biological processes cause cancers"

I don't understand this: "Moreover, regulations and reactions happen at both intra-level and inter-level modeled in the cell system view. Graph neural network styled operations for intra-level (hyper)edges and deep neural network styled operations for inter-level mappings, are applied respectively." 

Ref "Yao et al., 2030"
Incorrectly cited: "Consortium et al. (2018)".

clarity,_quality,_novelty_and_reproducibility: In general, I found the text very hard to understand, despite being familiar with this area. Many terms not explicitly defined, such as "biological factor", "level", "[neural network] hyperedge", "knowledge". The text includes a great deal of imprecise language such as "almost transparent", "as much as possible", "most typical"
summary_of_the_review: The problem of using prior knowledge to design transparent NN architectures is important and the authors' approach seems reasonable. However, there is a great deal of literature on this topic already and I could not tell which parts of the proposed approach are novel.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed a deep learning model for simulating biological processes. The main features of the proposed technique is the ability to integrate biological knowledge into the model, and the explainability of the model.

While the reviewers appreciate the contributions of the paper, they believe the paper is not yet ready for publication. The authors need to better position their work by discussing other techniques for integrating biological knowledge with deep learning and contrasting with them. They also have to improve their experimental evaluation and compare with state of the art techniques, including black-box ones.  Finally, the authors should improve the writing to make the paper clearer and more accessible to a ML audience with limited background in bioinformatics.  While the authors did try to address these issues during the discussion period, the changes are extensive enough to require a fresh review of the paper. 
justification_for_why_not_lower_score: Most reviewers believe the paper is not ready for acceptance at this time. The authors need to better position their work by discussing other techniques for integrating biological knowledge with deep learning and contrasting with them. They also have to improve their experimental evaluation and compare with state of the art techniques, including black-box ones.  Finally, the authors should improve the writing to make the paper clearer and more accessible to a ML audience with limited background in bioinformatics.
Decision: Reject