An Effective GCN-based Hierarchical Multi-label classification for Protein Function Prediction
Authors: Kyudam Choi
Yurim Lee
Cheongwon Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: Bioinformatics
Protein function prediction
Machine Learning

Abstract: We propose an effective method to improve Protein Function Prediction (PFP) utilizing hierarchical features of Gene Ontology (GO) terms. Our method consists of a language model for encoding the protein sequence and a Graph Convolutional Network (GCN) for representing Go terms. To reflect the hierarchical structure of GO to GCN, we employ node(GO term)-wise representations containing the whole hierarchical information. Our algorithm shows effectiveness in a large-scale graph by expanding the GO graph compared to previous models. Experimental results show that our method outperformed state-of-the-art PFP approaches.

Under review as a conference paper at ICLR 2022
AN EFFECTIVE GCN-BASED HIERARCHICAL MULTI- LABEL CLASSIFICATION FOR PROTEIN FUNCTION PRE- DICTION
Anonymous authors Paper under double-bling review
ABSTRACT
We propose effective method to improve Protein Function Prediction (PFP) utilizing hierarchical features of Gene Ontology (GO) terms. Our method consists of a language model for encoding the protein sequence and Graph Convolutional Network (GCN) for representing Go terms. To reflect the hierarchical structure of GO to GCN, we employ node(GO term)-wise representations containing the whole hierarchical information. Our algorithm shows effectiveness in large-scale graph by expanding the GO graph compared to previous models. Experimental results show that our method outperformed state-of-the-ar PFP approaches.
INTRODUCTION
Protein Function Prediction (PFP) is one of the key challenges in the post- post-genomic era (Zhou et al. 2019 Li et al. 2018). With large numbers of genomes heing sequenced every year, the number of novel proteins being discovered expanding as well (Spalević et al.] (020). On the other side, protein functions are reliably determined in wet-lab experiments which are cumbersome and high- cost. As a result, the number of novel protein sequences without function annotations is rapidly expanding. In fact, UniRef100 (Consortium) 2019) contains over 220M (million) protein sequences of which less than IM have function annotations proved by experiments (Littmann et al.) 2021). Fast and accurate PFP is especially important biomedical and pharmaceutical applications which are associated with specific protein functions.
Protein functions are defined by Gene Ontology (GO) composed of a directed acyelic graph (DAG) (Ashburner et al.) 2000). There are three GO domains: Molecular Function Ontology (MFO), Biological Process Ontology (BPO), and Cellular Component Ontology (CCO), where each node represents one function called GO term, and each edge represents hierarchical relation between two GO terms, such as 'is_a", pfrrt.o'', and etc. Since one protein is usually represented by multiple function annotations, PFP can be regarded as hierarchical multi-label classification (HMC).
There are two groups of existing approaches for PFP: local approaches and global approaches. Local approaches usually constructed classifier for each label (GO term) for few labels of the same hierarchy level (Lobley et al.) 2008 Minneci et al.] 2013] Cozzetto et al., 2016 Rifaioglu et al. 2019). On the other hand, global approaches constructed single classifier for multiple labels. The initial global approaches considered PFP to flat multi-label classification, ignoring the hierarchical structure of GO, and considering each label independently (Kulmanov & Hoehndorf] 2020). Recent global approaches have constructed a structure encoder to learn the correlation among labels (Zhou al. 2020 Cao Shen (021). However, these results showed that existing global approach-based models had limitations in representing correlations between GO terms by learning large-scale hierarchical graph of GO.
One of the structure encoders in global approach-based models used Graph Convolutional Network (GCN). GCN has been applied learning representation of node features. Nevertheless, in the case of applying GCN to large-scale hierarchical graph, it was difficult to obtain information among long-distn long-distance (Chen et al. 2019 Zeng et al.) 2021) and unable to obtain adequate structure information since adjacent nodes did not contain any hierarchical features (Hu et al. 2019). To overcome these shortcomings, we build node-wise wise representations containing the whole hierarchical information, which is involved relationship between long-distance nodes and structure information.

---

Under review as a conference paper at ICLR 2022
In this paper, we propose novel PFP model that combines pre-trained Language Model (LM) and GCN- model including new node-wise representations. A pre-trained LM as a sequence encoder (Littmann et al., 202I) extracts general ad helpful sequence features. GCN-baser model as a structure encoder blends hierarchical information into a graph representation to improve GCN performance in large-scale hierarchical graph of GO. To predict the probability of each GO term representing target protein functions, the prediction layer is constructed as a dot product of outputs of two encoders. The experimental results show that our method achieves performance improvement compared to the-state-off tfe art models, especially in the most difficult BPO.
2
RELATED WORK
2.1 PROTEIN SEQUENCE FEATURE EXTRACTION
Protein sequences contain multiple biophysical features related to function and structure. Initially, protein biophysical features such as motifs, sequence profiles, and secondary structures were calcu- lated from suite of programs and then utilized as protein sequence feature vectors by combining them (Lobley et al. 2008 Minneci et al.) 2013; Cozzetto et al. 2016: Rifaioglu et al. 2019). While these methods intuitively utilized the relationship between protein features and its biological func- tions, it required deep knowledge of proteomics and had high-cost.
Various deep learning architectures that extract high-level biophysical features of protein sequences have been proposed. Convolutional Neural Network (CNN) is one of the architectures as a sequence encoder learn sequence patterns or motifs that are related to functions (Xu al. 2020). Therefore. ID CNN was utilized as effective sequence encoder in previous researches (Kulmanov & Hoehndorf
2020
Zhou al. 2020 Kulmanov al. 2018).
With the advent of transformers (Vaswani et al., 2017). which is attention- model, in Natural Language Processing (NLP), various attention based LMs were applied to protein sequence embed- ding (Rao et al. 2019: Vig et al., 2020 Rives et al., [021] Heinzinger et al. 2019 Elnaggar et al.) 2020 As protein sequences can be considered as sentences, these learned the relationship between amino acids constituting the sequence and learned contextual information. Vec
Vec (Heinzinger et al.) 2019) and ProtBert (Elnaggar et al. 22220). which were learned protein sequences using ElMo (Pe- ters al. 2018) and BERT (Devlin et al. 2018), showed that these mostly extracted biophysical features of protein structures and functions, such as secondary structures, binding sites, and homol- ogy detections.
2.2 PROTEIN FUNCTION PREDICTION (PFP)
PFP methods can be categorized into two different approaches which are local and global. Local approaches commonly employed single or few multi-label classifiers to each or few GO terms.
These approaches included FFpred (Lobley et al. 2008 Minneci et al.) 2013 Cozzetto 2013eCozzetto et al.) 2016)
and DEEPred (Rifaioglu et al., 2019). FFpred predicted one GO term by multiple Support Vector Machines (SVMs) trained with radial basis function kernels to recognize protein sequence patterns associated with the GO term (Cozzetto et al.) 2016). DEEPred created a multi-label classification model using deep neural network for each GO hierarchical level (Rifaioglu et al. 2011). Each model could carry out five GO terms in most labels. Even though this procedure generated 1,101 different models concerning all GO domains, this still needed numerous models for PFP (Rifaioglu et al. 2019). Ultimately, local approaches required expensive costs by training numerous models.
On the other hand, global approaches respectively constructed one classifier model for MFO, BPO, and CCO. The initial global approaches considered PFP as flat multi-label classification. They fo- cused on extracting rulatid features from the protein sequence. One of the function-related features is motifs, called sequence patterns. DeepGoPlus (Kulmanov & Hoehndorf, (020) encoded sequence to extract motifs using ID CNN and then predicted the probability annotating each GO term using one fully connected layer. Compared to local approach-based previous methods, Deep- GoPlus achieved improved performance in PFP, despite its simple model architecture. This model resulted in poor performance when the number of GO terms increases. The recent global approaches expanded that built a structure encoder to improve performance. In DeepGOA (Zhou et al.) 2020), all GO terms were regarded correlated labels contrary to DeepGoPlus where they were regarded
2

---

Under review as a conference paper at ICLR 2022
MESGKORTLAGWTI
Model
Pictein festum wector
Protein testurn veder
GD Embedding
Node feaure N" RM
GCN Layers
Adjacmnc Lo
Probabliics e R
Gene Omtology
Figure I: The network architecture this work. we defined that L is the protein sequence length, is the hidden dimension size and N is the number of GO terms in each GO domain.
as independent labels. This enabled GCN to learn more effectively GO terms. They showed im- proved performance compared to DeepGoPlus, although the same protein sequence encoding was used. They extracted patterns using ID CNN, but could not extract various features related func- tions other than patterns. TALE (Cao & Shen 2021) implied transformer encoder (Vaswani et al.) 2017) for sequence encoder and embedded GO with hierarchical information each GO term. They learned various and specific features related to function the transformer encoder. However, this indirectly learned the correlation among GO terms since they did not learn GO itself and just use hierarchical information of each GO term.
One of the common problems of global approaches is that they partially used GO terms. In general, cut-off criterion in each GO domain approaches, which had the cut-off criterion, utilized about 13% of the total GO terms. Although TALE (Cao & Shen 2021), which is the latest model, had the cut-off criterion 1, this still utilized only about 60% GO terms for their model. In this paper, we propose a method that effectively learns the relation between GO terms in an extended data using over 85% GO terms. This model results in similar or higher performance to the latest models used fewer GO terms.
number of annotations such as 25, 50, and 150. Global
PROPOSED METHOD
In this section, we describe the details of our model. The overall architecture is shown in Figure I This model has two inputs: protein sequence and the hierarchical graph GO. A protein sequence is encoded by pre-trained LM and is reduced dimension to the feature vector size of GO respectively. A GO graph is represented to a large-scale adjacency matrix and node-wise feature matrix. These matrices are inputs to GCN for learning the hierarchical representation of GO terms. The prediction layer is built as a dot product of protein feature vector and a GO terms vector to predict the probability of annotated GO term of each protein sequence. We explain the technicality of these process in this subsection.
PROTEIN SEQUENCE ENCODING
We employ pre-trained LM as a sequence encoder. As we mentioned in related work, pre-trained LM, such as Veq (Heinzinger et al. 2019) and ProtBert (Elnaggar et al. 2020). already proved their performance to capture rudimentary features of proteins such as secondary structures, biolog- ical activities, and functions (Rives et al.) 2021 Vig et al.) (020). Especially, it was showed that Veq (Heinzinger et al. 2019) better than ProtBert [Elnagaar et al. 2020) to extract high-level features related functions for PFP (Littmann et al.) 2021). Seqvec (Heinzinger et al. 2019) is uti- lized as protein sequence encoder. This makes the various lengths of protein sequences to 1024 representation vectors with high-level biophysical features. Protein sequence representations converted to P e xx www-dimensional representation vectors by fully connected layer to combine GO term vectors.
3

---

Under review as a conference paper at ICLR 2022
3.2 HIERARCHICAL REPRESENTATION OF GO TERM
Initial node features H RNNN represented one-hot encoding matrix where the i th row GO term and its ancestors are 1. The GO term Embedding layer reduces dimension by converting sparse matrix to H NNx xde dense matrix for preventing overfitting and reducing training time in GCN. It indicate that node features contain its physical location and conceptual information in hierarchical graph. N number of GO terms(node) and do is scale of dimension, which the maximum depth in each domain for containing hierarchical information in the dense matrix.
The adjacency matrix A R contains the relationship between GO terms. When parent node is f and the children node is the adjacency matrix is combined prior probability P(U, with Information Content(IC) (Song al., 2013) that measures semantic similarity between and The existing adjacency matrix is usually built by one-hot encoding or the prior probability. The one-hot encoding can not involve any additional information other than connection information between GO terms. The prior probability can involve relational information. Nevertheless, apply the inveterate label imbalanced problem in the PFP dataset to the adjacency matrix due to being highly dependent on the training dataset. We solve this problem that adding IC, which is less affected by the training dataset (Zouu et al. 2020). The adjacency matrix defined as follows:
IC(s)
A P(U,(U)) + Eiechita IC(i)
Prior probability P(U)\U)) is calculated as follows:
P(U.IU.) PUU,NU, P(U.) N
P(U.) P(U.) N,
Where U, means a number of annotations in the training dataset, P(U means the conditional
probability that and are annotated in the same protein sequence. IC is calculated follows:
IC(K) -ogp(kk)
p(k)
freg(k) Tregtroat]
(3)
freq(k) = Uk + Evechita(yy freq(i)
Where p(k) is probability of each GO term in the GO dataset, freq(k) is frequency of t and child(k) is every children of k.
Initial node features HD updated H' Nxd with node features of adjacent nodes through Ith GCN layer (1 < 1 < M). GCN layer is represented as follows:
H (AH'W
3.3 PREDICTION LAYER AND LOSS FUNCTION
The prediction layer builds dot product of protein sequence feature vectors and GO feature vectors to finally predict the probability of each GO term for the protein sequence. We use the sigmoid function in the prediction layer. The prediction layer is defined as follows:
Y ( P)
(5)
We use binary-cross entropy the loss function since it is binary problem at each GO term.
4

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
4.1 DATASET
The CA et al. 2019) dataset, which was used in an international protein function pre- diction competition, was used for our experiment. According to CAFA3, experimental annotations have one of 8 experimental evidence codes: EXP, IDA, IPI, IMP, IGI, IEP, TAS, and IC. The train- ing dataset includes protein sequences with all known experimental annotations before September 2016. The test dataset includes protein sequences with their experimental annotations known be- tween September 2016 and November 2017. The GO dataset is version of May 31, 2016. This GO dataset is also used in CAFA3. GO dataset has 1 11,888 GO terms in molecular function(MFO), 30,546 GO terms in biological process(BPO) and 4241 GO terms in cellular componen((CCO).
We propagated annotations using the true path rule. For example, if protein sequence 'P" has annotation ''' and GO term 'B' an ancestor of ''' protein sequence "P" contains annotations both 'A' and 'B'. Among various type relations, such as 'is.a', , ppartoof.. and etc, we propagated with only an 'is type relation as ancestor GO term, as same as a CAFA3 assessment tool. We constructed hierarchical graph of GO as input excepting isolated GO term. Eventually, we utilized over 85% of all GO terms in MFO, over 90% of all GO terms BPO and CCO. As result, our model utilized about 15% larger GO graph than previous models.
Dataset
Statistics
MFO
ВPO
CCO
Seq in Training Set
35,086
50,813
49,328
CAFA3
Seq in Test Set
1,101
2,145
1,097
Number of GO terms
10,236
28,678
3,905
Table 1: Statisties of sequences and GO terms in CAFA3
4.2 PERFORMANCE ON THE CAFA3
Experimental settings We constructed each GO domain's 's model since each domain (MFO, BPO, CCO) had a different number of GO terms. Sequence and GO term's 's hidden dimensions d were each GO graph's maximum depth, which was 41 dimensions of MFO, 80 dimensions of BPO, and 54 dimensions of CCO. Although the maximum depth of BPO was 155 since there were only 42 GO terms with a depth of or more, we used maximum depth of BPO 80 for efficient training time.
Evaluation We evaluated our model using maximum protein-centric F-measure (F (Fmaa) and area under the precision-recall curve (AUPR). Fmax the official assessment score used in CAFA3. Fmar is defined as follows:
2pr(t) re(() pr(t) rc(()
Fmox = mar
(6)
Where pr( and re(() is average precision and recall in threshold t. pr(() and re(t) are calculated as follows:
pr(() = c(T)
Σ G(t) tredf(()) Li=1 trall
re(t) Σ
ye (ra())(
Where G(t) is the number of proteins with at least one GO term in our test dataset, n is the number of all proteins in the test dataset. fr; i-th GO term's 's true vector. [tr;]] l is that if the i-th GO term true, tr; is I; otherwise zero. (rt(t) i-th GO term's predicted vector at threshold t. Thresholds t [0, 1] increase with 0.01 steps.
5

---

Under review as a conference paper at ICLR 2022
On the other hand, AUPR is used to evaluate classification problems, especially highly imbalanced- label classification. We used AuPRC on each GO term to evaluate the performance.
Method
Fmar
AUPR
MFO
BPO
CCO
MFO
ВPО
CCO
Naive
0.331
0.253
0.541
0.312
0.173
0.483
DIAMONDScore
0,532
0.382
0,523
0.461
0.304
0,500
DeepGoCNN
0.411
0.388
0.582
0.402
0.213
0.523
TALE
0.548
0.398
0.654
0.471
0.317
0.626
Ours
0.518
0.470
0.637
0.476
0.368
0.626
MFO
BPO
OCO
Nahve
DIAMORDSKare DerpSuChN Ours
04
Scores
Scores
0.2
0.2
0.2
0.1
0.2
0. Finax
AURA
0.0
Fmax
AUFR
0.
Trma
AURR
Measures
Measures
Measures
Figure 2: The performance of our model against baseline models on the CAFA3
Baseline Models and Results We compared our model using baseline models. The baseline mod- els were Naive, DIAMONDScore, DeepGoCNN, and TALE. Naive predicted protein functions ing prior probability. DIAMONDScoree predicted protein functions based on the sequence similar- ity measured using the Diamond tool (Buchfink al., 2015). DeepGoCNN was the most famous sequence-based PFP model, using 1D CNN protein sequence encoder and a flat multi-label clas- sifier (Kulmanov & Hochndorí] (020). TALE was the state-of-the-art sequence-based PFP model, using transformer encoder for protein sequences encoding and embedding GO terms with hi- erarchical information (Cao & Shen 2021). Results were shown in Figure 2 Overall, our model achieved the best performance on all domains in AUPR and on biological process (BPO) in F Fmar. Our model especially improved from 0.398 to 0.470 on BPO (by 18%) compared to TALE. All the baseline models did not exceed 0.4 in Fmax on BPO, which had the most complex and large-scale hierarchical graph. It proved that our model was effective in a large-scale hierarchical graph, as showed highly improved performance on BPO.
CONCLUSION
In this paper, we proposed an effective GCN-based model to improve the protein function prediction. We build initial node-wise representations involving the whole hierarchical information to overcome the shortcoming of GCN-base model in a large-scale graph. After that, we combined a pre-trained LM and an effective GCN-based model to predict protein function.
Experimental results showed that in Fmaz, our model outperformed the related state-of-the-are methods BPO, which was the highly difficult domain with the most large-scale hierarchical graph. Moreover in AUPR, our model outperformed the related of-te-of-the-arr methods on all domains. We demonstrated that our model was especially effective in large-scale graph having deep depth.
Learning correlation of GO terms is an important part of PFP since it is the large-scale hierarchical multi-label classification. It is necessary extract the features of the protein sequence related to

---

Under review as a conference paper at ICLR 2022
the function from the PFP. We used pre-trained LM in the sequence encoder part. It properly extract the features of the protein sequence, but does not train the encoder while training the model. We will work later sequence encoders that extract function-related features using different L.M..
REPRODUCIBILITY STATEMENT
The learning was processed with epochs and 32 batch sizes. We used ReLU activation function, 1e-3 the learning rate, and Adam optimizer (Kingma & Ba 2014). We simulated our model with NVIDIA TITAN Xp. The code for reproducibility is posted at the anonymous GitHub repository :
4open. wttpeeeeeee///ppprrooo..abbloo.a code
REFERENCES
Michael Ashburner, Catherine A Ball, Judith Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification of biology. Nature genetics, 25(1):25-29, 2000.
Benjamin Buchfink, Chao Xie, and Daniel H Huson. Fast and sensitive protein alignment using diamond. Nature methods, 12(1):59-00)) 60, 2015.
Yue Cao and Yang Shen. Tale: Transformer-baseee based protein function annotation with joint sequence- label embedding. Bioinformatics, 2021.
Deli Chen, Xiaoqian Liu, Yankai Lin, Peng Li, Jie Zhou, Qi Su, and Xu Sun. Highwaygraph: Mod- elling long-distance node relations for improving general graph neural network. arXiv preprint arXiv:1911.03904, 2019.
UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47 (D1):D506-D515, 2019.
Domenico Cozzetto, Federico Minneci, Hannah Currant, and David T Jones. Ffpred 3: feature-based function prediction for all gene ontology domains. Scientific reports, 6(1):1-11, 2016.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: towards crack- ing the language of life's s code through supervueee deep learning and high performance com- puting. arXiv preprint arXiv:2007.0622 2020.
Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost. Modeling aspects of the language of life through transfer-learning protein sequences. BMC bioinformatics, 20(1):1-17) 2019.
Fenyu Hu, Yanqiao Zhu, Shu Wu, Liang Wang, and Tieniu Tan. Hierarchical graph convolutional networks for semi-supervised node classification. arXiv preprint 1902.066. 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Maxat Kulmanov and Robert Hoehndorf. Deepgoplus: improved protein function prediction from sequence. Bioinformatics, 36(2):422-47) 429, 2020.
Maxat Kulmanov, Mohammed Asif Khan, and Robert Hoehndorf. Deepgo: predicting protein func- tions from sequence and interactions using a deep ontology-aware classifier. Bioinformatics, 34 (4):660-668, 2018.
Zejun Li, Bo Liao, Yun Li, Wenhua Liu, Min Chen, and Lijun Cai. Gene function prediction based on combining gene ontology hierarchy with multi-in instiiinnttnce multi-labe learning. RSC advances, 8(50))28503-2809, 2018.
7

---

Under review as a conference paper at ICLR 2022
Maria Littmann, Michael Heinzinger, Christian Dallago, Tobias Olenyi, and Burkhard Rost. Em- beddings from deep learning transfer go annotations beyond homology. Scientific reports, 11(1): 1-14, 2021.
Anna Lobley, Timothy Nugent, Christine A Orengo, and David T Jones. Ffpred: an integrated feature-based function prediction server for vertebrate proteomes. Nucleic acids research, 36 (suppl2):W27-WW30) 2008
Federico Minneci, Damiano Piovesan, Domenico Cozzetto, and David T Jones. Ffpred 2.0: im- proved homology-independent prediction of gene ontology terms for eukaryotic protein se- quences. PLoS One, 8(5):e63754, 2013.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv: 182.05188,..55665, 2018.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Picter Abbeel, and Yum S Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32:9689, 2019.
Ahmet Sureyya Rifaioglu, Tunca Dogan, Maria Jesus Martin, Rengul Cetin-Atalay, and Volkan Ata- lay. Deepred: automated protein function prediction with multi-task feed-forward deep neural networks. Scientific reports, 9(1): 1-,1 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Xuebo Song, Lin Li, Pradip K Srimani, S Yu Philip, and James 7. Wang. Measure the semantic simi- larity go terms using aggregate information content. JEEE/ACM transactions on computational biology and bioinformatics, 11(3):468-476 2013.
Stefan Spalević, Petar Veličković, Jovana Kovačević, and Mladen Nikolić. Hierachial protein func- tion prediction with tails-gnns. arXiv preprint arXiv:2007. /2804, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models. arXiv preprint TrXii:2006.1522. 2020.
Yuting Xu, Deeptak Verma, Robert P Sheridan, Andy Liaw, Junshui Ma. Nicholas M Marshall, John McIntosh, Edward C Sherer, Vladimir Svetnik, and Jennifer M Johnston. Deep dive into machine learning models for protein engineering. Journal of chemical information and modeling. 60(6): 2773-2790, 2020.
Daojian Zeng, Chao Zhao, and Zhe Quan. Cid-gen: An effective graph convolutional networks for chemical-induced disease relation extraction. Frontiers in Genetics, 12:1 15, 2021.
Guangjie Zhou, Jun Wang, Xiangliang Zhang, Maozu Guo, and Guoxian Yu. Predicting functions of maize proteins using graph convolutional network. BMC bioinformatics, 21(16):1-16, 2020.
Naihui Zhou, Yuxiang Jiang, Timothy R Bergquist, Alexandra Lee, Balint Z Kacsoh, Alex W Crocker, Kimberley A Lewis, George Georghiou, Huy N Nguyen, Md Nafiz Hamid, et al. The cafa challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens. Genome biology, 20(1): 2019.
8


Reviews:
[{"review_id": "TDra8kjtY0r", "reviewer": "Reviewer_mvKR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Impression\nI believe the proposed method to be interesting and the contribution to the GO embedding technique \nto be valuable for the PFP communitt. However, I found the paper very difficult to read, and I have several\nconcerns about this study, which I detail below.\n\n# Major Concerns\n\n1. Evaluation is lacking representative state-of-the-art competitors.\n\nThe authors compare their method against 4 competitors: \n * Naive: a simple frequency based predictor.\n * DIAMONDScore: A sequence similarity based predictor.\n * DeepGOCNN: A Deep Learning method that uses a Convolutional Neural Network\n * TALE: A transformer-based PFP method.\n\nNotably, the authors decided not to compare with GOLabeler, the best performing method\nfrom the CAFA3 competition. I also find it very strange that they decided to go only with\nDeepGOCNN, which is consistently outperformed by the full architecture (DeepGOPlus) in the paper\nby Kulmanov and Hoehndorf (Bioinformatics, 2020). \n\nThe authors claim their method surpasses the state-of-the-art, but this is not properly shown\nin my opinion. How does the proposed method compare to GOLabeler and DeepGOPlus?\n\n\n2. The proposed method is not explained in the paper.\n\nThe method proposed by the authorsis not very well explained in the manuscript. \nI would not have understood the architecture without careful studying of the code.\n\nBelow are some questions that would have remained unanswered had I not been willing to\ncarefully inspect and study the code myself:\n * $H^0$ gets compressed from a dimensionality of $N \\times N$ to $N \\times d_0$, but this embedding layer is never described.\n * Node features get updated through layers $l$ using equation 4. What is $\\hat{A}$? What is $W^l$, perhaps the weights of this layer? If so, why is this not properly explained?\n * When explaining equation 2, what are $N_s$ and $N_t$?\n\n3. The code is lacking instructions or documentation.\n\nThe understanding of this manuscript requires the reader to be a seasoned Python programmer with knowledge of several very specialised frameworks.\nMoreover, the provided repository comes with no instruction or documentation on how to use the code to reproduce their results.\nHad I not been willing to spend several hours carefully studying this code, I would consider this paper as not reproducible.\n\n# Minor Comments\n\nSection 2 does not represent a good summary of the field in my opinion. Based on the brief \ndescription of the field in this section, the complexity and wide variety of existing protein\nfunction prediction methods is not evident. Moreover, the state-of-the-art is hardly reflected \nin this section, which fails to mention the top-performing methods in the CAFA competition.\n\nThe spelling of existing models contains several mistakes. Examples are \"DeepGoPlus\" (the correct spelling is DeepGOPlus)\nand DeepGoCNN (the correct spelling is DeepGOCNN).\n\nThere are several grammatical and words that seem to be expressing the wrong idea. For example, in Section 2.2, the authors state\n\"TALE (Cao & Shen, 2021) implied transformer encoder [...] for a sequence encoder and embedded GO with hierarchical information of each GO term.\"\nIt seems to me that \"implied\" should be replaced by \"applied\", to describe the work by Cao & Shen more accurately. Several other instances exist in\nthe text.", "summary_of_the_paper": "The authors introduce a GCN-based hierarchical model for protein function prediction.\nThe idea is built on top of using existing protein embedding techniques and using a GCN \nto embed GO terms and their hierarchical information, encoded as node features.", "summary_of_the_review": "This article proposes an interesting architecture for PFP, but the evaluation is not very complete, and the explanation of the method is very confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MybJqyOqE2I", "reviewer": "Reviewer_k9w6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Model uses more of the GO terms than other work that trim infrequent terms\n- Model achieves better performance on BPO term annotation prediction\n\nWeaknesses:\n- Limited novelty as it combines two existing models, closely following the approach of [1] but replacing the CNN-based sequence embedding with [2].\n- Discussion of related work and empirical analysis is lacking as not all appropriate baselines are considered and novelty vs existing work is not properly described/motivated. \n\nDetailed comments: \nApproach:\nI believe just after equation (2) where the text says that “U_t means a number of annotations in the training dataset,” this should be N_t. Otherwise, what are N_t and N_s? These are not defined anywhere. Likewise, equation (3) should be freq(k) = N_k + \\sum_ …. This is the same adjacency weighting scheme used by DeepGOA [1]. DeepGOA was not included in the experiments to validate the choice of sequence embedding, this method’s main difference with DeepGOA.\n\nRegarding how many GO terms to include in the model: An ablation study could be included to justify using “85% of all terms” vs the 13% (according to this paper) used in previous work. It is not well justified to include more terms without an experiment to show the advantage of this design decision. More terms may indeed cause overfitting and other issues, eg increasing the path length between correlated/useful GCN nodes. As this seems to be a main contribution of this work, such a study would help strengthen the contribution.\n\nRelated works:\nThe related works section, especially the last two paragraphs of 2.2 which discuss the most closely-related works is unclear. It does not clearly position the current work in context of these other deep structured PFP models and it fails to properly motivate the work by clearly describing shortcomings or limitations of these past works. This is not to say that these works do not have limitations, just that this text does not make it clear what gaps it is filling/addressing. In particular, the differences with DeepGOA [1] should be clearly discussed. \n\n“Cut off-criterion” is not well explained. Is this referring to the number of times a GO term had to be seen in a dataset in order to include it in the GO DAG? If so, please make sure to clarify this point.\n\nA closely related PFP work that I suggest be included in related work is: \nLi J, Wang L, Zhang X, Liu B, Wang Y. 2020. Gonet: a deep network to annotate proteins\nvia recurrent convolution networks. In: 2020 IEEE international conference on\nbioinformatics and biomedicine (BIBM), volume 2. Piscataway: IEEE, 29–34\n\nMostly minor comments on clarity and text:  \nSeveral typos and language ambiguities should be addressed.   \n-Abstract: “Go terms” should be “GO terms”  \n-Section 1   \n“On the other side,” not clear what this means. On the other side of what?  \n“Protein functions are defined by Gene Ontology (GO) composed of a …” -> “Protein functions are *described* by the Gene Ontology (GO) composed of a …”  \n“GCN has been applied…” -> “GCNs have been applied…”  \n“...which is involved relationship between…” What does this mean? This language is unclear. Is this saying the hierarchical information is *encoding* relationships between nodes at long distances? That it is *capturing* that relationship? Or something else?  \n“Extracts general ad helpful” -> “Extracts general *and* helpful”  \n-Section 2  \n“it required deep knowledge of proteomics and had high-cost” -> “*they* required deep knowledge of proteomics and had *high cost*.” Also, what is meant by “high cost” here?  \n“...(CNN) is one of the architectures as a sequence…”->“...(CNN) is one of the architectures *used* as a sequence…”  \n“transformers…,which is attention-based model” ->“transformers…,which *are* attention-based models”  \n“The recent global approaches expanded that built a structure encoder…” I am not sure what this means  \n“TALE implied …”-> “TALE employed …”  \n-Section 3  \n“A GO graph is represented to” -> “A GO graph is represented *by*”  \nWhat does “large-scale adjacency matrix” mean in this context?  \n“We explain the technicality of these process in this subsection” -> “We explain the *technical details* of *this* process in *the following* subsection”   \n“It was showed that” -> “It was shown that”  \n“...to extract high-level features related functions for PFP.” This is not clear. To what functions does this sentence refer, functions of the features? Functions of the protein that are to be learned? It is not clear.  \n“Seqvec” -> “SeqVec”  \n“The adjacency matrix is combined prior probability”->“The adjacency matrix combines prior probability...”  \n“Nevertheless, it apply the inveterate…” It is not clear what “applies” and what it applies. Is this the wrong word here or what is the sentence saying?  \n“label imbalanced problem”->“label imbalance problem”  \n-Section 4  \n“The GO dataset is a version of May 31, 2016”->“The GO dataset is a version *from* May 31, 2016”?  \n“as same as a CAFA3 assessment tool” ->“same as a CAFA3 assessment tool”  \n“GO as input excepting isolated”->“GO as input excluding isolated”  \n", "summary_of_the_paper": "This paper presents a model to predict Gene Ontology (GO) term annotations for protein function. The model uses an existing method, SeqVec [2], to encode the protein sequence and a GCN on the Gene Ontology (GO) DAGs to encode the structure of term relationships. Like in DeepGOA[1], the graph is weighted by functions of term frequencies. Sequence embedding is reduced via FC layers to dimension d, where d is equal to the DAG depth for the ontology being predicted. The final prediction is a dot product between the GCN encoding and the reduced sequence embedding. The model largely combines these two existing models.\n\n[1] Zhou, Guangjie, et al. \"Predicting functions of maize proteins using graph convolutional network.\" BMC bioinformatics 21.16 (2020): 1-16.  \n[2] Heinzinger, Michael, et al. \"Modeling aspects of the language of life through transfer-learning protein sequences.\" BMC bioinformatics 20.1 (2019): 1-17.  \n[3] Cao, Yue, and Yang Shen. \"TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding.\" Bioinformatics 37.18 (2021): 2825-2833.", "summary_of_the_review": "Technical contributions are very limited since the GO embedding module uses the same structure weighting, architecture, and loss function as existing method DeepGOA [1] and replaces its sequence embedding module with an existing protein sequence encoder, SeqVec [2]. Empirical results do not significantly improve, or in some cases even match, the previous method TALE [3], with the exception of prediction on the BPO.\n\nThe main difference is in the dataset in that it uses all GO terms that have a `is_a’ relation in the DAG, rather than the common approach of removing terms that are not seen or seldom seen in training. However later in the text it states that more deep BPO terms were also trimmed when the maximum depth was cut off to 80, so it is not clear which terms are used to evaluate after all. \n\nMotivation and comparison to prior work is very limited as the paper does not properly discuss differences to prior work or motivate the decision made. In particular, the choice of term “cut off” (which terms are used in training) are not empirically validated; the use of these particular architectures are not well-supported as the component modules are not used as baselines in experiments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KTHPxdEjQqU", "reviewer": "Reviewer_evqx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros:\n\n- The idea of this paper is very intuitive and understandable, including a language model for encoding the protein sequence and a Graph Convolutional Network (GCN) for representing Go terms. \n- Overall, the paper is well written. In particular, the INTRODUCTION section has a nice flow and puts the related works into context. Despite the method having limited novelty (LM + GCN), the method has been well-motivated by pointing out the limitations in SOTA methods. \n- The EXPERIMENTS section is well structured. In partuical, the author provides the statistics of sequences and GO terms in CAFA3, the experimental settings and evaluation are also introduced in great detail.\n\n##########################################################################\n\nCons:\n\n- The core idea of this paper is the PFP model, which combines a pre-trained Language Model (LM) and GCN-based model including new node-wise representations. However, both LM and GCN are now very mature methods, and the combination of the two is limited novelty. \n- The experimental part of this paper is only compared on one dataset: CAFA3. It is difficult to convince others that the proposed method is really effective.\n- In the comparison between the author's method and the baseline methods, the TALE is superior to the proposed method in $F_{max}$, but the author did not give a reasonable explanation.\n", "summary_of_the_paper": "This paper proposes a novel PFP model that combines a pre-trained Language Model (LM) and GCN-based model including new node-wise representations, to improve Protein Function Prediction (PFP) utilizing hierarchical features of Gene Ontology (GO) terms. The experimental results show that the proposed method achieves performance improvement compared to SOTA models, especially in the most difficult BPO.", "summary_of_the_review": "Considering the above pros and cons, I think there is still a lot of room for improvement in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3NEqaf_fta5", "reviewer": "Reviewer_BnVc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The paper is easy to read.\n\nWeakness: Overall the proposed method is very similar to DeepGOA, which the paper cited but did not benchmark with. The main difference is that DeepGOA used a CNN model trained from scratch to model protein sequences while here a pretrained language model was used. \n\nComment: Some ablation study on different components of the model would be helpful. Particularly the sequence embedding part of the model. Better protein language models e.g. ProtBert (cited), ESM may help the performance. ", "summary_of_the_paper": "The paper proposes a method to predict protein functions from Gene Ontology (GO) and protein sequences. The protein sequences are embedded with a pretrained protein language model (SeqVec) and the GO network is modelled with a graph convolutional neural network. The method was benchmarked using CAFA3 competition datasets. Improved model performance was shown against a Naive baseline, DIAMONDScore, DeepGoCNN, and TALE. \n", "summary_of_the_review": "I find the paper not particularly novel. The paper needs more discussion and comparison with DeepGOA. The method itself is not novel enough for ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to predict protein functions from Gene Ontology (GO) and protein sequences. The protein sequences are embedded with a pretrained protein language model (SeqVec) and the GO network is modelled with a graph convolutional neural network.

Reviewers found the paper well-written and structured. At the same time, they found the novelty of the paper limited. Two reviewers pointed out that the paper is very similar to DeepGOA, which the authors cite but don't compare against. Overall, there is consensus among the reviewers that the paper is not suitable for ICLR.

The authors didn't submit a rebuttal.

We encourage the authors to take into account reviewer comments to improve the paper. Since it is more on the application side, perhaps a computational biology conference / workshop would be more appropriate for this paper.
Decision: Reject