An Effective GCN-based Hierarchical Multi-label classification for Protein Function Prediction

1 INTRODUCTION . Protein Function Prediction ( PFP ) is one of the key challenges in the post-genomic era ( Zhou et al. , 2019 ; Li et al. , 2018 ) . With large numbers of genomes being sequenced every year , the number of novel proteins being discovered is expanding as well ( Spalević et al. , 2020 ) . On the other side , protein functions are reliably determined in wet-lab experiments which are cumbersome and highcost . As a result , the number of novel protein sequences without function annotations is rapidly expanding . In fact , UniRef100 ( Consortium , 2019 ) contains over 220M ( million ) protein sequences of which less than 1M have function annotations proved by experiments ( Littmann et al. , 2021 ) . Fast and accurate PFP is especially important in biomedical and pharmaceutical applications which are associated with specific protein functions . Protein functions are defined by Gene Ontology ( GO ) composed of a directed acyclic graph ( DAG ) ( Ashburner et al. , 2000 ) . There are three GO domains : Molecular Function Ontology ( MFO ) , Biological Process Ontology ( BPO ) , and Cellular Component Ontology ( CCO ) , where each node represents one function called GO term , and each edge represents a hierarchical relation between two GO terms , such as ’ is a ’ , ’ part of ’ , and etc . Since one protein is usually represented by multiple function annotations , PFP can be regarded as hierarchical multi-label classification ( HMC ) . There are two groups of existing approaches for PFP : local approaches and global approaches . Local approaches usually constructed a classifier for each label ( GO term ) or for a few labels of the same hierarchy level ( Lobley et al. , 2008 ; Minneci et al. , 2013 ; Cozzetto et al. , 2016 ; Rifaioglu et al. , 2019 ) . On the other hand , global approaches constructed a single classifier for multiple labels . The initial global approaches considered PFP to flat multi-label classification , ignoring the hierarchical structure of GO , and considering each label independently ( Kulmanov & Hoehndorf , 2020 ) . Recent global approaches have constructed a structure encoder to learn the correlation among labels ( Zhou et al. , 2020 ; Cao & Shen , 2021 ) . However , these results showed that existing global approach-based models had limitations in representing correlations between GO terms by learning a large-scale hierarchical graph of GO . One of the structure encoders in global approach-based models used Graph Convolutional Network ( GCN ) . GCN has been applied in learning representation of node features . Nevertheless , in the case of applying GCN to a large-scale hierarchical graph , it was difficult to obtain information among long-distance ( Chen et al. , 2019 ; Zeng et al. , 2021 ) and unable to obtain adequate structure information since adjacent nodes did not contain any hierarchical features ( Hu et al. , 2019 ) . To overcome these shortcomings , we build node-wise representations containing the whole hierarchical information , which is involved relationship between long-distance nodes and structure information . In this paper , we propose a novel PFP model that combines a pre-trained Language Model ( LM ) and GCN-based model including new node-wise representations . A pre-trained LM as a sequence encoder ( Littmann et al. , 2021 ) extracts general ad helpful sequence features . GCN-based model as a structure encoder blends hierarchical information into a graph representation to improve GCN performance in a large-scale hierarchical graph of GO . To predict the probability of each GO term representing target protein functions , the prediction layer is constructed as a dot product of outputs of two encoders . The experimental results show that our method achieves performance improvement compared to the-state-of-the art models , especially in the most difficult BPO . 2 RELATED WORK . 2.1 PROTEIN SEQUENCE FEATURE EXTRACTION . Protein sequences contain multiple biophysical features related to function and structure . Initially , protein biophysical features such as motifs , sequence profiles , and secondary structures were calculated from a suite of programs and then utilized as protein sequence feature vectors by combining them ( Lobley et al. , 2008 ; Minneci et al. , 2013 ; Cozzetto et al. , 2016 ; Rifaioglu et al. , 2019 ) . While these methods intuitively utilized the relationship between protein features and its biological functions , it required deep knowledge of proteomics and had high-cost . Various deep learning architectures that extract high-level biophysical features of protein sequences have been proposed . Convolutional Neural Network ( CNN ) is one of the architectures as a sequence encoder to learn sequence patterns or motifs that are related to functions ( Xu et al. , 2020 ) . Therefore , 1D CNN was utilized as effective sequence encoder in previous researches ( Kulmanov & Hoehndorf , 2020 ; Zhou et al. , 2020 ; Kulmanov et al. , 2018 ) . With the advent of transformers ( Vaswani et al. , 2017 ) , which is attention-based model , in Natural Language Processing ( NLP ) , various attention-based LMs were applied to protein sequence embedding ( Rao et al. , 2019 ; Vig et al. , 2020 ; Rives et al. , 2021 ; Heinzinger et al. , 2019 ; Elnaggar et al. , 2020 ) . As protein sequences can be considered as sentences , these learned the relationship between amino acids constituting the sequence and learned contextual information . SeqVec ( Heinzinger et al. , 2019 ) and ProtBert ( Elnaggar et al. , 2020 ) , which were learned protein sequences using ElMo ( Peters et al. , 2018 ) and BERT ( Devlin et al. , 2018 ) , showed that these mostly extracted biophysical features of protein structures and functions , such as secondary structures , binding sites , and homology detections . 2.2 PROTEIN FUNCTION PREDICTION ( PFP ) . PFP methods can be categorized into two different approaches which are local and global . Local approaches commonly employed single or few multi-label classifiers to each or few GO terms . These approaches included FFpred ( Lobley et al. , 2008 ; Minneci et al. , 2013 ; Cozzetto et al. , 2016 ) and DEEPred ( Rifaioglu et al. , 2019 ) . FFpred predicted one GO term by multiple Support Vector Machines ( SVMs ) trained with radial basis function kernels to recognize protein sequence patterns associated with the GO term ( Cozzetto et al. , 2016 ) . DEEPred created a multi-label classification model using deep neural network for each GO hierarchical level ( Rifaioglu et al. , 2019 ) . Each model could carry out five GO terms in most labels . Even though this procedure generated 1,101 different models concerning all GO domains , this still needed numerous models for PFP ( Rifaioglu et al. , 2019 ) . Ultimately , local approaches required expensive costs by training numerous models . On the other hand , global approaches respectively constructed one classifier model for MFO , BPO , and CCO . The initial global approaches considered PFP as flat multi-label classification . They focused on extracting function-related features from the protein sequence . One of the function-related features is motifs , called sequence patterns . DeepGoPlus ( Kulmanov & Hoehndorf , 2020 ) encoded sequence to extract motifs using 1D CNN and then predicted the probability of annotating each GO term using one fully connected layer . Compared to local approach-based previous methods , DeepGoPlus achieved improved performance in PFP , despite its simple model architecture . This model resulted in poor performance when the number of GO terms increases . The recent global approaches expanded that built a structure encoder to improve performance . In DeepGOA ( Zhou et al. , 2020 ) , all GO terms were regarded as correlated labels contrary to DeepGoPlus where they were regarded as independent labels . This enabled GCN to learn more effectively GO terms . They showed improved performance compared to DeepGoPlus , although the same protein sequence encoding was used . They extracted patterns using 1D CNN , but could not extract various features related to functions other than patterns . TALE ( Cao & Shen , 2021 ) implied transformer encoder ( Vaswani et al. , 2017 ) for sequence encoder and embedded GO with hierarchical information of each GO term . They learned various and specific features related to function as the transformer encoder . However , this indirectly learned the correlation among GO terms since they did not learn GO itself and just use hierarchical information of each GO term . One of the common problems of global approaches is that they partially used GO terms . In general , a cut-off criterion in each GO domain was a number of annotations such as 25 , 50 , and 150 . Global approaches , which had the cut-off criterion , utilized about 13 % of the total GO terms . Although TALE ( Cao & Shen , 2021 ) , which is the latest model , had the cut-off criterion to 1 , this still utilized only about 60 % GO terms for their model . In this paper , we propose a method that effectively learns the relation between GO terms in an extended data using over 85 % GO terms . This model results in similar or higher performance to the latest models used fewer GO terms . 3 PROPOSED METHOD . In this section , we describe the details of our model . The overall architecture is shown in Figure 1 . This model has two inputs : a protein sequence and the hierarchical graph of GO . A protein sequence is encoded by pre-trained LM and is reduced dimension to the feature vector size of GO respectively . A GO graph is represented to a large-scale adjacency matrix and node-wise feature matrix . These matrices are inputs to GCN for learning the hierarchical representation of GO terms . The prediction layer is built as a dot product of a protein feature vector and a GO terms vector to predict the probability of annotated GO term of each protein sequence . We explain the technicality of these process in this subsection . 3.1 PROTEIN SEQUENCE ENCODING . We employ pre-trained LM as a sequence encoder . As we mentioned in related work , pre-trained LM , such as SeqVec ( Heinzinger et al. , 2019 ) and ProtBert ( Elnaggar et al. , 2020 ) , already proved their performance to capture rudimentary features of proteins such as secondary structures , biological activities , and functions ( Rives et al. , 2021 ; Vig et al. , 2020 ) . Especially , it was showed that SeqVec ( Heinzinger et al. , 2019 ) is better than ProtBert ( Elnaggar et al. , 2020 ) to extract high-level features related functions for PFP ( Littmann et al. , 2021 ) . Seqvec ( Heinzinger et al. , 2019 ) is utilized as a protein sequence encoder . This makes the various lengths of protein sequences to 1×1024 representation vectors with high-level biophysical features . Protein sequence representations are converted to P ∈ R1×d low-dimensional representation vectors by fully connected layer to combine GO term vectors . 3.2 HIERARCHICAL REPRESENTATION OF GO TERM . Initial node features H0 ∈ RN×N are represented as a one-hot encoding matrix where the i th row GO term and its ancestors are 1 . The GO term Embedding layer reduces dimension by converting sparse matrix to H0 ∈ RN×d0 dense matrix for preventing overfitting and reducing training time in GCN . It indicate that node features contain its physical location and conceptual information in a hierarchical graph . N is a number of GO terms ( node ) and d0 is a scale of dimension , which is the maximum depth in each domain for containing hierarchical information in the dense matrix . The adjacency matrix A ∈ RN×N contains the relationship between GO terms . When a parent node is t and the children node is s , the adjacency matrix is combined prior probability P ( Us|Ut ) with Information Content ( IC ) ( Song et al. , 2013 ) that measures semantic similarity between t and s. The existing adjacency matrix is usually built by one-hot encoding or the prior probability . The one-hot encoding can not involve any additional information other than connection information between GO terms . The prior probability can involve relational information . Nevertheless , it apply the inveterate label imbalanced problem in the PFP dataset to the adjacency matrix due to being highly dependent on the training dataset . We solve this problem that adding IC , which is less affected by the training dataset ( Zhou et al. , 2020 ) . The adjacency matrix is defined as follows : A = P ( Us|Ut ) + IC ( s ) ∑ i∈child ( t ) IC ( i ) ( 1 ) Prior probability P ( Us|Ut ) is calculated as follows : P ( Us|Ut ) = P ( Us ⋂ Ut ) P ( Ut ) = P ( Us ) P ( Ut ) = Ns Nt ( 2 ) Where Ut means a number of annotations in the training dataset , P ( Us|Ut ) means the conditional probability that t and s are annotated in the same protein sequence . IC is calculated as follows : IC ( k ) = −logp ( k ) p ( k ) = freq ( k ) freq ( root ) freq ( k ) = Uk + ∑ i∈child ( k ) freq ( i ) ( 3 ) Where p ( k ) is probability of each GO term k in the GO dataset , freq ( k ) is frequency of t and child ( k ) is every children of k. Initial node features H0 are updated H l ∈ RN×d with node features of adjacent nodes through lth GCN layer ( 1 ≤ l ≤M ) . GCN layer is represented as follows : H l+1 = ReLU ( ÂH lW l ) ( 4 )

Summary: This paper presents a model to predict Gene Ontology (GO) term annotations for protein function. The model uses an existing method, SeqVec [2], to encode the protein sequence and a GCN on the Gene Ontology (GO) DAGs to encode the structure of term relationships. Like in DeepGOA[1], the graph is weighted by functions of term frequencies. Sequence embedding is reduced via FC layers to dimension d, where d is equal to the DAG depth for the ontology being predicted. The final prediction is a dot product between the GCN encoding and the reduced sequence embedding. The model largely combines these two existing models.  [1] Zhou, Guangjie, et al. "Predicting functions of maize proteins using graph convolutional network." BMC bioinformatics 21.16 (2020): 1-16.   [2] Heinzinger, Michael, et al. "Modeling aspects of the language of life through transfer-learning protein sequences." BMC bioinformatics 20.1 (2019): 1-17.   [3] Cao, Yue, and Yang Shen. "TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding." Bioinformatics 37.18 (2021): 2825-2833.