OntoProtein: Protein Pretraining With Gene Ontology Embedding
Authors: Ningyu Zhang
Zhen Bi
Xiaozhuan Liang
Siyuan Cheng
Haosen Hong
Shumin Deng
Qiang Zhang
Jiazhang Lian
Huajun Chen
Venue: ICLR 2022 Poster (2022)
Keywords: pre-trained language model
knowledge graph
protein representation

Abstract: Self-supervised protein language models have proved their effectiveness in learning the proteins representations. With the increasing computational power, current protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve remarkable improvement. However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better protein representations. We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge. In this work, we propose OntoProtein, the first general framework that makes use of structure in GO (Gene Ontology) into protein pre-training models. We construct a novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph. We propose novel contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and protein embedding during pre-training.  Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction.

Published as a conference paper at ICLR 2022
ONTOPROTEIN: PROTEIN PRETRAINING WITH GENE ONTOLOGY EMBEDDING
Ningyu Zhang 12.33 Zhen 2 Xiaozhuan Liang?. Siyuan Cheng
Haosen Hong
Shumin Dengl4 1.3 Qiang Zhang Jiazhang Lian'
Huajun Chen 13341
*College of Computer Science and Technology. Zhejiang University
Schooo of Software Technology, Zhejiang University
3 "Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies
"annghoou Innovation Center, Zhejiang University
2 Shangninggya bi zhen zju, 1 annggiiia aazhuan,,, 22151070 edw.ju.eduuu.e edu.cn [231 sm, 12028071, jzli azliann an, qiang. zhang cs, hua junsi r @z ju. a...jued.. edu. cn
ABSTRACT
Self-super protein language models have proved their effectiveness in learn- ing the proteins representations. With the increasing computational power, cur- rent protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve re- markable improvement. However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowl- edge facts for better protein representations. We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge. In this work, we propose OntoProtein, the first general framework that makes use structure GO (Gene Ontology) into protein pre-training models. We construct novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph. We propose novel contrastive learning with knowledge-aware negative sampling jointly optimize the knowledge graph and protein embedding during pre-training. Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction"
INTRODUCTION
Protein science, the fundamental macromolecules governing biology and life itself. has led markable advances in understanding the disease therapies and human health (Vig et al.] (202T). As sequence of amino acids, protein be viewed precisely as language, indicating that they may be modeled using neural networks that have been developed for natural language processing (NLP). Recent self-supervised pre-trained protein language models (PLMs) such ESM (Rao al. (02Ib)), ProteinBERT (Brandes et al. (2021)). ProtTrans (Elnaggar et al. (2200) which can learn powerful protein representations, have achieved promising results in understanding the structure and functionality of the protein. Yet existing PLMs for protein representation learning generally cannot sufficiently capture the biology factual knowledge, which is crucial for many protein tasks but is usually sparse and has diverse and complex forms in sequence.
By contrast, knowledge graphs (KGs) from gene ontology contain extensive biology structural facts, and knowledge embedding (KE) approaches (Bordes et al. (2013), ZZeng et al. (202T)) can efficiently embed them into continuous vectors of entities and relations. For example, as shown in Figure I without knowing PEX5 has specific biological processes and cellular components,
Equal contribution and shared co-first authorship.
Corresponding author.
"ode and datasets are available in ht http:///ggghhbaaaoo////////ooooa
http: httt:://geneane http://ggeenntttlgyy..oo

---

Published as a conference paper at ICLR 2022
90 VOBCa NY Pundron
Chamal
Plate
ADGM
treblay
Ausurday
GO Steley Proooss
CHASSS
peniie meae aeee dech
Prosein
Moleoular Functon
Biological Prociis
Deluuar Component
Interactio
Cat
ASGA
Tornina
Figure 1: Left: A protein example with biology knowledge (molecular function, biological pro- cess and cellular component): K (potassium ion) Cyelic nucleotide- gated cation channel protein. Right: The corresponding sub-graph regarding K* carrier proteins in ProteinK Yellow nodes are protein sequences and blue nodes are GO (Gene Ontology) entities with biological descriptions.
is challenging to recognize its interaction with other proteins. Furthermore, since protein's 's shape determines its function, is more convenient for models to identify protein's functions with the prior knowledge of protein functions having similar shapes. Hence, considering rich knowledge can lead to better protein representation and benefits various biology applications, e.g., protein contact prediction, protein function prediction, and protein-proted interaction prediction. However, differ- ent from knowledge-enhanl al. (2021a)) protein sequence and gene ontology are two different types of data. Note that protein sequence is composed of amino acids while gene ontology is a knowledge graph with text descrip- tion; thus, severe issues of structured knowledge encoding and heterogeneous information fusion remain.
approaches in NLP (Zhngg et al. (2019b), Waan et al. 2202 Waan
In this paper, we take the first to propose protein pre-training with gene ontology embedding (OntoProtein), which is the first general framework to integrate external knowledge graphs into protein pre-training. We propose a hybrid encoder to represent language text and protein sequence and introduce contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and the protein sequence embedding during pre-training. For the KE objective, encode the node descriptions (go annotations) as their corresponding entity embeddings and then optimize them following vanilla KE approaches (Bordes et al. (2013). We further leverage gene ontology of molecular function, cellular component, and biological process and introduce knowledge-aware negative sampling method for the KE objective. For the MLM (Mask Language Modeling) objective, we follow the approach of existing protein pre-training approaches (Rao al. 2021b).. OntoProtein has the following strengths:
(1) OntoProtein inherits the strong ability of protein understanding from PLMs with the MLM ob- ject. (2) OntoProtein can integrate biology knowledge into protein representation with the supervi- sion from KG by the KE object. (3) OntoProtein constitutes model-agnostic method and is readily pluggable into a wide range of protein tasks without additional inference overhead since we do not modify model architecture but add new training objectives.
For pre-training and evaluating OntoProtein, we need a knowledge graph with large-scale biology knowledge facts aligned with protein sequences. Therefore, we construct ProteinKG25, which con- tains about 612,483 entities, 4,990,097 triples, and aligned node descriptions from GO annotations. To the best of our knowledge, it is the first large-scale KG dataset to facilitate protein pre-training- We deliver data splits for both the inductive and the transductive settings to promote future research.
To summarize, our contribution is three-fold: (I) We propose OntoProtein, the first knowledge- enhanced protein pre-training approach that brings promising improvements to wide range of pro- tein tasks. (2) By contrastive learning with knowledge-awer awwrre aware sampling to jointly optimize knowl- edge and protein embedding, OntoProtein shows its effectiveness in widespread downstream tasks, including protein function prediction, protein-proot proteiproonn interaction prediction, contact prediction, and so on. (3) We construct and release the ProteinKG225. novel large-scale KG dataset, promoting the
2

---

Published as a conference paper at ICLR 2022
Text enhtamedd
eniledding
Knowledge Graph
negative samplig
Protein
TRa3IN raque
Input
Proteis GO Knowledge Graph
Enriched with descciptions
Objective for Lys
Sequences
Shared Protein Encoder
b Knowledge Graph Embedding
Protein Encoder
Protcin Function Prodiction
matt Burr Eaper
Fruch Exeaer
Maa holtg
Layer
Classifer
[M]
60: 00000019319
00:0019151
60 0019870
Obiertin for busa
Masked Protein Model
Downstream Tasks
Figure 2: Overview of our proposed OntoProtein, which jointly optimize knowledge graph embed- ding and masked protein model (Best viewed color.).
research on protein language pre-training. (4) We conduct extensive experiments in widespread pro- tein tasks, including TAPE benchmark, protein-protein interaction prediction, and protein function prediction, which demonstrate the effectiveness of our proposed approach.
METHODOLOGIES
We begin to introduce our approach of protein pre-training with ontology embedding (OntoProtein), shown in Figure OntoProtein incorporates external knowledge from Gene Ontology (Go) into language representations by jointly optimizing two objectives. We will first introduce the hybrid encoder, masked protein modeling, and knowledge encoder, and then will present the details of contrastive learning with knowledge-aware negative sampling. Finally, we will illustrate the overall pre-training objects.
2.1 HYBRID ENCODER
We first introduce the hybrid encoder to represent protein and GO knowledge. For the protein encoder, use the pre-trained ProtBert from Elnaggar al. (2020). ProtBert pre-trained using the BERT architecture with UniRef100 datasets. Compared to BERT Devlin et al. (2019), ProtBert encodes amino acid sequences into token level sentence level representations, which can be used for downstream protein tasks such as contacts prediction tasks. The encoder takes protein sequence of N tokens (#1, ..., N., ) as inputs, and computes contextualized amnio acid representation H Protein and sequence representation Protein via mean pooling. To bridge the gap between text and protein, we utilize affine transformation (an extra linear layer) project those representation the same space. We will discuss details of learning protein representation in Section Mask Protein Modeling
For the Go encoder, we leverage BERT (Devlin et al. (2019). ) a Transformer (Vaswani et al. (017) based text encoder for biological descriptions in Gene Ontology entities. Specifically, we utilize the pre-trained language model from (Gu et al. (000 The encoder takes a sequence of tokens (II...., xN) inputs, and computes Go representations Hco N by averaging all the token embeddings.
https://hugginggface co..aaac....ccccaoooaaeoaaoooaaa http://iccrccfftff/t/cc /mma:///amaaccccoooooo///////mmm Pww.eo.iaaa-aaaa.aarar
www.sedd-bbstraccte... fulltes

---

Published as a conference paper at ICLR 2022
Since the relations in Gene Ontology important for representing the knowledge of biology fea- tures, thus, we utilize a relation encoder with the random initialization, and those embeddings of relations will be optimized and updated during pre-training.
2.2 KNOWLEDGE EMBEDDING
We leverage the knowledge embedding (KE) objective to obtain representations in the pre-trainn training process since Gene Ontology is actually factual knowledge graph. Similar to Bordes et al. (2013), we use distributed representations to encode entities and relations. The knowledge graph here con- sists of lots of triples to describe relational facts. We define a triplet as (h, r, 1), where h and & are head and tail entities, r is the relation whose type usually is pre-defined in the schema Note that there are two different types of nodes EGO and Eprotein in our knowledge graph. EGO is denoted as nodes that exist in the gene ontology, such molecular function or cellular component nodes, and ego can be described by annotation texts. Cprotein is the protein node that links to the gene ontology, and we also represent Cprotein with amnio acids sequences. Concretely, the triplets in this knowledge graph can be divided into two groups, triple and triple Protein200 To integrate multi-modal descriptions into the same semantic space and address the heterogeneous information fusion issue, we utilize hybrid encoders introduced in the previous Section. Note that protein en- coder and GO encoder represent protein sequence and GO annotations separately.
2.3 MASKED PROTEIN MODELING
We use masked protein modeling to optimize protein representations. The masked protein modeling similar to masked language modeling (MLM).. During model pre-training, we use 15% proba- bility mask each token (amino acid) and leverage masked tokens. We initialize our model with the pre-trained model of ProtBert and regard fMLM LM of the overall objectives of OntoProtein by jointly training KE (knowledge embedding) and MLM. Our approach is agnessag, and other pre-trained models can also be leveraged.
cooss-y loss ÊMLM to estimate these
2.4 CONTRASTIVE LEARNING WITH KNOWLEDGE-AWARD NEGATIVE SAMPLING
Knowledge embedding (KE) is learn low-dimensional representations for entities and relations, and contrastive estimation represents scalable and effective method for inferring connectivity pat- terns. Note that a crucial aspect of contrastive learning approaches the choice of corruption distri- bution that generates hard negative samples, which force the embedding model to learn discrimina- tive representations and find critical characteristics of observed data. However, previous approaches either employ too simple corruption distributions, i.c., uniform, yielding easy uninformative neg- atives, or sophisticated adversarial distributions with challenging optimization schemes. Thus, in negative sampling, inexpen-
this paper, propose contrastive learning with knowledge- sive negative sampling strategy that utilizes the rich GO knowledge to sample negative samples. Formally, the KE objective can be defined as:
(K = - log - -dd,)(),t)
Σ log o (d(h), ()) -7)
(I)
(K,+t) is the negative sample, in which head or tail entities are random sampled construct the corrupt triples. is the number of negative samples, is the sigmoid function, and y means the margin. d is the scoring function, and we use TransE (Bordes al. (2013) for simplicity, where
(h, 1/2 + t||
Specifically, we define triple sets and entity sets as T and E, all triplets are divided into two groups If the head entity is protein node and the tail entity is GO node, we denote the triple as T Trrotein-Go Similarly, if head and tail entities are both GO nodes, we denote them as TGo-Go -ooo--o-oo As Gene Ontol- ogy describes the knowledge of the biological domain concerning three aspects, all entities in Gene Ontology belong to MFO (Molecular Function), CCO (Cellular Component), or BPO (Biological Process).
*The schema of the knowledge graph can found in Appendix A.1
4

---

Published as a conference paper at ICLR 2022
Cellular Component
Molecular Function
Biological Process
120000
150000
100000
80000
40000
50000
20000
0
torm
annotated
term annocated
prosein annotaced
Cellular Component
Molecular Function
Biological Process
250000
Leaf
400000
Leaf
Leaf
Non-keaf
Non-leal
250000
Non-leal
300000
200000
150000
200000
100000
100000
50000
100000
50000
0
10
12
Level of GO Serm
Level of GO torm
Level GO tarm
Figure 3: Top: Data Distribution of GO Terms. Bottom:
Statistics of Protein-Go GO Term.
To avoid plain negative samples, for those TGo-Go triples, we sample triples by replacing entities with the same aspect (MFO, CCO, BPO Finally, define the negative triple sets T' and positive triple as (h, r, t), the negative sampling process can be described as follows:
Coo-co ((h', r,, R' E W' U ((h,r,t) e EFIEFE)
T Trrotcin [(h, 1 EtE
where E' E
and we only replace the tail entities for Prottein-Ge GO triples.
2.5 PRE-TRAINING OBJECTIVE
We adopt the mask protein modeling object and knowledge embedding objective to construct the overall object of the OntoProtein. We jointly optimize the overall object as follows:
f=
al + fMLM
where a is the hyper-parameter. Our approach can embedded into existing fine-tuning scenarios.
EXPERIMENT
Extensive experiments have been conducted prove the effectiveness of our approach. In the pre- training stage, we construct a new knowledge graph dataset that consists of Gene Ontology and public annotated proteins. Our proposed model is pre-trained with this dataset and evaluated in several downstream tasks. We evaluate OntoProtein in protein function prediction, protein-protein interaction and TAPE benchmark (Rao al. (019)).
3.1 DATASETS
Pre-training Dataset To incorporate Gene Ontology knowledge into language models, we build a new pre-training dataset called ProteinKG220) which is a large-scale KG dataset with aligned descriptions and protein sequences respectively to GO terms
L and proteins entities. Gene Ontology
For Pprotein-GG triples, it is also intuitive to replace the proteins with their homologous proteins to gen- crate hard negative triples, and we leave this for future works.
Shttps: / //n::////jnpppp p.
The structure of GO can he described in terms of graph, where each GO term is node, and the relation- ships between the terms edges hetween the nodes.

---

Published as a conference paper at ICLR 2022
Structure SS-Q8
Evolutionary Homology
Engineering Stability
Method
SS-Q3
Contact
Fluorescene
LSTM
0.75
0.59
0.26
0.26
0.67
0.69
TAPE Transformer
0.73
0.59
0.25
0.21
0.68
0.73
ResNet
0.75
0.58
0.25
0.17
0.21
0.73
MSA Transformer
0.73
0.49
ProtBert
0.81
0.67
0.35
0.29
0.61
0.82
OntoProtein
0.82
0.68
0.40
0.24
0.66
0.75
Table 1:
Results on TAPE Benchmark. SS a secondary structure task that evaluates in CB513. In contact prediction, test medium- and long-range using P@L/2 metrics. In protein engineering tasks, we test fluorescence and stability prediction using spearman's 's metric.
consists of set of GO terms (or concepts) with relations that operate between them, e.g.. molecular function terms describe activities that occur at the molecular level. A GO annotation is statement about the function of particular gene or gene product, c.g., the gene product "cytochrome c" can be described by the molecular function oxidoreductase activity. Due to the connection between Gene Ontology and Gene Annotations, we combine the two structures into unified knowledge graph. For each GO term in Gene Ontology, align it to its corresponding name and description and concatenate them by colon as an entire description. For each protein Gene annotation, we align it the Swoss-Prof] protein knowledge database, and extract its corresponding sequence its description. ProteinKG25, there exists 4,990,097 triples, including 4,879,951 Tprotein-GO and the statistics of our ProteinKG25, Detailed construction
110,146 TGo-Go triples. procedure and analysis of pre-train datasets can be found in Appendix A.1
Downstream Task Dataset We use TAPE as the benchmark (Rao al, (019)) to evaluate protein representation learning. There are three types of tasks in TAPE, including structure, evolutionary, and engineering for proteins. Following Rao et al. (2021a). we select 6 representative datasets including secondary structure (SS), contact prediction to evaluate OntoProtein.
Protein-protein interactions (PPI) are physical contacts of high specificity established between two or more protein molecules; we regard PPI as a sequence classification task and use three datasets with different sizes for evaluation. STRING is built by Lv et al. (2021), which contains 15,335 proteins and 593,397 PPIs. We also use SHS27k and SHS148K, which are generated by Chen et al. 2019).
Protein function prediction aims assign biological or biochemical roles proteins, and we also regard this task a sequence classification task. We build new evaluation dataset based our ProteinKG25 following the standard CAFA protocol (Zhou et al. (2019)). Specifically, we design two evaluation settings, the transductive setting and the inductive setting, which simulate two sce- narios gene annotation in reality. the transductive setting, the model can generate embeddings of unseen protein entities with entity descriptions. On the contrary, for the inductive setting, those entities have occurred in the pre-training stage. The detailed construction of the dataset can be found in Appendix A. As shown in Figure proteins are, average, annotated by 2 terms in CCO, in MFO, and 3 in BPO, indicating that protein function prediction can be viewed as multi-label problem. Notably, we notice that leaf GO terms tend have more specific concepts than non-leaf' GO terms. Meanwhile, there exists a challenging long-tail issue for the function prediction task.
3.2 RESULTS
TAPE BENCHMARK
Baselines In TAPE, we evaluate our OntoProtein compared with five baselines. The first is the model with LSTM encoding of the input amino acid sequence, which provides simple baseline. The second is TAPE Transformer that provides a basic transformer baseline. We further select ResNet from He al. 2016) as a baseline. The forth is the MSA Transformer (Rao al. (2021a))
httpp:: wttp:/://wwwwwwp..roooooo

---

Published as a conference paper at ICLR 2022
SHS27k
SHS148k
STRING
Methods
BFS
DFS
BFS
DFS
BFS
DFS
DPPI
41.43
46.12
52.12
52.03
56.68
66.82
DNN-PPI
48.09
54.34
57.40
58.42
53.05
64.94
PIPR
44.48
57.80
61.83
63.98
55.65
67.45
GNN-PPI
63.81
74.72
71.37
82.67
78.37
91.07
GNN-PPI (ProtBert) 70.94
73.36
70.32
78.86
67.61
87.44
GNN-PPI (OntoProtein)) 72.26
78.89
75.23
77.52
76.71
91.45
Table Protein-Proteiie Interaction Prediction Results, Breath-First Search (BFS) and Depth-First Search (DFS) are strategies that split the training and testing PPI datasets.
Transductive
Inductive
Method
BPO
MFO
CCO
BPO
MFO
CCO
ProtBert
0.58
0.13
8.47
0.64
0.33
9.27
OntoProtein
0.62
0.13
8.46
0.66
0.25
8.37
Table 3: Protein Function Prediction Results on three sub-sets with two settings. BPO refers to Biological Process, MFO refers to Molecular Function, and CCO refers to Cellular Component.
Note that MSA Transformer takes advantage of multiple sequence alignments (MSAs) and is the current state-of-the-art approach. Finally, we use ProtBert (Elnaggar et al. (2020)) with 30 layers of BERT encoder, which the largest pre-trained model among baselines.
Results We detail the experimental result on TAPE in Table [ Concretely, notice that On- toProtein yields better performance in all token level tests. For the second structure (SS-Q3 and SS-Q8) and contact prediction, OntoProtein outperforms TAPE Transformer and ProtBert, show- ing that it can benefit from those informative biology knowledge graphs in pre-training. Moreover, OntoProtein can achieve comparable performance with MSA transformer. Note that our proposed OntoProtein does not leverage the information from MSAs. However, with external gene ontology knowledge injection, OntoProtein can obtain promising performance. In sequence level tasks, On- toProtein can achieve better performance than ProtBert in fluorescence prediction. However, we observe that OntoProtein does not perform well protein engineering, homology, and stability pre- diction, which are all regression tasks. We think this is due to the lack of sequence-level objectives our pre-training object, and we leave this for future work.
PROTHIN-PROTEN INTERACTION
Baselines We choose four representative methods as baselines for protein-protein interaction. PIPR (Chen et al. (2019),, DNN-PPI (Li et al.] (2018)) and DPPI (Hashemifar et al. (2018) are graph neural network based method
deep learning based methods. GNN-PPI (Lv et al. (202T) for better inter-novel-protein interaction prediction. To evaluate our OntoProtein, we replace the initial protein embedding part of GNN-PPI with ProtBERT and OntoProtein as baselines.
Results From Table observe that the performance of OntoProtein is better than PIPR, which demonstrates that external structure knowledge can be beneficial for protein-protein interaction pre- diction. We also notice th method achieve promising improvement in smaller dataset SHS2K, even outperforming GNN-PPI and GNN-PPI (ProtBert). With a larger size of datasets, OntoProtein can still obtain comparable performance to GNN-PPI and GNN-PPI (ProtBert).
PROTEIN FUNCTION PREDICTION
Baselines
For simplicity, leverage Seq2Vec (Littmann et al. (021 the backbone for fair comparison and initialize embeddings with ProtBert and our OntoProtein. Note that our approach is model-agnostic, and other backbones also be leveraged.

---

Published as a conference paper at ICLR 2022
S seq < 12
12 S sey <
24 seg
P@L P@L/2 P@L/5
P@L
P@L/2 P@L/S
P@L
L
P@L/2
L@L/S
TAPE Transformer
0.28
0.35
0.46
0.19
0.25
0.33
0.17
0.20
0.24
LSTM
0.26
0.36
0.49
0.20
0.26
0.34
0.20
0.23
0.27
ResNet
0.25
0.34
0.46
0.28
0.25
0.35
0.10
0.13
0.17
ProtBert 0.30
0,40
0.52
0.27
0.35
0.47
0.20
0.26
0.34
OntoProtein 0.37
0.46
0.57
0.32
0.40
0.50
0.24
0.31
0.39
Table 4: Ablation study of contact prediction. seg refers to the sequence length between amino acids. "@K""N is precision for the top K contacts and L is the length of the protein.
We randomly select protein from the contact test dataset for visual analysis. Left: We visualize the 7th head in the last attention layer in OntoProtein. Right: It the contact label matrix.
Figure 4:
Results We split the test sets into three subsets (BPO, MFO, and CCO) and evaluate the perfor- mance of models separately. From Table 3 we notice that our OntoProtein can yield 4% im- provement with transductive setting and 2% advancement with inductive setting in BPO, further demonstrating the effectiveness of our proposed approach. We also observe that OntoProtein ob- tain comparable performance other subsets. Note that there exists a severe long-tail issue in the dataset, and knowledge injecting may affect the representation learning for the head but weaken the tail representation, thus cause performance degradation. We leave this for future works.
3.3 ANALYSIS
Table illustrates a detailed experimental analysis on the contact prediction. To further analyze the model's performance, we conduct experiments probe the performance of different sequences. Specifically, protein sequence lengths from short-range (6 < seq < 12) to long-range (24 I seq) are tested with three metrics (P@L. L, L/2, P@ L/5). We choose several basic algorithms such LSTM and TAPE transformer as baselines. For fairness, ProtBert is also leveraged for comparison. It can be seen that the performance of OntoProtein exceeds all other methods in all test settings, which is reasonable because the knowledge injected from Gene Ontology is beneficial. Further, we random sample a protein instance from the test dataset and analyze its attention weight of OntoProtein. We conduct visualization analysis as shown in Figure to compare the contacts among amino acids with the contact label matrix.
3.4 DISCUSSION
Applying techniques from NLP to proteins opens new opportunities extract information from proteins in self-supervised, data-driven way. Here we show for the first time that injecting external knowledge from gene ontology can help to learn protein representation better, thus, boosting the downstream protein tasks. However, the gains in pre-trained models using large-scale corpus is still relatively small. Note that the knowledge graph ProteinKG25 can only cover small subset of all proteins, thus. limiting the advancement. We will continue to maintain the knowledge graph by adding new facts from Gene Ontology. Besides, previous studies (Liu al. (2020); /Zhang al. (2021a)) indicate that not all external knowledge
proposed OntoProtein compared to previous
8

---

Published as a conference paper at ICLR 2022
beneficial for downstream tasks, and it is necessary investigate when and how to inject external knowledge into pre- pre-trained models effectively. Finally, our proposed approach can be viewed as jointly pre-training human language and protein (the language of life). Our motivation is to crack the language of life's code with gene knowledge injected protein pre-training. Our work is but a small step in this direction.
RELATED WORK
4.1 PRE-TRAINED LANGUAGE MODELS
Up to now, various efforts have been devoted to exploring large-scale PTMs, either for NLP (Peters
jtt al. (2018); Devlin al. (2019)), or for CV (Tan & Bansal 2019)). Fine-tuning large-scale PTMs
such as EL (Peters et al. (2018)), GPT3 (Brown et al. (0200). BERT (Devlin et al. (2019). ).
XLNet YYng et al. (2019) UniLM (Dong et al. (2019)) for specific Al tasks instead of learning models from scratch has also become consensus (Han et al. (202I). Apart from the of large scale language models for natural language processing, there has been considerable interest in develop- ing similar models for proteins (Xiao al. (2021): y) Rives et al. (202T)). |ao et al. (2021a) is the first to study protein Transformer language models, demonstrating that information about residue- residue contacts can be recovered from the learned representations by linear projections supervised with protein structures, VVgg et al. (2021) performs an extensive analysis of Transformer attention, identifying correspondences to biologically relevant features, and also finds that different layers of the model are responsible for learning different features. Elnaggar et al. (020)) proposes ProtTrans, which explores the limits of up-scaling language models trained on proteins as well as protein se- quence databases and compares the effects of auto-regressive and auto-encoding pre-training upon the success of the subsequent supervised training. Human-curated or domain domain-specific knowledge is essential for downstream tasks, which is extensively studied such as Himmelstein & Baranzini (2015), Smaili et al. (2018), Smaili et al. (2019), Hao et al. (2020). [oannidis et al. (2020). How- ever these pre-training methods do not explicitly consider external knowledge like proposed OntoProtein.
4.2 KNOWLEDGE-ENIANCHE LANGUAGE MODELS
Background knowledge has been considered as an indispensable part of language understanding
((Zhang et al.] 2 Deng et al.] 2021 Li et al.
2021: [Zhang et al.) 2019a: Yu et al., 2020;; iZhu
cct al. 2021 Zhang et | [0| lb] Chen et al. 2002
Zhang et al. 2022b Silvestri et al. | [021| Zhhng
al. 2021c Yao et al. 2022 Zhang et al.
2022a)),
which has inspired
models
including ERNIE (Tsinghua) (Zhang et al.
(2019b),
ERNIE (Baidu) (Sun et al. ))019), KnowBERT
(Peters et al.
(2019)). WKLM (Xiong al.
(2020)). LUKE (Yamada et al. (2020)) KEPLER (Wang
cet al.
(2021b)) K-BERT (Liu et al. (2020). K-Adaptor (Wang et al. al. (202Ta)). and CoLAKE (Sun
al.
2020). ERNIE (Zhang al. (2019b) injects relational knowledge into the pre-trained model
BERT. which aligns entities from Wikipedia to facts in WikiData. KEPLER (Wang et al. (2021b)) jointly optimizes knowledge embedding and pre-trained language representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also effectively learn KE through the abundant information in the text.
Inspired by these works, we propose OntoProtein that integrates external knowledge graphs into protein pre-training. To the best of our knowledge, we are the first to inject gene ontology knowledge into protein language models.
5 CONCLUSION AND FUTURE WORK
In this paper, we take the first step to integrating external factual knowledge from gene ontology into protein language models. We present protein pretraining with gene ontology embedding (On- toProtein), which is the first general framework to integrate external knowledge graphs into protein pre-training. Experimental results on widespread protein tasks demonstrate that efficient knowledge injection helps understand and uncover the grammar of life. Besides, OntoProtein is compatible with the model parameters of lots of pre-trained protein language models, which means that users can directly adopt the available pre-trained parameters on OntoProtein without modifying the

---

Published as a conference paper at ICLR 2022
chitecture. These positive results point to future work in (1) improving OntoProtein by injecting more informative knowledge with gene ontology selection; (2) extending this approach to sequence generating tasks for protein design.
ACKNOWLEDGMENTS
We want to express gratitude to the anonymous reviewers for their hard work and kind comments. This work is funded by MSFCU199222/NFFFX1440000 National Key R&D Program of China (Funding
Zhejiang Provincial Natural Science Foundation of China (No. LGG22F03001 I), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduc- tion Programme 1.)-G..
REPRODUCIBILITY STATEMENT
Our code and datasets are all available in the htttps:// com/ junipp/ ontoProtein for reproducibility. Hyper-parametts are provided in the Appendix A.3
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Christopher C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference Neural Information Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp- 2787-2795, 2013. URL httttp:://prroceeeee www.eeeedigggg.. www.neur........ 0oc//paper/2013/heae Icecc 2ataaaa8822222111133222222222822 Abs.teataaa----.aaaataaaa html
Abstract
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: A univer- sal deep-leamnng model of protein sequence and function. bioRxiv, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Kerbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Πla Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https::/pppoceeiigg..s neur wpw.urrpp..... CC/ wwtp.//pppp.//2////a...... 8wtb88ofbalc.... 42f4aa
1457c0d6bfcb453 22 Abst html
Muhao Chen, Chelsea J.-T. Ju, Guangyu Zhou, Xuelu Chen, Tianran Zhang, Kai-Wei Chang, Carlo Zaniolo, and Wei Wang. Multifaceted protein-protein interaction prediction based siamese residual RCNN. Bioinform., 35(1)):: 35(14):i305-i314, 2019, doi: 10. (093/ooinormatiissh222222 URL
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. CoRR, abs/2104.07650, 2021. URL
2104. 07650
Shumin Deng, Ningyu Zhang, Luoqiu Li, Chen Hui, Huaixiao Tou, Mosha Chen, Fei Huang, and Huajun Chen. Ontoed: Low-resource event detection with ontology embedding. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Papers), Virtual Event. Au- gust 1-6, 2021, pp. 2828-2833 Association for Computational Linguistics, 2021. doi: 10, 18653/ 220
v1/2021 acl- 220. URL URL.https / wttp///...163/////// 2200.
10

---

Published as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- vre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics. 2019. doi: 10.18653/v1/n194 1423. URL https: https://doi..o org.10....0oooo waarg//...1653///. 1ww.1/16631vv.
1423
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042 13054, 2019. URL http::: Abstract. Abstract.htm]
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing. CoRR, has/2007.06222. 2020. URL https:
7/arxiv. wrg.vvrrg/aaae 2aw.bb/s200000622..
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Nomain-specifis language model pretraining for biomedical natural language processing, 2020.
Xu Han. Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yugi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu. Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. Pre-trained models: Past, present and future. CoRR, abs/2106.07 139, 2021. URLpttpss: http://aaxxvvorreeeeeee
abs/ fess/2106.co 07139
Junheng Hao, Chelsea J-T Ju, Muhao Chen. Yizhou Sun. Carlo Zaniolo, and Wei Wang. Bio-joie: Joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp- 1-10, 2020.
Somaye Hashemifar, Behnam Neyshabur, Aly A. Khan, and Jinbo Xu. Predicting protein-protein interactions through sequence-basee based deep learning. Bioinform. 34(17):800--8800 2018. doi: 10.1093/bionfoomaaics//o// URL bty573
10. http:///w.o...10////ooooooooeae.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: N..110/CVVR.22166. URL. httttstts:://// //op:::///.i.. .arllorg//00 1109 2016. 90
Daniel S Himmelstein and Sergio E Baranzini. Heterogeneous network edge prediction: a data integration approach to prioritize disease- associated genes. PLoS computational biology. 11(7): c1004259, 2015.
Vassilis N loannidis, Xiang Song, Saurav Manchanda, Mufei Li, Xiaoqin Pan, Da Zheng, Xia Ning. Xiangxiang Zeng, and George Karypis. Drkg-drug repurposing knowledge graph for covid-19, 2020.
Chengxi Li, Feiyu Gao, Jiajun Bu, Lu Xu, Xiang Chen, Yu Gu, Zirui Shao, Qi Zheng, Ningyu Zhang, Yongpan Wang, and Zhi Yu. Sentiprompt: Sentiment knowledge enhanced prompt-tuning for aspect-based sentiment analysis. CoRR, abs/2109.08306 2021. URL pettss//arrae
org/ htt//bb//2109 08306
Hang Li, Xiu-Jun Gong, Hua Yu, and Chang Zhou. Deep neural network based predictions of protein interactions using primary sequences. Molecules, 23(8):1923, 2018.
11

---

Published as a conference paper at ICLR 2022
Maria Littmann, Michael Heinzinger, Christian Dallago, Tobias Olenyi, and Burkhard Rost. Em- beddings from deep learning transfer go annotations beyond homology. Scientific reports, 11(1): 1-14, 2021.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Haotang Deng, and Ping Wang. K-BERT: en- abling language representation with knowledge graph. In The Thirty-Fourty AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 2901-2908 AAAI Press, AAAT/ 2020. URL httthhtts:///et http::://aaa...rrgoo o is index. http://AAAaaat..lleleeee/e//////
Guofeng Lv, Zhiqiang Hu, Yanguang Bi, and Shaoting Zhang. Learning unknown from corre- lations: Graph neural network for inter-novel-proeri interaction prediction. In Zhi-Hur Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 3677-3683 ijcai.org, 2021. doi: 10.24963/ijcai. 2021/506. BtLpttts::// / doi orgg / 0 943 paaai.caai... 2021 /506
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024 8035, 2019. URL htt.://2....22222...000012222222
neuris...................... CC/ httpp/pppr/////aasseee html
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pp. Association for Computational Linguistics, 2018. doi: 10. 10.18653/v1/nl 1202 URL https: 1202
10. www.11865vvvnnln //
Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 43- 54. Association for Computational Linguisties, 2019. doi: 10. 10.11653//1/DD11-1000 URL U //doi.org/I0 1000 18100//1111.11111111110 1005
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with TAPE. In Hanna M. Wal- lach, Hugo Larochelle, Alina Beygelzimer, Florence Alccé-BBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, 2019. URL htt.tttpps//pppcceeiinn/ooooeeee neur ps cc/paper/ htt.ttttttttttttt......
BC, Canada, pp. 1t11://aaa/3116666666333337772 htt.://ee77722227000eeeeee222113
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8844 8856. PMLR, 2021a. URL httt:://proceee ngs ml ppwss/vv33 https///13/r/aaa1aaaaaa raw2ia..thaa
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
b021b. URL
12

---

Published as a conference paper at ICLR 2022
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sei. USA, 1 (18(15))220112311 18, 2021. doi: 10.1073/pnas.200 20162391 URL. URI.https://dol org 10030.1.03//00 pww.://ppaa...
6239118
Mattia Silvestri, Michele Lombardi, and Michela Milano. Injecting domain knowledge in neural net- works: A controlled experiment on a constrained problem. In Peter J. Stuckey (ed.), Integration of Constraint Programming, Artificial Intelligence, and Operations Reseurch 18th International Conference, CPAIOR 202 Vienna, Austria, July 5-8, 2021, Proceedings, volume 12735 of Lec- ture Notes in Computer Science, pp. 266-282. Springer. 2021. doi: 10. 10.1007/978 0-188-3--00--------0 030-78230-6, 17.
10 1007 3-000-000000000 78230- 730-00000--666 17
Fatima Zohra Smaili, Xin Gao, and Robert Hochndorf. of biological entities and their ontology-basee annotations. Bioinformatics, 34(13):152-i60, 2018.
Onto2vec: joint vector-based representation
Fatima Zohra Smaili, Xin Gao, and Robert Hoehndorf. Opa2vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction. Bioinformatics, 35(1)): 2133-2140, 2019.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. Colake: Contextualized language and knowledge embedding. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 1660-3670, In- ternational Committee on Computational Linguistics, 2020. doi: 10.18653/v1/2020 327. htRL:ttt:://// //ooip orgg 18653 21w...11116331122222.12
coling main.
ng ma i 327
Yu Sun, Shuohuan Wang. Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu, ERNIE: enhanced representation through knowledge integration. CoRR, 00521904409223, 2019. URL http::///////
1904.000000 09223
Hao Tan and Mohit Bansal. LXMERT: learning mrdssttyyality encoder representations from trans- formers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNIP-IJCNN 2019, Hong Kong, China, November 3-7, 2019, pp. 5099-5110., Association for Computational Linguistics, 2019. URL
doi: 1L.18653/11/D19-11
htt.t:tpp//// //doi org/ 18653/11/1D1399 1514
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL http:psss//ppocceelngg...coco
ps. CC pccppaper///eeeeee / 201 7/hash/ 4 html
Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httts:://penrews Ywt L2vLuu77
net/ for.mrrrrrdr
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL/JCNLP 2021, Online Event, August 1-6, 2021, volume ACL/ICNNPP 2021 of Findings of ACL. pp. 1405-1418, Association for Computational Linguistics, 2021a. doi: 10.18653/v1/2021. 18653/v1/2022 fi minnnnnnns-all acl.122
findiigg..iinn-ace 121. URL https: / /oo. orq/1c.
13

---

Published as a conference paper at ICLR 2022
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representa- 194, 2021b. URL
tion. Trans. Assoc. Comput. Linguistics,
org/
o hwwpiiiwwpppppccclaaiiiiccc
Yijia Xiao, Jiezhong Qiu, Ziang Li, Chang- Yu Hsieh, and Jie Tang. Modeling protein using large- scale pretrain language model. CoRR, 0bs/2108.004335 2021. URL ltttps://axxiv org/
abs 08.08.1 07435
Wenhan Xiong. Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclope- dia: Weakly supervised wwwwwldge-pretrainge language model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL wtt.thttt/::/mmmmtee
net. forum?oruum//m dd 2m64tDH
Ikya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. LUKE; deep contextualized entity representations with entity-awan self-attention. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6442 6454. Association for Computational Linguistics, 2020. doi: 10. /00.6/////020.mmmlppp.. manp-mman..52. 5233. URL
https:://oi.com http///ii.ogg/////// 18653 20/55//11/22000 e.r.l.mmrllgg.. ma maain.... 523
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bucc Emily B. Fox, and Roman Garnett (eds.), Advances in Neurul Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5764, 2019. URL ht http::///occceepeeeeooooooo www.iinnnnn.nnusssee.eu newrips 7cc69 Abstract. htt.rracctttmml
pacpppper/
Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, and Huajun Chen. Kformer: Knowledge injection in transformer foed-forward layers. CoRR, abs/2201.05742, 2022. URL
/ttpss///xxxv.oorr//////00. wwtp:/s///aaa////ooo 05742
Haiyang Yu, Ningyu Zhang, Shumin Deng, Hongbin Ye, Wei Zhang, and Huajun Chen. Bridg- ing text and knowledge with multi-prototype embedding for few-shot relational triple extraction. In Donia Scott, Núria Bel, and Chengqing Zong (eds.). Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), Decem- ber 8-13, 2020, pp. 6399-641 6410. International Committee on Computational Linguistics, 2020. doi: 10.18653/v1/2020.co col inn ma m...mmain5553.. 563
www.::0.ooing--mi.55.5. URL https:://dat http://....0//10111165/1/12/2220
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang. Xi Chen, Wei Zhang, and Huajun Chen. Long-tail relation extraction via knowledge graph embeddings and graph convolution net- works. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HI.T 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 10. 10.18653/v1/m 10153//1/n19-/11 1306. URL ht /ttt:::://oi..oo/11...BBB.
Association for Computational Linguistics, 2019a. doi: 1306
Ningyu Zhang, Shumin Deng, Xu Cheng, Xi Chen, Yichi Zhang, Wei Zhang, and Huajun Chen. Drop redundant, shrink irrelevant: Selective knowledge injection for language pretraining- In Zhi-Hua Zhou (cd.). Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 4007- 4014. orgai.org, 2021a. doi: 0.24963/ijeai.222155522 URL htthhttps//d/l. org 11../....444... 1jcai. 2021/502222222
Ningyu Zhang, Qianghuai Jia, Shumin Deng, Xiang Chen, Hongbin Ye, Hui Chen, Huaixiao Tou, Gang Huang. Zhao Wang, Nengwei Hua, and Huajun Chen. Alicg: Fine-graine -rainer and evolvable conceptual graph construction for semantic search at alibaba. In Feida Zhu, Beng Chin Ooi, and Chunyan Miao (eds.), KDD 211: The 27th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pp. 3895-3905. ACM,
14

---

Published as a conference paper at ICLR 2022
2021b. doi: 10.1145/3447558.34470777 3467057
Ningyu Zhang, Hongbin Ye, Jiacheng Yang, Shumin Deng, Chuanqi Tan, Mosha Chen, Songfang Huang, Fei Huang, and Huajun Chen. LOGEN: few-shot logical knowledge--onnditioned text generation with self-training. CoRR, abs/2112.01404, 2021c. URL https::/aaxiv.orrr
abs /esa/2112. 2.12.011111. 01404
Ningyu Zhang, Xin Xie, Xiang Chen, Shumin Deng, Chuanqi Tan, Fei Huang, Xu Cheng, and Huajun Chen. Reasoning through memorization: Nearest neighbor knowledge graph embeddings. CoRR, abs/2201.05555 2022a.
org. abs / 2201. 05575
Ningyu Zhang, Xin Xu, Liankuan Tao, Haiyang Yu, Hongbin Ye, Xin Xie, Xiang Chen, Zhoubo Li, Lei Li, Xiaozhuan Liang. Yunzhi Yao, Shumin Deng, Zhenru Zhang, Chuanqi Tan, Fei Huang, Guozhou Zheng, and Huajun Chen. Deepke: A deep learning based knowledge extraction toolkit for knowledge base population. CoRR, abs/2201 1..3333,, 2022b. httptttts///axiii
abs/ 2201. 03335
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE:
hanced language representation with informative entities. Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 1441-1451, Association for Computational Linguistics, 2019b. doi: 10.18653/11/11-11111 URL https::///// / ttt. 11..1..1..1111..1 v/pp5/////pp/v 1139.
Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Xu Ming, and Yefeng Zheng. PRGC: potential relation and global correspondence based joint relational triple extraction. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the Ith International Joint Conference on Natural Language Processing. ACI/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 6225-6235. Associa- tion for Computational Linguistics, 2021, doi: 10. 10.18653/1/20211aad/ acl-lclng--88 URL https:
7/doi //doi.
18653/v661vv/ 1ttp://0..1635311122
aacl aan-llong 486
Naihui Zhou, Yuxiang Jiang, Timothy R Bergquist, Alexandra Lee, Balint Z Kacsoh, Alex W Crocker, Kimberley A Lewis, George Georghiou, Huy N Nguyen, Md Nafiz Hamid, et al. The cafa challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens. Genome biology, 20(1):1-23, 2019.
Yushan Zhu, Huaixiao Tou, Wen Zhang, Ganqiang Ye, Hui Chen, Ningyu Zhang, and Huajun Chen. Knowledge perceived multi-modal pretraining in e-commerce. CoRR, abs/2109.00895, 2021.
wttphttt///aaaa org. abs/2109. 00895
A APPENDIX
A.1 CONSTRUCTION OF PROTEINKG25
To incorporate Gene Ontology knowledge into language models and train OntoProtein, we construct ProteinKG25,, a large-scale KG dataset with aligned descriptions and protein sequences respectively to GO terms and proteins entities. We design two evaluation schemes, the transductive and the inductive settings, which simulate two scenarios of gene annotation reality.
We use the latest Gene Ontology and Gene Annotations released in April 2020. Gene Ontology de- picts the relation between GO terms using GO-GO triplet, and Gene Annotation depicts the relations between protein and GO term using Protein-GO triplet. Due to this connectivity, we combine the two structures into a unified knowledge graph. For each GO term Gene Ontology, we align it to its corresponding name and description and concatenate them by colon For each protein in Gene annotation, we align it to the Swiss-Pras and extract its corresponding se- quence as its description. The final KG contains 4,990.097 triplets (4,879,951 Protein-GO triplets and 1 110,146 GO-GO triplets), 612,483 entities (565,254 proteins and 47,229 GO terms) and relations.
entire description.
15

---

Published as a conference paper at ICLR 2022
Due to the tree-like hierarchical structure of Gene Ontology, define the depth of GO terms using the shortest path from current GO terms to the root node of GO terms. The distribution of Protein- GO triplets with respect to the depth of GO term is shown at the bottom of Figure [ Usually, the deeper a GO term is located, the more concrete definition the GO term has, e.g., the small molecule biosynthetic process is the child node of the biosynthetic process. We notice that the number of gene annotations involved with leaf GO terms is a relatively small percentage in all three types of ontologies. We think there are two possibilities: (1) The complexity of annotations of some concrete GO terms, e.g., the identification of whether protein is involved in the hexose biosynthetic process. (2) The relatively small number of proteins in nature involved with some specific GO terms intrinsic to these GO terms.
We further pre-process the ProteinKG25 dataset as follows. We observe that the relations of Protein- Go have long-tailed distribution, and mostly focused on involved in, part of and enables. Such data distribution will seriously affect the protein feature embedding during pre-training, so we pre- process our dataset add more precise and fine-grained relations. Figure ( illlustrases the relation distribution of the original model and the distribution after being pre-poccessed. We search for GO terms whose frequency of occurrence in ProteinKG25 is the top 10 in MF and CC, top 20 in BP, then we form a new type Protein2GO relation by their corresponding relationship such as parts of cytoplasm.
A.2 DOWNSTREAM TASK DEFINITION
We list the detailed definition of downstream tasks and its corresponding or similar tasks in nature language processing.
Secondary Structure Prediction is token-level task and similar NER (Name Entity Recognition). Each token (amino acid) dq is mapped to a label E (Helia, Strand, Other).
Contact Prediction is token-level matching task. Each token (amino acid) pair x; Ty of sequence (protein) x is mapped to a label Vij E (0, 1}.
Remote Homology Detection a sequence-level classification task. Each input sequence (protein) is mapped to a label E {1,..., 1195) which represents different possible protein folds.
Fluorescence Landscape Prediction and Stability Landscape Prediction are regression tasks where each sequence (protein) x is mapped to a label E R.
Protein Protein Interface is a sequence-level matching task. Each sequence (protein) pair
Iy . &y mapped label yiy {0. 1}.
Protein Function Prediction is sequence-leve classification task knowledge graph completion task to prediction link of a protein to the Gene Ontology.
A.3 EXPERIMENTAL SETTINGS
This section details the training procedures and hyperparameters for each of the datasets. We utilize Pytorch (Paszke al. 2019)) conduct experiments with Nvidia V100 GPUs. In pre-training of OntoProtein, similar to Elnaggar et al. (2020), we use the same training protocol such as optimizer, learning rate schedule on BERT model. We set y 12.0 and the number of negative sampling to 128 in Equation]
A.4 DATAFLOW OF ONTOPROTEIN
We use batches (protein-seq, protein-go, go-go) to jointly train the model, which is shown in Figure
16

---

Published as a conference paper at ICLR 2022
1.75
1.25
Holh ALF urreaar Spae
If att Lon
can and use 2 Lesesrii sox
cii as see. steeid uepeod te La
ter aere dhe Pelioo
cumn aill It te
cume cotin ad senons
coun do chay Bad uf phepps
aour soo que ans "ooaaaaa"" SUStION
2013
is oo ooooooo
perio maceuss
620032
520032
300000
200000
120002
Riliit dftee SalAe
an Peoee 14
18t Sinoor WONEO a Dirpou
RilOE Ai
LIP u parpon
Figure Top: Initial relation distribution. Bottom: Pre- precrrcessed relation distribution.
training set
test set
Apr, 2020
Aug, 2020
Apr, 2021
time
validation set
Figure 6: The timeline of the three Gene Annotation datasets. To generate pre-training set and evaluation set of protein function prediction, we choose three Gene Annotation datasets in different periods.
17

---

Published as a conference paper at ICLR 2022
Entity Type
Relation Type
Molecular function Cellular component
enables, contributes_to located part_of. is is_active_in colocalizes.with
pcts_uustreamm_ff... -fttaooooooooiiinnnnn involved in, acts www..uustteam... of,
Biological process
acts wwwtuuamsam...... oo wwwstooeeeeeiiieeeeeeie effect. acts crtssuustream.c _ff, negative effeect acts uissuuptteamm or within pwwptooeeseieiieeeee effect, ccts_upstreem_oof.oo www..o/wwwiiii.nggiiieeeeeeeee _effect
Table 5: Entity categories of GO terms and specific relation types of these entity categories.
Task
epoch
batch,size
warmup_ratio
learning.rate
frozen.bert
optimizer
ss3
5
32
0,08
3e-5
False
Adam
ss8
5
32
0.08
3e-5
False
Adam
stability
0.08
3e-5
False
Adan
fluorescence;
25
3c-5
Truc
Adam
remote homology
10
64
0,08
3c-5
False
Adam W
contact
10
0.08
3e-5
False
Adam W
Table 6: Hyper-parameters for the downstream task.
Dataloader
ProteinKG25 Database
enooder
Knowledge Semantic Space
GO-GO
Protein
Protein-GO
Protein Sequence Database
Protein encoder
Mask Language Model
Protein Sequences
Figure 7: The dataflow of OntoProtein.
TERM
DESCRIPTION
GO annotation
A statement about the function of a particular gene A standard vocabulary term for biological function annotation A specific definition of GO term
GO term
GO statement
Table 7: Terms descriptions in Gene Ontology. GO annotations are created by associating gene or gene product with GO term. Four pieces of information uniquely identify a GO annotation: Gene Product, Go term, Reference, Evidence. And each Go term contains description text, and this is Go statement.
18


Reviews:
[{"review_id": "eJ3dPzEze_m", "reviewer": "Reviewer_fXpi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept: \n* Interdisciplinary study on extending the language model and KG embedding into protein embedding to empower bio appications\n* Insightful strategy and observation on KG negative sampling \n* Extensive experiments on multiple tasks and ablation study (attention analysis) are conducted. New useful resources of enhanced protein datasets are created and available publicly with strong reproducibility. \n\nReasons to reject: \n* The information fusion of text token and protein sequence tokens may face modality challenges in one KG embedding model. \n* Some missing references and potential baselines on the PPI task and other works that jointly trained on protein and gene ontology (see more in detailed comments)\n* Lack of quality assessment of curated dataset ProteinKG25. \n\nDetailed comments: \n\n* One of the major concerns of this paper is that some existing works with joint training on protein with other domains. Examples are [1,2,3] with PPI predictions as one task for evaluation (can be added to PPI baselines). As for PPI baselines, since the sequences have been added as input, the current evaluation only considered GNN-PPI variations, however, there is a large collection of existing work relying on protein sequences (potentially with the help of PSI-BLAST and multiple sequence alignment) and follow-up works of PIPR. It is necessary to show that the purely sequence-based model cannot compete with the proposed OntoProtein with an additional facet of GO knowledge.\n* In OntoProtein, the GO entity embeddings are based on text description while proteins are encoded with their amino acid sequence tokens through a shared model in MPM. The features from different modalities are jointly optimized together through KE loss with TransE. Two features are computed in the same embedding space with a simple translational distance which does not seem to make much sense, especially there are two types of triples (Protein-GO and GO-GO) trained together. This requires more justification on what the embeddings aim to learn. \n* It seems that the performance of OntoProtein does not significantly outperform other baselines in most cases on the TAPE benchmark. What is the benefit of OntoProtein compared with other models (especially ProtBERT)? \n*  It is interesting to see how different aspects can be leveraged into better protein presentations. Since the paper claims that the protein representation learning benefits from gene ontology, and GO can be divided into BP/MF/CO components, it would be interesting to see how individual subsets in GO can result in different improvements. Also, it is said that gene ontology (three subsets) and PPI are not independent of each other (some protein functions are annotated due to their known binding activities with other proteins). It is suggested to see whether such correlations can be identified in the current training framework or the OntoProtein can help solve the incompleteness. \n* There is no quality assessment of the newly curated dataset ProteinKG25. The understanding is that the authors successfully made one collection aligned with external resources of annotations to generate the protein-centric KG. But it is unclear about the quality (accuracy and completeness). It is also suggested to make a comparison of other benchmarks such as HetioNet[4] and DRKG[5] in terms of the detailed information or KG schema.\n* No sensitivity study on the hyperparameter \\alpha in pre-training objective which balances MLM and KE losses.\n\nMinor suggestions and corrections:\n* Figure 1 is somewhat confusing about the nodes and edges, especially the left subfigure and the number annotations. It would be beneficial to layout one snapshot of created Protein\n* The acronym “MLM” (masked language model) was used first in the introduction without full name (Page 2 line 10).\n\nReferences: \n* [1] Onto2Vec: Smaili, F. Z., Gao, X., & Hoehndorf, R. (2018). Onto2vec: joint vector-based representation of biological entities and their ontology-based annotations. Bioinformatics, 34(13), i52-i60.\n* [2] OPA2Vec: Smaili, F. Z., Gao, X., & Hoehndorf, R. (2019). Opa2vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction. Bioinformatics, 35(12), 2133-2140.\n* [3] Bio-JOIE: Hao, J., Ju, C. J. T., Chen, M., Sun, Y., Zaniolo, C., & Wang, W. (2020, September). Bio-JOIE: Joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (pp. 1-10).\n* [4] HetioNet: Himmelstein, D. S., & Baranzini, S. E. (2015). Heterogeneous network edge prediction: a data integration approach to prioritize disease-associated genes. PLoS computational biology, 11(7), e1004259.\n* [5] DRKG: Ioannidis, Vassilis N., Song, Xiang, Manchanda, Saurav, Li, Mufei, Pan, Xiaoqin, Zheng, Da, Ning, Xia, Zeng, Xiangxiang & Karypis, George. (2020) Drug Repurposing Knowledge Graph for Covid-19. https://github.com/gnn4dr/DRKG/\n", "summary_of_the_paper": "This paper introduces OntoProtein a comprehensive pre-training framework for protein embedding with the knowledge of gene ontology (GO). More specifically, OntoProtein jointly optimizes on both masked Protein Model and Knowledge Graph Embedding model which results in knowledge-aware protein embedding for downstream applications including protein-protein interactions and protein GO association prediction. The authors also create a new benchmark of proteins with aligned annotations to facilitate the OntoProtein training.", "summary_of_the_review": "This paper has merits in multiple directions. It applies the state-of-the-art language modeling in NLP in protein representation learning named OntoProtein. Also, OntoProtein help creates a new augmented protein benchmark with aligned GO annotations (enhanced views for heterogenous BioKG with multiple domains). The knowledge-aware negative sampling strategy has underlying insights on utilizing the hierarchically structured gene ontology which can be generalized onto KG negative sampling with ontology information, not limited in bio domain. The critics of this work are mostly about some missing references and baseline approaches, as well as the limitation on combinations of existing works.  Overall, the reviewer agrees that the merits of this work outweigh the flaws. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M11kX3Tw5Ae", "reviewer": "Reviewer_GdWW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The research direction is important and well motivated -- a lot of valuable information is captured in gene ontologies and ought to be valuable to improve protein representations and performance on downstream task.\n2. The paper reads very nicely and easy to follow through, except for a few subsections to clarify / re-write (in particular section 2.4, see below) and would benefit for another read through to correct typos.\n3. The method is applied and tested across a diverse set of experiments.\n\n**Weaknesses**\n1. Methodological contributions are limited to leveraging existing multi-relational data embedding methods (Bordes et al.) with no adaptation to the specificities of the GO modality. The negative sampling is very close to random sampling (with constraints based on entity group type).\n2. This translates into marginal benefits across all experimental setups when comparing with the ProtBert baseline (acknowledged by authors in section 3.4). Some of the claims in section 3.2 do not seem backed up by the results. For example, in the PFP experiments, you cannot simultaneously claim that your method outperforms baselines in BPO and that it performs comparably with other methods in other settings (since baselines outperforms OntoProtein in the latter settings by a larger margin that Ontoprotein outperforms them in the BPO setting).\n\n**Clarifying questions**\n1. Section 3.1: “Specifically, we notice that the frequency of leaf GO terms involved in gene annotations and those more specific concepts relative to their parent nodes are less to non-leaf GO terms” -- could you please clarify what you meant here?\n2. Section 2.4. Equation 1: you seem to have a typo in equation 1 since it includes r’ but negative triplets involve the same r (r=r’?) since you only negatively sample the leading/tail entities h’ and t’.\n3. Section 2.4. Equation 2: why is d not an explicit function of t? Shouldn’t it be d(h+r, t) following Bordes et al?\n4. Section 2.4. Equation 3 is confusing. The ‘|’ operator is not standard for sets / not defined anywhere (did you mean union?). Additionally, you state that the negative sampling is achieved by sampling h’ and t’ at random from the same family as h and t resp. -- which I understand means for example that if h is E_{MFO} then also h’ is in E_{MFO} -- but that is not what is expressed mathematically by equation (3). Could you please clarify?\n5. Section 2.4: please clarify which embeddings you use exactly here? Are you using aggregate embeddings H_{GO} and H_{Protein} and none of the token level embeddings?\n6. Section 2.4: Do you do any negative sampling for T_{protein--GO} triplets? Are you limiting to negatively sampling tail (GO) entities?\n7. Section 2.5: How do you form input batches to jointly train this objective? \n8. Figure 4: How is it different from what one would obtain with ProtBert?\n\n**Minor points**\n1. Introduction: \"different from knowledge-enhanced approaches in NLP\" → please cite\n2. Introduction: “text deceptions” (towards the top of page 2) -- did you mean \"test descriptions\"? \n3. Would suggest that you give 1 or 2 concrete examples to illustrate the different objects described in 2.2 and appendix A.1. Table 7 is also a bit confusing (e.g. difference between GO term Vs GO statement?)\n4. The write up of section 2 could be improved -- there was a lot of overlap between section 2.1., 2.2 and 2.3 which was making some information redundant / making things more difficult to read\n5. Section 2: would suggest to include an overall architecture diagram to show how an example input triplet is processed (eg., what goes to which encoder type?).\n6. Section 3: bolding of results is not consistent / misleading. Always bold the best result in each column.\n7. Section 3: would be helpful to have MSA transformer everywhere to put things in perspective (bearing in mind it does have access to MSA data that you do not explicitly leverage).\n", "summary_of_the_paper": "This paper introduces a method to enrich the representations that are learnt by protein language models with knowledge encapsulated in gene ontologies. To do so, it curates a knowledge graph (ProteinKG25) and applies existing methods in multi-relational data embedding (Bordes et al.) to jointly train knowledge embeddings and protein embeddings, with the objective to enhance the latter and subsequently improve the performance on various downstream tasks (namely TAPE benchmark, protein-protein interaction and protein function prediction).", "summary_of_the_review": "The research direction of this paper is very important and that the work described here is adequately framed and motivated. A valuable contribution is the ProteinKG25 knowledge graph put together by authors, which could be valuable to other researchers. While there are several points to clear out (see main review), the ideas are overall well presented and the paper reads nicely. \nHowever, the methodological and empirical contributions of this work are somewhat limited: the Knowledge Embedding for the knowledge graph (while applied for the first time to GO data) is largely borrowed from Bordes et al. and not particularly tailored to this very specific data modality; the negative sampling is only marginally different from random sampling (constraining on entity group). To the author’s own account “the gains in our proposed OntoProtein compared to previous pre-trained models using large-scale corpus is still relatively small” since “the knowledge graph ProteinKG25 can only cover a small subset of all proteins, thus, limiting the advancement”. In my view, this is precisely the main challenge to resolve for this type of work attempting to instill established knowledge into large protein language models, which would have made this work a very meaningful contribution.\nAll things considered, I’m leaning weak reject given the minor methodological contributions and weak experimental results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JqjUOYwn1DV", "reviewer": "Reviewer_XNXb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good paper but more efforts are needed.", "summary_of_the_paper": "This paper argues that informative biology knowledge in KGs can enhance protein representation with external knowledge. To show this, the author proposes a computational framework named OntoProtein that makes use of structure in Gene Ontology into protein pre-training models. This paper conducts various experiments to show the superiority of the proposed method and the benefits of the information of Gene Ontology.", "summary_of_the_review": "\npros:\n1. In computational biology, the prevailing approaches of learning protein representation rarely consider\nincorporating knowledge graphs (KGs), which can provide rich structured knowledge facts. This paper seems to be the first to impose the knowledge graphs of protein.\n\n2. This paper provides a meaningful architecture for simultaneously learning protein knowledge and embedding. \n\n3. To demonstrate the effectiveness of the proposed method, the author constructs a large-scale KG dataset, promoting the research on protein language pre-training.\n\nCons:\n \n1. To make the paper more friendly for the general ML/DL researchers, I think it should be made clearer the definition of the downstream tasks used in this paper.  \n\n2. This paper usually considers one or two baselines in the PPI task, which is are not enough and convincing. It is possible to compare the proposed method with [1] and [2]? If not, please provide the reasons.\n\n3. The presentation of this paper should be further improved. For example, Figures 1 and 4 are hard to read.\n\n[1] Patel, S., Tripathi, R., Kumari, V., & Varadwaj, P. (2017). DeepInteract: deep neural network based protein-protein interaction prediction tool. Current Bioinformatics, 12(6), 551-557.\n\n[2] Liu, X., Luo, Y., Li, P., Song, S., & Peng, J. (2021). Deep geometric representations for modeling effects of mutations on protein-protein binding affinity. PLoS computational biology, 17(8), e1009284.\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduce a protein pretraining framework that enhances representations learnt from protein language modeling with knowledge graph embeddings. The new framework, OntoProtein, optimizes jointly a masked Protein objective and a Knowledge Graph Embedding objective producing knowledge-aware protein embeddings. These embeddings are evaluated on downstream tasks including protein-protein interactions and protein GO association prediction. The paper also introduces a new large-scale KG dataset, ProteinKG25.

The reviewers were in agreement that the paper presents an important research direction and that the work is well framed and motivated. The dataset contribution was also considered important by the reviewers and they are in agreement that the paper is clear and generally easy to understand. Reviewers were concerned that the novelty of the work is in the application of existing techniques to a new domain rather than introducing new domains, but generally the reviewers considered the novelty to be sufficient for publication. There were some other concerns about missing references and some of the presentation, but the authors addressed these concerns in their response and in the updated version that they produced.
Decision: Accept (Poster)