Protein Representation Learning via Knowledge Enhanced Primary Structure Reasoning
Authors: Hong-Yu Zhou
Yunxiang Fu
Zhicheng Zhang
Bian Cheng
Yizhou Yu
Venue: ICLR 2023 poster (2023)
Keywords: Protein Science
Representation Learning
Knowledge Graph

Abstract: Protein representation learning has primarily benefited from the remarkable development of language models (LMs). Accordingly, pre-trained protein models also suffer from a problem in LMs: a lack of factual knowledge. The recent solution models the relationships between protein and associated knowledge terms as the knowledge encoding objective. However, it fails to explore the relationships at a more granular level, i.e., the token level. To mitigate this, we propose Knowledge-exploited Auto-encoder for Protein (KeAP), which performs token-level knowledge graph exploration for protein representation learning. In practice, non-masked amino acids iteratively query the associated knowledge tokens to extract and integrate helpful information for restoring masked amino acids via attention. We show that KeAP can consistently outperform the previous counterpart on 9 representative downstream applications, sometimes surpassing it by large margins. These results suggest that KeAP provides an alternative yet effective way to perform knowledge enhanced protein representation learning.

Published as a conference paper at ICLR 2023
PROTEIN REPRESENTATION LEARNING VIA KNOWL- EDGE ENHANCED PRIMARY STRUCTURE MODELING
Hong- Yu Zhou Yunxiang F Zhicheng Zhang
Cheng Bian'
Yizhou Yu
1 Departmeett of Computer Science, The University of Hong Kong
2 Xiaohe Healthcare, ByteDance JJncsiTTcch 4OPO HealthL.ab
com, yunxiange connect. w...hkuh.. .k, zhangzhichengL 3mmails...........mllmmmmlmlllm ucas. edu. en, bianl maain19922222gmaaii. com, yi vizhouy@acm. org
whuzhouhongyu@gman
ABSTRACT
Protein representation learning has primarily benefited from the remarkable de- velopment of language models (LMs). Accordingly, pre-trained protein models also suffer from a problem in LMs: a lack of factual knowledge. The recent so- lution models the relationships between protein and associated knowledge terms as the knowledge encoding objective. However, it fails to explore the relation- ships at more granular level, i.e., the token level. To mitigate this, we propose Knoxledge-exploieed Auto-encoder for Protein (KeAP), which performs token- level knowledge graph exploration for protein representation learning. In prac- tice, non-masked amino acids iteratively query the associated knowledge tokens to extract and integrate helpful information for restoring masked amino acids via attention. We show that KeAP can consistently outperform the previous coun- terpart on representative downstream applications, sometimes surpassing it by large margins. These results suggest that KeAP provides an alternative yet effec- tive way to perform knowledge enhanced protein representation learning. Code and models are available at https: //t:t//gitii thub. com BAImm/RLMMM/KSK KeAP
INTRODUCTION
The unprecedented success of AlphaFold (Jumper et al.] 2021 Senior et al. 2020) has sparked the public's interest in artificial intelligence-base protein science, which in turn promotes scientists to develop more powerful deep neural networks for protein. At present, major challenge faced by researchers is how learn generalized representation from vast amount of protein data. An anal- ogous problem also exists in natural language processing (NLP), while the recent development of big language models (Devlin al. 2018 Brown et al., 2020) offers a viable solution: unsupervised pre-training with self-supervision. In practice, by viewing amino acids as language tokens, we can easily transfer existing unsupervised pre-training techniques from NLP to protein, and the effective-
ness of these techniques has been verified in protein representation learning Rao et al. (2019) Alley
ctt al. (2019); Elnaggar et al. (202I): Unsal et al. (2022).
However, as pointed out by (Peters et al. 2019 Zhang al. 2019 Sun et al. 2020 Wang et al. 2021], pre-trained language models often suffer from a lack of factual knowledge. To alleviate sim- ilar problems appearing in protein models, Zhang et al. (2022) proposed OntoProtein that explicitly injects factual biological knowledge into the pre-trained model, leading to observable improvements on several downstream protein analysis tasks, such as amino acid contact prediction and protein- protein interaction identification.
In practice, OntoProtein leverages the masked language modeling (MLM) (Devlin al. 2018) and TransE [Bordes et al. 2013) objectives to perform structure and knowledge encoding, respectively. Specifically, the TransE objective is applied to triplets from knowledge graphs, where each triplet can be formalized as (Protein, Relation, Attribute). The relation and attribute terms described us- ing natural language are from the gene ontologies (Ashburner al. (000)) associated with protein.
First two authors contributed equally. Yunxiang's 's work was done at ByteDance
1 Cooresponning author.

---

Published as a conference paper at ICLR 2023
However, OntoProtein only models the relationships on top of the contextual representations of pro- tein (averaging amino acid representations) and textual knowledge (averaging word representations), preventing it from exploring knowledge graphs at more granular level, i..., the token level.
We propose KeAP (Knowledge-exploite) Auto-encoder for Protein) to perform knowledge enhanced protein representation learning. To address the granularity issue of OntoProtein, KeAP performs token-level protein-knowledge exploration using the cross-attention mechanism. Specifically, each amino acid itera- tively queries each word from relation and attribute terms to extract useful, relevant information using QKV Attention (Vaswani et al. 2017). The extracted information is then integrated into the protein representation via residual learning (He et al. 2016). The train- ing process is guided only by the MLM objective, while OntoPro- tein uses contrastive learning and masked modeling simultaneously. Moreover, we propose to explore the knowledge in a cascaded man- ner by first extracting information from relation terms and then from attribute terms, which performs more effective knowledge encod- ing.
Exls To bakh
Figure I: Transfer learning performance of ProtBert, OntoProtein, and our KeAP on downstream protein analysis tasks, S-, M-, and L- Contact stand for short-range, medium- raddum--aane,, and long- range contact prediction. PPI denotes the protein-protein interaction prediction. means the model is trained with the full ProteinKG25.
KeAP has two advantages over OntoProtein (Zhang et al. 2022). First, KeAP explores knowledge graphs a more granular ar level by applying cross-attention to sequences of amino acids and words from relation and attributes. Second, KeAP provides a neat solution for knowledge enhanced protein pre-training. The encoder-decoder architecture in KeAP can be trained using the MLM objective only (both contrastive loss and MLM are used in OntoProtein), making the whole framework easy to optimize and implement.
Experimental results verify the performance superiority of KeAP over OntoProtein. In Fig. [ we fine-tune the pre-trained protein models 9 downstream applications. We see that KeAP outperforms OntoProtein all 9 tasks mostly by obvious margins, such as amino acid contact prediction and protein-prote interaction (PPI) identification. Compared to ProtBert, KeAP achieves better results on 8 tasks while performing comparably on protein stability prediction. In contrast, OntoProtein produces unsatisfactory results on homology, stability, and binding affinity prediction.
2 RELATED WORKS
2.1 REPRESENTATION LEARNING FOR PROTEIN
How to learn generalized protein representation has recently become a hot topic in protein science, inspired by the widespread of representation learning in language models (Devlin et al., 2018
Radford et al. 2019 Yang et al.) 2019; Sarzynska- Wawer et al. 2021). [epler & Berger (0188)
introduced a multi-task protein representation learning framework, which obtains supervision sig- nals from protein-protein structural similarity and individual amino acid contact maps. Due to a
plethora of uncharacterized protein data, self-supervised pre-training (Alley al., 2019 Rao al. 2019) was proposed to directly learn representation from chains of amino acids, where tremendous and significant efforts were made to improve the pre-training result by scaling up the size of the
model and dataset (Elnaggar et al.] 2021 Rives et al.) 2021 Vig et al. 2020 Rao et al. 2020) Yang al. 2022 Nijkamp al. 2022] Ferruz et al.] [022 Chen et al. 2022) ). contrast, protein-related factual knowledge, providing abundant descriptive information for protein, has been long ignored and largely unexploited, OntoProtein (Zhang et al. 2022) first showed that we can improve the per- formance of pre-trained models downstream tasks by explicitly injecting the factual biological knowledge associated with protein sequences into pre-training.
In practice, OntoProtein proposed to reconstruct masked amino acids while minimizing the embed- ding distance between contextual representations protein and associated knowledge terms. One potential pitfall of this operation is that it fails explore the relationships between protein and
2

---

Published as a conference paper at ICLR 2023
knowledge granular level, ie., the token level. In comparison, KeAP overcomes this limitation by performing token-level protein-knond -nowledge exploration via cross-attention modules.
2.2 KNOWLEEDGE ENHANCED PRE-TRAINED LANGUAGE MODELS
Knowledge integration has been treated as reliable way to improve modern language mod- els (Zhang ct al. 2019; Sun et al. 2020 Liu et al. 2019; Vulié et al.) 2020; Petroni et al. 2019
Roberts al. 2020 Wang et al. 2021 Yao et al.) 2019
He et al. 2020 Qin al. 2021 Madani et al. 2020 Chen et al. 2023 Zhou et al. (022). Xie et al. (201)) proposed to per-
et al. 2020
form end-to-end representation learning on triplets extracted from knowledge graphs, while Wang
et al. (021) further bridged the gap between knowledge graphs and pre-trained language models by treating entity descriptions as entity embeddings and jointly training the knowledge encoding (i.e.,
TransE (Bordes et al. 2013) and MLM objectives.
Motivated by (Wang al., 2021). OntoProtein (Zhang et al. 2022) used the MLM and TransE (Bor- des et al. 2013) as two training objectives when learning protein representation on knowledge graphs. However, OntoProtein fails to explore the knowledge graphs at a more granular level, i.e., the token level. In contrast, KeAP performs token-level mootein-knowledee exploration via the at- tention mechanism and provides a neat solution for knowledge enhanced protein pre-training by training encoder-decoder architecture using only the MLM objective.
METHODOLOGIES
As shown in Fig. 2 for each triplet (Protein, Relation, Attribute) in the knowledge graphs, we apply random masking to the amino acid sequence while treating the relation and attribute terms as the associated factual knowledge. After feature extraction, representations of each triplet are sent to the protein decoder for reconstructing missing amino acids. The decoder model comprises N stacked protein-knowledge exploration (PiK) blocks. In each block, the amino acid representation iteratively queries, extracts, and integrates helpful, relevant information from word representations of associated relation and attribute terms in cascaded manner. The resulting representation is used to restore the masked amino acids, guided by the MLM objective. After pre-training, the protein encoder be transferred various downstream tasks.
3.1 PRELIMINARY: PROTEIN AND BIOLOGICAL KNOWLEDGE
KeAP is trained on a knowledge graph that consists of about five million triplets from Pro- t25 (Zhang et al., 2022). Each triplet is in the format of (Protein, Relation, Attribute). Protein can be viewed as sequence of amino acids, while both Relation and Auribble are factual knowl- edge terms (denoted gene ontologies in Zhang al. (2022) described using natural language. Specifically, Relation and Attribute provide knowledge in biology that is associated with Protein, such molecular function, biological process, and cellular components.
During the training stage, each protein is passed to the protein encoder, resulting in the protein representation fo E IR. xD The superscript 0 is the layer index. Lp denotes the length of the amino acid sequence. D stands for the feature dimension. In practice, the protein encoder has BERT-like architecture (Devlin al.||20 2018). Similarly, we forward the associated knowledge terms to the language encoder to obtain knowledge representations, i.e., fr E LR and fa E DD T, and T. denote the lengths of the relation and attribute terms, respectively. The reason for using embeddings for protein and biological
two encoders is that would like to extract knowledge.
3.2 LEVEN PROTEIN- -EOEENNNK EXPLORATION
KeAP uses surrogate task to perform knowledge enhanced pre-training, i.e., exploring knowledge graphs for protein primary structure modeling. In this way, KeAP asks the protein representation to treat each amino
be aware of what knowledge is helpful masked protein modeling, In practice, acid as a query, while the words from associated relation and attribute terms are attended to as keys and values in order. Taking the i-th layer example, the inputs the protein decoder include
3

---

Published as a conference paper at ICLR 2023
Protein decoder
Triplet from KG
Protein Soans
MIDYLLE IRVLFDP EFSSEM..
Protein encoder
Norm
Nor
i-th block
CICY Att.
GKV ALL
MLM
Rolation
enables RNA bind inn
MLP
Language encoder
Norm
Atriboos
Binding malecule or portion
N PIK
Figure 2: Overview. Given triplet (Protein, Relation, Attribute) from the knowledge graph, KeAP randomly masks the imput amino acid sequence and treats the relation and attribute terms as as- sociated factual knowledge. Then, the triplet is passed to encoders and the output representations are regarded as the protein decoder's inputs. The decoder asks each amino acid representation to iteratively query words from the associated knowledge terms in cascaded manner, extracting and integrating helpful, relevant information for restoring masked amino acids. PiK and MLM stand for the mooteinkknowlede exploration block and masked language modeling, respectively.
fp. fr, and fa. The relation representation fr is firstly queried by fg as the key and value:
Up K V, No Norm(f-)Wi, Norm(f.)
where W Wk. and Wy are learnable matrices. Norm stands for the layer normalization (Ba et al. 2016).
Then, QKV Attention (QKV-A) (Vawwani et al. 2017) is applied to (o) K V,',, where represen- tations of amino acids extract helptul, relevant knowledge from the latent word embeddings of the associated relation term. The obtained knowledge representation sy. stores the helpful information for restoring missing amino acids. We then add up st and f/ to integrate knowledge, resulting in the representation fl-
5 QKV-A(Q), K Vi),
f Norm + S
Next, we use f to query the attribute term. The whole query, extraction, and integration process similar to that of the relation term:
Q K3 a: V = Norm W Norm(Ja)Wk Norm(fa)
8 QKV-A(Q,, K Vo),
f: Nfrm(f)) + sy.
The resulting representation L integrates the helpful, relevant biological knowledge that benefits the restoration of missing amino acids. We finally forward Fü through a residual multi-layer perceptron (MLP) to obtain the output representation of the i-th PiK block, which also serves as the input to the i+1-th block.
3.3 MASKED LANGUAGE MODELING OBJECTIVE
For each input protein, randomly mask 20% amino acids in the sequence. Moreover, each masked amino acid has an 80% chance of being masked for prediction, a 10% chance of being replaced by random amino acid, and 10% chance of remaining unchanged. Suppose the number of masked amino acids M and denotes the j-th amino acid. The training objective LMLM to be minimized follows:
M
LMLM = - log
Σ ==
P (az
IN Og. θp)
(4)
4

---

Published as a conference paper at ICLR 2023
Øg and D denote the parameters of the protein encoder and decoder, respectively. We initialize the language encoder using PubMedBERT (Gu et al. 2021 and do not update it in the training phase.
EXPERIMENTS AND ANALYSES
In this section, we extensively evaluate the generalization ability of the learned protein represen- tation by fine-tuning the pre-trained model wide range of downstream applications, including amino acid contact prediction, protein homology detection, protein stability prediction, protein- protein interaction identification, protein-protein binding affinity prediction, and semantic similarity inference. Besides, we also provide ablations and failure analyses to facilitate the understanding of KeAP. Unless otherwise specified, follow the pre-training and fine-tuning protocols used by OntoProtein (refer to the appendix for more details), such as training strategies and dataset split. The pre-trained models of ProtBert (Elnaggar al.) 202T) OntoProtein (Zhang et al. 2022). and our KeAP share the same number of network parameters. Average results are reported over three independent training runs.
4.1 DATASET FOR PRE-TRAINISG
Gr0 (Zhang et al. 2022) provides knowledge graph that consists of approximately five million triplets, with nearly 600k protein, 50k attribute terms, and relation terms included. The attribute and relation terms described using natural language are extracted from Gene Ontology which is the world's largest source of information on the functions of genes and gene products (c.g., protein).
is worth noting that we pre-train KeAP and report its experimental results in two different settings where the pre-training datasets differ. In the first setting, remove 22,978 amino acid sequences that appear in downstream tasks. Accordingly, 396,993 triplets are removed from PooteinKG25, and the size of the new dataset is 91.87% of the original. In the second setting (denoted with ?), the same pre- pre-training data as in OntoProtein (Zhang et al. 2022)
4.2 AMINO ACID CONTACT PREDICTION
6 S seq < 12
12 S seq <24
24 S seq
Methods
P@L
P@ L/2
P@L/5
P@L
P@L/2
P@L/5
P@L
P@L/2
P@ L/5
LSTM
0.36
0.49
0.26
0.34
0.20
0.23
0.27
ResNet
0.25
0.34
0.46
0.28
0.25
0.35
0.10
0.13
0.17
Transformer
0.28
0.35
0.46
0.19
0.25
0.33
0.17
0.20
0.24
ProtBert
0.30
0.40
0.52
0.27
0.35
0.47
0.20
0.26
0.34
ESM-1h
0.38
0.48
0.62
0.33
0.43
0.56
0.26
0.34
0.45
KeAP
0.41
0.51
0.63
0.36
0.45
0.54
0.28
0.35
0.43
OntoProtein
0.37
0.46
0.57
0.32
0.40
0.50
0.24
0.31
0.39
KeAP
0.41
0.52
0.62
0.36
0.46
0.57
0.29
0.37
0.46
Table 1: Comparisons amino acid contact prediction. seq indicates the distance (i.e., the number of amino acids) between two selected amino acids. P@L. P@ I./2, P@ L/S denote the precision scores calculated upon top L (i.e., I. most likely contacts), top 1.22, and top 1/5 predictions, spectively, In each setting, the best results are bolded. Models with i are pre-trained using the full Pr0teinKG25.
Overview. Given an input protein molecule that comprises a chain of amino acids, the protein model is asked to predict whether any two amino acids (from the same sequence) are in contact or not. To achieve this goal, our model outputs probability contact matrix for each input protein, where the row and column numbers correspond to the indices of two amino acids. We performed experiments on the dataset collected and organized by (AlQuraishi 2019 Rao et 2019). The evaluation metric is precision.
http::/ http:///gementtallgy.oor
5

---

Published as a conference paper at ICLR 2023
Baselines. Following (Zhang al. 2022). included six protein analysis models baselines. Specifically, we used variants of LSTM (Hochreiter Schmidhuber 1997). ResNet (He et al. 2016), and Transformer /Vaswani al. 2017) proposed by the TAPE benchmark (Rao al. 2019)) ProtBert (Elnggaar et al. (021) is a 30-layer BERT-like model pre-trained on (ni (Suzek al.] 2007| 2013). ESM-1b
the UniRe150 (Suzek et al. 2007 2015) representative sequences. OntoProtein Zhang et al. (2022)
33-layer transformer pre-trained UR50/S, which
the most recent knowledge-baseer pre-training methodology.
Results. Table T presents the experimental results of amino acid contact prediction. In the first setting, we see that ProtBert and ESM-Ib lb are two most competitive baselines. Specifically, KeAP outperforms ProtBert by large margins in short- (6 S seq < 12), medium- (12 S seq < 24), and long-range (seq 2 24) contact predictions. Compared to ESM-Ib, KeAP is more advantageous in short-range prediction while achieving competitive results in medium- and long-range prediction tasks. In the second setting, we see that KeAP consistently surpasses OntoProtein regardless of the the distance between two amino acids. Particularly noteworthy is the fact that KeAP surpasses On- toProtein by large margins. Considering that OntoProtein also performs knowledge encoding, the clear performance advantages (6%) over OntoProtein demonstrate that KeAP provides more com- petitive choice for knowledge enhanced protein representation learning. We believe the performance gains brought by KeAP can be attributed the proposed token-level knowledge graph exploration methodology. This enables the pre-trained model to understand the knowledge context better, pro- ducing a better-comtextualized protein representation for contact prediction than directly applying meaning pooling.
4.3 HOMOLOGY DETECTION AND STABILITY PREDICTION
Overview of homology detection. Predicting the remote homology of protein can be formalized as a molecule-Ievel classification task. Given a protein molecule as the input, we ask the homology detec- tion model to predict the right protein fold type. In our case, there are 1,195 different types of protein folds, making it quite challenging task. The data are from Hou et al. (2018) and we report average accuracy the fold-level heldout set.
Methods
Homology
Stability
LSTM
0.26
0.69
ResNet
0.17
0.73
Transformer
0.21
0.73
ProtBert
0.29
0.82
ProteinBert
0.22
0.76
ESM-1b
0.11
0.77
KeAP
0.29
0.82
Overview of stability prediction. In this regres-
OntoProtein
0.24
0.75
sion task, we aim predict the intrinsic stability of the protein molecule, which measures the protein's ability to maintain its fold under extreme condi- tions. In practice, high-accuracy stability predic- tion can benefit drug discovery, making it easier to construct stable protein molecules. We evaluate the model performance by calculating Spearman's 's rank correlation scores on the whole test set Rocklin et al. 2017
KeAPt
0.30
0.82
Table Comparisons on protein homology detection and stability prediction. In each set- ting, the best results are bolded. Models with are pre-trained using the full Pr25
Baselines. Besides six approaches presented in Table 1 we add one more baseline Protein- Bert (Brandes et al. 2022). which pre-trains the protein model to restore missing amino acids and associated attribute annotations simultaneously.
Results. From Table 2 we see that the knowledge-based pre-training methodologies. i.e., Protein- Bert and OntoProtein, fail to display favorable results on both tasks. ZZhang et al.) (2022) claimed that the failure is due to the lack of sequence-level objectives in pre-training. In contrast, performs on par with ProtBert on homology detection and protein stability prediction. We believe the success of KeAP can be partly attributed to the token-level knowledge graph exploration.
KeAP
4.4 PROTTENN INTERACTION IDENTIFICATION
Overview. Protein-protein interactions (PPI) refer to the physical contacts between more amino acid sequences. this paper, we only study the two-protein cases, where a pair of protein molecules serve as the inputs. The goal is to predict the interaction type(s) of each protein pair. There

---

Published as a conference paper at ICLR 2023
SHS27K
SHS148K
STRING
Methods
BFS
DFS
Avg
BFS
DFS
Avg
BFS
DFS
Avg
DNN-PPI
48.09
54.34
51.22
57.40
58.42
57.91
53.05
64,94
59.00
DPPI
41.43
46.12
43.77
52.12
52.03
52.08
56.68
66.82
61.75
PIPR
44.48
57.80
51.14
61.83
63.98
62.91
55.65
67.45
61.55
GNN-PPI
63.81
74.72
69.27
71.37
82.67
77.02
78.37
91.07
84.72
ProtBert
70.94
73.36
72.15
70.32
78.86
74.59
67.61
87.44
77.53
ESM-1b
74.92
78.83
76.88
77.49
82.13
79.31
78.54
88.59
83.57
KeAP
78.58
77.54
78.06
77.22
84.74
80,98
81.44
89.77
85.61
OntoProtein
72.26
78.89
75.58
75.23
77.52
76.38
76.71
91.45
84.08
KeAP"
79.17
79.77
79.47
75.67
83.04
79.36
80.78
89.02
84.90
Table 3: Comparisons on PPI identification. Experiments were performed on three datasets, whose Fl scores are presented. The best results in each setting are bolded. Models with pre-trained using the full G20
are 7 types included in experiments, which are reaction, binding, post-translational modifications, activation, inhibition, catalysis, and expression. The problem of PPI prediction can be formalized multi-label classification problem. We perform experiments SHS148K (Chen et al.] 2019). and STRING (Lv et al., [021]. SHS27K and SHS 148K can be regarded as two subsets of STRING, where protein with fewer than 50 amino acids or 2 40% sequence identity is excluded. Breadth-Fin Search (BFS) and Depth-Firee First Search (DFS) are used generate test from the aforementioned three datasets. F1 score used as the default evaluation metric.
SHS27K (Chen et al.) (019),
Baselines. Following Zhang et al. (2022). we introduce DPPI (Hashemifar et al. 2018). DNN- PPI (Li al. 2018). PIPR (Chen et al. (019),, and GNN-PPI (Lv et al. 2021) as 4 more baselines in addition ProtBert, ESM-Ib, and OntoProtein.
Results. Experimental results are displayed in Table 3 In the first setting, we see that ESM-1b the best performing baseline. Nonetheless, our KeAP still achieves the best average perfor- mance, outperforming ESSM-Ib on all three datasets by 1% at least. In the second setting, KeAP outperforms OntoProtein by about 4%, 3%, and 1% on SHS27K, SHS 148K, and STRING, re- spectively. The trend of declining performance can be attributed to the increasing amount of fine- tuning data (from SHS27K to STRING) that reduces the impact of pre-training. Specifically, the advantages of KeAP are quite obvious SHS27K which has the least number of protein, in- dicating the effectiveness of the protein representation from KeAP with limited fine-tuning data. As the amount of training data increases (from SHS27K to STRING), ProtBert and OntoProtein gradually dis- play inferior performance, compared GNN-PPI. In contrast, our KeAP still performs competitively and sur- passes GNN-PPI by an obvious margin on BFS.
Methods
Affinity (L)
PIPR
0.63
ProtBert ESM-1b
0.58
0.50
KeAP
0.52
4.5 PROTEIN-PROTHH BINDING AFFINITY ESTIMATION
OntoProtein 0.59
KeAPt
0.51
Overview. In this task, we aim to evaluate the ability of the protein representation estimate the change of the binding affinity due to mutations of protein. In practice, each pair of protein mapped
Comparisons on protein- protein binding affinity prediction. The best result in each setting is bolded. + means the lower the better. Models with are pre-trained using the full Pro- teinKG25.
Table 4:
real value (thus this is a regression task), indicating the binding affinity change. We follow (Unsal et al.) 2022) to apply bayesian ridge regression to the result of the element-wise multiplica- tion of representation extracted from pre-trained protein models for predicting the binding affinity. We used the SKEMPI dataset from (Moal Fernández Recio 2012) and report the mean square error of 10-fold cross-validation.
7

---

Published as a conference paper at ICLR 2023
Baselines. We include PIPR, ProtBert, ESM-Ib, and OntoProtein comparative baselines.
Results. Table presents the experimental results on the binding affinity estimation task. We that KeAP outperforms PIPR. ProtBert, and ESM-Ib lb by substantial margins. Considering the region-level structural feature plays vital role in this task (Unsal et al. 2022), we believe the obvious performance advantage of KeAP again verifies the effectiveness of the proposed token-level knowledge graph exploration methodology. Nonetheless, there still a performance gap between KeAP and ESM-Ib, which may be attributed the difference in network architecture.
4.6 SEMANTIC SIMILARITY INFERENCE
Overview. this task, given two interacting protein molecules and their associated attribute terms, we first calculate the Manhattan Sim- ilarity between their representations. Then, we calculate the Lin Similarity between their associated attribute terms following instruc- tions from Unsal et al. (2022). Finally, Spear- man's rank correlation is calculated between the Manhattan Similarity scores and Lin Sim- ilarity scores, where the Lin Similarity scores are treated as ground truths, and the Manhat- Similarity scores are regarded as predic- tions. Specifically, we divide protein attributes into three groups: molecular function (MF), bi- ological process (BP), and cellular component (CC), and report the correlation scores for each group in Table
Methods
MF
BP
CC
Avg
MSA Transformer
0.38
0.31
0.30
0.33
ProtT5-XL
0.57
0.21
0.40
0.39
ProtBert
0.41
0.35
0.36
0.37
ESM-1b
0.38
0.42
0.37
0.39
KeAP
0.42
0.41
0.39
0.41
OntoProtein'
0.41
0.36
0.36
0.38
KeAPt
0.41
0.41
0.40
0.41
Table Comparisons on semantic similarity in- ference. The best results in each setting bolded. Models with f are pre-trained using the full ProteinKG25.
Baselines. addition to ProtBert and Onto-
Protein, introduce three powerful pre-trained protein models for comparisons, which include
MSA Transformer (Rao et al. 202T), ESM-1b (Rives et al. 202T). and Pro (Elnaggar et al.
2021
Results. Table presents the similarity inference results. We see that KeAP achieves the best average result even compared larger (with more parameters) protein models, such as ESM-1b and Pro Specifically, ProiT5-XL produces the best performance on MF and CC, while ESM-Ib performs the best BP. Compared ESM-1b and ProCTS-XL, our KeAP gets 2nd place on MF and achieves the highest score on CC. These results demonstrate the potential of KeAP in outperforming big protein models, and it would be interesting if KeAP could be integrated into bigger models. Again, KeAP outperforms OntoProtein by substantial margins.
Contact
Homology
Stability
PPI
Affinity
Semantic Similarity
Performance changes
-1%
-1%
0%
+1%
-1%
0%
Table 6: Performance changes when removing the proteins that appear in downstream tasks from the pre-training dataset (ProteinKG25). Average results are reported on contact prediction and PPT tasks.
Strutegies
Contact
Methods
SS-Q3
SS-Q8
Fluorescence
Ratios
Contact
Homology
PPI
Protßert
0.81
0.67
0.67
KeAP
0.45
15%
0.44
0.27
76.72
ESM-Ib
0.71
0.65
20%
0.45
0.29
78.06
Cascaded
0.44
KeAP
0.82
0.67
0.65
77.34
PiK
0.38
OntoProteinn
0.68
0.67
Tri. Match
0.45
KcAPi
0.82
0.68
0.67
Table 7: Ablations of mask ratios. Medium-rant range P@@ L/2 results are re- ported for contact prediction.
Table 8: Investigation of knowledge exploita- tion strategies.
Table 9: Failure analysis. We report results on three tasks from 2019).
(Rao et
Manhattan Similarity Manhattan Distance (normalized).
8

---

Published as a conference paper at ICLR 2023
5 ABLATION AND DISCUSSION
We present ablation study results in Tables and g where we investigate the impacts of using different pre-training datasets, different mask ratios, and different knowledge integration strategies, respectively. Table presents the performance on three tasks from Rao et al. (2019). SS-Q3 and SS-Q8 are two secondary structure prediction tasks (Klausen et al. 2019 different numbers of local structures. Fluorescence is regression task, where the model is asked to predict the olg-fluureeeence intensity of each protein.
Cuff & Barton 1999) with
5.1 ABLATION STUDY
Table presents the performance drops when we remove the proteins that appear in downstream tasks from ProteinKG25. We see that the pre-trained model shows slightly worse results on three out of six downstream tasks: contact prediction, homology detection, and affinity prediction, while performing competitively on PPI, stability prediction, and semantic similarity inference. These comparisons imply that we may not have worry too much about the consequences of data leakage. We will continue to investigate this problem in future work.
As shown in Table 7 the 20% mask ratio performs the best two (homology and PPI) of the three downstream tasks, which is the primary reason that we choose 20% as the default mask ratio. is interesting that larger ratio (i.e., 25%) leads to better performance the contact prediction task. We leave the exploration of larger mask ratios for future work.
In addition to the mask ratio, also study the impacts of using different knowledge exploitation strategies. First of all, removing the cascaded exploitation strategy (presented as Cascaded in Ta- ble() results in a 1-percent performance drop on contact prediction, implying exploring the factual knowledge in a cascaded manner is a more effective choice. Then, we remove the proposed protein- knowledge exploration block (denoted auto-encoder trained with the MLM objective. We find that this activity leads to an 7-percent per- formance drop on the contact prediction task, reflecting into protein pre-training. Besides, we add Triplet Matching training objective (appeared Match in Table 8) to KeAP, where we randomly replace the associated attribute term with a differ- ent one and train the model to tell whether the input triplet is matched not. The idea is similar to that of the knowledge-awarc contrastive learning proposed by OntoProtein (Zhang et al. 2022), protein representation. From Table g
PiK in Table 8). which means KeAP is simplified to an
necessity of incorporating knowledge Tri..
which is learning the knowledge-awee the matching objective does not bring performance improvements to KeAP, indicating that our pro- posed exploration strategy for knowledge graphs may already master the information introduced by the Triplet Matching objective.
that adding
5.2 FAILURE CASE ANALYSIS
We report the experimental results on three tasks from Rao et al. (2019), where KeAP performs on with or worse than ProtBert, ESM-1b, or OntoProtein. Specifically, in SS-Q3 and SS-Q8, the model asked predict the secondary structure of each amino acid, which heavily relies on the lo- cal information contained in the protein representation. We think the non-significant performance of KeAP is due to the lack of the incorporation of local details when performing the knowledge encod- ing. Similarly, on the Fluorescence task, KeAP also fails to achieve observable progress when asked to distinguish very similar protein molccules. Considering the same issues also exist in ProtBert and OntoProtein, believe it is necessary pay more attention how to improve the performance local prediction tasks by integrating more local information during the pre-training stage. We will continue explore this issue in the future.
CONCLUSION AND FUTURE WORK
KeAP performs token-level knowledge graph exploration using cross-modal attention, providing a neat solution for knowledge enhanced protein pre-training. KeAP outperforms the previous knowledge enhanced counterpart 9 downstream applications, sometimes by substantial margins, demonstrating the performance superiority of KeAP. In the future, we will investigate how to deploy KeAP specific applications where the factual knowledge makes greater impact.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-bar based deep representation learning. Nature methods,
1322, 2019.
Mohammed AlQuraishi. ProteinNet: a standardized data set for machine learning of protein struc- ture. BMC bioinformatics, 20(1):1-10, 2019.
Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan Davis, Kara Dolinski, Selina S Dwight, Janan Hppig, et al. Gene Ontology: tool for the unification of biology. Nature genetics, 25(1):25-29, 2000.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint w1X1v:1607.06450. 2016.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2018.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information pro- cessing systems, 26, 2013.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: A universal deep-learning model of protein sequence and function. Bioinformaties, 38(8):2102- 21 2110, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Can Chen, Jingho Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-
supervised learning, arXiv preprint wrXiv:2204.04213 2022.
Can Chen. Yingxue Zhang, Xue Liu, and Mark Coates. Bidirectional learning for offline model- based biological sequence design. arXiv preprint arXiv:2301.0293/1 20233,
Muhao Chen, Chelsea J-T Ju, Guangyu Zhou, Xuelu Chen, Tianran Zhang. Kai-Wei Chang, Carlo Zaniolo, and Wei Wang. Multifaceted protein-protein interaction prediction based on siamese residual RCNN. Bioinformatics, 35(14):335-ii314 2019.
James Cuff and Geoffrey J Barton. Evaluation and improvement of multiple sequence methods for protein secondary structure prediction. Proteins: Structure, Function, and Bioinformatics, 34 (4):508-519, 1999.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1110::18100048000 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. ProtTrans: Towards crack- ing the language of lifes code through self- self-supervised deep learning and high performance com- puting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nature communications, 13(1):1-10, 2022.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifie language model pretraining for biomedical natural language processing, ACM Transactions on Computing for Healthcare (HEALTH), 3(1); 1-23, 2021.
Somaye Hashemifar, Behnam Neyshabur, Aly A Khan, and Jinbo Xu. Predicting protein-protein interactions through requence-based deep learning. Bioinformatics, 34(17):8802-i810 2018.
10

---

Published as a conference paper at ICLR 2023
Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. BERT-MK: Integrating graph contextualized knowledge into pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2181-228199 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Jie Hou, Badri Adhikari, and Jianlin Cheng. DeepSF: deep convolutional neural network for map- ping protein sequences to folds. Bioinformatics, 34(8):1295-1305 2018.
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with alphafold. Nature, 596:583 - 589, 2021.
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Jurtz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, et al. NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 2019.
Hang Li, Xiu-Jun Gong, Hua Yu, and Chang Zhou. Deep neural network based predictions of protein interactions using primary sequences. Molecules, 23(8):1923, 2018.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1073-1094, 2019.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2901-2908, 2020.
Guofeng Lv, Zhiqiang Hu, Yanguang Bi, and Shaoting Zhang. Learning Unknown from Correla- interaction prediction. In Proceedings of the
tions: Graph neural network for Thirtieth International Joint Conference on Artificial Intelligence, pp. 3677-3683, 2021.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint twxiv:2004.034977 2020.
lain H Moal and Juan FemándezzRecio SKEMPI: a structural kinetic and energetic database of mutant protein interactions and its use in empirical models. Bioinformatics, 28(20):2600-266 2012.
Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. ProGen2: exploring the boundaries of protein language models. arXiv preprini arXiv:2206. 13517. 2022.
Matthew E. Peters, Mark Neumann, IV RobertL.LLgan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In EMNLP, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLL--JCNIIP), pp. 2463-2473, 2019.

---

Published as a conference paper at ICLR 2023
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. ERICA: Improving entity and relation understanding for pre-trained lan- guage models via contrastive learning. In Joint Conference of the 59th Anmual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Nat- ural Language Processing, ACI-IJCNLP 2021, pp. 3350-3363. Association for Computational Linguistics (ACL), 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with TAPE. Advances in neural information processing systems, 32, 2019.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations, 2020.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. In International Conference on Machine Learning, pp- 0844-8856. PMLR, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e01622311 2021.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, 2020.
Gabriel Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, et al. Global analysis protein folding using massively parallel design, synthesis, and testing. Science, 357 (6347):168-177 2017.
Justyna Sarzynska- -aarskkk-Wawer,,rrr Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewiez, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304: 14135, 2021.
Andrew W. Senior, Richard Evans, John M. Jumper, James Kirkpatrick, Sifre. Tim Green, Chongli Qin, Augustin Zidek. Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu, and Demis Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577:706-77 -700067--1,000 2020.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8968 09788897,.000 2020.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 13(10):1282-1253 2007.
Baris E Suzek, Yugi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consor- tium. Uniref clusters: comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6):926-933 2015.
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Doğan. Learning functional properties of proteins with language models. Nature Machine Intelligence, (3):2272245, 2022.
12

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Nazneen Rajani, et al. BERTology Meets Biology: Interpreting attention in protein language models. In International Conference on Learn- ing Representations, 2020.
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pp. 7222-7240, 2020.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: unified model for knowledge embedding and pre-trained language represen- tation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. Representation learning of knowledge graphs with entity descriptions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Kevin K Yang, Alex Xijie Lu, and Nicolo Fusi. Convolutions competitive with transformers for protein sequence pretraining. ICLR2022 Machine Learning for Drug Discovery, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.
XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Liang Yao, Chengsheng Mao, and Yuan Luo. KG-BERT: Bert for knowledge graph completion.
arXiv preprint arXiv: 1999000399,, 2019.
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. OntoProtein: Protein pretraining with gene ontology embedding. International Conference on Learning Representations, 2022.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451, 2019.
Hong- Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via cross-supervision between images and free- text radiology reports. Nature Machine Intelligence, 4(1):32-40, 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A. HYPER-PAR IOPER-PARAMETERS FOR FINE-TUNING
The syper-parameters for fine-tuning are provided in Table T0 Specifically, we follow the hyper- parameter settings in GNN-PPI (Ly et al. 2021 for PPI prediction. For protein binding affinity pre- diction and semantic similarity inference, we follow the fine-tuning configurations in PROBE (Unsal
al. 2022
Task
Epoch
Batch size
Warmup ratio
Learning rate
Freeze Bert
Optimizer
Contact
5
8
0.08
3e-5
false
Adam W
Homology
10
32
0.08
4e-5
false
Adam
Stability
5
64
0.08
le-5
false
AdamW W
SS-Q3
5
32
0.08
3e-5
false
Adam
SS-Q8
5
32
0.08
3e-5
false
AdamW W
Fluorescence
15
64
0.10
le-3
true
Adam
Table 10: Hyper-parameters for fine-tuning.
A.2
PROBABILITY CONTACT MAPS
In Fig. we present two randomly picked samples for visual analysis, where KeAP is able to detect contacts missed by ProtBert and OntoProtein with high confidence.
Figure 3: Ground truths and predicted probability contact maps. We compare the predictions of ProtBert and OntoProtein with ours, where the 1st, 2nd, and 3rd rows include the all, top L, and top L/2 predictions, respectively. For top L and L/2 predictions, the precision scores are reported. More examples are in the appendix.
14


Reviews:
[{"review_id": "tzxua-ID47", "reviewer": "Reviewer_oJan", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To overcome the semantic gap between protein sequences and natural language, this paper proposes KeAP (Knowledge-exploited Auto-encoder for Protein) to perform knowledge enhanced protein representation learning. Specifically, it performs protein-centric knowledge exploitation via the attention mechanism, and uses the MLM objective to make it easier to optimize. The generalization ability of the learned protein representation are evaluated by fine-tuning the pre-trained model on a wide range of downstream applications. Ablations and failure analyses are provided to facilitate the understanding of the model.", "summary_of_the_review": "The paper introduces how domain knowledge can be implicitly incorporated into protein representation learning. The authors use several popular ideas, such as the attention mechanism and MLM objective and show its effectiveness on multiple downstream tasks. Unfortunately, the author's writing logic is not clear enough, making the article not particularly smooth to read.  ", "strength_and_weaknesses": "Strength:\n\n1.This paper proposes a new knowledge-enhanced method of Knowledge-exploited Autoencoder for protein representation learning.\n\n2.Experimental results show that the proposed approach achieves better performance than baselines.\n\n3.This paper also provides ablations and discussions of failure cases, which help readers understand the model.\n\nWeakness:\n\n1.It looks like the authors have made some small improvements on the basis of the existing method OntoProtein. Specifically, both of these two methods aim to incorporate external domain knowledge (KG) into protein representation learning. It is questionable whether KeAP actually provides a great contribution or innovation to the community.\n\n2.The three colors in the legend of the Figure1 are not very distinguishable, so it takes a lot of effort to distinguish the corresponding method of each color. In addition, I think the positions of the figures and the tables should be moved to the corresponding part of the text to make it easier for readers to read. For example, Figure1,2,3 and Table1,2,6,7,8. Figure3 is too small to be seen clearly, affecting the reader's understanding of the experiment.\n\n3.I think there are some problems with the author's writing logic. Since the concepts of ProteinKG25 and Gene Ontology are mentioned for the first time in Section4, if readers are not familiar with the settings of OntoProtein, they will not know what relations are described in the protein-related knowledge graph. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b2Eyoo1sEO-", "reviewer": "Reviewer_cqj2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel protein sequence pre-training method, Knowledge-exploited Auto-encoder for Proteins (KeAP). Instead of injecting biomedical knowledge by directly applying knowledge embedding constraints as OntoProtein, KeAP employ a cross-attention mechanism to attentively inject relevant relation and attribute representations into protein representations. Upon such enhanced protein representations, masked residue type prediction loss is applied for representation learning. On contact prediction, homology detection, stability prediction and PPI prediction benchmarks, authors demonstrate the superiority of KeAP over OntoProtein and ProtBert (the initial protein sequence encoder of KeAP).  ", "summary_of_the_review": "In summary, I am convinced by the motivation of learning better protein representations guided by biomedical texts and the cross-attention technique to achieve this goal. However, I have concerns on the input data organization during pre-training and the completeness of performance comparison. Therefore, I think the current manuscript is on the border and expect authors' efforts during rebuttal. ", "strength_and_weaknesses": "Strength:\n1. It is a novel idea to enhance protein representations with associated biomedical texts (e.g., the relevant molecular functions and biological processes). The proposed cross-attention fusion scheme is technically sound to achieve this goal.\n2. KeAP achieves some obvious performance improvements over the previous SOTA OntoProtein on contact prediction and PPI prediction tasks.  \n\n\nWeakness:\n1. KeAP does not fully utilize the biomedical texts associated with a protein. For each forward pass, KeAP adopts a single (protein, relation, attribute) triplet and enhance the protein representation with only the relation and attribute texts in this triplet. However, in the biological KG used by authors, each protein can appear in multiple triplets, where these triplets describe orthogonal properties of the protein. Therefore, it is more promising to combine different text descriptions associated with a protein and inject relevant textual information across the combined text. I am curious and willing to see the performance of KeAP under such an improved input format. \n2. Some important baselines are missed in performance comparison. Authors are suggested to include the results of the SOTA protein language models, ESM-1b and ProtT5, into Tables 1, 2 and 3.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Hs4Cpq7U89", "reviewer": "Reviewer_mLUJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary of the paper\nOntoProtein is prior art that integrates additional information from Gene Ontology into protein presentation. OntoProtein chooses the MLM and TransE as two training objectives when learning protein representation on the Gene Ontology knowledge graphs.\nThe authors claim that the TransE objective has an issue because, in Gene Ontology, a triple usually concerns (Protein, Relation in natural language, and Property in natural language). While the initial embedding space of proteins and texts differs,  \"OntoProtein did not take into account the domain gap between protein and natural language, and this semantic gap may make the TransE objective a suboptimal choice \".\nTherefore, the authors proposed a new learning objective where only masked language model loss is considered during the training process. Specifically, random amino acids are masked as usual. Initial embeddings of masked protein sequences and texts (relations and properties) are obtained using pretrained models for proteins and texts respectively. \nThese initial embeddings are used by a decoder where attention layers (QKV Vaswani et al.) are used to obtain relation/property attention, given the initial embedding of proteins as a query.  The main objective of the decoder is to learn attentive weights to knowledge graph triples associated with the protein such that it helps predict masked amino acids in the sequence.\nThe authors claimed that this solution learned to incorporate knowledge implicitly into the protein representation and solved the semantic gap issues. The authors demonstrated the significance of the results compared to OntoProtein in 9 different downstream tasks.\n\n", "summary_of_the_review": "Summary of the comments:\nAlthough I see this as an interesting research work I am concerned about its evaluation methods with potential leakage. I am also concerned about its baseline methods to compare. More implementation details are needed to reproduce the results and stronger evidence to support the claim about \"semantic gaps\"\n", "strength_and_weaknesses": "Strengths\nThis a well-written paper.\nSignificant experimental results compared to OntoProtein.\nA good application paper.\n\nWeaknesses\n\nAdditional comparisons to state-of-the-art language models are needed:\nRecent language models for proteins such as ESM-1B, ESM-2, and MSA-Transformer demonstrate better results on TAPE benchmark datasets. I see you compare your approach with these models on only similarity prediction tasks. Why don't you compare these methods to the other benchmark datasets like contact prediction, homology, stability, protein-protein interaction, and affinity binding? \n\nA potential flaw in experiments:\nPotential leakage happens if proteins in the test sets of the benchmark datasets are not removed from the training, do you check the overlapping between test sets and the ProteinKG25. If that is not the case, the reported results might be over-optimistic.\n\nRelated to semantic gaps:\nThe authors claim that semantic gaps are the main issues of OntoProtein but it is very hard to see any evidence demonstrated in the paper to support that claim. Since this is the main claim against OntoProtein, this needs to be supported by a piece of strong evidence with careful analysis.\n\nReproducity:\nI checked both the paper and the appendix version, this information is needed to clarify the implementation details:\n+ what is the representation of proteins used in the encoder? I understand it is a bert-like encoder but whether you use pretrained models or train everything from scratch? \n+ potential leakage happens if proteins in the test sets of the benchmark datasets are not removed from the training, do you check the overlapping between test sets and the ProteinKG25 ?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jkux9r4YZRm", "reviewer": "Reviewer_6wQd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "LM pretraining on protein sequences has become popular in recent years. One difference between proteins and NLP is that for proteins there is often lots of structured/unstructured knowledge graph information for each pretraining example. This paper extends masked language modeling (BERT/MLM) by allow the model to also attend to an encoding of textual side information. This improves the performance of models on a variety of benchmark tasks.", "summary_of_the_review": "Can you comment on the relationship between your work and ProGen (https://arxiv.org/abs/2004.03497)? This also uses protein KG information in a large language model. \n\nWhen you use your model to get per-residue protein embeddings for downstream tasks, I'm assuming that you pass correct / ground truth KG information. For some of the tasks, does this present a data leakage problem, where there is information provided at evaluation time that reveals information about test-set labels? For example, for the protein-protein interaction task, you're providing GO terms for proteins in the test set. With this, are you effectively learning a model about which GO terms interact with other GO terms?\n\nI was disappointed that, besides Table 7, the experiments don't provide a direct ablation that removes the ontology information. The ProteinBert results are from a separate paper. Surely there are other differences between the systems, such as hyper-parameters. \n", "strength_and_weaknesses": "=Strengths=\nThorough evaluation on lots of downstream tasks\nLots of citations to recent work in NLP that uses similar ideas.\nThe general modeling idea (established in the OntoProtein paper) is good.\n\n=Weaknesses=\nMethodological contribution is very minor.\nWriting is confusing. Lots of poorly-defined jargon (see below).\nPotential issues regarding data leakage for test set (see below).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster