Multi-level Protein Structure Pre-training via Prompt Learning
Authors: Zeyuan Wang
Qiang Zhang
Shuang-Wei HU
Haoran Yu
Xurui Jin
Zhichen Gong
Huajun Chen
Venue: ICLR 2023 poster (2023)
Keywords: protein representation learning
prompt learning
multi-task learning
multi-level structure

Abstract: A protein can focus on different structure levels to implement its functions. Each structure has its own merit and driving forces in describing some specific characteristics, and they cannot replace each other. Most existing function prediction methods take the tertiary structure as input, unintentionally ignoring the other levels of protein structures. Considering protein sequences can determine multi-level structures, in this paper, we aim to realize the comprehensive potential of protein sequences for function prediction. Specifically, we propose a new prompt-guided multi-task pre-training and fine-tuning framework, and the resulting protein model is called PromptProtein. Through the prompt-guided multi-task pre-training, we learn multiple prompt signals to steer the model to focus on different structure levels. We also design a prompt fine-tuning module to provide downstream tasks the on-demand flexibility of utilizing respective levels of structure information. Extensive experiments on function prediction and protein engineering show that PromptProtein outperforms state-of-the-art methods by large margins.

Published as a conference paper at ICLR 2023
MULTI-LEVEL PROTEIN STRUCTURE PRE-TRAINING WITH PROMPT LEARNING
Zeyuan Wang! 127
Qiang Zhang Haoran Yu
Shuangwei Hu
Xurui Jin
Zhichen Gong26 2,6
Huajun Chen 1,2,7,8
'College of Computer Science and Technology, Zhejiang University
ZzJU-Has Hangzhuu Global Scientific and Technological Innovation Center
CColleee of Chemical and Biological Engineering. Zhejiang University
Vecx Biomedicines Inc., 'MindRank Al Ltd., University College London AZFT Joint Lab for Knowledge Engine, East China Sea Laboratory (yuanzew, giang. zhang. .aggg.cs, cs, yuhaoran, huajunsir] edu en shuangwei@vedt bio, xurui surui@mindrank ai, ucabzgo@ucl. ac uk
ABSTRACT
A protein can focus on different structure levels to implement its functions. Each structure has its own merit and driving forces in describing specific characteristies, and they cannot replace each other. Most existing function prediction methods take either the primary or the tertiary structure input, unintentionally ignoring the other levels of protein structures. Considering protein sequences can determine multi-level structures, in this paper, aim realize the comprehensive poten- tial of protein sequences for function prediction. Specifically, we propose a new prompt-guided multi-task pre-training and fine-tuning framework. Through the prompt-guided multi-task pre-training, we learn multiple prompt signals to steer the model, called PromptProtein, to focus on different levels of structures. We also design a prompt fine-tuning module to provide downstream tasks the on-demand flexibility of utilizing respective levels structural information. Extensive exper- iments function prediction and protein engineering show that PromptProtein outperforms state-of of-the-art methods by large margins. To the best our knowl- edge, this is the first prompt-base based pre-trained protein model.
INTRODUCTION
Pre-trained language models (PTLMs) have prevailed in natural language processing (NLP). Re- cently, some methods (Alley et al., 2019 Elnaggar et al.] [002]: Rives et al. (021) use PTLMs to encode protein sequences to predict biological functions, which called pre- trained protein models (PTPMs). In contrast to natural languages, there are four distinct levels of protein structures (Kessel & Tal 2018). The primal is the protein sequence consisting of amino acids, the second refers to the local folded structures (c.g., a helix and / pleated sheet), the tertiary describes the natural folded three-dimensional structure, and the quaternary is a protein multimer comprising multiple polypeptides. A protein can focus different structure levels to implement its specific functions, including reserving a piece of the sequence, manifesting the whole 3D structure as conformational elements, or even cooperating with other proteins. Therefore, when predicting protein functions, it vital to flexibly utilize multi-level structural information.
AlphaFold2 (Jumper et al. 2021 makes great progress in the tertiary structure prediction based on protein sequences. However, directly learning from predicted structures can be unachievable as the prediction of proteins without homologous sequences is inaccurate. More importantly, the quater- nary structure of protein multimers which faithfully depicts protein functions is usually different from the tertiary (see Figure [ and reliable predictive models have not been released. Fortunately, protein sequences are easy to obtain and can determine all the other levels of structures. This pa- per aims realize the full potential of protein sequences in function prediction by prompting
"Equal contribution and shared co-first authorship. "orresponniig author.

---

Published as a conference paper at ICLR 2023
PTPM to exploit all levels of protein structures during pre-training. The main challenges two- fold: 1) how to design proper pre-training tasks for different protein structures? and 2) how to efficiently integrate these tasks in the pre-training phase and transfer the implicit protein structure knowledge for function prediction in fine-tuning phase.
For the first challenge, mentary pre-training tasks across multiple structure levels, targeting both fine and coarse resolutions. Specifically, we use the de facto Mask Language Modeling (MLM) task to exploit the primary struc- ture information, where the model needs to predict randomly masked amino acids a protein. For the secondary and tertiary structure, we propose the alpha-carbon CooRDinate prediction (CRD) task, where the model should output the relative posi- tions between residues. For the quaternary structure, propose the Protein-Protein Interaction predic- tion (PPI) task, where the model is required to esti- mate the interaction probability. We collect millions of data covering different levels of protein structures from UniRef50 (Consortium) 2021), Protein Data Bank (Berman et al., 2000), and STRING (Szklar-
design three comple-
Figure I: comparison of protein CDK1 in the tertiary (left) and quaternary (right) structures.
czyk al. 2019).
For the second challenge, a straightforward strategy is to leverage multi-task learning to combine the losses of different pre-training tasks. However, many works (Wu al., 2019 Yu et al., 2020) find that task interference is common when tasks are diverse. This problem can be more severe in multi-task pre-training due to the gap between pre-training and downstream tasks, causing negative knowledge transfer. For example, BERT (Kenton & Toutanova, 2019h leverages MI. and Next Sentence Prediction (NSP) to learn the sequential dependency and sentence relationship simulta- neously, while RoBERTa (Liu et al. 2019) finds the performance will be slightly improved when removing the NSP loss. We postulate this problem also exists in multi-level protein structures, as different structures can be inconsonant. The MLM task emphasizes the neighboring relations along the sequence, while the CRD task shall focus more on long-range amino acid pairs which can be spatially close in the tertiary structure.
To address this challenge, inspired by recent prompt learning, we propose a prompt-guided multi- task pre-training and fine-tuning framework, and the resulting protein model is called PromptProtein. The prompt-guided multi-task pre-training associates multiple pre-training tasks with dedicated sen- tinel tokens, called prompts. To utilize the prompt tokens, we introduce prompt-aware attention module, which modifies two components of the Transformer architecture: 1) Attention mask, which designed to block attention calculation from input data to a prompt as a prompt should be task- dependent instead of sample-dependent 2) For skip connection, a prompt is used to calculate skip weight, which can filter out task-irrelevant information. prompt fine-tuning module to coordinate all prompt tokens, such that the model is capable of lever- aging multi-level protein structure information flexibly, enabling the positive transfer of learned structural knowledge to downstream tasks.
the fine-tuning phase, we propose a
We conduct experiments on function prediction and protein engineering as downstream tasks, where PromptProtein significantly outperforms state-of-the-an on all datasets, especially on low-resource protein engineering tasks where PromptProtein achieves an average improvement of 17.0%.
RELATED WORKS
Protein Representation Models. Proteins have complex structures that determine their biological functions (Epstein et al. 1963). A growing body of work focuses on how to leverage structural in-
formation. Since evolution through natural selection has spoken protein sequences as their "natural
language". various natural language processing methods have been extended to proteins. Asgari
& Mofrad (2013); Yang et al. (2018) apply word embedding algorithms (Mikolov et al.) (013) to obtain protein representations. Dalkiran al. (2018); Öztürk al. (2018) one-dimensional con-
2

---

Published as a conference paper at ICLR 2023
Prompt Pool
MASLSCYIMaaKDKKWWT
Mask Languag5, Modeling
MASLSCV[MaskJKKMVV
[MLM]
[CRD]
Prompt-Aware
hp
Aipha-Carbon, Prediction
Transfomer
[PPI]
Parameter
Interaction Prediction
MFF
trpA
Sharing
To(-)
Composed Prompt
p'
Prompt-Aware Transfomer
h
Tasks
Stability Prediction
Fibness Prediction
Function Annotation
MASLSCVSDKMMVT
The architecture overview of PrompiProtein. In the pre- pre-training stage, we pre-train our model with three structure-relater related tasks, including mask language modeling, alpha-carbon predic- tion, and protein-protein interaction prediction. For each task, the model takes the protein sequence and the task-specific token as input and learns to produce a representation encoding the correspond- ing structure information. In the fine-tuning stage, a prompt-tuning module rp(-) (-) can flexibly com- bine structure information via the learned prompt tokens for diverse downstream tasks.
Figure 2:
volutional neural networks to predict the functions. Furthermore, Alley et al. (2019); Elnaggar et al. (2021); Rives et al. (202I) explore whether the pre-training and fine-tuning paradigm, the trans- former architectures, and the objective functions can effectively transfer from natural languages to proteins. Zhang et al. (202 align the amino acid sequence and the text sequence obtain informa- tive protein representation. To utilize the tertiary structure, [Hemmsiila et al. (222)); Somnath et al. 2021 Ganea et al. (202T); Zhang et al. (2022) build protein graphs and employ message-passing neural networks to produce structure-aware representations, |Bepler& & Berger (202T) employ contact map prediction and structural similarity prediction to pre-train the protein model. Although primary and tertiary structures have been studied, few works try to enrich protein representation with the qua- ternary structure which faithfully depicts protein functions. In this paper, we show that systematic modeling and flexible utilization of multi-level structures are the keys to improving the performance of function prediction and protein engineering.
Multi-task Learning. The goal of multi-task learning is to take advantage of inductive transfer across tasks and achieve better generalization performance. When tasks are diverse, using a naive shared MTL model can suffer from task interference. Prior methods have been proposed to de- conflict gradients from different tasks. [Cen et al. (2018) dynamically adjust gradient magnitudes so different tasks can be trained at similar scales. Yu et al. (2020) take the gradient direction into account and drop the projection of one task gradient direction onto another they are conflicting. Rather than clipping the conflict gradient direction, Pavaloy & Valera (201)) learn rotation ma- trix for each task to bring different optima closer to each other. However, these methods are not designed for multi-task pre-training and cannot properly deal with the knowledge transferability to downstream tasks. We provide schematic comparison of these methods in Appendix A.]
Prompts for Pre-trained Models. In-context learning (Brown et al., 2020) is introduced to steer the pre-trained model to produce task-desired representations. In the NLP area, the prevailing ap- proaches to designing prompts can be divided into two categories: discrete prompt designing and continuous prompt tuning. The discrete prompt technique (Schick & Schütze 202T) adds task de- scription tokens from a vocabulary the context obtain enriched sentence embeddings. However, the hand-crafted prompts may provide disturbance of human bias and are limited to discrete vocab- ulary spaces. contrast. [i Liang (202T): /Zhang al. (202Tb) generate optimal prompt vectors in continuous spaces. Inspired by these works, we extend the concept of prompt tuning to the pre-training stage, associate multi-level protein structural information with dedicated prompt tokens during pre-training, and adaptively combine these learned prompts for downstream tasks.
METHODOLOGGY
To acquire multiple information from the input data x, conventional multi-task learning usually produces universal representation h. The whole objective can be formulated weighted sum of individual task objectives: L Σ. (h)), where {a;} are the hyper- parameters to balance these losses. However, multi-level protein structures can be inconsonant: the primary structure focuses
3

---

Published as a conference paper at ICLR 2023
more the dependency along the sequence, whereas the tertiary and quaternary structure weights more on the spatial organization, which can cause the problem of task interference. This problem can lead to more severe negative transfer in multi-task pre-training due to the gap between pre-training and downstream tasks. To solve this problem, we propose a prompt-guided multi-task pre-training and fine-tuning framework that utilizes a prompt token p to produce task-specifie representation hp. Multiple learned tokens can be flexibly combined to steer the pre-trained model for various downstream tasks, bridging the gap between pre-training and downstream tasks.
This section first describes how to use prompts to modify the Transformer architecture, such that different tasks can be processed by different neural layers and reduce task interference. Then we present the three pre-training tasks to acquire multi-level protein structural information: (1) masked language modeling, (2) alpha-carbon coordinate prediction, and (3) protein-protee diction. Finally, we introduce the prompt-guie -uided pre-training and fine-tuning framework where multiple information can be acquired in the pre-training stage and combined on-demand for down- stream tasks. The resulting Promp(Protein model is illustrated in Figure 2
interaction pre-
3.1 PROMPT-AWARE ATTENTION MODULE
To reduce interference between pre-training tasks, we use the prompt token to modify the Trans- former architecture so that multiple information can be effectively acquired by the pre- pre-trained model. Specifically, we modify two parts of the Transformer: attention mask and skip connection, and the resulting architecture is called Prompt-aware Transformer. Given input protein sequence and a prompt token p, we define the whole input Tp denote zp = 2: lp where || is concatenation. Let (1) o be the i-th token of the whole input and hp be the representation of xp at the (-th layer.
Attention mask. The conventional self- attention is formulated as: Attn(he') (oftmax((K")///i)
Skip Connection
where Q. K, and V
are the linear projection of ho Each token in the whole sequence can attend to others at any position which means the condition prompt will be affected by the input sequence. A more reasonable way is to keep only the effect of the prompt on the input sequence and eliminate the reverse effect, prompt should be task-dependent instead of sample-dependen As illustrated in Figure 33 we design attention mask matrix M to fulfill this requirement. Let Mij denote the -elem)- of the mask matrix, and we define:
MatMul
Wo 2
P
Softmax
4
Linear
22
Scale
Spt
Attention Mask
MatMul
Linear
Linear
Linear
o
22 33
P
0, ml p and nhj E
My =
(1) Figure 3: Prompt-aware Attention Module. A pink circle represents an amino acid token and purple circle represents a prompt token. We decouple prompt tokens from amino acid tokens by the attention mask. The embed- ding of decoupled prompt token determines the weight of the residual connection. In the fine-tuning stage, we use prompt-tuning module (+)-) to learn the downstream task- desired composed prompt.
1, others.
Skip connection. Skip connection enables deep neural networks easier to train (He et al.] 2016). To encourage different tasks to be processed by differ- ent layers and reduce task interference, we design a weighted skip connection. That is, the prompt to- ken is used to calculate weight for the output of the attention module, The whole process can be:
(1+1)) (!) p (1 (1-99) o Attn(h()).
(2)
where sp scalar, is linear projection of I-th layer embedding of prompt p. After L layers of the prompt-aware attention module, we have the task-specific representation hp = hp
3.2 PROTEIN MULTI-LEVEL STRUCTURES LEARNING
To acquire multi-level protein structure information, we consider three complementary pre-training tasks: (1) masked language modeling, which has been commonly used by existing PTPMS and can
4

---

Published as a conference paper at ICLR 2023
capture the primary structure information; (2) coordinate prediction, which acquires the secondary and tertiary structure; and (3) interaction prediction, which acquires the quaternary structure.
Masked language modeling. This task uses all available amino acid tokens to recover the masked ones. Let Y be the set of masked out tokens, and be the vocabulary of amino acid tokens. The MLM loss is formulated:
exp(plyl hpp)) (lyhp)) E Lvey exp(p(v(lt)))
Cucu(pp)
Σ NEY
log ((yl hp))
Iipha-Carbon Coordinate Prediction. Since a secondary structure can be inferred from the pro- tein 3D coordinates (Kabsch & Sander 1983), we use an a-C coordinate prediction task to learn both secondary and tertiary structures. Given the sequence length |ar|, we denote the ground-truth naturally folded 3D structure of protein as Z E R|a[xa and the structure predictor, 2-layer MLP network, as then the predicted structure is k(hp) E RR/=|x3, By translating and rotating (Kabsch) 1976) the predicted structure, we can get the minimal root mean square deviation between ground- truth and predicted structure, and the loss is calculated based this deviation. In this way, there is no need to consider spatial invariance or cquivariance, but only need to focus on the relative positions between residues. The CRD loss be calculated the mean square error (MSE):
LeRD hp) = MSE Z, Kabsh(ro(hpp (1)())))).
(4)
Prottinn Interaction prediction. To acquire the quaternary structure information, we con- duct the third pre-training task: predicting whether the m-th and n-th proteins can interact with each other within batched data. Let he be the m-th protein in mini-batch and ym., is the ground-truth We first calculate pair-aware protein representation hm. then formulate the PPI loss:
(h W W)gW)>
A Sigmoid
Vd |eann (Attum p)),
m.n p
mean(Attn hp)
Lepi (hp)
Σ
BCE( p(Um,,) h hm
(5)
m.EEN
where W E Rdw is projection matrix, BCE is the binary cross-entropy loss function, N is the batch size. More details of the pre-training tasks are provided in Appendix A.2
3.3 PROMPT-GUIDED MULTI-TASK PRE-TRAINING AND FINE-TUNING
Corresponding to the three pre-training tasks, the prompt can be instantiated as one of the three tokens, i.c., p P [[LM), (CRD), [PPI]]. The task-specific representation is thus denoted as h [CRD]. hppp))- The objective function of the prompt-guided multi-task pre-training can be formulated as:
+ Cz cRD) + 0z3 a:gCpr(h)ppr)
(6) 6 both model parameters 1 and prompts
When we pre-train a model with multiple tasks as Equation p are optimized. In this way, the model does not necessarily need to learn the optimal representation for all tasks, but only needs to learn the respective optimal representation for each task. Hence, the problem of task interference can be alleviated.
Furthermore, to bridge the gap between pre-training and downstream tasks, since the model can ac- quire each type of information conditioned on the learned prompt tokens, we can combine these to- kens with prompt-tuning to flexibly mix the acquired information on-demand. We denote prompt- tuning module as To (-), and the downstream task-de protein representation hy can be obtained by feeding the tuned prompt p'
p'
= To (P P(caD) PPPr))
Then the pre-trained model can produce hpp and conduct predictions for the downstream task of interest. Equation Saniiionnshai shows how to flexibly utilize the pre-training task information at the fine-tuning stage. Note that, in the pre-training stage, only append one prompt to acquire one type of task- specific information, while in the fine-tuning stage, we feed all the learned prompt tokens to Te(-) and flexibly combine the acquired information. Here, we leverage linear layer as our prompt-tuning module to combine three learned prompts. For sake of understanding, we provide the pseudo-code of the prompt-guided multi-task pre-training and fine-tuning framework in Appendix A.3
5

---

Published as a conference paper at ICLR 2023
Table 1: Model performance on EC numbers and GO terms prediction tasks. the results taken
from Wang et al. (2022), I the results taken from Zhang et al. (022).
BC
GO-BP
GO-MF
GO-CC
DATASET
AUPR
Fmux
AUPRpair
F
AUPRpai
Fmnx
AUPRpaiz
Fmax
CNN
0,540
0.545
0.165
0.244
0.380
0.354
0.261
0.387
RESNET
0.137
0.187
0.166
0.280
0.281
0.267
0.266
0.403
LSTM
0.032
0.082
0.130
0.248
0.100
0.166
0.150
0.320
TRANSPORMER
0.187
0.219
0.135
0.257
0.172
0.240
0.170
0,380
GAT
0,320
0.368
0.171
0.284
0.329
0.317
0.249
0,385
GVP
0.482
0.489
0.224
0.326
0.458
0.426
0.278
0.420
DEEPFRI
0.547
0,631
0.282
0.399
0.462
0.465
0.363
0.460
GrarNet Edge!
0.892
0.874
0.292
0.596
0.650
0.336
0.486
ESM - 1bb
0.889
0.864
0.343
0,470
0.639
0.657
0.384
0.488
ProtBERT BED
0.859
0.838
0.188
0.279
0.464
0.456
0.234
0,408
LM GVP
0.710
0.664
0.302
0.580
0.545
0.423
0.527
MT-LSTM
0.851
0.817
0.324
0.442
0.608
0.591
0.381
0.492
MTL
0.892
0.869
0.325
0.445
0.651
0.640
0.415
0.503
GRADNORM
0.893
0.874
0.331
0.466
0.647
0.643
0.415
0.504
ROTOGRAD
0.895
0.876
0.334
0.470
0.648
0.638
0.416
0.509
PROMPTPROTEIN (OURS)
0.915
0.888
0,363
0,495
0.665
0.677
0,457
0.551
4 EXPERIMENTS
4.1 PRE-TRAINING SETUP
For the primary structural information, we use UniRef50 (Suzek et al. 2015) which is a clustering of UniRef90 seed sequences 50% sequence identity. For the secondary and tertiary structural information, we use Protein Data Bank (PDB) (Berman et al.) 2000), which includes 200,000 pro- tein 3D structures obtained by experimental methods. For the quaternary structure information, we use the STRING dataset (Szklarezyk et al. 2019) that contains amino acid sequences and protein- protein interaction pairs. In the STRING dataset, protein interactions are divided into 7 categories. We selected the physical-only interaction subset from STRING which contains 65 million protein sequences from 14,095 species and 2.7 billion protein- interaction pairs.
We implement PromptProtein using Pytorch (Paszke al. 2019) and Fairseq (Ott al. 2019) PrompiProtein has 650M parameters with 33 layers and 20 attention heads. The embedding size is 1280. The learning rate is 1 x 10 with weight decay. We use an inverse square root learning rate schedule. All models are trained on 2x AxA100 40G GPUs for 270k steps of updates. After pre-training, the average error of the coordinate prediction task accuracy of physical binding prediction is greater than 90.0%. Unless otherwise specified, we use this model in all downstream experiments. The source code will be available online. Please refer to Appendix B for the details of all the pre-training and downstream task dataset statistics.
single residue is 5 X, and the
4.2 DOWNSTREAM TASKS: FUNCTION ANNOTATION
Datasets and Metrics. Gene ontology (GO) terms and enzyme commission (EC) numbers are two standard classification schemes that organize myriad protein functions. These function pre- diction tasks can be regarded as multiple binary classification tasks. We follow the dataset split method in (Glioorjjević et al. 202T). The evaluation metrics are protein-centric maximum F-score (Fmax) and term-centric area under precision- (AUPR) curve, which are used in the CAFA challenges (Radivojac et al. 2013).
Baselines. There are four categories of baselines. (1) Sequence- based encoders. CNN Shanehsaz- zadeh et al., 2020). ResNet, LSTM, and Transformer (Rao et al. (201)) only take amino acid se- quence as input: (2) Geometric learning method. GAT (Veličković et al. 2018), GVP (Ling et al.) al. 202I). and GearNet-Edge (pre-trained by Multiview Con- trast) (Zhang et al.) (022) take protein 3D coordinates as additional input to obtain informative representation; (3) Pre- teaitraieeddddd protein models. ESM-1b (Rives al. 2021), ProtBERT-BFD (EI- naggar et al.] 202T), and LM-GVP (Wang et al. 2022) learn the pattern from large protein corpus. contact map and structure similarity to enrich the embed-
2020), DeepFRI GGliorijeevć
MT-LSTM (Bepler& & Berger 2021

---

Published as a conference paper at ICLR 2023
Table 2: Model performance protein engineering tasks.
Results with two decimal places
token from Dallago et al. 2022I)
THERMO
AAV
DATASET
STABILITY
FLUORE.
1-vs-R
1-Vs-R
2-vs-R
3-Vs-R
CNN
0.51
0.67
0.34
0.48
0.17
0.32
0.83
RESNET
0.73
0.21
0.353
0.173
0.117
0.210
0.291
LSTM
0.69
0.67
0.317
0.215
0.124
0.349
0.491
CEM-UNTRAINED
0.452
0.337
0.36
0.01
0.05
0.05
0.46
ESM-IE
0.71
0.68
0.68
0.04
0.32
0.36
0.54
ESM-IV
0.726
0.507
0.18
0.32
0.32
0.77
PROTBERT-BFD
0.732
0.675
0.651
0.234
0.303
0.387
0,654
LSTM-MT
0.741
0,648
0.665
0.258
0.335
0.402
0.741
PROMPTPROTEIN (OURS)
0.767
0,683
0.694
0,551
0.403
0.550
0.783
dings. (4) Multi-task learning framework. We employ naive multi-task learning (MTL) and two optimization methods (GradNorm (Chen et al. (2188), RotoGram (Javaloy & Valera 2021
Results. We present the evaluation results of proposed PromptProtein and state-of-the-art baselines in Table [ Compared with all baselines, PromptProtein achieves new state-of-the-ard performance all tasks, which indicates that systematic modeling of multi-level structure information is bene- ficial. Although the multi-task learning baselines integrate the same information as PromptProtein, they cannot learn multiple information well and transfer properly downstream tasks. Their inferior performance in GO-BP and GO-CC suggests that there is gap between downstream task-desired representations and universal pre- trained representations. Flexible composing of structural informa- tion significantly improves the performance of the model for downstream tasks.
4.3 DOWNSTREAM TASKS: PROTEIN ENGINEERING TASKS
Datasets and Metrics. Protein engineering is regarded sequence regression task that, given protein, models are required to identify the functional strength, often termed the fitness landscape. Here, we employ five datasets (stability, fluorescence, thermostability, AAV, and GB1) coming from TAPE (Rao et al., 2019) and FLIP (Dallago et al. 2021) to evaluate whether the model can produce accurate quantitative predictions of these functions. We report the commonly-used Spearman's (rank correlation coefficient) to measure the degree to which the landscape was learned. Results of other tasks on FI.IP can found in Appendix
Baselines. For proteins without 3D structures, geometric methods cannot directly apply to these tasks. We choose sequence-bar based methods (CNN, LSTM, Transformer) and pre-trained protein methods (ESM-Ib, ESM-Iv (Meier et al.) (02T). ProteinBert-BFD. LSTM-MT) as baselines for protein engineering tasks. As Asaaalaa et al. (2021) purport that the various pooling choices perform inconsistently across datasets and splits, for fair comparison, utilize the mean pooling method to obtain protein representation.
Results. From Table 2 observe that PromptProtein obtains better performance than all baselines. It confirms that pre-training on struc- pro-
Table 3: Ablation of PromptProtein with different compo- nents.
METHOD
GB1
AAV
THERMO
tural objectives contributes tein engineering tasks and system- atic modeling of protein multi-level structure leads to further improve- ments. Note that LSTM-MT,, which leverages the tertiary structural in- formation to enhance protein repre- sentations, cannot surpass ESM-1b on all datasets, while our proposed approach obtains superior performances. This observation demonstrates that not all structural information leads to positive transfer and flexible utilization of structural information is the key to improved performance. Moreover, PromptProtein can obtain 17. .00%% improvement average in low-resource settings of the AAV and GB1 datasets, compared
CONVENTIONAL MTL.
0.238
0.525
0.651
PROMPTPROTEIN
0.279
0.544
0.672
ATTENTION MASK
0.264
0.531
0.663
LAYEER SKIP
0.270
0.520
0.659
.LMM OBJECTIVE
0.240
0.493
0.629
CRD OBJECTIVE
0.262
0.535
0.647
-PI OBJECTIVE
0.253
0.532
0.654

---

Published as a conference paper at ICLR 2023
10
20
30
33
1.0
[MLM]
1.0
-0.4
-0.7
[CRD]
-0.4
1.0
0.0
[PPI]
[PPi]
-0.7
0.8
1.0
-1.0
(MLM)
(CRD]
0.8
0.9
1.0
1.1
1.2
1.3
(a) Connection Skip Visualization
(b) Prompt Correlation
Figure 4: Skip connection visualization and prompt correlation. (a) We visualize the learned skip weight all neural layers. The darkness of block represents the weight of that block utilized for the given prompt. (b) We provide the Pearson's 's correlation between skip weights. The skip patterns between the [MLM] prompt and the other two prompts are negatively correlated, whereas the pattern between the tertiary and quaternary structures is positively correlated.
to the well-performed PTPM baselines. These results indicate that the prompt-guiding PTPM is better few-shot learner.
4.4 ABLATION STUDY
The ablation study is conducted to validate the effectiveness of designed modules in PromptProtein, ie., prompts, attention mask, skip connection. As illustrated in Table 3 the performance will decay if any one of the modules absent, demonstrating that all the modules are advantagcous. Furthermore, notice that skip connection contributes most to the performance, confirming the necessity reducing task interference.
4.5 ANALYSIS AND DISCUSSION
How do prompts determine the processing pathways of structural information?
In Figure a), we visualize the skip weights of three pre-trane pre-trained prompts at different neural layers, and compute the Pearson's correlation (Benesty et al. 2009) of these skip weights to measure the mutual correlations between the pre-training tasks (Figure ()). We have the following key obser- vations. (a) The skip weights are similar in the bottom layers (1-13) across all prompts, indicating all three tasks are processed by these layers. The MLM task information mainly acquired by the middle layers (14-29), whereas the CRD and PPI information is more acquired by the top layers (30-33). (b) We clearly observe that the [CRD) and [PPI] prompts are more correlated. This is con- sistent with the intuition that the tertiary and quaternary levels are 3D structures whose amino acids attend to spatially adjacent neighbors, resulting in similar skip weight patterns. Further analysis of the model layer be found in Appendix B.3
Can PromptProtein learn multi-level structures?
To examine whether prompt-guided pre-training learn multiple structure information, duct experiments to visualize the protein representations conditioned on different pre-trained prompt tokens. We t-SNE (van der Maaten Hinton, 2008) reduce the dimension of embeddings. Figure (aa) illustrates amino acid embeddings conditioned on [MMM] We observe that amino acid embeddings in a protein are grouped according to their type. Figure 5(b) illustrates amino acid em- beddings conditioned on [CRD]. We find that amino acids are linearly arranged in 2D space along their sequence the protein. To obtain more accurate relationship between representations and structures, we compare the protein contact map and the coordinate of embedding. The strong corre- lation between them demonstrates the CRD objective can effectively learn information about protein 3D structures. In Figure c). we visualize the amino acid embeddings with traditional multi-task pre- pre-training and highlight serine (a class of amino acids). The embeddings attempt to merge multi- ple structural features at the same time, which leads an unclear pattern. These results show that prompt-guided pre- pre-traine training mitigates task interference and allows the multiple structure information to be learned well, resulting promising performance.
8

---

Published as a conference paper at ICLR 2023
[MLM] Guided Embedding
[CRD] Guided Embedding Without Prompt-Guided Embedding
Axplainebe Bezwatest Diarterr
Yc
QQ
R
N
Hpdiophatti ageaeis
thargedianatt
Unagas
Hydrophetic aronviie
(a)
(b)
(c)
Figure 5: The comparison of amino acid embeddings with different learning methods. We visu- alize protein representations from prompt-guided multi-task pre-training in (a) and (b), and naive multi-task learning in (c). Each letter represents an amino acid and is colored according to the physicochemical properties in (a), and the secondary structure types in (b) and (c). The superscripts of letters represent the sequential number of amino acids from the C-terminal to the N-terminal.
Dittnce
ML Fedwesling
Rebensing
TAF
up
Amno Acid Sequence (TAR4)
(a)
(G)
Figure 6: Visualization of the difference of [MLM] and [PPI] prompts. The two proteins are Tran- scription initiation factor TFIID subunit4 (TAF4) and Transcription initiation factor TFIID subumit (TAFS). Left: Visualize the embedding of amino acids conditioned on [MLM] and [PPI] prompts (TAF4) by MDS. Middle: Visualize distances of corresponding amino acids in (a). Right: Visualize the amino acids with the most variation embeddings (red).
Furthermore, since the [PPI] prompt is trained provide quaternary structure information, ana- lyze what exactly the amino acid representations have changed. As shown in Figure 6[a), we firstly visualize the embeddings of amino acids of the TAF4 protein conditioned on [MLM] or [PPI] based on MDS (Kruskal 1964). Then we calculate the distances between two embeddings of the same amino acid and plot them Vigureo((b, We mark 30 amino acids with the most variation embed- dings in red (Figure (()). The observation that marked amino acids are all on the protein surface is consistent with fact that modeling the quaternary structure cares about the surface conformation, not the core (aan al. 2008).
5 CONCLUSION AND FUTURE WORK
In this paper, we extend the concept of prompts from NLP to protein representations. We propose the prompt-guided multi-task pre-training and fine-tuning framework. With this framework, we pro- pose three complementary pre-training structures to acquire multi-level structure information, and flexibly combine them for various downstream tasks. Experimental results on function prediction and protein engineering show that the proposed approach can produce satisfactory improvements when compared to the conventional PTPMs. The improvement is especially significant in low- interested in examining the effectiveness of the proposed
resource settings. In the future, prompt-guided multi-tase pre-training and fine-tuning framework domains where hierarchical task information required in the pre-training stage.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is funded by NSFC91846204//11BB22 and sponsored by CAAl-Huawei MindSpore Open Fund. We want express gratitude the anonymous reviewers for their hard work and kind comments and Hangzhou AI Computing Center for their technical support. Xurui Jin is the employee of the MindRank AI Ltd.
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):115-1322. 2019.
Ehsaneddin Asgari and Mohammad RK Mofrad. Continuous distributed representation of biological sequences for deep proteomics and genomics. PloS one, (0(11):e01410 2015.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Springer, 2009.
Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function.
Cell Systems, 12(6))654-669, 2021.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Πlya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick Riley, George M Church, Lucy Colwell, and Eric D Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology. 39(6):691-696, 2021.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing deep multitask networks. International conference on machine learning, pp. 794-803. PMLR, 2018
UniProt Consortium. Uniprot: the universal protein knowledgebase in 2021. Nucleic acids research, 49(00):D480--449, 2021.
Alperen Dalkiran, Ahmet Sureyya Rifaioglu, Maria Jesus Martin, Rengul Cetin-Atalay, Volkan Ata- lay, and Tunca Dogan. Eepred: tool for the prediction of the enzymatic functions of protein sequences based on the ec nomenclature. BMC bioinformatics, 19(1):1-13, 2018.
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, ((D1)):1140-DD DIL-D1144 40461146, 2014.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Towards crack- ing the language of lifes code through self-supervised deep learning and high performance com- puting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Charles J Epstein, Robert F Goldberger, and Christian B Anfinsen. The genetic control of tertiary protein structure: studies with model systems. In Cold Spring Harbor symposia on quantitative biology, volume 28, pp. 439 449. Cold Spring Harbor Laboratory Press, 1963.
10

---

Published as a conference paper at ICLR 2023
Octavian-Euger Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se (3)-equivariant models for end-to-end rigid protein docking. In International Conference Learning Representations, 2021.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):1-14, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Ko- zlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. intrinsic-extrinst convolution and pooling for learning on 3d protein structures. International Conference on Learning Represen- lutions, 2020.
Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, et al. Meltome atlas-thermal pro- teome stability across the of life. Nature methods, 17(5):495-503, 2020.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference on Learning Representations, 2021.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. International Conference on Learning Representations, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873);583-589) 2021.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 (5):922-923, 1976.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):557--263, 1983.
Jacob Devlin Ming-Wi Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAAC--HLT, pp. 4171- 4186, 2019.
Amit Kessel and Nir Ben-Tal. Introduction to proteins: structure, function, and motion. Chapman and Hall/CRC, 2018.
Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to nonmetric hypothe- sis. Psychometrika, 29(1):1 1-2,, 1964.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pp. 0882-597, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint mrXiv:9977 1692, 2019.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction the effects mutations on protein function. Advances in Neural Information Processing Systems, 34:29287-29303, 2021.

---

Published as a conference paper at ICLR 2023
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3781 2013.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Hakime Öztürk, Arzucan Özgür,, and Elif Ozkirimli. Deepdta: deep drug-target binding affinity prediction. Bioinformatics, 34(17)i821-i825) 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library, Advances in neural information processing systems, 32, 2019.
Predrag Radivojac, Wyatt T Clark, Tal Ronnen Oron, Alexandra M Schnoes, Tobias Wittkop, Artem Sokolov, Kiley Graim, Christopher Funk, Karin Verspoor, Asa Ben-Hur, et al. A large-scale evaluation of computational protein function prediction. Nature methods, 10(3):221-221 2013.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural infor- mation processing systems, 32, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15)c2016239111 2021.
Gabriel Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, al. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, 357 (6347):168-11)) 175,, 2017.
Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Ivankov, Nina Bozhanova, Mikhail Baranov, Onuralp Soyle- mez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397 401, 2016.
Timo Schick and Hinrich Schüitze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021.
Amir Shanchsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv preprint arXlv:2011.03443, 2020.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34:25244-25255 2021.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(11):1026-1028 2017.
Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consor- tium. Uniref clusters: comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 3 3166)922-933,, 2015.
Damian Szklarezyk, Annika L Gable, David Lyon, Alexander Junge, Stefan Wyder, Jaime Huerta- Cepas, Milan Simonovic, Nadezhda T Doncheva, John H Morris, Peer Bork, et al. String v1 1: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic acids research, 47(D1):D07-DD133 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, ((8)) 2008
12

---

Published as a conference paper at ICLR 2023
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property predic- tion. Scientific reports, 12(1):1-12, 2022.
Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes facilitated by indirect paths. Elife, S:e16965, 2016.
Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and improving information trans- fer in multi-task learning. In International Conference Learning Representations, 2019.
Changhui Yan, Feihong Wu, Robert L Jemigan, Drena Dobbs, and Vasant Honavar. Characterization of protein-protein interfaces. The protein journal, 27(1): 2008.
Kevin K Yang. Zachary Wu, Claire N Bedbrook, and Frances H Arnold. Learned protein embed- dings for machine learning. Bioinformatics, 34(15):6622-6453 2018.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. In International Conference on Learning Representations, 2021a.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations, 2021b.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint 220 2022.
13

---

Published as a conference paper at ICLR 2023
A MORE DETAILS OF PROMPTPROTEIN
A. PROMPT-GUIDED MULTI- TASK PRE- TRAININGG
One of the key problem to multi-task learning is what to share. Naive and gradient- methods try learn shared MTL model. To overcome between task interference between tasks, they adjust magnitude and direction of gradients. However, negative transfer between pre-training and downstream tasks cannot be mitigated. To realize the potential of positive transfer, multi-task pre- training requires to learn and use task differences on-demand. Both adapter-based approaches and proposed prompt-based approaches
learn task differences, whereas, for the flexibility of input, only prompt-based approach can use them on-demand. Figure 7 compares the mentioned multi-task methods.
Cly hy
Clyi. (b)h)
Cly.. Wul
Cla Su. I4,
Grad Clip
hOOOO
MTL model
MTL model
MTL model
MTL model
Adapter
xc
T
(b)
(c) Adapter-based
(d) Prompt-based (ours)
Figure 7: Comparison of multi-task pre- pre-Training Training.
A.2 PRE-TRAINING TASKS
In Figure we illustrate our proposed two additional pre-training tasks.
ApphonCaroon Coordinate Prediction. We use MLP network project protein embeddings into 3D space. To equip the model with 3D invariance, after predicting the protein coordinates, we first recenter the ground-truth coordinate Z and predicted coordinate 2 and then employ Kabsch algo- rithm (Kabsch 1976) to calculate the optimal rotation matrix that minimizes the root mean squared deviation. We first calculate cross-covariance matrix between two sets of coordinates:// Z 2. Then the covariance matrix can be decomposed by singular value decomposition: H = UEVT. The optimal rotation matrix R be formulated as: R == UVT.
Protein- Prottin Interaction Prediction. Since the limitation of GPU memory, it is not feasible to input two proteins the same sequence. Instead, we leverage the representations of proteins to calculate protein-pair attention in decoder. Then the pair-aware protein representations can be obtained by multiplication of protein representations and the attention.
A.3 ALGORITHMS FOR PROMPT-GUIDED MULTI-TASK PRE-TRAINING AND FINE-TUNING
To more easily appreciate the whole procedure of the prompt-guided multi-task pre-training and fine-tuning framework, we provide pseudo codes follows.
B ADDITIONAL DETAILS OF EXPERIMENTAL SETTING
B. PRE-TRAINING DATASET
To exploit primary structure information, language modeling has been prove effective (Elnaggar
cet al. 2021 Alley et al. 2019). We follow Rives et al. (202T) to use UniRef50 (Consortium) (021) dated March 28, 2018. 10% of UniRef50 clusters are randomly selected held-out evaluation set.
14

---

Published as a conference paper at ICLR 2023
Kabsh Rotation
ARFS Protein
Pair-aue Fair-aware Proteins
Representation
Centering
trpA Protein O
Predicted Structure
Linear Projection
Attention
C PPPI
C [CRDD]
Structure
Figure 8: The Overview Two Additional Pre- training Tasks.
Left:
Coordinate
Prediction. Right: Protein-Protein Interaction Prediction.
Algorithm 1: Prompt-Guided Multi-Task Pre-Training
Data:
Input protein æ, prompt pool p E P = [MLM) [CRD], [PPI]) } task objectives Lp.
the learning rate Ç. Result: Model parameters v/
while not converge do for P do
Initialize the task-specific input Tpp x
Compute the feature hp fa vb)
|| fg contain L layers Prompt-aware Attention Module Compute the loss (h,p) according to Equation 5
end for
Update the model parameters = 1h Ep(op (V.Cp) according Equation 6 Update the prompt parameters p p ap ppp.p Vp
end while
Algorithm 2: Prompt-Guided Fine-tuning
Data:
Input protein æ, downstream task object L' p. learned prompt pool P = [MLM], [CRD], [PPI]] pre-trained model parameters dh, the learning rate
Result:
Prompt-tuning module parameters 0.
while not converge do
Compute combined prompt p' = Te(p) according to Equation 7 Initialize the input Ep = x0pp'
Compute the feature hpp = (b)
Compute the loss Cp' = Lp' (h,')
Update the prompt-tuning module parameters 0 = 0 -VCVoLC
end while
15

---

Published as a conference paper at ICLR 2023
For secondary and tertiary structure information, extract data from Protein Data Bank (PDB) (Berman et al. 2000). For compatibility with pre-trained protein models, we only use pro- teins whose amino acid sequence length is less than 1,024. There many ways to define the coordinates of protein residues. Here we use the coordinates of carbon alpha atoms.
The pre-training dataset for quaternary structures is constructed based the latest STRING (Szklar- czyk et al. 2019) database with only the physical-only mode, which means edges between the pro- tein pairs indicate evidence of their binding or forming physical complex. The database contains in total 65 million protein sequences from 14,094 species and 2.7 billion protein-protein interaction pairs, Note that there no edge between proteins that come from different species.
We observe that the PPI network has a problem of uneven distribution, as illustrated in Figure 9 the largest network contains 60,000 proteins and 3.5 10 edges. Such data distributions can lead models to over-focus on proteins from a single species. We pre-process our dataset by choosing the species networks with comparable sizes. Figure T0 illustrates the data distribution after pre- processing.
Specie
Specte Num
14000
6000
3000
10000
8000
2000
4000
2000
Node Num
Edge Num
0
10000
20000
30000
40000
50000
60000
0.0
0.5
1.0
3.5
2.0
2.5
3.0
35
107
Figure 9: Visualization of the number of nodes and the number of edges in the original database.
Specie Num
Specie Num
600
400
300
400
250
200
150
200
300
0
20000
40000
60000
80000
Edge Num
Figure 10: Visualization of the number of nodes and the number of edges in the pre-processed database.
B.2 DOWNSTREAM TASK DATASETS.
The statistical results of the downstream datasets are shown in Table
Evaluation Metrics For multiple binary classification tasks, employ protein centric maximum F-score F max and pair-centric area under precision-red recall curve AUPRpair to evaluate protein mod- els. For regression tasks, we employ spearman's 's correlation p evaluate protein models.
16

---

Published as a conference paper at ICLR 2023
Table 4: Statistics of the downstream datasets.
DATASET
#TRAIN #VALIDATION
#TEST
TASK
ENZYME COMMISSION
15,551
1,729
1,919
CLASSIFICATION
GENE ONTOLOGY
29,902
3,323
3,416
CLASSIFICATION
STARILITY
53,679
2,447
12,839
REGRESSION
FLUORESCENCE
21,446
5,362
27,217
REGRESSION
THERMOSTABILITY
24,817
3,314
REGRESSION
AAV (I-VS-REST)
1,170
81,413
REGRESSION
GB1 (1-VS-REST)
29
8,704
REGRESSION
GB1 (2-VS-REST)
427
8,306
REGRESSION
SABDAB
345
48
99
REGRESSION
F maxx- Given target protein i, we denote its experimentally determined function terms T3. Given set of decision threshold E [0, 1], we denote predicted function terms as (t). The precision and recall of this protein can be formulated as:
precision, (() 4 E,
PT.
(8)
E
recall, T
-
where [[.] is an indicator function that is equal to 1 if and only if the condition is true. Combining these two measures, the max defined as the maximum value of F-measure:
F max max 2. precision(() precision(()-reaal(()) recall(()
precision(t) + recall(t) }.
(10)
where precision(t) M(O) Σn precision, (t), and precision(t) 3/4 Eg recall,tt) The
is denoted as the number of proteins and (t) is denoted as the number of proteins on which at least one prediction result is above threshold t.
AUPRpair- The pair-centric area under precision-recal curve is exactly the micro average precision score where precision and recall are for each term f:
precision (t) E, e P((
(11)
E.4 E P.(O]
recall, (t) = E, E P((
(12)
p. Spearman's 's is nonparametric measure of rank correlation for trouhd-truth Y and predicted Y landscape. We denote R() as ranks. The correlation coefficient is:
cov (R(Y),
(13)
where cov(-.)) is the covariance of the variables, and R is the standard deviations the rank variables.
Enzyme Commission and Gene Ontology. EC numbers are selected from the third and fourth levels of the EC tree, forming 538 binary classification tasks. GO terms are hierachically organized into three ontologies biological process (1943 binary classification tasks), molecular function (489 binary classification tasks), and cellular component (320 binary classification tasks). Following DeepFRI (Gligorijević al. 202T), use the protein sequences in the test set with 95% sequence identity to the training set.
Stability Landscape Prediction (Rocklin et al., 2017). This is regression task that maps each protein to a label, measuring the most extreme case where the protein maintains its fold above a concentration threshold. This task aims to test the ability to generalize from broad sampling of
17

---

Published as a conference paper at ICLR 2023
relevant sequences to local neighborhood of few sequences. The train set includes proteins from experimental design, while the test set contains single mutants.
Fluorescence Landscape Prediction (Sarkisyan et al. 2016). This is regression task that maps protein to a label corresponding to the flofluuoeeceenc intensity This task aims to test the ability to distinguish mutants. The train set includes triple mutants of the wild-type green fluorescent protein (GFP), while the test set contains more mutants.
Thermostability Landscape Prediction (Jarzab et al., 2020). This is regression task that maps a protein to thermostabillity label. We adopt the mixed split proposed by Dallago et al. (2021) that using MM-seqs2 (Steinegger & Söding 2017) threshold of 20% sequence identity creates split. The train set includes all sequences in 80% of clusters, while the test contains the remaining 20% of clusters.
Adeno-associated virus (AAV) Landscape Prediction (Bryant et al. 2021). This is regression task that predicts fitness for long mutated sequence. We adopt the 1-vs-rest split, where wild type and single mutants are assigned to train set, while test set contains the rest. This split are common protein engineering application.
GB1 Landscape Prediction (Wu et al. 2016). This is regression task that predicts the effects of interactions between mutations. We adopt the 1-vs-rest (and 2-vs-rest) split, where wild type and single mutants (and double mutants) are assigned to train set, while test set contains the rest.
Antibody-antigen Affinity Prediction (Dunbar et al., 2014). This a regression task that takes pair of proteins as input and predicts the affinity between them. We adopt random split which contains 493 pairs, 431 antibodies and 401 antigens.
Table 5: Model performance on FLIP benchmark.
THERMO
AAV
GB1
DATASET
MIXED
HUMAN
HUMAN- CELL
I-vs-R
2-v5-R
1-vs-R
2-v5-R
3-V5-R
LOW-VS-HICE HIGH
ESM-18
0.68
691)
0.75(0.673)
0.04
0.26
0.32
0.36
0.54
0.13
OURS
0.683
0.702
0,684
0.551
0.395
0.403
0,550
0.783
0.294
To illustrate the advantage of prompt-tuning in low-resource scenarios, we only selected subset of tasks in the FLIP benchmark. In Table 5 we report the performance of our model on other tasks. Note that, although we use the reported results of esm-1b 1b in the above table, we additionally provide the reproduced results of esm-lb on Thermo(Human) and Therno(Human-cell), These values (surrounded by brackets) are lower than reported.
0.10
0.08
0.06
0.04
0.02
0.00
Figure 11: Attention visualization. We select GBI protein as an example and visualize attentions of the 15-th layer (high skip weight for (MLM) and the 33-th layer (high skip weight for [CRD] and [PPI]).
18

---

Published as a conference paper at ICLR 2023
B.3 ANALYSIS NEURAL LAYERS
In Figure a)) we visualize the attentions in the 15-th layer (a high skip weight for [MLM] and the 33-th layer (a high skip weight for [CRD] and [PPI]. We observe that amino acid in the 15-th layer can only attend to the local neighbors in the sequence, whereas the amino acid in the 33-th layer can attend those amino acids that are more distant along the sequence but potentially closer in the 3D space. This observation demonstrates the primary structural knowledge learned by MLM pays more attention to sequential dependency, whereas the tertiary structural and quaternary structural knowledge learned by CRD and PPI tasks can capture the information from adjacent amino acids in the 3D space.
B.4 ADDITIONAL EXPERIMENT RESULTS
Do downstream tasks benefit from the acquired information on-demand by prompt tuning?
To further analyze the importance of prompt-guided fine-tuning, we conduct an ablation study on the binding affinity prediction task on the SAbDab dataset (Dunbar al., 2014). From Figure [2 we observe that PromptProtein performs worst without any prompt tokens. In contrast, adding either of the three prompt tokens, especially the token corresponding to the PPI task, can significantly im- prove performance. By combining different prompts without prompt tuning, we can obtain protein representations enhanced by multiple structural information. By doing that, find the combination of the [MLM] and [PPI] prompts empowers PromptProtein to achieve the best performance. It is also notable that, by comparing the results of adding [MLM] and [PPI] prompts and adding prompts, the [CRD] prompt leads to performance decrease. These results evidence that not all structure in- formation from pre-training is beneficial for downstream tasks, and adaptively combining acquired information via prompt-tuning leads to better performance.
Binding Affinity Prediction Task (SAbDab)
0.62
Spearman's Correlation
0.60
0.58
0.56
0.54
0.52
0.50
None
CRD
MLM
PPI
PR1-CRD
MUM+CRD MLM+PPI
ALL
Figure 12: Ablation of PromptProtein with different prompt tokens on SAbDab (spearman's
19


Reviews:
[{"review_id": "umyiNhlKrhy", "reviewer": "Reviewer_rpyU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a novel prompt-guided multi-task pre-training and fine-tuning framework for protein structure pre-training. Multi-level supervised information, including masked language modeling (MLM), CA coordinate prediction (CRD), and protein-protein interaction (PPI), are integrated into one unified learning framework without interfering with each other. Extensive empirical evaluation and embedding visualization validate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed prompt-based learning framework for protein structure pre-training is novel, and indeed resolves the possible interference between different learning tasks. A few minor issues need to be resolved in the author feedback to further improve the overall quality.", "strength_and_weaknesses": "Pros:\n1. The overall motivation is well founded and executed. Combining different level of supervised information for protein structure pre-training indeed leads to more informative embeddings, and the prompt-based multi-task learning avoids possible interference between different tasks. The visualization in Figure 4 and 5 provides clear evidence to support the proposed learning framework.\n2. Extensive experiments are conducted on two types of down-stream tasks, function annotation and protein engineering. Multiple benchmark datasets are used, and many closely-related baselines are included for comparison, which makes the empirical evaluation hightly convincing.\n\nCons:\n1. It seems that some details are missing (please correct me if I failed to find them in the manuscript). This includes the design of structure predictor (a MLP network?), and the prompt tuning module (linear combination of MLM/CRD/PPI prompts with learnable coefficients?).\n2. In Equation (2), it looks like that the linear projection of prompt is layer independent, i.e., the weight is constant in all the layers, which conflicts with Figure 4. Please clarify.\n3. Currently, the CRD task corresponds to the second and third levels of protein structures. Since the second level of protein structures refers to secondary structures (alpha-helix and beta-sheet), it may be more natural to introduce secondary structure prediction as learning task for this level. The training data can be the same as the CRD task, but with different supervised information.\n4. Authors mentioned that the average error of coordinate prediction task on a single residue is 5A, which may indicate that the model is well trained to minimize this loss. This could be caused from the relatively simple design of structure predictor (if it is a MLP network), which I assume may not work well if the amino-acid sequence is long (e.g. more than 100 amino-acid residues). It should be helpful to use more sophisticated structure predictors (e.g. AlphaFold2’s structure module), but this may be out of the scope of this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tiHp30iMpg", "reviewer": "Reviewer_iqu6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a prompt-guided multi-task pre-training and fine-tuning framework for protein function prediction.  The authors borrow the techniques from NLP and successfully applied to two tasks and show promising performance. ", "summary_of_the_review": "The work successfully transfers NLP techniques to protein study.  The choice of the techniques including prompt-guided multi-task pre-training and fine-tuning are tailored to apply onto function prediction and protein engineering.  The work is thorough and thought provoking.  \n\n", "strength_and_weaknesses": "Strengths\n\n1. The paper is well organized and written.\n2. Some NLP techniques including pre training and prompt tuning are applied to protein structure study.\n3. Empirical study shows good results.\n\nWeaknesses\n\nI would not say this to be a weakness.  The authors lend key techniques from NLP, but protein structure has four distinct levels.\nWhich levels are being helped by NLP techniques, and which levels are not?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rWv92nnN39O", "reviewer": "Reviewer_9Vnq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces PromptProtein, a prompt tuning-based protein language model that combines pre-training tasks at the sequence, structure and protein-protein interaction levels. The model is then fine-tuned on function annotation and protein engineering downstream tasks.\n\nIn a nutshell, PromptProtein introduces learnable prompt embeddings for each of the three pre-training tasks. During fine-tuning, a small network learns to combine the three task-specific prompt embeddings into a prompt embedding that is specific to the downstream task at hand. The authors also propose two minor changes to the transformer architecture. Namely:\n+ Attention from prompt tokens to sequence tokens is masked.\n+ A learnable gate that is a linear function of the prompt token is applied to the skip connections.\n\nResults on both function annotation and protein engineering downstream tasks are favourable relative to the baselines.\n", "summary_of_the_review": "Within the extremely active field of protein language models, this paper introduces the following changes over the baselines:\n\n1. While many baselines (e.g. ESM) pre-train only on sequences, the proposed approach pre-trains on three different tasks: masked language modelling on sequences, structure prediction and protein-protein interaction prediction.\n2. The proposed approach relies on prompt tuning for (1) fine-tuning and (2) to allow for a degree of task specialisation at pre-training time.\n\nWhile these changes are minor from a purely methodological viewpoint, I believe they are of interest to the protein language modelling literature. Moreover, these innovations appear to translate into improved downstream performance in function annotation and protein engineering tasks.\n\nHowever, this work is not without limitations. Most notably, in my opinion, the experiments currently have two main shortcomings:\n\n1. For most of the FLIP tasks (Thermostability, AAV, GB1), the authors show results only for a small subset of task variants. Instead, performance should be shown for *all* of the variants in the FLIP benchmark to prevent any potential cherry-picking.\n2. The ablations do not provide sufficiently compelling evidence that the performance improvements are due to the specifics of PromptProtein and not e.g. the use of prompt tuning in general.\n\nTo this end, I recommend:\n1. Providing experimental results for all subtasks in the FLIP benchmark.\n2. Extend the ablation study to all tasks instead of GB1 (2-vs-rest) only.\n3. To study the importance of pre-training with different prompt tokens for each task, compare to an ablation baseline that has a single learnable prompt of length three that is shared across the three pre-trained tasks, as opposed to three task-specific prompts of length one. This would help understand to which extent the performance improvement may be simply due to the extra expressivity in the skip-connections or, most importantly, using prompt tuning for fine-tuning to the downstream tasks as opposed to e.g. full model fine-tuning.\n4. To study the importance of multi-task pre-training, results should be shown for ablation baselines trained without each of the three tasks, one at a time.\n\nBecause of this, I slightly lean towards weak rejection of the manuscript for now, but strongly encourage the authors to address these issues in the rebuttal.\n", "strength_and_weaknesses": "Strengths:\n+ Good results on function annotation and protein engineering tasks.\n\nWeaknesses:\n+ Results shown only for a subset of sub-tasks in the FLIP benchmark.\n+ Ablation experiments are insufficient to disentangle the importance of the proposed architectural and task-related changes.\n+ Insufficient details for full reproducibility in the manuscript.\n\nOther weaknesses:\n+ Limited methodological novelty (not critical for an application-focused paper).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Uu7jUXPYmh", "reviewer": "Reviewer_M4p8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a prompt-based multi-task framework for pre-training and fine-tuning protein sequence representations. From a methodological standpoint, the paper adapts the prompt fine-tuning idea (ie., a differentiable continuous prompt token is pre-trained instead of using a discrete prompt) from the NLP literature to protein modeling with large transformer networks. It then makes two contributions over the standard transformer architecture 1) A specific masking scheme for the prompt token is used to keep only the effect of the prompt on the input sequence and eliminate the opposite effect (ie., the prompt should be task-dependent and not sample-dependent) 2) Learned task-specific layer-specific skip connection linear projections to let the network learn different weights for each task at each layer. The multi-task pre-training involves three kinds of pre-training: a) masked-language modeling with sequence-based information only (MLM) b) Prediction of the alpha-carbons positions (CRD) c) prediction of protein-protein interactions (PPI). Experiments on function annotation and protein engineering (FLIP benchmark) demonstrate the benefits of the different ideas introduced in this work.", "summary_of_the_review": "This paper is aiming to address a very important area in protein modeling: learning rich sequence embeddings by leveraging multiple pre-training tasks jointly. The approach is sound and the methodological section is overall very clearly presented. There are a few concerns with respect to some of the results and claims as detailed above. Given the several strengths of the work, I would be willing to increase my score if authors address these points during rebuttal.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------\n[UPDATE POST REBUTTAL]\n\nMy most important concerns have been alleviated during rebuttal and I have increased my score accordingly.", "strength_and_weaknesses": "**Strengths**\n- The idea of combining different pre-training tasks seems very sensible for proteins given the diversity of modalities that characterize them (eg., via their primary, secondary, tertiary and quaternary structure). As noted by the authors, there is however a high risk of task interference when one wants to obtain pre-trained representations that combine these different modalities / tasks. The approach suggested by authors appears to be doing a fine job at efficiently combining these different modalities given the performance reported in sections 4.2. and 4.3.\n- The introduced prompt masking and skip connections both appear to be critical to strong empirical performance — the latter seems to be particularly important to mitigate task interference as evidenced by the ablations in section 4.4.\n- The paper is very clear overall (in particular the methodology section) with nice visuals facilitating understanding and additional analyses in section 4.5 to help investigate the source of the performance lift.\n\n**Weaknesses**\n- The ablations for the different pre-training tasks in section 4.5 / Figure 6 are a bit puzzling. It does seem that the CRD task has destructive value on that particular binding affinity prediction task since: a) the performance of CRD + MLM or CRD + PPI leads to both lower performance Vs MLM or PPI alone respectively b) the performance of CRD + MLM + PPI is also lower vs just using MLM + PPI. This seems particularly important from a practical standpoint, and additional experiments are needed to confirm whether: 1) that problem applies to other downstream tasks or is just specific to binding affinity prediction — and if so, why?  2) there is something fundamentally wrong with the CRD pre-training as currently implemented? 3) there is a way to anticipate ex ante (or post fine tuning) which tokens should be used to ensure optimal task performance ?\n- The ablation in section in Table 3 is a bit puzzling as well: it appears that the performance of PromptProtein without layer skip is lower than the performance from the conventional MTL. Could you please explain why that might be the case? (I would have assumed intermediate performance between conventional MTL and full PromptProtein as I presume the attention masks are still used in that ablation?)\n- Several points (in section 4 primarily) were not fully clear (see clarity paragraph below).\n- The following claim in conclusion does not seem fully substantiated: “PromptProtein beats state-of-the-art baselines by significant margins”. Authors do report the relevant baselines listed in the FLIP paper [1]. But since that paper was released, several methods have shown markedly superior performance for protein modeling & achieving high spearman with deep mutational scanning assays — see for example, [2] and [3]. I would suggest adding these two baselines to the analysis or tone done the SOTA claims.\n\n\n------------------------------------------------------------------------------------------------------------------------\n[1] Dallago, C., Mou, J., Johnston, K.E., Wittmann, B.J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K.K. (2022). FLIP: Benchmark tasks in fitness landscape inference for proteins. bioRxiv.\n\n[2] Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B.L., Sercu, T., Lerer, A., & Rives, A. (2022). Learning inverse folding from millions of predicted structures. bioRxiv.\n\n[3] Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A.N., Marks, D.S., & Gal, Y. (2022). Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. ICML.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster