Title: Linker-Tuning: Optimizing Continuous Prompts for Heterodimeric Protein Prediction
Venue: NeurIPS 2023
Accepted: no
Keywords: protein structure prediction

protein language models

parameter-efficient training
Abstract: Predicting the structure of interacting chains is crucial for understanding biological systems and developing new drugs. Large-scale Pre-trained Protein Language models (PLMs), such as ESM-2, have shown an impressive ability to extract biologically meaningful representations for protein contact and structure prediction. In this paper, we show that ESMFold, which has been successful in computing accurate atomic structures for single-chain proteins, can be adapted to predict the heterodimer structures in a lightweight manner.
We propose Linker-tuning, which learns a continuous prompt to connect the two chains in a dimer before running it as a single sequence in ESMFold.
Experiment results show that our method is significantly better than the ESMFold-Linker baseline, with relative improvements of +28.13\% and +54.55\% in DockQ score on the i.i.d heterodimer test set and the out-of-distribution (OOD) test set HeteroTest2, respectively. Notably, on the antibody heavy chain light chain (VH-VL) test set, our method successfully predicts all the heavy chain light chain docking interfaces, with 46/68 medium-quality and 22/68 high-quality predictions, while being $9\times$ faster than AF-Multimer.

Abstract

Predicting the structure of interacting chains is crucial for understanding biological systems and developing new drugs. Large-scale Pre-trained Protein Language models (PLMs), such as ESM-2, have shown an impressive ability to extract biologically meaningful representations for protein contact and structure prediction. In this paper, we show that ESMFold, which has been successful in computing accurate atomic structures for single-chain proteins, can be adapted to predict the heterodimer structures in a lightweight manner. We propose Linker-tuning, which learns a continuous prompt to connect the two chains in a dimer before running it as a single sequence in ESMFold. Experiment results show that our method is significantly better than the ESMFold-Linker baseline, with relative improvements of +28.13% and +54.55% in DockQ score on the i.i.d heterodimer test set and the out-of-distribution (OOD) test set HeteroTest2, respectively. Notably, on the antibody heavy chain light chain (VH-VL) test set, our method successfully predicts all the heavy chain light chain docking interfaces, with 46/68 medium-quality and 22/68 high-quality predictions, while being 9 × faster than AF-Multimer.

1 Introduction

Proteins are large biomolecules essential to life. They are sequences compromised of 20 types of amino acids and fold into three-dimensional (3D) structures to carry out functions. Predicting the 3D structures of proteins from amino acid sequences is a long-standing challenge in computational biology. It is important for the mechanical understanding of protein functions as well as for designing new drugs. In 2021, AlphaFold2 (AF2) strikes a huge success in solving this challenge, achieving near experimental accuracy on protein structure prediction [1]. However, this system heavily relies on Multiple Sequence Alignments (MSAs) to extract the evolutionary information, but MSAs are not always available or high quality, especially for orphan proteins and fast-evolving antibodies [2].

Inspired by the success of transformer language models in the field of Natural Language Processing (NLP), there is a line of work resorting to large-scale PLMs for protein structure prediction [2, 3, 4, 5]. These PLM-based models, such as ESMFold [3], take only amino acid sequences as input, eliminating the need for MSAs. Powered by PLMs, they show strong abilities in capturing protein structure information [6, 7]. And they are able to predict protein 3D structures at the atomic level with high accuracy while being an order of magnitude faster than AF2. However, these models are developed for predicting the structures of single-chain proteins and it is not clear how to use them to predict multi-chain protein structures.

To adapt these models for protein complex prediction, some researchers have proposed to use a 33 poly-Glycine linker to join chains and input the linked sequences to the model to predict complex 34 structures [8, 9]. The rationale is that the model should identify the linker segment as unstructured 35 and fold the linked sequence in a similar way to multiple chains. Experimental result on AF2 shows 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

that this approach is simple yet effective. However, for the PLM-based models, whether a linker is effective or not for protein complex prediction remains unexplored. In the work of ESMFold, they briefly mention that they use a 25-residue poly-Glycine linker (denoted as G25 in the following) to join different chains for a specific protein complex example [3]. But they do not test the performance of the linker systematically. Based on existing work, we would like to investigate the following questions in this paper: 1) How well can a G25 linker perform on protein complex prediction? 2) Can we optimize the linker to achieve a better result? And how?

Viewing proteins as the language of life, linkers in fact are the same things as prompts in natural language. Inspired by prompt engineering [10, 11] in NLP, we propose Linker-tuning, which is to automatically learn a linker for the PLM-based model ESMFold on the task of heterodimeric protein structure prediction. Our goal is to find a linker that can link the two chains of a heterodimer so the structure prediction model can fold it in a similar way as a single-chain protein. How to best achieve this goal, however, is non-trivial and remains under-explored for the complicated protein structure prediction model. Through preliminary analysis, we find that it is better to place linker optimization at the Folding Module instead of at the PLM, which is different from intuition.

Considering ESMFold is a model with large-scale pre-trained PLM ESM2 that scales up to 15B parameters, to accelerate the linker learning procedure, we train and select our model on a proxy task called distogram prediction [12], a task that aims to predict inter-residue distance bins in the 3D space for each pair of residues in a given protein. After training, we test our learned linker on the 3D structure prediction task on three datasets to investigate the generalization ability of our method.

In summary, our main contributions are as follows:

We propose Linker-tuning, a lightweight adaptation method that automatically learns a linker in the continuous space to adapt the single-chain ESMFold for heterodimer structure prediction.

We show that our method outperforms the ESMFold-Linker baseline by large margins on both contact and structure prediction tasks on the heterodimer test set.

Wefind that our method generalizes well to predict heterodimers with low sequence similarity and antibody VH-VL complex.

2 Biological background

Linker In biology, linkers are short amino acid sequences created in nature to separate multiple domains in a single protein [13]. Biologists have found that linkers rich in Glycine act as independent units and do not affect the function of the individual proteins to which they attach [14, 15]. Therefore, we can use the Glycine-rich linker to join interacting chains to make it a single sequence, hoping it folds in the way they suppose to. Grounded in biological principles, we further extend the natural discrete linkers to virtual continuous linkers for better protein complex structure prediction.

Distogram and contact map The 3D structure of a protein is expressed as ( x, y, z ) coordinates of the residues' atoms in the form of a pdb file [16]. The distance between two residues in a protein 3D structure is defined as the Euclidean distance between their C β atoms ( C α for Glycine). Binning all the inter-residue distances in a protein into k distance bins, we can obtain the distogram matrix [12]. For a protein with L residues, the distogram d is an L × L matrix, with entry d ij referring to the distance category of residue i and j . In a coarser granularity, we can compute the contact map c ∈ R L × L , where c ij = 1 means the distance between residue i and j is less than or equal to 8Å. For protein complexes, we are especially interested in the inter-chain contact maps where the contacts are formed by two residues from different chains. The inter-chain contact map reflects the interface of interacting proteins, which is essential for predicting the 3D structure of the complex.

3 Related work

3.1 Protein structure prediction

Single-chain protein structure prediction In recent years, single-chain protein structure prediction 84 has attracted increasing attention from researchers in the Artificial Intelligence (AI) community, 85

mainly due to the ground-breaking success of the deep learning model AF2. Deep learning based 86 protein structure prediction methods can be classified into two main categories: 1) MSA-based 87 methods, such as AF2, that take protein sequences and MSAs as input and predict 3D structures 88 [1, 17, 18]; 2) PLM-based methods, such as ESMFold, that take only protein sequences as input 89 and predict 3D structures [3, 2, 4, 5, 19, 20, 21, 22, 23]. PLM-based methods do not rely on MSAs, 90 which are time-consuming in searching homologs and not always available for some proteins like 91 orphan proteins. Instead, they adopt large-scale pre-trained PLMs to learn evolutionary and structural 92 meaningful representations for 3D structure prediction. In this work, we build our method upon PLM93 based methods. Specifically, we adopt ESMFold [3] as the backbone since its code and pre-trained 94 weights are all released and convenient to use. The overall architecture of ESMFold contains two 95 parts: 1) ESM2 : a PLM pre-trained with masked language modeling objective and scales up to 15B 96 parameters; 2) Folding Module : contains Folding Trunk (similar to Evoformer in AF2) and Structure 97 Module (same as the one in AF2), which are responsible for structure folding. 98

99

100

101

102

103

104

105

106

107

108

109

110

111

Multi-chain protein structure prediction In biology, multi-chain proteins are protein complexes formed by interacting single-chain proteins where the interactions are driven by the same physical forces as protein folding [24]. Recently, there is a line of work repurposing single-chain AF2 for protein complex structure prediction. The methods can be summarized into two main categories: 1) input-adapted methods that provide AF2 with pseudo-multimer inputs either by adding a large number to the residue_index between chains to indicate chain break [25, 26, 27, 28] or using a linker to join chains [8, 9]; and 2) training-adapted methods that retrain AF2 on multimeric proteins, such as AF-Multimer, the state-of-the-art (SOTA) method [29]. On the one hand, the two types of methods either do not update any parameters, or update all parameters of the base model, while our method falls in between, adding only a tiny number of extra parameters to the base model. On the other hand, existing work mainly focuses on the MSA-based method AF2, with little attention being paid to the PLM-based methods. In this work, we focus on adapting the PLM-based methods for two-chain protein structure prediction, which has not yet been explored.

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

3.2 Prompt engineering

In the NLP community, with the rise of large-scale pre-trained language models (LMs) such as GPT-3 [30], 'pre-train, prompt, and predict" has become a prevalent paradigm to steer the LM to perform a wide range of downstream tasks [10]. In this paradigm, the downstream tasks are reformulated in a form that is similar to the LM pre-training task using a textual prompt [30, 31]. The key challenge in prompt-based learning is to find the right prompt for a specific task, termed 'prompt engineering". There is a line of work that automatically search the right prompts for downstream tasks [32, 33]. In particular, instead of natural language prompts, some researchers propose to use continuous prompts, directly performing prompting in the embedding space of the LM [34, 11]. In their experiment, continuous prompts achieve strong results in both language understanding and generation tasks. In this work, we follow the idea of continuous prompting, searching for the linkers in the continuous space.

4 Method: Linker-tuning

To adapt the single-chain model for multi-chain protein structure prediction, we propose a lightweight adaptation method called Linker-tuning and a novel weighted distogram loss. The basic idea of our method is to optimize linkers, i.e., prompts, in the embedding space of ESMFold.

4.1 Problem formulation

Continuous linker tuning of ESMFold for protein complex structure prediction is a continuous optimization problem. Our goal is to find a linker that maximizes the performance of ESMFold on protein complex prediction. To be specific, we first denote training data as D train = { ( x 1 , y 1 ) , ..., ( x n , y n ) } where x i = ( x A i , x B i ) and x A i , x B i represent the amino acid sequences of two chains, y i is the structure of protein x i . For a specified linker length L , the linker optimization problem is defined as follows:

134

135

136

137

seq rep r

predicted distogram

ESM2

Distogram

Head

64 bins

r:=r + E(x)

weighted

distogram loss

E

EL

x

AECDGG...GGHIJKK

AECDt1t2 ..tLHIJKK

Input seq:AECD:HIJKK

Chain A Linker Chain B

true structure

true distogram

B. Inference

Recycling

ESM2

FoldingModule

Distogram

ESMFold-Linker*

E

EL

Head

AECDGG...GGHIJKK

where l denotes a linker, E L ⊂ R L × d denotes a specific embedding space with embedding dimension of d , L ( x i , y i , l ) denotes complex structure prediction loss w.r.t. protein ( x i , y i ) using linker l . Therefore, the linker optimization is placed at the task level instead of at the instance level.

4.2 Model architecture

Our method is implemented based on ESMFold, a PLM-based strong structure prediction model. 138 As shown in Figure 1, we place the continuous linker at Folding Module of ESMFold, which takes 139 both the sequence representation from ESM2 and the amino acid sequence as input. There are two 140 main reasons that motivate us to place the continuous linker at Folding Module instead of at ESM2. 141 First, we can utilize the pre-trained distogram head while avoiding backpropagating to the giant 142 ESM2 model. If we put it on the ESM2 side, the combined depth of training will go up to 104 143 layers, making it easily suffer from gradient vanishing and exploding. Second, preliminary analysis 144 on inter-chain contact prediction (shown in Table 4) shows that using Folding Module on top of 145 ESM2-3B increases prediction precision dramatically over ESM2-3B while ESM2-3B just performs 146 slightly better ESM2-650M, implying that Folding Module is more sensitive to structure prediction 147 and easier to control. 148

We implement a plug-in linker embedding module, which contains L × d learnable parameters where 149 d is the embedding dimension of Folding Module. During training, only the linker embedding module 150 is trainable, while all the original parameters in ESMFold are frozen. Therefore, ESM2 is just a 151 sequence feature extractor that generates features for Folding Module. As shown in Figure 1(A), we 152 first use a poly-Glycine linker of the same length as the continuous linker to join different chains for 153 the ESM2 input. Then we obtain the protein sequence representation and input it to Folding Module 154 along with the chains connected by the continuous linker. Finally, the distogram head outputs a 155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

probability distribution p D ij ∈ R 64 of each residue pair ( i, j ) on 64 distance bins, which is used for computing the loss function. After training, we view ESMFold and the linker embedding module as a whole and name it as ESMFold-Linker*. As shown in Figure 1(B), it can be used to predict the distograms as well as the 3D coordinates of all the residues for multi-chain protein sequences.

4.3 Weighted distogram loss

Intuitively, to predict the structure of a protein complex, we need to know two things: 1) the structures of each chain, on which ESMFold has been trained; and 2) the interaction interface between chains, which ESMFold has never seen before. Therefore, we propose to weight the intra-chain predictions and inter-chain predictions differently, with a focus on learning better interface between chains.

Formally, let N A , N B be the number of residues in two chains in a protein complex, N = N A + N B be the total number of residues in the protein complex. Let y ij ∈ R 64 denote the one-hot labels of the 3D space distance bins between residue pair ( i, j ) and p ij ∈ R 64 be the corresponding predicted probability. We define a weighted distogram loss for a protein complex as follows:

where L 1 ( ., . ) denotes the single-chain distogram loss given as follows:

and L 2 ( x, y, l ) denotes the inter-chain distogram loss defined as follows:

and λ ≥ 2 is a hyperparameter controlling the attention we place on the interface of a protein complex. In our method, we use the weighted distogram loss as the training objective and validation metric.

5 Experiments

5.1 Experiment setting

Datasets We mainly perform experiments on heteromers of two chains. For training, we use the dataset from APOC [35], which contains heterodimers released in the Protein Data Bank (PDB) before 2018-09-30. After filtering out similar sequences at a 40% sequence identity threshold, it is split into train/valid/test 1 sets by CDPred [36]. We further filter out those proteins that contain missing C β coordinates ( C α for Glycine) in the pdb file. The resulting train/valid/test sample sizes are 2,946/193/172, respectively. The average number of residues in the test set is 367, with a maximum of 998. Furthermore, we use the largest blind test set HeteroTest2 2 from CDPred, which contains 55 heterodimers released in PDB between 2021-09-01 to 2021-10-20 [36]. The average number of residues is 505, with a maximum of 979. In addition, we use the antibody VH-VL test set from XtrimoDock [37]. It contains 68 samples released in PDB after 2022-02-01. Each sample consists of one heavy and one light chain, forming the fragment variable region (Fv), which is a critical part of antigen binding. The average number of residues is 231, with a range of [223, 244].

Models We use ESMFold-v1 3 as our backbone model. ESMFold-v1 consists of a 3B ESM2 model and a 670M Folding Module, which is the largest yet publicly available ESMFold checkpoint. For the Linker-tuning method, the linker length L is set to 25, equal to the length of the manual poly-Glycine linker. So the plug-in linker embedding module contains 0.027M parameters. We initialize the linker embedding using the embedding of Glycine. During training, only the linker embedding module is trainable, while all the original parameters in ESMFold are frozen. The hyperparameter λ in the

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

weighted distogram loss is set to 4. We train the model on a single Nvidia A100 80GB GPU with batch_size=1 and num_epoch=15. The protein sequences in the training set are cropped to 225 residues to fit in GPU memory using the multi-chain cropping algorithm from AF-Multimer [29]. The number of recycles is set to 1 during training to reduce computation. We use Adam optimizer with a learning rate of 5e-4. We select the best model based on the validation weighted distogram loss. During inference, the number of recycles is set to 3.

Baselines We compare our method with several baselines and one SOTA model as follows:

ESMFold-Linker: ESMFold-v1 with chains joined by the G25 linker as input.

ESMFold-Gap: ESMFold-v1 with residue_index_offset set to 512.

AlphaFold-Linker [29]: AF2 with a 21 residue repeated Glycine-Glycine-Serine linker.

HDOCK [38]: rigid docking with single chains predicted by AF2.

AF-Multimer(v3 best) [29]: AF-Multimer contains five models that are trained on all protein structures released in PDB before 2021-09-30. We take the best prediction from the five AF-Multimer models.

Metrics For protein complex 3D structure prediction, we use DockQ [39] to evaluate the quality of the predicted interfaces. As defined by Critical Assessment of PRediction Interactions (CAPRI), interfaces with DockQ < 0 . 23 means incorrect prediction, interfaces with 0 . 23 ≤ DockQ < 0 . 49 means acceptable prediction, 0 . 49 ≤ DockQ < 0 . 80 means medium quality prediction, and DockQ ≥ 0 . 80 means high-quality prediction. To evaluate the whole predicted protein complex structure rather than the interfaces, we adopt two commonly used global structure metrics, namely, Root Mean Squared Deviation (RMSD), and Template-Modeling Score (TM-Score) [40]. Besides, we use the topk precision as an evaluation metric for inter-chain contact prediction. We set k = N s / 5 , where N s is the minimum chain length for a given protein complex.

5.2 General heterodimer structure prediction

Table 1 shows the protein complex structure prediction results of our methods and the baselines on the heterodimer test set and HeteroTest2. On the i.i.d. heterodimer test set, ESMFold-Linker achieves a 0.32 DockQ score and a 0.76 TM-score on average. By optimizing the linker, our model, i.e., ESMFold-Linker*, achieves a 0.36 DockQ score and a 0.79 TM-score on average on the same test set, outperforming the ESMFold-Linker baseline by 13.61% and 3.28%, respectively. Interestingly, the gain of the interface quality (13.61%) is much larger than the gain of the whole structure quality (3.28%), indicating that our learned linker mainly improves the interfaces more than the overall structures. We further improve the ESMFold-Linker* by incorporating a large chain break, which adds a large number to the residue index in Folding Module. And the model ESMFold-Linker*Gap achieves a 0.41 DockQ score and 0.80 TM-score, outperforming ESMFold-Linker by 28.13% and 5.26%, respectively. On the OOD test set HeteroTest2, we observe similar results. ESMFoldLinker*-Gap surpasses ESMFold-Linker by 54.55% DockQ score and 4.82% TM-score, respectively, suggesting that our learned linker can generalize well to OOD data.

Compared to AlphaFold-Linker, a model that takes linked sequences and MSAs as input, our best model ESMFold-Linker*-Gap achieves similar DockQ scores on both test sets, with lower values in RMSD. Meanwhile, it outperforms the classic docking method HDOCK with AF2 predicted chains as input in terms of DockQ score and RMSD. Furthermore, we compare it with the SOTA model AF-Multimer. 4 From Table 1, we can see there is still a large gap between our method and the AF-Multimer(v1 best) on HeteroTest2. There are three main reasons responsible for this gap: 1) The base model for AF-Multimer is AF2, which is a model stronger than ESMFold in general, especially for those proteins that have high-quality MSAs; 2) AF-Multimer is a fully fine-tuned version of AF2 on a larger protein complex structure dataset while our model is a prompt tuning method trained only on the heterodimer dataset; 3) AF-Multimer ensembles five models, while we only use one model. However, our method is able to predict some proteins that are hard for both ESMFold-Linker and AF-Multimer. As shown in Figure 2, ESMFold-Linker* successfully predicts the interface of the

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

ESMFold-Linker

ESMFold-Linker*(ours)

AF-Multimer(v3best)

TM-score=0.59,DockQ=0.13

TM-score=0.90,DockQ=0.39

TM-score=0.52,DockQ=0.02

membrane protein 7D7F_AD with a DockQ score of 0.39 while ESMFold-Linker and AF-Multimer cannot predict the interface correctly.

5.3 Antibody heavy chain light chain docking

We further test our method on antibodies, an important type of protein in designing new drugs. Particularly, we focus on the heavy chain and the light chain docking. Table 2 shows the structure prediction results of MSA-free methods (first two methods) and MSA-based methods (last four methods) on the VH-VL test set. As an MSA-free model, ESMFold-Linker predicts all the interfaces successfully with an average DockQ score of 0.737, better than the classical docking method HDOCK. But it still lacks behind AlphaFold-Linker. For the three linker-based models, the distributions of their DockQ scores are shown in Figure 3. Equipped with the optimized linker, ESMFold-Linker* achieves an average DockQ score of 0.753, with 7 more high-quality interface predictions than ESMFoldLinker and 5 more high-quality interface predictions than AlphaFold-Linker. This result indicates that our learned linker trained on the general heterodimer dataset generalizes well to antibody data. Although the interface prediction performance of our method still falls behind AF-Multimer(v3 best), the gap in the DockQ score is much smaller compared to the case in HeteroTest2. Besides, it is quite close in TM-score to XtrimoDock [37], which is trained on an antibody-antigen dataset. Given our method only requires sequences as input, it can be a potentially useful model in the scenario of antibody design where the evolving antibody might not have MSAs.

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

0.9

0.8

0.7

0.6

0.5

ESMFold-Linker

AlphaFold-LinkerESMFold-Linker*(ours)

6 Analysis and Discussion

ESMFold-Linker* is 9 × faster than AF-Multimer in inference

We report the structure inference time of the MSA-free methods (ESMFold-Gap, ESMFold-Linker, and ESMFold-Linker*) and the SOTA MSA-based model AF-Multimer on the VH-VL dataset using A100 80G GPU. Table 3 shows the total model inference time on the VH-VL test set, where AF-Multimer's time is only for one model, excluding the time of MSA search. As shown in Table 3, on the VH-VL test set with an average sequence length of 231, both ESMFold-Linker and ESMFold-Linker* take 4 minutes to run the inference, which is 9 × faster than AF-Multimer.

Large chain break or linker, or both? We perform an ablation study on ESMFold with chain break and linker to better understand the contribution of each operation. Table 4 shows the comparison of inter-chain contact prediction precision of ESMFold-based methods on the heterodimer test set and HeteroTest2. 5 As shown in Table 4, it is hard to tell whether ESMFold-Linker or ESMFold-Gap is better. However, combining the two (ESMFold-Linker-Gap) provides significant performance gains over using either operation alone on both datasets. We observe similar effects in our method when incorporating chain break with the optimized linker. Compared to using a chain break, the major limitation of using a linker is that it increases the computation cost (shown in Table 3). But we can enjoy the advantage of a large degree of freedom for improvement and better performance. Empirically, combining the two gives a better performance than just using each of them.

The learned linker allows more chain twist while rarely interacting with the chains In Figure 4, 281 we visualize the predicted contact maps of two proteins with the linker inside to understand how the 282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

Ground truth

ESMFold-Linker

ESMFold-Linker*(ours)

A

H

Linker

Linker*

0

100

200

300

400

0

100

200

300

400

100

200

300

400

B

Linker

Linker*

Z

Z

0

50

100

150

200250

300

0

100

200

300

0

100

200

300

linker interacts with the chains. The two proteins are 7VYR_HL and 7WPE_YZ, corresponding to a good case (0.77 DockQ score) and a bad case (0.01 DockQ score) in our model ESMFold-Linker*. As shown in Figure 4, both the G25 linker (middle) and our learned linker (right) seem to rarely interact with the protein chains in both cases. This result indicates that ESMFold is able to recognize the linker part as a disordered region and fold the connected sequences as multi-domain proteins. Furthermore, there are more predicted contacts using the learned linker than using the G25 linker in both cases. This result suggests that the learned linker allows the connecting chains to freely twist and rotate to recruit binding partners more than the manual linker.

Limitations Our method has some limitations. First, if the base model (ESMFold-v1) is not good at predicting a certain type of protein complexes, such as the heterodimers in HeteroTest2, adding an optimized linker can not make it a strong model for that type of data since the trainable parameter size is very small. Second, our method is tested on heterodimers, whether it generalizes to homodimers or multi-chain proteins is unknown. Third, the linker is only optimized at the Folding Module, while the linker at ESM2 remains constant. And the linker length is treated as a hyperparameter, which can be further optimized to improve performance and speed.

7 Conclusions and future work

The use of prompts in protein structure prediction models is not always clear due to the high complexity of models and a general lack of biological knowledge for AI researchers. In this work, we have proposed Linker-tuning, a prompt tuning method to adapt the single-chain pre-trained ESMFold for heterodimer structure prediction. As proof-of-concept, we showcase that we can place a soft prompt in ESMFold. The task is reformulated as a pre-trained task itself under the biological prior. Experiments show that merely tuning a prompt on ESMFold can significantly improve the predicted complex structure quality over the discrete prompt handcrafted with strong biological insight. Hopefully, our work can inspire more work on AI for Protein Science.

There are two directions for future work. Firstly, we would like to extend our work to antibodyantigen structure prediction, a critical task with direct relevance to drug design. Secondly, we are going to explore structural-aware antibody design using our method since it is efficient and fast. By pursuing these directions, our objective is to make progressive contributions towards the development of effective drugs for disease treatment and pain relief.

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

References

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv , pages 2022-07, 2022.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv , 2022.

Xiaomin Fang, Fan Wang, Lihang Liu, Jingzhou He, Dayong Lin, Yingfei Xiang, Xiaonan Zhang, Hua Wu, Hui Li, and Le Song. Helixfold-single: Msa-free protein structure prediction by using protein language model as an alternative. arXiv preprint arXiv:2207.13921 , 2022.

Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M Church, et al. Single-sequence protein structure prediction using a language model and deep learning. Nature Biotechnology , 40(11):1617-1623, 2022.

Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations , 2020.

Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences , 118(15):e2016239118, 2021.

Junsu Ko and Juyong Lee. Can alphafold2 predict protein-peptide complex structures accurately? BioRxiv , 2021.

Tomer Tsaban, Julia K Varga, Orly Avraham, Ziv Ben-Aharon, Alisa Khramushin, and Ora SchuelerFurman. Harnessing protein folding neural networks for peptide-protein docking. Nature communications , 13(1):1-12, 2022.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv preprint arXiv:2103.10385 , 2021.

Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature , 577(7792):706-710, 2020.

Vishnu Priyanka Reddy Chichili, Veerendra Kumar, and Jayaraman Sivaraman. Linkers in the structural biology of protein-protein interactions. Protein science , 22(2):153-167, 2013.

Athena D Nagi and Lynne Regan. An inverse correlation between loop length and stability in a four-helixbundle protein. Folding and Design , 2(1):67-75, 1997.

Janet E Deane, Daniel P Ryan, Margaret Sunde, Megan J Maher, J Mitchell Guss, Jane E Visvader, and Jacqueline M Matthews. Tandem lim domains provide synergistic binding in the lmo4: Ldb1 complex. The EMBO journal , 23(18):3589-3598, 2004.

Stephen K Burley, Helen M Berman, Charmi Bhikadiya, Chunxiao Bi, Li Chen, Luigi Di Costanzo, Cole Christie, Jose M Duarte, Shuchismita Dutta, Zukang Feng, et al. Protein data bank: the single global archive for 3d macromolecular structure data. Nucleic Acids Research , 47(D1), 2018.

Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science , 373(6557):871-876, 2021.

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences , 117(3):1496-1503, 2020.

Wenkai Wang, Zhenling Peng, and Jianyi Yang. Single-sequence protein structure prediction using supervised transformer protein language models. Nature Computational Science , 2(12):804-814, 2022.

Yining Wang, Xumeng Gong, Shaochuan Li, Bing Yang, YiWu Sun, Chuan Shi, Hui Li, Yangang Wang, Cheng Yang, and Le Song. xtrimoabfold: De novo antibody structure prediction without msa. arXiv preprint arXiv:2212.00735v3 , 2022.

Jinhua Zhu, Zhenyu He, Ziyao Li, Guolin Ke, and Linfeng Zhang. Uni-fold musse: De novo protein complex prediction with protein language models. bioRxiv , pages 2023-02, 2023.

Brennan Abanades, Wing Ki Wong, Fergus Boyles, Guy Georges, Alexander Bujotzek, and Charlotte M Deane. Immunebuilder: Deep-learning models for predicting the structures of immune proteins. bioRxiv , pages 2022-11, 2022.

Yining Wang, Xumeng Gong, Shaochuan Li, Bing Yang, Yiwu Sun, Yujie Luo, Hui Li, and Le Song. Fast de novo antibody structure prediction with atomic accuracy. Cancer Research , 83(7_Supplement):4296-4296, 2023.

Ozlem Keskin, Attila Gursoy, Buyong Ma, and Ruth Nussinov. Principles of protein- protein interactions: what are the preferred ways for proteins to interact? Chemical reviews , 108(4):1225-1244, 2008.

Ian R Humphreys, Jimin Pei, Minkyung Baek, Aditya Krishnakumar, Ivan Anishchenko, Sergey Ovchinnikov, Jing Zhang, Travis J Ness, Sudeep Banjade, Saket R Bagde, et al. Computed structures of core eukaryotic protein complexes. Science , 374(6573):eabm4805, 2021.

Patrick Bryant, Gabriele Pozzati, and Arne Elofsson. Improved prediction of protein-protein interactions using alphafold2. Nature communications , 13(1):1265, 2022.

Mu Gao, Davi Nakajima An, Jerry M Parks, and Jeffrey Skolnick. Af2complex predicts direct physical interactions in multimeric proteins with deep learning. Nature communications , 13(1):1744, 2022.

Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods , 19(6):679-682, 2022.

Richard Evans, Michael O'Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Žídek, Russ Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with alphafoldmultimer. BioRxiv , pages 2021-10, 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 255-269, 2021.

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4222-4235, 2020.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3816-3830, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 45824597, 2021.

Mu Gao and Jeffrey Skolnick. Apoc: large-scale identification of similar protein pockets. Bioinformatics , 29(5):597-604, 2013.

Zhiye Guo, Jian Liu, Jeffrey Skolnick, and Jianlin Cheng. Prediction of inter-chain distance maps of 411 protein complexes with 2d attention-based deep neural networks. Nature Communications , 13(1):6963, 412 2022. 413

414

415

416

Yujie Luo, Shaochuan Li, Yiwu Sun, Ruijia Wang, Tingting Tang, Beiqi Hongdu, Xingyi Cheng, Chuan Shi, Hui Li, and Le Song. xtrimodock: Rigid protein docking via cross-modal representation learning and spectral algorithm. bioRxiv , pages 2023-02, 2023.

417

418

419

420

421

422

Yumeng Yan, Huanyu Tao, Jiahua He, and Sheng-You Huang. The hdock server for integrated proteinprotein docking. Nature protocols , 15(5):1829-1852, 2020.

Sankar Basu and Björn Wallner. Dockq: a quality measure for protein-protein docking models. PloS one , 11(8):e0161879, 2016.

Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics , 57(4):702-710, 2004.

Gianluca Tomasello, Ilaria Armenia, and Gianluca Molla. The protein imager: a full-featured online 423 molecular viewer interface with server-side hq-rendering capabilities. Bioinformatics , 36(9):2909-2911, 424 2020. 425

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a strategy of predicting heterodimers with ESMFold. \n\nChains of heterodimers are linked by glycine linkers and inputed to esm. The output representations are then added with a learnable embedding layer, and then folded with the folding module in ESMFold. The finetuning is only done for the learnable embedding, with a weighted distogram loss.\n\nEvaluations are done on 3 datasets, while one of them should be considered pointless. Some elevation is gained by the method, compared with directly using links.\n\nSTRENGTHS:\n1. The writing of the paper is clear.\n\n2. The proposed method is reasonable and intuitive.\n\n3. The method achieves comparable performances with other linker-based hacking strategies. Slight elevation of performances is gained (TMscore 0.62->0.65, ~0.03) from finetuning compared with directly using linkers.\n\nWEAKNESSES:\nMethod:\n\n3. The proposed idea, i.e. exploiting ESM models to predict protein complexes by architecture adjusting and finetuning has already been explored previously [1]. Seems that their implementations are more \"neat\": no external linkers are involved; permutation invariances are regarded; they solve complexes with arbitrary numbers of chains, both homomers and heteromers; and the performance elevation is more significant (TMscore 0.27->0.66, ~0.39 in their benchmark). However, no discussion let alone comparison is shown in this paper.\n\n4. In fact, I don't know why linkers are needed to fold multimers at all: simply modifying the relative position indices suffices to tell the model that the residues are from separate chains. Involving linkers will implicitly pose in the model a geometrical constraint on the C-terminal of chain A and N-terminal of chain B (as they'll have relative position of +-L). Also the running time can be (slightly) added. Also, I don't personally like the saying that connects the linker idea to \"prompts\": they are totally different things. Doing so is more of attempting to ride the wave of LLM.\n\nEvaluation:\n\n5. The elevation of performances is in all sense too marginal. The results simply tell me that both ESMFold-linker and the proposed method are not reliable (avg DockQ of 0.11/0.17), instead of telling me that the proposed method is useful. In this case, maybe the percentage of success is a better metric to show.\n\n6. The VH-VL docking benchmark is totally pointless and lacks commonsense. One who has basic senses of the domain knows that all protein folding efforts on antibodies should focus on Ab-Ag instead of VH-VL, because all interaction modes between VH-VL are the same, i.e. they fold almost identically (In table 2, as one can expect, all TM-scores are above 0.92). Therefore, they shouldn't be used as heterodimer folding or protein docking benchmarks. Even if one focuses on the structures of VH-VL, the metrics on the CDR loops should be independently reported, rather than global RMSD. \n\n[1] Zhu et al, Uni-Fold MuSSe: De Novo Protein Complex Prediction with Protein Language Models. https://www.biorxiv.org/content/10.1101/2023.02.14.528571v1.full.pdf\n\nQUESTIONS:\n8. I don't know how the standard deviation of RMSD reachs 8.39 when the mean is 8.59 (0.35+- 0.36 for DockQ). This is so counter-intuitive. The authors may need a histogram to explain this.\n\n9. Why is AF-multimer performances in Table 1 (left) missing? \n\n10. Boxplots are shown only for VH-VL. Why not show them for Table 1 benchmarks?\n\nLIMITATIONS:\nlimitations are addressed in section 6.\n\nThe suggestion would be \n\n11. More solid benchmarks.\n\n12. Discussions and comparisons with [1].", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author applies prompt tuning in the heterodimeric protein prediction task. Instead of using the poly-Glycine linker, this method automatically finds the best linker in the continuous space. The author compares this method with several existing methods including the current start-of-art algorithm AF-multimer, the best PLM-based algorithm ESMFold-Linker, and the rigid docking algorithm HDOCK. The performance shows this method which is PLM-based is better than ESMFold-Linker but worse than AF-multimer.\n\nSTRENGTHS:\n1. The novelty and contribution of the paper mentioned in the paper are clear and correct.\n2. The motivation behind the method and its design are well-founded and logical.\n3. Overall, the writing is great.\n\nWEAKNESSES:\n1. Based on my understanding, the methodology seems to be limited in its applicability to the pre-trained language model-based protein structure prediction method, which is not considered the most accurate algorithm for protein structure prediction. Furthermore, the performance of this methodology appears to be influenced by the linker's position and the location suggested in the paper is not the PLM part, raising concerns about its compatibility with other PLM-based methods like Omegafold. Consequently, the paper's value may diminish if there are more robust folding algorithms available.\n2. Given that all other methods are unsupervised, there is a possibility of the method benefiting from overfitting. Regarding the antibody dataset, from my understanding, antibodies generally exhibit a rigid overall structure except for the six CDR loops. Consequently, I suspect that the flexible CDR loop will not lead to significant variations in the docking of the light chain and heavy chain. Unless supported by evidence, I prefer to believe that the results obtained from the antibody dataset are not reliable and it may not be an appropriate dataset for this task. As for the Heterodimer test, while a 40% threshold seems acceptable, I believe setting a lower threshold, preferably below 30%, would be better.\n3. In comparison to the state-of-the-art method AF-multimer, this method exhibits a significant decrease in performance. When people have the opportunity to utilize a substantial number of CPUs for preparing MSA information, the speed advantage offered by this method does not compensate for its accuracy limitations. Moreover, as far as I understand, there aren't a lot of downstream tasks that necessitate high-throughput calculations.\n\nQUESTIONS:\n1. The definition of the Gap model is not clear. It seems it is the version that you set the residue_index_offset for the second chain as 512. If the understanding is correct, is there any explanation for the improvement? It seems like this trivial strategy can achieve 50% of your improvement (according to the DockQ score for the Heterodimer test). Is that possible to also apply this trick to AF-multimer and AlphaFold-Linker?\n2. The reason why you choose to use a 21-residue linker for AlphaFold-Linker which is different from the number you used for ESM.\n3. Could you show the distribution of the score of the metrics for the Heterodimer test and HeteroTest2? It seems the standard deviation of scores is extremely high. In this case, it seems it is necessary to prove the difference is statistically significant for some key comparisons.\n\nLIMITATIONS:\nThe limitation is well addressed and there is no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Inspired by the prompt tuning technique used in the field of NLP, the authors leverage the prompt tuning to adapt the single-chain pre-trained ESMFold for heterodimer protein structure prediction. To be specific, a learnable soft prompt is placed between protein chains. With such links, the pre-trained ESMFold treats complex structure prediction as the monomer structure prediction, and the prompt is tuned on the heterodimer dataset.

They show model with such a trick can outperform ESMFold-Linker baseline by large margins on both contact and structure prediction tasks on the heterodimer test set.

STRENGTHS:
1 Leveraging the prompt tuning idea in multimer structure prediction using a single-chain pre-trained model is quite novel and interesting.

2. The proposed trick does improve the performance of the ESMFOLD-link baseline. The trick can potentially be applied to different single-chain pre-trained protein structure prediction models and can be considered a general trick.

WEAKNESSES:
1. The link-tuning idea is only validated on ESMFold. I'm curious about its effectiveness when applied to other protein structure prediction (PSP) models, e.g., Alphafold, and Omegafold. Prompt tuning is a general trick in NLP. Justifying the generalizability of the link-tuning trick on different PSP models can definitely make the manuscript stronger.

2. The idea of Linker-tuning is simple and effective, but I feel the current manuscript is not informative enough to be published on NeurIPS conference. If the authors can justify the generalizability of the link-tuning trick, I'm willing to adjust my score.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This work predicts the structure of heterodimeric protein chains by optimizing poly-G linkers that connect two chains of a heterodimer.

STRENGTHS:
- This research, compared to other existing deep learning methods, finds an alternative to protein complex prediction methods, which is to make use of poly-G linkers. The idea connects closely to many biological applications and thus has more potential impacts on wider communities. 
- The evaluation metrics cover various aspects to comprehensively assess the model's performance.

WEAKNESSES:
- ProteinMPNN as another famous multimeric structure prediction tool should have been compared.
- It seems very often the proposed method does not achieve the optimal performance (in Table 1 and Table 2).

QUESTIONS:
- It is not very clear to non-LM experts how adding a linker is an analogue to the prompt in LM.

LIMITATIONS:
No potential negative societal impacts were discussed in the main text.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07