Title: DNDesign: Denoising is All You Need for Protein Inverse Folding
Venue: NeurIPS 2023
Accepted: no
Keywords: Protein sequence design

Inverse folding

Denoising

Protein modeling
Abstract: Based on the central dogma that protein structure determines its functionality, an important approach for protein sequence design is to identify promising sequences that fold into pre-designed structures based on domain knowledge. Numerous studies have introduced deep generative model-based inverse-folding, which utilizes various generative models to translate fixed backbones to corresponding sequences. In this work, we reveal that denoising training enables models to deeply capture the protein energy landscape, which previous models do not fully leverage. Based on this, we propose a novel Denoising-enhanced protein fixed backbone design (DNDesign), which combines conventional inverse-folding networks with a novel plug-in module, which learns the physical understanding via denoising training and transfers the knowledge to the entire network. Through extensive experiments, we demonstrate that DNDesign can easily be integrated into state-of-the-art models and improve performance in multiple modes, including auto-regressive, non-auto-regressive, and scaled-up scenarios. Furthermore, we introduce a fixed backbone conservation analysis based on potential energy changes, which confirms that DNDesign ensures more energetically favorable inverse-folding.

1

Abstract

Based on the central dogma that protein structure determines its functionality, an important approach for protein sequence design is to identify promising sequences that fold into pre-designed structures based on domain knowledge. Numerous studies have introduced deep generative model-based inverse-folding, which utilizes various generative models to translate fixed backbones to corresponding sequences. In this work, we reveal that denoising training enables models to deeply capture the protein energy landscape, which previous models do not fully leverage. Based on this, we propose a novel Denoising-enhanced protein fixed backbone design (DNDesign), which combines conventional inverse-folding networks with a novel plug-in module, which learns the physical understanding via denoising training and transfers the knowledge to the entire network. Through extensive experiments, we demonstrate that DNDesign can easily be integrated into state-of-the-art models and improve performance in multiple modes, including auto-regressive, non-autoregressive, and scaled-up scenarios. Furthermore, we introduce a fixed backbone conservation analysis based on potential energy changes, which confirms that DNDesign ensures more energetically favorable inverse-folding.

1 Introduction

Proteins play a crucial role in various biological processes, serving as the primary components of 18 cellular machinery and exhibiting specific functionalities based on their three-dimensional structures. 19 Therefore, designing unique protein sequences that fold into the pre-designed structures to have 20 targeted functionality is of great importance in various biological applications, including pharmaceu21 ticals, biofuels, biosensors, and agriculture [1, 2, 3, 4, 5]. Mutagenesis aided with high-throughput 22 experiments and physics-based computational methods have been proposed to address this problem 23 in the past [6, 7]. 24

25

26

27

28

29

30

31

32

Recently, a new paradigm employing a deep generative model conditioning on three-dimensional structures has gained attention, and its results have been encouraging. These models typically represent a given fixed protein backbone as a 3D graph and leverage advanced generation techniques such as auto-regressive, non-auto-regressive, or adapter-based methods to translate fixed protein backbones into corresponding sequences that can achieve specific biological purposes [8]. For example, Daurass et al. [9] experimentally proved that the deep inverse folding neural networks(IFNN) generates protein sequences with targeted functionality at a higher success rate than Rosetta, a wellestablished software widely used in the protein community.

33

As the folding structure is a consequence of the physics that causes folding, acquiring meaningful

physical knowledge of folding structures, called folding energy landscape, directly from data would 34

benefit generative models. The currently proposed DIFM has utilized geometric-related features 35

and geometric neural networks to maximize the likelihood of the given training data, aiming to 36

comprehend the protein world and perform inverse-folding based on the physical understanding. 37 However, the experimentally validated structural data is limited to about 200K [10], and the number 38 of structurally distinct and unique proteins practically used for inverse-folding training is only around 39 30K [11], which is insufficient to train a model to grasp the nature of complex protein folding systems 40 fully. In addition, considering de novo proteins with significantly different characteristics from those 41 discovered thus far, the data scarcity problem worsens more. 42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

In this study, we present DNDesign, a DeNoising-enhanced protein DESIGN framework, which maximizes the model's physical understanding directly from training data. First, we prove that denoising training directly on a three-dimensional backbone structure is equivalent to the direction of energy minimization, finally enabling the models to learn the folding energy landscape. Then, we suggest a novel folding physics learning plug-in module (FPLM), which can be integrated into existing inverse-folding neural networks. First, IFNN takes the original structure, while FPLM inputs the perturbed structures obtained by adding noises to the original structure. Then, FPLM learns the folding energy landscape by moving the perturbed structure to the energetically stable state. To effectively transfer the folding knowledge to IFNN, FPLM employs five novel operations: (1) force feature initialization, (2) force-node attention, (3) force-edge attention, (4) global force attention, and (5) force biasing to transfer the physical inductive bias to the entire network. Notably, the force biasing operation allows the sequence decoder to conduct sampling while considering folding physics. Finally, DNDesign enables IFNN to learn four pieces of information, including original structure, intermediate structure, folding physics, and the direction of energy minimization using given data, enhancing previous methods which only know the original structure.

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

To demonstrate the novelty of DNDesign, we conduct a series of protein sequence design tasks. Particularly, to showcase the "easy-to-use" nature and generalization of DNDesign, we apply DNDesign to three representative IFNN settings, including auto-regressive (AR), non-auto-regressive (NAR), and scaled-up settings. Remarkably, DNDesign consistently improves the previous method in all experiments, proving that more physical understanding from denoising training benefits models to conduct successful inverse-folding. In addition to considering sequence consistency estimated using the sequence recovery metric, we evaluate the structure conservation by measuring the potential energy change caused by generated sequences after structure relaxation. Based on this metric, we have demonstrated that DNDesign enables models to generate more energetically favorable sequences. To our knowledge, this is the first work comparing structure conservation using potential energy change for evaluating deep inveres-folding models. In addition, extensive ablation and analysis are provided to help the understanding of DNDesign. Our contributions are as follows:

We propose DNDesign, which enables the inverse-folding model to capture the deep understanding of folding physics that previous models do not fully exploit.

We prove how DNDesign learns folding physics directly from data and show that DNDesign improves the state-of-the-art model on various protein sequence design benchmarks in auto-regressive, non-auto-regressive, and scaled-up scenarios.

We introduce a fixed backbone conservation task based on potential energy change from newly generated sequences. The analysis proves that DNDesign generates energetically favorable sequences, leading to more fine-grained protein design.

2 Preliminaries and Related works

2.1 Protein representation

A protein is a sequence P = { A 1 , ..., A N } where A i is a residue of 20 types of amino acids. Each 80 amino acid is composed of backbone atoms including C, N, C α , O and side chain R i consisting 81 of atoms, which determine the property of the amino acid, and each atom has coordinate x ∈ R 3 . 82 Following ingraham et al [12], each residue has geometry derived from backbone atoms, called local 83 coordinate system g i = [ b i , n i , b i × n i ] , where, 84

Because each side-chain corresponds to each amino acid residue s , a protein can be represented as a 85 sequence of residue and geometry pairs, P = { ( s i , g 1 ) , ..., ( s N , g N ) } as depicted in Figure 1. 86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

Backbone generation

C

C

𝛼

N

O

𝐺

!"#

𝐺

!

𝐺

!$#

R

A

B

C

C

𝛼

C

𝛼

N

C

N

C

O

O

R

R

C

C

𝛼

N

O

?

?

?

C

𝛼

C

𝛼

N

C

N

C

O

O

…ABC…

C

C

𝛼

N

O

𝐺

!"#

𝐺

!

?

R

A

B

?

C

𝛼

C

𝛼

N

C

N

C

O

O

R

?

C

C

𝛼

N

O

𝐺

!"#

𝐺

!

𝐺

!$#

?

?

?

?

C

𝛼

C

𝛼

N

C

N

C

O

O

?

?

Structure-sequence co-design

Inverse-folding

C

𝛼

N

C

O

𝑢

𝑣

𝑏

𝑛

𝑏×𝑛

Local frame

2.2 Structure-based Protein Design

Because we can reconstruct side-chain R using both backbone G and residues S , structure-based protein design only focuses on the generation of backbone G and residues S . Neural structure-based protein design can be categorized into three tasks, (1) backbone generation, (2) structure-sequence co-design, and (3) inverse-folding depending on design scenarios as illustrated in Figure 1.

Backbone generation Backbone generation can be formulated as an end-to-end structure-tostructure generation as f θ : G → G . We train parameterized encoder-decoder neural networks f θ to sample diverse and plausible backbone structures. We note that this approach requires a model which can assign amino acids are required to obtain complete proteins [13].

Structure-sequence co-design The structure-sequence co-design approach generates sequences and backbone structures simultaneously as f θ : ( G,S ) → ( G,S ) . By generating sequences autoregressively and refining predicted structures iteratively, this co-design approach enables the adaptive generation of optimized sequences compatible with the flexible backbone. This approach is suitable for scenarios such as antibody design, where global structure tends to be changed depending on the designed antibody [14].

Inverse-folding Inverse-folding is an approach that aims to 'caption' or 'translate' a provided backbone structure into potential protein sequences as f θ : G → S . Inverse-folding becomes particularly valuable for protein functional design problems, such as ligand binding site, enzyme, and binder design, which require fine-grained control over side-chain combinations. Since there exist numerous sequences that can fold into a specific backbone conformation [15], inverse-folding approach enables protein engineers to identify diverse and promising protein sequences with optimized functional properties [1, 2, 3, 4, 5].

2.3 Importance of physical understanding for structure-based protein design

The folding structure of proteins is governed by the physics that determines the folding process.

110 Therefore, a higher level of understanding of physics enables a model to perform well in structure111 related tasks. For instance, AlphaFold2 [16] achieved nearly experimental-level accuracy in structure 112 prediction and has revolutionized protein fields since its publication. Recently, [17] proved that 113 the success of AlphaFold2 [18] is attributed to the data-driven understanding of the folding energy 114 landscape. Additionally, RFDiffusion [13] demonstrated that physically plausible and synthesizable 115 structures were created when generative models were trained upon pre-trained protein structures 116 model, especially, RoseTTAFold [18] that comprehend the mechanism of folding. On the other hand, 117 RefineDiff [19] showed that physics learning enhances protein representation learning for various 118 downstream tasks. 119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

3 Related works

3.1 Deep generative protein design

Deep generative models for protein design can be classified into four categories: sequence-tosequence, structure-to-structure, structure-to-sequence, and structure-sequence co-design. In the sequence-to-sequence category, several notable works, such as RITA [20], DARK [21], ProtGPT2 [22], and ProGen2 [23] have applied language modeling techniques to analyze large protein sequence databases and successfully generated novel protein sequences. Moving on to the structure-tostructure category, deep generative models, such as variational autoencoders [24], generative adversarial networks [25], or diffusion models [26, 27], have been applied to this problem and shown promising results for generating diverse and structurally plausible protein backbones [28, 29, 13, 30, 31, 32]. Third, in the structure-to-sequence category, deep generative inverse-folding models, such as GraphTrans, GVP, ProteinMPNN, PiFold [12, 33, 34, 35, 36, 9] have been proposed. These models predict or generate protein sequences corresponding to a given protein backbone structure, allowing for the design of functional proteins with specific structural constraints. Last, DiffAb [37] and RefineGNN [38] generate structure and sequence simultaneously through iterative refinement. Our proposed DNDesign falls within the category of deep generative inverse-folding models. DNDesign differentiates itself from previous approaches by utilizing denoising to strengthen the understanding of models.

3.2 Denoising training for chemical and biology system modeling

In order to effectively model chemical and biological systems, acquiring meaningful physical knowledge directly from data plays a crucial role in successful representation learning and downstream tasks. Over the past decade, numerous studies have proposed novel deep learning architectures and training methods that demonstrate remarkable performance in approximating interactions and quantum properties [39, 40, 41, 36, 42]. Recently, several works have theoretically established the equivalence between denoising training on biological molecule data and learning the underlying physical force field [43, 44, 19]. Building upon this finding, Godwin et al. [45] successfully addressed the over-smoothing challenge in graph neural networks (GNNs) by employing denoising training on three-dimensional coordinates as an auxiliary loss. Their approach achieved state-of-the-art results on various quantum chemistry downstream tasks. Similarly, [43] utilized denoising as a pre-training objective on the 3M molecular dataset and demonstrated that denoising-based learning produces high-quality representations based on a data-driven force field, effectively solving different molecular benchmarks. In this work, we adopt a similar denoising-based approach to that of [45, 19]. Our work is similar to [19], which utilized denoising to learn folding physics, but they focused on representation learning. Otherwise, our work first extend denoising to protein sequence generation. Despite this difference, the inductive bias gained from DNDesign is identical to that of the two aforementioned papers.

4 Methods

4.1 Overview

DEDesign integrates conventional inverse-folding networks (IFNN) with denoising networks (FPLM). In this section, we begin by elucidating the process of protein featurization. Subsequently, Section 4.3 outlines the model architecture and describes the feature update operation in IFNN. Next, section 4.4 highlights the correlation between the denoising process and the acquisition of folding physics knowledge. Finally, Section 4.5 describes the five operations utilized to transfer data-driven folding physics across the entire network.

4.2 Featurization

In this work, we regard a protein as a k-NN 3D graph G ( V, E, X ) , where V , E , X are node features, edge features, and xyz coordinates, respectively. As suggested in [12], node and edge features are constructed considering pairwise distances, angle, torsion, and directions using the four backbone atoms of each node. Each node and edge feature is input to a single fully-connected layer and converted into an embedding vector, h 0 , e 0 , respectively. More details are illustrated in Appendix A.

170

Featurizer

Node-Edge

Cross-update

Groud truth noise

Self-Attention

Original Protein

Projection

Force logits

Featurizer

Node update

With force

Edge features

Force context

Attention

Sequence Decoder

Force features

Folding physics learning module (FPLM)

Conventional inverse-folding networks (IFNN)

Force gradient

Projection

Force constraint

Force field initialization

Cross-entropy

objective

Predicted noise

Denoising objective

Protein sequence decoder

Logits

Edge update

With Force

4.3 Conventional inverse-folding model modeling

In this subsection, we will briefly describe the IFNN used in this work. As shown in Figure 2, we 171 adopt a state-of-the-art inverse-folding encoder, called PiGNN proposed by [46]. We first update 172 node and edge features in order based on k-NN neighbors. And, the node features h l i and edge 173 features e l i are obtained using a featurizer and updated using a simplified graph transformer [47]. 174 After computing the node value using the edge features and neighbor node features, we multiply 175 the softmax values of the attention weights obtained from h l i and e l i to the node value and add the 176 weights to the node features. Finally, we update the h l i to obtain the next layer node features h l i 177 using a learnable weight function with normalization and residual connection. Edge features e l ji are 178 calculated by passing concatenation of node feature h i , neighbor node feature h j , and edge feature 179 e i,j to the MLP ψ e . 180

We use two types of inverse folding decoders to predict logits: (1) auto-regressive (AR) and (2) 181 non-auto-regressive (NAR). For AR, we employ the sequence decoder used in GraphTrans [12] and 182 utilize random decoding introduced by ProteinMPNN [9]. For NAR, we use a linear layer as [33]. 183 The decoder inputs node feature h j and outputs logits. The whole IFNN is trained by minimizing the 184 negative log-likelihood L AR of the data. 185

4.4 Folding physics learning via denoising 186

𝐺

!

"

𝐺

",$

"

𝐺

",%

"

𝐺

",&

Previous methods only know

DNDesign knows all

Folding energy landscape

Stable

Unstable

Energy (E)

Intermediate state (noised data)

Equilibrium state (training data)

Folding dynamics

Denoising (direction to lower E)

Folding energy landscape The protein structure G = { g 1 , ..., g N } that a protein P with sequence 187 S = { s 1 , ..., s N } can have is diverse, and each structure corresponds to a specific potential energy E 188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

based on the folding physics, represented as the folding energy landscape. Considering the dynamics that structure prefers energetically stable states, each protein structure G used in training data can be regarded as the most stable, i.e., energetically lowest structure G L among the various folding states that each sequence can possess.

Figure 3 illustrates four crucial pieces of information contained within the energy landscape: (1) stable or low energy state G L , (2) unstable or high energy state ˜ G H , (3) folding physics, i.g, energy potential, and (4) the direction or gradient ∇ G that transforms the unstable state ˜ G H to the stable state G L .

Perturbation on backbone structure Firstly, since we only have G L in the training data, we obtain ˜ G H by adding Gaussian noise to G L . G is a sequence of local frames g , where each g consists of a C α coordinate vector, x , and an SO 3 vector, r . To account for this nature, we follow the forward noising procedure used in [48, 37]. For x , we use random three-dimensional Gaussian noise, which can be described as follows:

where N denotes a Gaussian distribution.

For perturbing r , we use an isotropic Gaussian distribution with a mean rotation scalar variance parameter in the SO3 space [48, 49, 50] as follow:

, where λ is a modification of the rotation matrix by scaling its rotation angle with the rotation axis fixed [51].

Denoising training Based on the remarkable reparameterization technique proposed by Ho et al. [26] and definition of training objective [37], we can optimize denoising objectives for the transition vector t and SO 3 vector r as follows:

, where ψ is a neural networks that predicts the perturbation on t and r . L t pos is mean squared error between added Gaussian noise ϵ j and predicted noises ψ ( ˜ G H , t ) and L t rot minimizes the discrepancy calculated by the inner product between the real and predicted orientation ψ ( ˜ G H , t ) .

Learning folding physics through denoising learning From a statistical perspective, denoising training can be seen as learning the Boltzmann distribution p physical ( G ) , which represents the probability distribution p physical ( G ) ∝ exp( -E ( G )) of the structure's energy E using a given protein structure G as a random quantity. Based on the definition, the derivative ∇ x log p physical of the folding energy potential E ( G ) corresponds to the force -∇ x E ( x ) acting on the backbone atoms, directing them towards energetically stable directions as follows:

Since p physical is unavailable in practice, we approximate it using data. Following [52], we can compute the log-likelihood of p 0 using

So, we can approximate p by approximating ˜ f . To do that, we use Gaussian noise q σ ( ˜ G T H | G L ) = N ( ˜ G T H ; G L , σ 2 I 3 N ) as used in other works [27, 26], then, we finally can match the score ∇ x log p physical by learning neural networks θ ( ˜ G T H , T ) that predict ∇ ˜ x log q σ ( ˜ G T H | G L ) .

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

This score-matching is the same as what we use in the DNDesign framework. So, we can conclude that denoising ˜ G H , which has become a higher energy state due to noise, towards G L is equivalent to learning the folding dynamics.

In summary, in DNDesign, we employ denoising training to learn the energy potential that determines the folding state, i.e., the folding energy landscape. Finally, unlike all previous methods that only train for (1), DNDesign allows for the training of (1), (2), (3), and (4) simultaneously, maximizing the model's physical understanding within the given data.

4.5 Physics understanding transfer

This section describes FPLM and the five operations to transfer the folding physics inductive bias of DENN to IFNN.

Feature embedding We first extract geometric features of perturbed structure and update the features using orientation-aware roto-translation invariant network networks used in [37]. We call the features force features and use the features in the following five operations.

Node initialization with force We add force features to the initial node features so that all node and edge features update in IFNN can be conducted using folding physics understanding.

Node update with force For node updates using force features, we use an attention module that combines two operations: self-attention and cross-attention. The two attentions are based on multi-headed full self-attention [53]. First, self-attention on node features is conducted. Then, cross-attention is performed between the self-attended node features and the force features. At this time, the query and key are force features, and through this, each node strongly interacts with the force state of all other nodes.

Edge update with force Edge updates with force features are performed similarly to edge updates with node features in IFNN. After concatenating the force feature f i , the force feature of the neighbor j node f j , and the edge features e ij corresponding to the ij pair, edge feature e l ji is obtained by projecting the concatenated features into the learnable MLP ψ f .

Global force context attention The global force state summing all local forces is an essential context for protein sequences. In the above two modules, node and edge updates are conducted using local forces. As the last module, we update node and edge features using the global force state features. The total global force state is simply calculated by taking the average of all force state features. To reduce computational costs, we apply an element-wise product using gated attention combined with a sigmoid function to each node and edge. By doing so, the global force information is transferred to all nodes and edges that comprise a protein.

Sequence sampling with end-to-end physics constraint We obtained logits using force features through linear layer and add the logits to the logits of IFNN as follow:

The ratio of force to decoder logits is empirically determined. In this work, we use 0.2 of α . By different ratios, we can generate diverse sequences by conditioning on physics features. This adds an additional sampling parameter to inverse-folding sampling, which only resort to probability-based sequence sampling previously.

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

5 Experiments

In this section, we compare FFDesign with the state-of-the-art deep generative inverse-folding models in three scenarios, including single-chain, multi-chain, and real-world datasets.

5.1 Experiment Setting

Implementation details We choose PiFold as IFNN and train PiFold and DNDesign-PiFold in AR, NAR, and scaled-up settings. Models are trained up to 100 in AR and NAR settings, and we set 150 epochs for the scaled-up scenario. All models are trained on 1 NVIDIA A100s with the Adam optimizer [54]. The batch size contains 6000 tokens, and the learning rate is set to 0.001 and decayed with OneCycle scheduler. More details are provided in the appendix. For reliable experiments, all results are obtained using three seeds.

Baselines We employ various graph-based inverse-folding models as baselines, including StructGNN, StructTrans [12], GCA [55], GVP [34], GVP-large [8], GVP-transformer [8], AlphaDesign [47], ESM-IF [8], ProteinMPNN [9], and PiFold [33].

5.2 Main Results

Single-chain sequence design CATH [11] is a widely used protein dataset to evaluate inversefolding models on single-chain sequence design tasks. For a fair comparison, we adopt the version of CATH4.2 as used in GraphTrans and GVP, PiFold. In CATH 4.2, 18024, 608, and 1120 proteins are used for training, validation, and testing, respectively. In the standard setting for this task, models are trained using training data and evaluated on three sets from the test set; short-chain, single-chain, and all-set, with perplexity and median recovery scores. Sequence recovery is a widely used metric for inverse-folding and measures how many residues of sampled sequences match that of the ground truth sequence at each position. The results on CATH 4.2 are shown in Table 1. Under similar conditions, the proposed DNDesign consistently improves the previous SOTA method on both perplexity and recovery in both auto-regressive and non-auto-regressive settings.

Scaling-up [8] proved that additional structure data predicted using AlphaFold2 gives remarkable improvement for sequence design. Likewise, we prepare ∼ 12M predicted structure data of Uniref50 [56] from AlphaFold2 database and train both PiFold and DNDesign-PiFold models using CATH 4.3 + Uniref50. Interestingly, the improvement from denoising becomes more evident in a scaled setting, as shown in Table 2. These results indicate that even with scaling up by adding

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

millions of structures, a model still accesses only the given stable structures. However, models trained using DNDesign can fully leverage all four pieces of information in folding physics, resulting in improved performance.

Other benchmarks We also conduct sequence design on multi-chain and real-world datasets, TS50, TS500 [57], and Ollikainen [58] datasets. As shown in Appendix A, performance gains from denoising still appear, proving the importance of folding physics understanding for protein inverse-folding design. Detailed results are provided in the appendix.

5.3 Analysis

In this section, we provide analyses to understand DNDesign more deeply. This section includes fixed backbone conservation using potential energy and an ablation study about five proposed operations. Additional analyses, such as core-surface analysis and other ablation studies, are provided appendix.

Fixed backbone conservation study Sequence recovery is suitable for measuring sequence consistency, but it cannot fully describe the potential energy behavior. In fixed backbone design, the ultimate goal is to generate sequences with the given structure. Therefore, it is necessary to evaluate whether the generated sequences conserve the fixed backbone structure. Conservation of the given structure implies that the generated sequences pose a structure that do not deviate far from the minimum energy state of the structure. Thus, we can evaluate whether the model generates energetically stable sequences by performing structure relaxation on the sampled proteins, which have corresponding new sequences and given backbone, and measuring the potential energy change. We utilize the Rosetta [59], a well-published computational tool, for structure relaxation. We use 282 structures after filtering structures having residues without coordinates. We generated nine sequences for each structure and performed relaxation for each (structure, sequence) pair. Then, we

6000

4000

2000

0

2000

E

0.0000

0.0002

0.0004

0.0006

0.0008

0.0010

0.0012

0.0014

Density

PiFold

DNDesign (PiFold)

500

0

500

E

0.00000

0.00025

0.00050

0.00075

0.00100

0.00125

0.00150

0.00175

Density

computed the change of potential energies. Interestingly, as shown in Figure 4, we observe that PiFold, with an enhanced understanding of folding physics from DNDesign, generates more sequences with potential energy change near zero, meaning complete structure conservation. This clearly demonstrates that our approach, which emphasizes the importance of physics in fixed backbone design and addresses it through denoising, works. To the best of our knowledge, this is the first work comparing fixed backbone conservation using potential energy in fixed backbone design.

Ablation study To understand the effectiveness of each additional force field supervision, we conduct an ablation study, as shown in Appendix B. All components show improvement over baseline, meaning that the folding physics inductive bias is effectively transferred to the entire networks.

6 Conclusion

In this study, we have demonstrated that denoising can learn folding physics, and based on this insight, we proposed the DNDesign framework. The proposed framework can easily be integrated into existing SOTA models and has shown performance improvements in various sequence design tasks across multiple settings. Particularly, we evaluated the impact of the acquired energy potential knowledge from the proposed framework by directly assessing its influence on potential energy through the fixed backbone conservation task. This evaluation provides evidence that the model trained with denoising generate energetically favorable sequences. This work sheds light on the significance of learning physics in structure-based protein sequence design, both theoretically and experimentally, and provides energy-based evidence. We hope that our work inspires future research in the structure-based design protein field.

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

References

Daniela Röthlisberger, Olga Khersonsky, Andrew M Wollacott, Lin Jiang, Jason DeChancie, Jamie Betker, Jasmine L Gallaher, Eric A Althoff, Alexandre Zanghellini, Orly Dym, et al. Kemp elimination catalysts by computational enzyme design. Nature , 453(7192):190-195, 2008.

Justin B Siegel, Alexandre Zanghellini, Helena M Lovick, Gert Kiss, Abigail R Lambert, Jennifer L St. Clair, Jasmine L Gallaher, Donald Hilvert, Michael H Gelb, Barry L Stoddard, et al. Computational design of an enzyme catalyst for a stereoselective bimolecular diels-alder reaction. Science , 329(5989):309-313, 2010.

Scott E Boyken, Zibo Chen, Benjamin Groves, Robert A Langan, Gustav Oberdorfer, Alex Ford, Jason M Gilmore, Chunfu Xu, Frank DiMaio, Jose Henrique Pereira, et al. De novo design of protein homo-oligomers with modular hydrogen-bond network-mediated specificity. Science , 352(6286):680-687, 2016.

Nathan H Joh, Tuo Wang, Manasi P Bhate, Rudresh Acharya, Yibing Wu, Michael Grabe, Mei Hong, Gevorg Grigoryan, and William F DeGrado. De novo design of a transmembrane zn2+-transporting four-helix bundle. Science , 346(6216):1520-1524, 2014.

Gevorg Grigoryan, Yong Ho Kim, Rudresh Acharya, Kevin Axelrod, Rishabh M Jain, Lauren Willis, Marija Drndic, James M Kikkawa, and William F DeGrado. Computational design of virus-like protein assemblies on carbon nanotube surfaces. Science , 332(6033):1071-1076, 2011.

Bassil I Dahiyat and Stephen L Mayo. Probing the role of packing specificity in protein design. Proceedings of the National Academy of Sciences , 94(19):10172-10177, 1997.

Arthur G Street and Stephen L Mayo. Computational protein design. Structure , 7(5):R105-R109, 1999.

Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International Conference on Machine Learning , pages 8946-8970. PMLR, 2022.

Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science , 378(6615):49-56, 2022.

Joel L Sussman, Dawei Lin, Jiansheng Jiang, Nancy O Manning, Jaime Prilusky, Otto Ritter, and Enrique E Abola. Protein data bank (pdb): database of three-dimensional structural information of biological macromolecules. Acta Crystallographica Section D: Biological Crystallography , 54(6):1078-1084, 1998.

Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton. Cath-a hierarchic classification of protein domain structures. Structure , 5(8):1093-1109, 1997.

John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems , 32, 2019.

Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv , 2022.

Matthew IJ Raybould, Claire Marks, Konrad Krawczyk, Bruck Taddese, Jaroslaw Nowak, Alan P Lewis, Alexander Bujotzek, Jiye Shi, and Charlotte M Deane. Five computational developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences , 116(10):4025-4030, 2019.

William R Pearson and Michael L Sierk. The limits of protein sequence comparison? Current opinion in structural biology , 15(3):254-260, 2005.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron392 neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 393 Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021. 394

395

396

James P Roney and Sergey Ovchinnikov. State-of-the-art estimation of protein model accuracy using alphafold. Physical Review Letters , 129(23):238101, 2022.

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science , 373(6557):871-876, 2021.

Yufei Huang, Lirong Wu, Haitao Lin, Jiangbin Zheng, Ge Wang, and Stan Z Li. Data-efficient protein 3d geometric pretraining via refinement of diffused protein structure decoy. arXiv preprint arXiv:2302.10888 , 2023.

Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. Rita: a study on scaling up generative protein sequence models. arXiv preprint arXiv:2205.05789 , 2022.

Lewis Moffat, Shaun M Kandathil, and David T Jones. Design in the dark: Learning deep generative models for de novo protein design. bioRxiv , 2022.

Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications , 13(1):1-10, 2022.

Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. arXiv preprint arXiv:2206.13517 , 2022.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Namrata Anand and Possu Huang. Generative modeling for protein structures. Advances in neural information processing systems , 31, 2018.

Hao Huang, Boulbaba Ben Amor, Xichan Lin, Fan Zhu, and Yi Fang. G-vae, a geometric convolutional vae for proteinstructure generation. arXiv preprint arXiv:2106.11920 , 2021.

Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277 , 2023.

Kevin E Wu, Kevin K Yang, Rianne van den Berg, James Y Zou, Alex X Lu, and Ava P Amini. Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.15611 , 2022.

Brian L Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.04119 , 2022.

Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. arXiv preprint arXiv:2209.12643 , 2022.

Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411 , 2020.

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell systems , 11(4):402-411, 2020.

Namrata Anand, Raphael Eguchi, Irimpan I Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications , 13(1):1-11, 2022.

Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigenspecific antibody design and optimization with diffusion-based generative models. bioRxiv , pages 2022-07, 2022.

Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. arXiv preprint arXiv:2110.04624 , 2021.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature , 577(7792):706-710, 2020.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

Michael Schaarschmidt, Morgane Riviere, Alex M Ganose, James S Spencer, Alexander L Gaunt, James Kirkpatrick, Simon Axelrod, Peter W Battaglia, and Jonathan Godwin. Learned force fields are ready for ground state catalyst discovery. arXiv preprint arXiv:2209.12466 , 2022.

Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133 , 2022.

Marloes Arts, Victor Garcia Satorras, Chin-Wei Huang, Daniel Zuegner, Marco Federici, Cecilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg. Two for one: Diffusion models and force fields for coarse-grained molecular dynamics. arXiv preprint arXiv:2302.00600 , 2023.

Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Veliˇ ckovi´ c, James Kirkpatrick, and Peter Battaglia. Simple gnn regularisation for 3d molecular property prediction & beyond. arXiv preprint arXiv:2106.07971 , 2021.

Zhangyang Gao, Cheng Tan, and Stan Z. Li. Pifold: Toward effective and efficient protein inverse folding. ArXiv , abs/2209.12643, 2022.

Zhangyang Gao, Cheng Tan, Stan Li, et al. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079 , 2022.

Adam Leach, Sebastian M Schmon, Matteo T Degiacomi, and Chris G Willcocks. Denoising diffusion probabilistic models on so (3) for rotational alignment. 2022.

Siegfried Matthies, J. Muller, and G. W. Vinel. On the normal distribution in the orientation space. Textures and Microstructures , 10:77-96, 1988.

Dmitry Nikolayev and Tatjana I. Savyolov. Normal distribution on the rotation group so(3). Textures and Microstructures , 29:201-233, 1997.

Jean Gallier and Dianna Xu. Computing exponentials of skew-symmetric matrices and logarithms of orthogonal matrices. International Journal of Robotics and Automation , 18(1):10-20, 2003.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary 486 differential equations. Advances in neural information processing systems , 31, 2018. 487

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, 488 Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information 489 processing systems , 30, 2017. 490

491

492

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

493

494

495

496

497

498

499

500

501

502

503

Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Z Li. Generative de novo protein design with global context. arXiv preprint arXiv:2204.10673 , 2022.

Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics , 31(6):926-932, 2015.

Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics , 82(10):2565-2573, 2014.

Noah Ollikainen and Tanja Kortemme. Computational protein design quantifies structural constraints on amino acid covariation. PLoS computational biology , 9(11):e1003313, 2013.

Rhiju Das and David Baker. Macromolecular modeling with rosetta. Annu. Rev. Biochem. , 504 77:363-382, 2008. 505

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present DNDesign, a denoising training module atop inverse folding networks (IFNN). The folding physics learning plug-in module (FPLM) is trained following score-matching with noise added to the protein backbone. It also contains five operations, including summation, cross-attention, and gated attention, that integrate the features from FPLM to IFNN. Experimental results show the PiFold with FPLM achieves superior performance on CATH 4.2 and 4.3, when compared to previous IFNNs. Besides, the work introduces a fixed backbone conservation analysis based on potential energy changes to evaluate the performance of IFNNs.\n\nSTRENGTHS:\n1. The protein inverse folding problem that the paper investigates is an emerging domain to apply deep learning techniques.\n2. The fixed backbone conservation analysis based on potential energy changes leverages physical prior to evaluate the performance of IFNNs.\n3. The idea of utilizing denoising to IFNNs is also connected to physical prior which is expected to boost the performance on inverse folding problems.\n\nWEAKNESSES:\n1. The improvement of DNDesign compared to the original PiFold may not be significant. \n2. The paper writing needs to be improved. Some notations and technical details are not clearly explained.\n3. DNDesign is claimed to be a plug-in for IFNNs. However, it is only tested with PiFold. \n\nPlease see \"Questions\" for more details.\n\nQUESTIONS:\nMajor questions:\n1. The authors mention proving denoise training is equivalent to learning the direction of energy minimization as a contribution. However, such relationship has been proven in previous works regarding molecules [1,2] and crystals [3]. In this work, the authors apply a similar denoise training strategy to biomolecules. Though the authors cite some of the works. The connection between this paper and previous works should be highlighted. \n2. In section 2.2, the authors mention that the side chains can be reconstructed by backbone and residue types, which can be inaccurate. The side chains have degrees of freedom that may not be fully reconstructed from the backbone and residue. \n3. In experiments, the authors only apply the proposed FPLM to PiFold and show improvements. How can FPLM be integrated into other IFNNs and how will FPLM affect their performance?\n4. Another concern is that the improvement from FPLM may not be significant. In terms of recovery rate, the improvements shown in Table 1 are less than 1%. Besides in Appendix A.1, FPLM shows even worse performance in PP and SR on TS50. And it is worse in SR on TS500 though slightly better in PP. Also in Appendix A.2 for multi-chain sequence design, FPLM shows better performance in 2 out of 4 models though achieving better averaged performance. Further validation of the gain of leveraging the proposed method can be helpful. \n5. Also, why apply denoise as an individual plug-in module instead of directly pre-training IFNNs via denoising? Have the authors applied the latter strategy by any chance?\n6. In section 5.3, the authors use Rosetta as an oracle in evaluating the potential energy. How accurate is Rosetta in evaluating the energy? The authors are encouraged to discuss the uncertainty. \n7. From my perspective, the title may overstate the contribution of the work. The proposed method is a plug-in to existing IFNN but not a general architecture that is \"all you need\". \n\nMinor questions:\n1. In line 35, what is \"DIFM\"?\n2. In line 84, $n_i$ is not defined in the following Eq 1. \n3. In Figure 1, the authors may consider denoting the side chains as $R_A$, $R_B$, and $R_C$ as they can be different. \n4. In line 198-199, the definition of local frames $g$ is different from line 84. And SO3 vector $\\bf r$ is not specified. \nIn line 233, what is \"DENN\"?\n\nReference\n\n[1] Zaidi, S., Schaarschmidt, M., Martens, J., Kim, H., Teh, Y.W., Sanchez-Gonzalez, A., Battaglia, P., Pascanu, R. and Godwin, J., 2022. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133.\n\n[2] Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H. and Tang, J., 2021. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728.\n\n[3] Xie, T., Fu, X., Ganea, O.E., Barzilay, R. and Jaakkola, T., 2021. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197.\n\nLIMITATIONS:\nThe authors have discussed potential limitations in Appendix B.1.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes DNDesign, a denoising-enhanced protein fixed backbone design method that effectively captures the protein energy landscape. By integrating denoising training and a plug-in module, DNDesign demonstrates its ability to generate promising protein sequences based on pre-designed structures. \nIn the inverse folding experiments, the method outperforms all the baseline methods. It also compares the diversity with the baseline, which demonstrates that the method can generate diverse suggestions for a design protein.\n\n\nupdate: I keep my score.\n\nSTRENGTHS:\nThe authors propose DNDesign, which enables the inverse-folding model to capture the deep understanding of folding physics that previous models do not fully exploit.\nThey prove how DNDesign learns folding physics directly from data and the method improves the state-of-the-art model on various protein sequence design benchmarks.\nThe authors further make a fixed backbone conservation task based on potential energy change from newly generated sequences. The analysis proves that DNDesign generates energetically favorable sequences. It is highly likely to work in real wet-lab experiments.\n\nWEAKNESSES:\nFigure 2 is a bit confusing. I think the 'noisy protein' and 'protein' should be a protein backbone without side chains?\nThe energy function and energy based distribution part is a little confusing to me. The distribution (force) is purely learning from the data? Or, the distribution is initialized with some force field, e.g. Rosetta. If the energy is purely learned from the training data, I think the submission should have some experiments to demonstrate that the energy is meaningful.\nsome typos, e.g. line 158, DEDesign.\n\nQUESTIONS:\n1. In Table 1, the PiFold paper report 51.66% recovery ratio, while this submission report 49.49% accuracy for PiFold. What's the difference between the original PiFold settings and this paper's PiFold settings?\n2. In Fixed backbone conservation study, the authors measure the Rosetta energy. I wonder whether the authors can also calculate the AlphaFold scores, which is measured in ProteinMPNN. It can further show the generated protein sequences is of high quality.\n3. The denoising step sounds time-consuming. Could the authors provide a comparison of the inference time of PiFold and the proposed method?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to use denoising diffusion probabilistic model and score-based model to solve the protein inverse folding problem, e.g., generate amino acid sequence given a protein backbone structure. Specifically, this paper used denoising diffusion model to generate unstable protein structure with higher dynamic energy and then used score-based model to learn the physical dynamic forces which drive a protein structure from unstable state to stable one. The learned physical forces are incorporated into a graph-based attention network to predict amino acid sequence. Experiments are conducted on CATH 4.2 and 4.3 datasets and compared with various approaches, showing superior performance (though not by a large margin) over the compared methods.\n\nSTRENGTHS:\n(+) Using score-based model to learn physical dynamics in protein folding is novel, informative and reasonable.\n\n(+) Code has been submitted along with the manuscript submission.\n\nWEAKNESSES:\n(-) The paper is not well organized and written, and the logic between each sections/subsections is not obvious and hard to follow.\n\n(-) The texts in Figure 2 are too small to see them clearly with ease.\n\n(-) The experimental results in Table 1 is a little bit marginal compared with PiFold, especial for NAR type.\n\nQUESTIONS:\nMethod part:\n\n1. In Figure 3 (left), what the relation among G_{H, 1}, G_{H, 2} and G_{H, 3}? Is G_{H, 2} transformed from G_{H, 1} driven by folding dynamic forces? \n\n2. How to ensure the perturbated backbone structures using Eq.2 - Eq.4 are realistic? I.e., such perturbated structures (or similar ones) do exist in nature?\n\n3. Subsections in Section 4.4 seem to be disconnected. What's the relation between \"denoising training\" (Line 207, DDPM) and \"Learning folding physics through denoising learning\" (Line 213, score-based model)? Actually DDPM is a concrete/special case of score-based model, I'm not sure how the authors train the two models (Eq.5 and Eq.8) at the same time in a single DNDesign model. \n\n4. What's the exact meaning of (1) (2) (3) (4) in Line 229?\n\n5. Could the authors provide a more detailed caption for Figure 2 to summarize the workflow of the proposed model?\n\nExperimental part:\n\n6. In Table 4 in Appendix A.4, how to compare the results with and without \"Learning folding physics through denoising learning\" (Line 213 in the main text)?\n\n7. Is it possible to provide some qualitative results? I.e., given a protein backbone structures visualization, show the predicted sequences and the ground-truth sequences?\n\nLIMITATIONS:\nThe authors did not mention any limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper combines the denoising pretraining technique with protein inverse folding models, achieving competitive results to baselines. The denoising pretraining has been proven to be effective in molecule and protein representation learning. Therefore, the rediscovery of the phenomenon in protein design is to some extent straightforward.\n\nSTRENGTHS:\n- The paper is clearly written. The presentation is good.\n- The methodology is reasonable and convincing.\n- The experimental results are positive and support the authors' claims.\n\nWEAKNESSES:\n- The innovation seems to be limited. As denoising pretraining has been proven to be effective in molecule [1] and protein [2,3] representation learning, its success in protein design is straightforward.\n\n- The denoising techniques and model design are largely proposed by existing work, which further limits the innovation of this paper\n\n- Compared to the PiFold baseline, the improvement is marginal.\n\n[1] Zaidi, Sheheryar, et al. \"Pre-training via denoising for molecular property prediction.\" arXiv preprint arXiv:2206.00133 (2022).\n\n[2] Huang, Yufei, et al. \"Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy.\" arXiv preprint arXiv:2302.10888 (2023).\n\n[3] Zhang, Zuobai, et al. \"Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction.\" arXiv preprint arXiv:2301.12068 (2023).\n\nQUESTIONS:\nQ1: How do you use the predicted structure data of AlphaFold2? Have you clustered and partitioned these data according to sequence or structure similarity?\n\nQ2: How does the noise scale affect the model? How do the authors adjust the $beta$ and $alpha$ parameters?\n\nQ3: What are the differences of your denoising strategy against previous works[2,3]?\n\n[2] Huang, Yufei, et al. \"Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy.\" arXiv preprint arXiv:2302.10888 (2023).\n\n[3] Zhang, Zuobai, et al. \"Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction.\" arXiv preprint arXiv:2301.12068 (2023).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}