Bridge-IF: Learning Inverse Protein Folding with Markov Bridges
Authors: Yiheng Zhu
Jialu Wu
Qiuyi Li
Jiahuan Yan
Mingze Yin
Wei Wu
Mingyang Li
Jieping Ye
Zheng Wang
Jian Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Inverse Protein Folding
Markov Bridges
Protein Language Models
Primary area: machine_learning_for_healthcare
TL;DR: A Markov bridge model for inverse protein folding

Abstract: Inverse protein folding is a fundamental task in computational protein design, which aims to design protein sequences that fold into the desired backbone structures. While the development of machine learning algorithms for this task has seen significant success, the prevailing approaches, which predominantly employ a discriminative formulation, frequently encounter the error accumulation issue and often fail to capture the extensive variety of plausible sequences. To fill these gaps, we propose Bridge-IF, a generative diffusion bridge model for inverse folding, which is designed to learn the probabilistic dependency between the distributions of backbone structures and protein sequences. Specifically, we harness an expressive structure encoder to propose a discrete, informative prior derived from structures, and establish a Markov bridge to connect this prior with native sequences. During the inference stage, Bridge-IF progressively refines the prior sequence, culminating in a more plausible design. Moreover, we introduce a reparameterization perspective on Markov bridge models, from which we derive a simplified loss function that facilitates more effective training. We also modulate protein language models (PLMs) with structural conditions to precisely approximate the Markov bridge process, thereby significantly enhancing generation performance while maintaining parameter-efficient training. Extensive experiments on well-established benchmarks demonstrate that Bridge-IF predominantly surpasses existing baselines in sequence recovery and excels in the design of plausible proteins with high foldability. The code is available at https://github.com/violet-sto/Bridge-IF.

Bridge-IF: Learning Inverse Protein Folding with Markov Bridges
Yiheng Zhu', Jialu Wu', Qiuyi Li Jiahuan Yan', Mingze Yin', Wei Wu',
Mingyang Li Jieping Ye', Zheng Wang"; Jian W4
Coollege of Computer Science & Technology and Liangzhu Laboratory, Zhejiang University Coolleee of Pharmaceutical Sciences, Zhejiang University
AAlibaba Cloud Computing
School of Public Health, Zhejiang University
"School of Artificial Intelligence and Data Science, University of Science and Technology of China
"The Second Affiliated Hospital Zhejiang University School of Medicine
{ Izhuyiheng2020, jialuwu, jyansir, yinmingze, wujian2000| @ @zju.edu.cn
(liqiuyi.lqy, sangheng. yejieping.ye wz388779) @alihaha-ine.com urara mail. usw..llutc.eeu...
Abstract
Inverse protein folding is a fundamental task in computational protein design, which aims to design protein sequences that fold into the desired backbone struc- tures. While the development of machine learning algorithms for this task has seen significant success, the prevailing approaches, which predominantly employ a discriminative formulation, frequently encounter the error accumulation issue and often fail to capture the extensive variety of plausible sequences. To fill these gaps, we propose Bridge-IF, a generative diffusion bridge model for inverse folding, which is designed to learn the probabilistic dependency between the distributions of backbone structures and protein sequences. Specifically, we harness an expressive structure encoder propose discrete, informative prior derived from structures, and establish a Markov bridge to connect this prior with native sequences. During the inference stage, Bridge-IF progressively refines the prior sequence, culminating in a more plausible design. Moreover, we introduce a reparameterization perspec- tive on Markov bridge models, from which we derive simplified loss function that facilitates more effective training. We also modulate protein language models (PLMs) with structural conditions precisely approximate the Markov bridge process, thereby significantly enhancing generation performance while maintaining parameter-efficien training. Extensive experiments on well-established bench- marks demonstrate that Bridge-IF predominantly surpasses existing baselines in sequence recovery and excels in the design plausible proteins with high foldabil- ity. The code is available at https: // thub. www/violet-sto.ooooooooooooooooo Brridger--ree-e IF
Introduction
Proteins are 3D folded linear chains of amino acids that execute the myriad of biological processes fundamental to life, such as catalysing metabolic reactions, mediating immune responses, and responding stimuli (23). Designing protein sequences that fold into desired 3D structures, known as inverse protein folding, is a crucial task with great potential for applications in protein engineering [29 66 [5]. Beyond estabblibd physics physics-based based methods like Rosetta (2), the considerable promise of leveraging geometric deep learning for protein structure modeling has given rise to an ongoing
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Deterministic Mapping
Markov Bridge Process
1=0
T=T
(s)s)
Px(x)
Py()
TKTIEVTDOREAEEV
TTRYTTKDENEAEKI
TGEI OVDDEENARRI
TTOYKYTDEEKARRL
~ -
y5
t=T T
Figure I: Overview Bridge-IF. Bridge-IF consists an expressive structure encoder supervised by native sequences for proposing discrete, deterministic prior, and Markov bridge model for learning the dependency between the distribution of prior sequences and the distribution of native sequences. During the inference stage, Bridge-IF progressively refines the prior sequence.
paradigm. This paradigm centered on deciphering the principles of protein design directly from data and predicting sequences corresponding specific structures 225 27
6 22
Despite substantial advancements, most existing approaches follow discriminative formulation for learning inverse folding (58), consequently encountering two principal obstacles: (i) Error accumulation issue. For instance, Transformer-ban basedd autoregressive models are constrained by their inherent sequential generation process and exposure bias, which prevents them from correcting preceding erroneous predictions. (ii) One-to-many mapping nature of the inverse folding problem. multitude of distinct amino acid sequences possess the capability fold into an identical protein backbone structure, a phenomenon exemplified by homologous proteins. Discriminative models are incapable of capturing the one-to-many mapping from the protein structure non-unique sequences, thereby facing difficulties in covering the broad spectrum of plausible solutions (58).
Recent studies have advanced the iterative refinement strategy optimize the previously generated results, aiming to reduce prediction errors (64. 14 42). These approaches employ refinement module to identify and correct inaccurately predicted amino acids. However, as the number of refinement iterations grows, managing the intermediate stages effectively becomes more challenging, potentially hindering sustained performance gains.
particularly their discrete extensions [3] which offer structured iterative refinement process with probabilistic interpretation, appear to be promising solution. Ir 58 is pioneer investigating diffusion models for inverse folding, leveraging the backbone structure to guide the denoising process on the amino acid residues. However, as diffusion models are designed to learn a single intractable data distribution, the prior distribution utilized by GraDe-IF is restricted to a simple noise distribution (i.c., uniform distribution across all residue types), which has little or no information about the distribution of native sequences. remains unclear whether this default formulation best suits conditional generative problems such as inverse protein folding, where the backbone structures provide significantly more information than random noise. Thus, exciting research question naturally arises: Can propose more strong and informative prior based on desired backbone structures to enhance the quality of samples and accelerate the inference process?
Diffusion-based generative models 45
I7
In this work, we propose Bridge-IF, a novel generative diffusion bridge model for inverse folding. Its core design is aimed at generating protein sequences from dructure-awan prior. As shown

---

in Figure 1 we leverage expressive structure encoder supervised by native sequences to propose discrete, deterministic prior based on desired structures, and build Markov bridge [T0] 24 between it and the native sequence. By approximating the reference Markov bridge process, Bridge-IF learns to progressively refine the prior sequence, resulting more plausible design. Furthermore, we present fresh reparameterizatioo perspective on Markov bridge models and derive simplified loss function that yields enhanced training effectiveness. Inspired by significant advances in protein language models (PLMs) for understanding proteins p [4], we innovatively integrate conditions, including timestep and structures, into PLMs to accurately approximate the Markov bridge process. This approach notably improves generation performance while ensuring parameter-efficien training. Empirically, we demonstrate that Bridge- outperforms state-of-the art baselines on several standard benchmarks and excels in the design of plausible proteins with high foldability.
To summarise, the main contributions of this work are as follows:
We introduce Bridge-IFF the first generative diffusion bridge model based on Markov bridges for inverse folding. We also offer a reparameterizztion perspective and derive a simplified loss function to facilitate effective training.
We innovatively adapt PLMS to effectively capture both timestep and structural information while ensuring the modified architecture is compatible with pre-trained weights.
Experiments verify that Bridge-IF achieves state-of-the-art performance on standard benchmarks.
2 Related work
2.1 Inverse protein folding
Meanwhile, the inverse problem of protein folding, which aims to infer an amino acid sequence that will fold into the desired structure, is gaining increasing attention (6. By representing protein backbone structures as a A--N graph. geometric deep learning has achieved remarkable progress in learning inverse folding 22 6 surpassing traditional physics-based approaches 22, and even facilitating the design of a range of experimentally validated proteins [ [56]. Modern deep learning- based inverse folding approaches typically comprise a structure encoder and sequence decoder. Depending on their decoding strategies, these approaches can be classified into three categories: autoregressive models, one-shot models, and iterative models. Most methods adopt the autoregressive decoding scheme to generate amino acid sequences [25 6 22 Given that autoregressive models tend to have low inference speed, some researchers have investigated one-shot methods that facilitate the parallel generation of multiple tokens [12 33) Since directly predicting highly plausible sequences is challenging, some works have shifted their attention to itcrative refinement [64][]]] 26 42, 58). For instance, LM-Design (64] and KW-dessgn [14] utilize the pre-trained knowledge from PLMs to reconstruct a native sequence from a corrupted version. The Potts model-based ChromaDesign 26 and Cerbon [4 employ iterative sampling techniques, including Markov chain Monte Carlo, to design protein sequences. GraDe-IF [58] further leverages the principles of discrete denoising diffusion probabilistic models [3), demonstrating strong capacity encompass diverse plausible solutions. In this work, we present the first generative diffusion bridge model for inverse folding.
Recently, Al algorithms have spurred a revolution in modeling protein folding 28 34
2.2 Diffusion models
Diffusion-based generative models 45 have showcased remarkable successes wide range of applications, ranging from image synthesis [8]. audio synthesis [31], to video generation [18). Generally, the essential idea behind these models is to define a forward diffusion process that gradually transforms the data into a simple prior distribution and learn a reverse denoising process to gradually recover original data samples from the prior distribution. While most existing methods are designed for modeling continuous data, a few efforts have extended diffusion models to discrete data
domains R3 52 36 Recently, diffusion models have also found utility in scientific discovery (55)
particularly in protein design [56] 59 N 15 58).
3

---

2.3 Schrödinger bridge problem
The Schrödinger bridge (SB) problem a classical antropy-regularinn optimal transport problem 32 4]. Given a data distribution, prior distribution, and a reference stochastic process between them, solving the SB problem amounts to finding the closest process the reference terms of Kullback-Leiibe
43
divergence on path spaces. This concept exhibits fundamental similarities to diffusion models [47]. particularly in the field of unconditional generative modeling [49 54 44
where the prior distribution assumes the form of Gaussian noise. Notably, SB formalism offers general framework for approximating the reference stochastic process by training on coupled samples
from two continuous distributions [19 46 35 65]. The recently proposed Markov bridge [IO [24]
has broadened the scope the SB, enabling it to model categorical distributions. this work, we
present the first diffusion bridge model for inverse protein folding.
3 Background
3.1 Problem formulation and notation
Generally, a protein can be represented as pair of amino acid sequence and structure (y. 8). where [y1. -22--..9m1l denotes its sequence of n residues with y: (1,2...., .0) indicating the type of the i-th residue, and s = [$1, $2...., Sn] E Rax4x3 denotes its structure with 8; representing the Cartesian coordinates of the i-th residue's ' backbone atoms (i.e., N, C-a, and C, with O optionally). The inverse protein folding problem aims to automatically identify the protein sequence that can fold into the given structure 8. Gíven that homologous proteins invariably exhibit similar structures, the solution for given structure is not unique [16]. Hence, an ideal model, parameterized by 0, should be capable of learning the underlying mapping from protein backbone structures to their corresponding sequence distributions py (3/ss.
3.2 Markoy bridge models
Markov bridge model [4 is general framework for learning the probabilistic dependency between two intractable discrete-vale valued distributions px and Py. For a pair of samples (xx y) ~ Px.y (x. y). it defines Markov process pinned to fixed start and end points =n and y through sequence random variables (z.)² that satisfies the Markov property,
p(ze| 20, Z1,.... 2t - 1, y) p(zz/=( V)- To pin the process at the end point zr y. we have an additional requirement, p(=r = y 11 =
(2)
Assuming that both px and py are categorical distributions with finite sample space (1,.... K}. we can represent data points as one-hot vectors: æ, y: 2t E (0, 1} and define the transition probabilities (Equation I as follows,
p(z++1/z, y) Cat (z++1i Q.z,,,
where Cat(- :P) categorical distribution with probabilities given by p, and Qe a transition matrix parameterized as
Q: = Qv(v) = B, + 1111))y1
(4) O. It is easy to see that can
where 35 is a schedule parameter transitioning from Su = 1 be efficiently sampled from p( p(z(+1/ |2 2r) = Cat (z+41: Q,20) with a cumulative product matrix Q Q.Q.-11.... B (1 3,) 1 K. where 3, I1. 8s.
Training Using the finite set of coupled samples {(x; v: pxp, Markov bridge model learns to sample y when only æ is available by approximating y with a neural network yo:
ÿ (2 (),
(5)
and defining approximated transition kernel,
(0(2+112;) = Cat (z++1: Q (1))..)
(6)
Pe is trained by optimizing the variational bound on negative log- likelihoooo log qe ())2), which has the following closed-form expression.
- log qa (y|z) < T. E.-i T-
E Be (1) DxL. (p(z+4 Ze4 1Zz, (z41))
(7)

---

Sampling To sample data point y starting from a given = x ~ (x(x)) one can iteratively predict ÿ (o(z,, t) and then derive 2441 (0(2<+1 (z)) Cat (2++1: ()) (1)z)) for
t= 0, I.
4
Methods
In this section, we introduce Bridge-IF, a Markov bridge-based model for inverse protein folding. Figure [ shows overview of our proposed Bridge-IF. Due space limitation, we present the detailed algorithm in Appendix A To begin, we describe how to extend Markov bridge techniques to facilitate the inverse protein folding task. Next, we propose a simplified training objective. Finally, we elucidate how to modulate pre-trained PLMs with structural conditions to approximate the Markov bridge process.
4.1 Overview of Bridge-IF
We frame the inverse protein folding problem as a generative problem of modeling a stochastic process between the distributions of backbone structures ps(s) and protein sequences py As previously discussed, diffusion bridge models, with their general properties of an unrestricted prior form, serves an ideal substitution for diffusion models in the presence of well-defined informative prior. Regrettably, to the best of our knowledge, no existing method can directly model the dependency between two distinct types of distributions: specifically, the continuous source distribution of backbone structures and the discrete target distribution of protein sequences.
To reconcile the differences between source and target distributions and streamline the modeling process, we propose introducing a discrete proposal distribution to serve as deterministic prior. We parameterize the proposal distribution using a structure encoder & S + X that is supervised by ground-truth target sequences, Recent advancements have demonstrated that an expressive encoder capable of directly predicting pretty good protein sequences in one-shot manner [2] This approach enables us to utilize structural information more effectively, rather than simply employing to guide the denoising process as previous diffusion-be baasdd methods like GraDe-IF [58] In this
work, we will take the discriminative model PiFold [12] as the structure encoder to produce a clean and deterministic prior x = E()). Upon this deterministic mapping from structure to sequence, we simplify the originally complex problem of modeling p( v)) into the more tractable problem of modeling p(ar, y). Then, we build a Markov bridge [10] [4 between the prior sequence and the native sequence model the stochastic process, leading to data-to-data process. As depicted in the lower half of Figure[] each sampling step progressively refines the prior sequence, which contains significant information about the target sequence, ultimately resulting in a more precise prediction.
Recall that the Markov bridge models are typically trained by optimizing the variational bound on negative likeliool log 9o ())(z) (Equation 7 which is analytically complicated and hard to optimize practice 63 62] Therefore, we here propose a reparameterization perspective on Markov bridge models, deriving simplified loss function for easier optimization $4.2
We build Markov bridges in the sequence space, treating the sequence representation as set of independent categorical random variables. To model the Markov bridge process, Q2 is applied separately each residue within protein sequence. Motivated by the impressive advancements in PLMs for understanding and generating proteins (9, 34
37 60), we advocate for employing PLMs to approximate the Markov bridge process. This approach capitalizes on the emergent evolutionary knowledge of proteins, learned from an extensive dataset of protein sequences. Additionally, we utilize the latent structural features extracted by the structure encoder to prompt PLMs, thereby guiding the generation of structurally coherent proteins. Formally, the final state of the Markov bridge process is approximated by ý = soo (zt, 8, l), foregoing the use of Equation 5 We investigate the integration of conditional information, such as timestep and structure, into PLMs, focusing on preserving their emergent knowledge and achieving parameter-efficient training $4.3)
4.2 Reparameterized Markov bridge models
Inspired by the similarities between Markov bridge models 24 and discrete diffusion models 3 63 we propose a reparameterization of the Markov bridge model characterized in 53.2 to enable more effective training. With the reparameterization trick, we introduce latent binary random variable
5

---

if U2 Bernouli( to indicate whether 21 has been transformed from Zo to 2T. Thus can be sampled from p( ((zz 2o: y) 2,20 (1 1, Accordingly, p(z Fr. y) can be equivalently written as:
2t p(ze+1|00, Fc, (11 B) + Be
if 0r
(8)
if 21 =
Using the teacher-foreing approach, can similarly define the approximation process
qo at+1 Ut, 21) ((1 (11-8))pozz))) + Bize
if ez
Proposition 4.1. The loss objective (0) for sequence a at the t-th step can be reduced to the form (0) = le log (z,, [)}
(10)
where d = 11-8 3,.
The full derivation is provided in C This derived expression of (0) formulates the training loss as a re-weighted standard multi-class cross-entropy loss function, which computed over tokens that have not been transformed to the ground truth zr- Following Ho et al. [17). set A to a constant practice. Compared to the simpler cross-entropy loss calculated across all tokens, this new formulation places greater weight tokens that require refinement. On the other hand, it conceptually simpler than the original training loss (Equation 77, which requires calculating the complicated KL divergence between two categorical distributions DKL. [p( ze+1 Ee g)llgg [=:)]-
4.3 Network architecture design space
We adopt pre-trained PLMs as the base network to approximate the final state of the Markoy bridge process. Typically, PLMs exclusively take protein sequences input during the pre-training stage, making it non-trivial to integrate timestep and structural conditions into the PLMs. Hence, innovatively tailor the Transformer blocks (50) effectively capture timestep and structural information, as depicted in Figure 2 To facilitate efficient training, the architecture of our model delicately designed for compatibility with the pre-trained weights. Our exposition emphasizes fundamental principles and the corresponding modifications to the base network.
4.3.1
AdaLasBias
Inspired by DiT [T), we explore replacing stan- dard layer norm layers in transformer blocks with adaptive layer norm (adaLN) to modulate the normalization's output based on both the timestep of the Markov bridge process and the backbone structure. The key idea is to regress the dimension-wind wise scale and shift parameters y and 3 of the layer norm from the sum of the timestep embedding and the pooled structure representation. In our situation, meaningful pre- trained parameters and 33 are readily accessi- ble. Upon commencing the fine-tuning stage, is crucial that these parameters are close to the pre-trained values to preserve the effectiveness of the original model, since a poor initialization could significantly deteriorate performance. For simplicity, we propose to predict bias A and A33 on the frozen original scalars and initialize the multi-layer perception (MLP) to output the zero-vector for all Ây and A33. We term the proposed variant of adaLN as AdaLN-Bias.
Structure
Adapter block
Up-project
Down-project
Struclure features
Structure Encoder
Cross-attention
Transformer block
Pooling
Timestep
Feed Forward
Layer Norm
Ay
Add
MLP
A
Self-altention
Ay.A
Layer Norm
Trainable
Frozen
Input tokens
Figure 2: Model architecture of Bridge-IF.

---

Considering that the pooled structure representation might only retain coarse-grained information, the network could consequently lack detailed understanding of the structure input and necessitate information derived from original structural features to compensate. We incorporate multi-head cross-attention module to the transformer block, enabling the network to flexibly interact with the Table Results comparison on the CATH dataset. Benchmarked results are quoted from Hsu et al. 22]. Zheng et al. [64]. Yi et al. [58]. Gao et al. [1]] t: "Single-chain in Hsu et al. [22 is defined differently. The best and suboptimal results are labeled with bold and underline.
Perplexity 4
Recovery Rate %
Model
Short
Single-chain
All
Short
Single-chain
All
StructGNN 25
8.29
8.74
6.40
29.44
28.26
35.91
GraphTrans
25
8,39
8.83
6.63
28, 14
28.46
35.82
GCA 48
7.09
7.49
6.05
32.62
31.10
37.64
CATH v4.2
GVP 27
7.23
7.84
5.36
30.60
28.95
39.47
AlphaDesign
7.32
7.63
6,30
34.16
32.66
41.31
ProteinMPNN 6
6.21
6,68
4.61
36.35
34.43
45.96
PiFold [12
6.04
6.31
4.55
39.84
38.53
51.66
GraDe-IF 158
5,49
6.21
4.35
45.27
42.77
52.21
With PLMs
LM-Design (ESM- 1b 650M)
6.77
6.46
4.52
37.88
42.47
55.65
KW-Design (ESM-2 650M)
14
6,05
5.29
3.90
43.32
46.30
57.38
Bridge-IF (ESM-1b 650M)
5.67
5.27
3,90
43.84
48.24
58.49
Bridge-IF (ESM-2 650M)
5.68
5.06
3.83
43.86
48.96
58.59
CATH v4.3
GVP-large [22
7.68
6.12
6.17
32.60
39.40
39.20
ESM-IF [22
8.18
6.33
6.44
31.30
38.50
38.30
+1.2M AF2 predicted data
6.05
4.00
4.01
38.10
51,50
51.60
With PLMs
LM-Design (ESM-IB lb 650M)
5.66
5.52
4.01
46.84
48.63
56.63
Bridge-IF (ESM-Ib 650M)
5.17
4.63
3.68
50.00
53.49
58.93
4.3.2 Structural adapter
structural features extracted from the structure encoder [64]. To facilitate pre-trained weights, we further integrate it into bottleneck adapter layer [21] with residual connection, preserving the input for the subsequent layers.
We stress that freeze all pre-trained parameters of the base network during training.
5 Experiments
In this section, we first demonstrate the effectiveness of our Bridge-IF on the standard CATH benchmark [40] Next, we assess Br for its applicability in de novo protein design. Moreover, we conduct several ablation studies empirically justify the key design choices. Further results pertaining the design of multi-chain protein complexes can be found in Appendix B.
5.1 Experimental protocol
Training setup We conduct experiments on both CATH v4.2 and CATH v4.3, where proteins categorized based on the CATH hierarchical classification of protein structure, ensure comprehensive analysis. Following the standard data splitting provided by Ingraham et al. 25
CATH v4.2 dataset consists of 18.024 proteins for training, 608 proteins for validation, and 1,120 proteins for testing. Following the standard data splitting provided by Hsu et al. [22). CATH v4.3 dataset consists 16,153 proteins for training, 1,457 proteins for validation, and 1.797 proteins for testing. For fair comparison with iterative models [64 [4]
we use pre-trained PiFold [2] to
propose the prior distribution. We use the cosine schedule
39 with number of timestep 7 25.
The model is trained up to 50 epochs by default on an NVIDIA 3090. We used the same training

---

settings as ProteinMPNN [6, where the batch size was set to approximately 6000 residues, and Adam optimizer 30 with noam learning rate scheduler [5T] was used.
Baselines We compare Bridge-IF with several state-of-the-arr baselines, categorized into three
groups: (1) autoregressive models, including StructGNN [25), GraphTrans [25], GCA [48] GVP 227
AlphaDesign [II., ESM-IF [22]. and ProteinMPNN [6;; (2) the one-shot model, Pifold 12: (3)
iterative models, including LM-Design
(64] KW-Design 14 and diffusion-based GraDe-IF
Evaluation We evaluate the generative quality using perplexity and recovery rate. Following previous studies [25 [2]] we report perplexity and median recovery rate on three settings, namely short proteins (length < 100), single-chain proteins (labeled with 1 chain in CATH), and all proteins.
5.2 Inverse folding
The performance of Bridge-IF, compared to competitive baselines, summarized in Table Bridge- IF demonstrates superior performance previous methods. We highlight the following: (I) Iterative models comprehensively surpass the previously dominant autoregressive and one-shot methods. (2) Our Bridge-IF outperforms LM-Design and KW-Design with the same pre-trained PLMs, supporting our hypothesis that the iterative refinement process should be modeled a probabilistic framework. (3) Compared with diffusion-barr based GraDe-IF, our Bridge-IF achieves better performance with fewer diffusion steps (25 vs. 500), demonstrating that our bridge-based formulation better leverage the structural prior.
Following Zheng et al. [64), we also study the impact of the scale of PLMs CATH v4.3.
LM-Design
BriggeIF
ESM 3R
We use ESM-2 series, with parameters rang- ing from 8M to 3B. As depicted in Figure the performance of Bridge-IF IF improves with model scaling, exhibiting distinct scaling law in logarithmic scale. Using ESM-2 at the same scale, we observe that Bridge-IF consistently obtains greater enhancements relative to LM- Design. Besides, Bridge-IF does exhibit any performance degradation, even when the small- est model (i.e, ESM-2 8M) is employed. Re- markably, the largest ESM2-3B- Bridge-IF attains a record- setting recovery rate of 61.27% on CATH v4.3.
ESM 6RMM
Recovery 8.
$5.0
$5.05
BSM- 35M
52.5
ESM-2 SM
PFFM
$0.0
50.02
47.5
16
45.0
101
variant of
parameterstritiios)
Figure 3: Performance comparison w.r.t. model scales of PLMs using ESM-2 series on CATH 4.3.
5.3 Foldability
Table 2: Numerical comparison foldability and recovery rate. Benchmarked results are quoted from Wang al. [53]. The best and suboptimal results are labeled with bold and underline.
While perplexity and recovery rate serve as ef- fective proxy metrics, it is imperative to recog- nize that these measurements may not accurately reflect the foldability of the designed protein sequences in real-world scenarios [58 53 Given that wet-lab assessment is extremely costly, we leverage the silico structure predic- tion model ESMFoold (34], to evaluate whether our designs can adhere to the structure condi- tion. Here assess the agreement of the native structures with the predicted structures using the TM-score [61], and follow the evaluation con- figurations as in Wang et al. [33] Specifically, use the small, high-quality test set of 82 sam- ples curated by Wang et al. (53] and randomly generate 100 sequences for each structure.
Model
TM-score
Recovery %
Native sequences Uniform
0.80
100.00
0.05
5.00
Natural frequencies
0.07
5.84
GraphTrans GVP
0.72
35.89
0.73
39.46
ProteinMPNN PiFold
0.80
41.44
0.71
44.86
LM-Design
0.73
51.23
Bridge-IF
0.81
54.08
We report the TM-score and recovery metrics
in Table 2
We observe that our Bridge-IF stands out the leading model, exhibiting both high

---

ID: 1007.A
PDB ID: 3ffv.A
PDB ID: 1uc2.B
Recovery Rate: 0.69
Recovery Rate 0.65
Recovery Rate 0.76 TM-score 0.99
TM-score: 0.91
TM-score 0.95
Figure 4: Folding comparison of our designed sequences (in blue) and the native sequences (in nude).
Table 3: Ablation studies of key design choices on CATH v4.2. "w/ AdaLN-Bias replaces the vanilla AdaLN with AdaLN-Bias. "w/ SCE" replaces the variational lower bound loss with simplified cross-entropy loss.
Prior
Architecture
Objective
Perplexity
Recovery Rate %
w/ pre-training
w/ AdaLN- Biaas
w/ SCE
Short
Single-chain
All
Short
Single-chain
All
V
6.51
6.30
4.23
43.17
44.29
56.53
5.98
5.27
3.89
43.45
48.01
57.92
6.52
6.40
4.28
43.43
44.01
56.43
5.68
5.06
3.83
43.86
48.96
58.59
foldability and a high recovery rate. Notably, the predicted structures of our redesigned sequences align more closely with the given structures than do the native sequences, implying better structural validity of our redesigns. Another interesting finding is that PiFold and LM-Design achieve high recovery via discriminative formulation but fall short on TM-score, indicating the limitation of siuuture-agnostie metrics. In contrast, probabilistic models Bridge-IF and ProteinMPNN perform exceptionally well on foldability. These results support our hypothesis that inverse protein folding should be modeled in probabilistic framework considering the absence of unique native sequence a given backbone structure. Figure showcases several instances where the folded structures of sequences designed by Bridge-IF are compared with reference crystal structures.
5.4
De novo protein design
Thus far, our experiments have been limited to accurate experimentally -ettemmiee structures. How- in real-world applications like de nove protein design, inverse folding models commonly used to design sequences for novel structures generated by backbone generation models [56] 26) Consequently, next evaluate Bridge-IF for its potential in such scenario. The experimental methodology is detailed as follows: we sample 10 backbones at every length [100, 105, 500] in intervals of 5 using Chroma [26). For each de design 8 sequences. Subsequently, these sequences are folded using ESMFold to identify the sequence with the highest TM-score (scTMM). We compare Bridge-IF with ProteinMPNN (6) which is widely used in de novo protein design (59 (7). Our results show that Bridge-IF surpasses ProteinMPNN in
structure, employ inverse folding models to
terms of scMM (0.73 vs. 0.69) and designability (0.85 vs. 0.80), using scMM 0.5 as the criterion.
5.5 Ablation Studies
We conduct ablation experiments on CATH v4.2 to verify the impact of key design choices, and present the results in Table
ProteinMPNN, with its order-agnostic modeling, can be viewed autoregressive diffusion model 20

---

5.5.1 Prior
We investigate two training strategies distinguished by their prior: 1) the structure encoder and the PLM are jointly trained; 2) the structure encoder is first pre-trained and remains frozen during the subsequent training of the PLM. We noted that the structure encoder is trained with equivalent objective in both strategies. The latter consistently yields higher-qualite protein sequences. Hence, has been established as our default configuration.
5.5.2 Training objective
We find that the proposed simplified cross-entropy loss works better than the variational lower bound loss (24), demonstrating that the inferior performance of the vanilla Markov bridge model may stem from harder optimization.
5.5.3 Network architecture
We observe that the performance of Bridge-IF further increases (57.92% 58.59%) when we replace the vanilla A with the proposed variant Adal NN-Bias. We highlight the use of Adal.N-Bias to enhance compatibility with pre-trained parameters when modulating a pre-trained Transformer model with additional conditions.
6 Conclusion
In this work, we introduce Bridge-IF, the first diffusion bridge model based on the Markov bridge process for inverse protein folding. Bridge-IF can gradually generate high-quality protein sequences from deterministic prior. Bridge-IF achieves -oth-oof-the-att performance in sequence recovery and foldability. Future work will focus on investigating more advanced structural encoders [38] and pre- tre-training Bridge-IF using more protein structure data predicted by AlphaFold2 [28] to further enhance performance. We also intend to apply Bridge-IF to guide protein engineering aimed at designing novel functional proteins. One potential limitation of the proposed Bridge-IF is its lack of validation through wet-lab experiments practical applications.
Acknowledgments and Disclosure of Funding
This research was partially supported by National Natural Science Foundation of China under grants No. 12326612., 12, Zhejiang Key R&D Program of China under grant No. 2023C03053 and No. 2024SSYS0026, Alibaba Research Intern Program.
References
[1] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Xijie Lu, Nicolo Fusi, Ava Pardis Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. hioRviv, pages 2023-09, 2023.
[2] Rebecca F Alford, Andrew Leaver-Fay Jeliazko R Jeliazkov, Matthew J O' O'Meara, Frank DiMaio, Hahnbeom Park, Maxim Shapovalov, Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6)3303-3044 2017.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 19999, 2021.
[4] Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. Optimal transport in systems and control. Annual Review of Control, Robotics, and Autonomous Systems, 4:89-113, 113,,11,,00 2021.
[5] Alexander Chu, Tianyu Lu, and Po-Ssu Huang. Sparks of function by de novo protein design. Nature Biotechnology, 22(2))203-215, 2024.
10

---

[6] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, al. Robust deep learning-based protein sequence design using proteinmpnn. Science, (): 2022.
[7] Valentin De Bortoli, James Thomton, Jeremy Heng, and Arnau Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:1 34:17695-17709 2021.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[9] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattem analysis and machine intelligence, 14(10):7112-713) 2021.
[10] Pat Fitzsimmons, Jim Pitman, and Marc Yor. Markovian bridges: construction, palm inter- pretation, and splicing. In Seminar on Stochastic Processes, 1992, pages 101-134, Springer, 1992.
[11] Zhangyang Gao, Cheng Tan, and Stan Z Li. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01077 2022.
[12] Zhangyang Gao, Cheng Tan, and Stan Z. Li. Pifold: Toward effective and efficient protein inverse folding. In The Eleventh International Conference Learning Representations, 2023.
URL htttps:://ppenreiiiiii wwwpeeewwwweieeeeeeeeeeeeeeeeeee htt.pp/eeeeeeeeeeeeeooooooototaa 1d=oMsN9TYJJJO
[13] Zhangyang Gao, Cheng Tan, Yijie Zhang, Xingran Chen, Lirong Wu, and Stan Z. Li. Pro- teininvbench: Benchmarking protein inverse folding on diverse tasks, models, and metrics. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. httptttsssssppreeeeeeeoe
[14] Zhangyang Gao, Cheng Tan, Xingran Chen, Yijie Zhang, Jun Xia, Siyuan Li, and Stan Z. Li. KW-design. Pushing the limit of protein design via knowledge refinement. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://ppenrviiit
net/forum? i ii==mpWWWgggii
[15] Nate Gruver, Samuel Don Stanton, Nathan C. Frey, Tim G. J. Rudner, Isidro Hotzel, Julien Lafrance- Cafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL http::////ppereeeeeeeeeeffrooeoe MXM
[16] Tymor Hamamsy, James T Morton, Robert Blackwell, Daniel Berenberg, Nicholas Carriero, Vladimir Gligorijevic, Charlic EM Strauss, Julia Koehler Leman, Kyunghyun Cho, and Richard Bonneau. Protein remote homology detection and structural alignment using deep learning. Nature biotechnology, pages 1-11, 2023.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-660 6851, 2020.
[18] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 8646, 2022.
[19] Lars Holdijk, Yuanqi Du, Priyank Jaini, Ferry Hooft, Bernd Ensing, and Max Welling, Path integral stochastic optimal control for sampling transition paths. In ICML 2022 2nd Al for Science Workshop. 2022.
[20] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. International Conference on Learning Representations, 2022. URL htttps://openreviiiie http:pppwww.pooeeeeeee/eoeaaaaaa

---

[21] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for nip. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[22] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International conference on machine learning, pages 8946-8970. PMIR, 2022.
[23] Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620);320-327, 2016.
[24] Ilia Igashov, Arne Schneuing, Marwin Segler, Michael M. Bronstein, and Bruno Correia. Retrobridge: Modeling retrosynthesis with markov bridges. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://pperreye
net/forum?
Vd=770DetV8le
[25] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[26] John B Ingraham, Max Baranov, 7ak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng- Ng-Thow-Hing, Erik R Van Vlack, al. Illumi- nating protein space with a programmable generative model. Nature, 62307989)11000-10778 2023.
[27] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021. URL httpss///ppenreviee wwtprreeviieeeeefrrrnnne. net/forum?
1d=1YL.DVVxx6J4
[28] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):5833555 2021.
[29] Hamed Khakzad, Ilia Igashov, Ame Schneuing, Casper Goverde, Michael Bronstein, and Bruno Correia. A new age in protein design empowered by deep learning. Cell Systems, 14(11): 925-939, 2023.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[31] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: http:////pp/eeeeee/////
[32] Christian Léonard. A survey of the schrödinger problem and some of its connections with optimal transport. Discrete & Continuous Dynamical Systems-A, 14(4):1533-155 2014.
[33] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 0144328-4343 2022.
[34] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction atomic-level protein structure with language model. Science, 379 2023.
[35] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. 12sb: image-to-image schrödinger bridge. In Proceedings of the 40th International Conference on Machine Learning, pages 22042-22062, 2023.
[36] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024. URL http:/tttt//ppeeeeee http::///ppppeeeeeeeeeeeeeeeeooe wttp://oror..t--....IIII.II
12

---

[37] Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et Large language models generate functional protein sequences across diverse families. Nature Biotechnology. 41)): 1099-1106, 2023.
[38] Weian Mao, Muzhi Zhu, Zheng Sun, Shuaike Shen, Lin Yuanbo Wu, Hao Chen, and Chunhua Shen. De novo protein design using geometric vector field networks. In The Twelfth International Conference on Learning Representations 2024. htt.:tttss://opreeeteeee wwwpearweieee..neeffrreeeeeee net/forum?
id=9UIGyJJpay
[39] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 81 8162-8171. PMLR, 2021.
[40] Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark Swindells, and Janet M Thornton. Cath-a hierarchic classification of protein domain structures. Structure, (8):1093-1109, 1997.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[42] Milong Ren, Chungong Yu, Dongbo Bu, and Haicang Zhang. Accurate and robust protein sequence design with carbondesign. Nature Machine Intelligence, 6(5):336-547, 2024.
[43] Erwin Schrödinger. Sur la théorie relativiste de I'électron et I'interprétation de la mécanique quantique. In Annales de l'institut Henri Poincaré, volume 2, pages 269-310, 1932.
[44] Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024.
[45] Jaseha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[46] Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence, pages 1985-1995, PMLR, 2023.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL htttps///ppenreiier pet//forumPid-PXXIII1RRRR
[48] Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu. and Stan Z Li. Global-context aware generative protein design. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[49] Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving schrödinger bridges via maximum likelihood. Entropy, 23(9): 2021.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URI http:::://rrooo /ttttp::///rroceeiingas inweeeeeeddnnnnenenne.ree.eee neurips. cc/ wc/pppppr... htppppppffllesseee/// files/ htppppppeeeess/ppp/////////////1 pappp/p////////eee $fie2445577eee11111b3cccccdd
Paper.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint 2xXi:2209.14734 2022.
13

---

[53] Chuanrui Wang, Bozitao Zhong, Zuobai Zhang, Narendra Chaudhary, Sanchit Misra, and Jian Tang. Pdb-struct: A comprehensive benchmark for structure- baseed protein design. In NeurIPS 2023 Workshop on New Frontiers of Al for Drug Discovery and Development, 2023.
[54] Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In International conference on machine learning, pages 10794-10806
PMLR, 2021.
[55] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-66, 2023.
[56] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen E Eisenach, Woody Ahern Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089 -100000999111-101111111111100000 1100, 2023.
[57] Kevin E Wu, Kevin Yang. Rianne van den Berg. Sarah Alamdari, James Zou, Alex Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature communications, 15(1):1059, 2024.
[58] Kai Yi, Bingxin Zhou, Yiqing Shen, Pietro Lio, and Yu Guang Wang. Graph denoising diffusion for inverse protein folding. In Thirtty-eventhh Conference on Neural Information Processing Systems, 2023. URL
[59] Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application protein backbone generation. In International Conference on Machine Learning, pages 40001-40039, PMLR, 2023.
[60] Mingze Yin, Hanjing Zhou, Yiheng Zhu, Miao Lin, Yixuan Wu, Jialu Wu, Hongxia Xu, Chang- Yu Hsieh, Tingjun Hou, Jintai Chen, et al. Multi-modal clip-informed protein editing. arXiv preprint arXiv:2407. 19296, 2024.
[61] Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-25 2005.
[62] Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying
discretee&coomiinuus-iiin discrete denoising diffusion. arXiv preprint arXiv:2402.03701, 2024.
[63] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.057331 2023.
[64] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure- informed language models are protein designers. In International Conference on Machine Learning, pages 42317-42338 PMLR, 2023.
[65] Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon. Denoising diffusion bridge models. In The Twelfth International Conference on Learning Representations, 2024. URL
https:://pperreveew..ttt wwtp:eeeeeeoooodddddneeeeeeeee
[66] Yiheng Zhu, Zitai Kong, Jialu Wu, Weize Liu, Yuqiang Han, Mingze Yin, Hongxia Xu, Chang- Yu Hsich, and Tingjun Hou. Generative ai for controllable protein sequence design: A survey, arXiv preprint 10255 2024.
14

---

A
Algorithms
The overall workflow of the training and sampling process are provided in Algorithm ! and Algo- rithm 2
Algorithm Training of the Bridge-IF
Input: coupled sample (s, y) ps,,, structure encoder &, neural network (Pp
x = =s)
B> Deterministic mapping from structure to sequence Sample time step and intermediate state Output of yog is vector of probabilities b Reference transition distribution Approximated transition distribution
t ~~ [(, ................................
1), 2t ~ Cat (z): 11)
x- so (2t p0(21,
p(z++1|21, Y) t- Cat (z++1: (z)))z
qa(2++1 21) +- Cat (Z+41; Qz)))
Minimize DxL. (p(z+41 (2t, y) go Ze+ l2))
Algorithm 2 Sampling
Input: starting point 8 ~ ps. structure encoder &, neural network y28
Zo +- {() for O,... T...TT- 1:
ÿ + (z((ztt] t)
Output of (p is vector of probabilities Approximated transition distribution
(0(=++1 (xt) t- Cat (z ())
Σt+1 20 (0(21+1/2))
Return 2T
B Additional results
B.1 Multi-chain protein complex design
Studying protein sequence design for multi- chain assemble structures is crucial for drug design. Next, we assess the capabilities of de- signing multi-chain complexes using the PDB dataset curated by Dauparas et al. [6), where sequences were clustered at 30% identity, sulting in 25,361 clusters. Following the stan- divided those clusters ran-
Table 4: Performance multi-chain protein com- plex dataset (in median recovery). Results of the original ProteinMPNN and GVP-Transformer were obtained using publicly available check- points.
dard data splitting, domly into three groups for training (23,358), validation (1,464), ensuring that neither the chains from the target chain nor the chains from the biounits of the target chain would be present in the other two groups.
Models
Rec. (t)
ProteinMPNN 6
50,00
ProteinMPNN CMLM [ProMMPNN-CMLM]
54,39
LM- L---Dsiggn (ProMMPNN- CMLM ESM-IB 1b 650M)
59.10
LM-Design (pretrained ProtMPNN-CMLMM freeze)
59.43
LM- A.--Dsignn (pretrained ProMPPNN-MML
tre-)ane)
59.43
LM-Design (PooMMPNN-CMLL BSM-2 650M)
59.81
As shown in Table 4 Bridge-IF also achieves similar improvements when extending to the PDB dataset, further validating its effectiveness and generalizability.
Bridge-IF (pretrained PiFold,freeze + ESM-2 650M)
61.26
These results show that Bridge-IF can not only design single-chain proteins, which are mostly studied in previous works but also be used for designing multi-chain protein complexes.
15

---

C Derivations for the variational bound of reparameterized Markov bridge models
We derive the variational bound on negative log- log-liicl log go (3/ ()) as discussed in Section 4.2
- -log ()())))) = log go 22
(0 20))
log / 2o (= u1.r (20 dvi:r dz1-T-1
log /
p( U1.T 2o. zT) p(z1:T; v1. 20, 2T)
9o (21:T, u1)) duier dz1-T-1
go (Z UI.T zo) duirr dz1-r-1
S
p(z):T. U1.T Zo: 2T) log
p p(21:T U1r Fo. Zr)
T E E..2:0....IIIIC(-(0
where
()
(p) (p((=)) (pL(p Ne 22, =T (1)0 U2, =1)] KL (p(v:)ll9o(v:))
We adopt the simplifying assumption that qa (v,) = p(v)). then (0) can be written
(0) Pe (1. =e, zr 2l =.)]]=
(11)
in which the KL divergence has the form
(y) if KL|p(ze+1| Uv, Re, (l)0lg(((4411)) lue, 2:)1 (L(()]|2))) =
(12)
vz 0
Given that KL ()) = -² log soa (ze, 1), we have
Ep(e.) KK [(p(=(+ /0,, 2c, zT x1))l19 (x))
- (1 3r v log (z).)) t)
D
Broader impacts
Inverse protein folding models, operating within the broader realm of bioinformatics and computa- tional biology, have significant impacts across various scientific and practical domains. These models, by enabling the design or prediction of protein sequences that fold into specific three-dimensionall structures, foster advancements in numerous fields. The broader impacts encompass several areas, including drug discovery, enzyme design, and synthetic biology.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification: See $4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[No]
Justification: The code and pre-trained models will be made publicly available upon accep- tance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooooSSumisssodsee
//nips.cc/ppp wwwppbblccgguiicuceccccccoeooopp
ssiiisoliccyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See $5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See
$5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "1b2N2MjnhV", "reviewer": "Reviewer_3QQQ", "summary": "In this paper the authors introduce Bridge-IF, an inverse folding approach based on a novel Markov bridge sampling formulation and both protein sequence and structure encoders. They show that leveraging both the information from pre-trained protein language models and a probabilistic, iterative sampling procedure leads to improved perplexity and sequence recovery on the CATH benchmark.", "questions": "Can the authors comment on any attempts to apply pLM and structure conditioning to alternative sampling methods, and the relative advantages of adapting the Markov bridge approach?\n\nI’m not sure what the authors mean by “discriminative formulation.” Autoregressive models are still generative models, not “discriminators” in the usual usage referring to classifiers.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Leveraging pre-trained pLMs for IF is a clear and seemingly effective idea. The demonstration in Fig 3 is also one of the clearest results to date on the effectiveness of pLM model size scaling for a downstream task. This is an exciting result and worth highlighting.", "weaknesses": "Because of the effectiveness of the pLM in approximating the Markov bridge process, it is not as clear how central the Markov bridge approach itself is to the results, versus augmenting an “autoregressive diffusion” model like ProteinMPNN or a more traditional diffusion model like GraDe-IF. It’s possible that the comparison to LM-design can clarify this point, if the authors include additional discussion. Figures 1 and 2 are particularly helpful for understanding the approach and architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> **W1**. Because of the effectiveness of the pLM in approximating the Markov bridge process, it is not as clear how central the Markov bridge approach itself is to the results, versus augmenting an “autoregressive diffusion” model like ProteinMPNN or a more traditional diffusion model like GraDe-IF. It’s possible that the comparison to LM-design can clarify this point, if the authors include additional discussion.\n>\n> **Q1**. Can the authors comment on any attempts to apply pLM and structure conditioning to alternative sampling methods, and the relative advantages of adapting the Markov bridge approach?\n\nThanks for pointing this out. To the best of our knowledge, LM-Design is among the first attempts to steer pLMs to perform protein inverse folding. The authors of LM-Design additionally introduced an autoregressive variant by adapting ProGen2, a widely-recognized autoregressive pLM, to function as a structure-based sequence designer (Appendix A.4, https://proceedings.mlr.press/v202/zheng23a/zheng23a.pdf). Their experimental results indicated that LM-Design (ESM-1b) significantly outperforms its autoregressive counterpart, LM-Design (ProGen2), thereby demonstrating that iterative refinement is more suitable for protein modeling than autoregressive methods. Furthermore, our Bridge-IF approach substantially surpasses LM-Design while utilizing the same pLM backbone, highlighting the efficacy of the Markov bridge in modeling the refinement process. Additionally, we implemented a pLM-augmented GraDe-IF by substituting the denoising network (i.e., an equivariant graph neural network) with the same pLM utilized in Bridge-IF. Despite this modification, our Bridge-IF continues to outperform GraDe-IF, underscoring the advantages of Markov bridges over diffusion models in better leveraging structural priors. These results emphasize the superiority of the Markov bridge method compared to other sampling techniques.\n\n|| Sequence recovery↑|\n|-|-|\n|LM-Design (ProGen2)|50.80|\n|GraDe-IF (ESM-1b 650M) |55.37|\n|LM-Design (ESM-1b 650M)|55.65|\n|Bridge-IF (ESM-1b 650M)|**58.49**|\n\n> **Q2**. I’m not sure what the authors mean by “discriminative formulation.” Autoregressive models are still generative models, not “discriminators” in the usual usage referring to classifiers.\n\nWe apologize for our unclear definition. We adopt the term \"discriminative formulation\" from ill-posed problems like retrosynthesis and super-resolution, where multiple valid and compatible predictions exist for a single input. In the context of protein inverse folding, we use \"discriminative formulation\" to emphasize that autoregressive and one-shot models formulate the problem as the prediction of the most probable sequence (typically the native sequence observed in the training data) for a given protein structure.\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "crytzUDtp6", "reviewer": "Reviewer_pPDQ", "summary": "The paper proposes a new method called Bridge-IF for the problem of inverse protein folding. The main objective is to develop a diffusion bridge generative model called Bridge-IF that can generate high-quality protein sequences from a structure-aware prior.  The authors claim that they outperform state-of-the-art baselines on standard benchmarks like CATH in terms of sequence recovery rate and perplexity.", "questions": "My questions are:\n- When you use CATH4.2 and CATH4.3 for evaluating PLM-based methods, there is a risk of data leakage. Because the CATH dataset may have been seen during the training of ESM models. Have you considered this? And how do you address this issue?\n- Why not report the original results of KWDesign? How does your setup differ from theirs?", "limitations": "The authors said: One potential limitation of the proposed Bridge-IF is its lack of validation through wet-lab experiments in practical applications", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "There are some strengths:\n- The paper is clearly written.\n- The consistent performance improvement across different model sizes.\n- Figures are beautiful.", "weaknesses": "Weakness include:\n- The authors have toned down the baseline results. For example, KWDesign reported recovery rates above 60% for CATH4.2 and CATH4.3 in the original paper. However, this paper drops the results to 58%. Why is this happening?\n- Compared to KWDesign, the results are not state-of-the-art. And, more importantly, this may indicate that the model could not obtain additional benefit from the diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your critical comments on our paper. However, we believe some of the concerns are caused by potential misunderstandings, and we hope that our response can address your concerns.\n\n> **W1**: The authors have toned down the baseline results. For example, KWDesign reported recovery rates above 60% for CATH4.2 and CATH4.3 in the original paper. However, this paper drops the results to 58%. Why is this happening?\n>\n> **W2**: Compared to KWDesign, the results are not state-of-the-art. And, more importantly, this may indicate that the model could not obtain additional benefit from the diffusion model.\n>\n> **Q2**: Why not report the original results of KWDesign? How does your setup differ from theirs?\n\nWe apologize for any misunderstandings caused by our unclear descriptions. Indeed, we **did not tone down** the baseline results. We aim to conduct a fair comparison using the same protein language model (pLM) backbone (i.e., ESM-2 650M) to verify the effectiveness of the proposed Markov-bridge-based method. The results of KW-Design are **quoted from the original paper** (see Table 5, https://openreview.net/pdf?id=mpqMVWgqjn). Specifically, the best performance of KW-Design (recovery rate 60.77% on CATH 4.2, 60.38% on CATH 4.3) is achieved by combining the ESM-2 650M and ESM-IF, which is pre-trained on large-scale structure data, leading an unfair comparison. Besides, in addition to refining sequences with pLM, KW-Design also introduces several complex mechanisms, such as virtual multiple sequence alignment, a confidence-aware gated attention mechanism, and memory retrieval, to enhance the model's performance. In contrast, our Bridge-IF is more simple and effective. Remarkably, despite KW-Design using more complex techniques and knowledge pre-trained on structural data, Bridge-IF with ESM-2 3B still outperforms ensemble KW-Design (recovery rate: 61.27% v.s. 60.38%) on CATH 4.3, as shown in our Figure 3.\n\n> When you use CATH4.2 and CATH4.3 for evaluating PLM-based methods, there is a risk of data leakage. Because the CATH dataset may have been seen during the training of ESM models. Have you considered this? And how do you address this issue?\n\nFirst of all, we remark that several recent works, such as LM-Design and KW-Design, have leveraged pLMs for protein inverse folding. Following these works, we used the same PLM and evaluated the proposed methods on the standard benchmark without any additional data processing. Therefore, the comparison is fair. \n\nHere are our thoughts about data leakage:\n- As pLMs such as ESMs are trained solely on sequence data, they cannot directly be applied for the inverse folding problem on their own, regardless of whether or not the proteins are part of a pLM's training data. As shown in the preliminary study of LM-Design, the authors found that directly using pLMs to revise the predicted sequences can only yield a marginal enhancement. Hence, the focus of the recent research is on how to better utilize pLMs to facilitate the inverse folding task.\n\n- To further eliminate the potential concern of model memorization and ensure that our model is capable of generalizing, we conducted experiments on a set of de novo protein structures. Specifically, we sample 10 backbones at every length [100, 105, . . . , 495, 500] in intervals of 5 (810 total samples) using Chroma [1], and generate 8 sequences for each backbone using different inverse folding methods. Then these sequences were folded using ESMFold to select the sequence with the highest TM-score (sc-TM). We found that Bridge-IF outperforms ProteinMPNN in terms of average sc-TM and Designability (sc-TM > 0.5), verifying Bridge-IF's ability to design novel sequences for de novo protein structures that were not present in its training data.\n\n|| scTM | designability (scTM > 0.5)\n|-|-|-|\n| ProteinMPNN  | 0.69 | 0.80 |\n| Bridge-IF | 0.73 | 0.85 |\n\n[1] Ingraham, John B., et al. \"Illuminating protein space with a programmable generative model.\" Nature 623.7989 (2023): 1070-1078.\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxJmG3Oa5Z", "reviewer": "Reviewer_r7sf", "summary": "The authors propose a Markov Bridge-based model for the protein inverse folding problem, incorporating a language model. This model outperforms all baseline methods on the CATH dataset. However, achieving state-of-the-art results on accurate crystal test datasets alone is not sufficient in practice. In real-world applications, protein inverse folding is often applied to structures with noise (e.g., structures generated by diffusion or predicted by software like AlphaFold).\n\nI expect the authors to add relevant experiments to illustrate the generalizability of the method.", "questions": "1. ESM-IF does not release a version that is not trained with AlphaFold DataBase. How did the authors evaluate this?\n\n2. According to my statistics, the CATH dataset has many sequences in the UniRef database, while ESM-1b and ESM-2 were trained using UniRef datasets, respectively. How do the authors deal with potential information leakage?\n\n3. Discriminative methods for sequence design based on the Potts model (like ChromaDesign [1] or CarbonDesign [2]) can also alleviate the problem of \"Error accumulation\" by sampling the amino acid pairs using a gradient-based or a multistep greedy method. Is the conclusion in line 32 too strong? I would like the author to add a relevant discussion.\n\n4. The authors mention \"accelerate the inference process\" in the motivation, but there is no comparative reasoning time. They used a pre-trained language model, a more complex network. Can the inference process be accelerated (especially compared to auto-regressive-based methods)?\n\n5. The TMscore of Bridge-IF's designed sequence exceeds that of the natural sequence. Does this indicate that the structure prediction software prefers Bridge-IF designed sequences? I would like to see a discussion from the authors about this point.\n\n[1] Illuminating protein space with a programmable generative model. Nature, 2023.\n\n[2] Accurate and robust protein sequence design with CarbonDesign. Nature Machine Intelligence, 2024.", "limitations": "The authors address limitations and potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors used the Markov bridge method for sequence design and incorporated a language model that demonstrated superior performance on the CATH dataset.\n\n2. I was pleased to see that the authors explored the performance of complex sequence design, and Bridge-IF also achieved superior performance on the complex dataset.", "weaknesses": "**Major**\n\nProtein inverse-folding software is crucial for de novo protein design. Experiments in Bridge-IF focus only on accurate experimentally determined structures, which does not prove that Bridge-IF generalizes better than other methods. In real-world applications, designs are often made for structures generated by de novo software or predicted by structure prediction software. Complementing inverse folding experiments on de novo proteins is essential. The experiment details are as follows:\n\nProteins of lengths 100-500 were designed using de novo methods (e.g., RFdiffusion or Chroma). For each de novo protein, 8 sequences were designed using each method, and then these sequences were folded using ESMFold to select the sequence with the highest TM-score.\n\nConsidering the rebuttal time constraints, showing a de novo design approach and comparing two of the strongest current baselines is sufficient to illustrate the superiority of Bridge-IF. These baselines are ChromaDesign (t=0.5 version) (https://github.com/generatebio/chroma/tree/main) and ProteinMPNN (input Ca atoms only, v_48_020 version) (https://github.com/dauparas/ProteinMPNN/tree/main/ca_model_weights).\n\nI would be happy to increase my score if Bridge-IF achieves significant improvements on the above experiments.\n\n**Minor**\n\n1. Lack of novelty. Using diffusion models and protein language models to design protein sequences is not new. This work seems to combine previous works (LM-Design, GraDe-IF) and make incremental changes.\n\n2. The improvement (1-2%) in recovery rate is limited.\n\n3. Lacks many baselines. The core motivation of Bridge-IF is to enhance the quality of samples. Although many inverse-folding methods are orthogonal to Bridge-IF, a detailed comparison is needed. Like SPDesign (http://zhanglab-bioinf.com/SPDesign/), ChromaDesign (t=0 version on CATH dataset) (https://github.com/generatebio/chroma/tree/main) , and InstructPLM (https://github.com/Eikor/InstructPLM?tab=readme-ov-file).\n\n4. The authors state that the discriminant model cannot handle the one-to-many case between protein sequence and structure. \nHowever, ProteinMPNN and ESM-IF provide sampling temperatures for generating diverse sequences, and ESM3 (also based on a discriminative model) can generate diverse structures and sequences. The article does not have a corresponding benchmark for why the diffusion-based model is better at handling one-to-many situations.\n\n6. ProteinMPNN provides many versions of models. If the authors want to compare recovery rates on experimentally determined structures, they should select the backbone atom input, add a model with little noise (0.02), and set the sampling temperature to 1e-5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> Experiment on de novo proteins.\n\nWe conducted experiments based on your suggested setups. We sample 10 backbones at every length [100, 500] in intervals of 5 using Chroma. Inspired by the finding of ProteinMPNN that training with noise improves performance for real applications, we trained a Bridge-IF with 0.2-Å noise on the dataset curated by ProteinMPNN for a fair comparison. \n\nNote that ChromaDesign is trained on the same training structure data as Chroma, it is not surprising that ChromaDesign achieves excellent performance in predicting sequences for structures generated by Chroma. In contrast, Bridge-IF and ProteinMPNN are trained on the same data and tested in the same manner, leading to a fair comparison. We found that Bridge-IF outperforms ProteinMPNN (Ca_only, v_48_020) in terms of the highest TM-score (scTM) and designability (scTM > 0.5). Because ProteinMPNN is widely used in de novo protein design and evaluating backbone generation models, we believe that Bridge-IF can generalize better to de novo structures and be applied to de novo design. Detailed distribution of the TM-score w.r.t. sequence length is presented in the PDF for global response.\n\n||scTM| designability (scTM > 0.5)\n|-|-|-|\n|ChromaDesign|0.86 | 0.98 |\n|ProteinMPNN| 0.69 | 0.80 |\n|Bridge-IF| 0.73 | 0.85 |\n\n> This work seems to combine previous works and make incremental changes.\n\nWe emphasize that Bridge-IF is not simply a combination of diffusion models and pLMs.  **We have sufficient novelty and contributions in both algorithms and network architectures:**\n\n- We present the first bridge-based model for inverse folding and offer a reparametrization of the Markov bridge model, which enhances the training of the vanilla Markov bridge model (Section 4,2). Additionally, we have implemented a pLM-augmented GraDe-IF by substituting the denoising network with ESM-1b 650M. Despite this modification, our Bridge-IF continues to outperform GraDe-IF (Recovery: 58.49 v.s. 55.37), underscoring the advantages of Markov bridges over diffusion models.\n- Our backbone is different from LM-Design with additional timestep conditions. To integrate timestep, our proposed adaLN-Bias is also different from adaLN used in DiT, designed for better integration with pre-trained parameters of LN (Section 4.3).\n\nThe effectiveness of the above design choices is verified with ablations (Section 5.4). \n\n> The improvement in recovery rate is limited.\n\nRecovery rate may not be the only criterion for evaluating the quality of the designed sequences. Bridge-IF is the first model that can accommodate both recovery and foldability. In Table 2, LM-Design can achieve performance closest to Bridge-IF on recovery, but the sequence foldability is significantly lower than that of Bridge-IF.\n\n> Lacks many baselines. \n\nFor SPDesign, the data partitioning differs from all baselines, the source code is not available, and the server is out of service currently, preventing a fair comparison. For Chroma (t=0), the inconsistent training data and unavailable training code also make comparison hard. We evaluated on the intersection of the Chroma and CATH 4.2 test set, and found Bridge-IF outperforms Chroma significantly (57.52% v.s. 51.28%). Compared with InstructPLM, Bridge-IF achieves a higher recovery rate (58.59 v.s. 57.51), despite using a smaller pLM (ESM-2 650M v.s. ProGen2 6.4B). We will add this discussion for a detailed comparison.\n\n> Benchmark for diversity.\n\nWe are sorry for not conducting a comparison of diversity due to the lack of benchmarks. For de novo protein design, we compute the diversity of 8 sequences generated for the same backbone according to [1], and find that Bridge-IF generates more diverse samples.\n\n||diversity| \n|-|-|\n|ChromaDesign|0.34| \n|ProteinMPNN|0.44| \n|Bridge-IF|0.56| \n\n[1] Biological sequence design with gflownets.\n\n> ESM-IF does not release a version that is not trained with AFDB.\n\nThe results of ESM-IF on CATH 4.3 are quoted from Table 1 in the original paper.\n\n> Potential information leakage.\n\nWe remark that recent works have leveraged pLMs for protein inverse folding. Following these works, we used the same PLM and evaluated on the standard benchmark without additional data processing. Besides, the promising performance on de novo protein design verified Bridge-IF's ability to design novel sequences that were not present in its training data.\n\n> Potts model can alleviate error accumulation.\n\nWe use discriminative formulation to emphasize that autoregressive and one-shot models formulate the problem as the prediction of the most probable sequence for a given protein structure. Hence, we do not want to include Potts model, whose sampling process is similar to diffusion models. We will add a relevant discussion about potts model.\n\n> Can the inference be accelerated, compared to autoregressive methods (ARM)?\n\nThe complexity of ARM is O(L) to the protein length L, while that of bridge/diffusion-based models is O(T) to the total timestep T. This fact demonstrates that our Bridge-IF can work with fewer number of function evaluations (NFE) than ARM, especially for larger proteins. Compared with GraDe-IF (500 NFE), Bridge-IF achieves better performance with fewer diffusion steps (25 NFE). We profiled the inference speed on a NVIDIA RTX 3090, averaging over the sampling time for sequences generated by Chroma.\n\n| Length|100-200|200-300|300-400|400-500|\n|-|-|-|-|-|\n| ChromaDesign | 4.67s| 6.74s| 8.24s| 10.29s| \n| ProteinMPNN | 1.52s | 4.52s|7.45s|13.03s|\n| Bridge-IF|1.04s| 1.56s|1.98s| 2.30s|\n\n> The TM-score of Bridge-IF's designed sequence exceeds natural sequences. \n\nThis phenomenon is also observed in the original papers of ProteinMPNN. We reasoned that Bridge-IF might design sequences for native backbones that better encode their structures, because evolution does not always prioritize structural stability. We will add this discussion in Section 5.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBaBx00OvZ", "reviewer": "Reviewer_HBUp", "summary": "This work introduces the bridge process generative model for inverse protein folding using Markov bridges to learn the probabilistic dependencies between protein backbone structures and sequences. This model aims to address the limitations of existing methods, such as error accumulation and the one-to-many mapping nature of inverse folding. Bridge-IF leverages an expressive structure encoder and integrates structural conditions into protein language models, achieving state-of-the-art accuracy on standard benchmarks.", "questions": "Please refer to the weaknesses part.", "limitations": "I do not think this work has potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "[+] The use of Markov bridges for inverse protein folding is novel. The method and background is well-introduced. \n[+] The Bridge-IF demonstrates superior performance in sequence recovery compared to existing baselines.\n[+] The integration of structural conditions into protein language models significantly uses pre-trained information, and therefore improves generation performance while maintaining parameter efficiency.", "weaknesses": "[-] While the authors show the accuracy / PPL and TM-scores, I wonder whether it is possible to show that, experimentally, the proposed method can make discovery of novel protein sequences, or engineer existing proteins. The GFP can be a common benchmark.\n[-] Instead of common benchmarks, I wonder whether the proposed method can show improvements on highly complex protein structures.\n[-] While introducing the background of bridges, I think the authors can do a more details discussion and introductions, e.g., including some missing references published in last two years Neurips / ICML conferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> While the authors show the accuracy / PPL and TM-scores, I wonder whether it is possible to show that, experimentally, the proposed method can make discovery of novel protein sequences, or engineer existing proteins. The GFP can be a common benchmark.\n\nThanks for your constructive comment. As wet-experimental assessments are exceptionally costly and time-consuming, we are currently unable to verify Bridge-IF’s ability to discover novel proteins. Nevertheless, we can leverage the zero-shot mutation effect predictive power of Bridge-IF to engineer existing proteins. Specifically, we score the mutants by conditioning the sequence likelihoods, measured as the evidence lower bound in Equation 7, on a known structure. We then compute the Spearman’s rank correlation between the model scores and the corresponding experimental measurements. On the common GFP benchmark [1], our Bridge-IF outperforms state-of-the-art inverse folding models, including ProteinMPNN and ESM-IF. The high correlation suggests that our model is likely to identify variants with enhanced fitness in real-world applications [2].\n\n|| Spearman’s rank correlation |\n|-|-|\n| ProteinMPNN | 0.602 |\n| ESM-IF | 0.713 |\n| Bridge-IF | 0.722 |\n\n[1] Sarkisyan, Karen S., et al. \"Local fitness landscape of the green fluorescent protein.\" Nature 533.7603 (2016): 397-401.\n\n[2] Hie, Brian L., et al. \"Efficient evolution of human antibodies from general protein language models.\" Nature Biotechnology 42.2 (2024): 275-283.\n\n> Instead of common benchmarks, I wonder whether the proposed method can show improvements on highly complex protein structures.\n\nOur Bridge-IF can achieve promising performance on the design of multi-chain protein complexes and de novo proteins. As shown in Appendix B.1, Bridge-IF outperforms ProteinMPNN and LM-Design on designing multi-chain proteins with highly complex structures. In addition, we also conduct experiments on de novo protein design, where inverse folding models are used to design sequence for structures generated by the structure generation model. Specifically, we sample 10 backbones at every length [100, 105, . . . , 495, 500] in intervals of 5 (810 total samples) using Chroma [3], and generate 8 sequences for each backbone using different inverse folding methods. Then these sequences were folded using ESMFold to select the sequence with the highest TM-score (sc-TM). We found that Bridge-IF outperforms ProteinMPNN in terms of average sc-TM and Designability (sc-TM > 0.5). We believe that these promising results demonstrate that Bridge-IF can generalize better to more complex structures and be applied to the design of multi-chain protein complexes and de novo proteins.\n\n|| scTM | designability (scTM > 0.5)\n|-|-|-|\n| ProteinMPNN  | 0.69 | 0.80 |\n| Bridge-IF | 0.73 | 0.85 |\n\n[3] Ingraham, John B., et al. \"Illuminating protein space with a programmable generative model.\" Nature 623.7989 (2023): 1070-1078.\n\n> While introducing the background of bridges, I think the authors can do a more details discussion and introductions, e.g., including some missing references published in last two years Neurips / ICML conferences.\n\nThanks for pointing this out. We will revise the related work to include more important references, to name a few [4,5,6], and offer a more detailed introduction.\n\n[4] Liu, Xingchao, et al. \"Let us build bridges: Understanding and extending diffusion generative models.\" arXiv preprint arXiv:2208.14699 (2022).\n\n[5] Zhou, Linqi, et al. \"Denoising Diffusion Bridge Models.\" The Twelfth ICLR.\n\n[6] Shi, Yuyang, et al. \"Diffusion Schrödinger bridge matching.\" NeurIPS 36 (2023).\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chairs and Reviewers,

In the author response period, we made diligent efforts to address reviewers' concerns and provided additional experimental results to further verify our contributions. The summary of our main efforts is presented as follows:

- We have provided a detailed explanation for the protocol, experimental details, and implementation details. (To all reviewers)

- We have further conducted diverse experiments (de novo protein design and zero-shot mutation effect prediction) to demonstrate the general applicability of the proposed method (To r7sf, HBUp, pPDQ).

- We have further conducted experiments to analyze the diversity and inference speed to demonstrate the advantages of the proposed method (To r7sf).

- We have further discussed and conducted experiments to analyze the impact of Markov bridge models (To 3QQQ).

- We have clarified the results of baselines (To pPDQ).

- We have expressed our thoughts about potential data leakage in the use of pre-trained protein language models (pLMs), and conducted experiments to demonstrate that performance improvements occur regardless of whether the protein sequences are included in the pLM's training data (To pPDQ, r7sf). 

In our individual responses, we provide detailed answers to all the specific questions raised by the reviewers. Further discussions are welcomed to facilitate the reviewing process toward a comprehensive evaluation of our work.

Meta-review: The authors' method seems to work quite well on an important problem, and is applying a technique that--while relatively newly studied in the literature--hasn't been applied to inverse folding before. Ultimately, there's straightforward justification for acceptance: they evaluate on the right datasets, and beat very competitive recent methods like PiFold. Finally, I do generally agree that the authors' results on de novo proteins do demonstrate that their method is quite competitive and generally answers the concern about information leakage. I *do* agree with Reviewer pPDQ that the authors' should include the remaining baselines here in the final version, but am nevertheless generally in favor of acceptance.
Decision: Accept (poster)