Bridge-IF: Learning Inverse Protein Folding with Markov Bridges

Authors: yiheng  zhu
jialu  wu
qiuyi  li
jiahuan  yan
mingze  yin
wei  wu
mingyang  li
jieping  ye
zheng  wang
jian  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:38:59

Abstract: Inverse protein folding is a fundamental task in computational protein design, which aims to design protein sequences that fold into the desired backbone structures. While the development of machine learning algorithms for this task has seen significant success, the prevailing approaches, which predominantly employ a discriminative formulation, frequently encounter the error accumulation issue and often fail to capture the extensive variety of plausible sequences. To fill these gaps, we propose Bridge-IF, a generative diffusion bridge model for inverse folding, which is designed to learn the probabilistic dependency between the distributions of backbone structures and protein sequences. Specifically, we harness an expressive structure encoder to propose a discrete, informative prior derived from structures, and establish a Markov bridge to connect this prior with native sequences. During the inference stage, Bridge-IF progressively refines the prior sequence, culminating in a more plausible design. Moreover, we introduce a reparameterization perspective on Markov bridge models, from which we derive a simplified loss function that facilitates more effective training. We also modulate protein language models (PLMs) with structural conditions to precisely approximate the Markov bridge process, thereby significantly enhancing generation performance while maintaining parameter-efficient training. Extensive experiments on well-established benchmarks demonstrate that Bridge-IF predominantly surpasses existing baselines in sequence recovery and excels in the design of plausible proteins with high foldability. The code is available at https://github.com/violet-sto/Bridge-IF.

Main paper: 
 introduction: 
 Proteins are 3D folded linear chains of amino acids that execute the myriad of biological processes fundamental to life, such as catalysing metabolic reactions, mediating immune responses, and responding to stimuli [23] . Designing protein sequences that fold into desired 3D structures, known as inverse protein folding, is a crucial task with great potential for applications in protein engineering [29, 66, 5] . Beyond long-established physics-based methods like Rosetta [2] , the considerable promise of leveraging geometric deep learning for protein structure modeling has given rise to an ongoing paradigm. This paradigm is centered on deciphering the principles of protein design directly from data and on predicting sequences corresponding to specific structures [25, 27, 6, 22] .Despite substantial advancements, most existing approaches follow a discriminative formulation for learning inverse folding [58] , consequently encountering two principal obstacles: (i) Error accumulation issue. For instance, Transformer-based autoregressive models are constrained by their inherent sequential generation process and exposure bias, which prevents them from correcting preceding erroneous predictions. (ii) One-to-many mapping nature of the inverse folding problem. A multitude of distinct amino acid sequences possess the capability to fold into an identical protein backbone structure, a phenomenon exemplified by homologous proteins. Discriminative models are incapable of capturing the one-to-many mapping from the protein structure to non-unique sequences, thereby facing difficulties in covering the broad spectrum of plausible solutions [58] .Recent studies have advanced the iterative refinement strategy to optimize the previously generated results, aiming to reduce prediction errors [64, 14, 42] . These approaches employ a refinement module to identify and correct inaccurately predicted amino acids. However, as the number of refinement iterations grows, managing the intermediate stages effectively becomes more challenging, potentially hindering sustained performance gains.Diffusion-based generative models [45, 17] , particularly their discrete extensions [3] , which offer a structured iterative refinement process with probabilistic interpretation, appear to be a promising solution. GraDe-IF [58] is a pioneer in investigating diffusion models for inverse folding, leveraging the backbone structure to guide the denoising process on the amino acid residues. However, as diffusion models are designed to learn a single intractable data distribution, the prior distribution utilized by GraDe-IF is restricted to a simple noise distribution (i.e., a uniform distribution across all residue types), which has little or no information about the distribution of native sequences. It remains unclear whether this default formulation best suits conditional generative problems such as inverse protein folding, where the backbone structures provide significantly more information than random noise. Thus, an exciting research question naturally arises: Can we propose a more strong and informative prior based on desired backbone structures to enhance the quality of samples and accelerate the inference process?In this work, we propose Bridge-IF, a novel generative diffusion bridge model for inverse folding. Its core design is aimed at generating protein sequences from a structure-aware prior. As shown in Figure 1 , we leverage an expressive structure encoder supervised by native sequences to propose a discrete, deterministic prior based on desired structures, and build a Markov bridge [10, 24] between it and the native sequence. By approximating the reference Markov bridge process, Bridge-IF learns to progressively refine the prior sequence, resulting in a more plausible design. Furthermore, we present a fresh reparameterization perspective on Markov bridge models and derive a simplified loss function that yields enhanced training effectiveness. Inspired by significant advances in protein language models (PLMs) for understanding proteins [9, 34] , we innovatively integrate conditions, including timestep and structures, into PLMs to accurately approximate the Markov bridge process. This approach notably improves generation performance while ensuring parameter-efficient training. Empirically, we demonstrate that Bridge-IF outperforms state-of-the-art baselines on several standard benchmarks and excels in the design of plausible proteins with high foldability.To summarise, the main contributions of this work are as follows:• We introduce Bridge-IF, the first generative diffusion bridge model based on Markov bridges for inverse folding. We also offer a reparameterization perspective and derive a simplified loss function to facilitate effective training.• We innovatively adapt PLMs to effectively capture both timestep and structural information while ensuring the modified architecture is compatible with pre-trained weights.• Experiments verify that Bridge-IF achieves state-of-the-art performance on standard benchmarks.2 Related work 
inverse protein folding: 
 Recently, AI algorithms have spurred a revolution in modeling protein folding [28, 34] . Meanwhile, the inverse problem of protein folding, which aims to infer an amino acid sequence that will fold into the desired structure, is gaining increasing attention [6] . By representing protein backbone structures as a k-NN graph, geometric deep learning has achieved remarkable progress in learning inverse folding [25, 6, 22] , surpassing traditional physics-based approaches [2] , and even facilitating the design of a range of experimentally validated proteins [6, 56] . Modern deep learning-based inverse folding approaches typically comprise a structure encoder and a sequence decoder. Depending on their decoding strategies, these approaches can be classified into three categories: autoregressive models, one-shot models, and iterative models. Most methods adopt the autoregressive decoding scheme to generate amino acid sequences [25, 6, 22] . Given that autoregressive models tend to have low inference speed, some researchers have investigated one-shot methods that facilitate the parallel generation of multiple tokens [12, 38] . Since directly predicting highly plausible sequences is challenging, some works have shifted their attention to iterative refinement [64, 14, 26, 42, 58] . For instance, LM-Design [64] and KW-design [14] utilize the pre-trained knowledge from PLMs to reconstruct a native sequence from a corrupted version. The Potts model-based ChromaDesign [26] and CarbonDesign [42] employ iterative sampling techniques, including Markov chain Monte Carlo, to design protein sequences. GraDe-IF [58] further leverages the principles of discrete denoising diffusion probabilistic models [3] , demonstrating a strong capacity to encompass diverse plausible solutions. In this work, we present the first generative diffusion bridge model for inverse folding.Diffusion-based generative models [45, 17] have showcased remarkable successes in a wide range of applications, ranging from image synthesis [8] , audio synthesis [31] , to video generation [18] . Generally, the essential idea behind these models is to define a forward diffusion process that gradually transforms the data into a simple prior distribution and learn a reverse denoising process to gradually recover original data samples from the prior distribution. While most existing methods are designed for modeling continuous data, a few efforts have extended diffusion models to discrete data domains [3, 33, 52, 36] . Recently, diffusion models have also found utility in scientific discovery [55] , particularly in protein design [56, 59, 1, 15, 58] .The Schrödinger bridge (SB) problem is a classical entropy-regularized optimal transport problem [43, 32, 4] . Given a data distribution, a prior distribution, and a reference stochastic process between them, solving the SB problem amounts to finding the closest process to the reference in terms of Kullback-Leibler divergence on path spaces. This concept exhibits fundamental similarities to diffusion models [47] , particularly in the field of unconditional generative modeling [49, 54, 7, 44] , where the prior distribution assumes the form of Gaussian noise. Notably, SB formalism offers a general framework for approximating the reference stochastic process by training on coupled samples from two continuous distributions [19, 46, 35, 65] . The recently proposed Markov bridge [10, 24] has broadened the scope of the SB, enabling it to model categorical distributions. In this work, we present the first diffusion bridge model for inverse protein folding.3 Background The inverse protein folding problem aims to automatically identify the protein sequence y that can fold into the given structure s. Given that homologous proteins invariably exhibit similar structures, the solution for a given structure is not unique [16] . Hence, an ideal model, parameterized by θ, should be capable of learning the underlying mapping from protein backbone structures to their corresponding sequence distributions p θ (y|s). 
markov bridge models: 
 Markov bridge model [24] is a general framework for learning the probabilistic dependency between two intractable discrete-valued distributions p X and p Y . For a pair of samples (x, y) ∼ p X ,Y (x, y), it defines a Markov process pinned to fixed start and end points z 0 = x and z T = y through a sequence of random variables (z t ) T t=0 that satisfies the Markov property, p(z t |z 0 , z 1 , . . . , z t-1 , y) = p(z t |z t-1 , y).(1) To pin the process at the end point z T = y, we have an additional requirement, p(z T = y|z T -1 , y) = 1.(2) Assuming that both p X and p Y are categorical distributions with a finite sample space {1, . . . , K}, we can represent data points as one-hot vectors: x, y, z t ∈ {0, 1} K , and define the transition probabilities (Equation 1) as follows,EQUATIONwhere Cat(• ; p) is a categorical distribution with probabilities given by p, and Q t is a transition matrix parameterized asEQUATIONwhere β t is a schedule parameter transitioning from β 0 = 1 to β T -1 = 0. It is easy to see that z t can be efficiently sampled from p(z t+1 |z 0 , z T ) = Cat z t+1 ; Q t z 0 with a cumulative product matrixQ t = Q t Q t-1 ...Q 0 = β t I K + (1 -β t )y1 ⊤ K , where β t = t s=0 β s .Training Using the finite set of coupled samples {(x i , y i )} D i=1 ∼ p X ,Y , Markov bridge model learns to sample y when only x is available by approximating y with a neural network φ θ :EQUATIONand defining an approximated transition kernel,EQUATIONφ θ is trained by optimizing the variational bound on negative log-likelihood log q θ (y|x), which has the following closed-form expression,EQUATIONSampling To sample a data point y ≡ z T starting from a given z 0 ≡ x ∼ p X (x), one can iteratively predict ŷ = φ θ (z t , t) and then derive z t+1 ∼ q θ (zt+1 |z t ) = Cat (z t+1 ; Q t ( ŷ)z t ) for t = 0, . . . , T -1. 
methods: 
 In this section, we introduce Bridge-IF, a Markov bridge-based model for inverse protein folding.Figure 1 shows an overview of our proposed Bridge-IF. Due to space limitation, we present the detailed algorithm in Appendix A. To begin, we describe how to extend Markov bridge techniques to facilitate the inverse protein folding task. Next, we propose a simplified training objective. Finally, we elucidate how to modulate pre-trained PLMs with structural conditions to approximate the Markov bridge process.We frame the inverse protein folding problem as a generative problem of modeling a stochastic process between the distributions of backbone structures p S (s) and protein sequences p Y (y). As previously discussed, diffusion bridge models, with their general properties of an unrestricted prior form, serves as an ideal substitution for diffusion models in the presence of a well-defined informative prior. Regrettably, to the best of our knowledge, no existing method can directly model the dependency between two distinct types of distributions: specifically, the continuous source distribution of backbone structures and the discrete target distribution of protein sequences.To reconcile the differences between source and target distributions and streamline the modeling process, we propose introducing a discrete proposal distribution to serve as a deterministic prior. We parameterize the proposal distribution using a structure encoder E : S → X that is supervised by ground-truth target sequences. Recent advancements have demonstrated that an expressive encoder is capable of directly predicting pretty good protein sequences in a one-shot manner [12] . This approach enables us to utilize structural information more effectively, rather than simply employing it to guide the denoising process as in previous diffusion-based methods like GraDe-IF [58] . In this work, we will take the discriminative model PiFold [12] as the structure encoder to produce a clean and deterministic prior x = E(s). Upon this deterministic mapping from structure to sequence, we simplify the originally complex problem of modeling p(s, y) into the more tractable problem of modeling p(x, y). Then, we build a Markov bridge [10, 24] between the prior sequence and the native sequence to model the stochastic process, leading to a data-to-data process. As depicted in the lower half of Figure 1 , each sampling step progressively refines the prior sequence, which contains significant information about the target sequence, ultimately resulting in a more precise prediction.Recall that the Markov bridge models are typically trained by optimizing the variational bound on negative log-likelihood log q θ (y|x) (Equation 7), which is analytically complicated and hard to optimize in practice [63, 62] . Therefore, we here propose a reparameterization perspective on Markov bridge models, deriving a simplified loss function for easier optimization ( §4.2).We build Markov bridges in the sequence space, treating the sequence representation as a set of independent categorical random variables. To model the Markov bridge process, Q t is applied separately to each residue within a protein sequence. Motivated by the impressive advancements in PLMs for understanding and generating proteins [9, 34, 37, 60] , we advocate for employing PLMs to approximate the Markov bridge process. This approach capitalizes on the emergent evolutionary knowledge of proteins, learned from an extensive dataset of protein sequences. Additionally, we utilize the latent structural features extracted by the structure encoder to prompt PLMs, thereby guiding the generation of structurally coherent proteins. Formally, the final state of the Markov bridge process is approximated by ŷ = φ θ (z t , s, t), foregoing the use of Equation 5. We investigate the integration of conditional information, such as timestep and structure, into PLMs, focusing on preserving their emergent knowledge and achieving parameter-efficient training ( §4.3).Inspired by the similarities between Markov bridge models [24] and discrete diffusion models [3, 63] EQUATIONUsing the teacher-forcing approach, we can similarly define the approximation process asEQUATIONProposition 4.1. The loss objective L t (θ) for sequence x at the t-th step can be reduced to the formEQUATIONwhereλ t = 1 -β t .The full derivation is provided in C. This derived expression of L t (θ) formulates the training loss as a re-weighted standard multi-class cross-entropy loss function, which is computed over tokens that have not been transformed to the ground truth y = z T . Following Ho et al. [17] , we set λ t to a constant 1 in practice. Compared to the simpler cross-entropy loss calculated across all tokens, this new formulation places greater weight on tokens that require refinement. On the other hand, it is conceptually simpler than the original training loss (Equation 7), which requires calculating the complicated KL divergence between two categorical distributions D KL [p(z t+1 |z t , y)∥q θ (z t+1 |z t )].We adopt pre-trained PLMs as the base network to approximate the final state of the Markov bridge process. Typically, PLMs exclusively take protein sequences as input during the pre-training stage, making it non-trivial to integrate timestep and structural conditions into the PLMs. Hence, we innovatively tailor the Transformer blocks [50] to effectively capture timestep and structural information, as depicted in Figure 2 . To facilitate efficient training, the architecture of our model is delicately designed for compatibility with the pre-trained weights. Our exposition emphasizes fundamental principles and the corresponding modifications to the base network. Inspired by DiT [41] , we explore replacing standard layer norm layers in transformer blocks with adaptive layer norm (adaLN) to modulate the normalization's output based on both the timestep of the Markov bridge process and the backbone structure. The key idea is to regress the dimension-wise scale and shift parameters γ and β of the layer norm from the sum of the timestep embedding and the pooled structure representation. In our situation, meaningful pretrained parameters γ and β are readily accessible. Upon commencing the fine-tuning stage, it is crucial that these parameters are close to the pre-trained values to preserve the effectiveness of the original model, since a poor initialization could significantly deteriorate performance. For simplicity, we propose to predict bias ∆γ and ∆β on the frozen original scalars and initialize the multi-layer perception (MLP) to output the zero-vector for all ∆γ and ∆β. We term the proposed variant of adaLN as adaLN-Bias. Considering that the pooled structure representation might only retain coarse-grained information, the network could consequently lack a detailed understanding of the structure input and necessitate information derived from original structural features to compensate. We incorporate a multi-head cross-attention module to the transformer block, enabling the network to flexibly interact with the structural features extracted from the structure encoder [64] . To facilitate pre-trained weights, we further integrate it into a bottleneck adapter layer [21] with residual connection, preserving the input for the subsequent layers.We stress that we freeze all pre-trained parameters of the base network during training. 
experiments: 
 In this section, we first demonstrate the effectiveness of our Bridge-IF on the standard CATH benchmark [40] . Next, we assess Bridge-IF for its applicability in de novo protein design. Moreover, we conduct several ablation studies to empirically justify the key design choices. Further results pertaining to the design of multi-chain protein complexes can be found in Appendix B.1.Training setup We conduct experiments on both CATH v4.2 and CATH v4.3, where proteins are categorized based on the CATH hierarchical classification of protein structure, to ensure a comprehensive analysis. Following the standard data splitting provided by Ingraham et al. [25] , CATH v4.2 dataset consists of 18,024 proteins for training, 608 proteins for validation, and 1,120 proteins for testing. Following the standard data splitting provided by Hsu et al. [22] , CATH v4.3 dataset consists of 16,153 proteins for training, 1,457 proteins for validation, and 1,797 proteins for testing. For a fair comparison with iterative models [64, 14] , we use pre-trained PiFold [12] to propose the prior distribution. We use the cosine schedule [39] with number of timestep T = 25.The model is trained up to 50 epochs by default on an NVIDIA 3090. We used the same training settings as ProteinMPNN [6] , where the batch size was set to approximately 6000 residues, and Adam optimizer [30] with noam learning rate scheduler [51] was used.Baselines We compare Bridge-IF with several state-of-the-art baselines, categorized into three groups: (1) autoregressive models, including StructGNN [25] , GraphTrans [25] , GCA [48] , GVP [27] , AlphaDesign [11] , ESM-IF [22] , and ProteinMPNN [6] ;(2) the one-shot model, PiFold [12] ; (3) iterative models, including LM-Design [64] , KW-Design [14] , and diffusion-based GraDe-IF [58] .Evaluation We evaluate the generative quality using perplexity and recovery rate. Following previous studies [25, 22] , we report perplexity and median recovery rate on three settings, namely short proteins (length ≤ 100), single-chain proteins (labeled with 1 chain in CATH), and all proteins.The performance of Bridge-IF, compared to competitive baselines, is summarized in Table 1 . Bridge-IF demonstrates superior performance over previous methods. We highlight the following: Following Zheng et al. [64] , we also study the impact of the scale of PLMs on CATH v4.3. We use ESM-2 series, with parameters ranging from 8M to 3B. As depicted in Figure 3 While perplexity and recovery rate serve as effective proxy metrics, it is imperative to recognize that these measurements may not accurately reflect the foldability of the designed protein sequences in real-world scenarios [58, 13, 53] .Given that wet-lab assessment is extremely costly, we leverage the in silico structure prediction model ESMFold [34] , to evaluate whether our designs can adhere to the structure condition. Here we assess the agreement of the native structures with the predicted structures using the TM-score [61] , and follow the evaluation configurations as in Wang et al. [53] . Specifically, we use the small, high-quality test set of 82 samples curated by Wang et al. [53] and randomly generate 100 sequences for each structure.We report the TM-score and recovery metrics in Table 2 . We observe that our Bridge-IF stands out as the leading model, exhibiting both high foldability and a high recovery rate. Notably, the predicted structures of our redesigned sequences align more closely with the given structures than do the native sequences, implying better structural validity of our redesigns. Another interesting finding is that PiFold and LM-Design achieve high recovery via a discriminative formulation but fall short on TM-score, indicating the limitation of structure-agnostic metrics. In contrast, probabilistic models Bridge-IF and ProteinMPNN,2 perform exceptionally well on foldability. These results support our hypothesis that inverse protein folding should be modeled in a probabilistic framework considering the absence of a unique native sequence for a given backbone structure. Figure 4 showcases several instances where the folded structures of sequences designed by Bridge-IF are compared with reference crystal structures.Thus far, our experiments have been limited to accurate experimentally-determined structures. However, in real-world applications like de novo protein design, inverse folding models are commonly used to design sequences for novel structures generated by backbone generation models [56, 26] . Consequently, we next evaluate Bridge-IF for its potential in such a scenario. The experimental methodology is detailed as follows: we sample 10 backbones at every length [100, 105, . . . , 500] in intervals of 5 using Chroma [26] . For each de novo structure, we employ inverse folding models to design 8 sequences. Subsequently, these sequences are folded using ESMFold to identify the sequence with the highest TM-score (scTM). We compare Bridge-IF with ProteinMPNN [6] , which is widely used in de novo protein design [59, 57] . Our results show that Bridge-IF surpasses ProteinMPNN in terms of scTM (0.73 vs. 0.69) and designability (0.85 vs. 0.80), using scTM > 0.5 as the criterion.We conduct ablation experiments on CATH v4.2 to verify the impact of key design choices, and present the results in Table 3 .We investigate two training strategies distinguished by their prior: 1) the structure encoder and the PLM are jointly trained; 2) the structure encoder is first pre-trained and remains frozen during the subsequent training of the PLM. We noted that the structure encoder is trained with an equivalent objective in both strategies. The latter consistently yields higher-quality protein sequences. Hence, it has been established as our default configuration.We find that the proposed simplified cross-entropy loss works better than the variational lower bound loss [24] , demonstrating that the inferior performance of the vanilla Markov bridge model may stem from a harder optimization.We observe that the performance of Bridge-IF further increases (57.92% → 58.59%) when we replace the vanilla AdaLN with the proposed variant AdaLN-Bias. We highlight the use of AdaLN-Bias to enhance compatibility with pre-trained parameters when modulating a pre-trained Transformer model with additional conditions. 
conclusion: 
 In this work, we introduce Bridge-IF, the first diffusion bridge model based on the Markov bridge process for inverse protein folding. Bridge-IF can gradually generate high-quality protein sequences from a deterministic prior. Bridge-IF achieves state-of-the-art performance in sequence recovery and foldability. Future work will focus on investigating more advanced structural encoders [38] and pre-training Bridge-IF using more protein structure data predicted by AlphaFold2 [28] to further enhance performance. We also intend to apply Bridge-IF to guide protein engineering aimed at designing novel functional proteins. One potential limitation of the proposed Bridge-IF is its lack of validation through wet-lab experiments in practical applications.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See §5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See §5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See §5.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The overall workflow of the training and sampling process are provided in Algorithm 1 and Algorithm 2.Input: coupled sample (s, y) ∼ p S,Y , structure encoder E, neural network φ θ x = E(s) ▷ Deterministic mapping from structure to sequence t ∼ U(0, . . . , T -1), Studying protein sequence design for multichain assemble structures is crucial for drug design. Next, we assess the capabilities of designing multi-chain complexes using the PDB dataset curated by Dauparas et al. [6] , where sequences were clustered at 30% identity, resulting in 25,361 clusters. Following the standard data splitting, we divided those clusters randomly into three groups for training (23, 358) , validation (1, 464) , ensuring that neither the chains from the target chain nor the chains from the biounits of the target chain would be present in the other two groups.z t ∼ Cat z t ; Q t-1 x ▷ Sample time step and intermediate state ŷ ← φ θ (z t , t) ▷ Output of φ θ is a vector of probabilities p(z t+1 |z t , y) ← Cat (z t+1 ; Q t (y)z t ) ▷ Reference transition distribution q θ (z t+1 |z t ) ← Cat (z t+1 ; Q t ( t ) ▷ Approximated transition distribution Minimize D KL (p(z t+1 |z t , y)∥q θ (z t+1 |z t )) Algorithm 2 Sampling Input: starting point s ∼ p S , structure encoder E, neural network φ θ z 0 ← E(s) for t in 0, ..., T -1: ŷ ← φ θ (z t , t) ▷ Output of φ θ is a vector of probabilities q θ (z t+1 |z t ) ← Cat (z t+1 ; Q t ( ŷ)z t ) ▷ Approximated transition distribution z t+1 ∼ q θ (z t+1 |z t ) Return z TAs shown in Table 4 , Bridge-IF also achieves similar improvements when extending to the PDB dataset, further validating its effectiveness and generalizability.These results show that Bridge-IF can not only design single-chain proteins, which are mostly studied in previous works but also be used for designing multi-chain protein complexes.We derive the variational bound on negative log-likelihood log q θ (y|x) as discussed in Section 4.2.-log q θ (y|x) = -log q θ (zT |z 0 ) = -log q θ (z 1:T , v 1:T |z 0 ) dv 1:T dz 1:T -1 = -log p(z 1:T , v 1:T |z 0 , z T ) p(z 1:T , v 1:T |z 0 , z T ) q θ (z 1:T , v 1:T |z 0 ) dv 1:T dz 1:T -1 ≤ -p(z 1:T , v 1:T |z 0 , z T ) log q θ (z 1:T , v 1:T |z 0 ) p(z 1:T , v 1:T |z 0 , z T ) dv 1:T dz 1:T -1 = T • E t∼U (0,...,T -1) L t (θ)whereL t (θ) = E p(zt|x,y) E p(vt) [KL(p(z t+1 |v t , z t , z T )||q (z t+1 |v t , z t ))] + KL (p(v t )∥q θ (v t )) .We adopt the simplifying assumption that q θ (v t ) = p(v t ), then L t (θ) can be written asEQUATIONin which the KL divergence has the formEQUATIONGiven that KL(y||φ θ (z t , t)) = -y T log φ θ (z t , t), we haveE p(vt) [KL [(p(z t+1 |v t , z t , z T )||q θ (z t+1 |v t , z t ))] = -(1 -β t )v t y T log φ θ (z t , t)Inverse protein folding models, operating within the broader realm of bioinformatics and computational biology, have significant impacts across various scientific and practical domains. These models, by enabling the design or prediction of protein sequences that fold into specific three-dimensional structures, foster advancements in numerous fields. The broader impacts encompass several areas, including drug discovery, enzyme design, and synthetic biology.Answer: [No] Justification: The code and pre-trained models will be made publicly available upon acceptance of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.ProteinMPNN, with its order-agnostic modeling, can be viewed as an autoregressive diffusion model[20]. 


Decision: The authors' method seems to work quite well on an important problem, and is applying a technique that--while relatively newly studied in the literature--hasn't been applied to inverse folding before. Ultimately, there's straightforward justification for acceptance: they evaluate on the right datasets, and beat very competitive recent methods like PiFold. Finally, I do generally agree that the authors' results on de novo proteins do demonstrate that their method is quite competitive and generally answers the concern about information leakage. I *do* agree with Reviewer pPDQ that the authors' should include the remaining baselines here in the final version, but am nevertheless generally in favor of acceptance.

Reviews: review: paper_summary: paper_summary: In this paper the authors introduce Bridge-IF, an inverse folding approach based on a novel Markov bridge sampling formulation and both protein sequence and structure encoders. They show that leveraging both the information from pre-trained protein language models and a probabilistic, iterative sampling procedure leads to improved perplexity and sequence recovery on the CATH benchmark.
strength_weakness: strength_weakness: Because of the effectiveness of the pLM in approximating the Markov bridge process, it is not as clear how central the Markov bridge approach itself is to the results, versus augmenting an “autoregressive diffusion” model like ProteinMPNN or a more traditional diffusion model like GraDe-IF. It’s possible that the comparison to LM-design can clarify this point, if the authors include additional discussion. Figures 1 and 2 are particularly helpful for understanding the approach and architecture.
questions: questions: Can the authors comment on any attempts to apply pLM and structure conditioning to alternative sampling methods, and the relative advantages of adapting the Markov bridge approach?

I’m not sure what the authors mean by “discriminative formulation.” Autoregressive models are still generative models, not “discriminators” in the usual usage referring to classifiers.
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a new method called Bridge-IF for the problem of inverse protein folding. The main objective is to develop a diffusion bridge generative model called Bridge-IF that can generate high-quality protein sequences from a structure-aware prior.  The authors claim that they outperform state-of-the-art baselines on standard benchmarks like CATH in terms of sequence recovery rate and perplexity.
strength_weakness: strength_weakness: Weakness include:
- The authors have toned down the baseline results. For example, KWDesign reported recovery rates above 60% for CATH4.2 and CATH4.3 in the original paper. However, this paper drops the results to 58%. Why is this happening?
- Compared to KWDesign, the results are not state-of-the-art. And, more importantly, this may indicate that the model could not obtain additional benefit from the diffusion model.
questions: questions: My questions are:
- When you use CATH4.2 and CATH4.3 for evaluating PLM-based methods, there is a risk of data leakage. Because the CATH dataset may have been seen during the training of ESM models. Have you considered this? And how do you address this issue?
- Why not report the original results of KWDesign? How does your setup differ from theirs?
limitations: limitations: The authors said: One potential limitation of the proposed Bridge-IF is its lack of validation through wet-lab experiments in practical applications
paper_summary: paper_summary: The authors propose a Markov Bridge-based model for the protein inverse folding problem, incorporating a language model. This model outperforms all baseline methods on the CATH dataset. However, achieving state-of-the-art results on accurate crystal test datasets alone is not sufficient in practice. In real-world applications, protein inverse folding is often applied to structures with noise (e.g., structures generated by diffusion or predicted by software like AlphaFold).

I expect the authors to add relevant experiments to illustrate the generalizability of the method.
strength_weakness: strength_weakness: **Major**

Protein inverse-folding software is crucial for de novo protein design. Experiments in Bridge-IF focus only on accurate experimentally determined structures, which does not prove that Bridge-IF generalizes better than other methods. In real-world applications, designs are often made for structures generated by de novo software or predicted by structure prediction software. Complementing inverse folding experiments on de novo proteins is essential. The experiment details are as follows:

Proteins of lengths 100-500 were designed using de novo methods (e.g., RFdiffusion or Chroma). For each de novo protein, 8 sequences were designed using each method, and then these sequences were folded using ESMFold to select the sequence with the highest TM-score.

Considering the rebuttal time constraints, showing a de novo design approach and comparing two of the strongest current baselines is sufficient to illustrate the superiority of Bridge-IF. These baselines are ChromaDesign (t=0.5 version) (https://github.com/generatebio/chroma/tree/main) and ProteinMPNN (input Ca atoms only, v_48_020 version) (https://github.com/dauparas/ProteinMPNN/tree/main/ca_model_weights).

I would be happy to increase my score if Bridge-IF achieves significant improvements on the above experiments.

**Minor**

1. Lack of novelty. Using diffusion models and protein language models to design protein sequences is not new. This work seems to combine previous works (LM-Design, GraDe-IF) and make incremental changes.

2. The improvement (1-2%) in recovery rate is limited.

3. Lacks many baselines. The core motivation of Bridge-IF is to enhance the quality of samples. Although many inverse-folding methods are orthogonal to Bridge-IF, a detailed comparison is needed. Like SPDesign (http://zhanglab-bioinf.com/SPDesign/), ChromaDesign (t=0 version on CATH dataset) (https://github.com/generatebio/chroma/tree/main) , and InstructPLM (https://github.com/Eikor/InstructPLM?tab=readme-ov-file).

4. The authors state that the discriminant model cannot handle the one-to-many case between protein sequence and structure. 
However, ProteinMPNN and ESM-IF provide sampling temperatures for generating diverse sequences, and ESM3 (also based on a discriminative model) can generate diverse structures and sequences. The article does not have a corresponding benchmark for why the diffusion-based model is better at handling one-to-many situations.

6. ProteinMPNN provides many versions of models. If the authors want to compare recovery rates on experimentally determined structures, they should select the backbone atom input, add a model with little noise (0.02), and set the sampling temperature to 1e-5.
questions: questions: 1. ESM-IF does not release a version that is not trained with AlphaFold DataBase. How did the authors evaluate this?

2. According to my statistics, the CATH dataset has many sequences in the UniRef database, while ESM-1b and ESM-2 were trained using UniRef datasets, respectively. How do the authors deal with potential information leakage?

3. Discriminative methods for sequence design based on the Potts model (like ChromaDesign [1] or CarbonDesign [2]) can also alleviate the problem of "Error accumulation" by sampling the amino acid pairs using a gradient-based or a multistep greedy method. Is the conclusion in line 32 too strong? I would like the author to add a relevant discussion.

4. The authors mention "accelerate the inference process" in the motivation, but there is no comparative reasoning time. They used a pre-trained language model, a more complex network. Can the inference process be accelerated (especially compared to auto-regressive-based methods)?

5. The TMscore of Bridge-IF's designed sequence exceeds that of the natural sequence. Does this indicate that the structure prediction software prefers Bridge-IF designed sequences? I would like to see a discussion from the authors about this point.

[1] Illuminating protein space with a programmable generative model. Nature, 2023.

[2] Accurate and robust protein sequence design with CarbonDesign. Nature Machine Intelligence, 2024.
limitations: limitations: The authors address limitations and potential negative societal impact.
paper_summary: paper_summary: This work introduces the bridge process generative model for inverse protein folding using Markov bridges to learn the probabilistic dependencies between protein backbone structures and sequences. This model aims to address the limitations of existing methods, such as error accumulation and the one-to-many mapping nature of inverse folding. Bridge-IF leverages an expressive structure encoder and integrates structural conditions into protein language models, achieving state-of-the-art accuracy on standard benchmarks.
strength_weakness: strength_weakness: [-] While the authors show the accuracy / PPL and TM-scores, I wonder whether it is possible to show that, experimentally, the proposed method can make discovery of novel protein sequences, or engineer existing proteins. The GFP can be a common benchmark.
[-] Instead of common benchmarks, I wonder whether the proposed method can show improvements on highly complex protein structures.
[-] While introducing the background of bridges, I think the authors can do a more details discussion and introductions, e.g., including some missing references published in last two years Neurips / ICML conferences.
questions: questions: Please refer to the weaknesses part.
limitations: limitations: I do not think this work has potential negative societal impact.

Comments: title: Reply to authors
Reponse
Thanks for your reply
Looking forward to your feedback
Thanks for the rebuttal
Response about related work
Response about de novo protein design
comment: We sincerely appreciate your valuable comments which greatly help us to improve our work and will incorporate the discussions and results into the final version.
We sincerely appreciate your valuable comments which greatly help us to improve our work and will incorporate these discussions and results into the final version.
We sincerely appreciate your valuable comments which greatly help us to improve our work and will incorporate the discussions and results into the final version.
Dear reviewer,

We would like to first express our sincere gratitude for your time and effort in reviewing our paper. We appreciate your valuable suggestions and will incorporate the discussions and results into the final version.

Considering the author-reviewer discussion is ending very soon, could you kindly check our responses.

We would greatly appreciate it if the reviewer would consider adjusting the score, based on our response and other review comments.

Sincerely, Authors
> You did many things unrelated to the key question, but have not yet convinced me. The best way to evaluate this is to compare an un-pretrained model with a pretrained model.

We agree that comparing a non-pretrained model with a pretrained model is a suitable approach (see the below response to other comments for a detailed discussion). However, in our initial rebuttal, we have conducted experiments on **de novo protein design, where the risk of data leakage is nonexistent, as the ESM model has not encountered these novel proteins during pre-training**. Our Bridge-IF outperforms robust and strong baselines ProteinMPNN and ChromaDesign, which have been validated with wet-lab experiments, also providing strong evidence to alleviate concerns regarding data leakage.

> ESM2 650M w/o pretraining works even worse than the baseline. But, with pretraining, the model can outperform baseline significantly. This shows the pretraining knowledge performs a key role in improving the performance, this is exactly what I say data leackage. 

The phenomenon that ESM2 650M w/ pretraining significantly outperforms ESM2 650M w/o pretraining can **only demonstrate the importance of pre-training, and it is not evidence for information leakage**. As previously emphasized, **we have only 20k training samples, which is extremely insufficient for training a large Transformer model with 650M parameters from scratch**. In practice, larger Transformers typically require more large-scale datasets for training to achieve good convergence and generalization capabilities [1,2,3]. Due to the lack of inductive biases in Transformers, directly training these large models on small-sized datasets always results in poor performance [2,3,4]. Taking ESM [4] used in our experiment as an example, in its original paper, we observe that **non-pretrained ESM achieved the worst performance across all tasks**, i.e., a performance drop of about 30% compared to the pre-trained version (**Table 2, 3, 4, and 5 in https://www.pnas.org/doi/pdf/10.1073/pnas.2016239118**), which is aligned with our results of Bridge-IF (ESM-2 650M w/o pretraining).

[1] Kaplan, et al. "Scaling laws for neural language models."

[2] Radford, et al. "Improving Language Understanding by Generative Pre-Training."

[3] Dosovitskiy, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." 

[4] Rives, et al. "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences."

> You also use ESM2 8M to prove pretraining do not improve the performance, even resulting in negative affect. However, this is not correlated to your main results, and all related works using ESM-650M find that pretraining is important to the performance. In your rebuttal, LM-Design (ESM-2 8M) even performs worse than ProteinMPNN, which is strange.

Based on the analysis above, we compare the pretrained and non-pretrained ESM2 8M to **address your concern about data leakage, rather than to prove that pre-training does not enhance performance**. As we only have 20k proteins for training, **when trained from scratch, the smallest ESM2 8M is the only model that achieves good performance**. The comparable performance of pretrained ESM2 8M and non-pretrained ESM2 8M proves that the performance improvement does not stem from the potential for data leakage

The results of LM-Design (ESM-2 8M) and ProteinMPNN are quoted from the original paper of LM-Design (Figure 6, https://proceedings.mlr.press/v202/zheng23a/zheng23a.pdf). The reason for this phenomenon is that even though ESM-2 8M has been pre-trained, **its limited model capacity leads to weaker generalization performance in fine-tuning tasks**. This experiment demonstrates that the improvement achieved through pre-training stems from **scaling capability of large models**, rather than from information leakage.

> I'm still confused how much performance gain comes from diffusion and pretraining, respectively. Therefore, we don't know how the data leakage issue affected your model. 

In the previous response, we compared Bridge-IF to LM-Design and GraDe-IF using the **same ESM-1b 650M** to verify the performance gain from our diffusion bridge model. In Figure 3 of our paper, we have shown **consistent performance improvement across pre-trained pLMs with different sizes**, which is acknowledged in your initial review. 

In summary, the experimental results indicate that: **1) both pre-training and the diffusion bridge model significantly enhance model performance and 2) the improvement derived from pre-training is primarily attributed to the scaling capability of large models. As the model size increases, it exhibits a greater ability to learn general evolutionary knowledge about proteins from extensive datasets**.

**We hope that our clearer explanation can alleviate your concerns. We kindly ask you to consider adjusting the scores. Please let us know if you have any additional questions.**
Thanks to the author's response, I have no additional questions about Bridge-IF
I think the final version of the paper after the rebuttal process of Bridge-IF could have the following changes:

(1) In the conclusion section, the authors only state further wet-experiment validation, and do not say what else can be explored in the machine learning direction, which is something that would have far-reaching implications for the machine learning community. For example, the authors can add some conclusions from the following aspects:

i) The exploration of input features, both PiFold and SPDesign discuss and illustrate the input features further, 

ii) Pre-training features, I am concerned that KW-Design and SPDesign use protein structure pre-training models, whereas Bridge-IF uses pre-training of sequences only, 

iii) The tendency of sequence design to be a function of features, e.g. CarbonDesign performed experiments on DMS and ClinVar to illustrate the link between sequence design and functional proteins. 

iv) Interpretability: what exactly is the framework improvement of the diffusion model over autoregressive, and why can auto-regressive text-based generation achieve very good results while sequence-generated diffusion would be better? 

v) Scaling, ESM-IF uses the AlphaFold Database as a training set, while ESM3 uses a wider range of data sources, how to use more data and scaling is also a concern of the field

(2)Presentation of experimental details: In CarbonDesign's comparison test, I found that not all CASP15 data used by CarbonDesign is publicly available, and there are new CASP15 data published after CarbonDesign's work, is the author's test set different from CarbonDesign? CarbonDesign demonstrated performance for orphan, and long sequences, and future work could explore these.

(3)Clear conclusions from the Rebuttal process.

(4) Considerable work has been done on protein sequence design, and the authors could consider summarising the Beidge-IF position more clearly in tabular form

I would like to thank the authors for their responses again during the rebuttal process and their contributions to the field of protein design and machine learning applications, and I would recommend acceptance of this article. 

Best wishes!
Thanks for the author rebuttal.
 
Your initial rebuttal is "As pLMs such as ESMs are trained solely on sequence data, they cannot directly be applied for the inverse folding problem on their own, regardless of whether or not the proteins are part of a pLM's training data. As shown in the preliminary study of LM-Design, the authors found that directly using pLMs to revise the predicted sequences can only yield a marginal enhancement. Hence, the focus of the recent research is on how to better utilize pLMs to facilitate the inverse folding task." But my question is about data leackage. Your initial rebuttal could not address my concern. You did many things unrelated to the key question, but have not yet convinced me. The best way to evaluate this is to compare an un-pretrained model with a pretrained model. 

In my request, you provided the results, where ESM2 650M w/o pretraining works even worse than the baseline. But, with pretraining, the model can outperform baseline significantly. This shows the pretraining knowledge performs a key role in improving the performance, this is exactly what I say data leackage. You also use ESM2 8M to prove pretraining do not improve the performance, even resuling in negative affect. However, this is not correlated to your main results, and all related works using ESM-650M find that pretraining is important to the performance.

Up to now, I'm still confused how much performace gain comes from diffusion and pretraining, respectively. Therefore, we don't know how the data leakage issue affected your model. In addition, in your rebuttal, LM-Design (ESM-2 8M) even performs worse than ProteinMPNN baseline, which is strange. This either indicates ESM-8M do not suitable for this task, or you have made some wrong in the experiments.

Considering above concerns, I retain my score.
We appreciate the reviewer for providing insightful comments!

> I can not agree with your claim, that is "pLMs to revise the predicted sequences can only yield a marginal enhancement". You provide me the evidence of LMDesign, where sequence predictions of the ProteinMPNN were fed as input into the ESM-1b, without any training. In this case, LMDesign improves the recovery from approximately 49% to 50%. However, with fintuning pLM, LMDesign increase the performance to 54.62%. Your model falls into the fintuning case, therefore, your claim is not correct.

Thanks for the comment. Due to the intermittent discussion, misunderstandings may have arisen regarding this claim. As stated in our initial rebuttal, our claim aims to indicate that pLMs pre-trained exclusively on sequence data cannot be directly applied to inverse folding problems without fine-tuning, even if they have encountered native sequences during the pre-training phase. Our second rebuttal is conducted to explain in detail.

We believe that this proof-of-concept experiment without fine-tuning can mitigate concerns about the potential for data leakage. If data leakage is significant, the pLMs should demonstrate a substantial improvement in predictions without fine-tuning, as nearly half (49%) of the input sequences were identical to the native sequences. For an analysis of the experiments in the fine-tuning case, please refer to the response below.

> ESM2 8M may be not enough, you should use the ESM-650M to see how much performance gain is obtained from pLM's pretraining. I want you to clarify how much each of pLM and diffusion contributes to the overall improvement.

The reason we chose ESM-2 8M is that the CATH v4.2 dataset only consists of 18,024 proteins for training, and we cannot train a larger Transformer model from scratch. Following your request, we utilized the ESM-2 650M as the backbone and trained it from scratch. However, as expected, Bridge-IF did not work and even produced worse predictions than the prior (50% for PiFold). We wish to emphasize that our performance achieved with pre-trained ESM-2 650M is not the result of information leakage; rather, training large Transformers from scratch presents inherent challenges. Conversely, experiments with ESM-2 8M reveal that Bridge-IF (w/o pre-training) achieved comparable or slightly superior performance compared to Bridge-IF (w/ pre-training). This phenomenon may be due to the inconsistency in the data distribution and training objectives during the pre-training and fine-tuning stages. This finding indicates that the observed performance improvements cannot be attributed to information leakage, thereby validating the efficacy of our model. 

|backbone| recovery rate | 
|-|-|
|ESM2 8M w/o pretraining | **51.27** |
|ESM2 8M | 51.18 |
|ESM2 650M w/o pretraining| 46.41|
|ESM2 650M|**58.59**|

Additionally, we conducted further ablation studies to verify the contribution of our diffusion bridge approach. Bridge-IF significantly outperforms both LM-Design and the traditional diffusion model GraDe-IF while **utilizing the same pLM backbone (ESM-1b 650M)**. This highlights the effectiveness of the Markov bridge in modeling the refinement process. The result for LM-Design is quoted from the original paper, while GraDe-IF was implemented by replacing the denoising network with the same pLM used in Bridge-IF during the early rebuttal stage.

||Sequence recovery↑|
|-|-|
|LM-Design (PiFold + ESM-1b 650M) | 55.65 |
|GraDe-IF (ESM-1b 650M)|55.37|
|Bridge-IF (PiFold + ESM-1b 650M)|**58.49**|

> ChromaDesign is not targeted to inverse folding. You can try other strong inverse folding baselines.

We apologize for not providing a clear description of ChromaDesign. Indeed, **ChromaDesign is an inverse folding model**. Chroma is a generative model capable of directly sampling novel protein structures and sequences. It comprises a backbone network trained as a diffusion model to generate structures and a design network (i.e., ChromaDesign, G.3 in https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-023-06728-8/MediaObjects/41586_2023_6728_MOESM1_ESM.pdf) that facilitates robust sequence prediction based on the generated structures (I.2, J.5, and J.6). Both ProteinMPNN and ChromaDesign have demonstrated their ability to generate de novo proteins through wet-lab experiments, showcasing they are strong inverse folding baselines on the de novo protein design. The performance of Bridge-IF surpasses that of ProteinMPNN and ChromaDesign, indicating that it can generalize to de novo proteins, alleviating concerns about information leakage.

We have made every effort to address your concerns regarding data leakage through a proof-of-concept experiment without fine-tuning, ablation studies in the fine-tuning context, and de novo protein design experiments. **If our response resolves your concerns, we kindly ask you to consider adjusting the scores. Please let us know if you have any additional questions**.
I thank the authors for their response. The related work discussion will be helpful and the additional experiments are useful for understanding the model performance. I will keep my weak accept score.
Thanks for the author rebuttal. I update my comments:

**Q1** I can not agree with your claim, that is "pLMs to revise the predicted sequences can only yield a marginal enhancement". You provide me the evidence of  LMDesign, where sequence predictions of the ProteinMPNN were fed as input into the ESM-1b, without any training. In this case, LMDesign improves the recovery from approximately 49% to 50%. **However, with fintuning pLM, LM design increase the performance to 54.62%.** Your model falls into the fintuning case, therefore, your claim is not correct.

**Q2** ESM-2 8M may be not enough, you should use the ESM-650M to see how much performance gain is obtained from pLM's pretraining.  In my experience, it is very strange that pLMs that are not pre-trained outperform pre-trained pLMs in this task. I’m not trying to make things difficult for you; I just want you to clarify how much each of pLM and diffusion contributes to the overall improvement. Finetuning ESM-650M is not expensive on CATH dataset. 

**Q3** ChromaDesign is not targeted to inverse folding. You can try other strong inverse folding baselines on the de-novo dataset. But the rebuttal ddl is approaching, you do not need to do such experiments.
Dear reviewer,

We would like to first express our sincere gratitude for your time and effort in reviewing our paper. We appreciate your valuable suggestions and will incorporate the discussions and results into the final version.

Considering the author-reviewer discussion is ending very soon, could you kindly check our responses and let us know if you have further concerns? We are more than willing to address any other concerns or questions.

We would greatly appreciate it if the reviewer would consider adjusting the score, based on our response and other review comments.  

Sincerely, Authors
We sincerely appreciate the reviewer for engaging in the discussion and providing further insightful comments which greatly help us to improve our work!

> I'm confused about the claim: "The authors found that directly using pLMs to revise the predicted sequences can only yield a marginal enhancement." Could you give me some evidence to support this?

We apologize for the oversight in accurately citing the source. In the proof-of-concept for LM-Design (Section 2.3, https://proceedings.mlr.press/v202/zheng23a/zheng23a.pdf), the authors studied whether a pre-trained pLM (ESM-1b 650M) is capable of refining predicted sequences generated by the advanced inverse folding model ProteinMPNN. As illustrated in Figure 1C, this approach yields immediate but marginal gains in sequence recovery, increasing from approximately 49% to 50%. While this experiment demonstrates the potential for further leveraging pLMs to enhance protein sequence design, it also indicates that even for the structure of a native protein sequence present in the ESM's pretraining sequence data, ESM still cannot significantly improve the prediction.

> Your goal is to design sequence, therefore, pLM actually brings the potential of data leackage. Could you provide the ablation results by removing the pLM on the CATH dataset? You can use the same model architecture, but do not load pretrained weights.

Thanks for your constructive suggestion. We have performed ablation studies on a non-pretrained model. As it is challenging to train a large model from scratch on limited training data, we opted for the smallest model (i.e., ESM-2 8M), as our backbone network. We quote the results of baselines from Figure 3 of our paper. We found that Bridge-IF, when not pre-trianed, slightly surpasses the performance of the Bridge-IF that underwent pretraining. Additionally, we observed an interesting phenomenon where LM-Design (ESM-2 8M) showed a decrease in performance relative to the ProteinMPNN. These observations suggest that the potential for data leakage is unlikely to contribute to Bridge-IF's superior performance.

|  | recovery rate | 
|-|-|
| ProteinMPNN | 48.18 |
| LM-Design (ESM-2 8M) | 47.58 |
| Bridge-IF (ESM-2 8M) | 51.18 |
| Bridge-IF (ESM-2 8M w/o pretraining) | **51.27** |

> For the new results, ProteinMPNN probably to be weak. Could you provide more baseline comparisons?

We would like to emphasize that the preliminary results presented in our initial rebuttal aimed to demonstrate that Bridge-IF can also generalize to de novo backbone structures for which the corresponding native sequences are unavailable. This capability addresses the potential concern regarding model memorization of the pre-training sequence data. Besides, it is important to note that de novo protein design presents a considerable challenge, since the de novo structures are not as high-resulution as experimentally determined structures used for model training. Most of the baselines compared in the main paper have not been specifically optimized for this task. Indeed, ProteinMPNN is a very robust model, which is widely utilized in de novo protein design and the evaluation of protein backbone generation models [1,2,3].

Fortunately, during our discussions with Reviewer r7sf, we have dived into the application of the inverse folding model for de novo protein design. We summarize our findings as follows: 1) Adding noise to the training data is crucial for robust de novo protein design; 2) There is a trade-off between diversity and sample quality, with models using a lower sampling temperature achieving better performance on this challenging task. Inspired by these findings, we successfully proposed an improved and robust Bridge-IF model for de novo protein design. Specifically, we trained Bridge-IF with 0.2-Å noise and sampled from it at a lower temperature. 

In experiments with structures generated by FrameDiff [2], compared with the state-of-the-art ChromaDesign [4], recently published in Nature, which made a lot of improvements in de novo protein sequences, our Bridge-IF demonstrates comparable performance in terms of scTM and designability (scTM > 0.5), while outperforming on the more stringent designability metric (scRMSD < 2Å).

|  | scTM | designability (scTM > 0.5) | designability (scRMSD < 2Å)
| - | - | - | - |
| ChromaDesign  | 0.86 | 0.99 | 0.41 |
| Bridge-IF | 0.86 | 0.99 | **0.44** |

[1] Watson, Joseph L., et al. "De novo design of protein structure and function with RFdiffusion." Nature 2023.

[2] Yim, Jason, et al. "SE (3) diffusion model with application to protein backbone generation." ICML 2023.

[3] Wu, Kevin E., et al. "Protein structure generation via folding diffusion." Nature communications 2024.

[4] Ingraham, John B., et al. "Illuminating protein space with a programmable generative model." Nature 2023.

**If our response resolves your concerns, we kindly ask you to consider adjusting the scores. Please let us know if you have any additional questions.**
I thank the authors for their response and the above clarifications. I have read the response and those to the other reviewers and will keep my score the same.
Thanks for the author response. I have some quesions:

**Q1** I'm confused about the claim:  "The authors found that directly using pLMs to revise the predicted sequences can only yield a marginal enhancement." Could you give me some evidence to support this? 

**Q2** Your goal is to design sequence, therefore, pLM actually brings the potential of data leackage. Could you provide the ablation results by removing the pLM on the CATH dataset? You can use the same model architecture, but do not load pretrained weights.

**Q3** For the new results, ProteinMPNN probably to be weak. Could you provide more baseline comparisons?
Thank you very much for your timely reply and recognition of our efforts! 

> Although the authors' responses are complete enough, for Question1, I still want to know under the standard experimental setup (100-500 lengths, using either 8 times or 1 sequence design method for each protein), for RFdiffusion or FrameDiff-designed proteins, the standard setups of ChromaDesign (t=0.01) and Bridge-IF (default setting) performance comparison. Because in my use and review, ChromaDesign is very robust software in de novo protein sequence design.

The results are presented in the table below. We also found that ChromaDesign is a robust software capable of generalizing to the backbone structures generated by FrameDiff.

|  | scTM | designability (scTM > 0.5) | designability (scRMSD < 2Å)
| - | - | - | - |
| ChromaDesign  (t=0.01)          | 0.86     | 0.99  | 0.41 |
| Bridge-IF (low temperature)    | 0.86     | 0.99  | **0.44** |
| Bridge-IF (default)                   | 0.81     | 0.90  | 0.36 |

**We appreciate your valuable comments and will incorporate the discussions and results into the final version to improve the impact and completeness of Bridge-IF.**
Thanks for your reply, the author has addressed most of my questions and I think the Bridge-IF experiment and results are complete.
**Therefore, I recommend acceptance of this article.** I have raised my score.

I recommend that the authors to (1) add relevant work (2) add relevant discussion (3) add the experiments in de novo to the paper, which will help to improve the impact and completeness of Bridge-IF. 

Although the authors' responses are complete enough, for Question1, I still want to know under the standard experimental setup (100-500 lengths, using either 8 times or 1 sequence design method for each protein), for RFdiffusion or FrameDiff-designed proteins, the standard setups of ChromaDesign (t=0.01) and Bridge-IF (defult setting) performance comparison. Because in my use and review, ChromaDesign is very robust software in de novo protein sequence design.
> The authors state that the Potts-model based approach has no error accumulation problems. Then the Potts-model approach is still a potential baseline as CarbonDesign (default), and I would like the authors to add a relevant comparison in the CATH dataset.

Thanks for the constructive comment. Unfortunately, the training script for CarbonDesign is unavailable, which complicates a fair comparison in the CATH dataset. Additionally, according to the description in the original paper, CarbonDesign was trained on PDB data prior to 2020. Given that the CATH4.2 dataset is partitioned based on PDB data from before 2019 [1], there is a potential risk of data leakage when directly validating CarbonDesign on the CATH4.2 test set. Therefore, we opted to test our Bridge-IF on the CASP15 test set for comparison with CarbonDesign. Our results indicate that Bridge-IF outperforms state-of-the-art baselines, including CarbonDesign.

|  | recovery rate (the results of baselines are quoted from CarbonDesign)| 
| -------- | -------- | 
| ProteinMPNN_002 | 0.48 |
| ESM-IF | 0.50 |
| CarbonDesign | 0.54 |
| Bridge-IF | **0.566** |

[1] Generative models for graph-based protein design, 2019.

> I would like to see more discussion from the author about the Potts model.

The Potts model, a class of energy-based models, is a statistical mechanics framework utilized to describe interactions among spins on a lattice. Potts models have been applied to capture the sequence coevolution information of proteins [2,3]. Typically, the parameters of the Potts model are inferred using plmDCA [4] or bmDCA [5], and Markov Chain Monte Carlo (MCMC) sampling [6] can be employed to generate novel sequences. In the context of inverse folding, ChromaDesign factorizes the conditional distribution of sequences as a conditional Potts model with the likelihood given by
$$
p_\theta(s|x) = \frac{1}{Z(x,\Theta)} \exp \left( -\sum_i h_i(s_i;x) - \sum_{i<j} J_{ij}(s_i,s_j|x) \right),
$$
where the conditional fields $h_i(s_i;x)$ and conditional couplings $J_{ij}(s_i,s_j|x)$ are parameterized by the node and edge embeddings of the graph neural network, respectively. Similarly, CarbonDesign proposes Inverseformer to derive representations from the input backbone structures while conditioning fields and couplings on the learned single and paired representations, respectively. 

We contend that the strength of the Potts model lies in its explicit formulation of probability distributions that align with domain knowledge. However, its limitations include a restriction to modeling only second-order effects, a requirement for significantly more iterations of Monte Carlo sampling, and potential issues with slow mode mixing.

[2] Hopf, Thomas A., et al. "Mutation effects predicted from sequence co-variation." Nature biotechnology.

[3] Russ, William P., et al. "An evolution-based model for designing chorismate mutase enzymes." Science。

[4] Ekeberg, Magnus, Tuomo Hartonen, and Erik Aurell. "Fast pseudolikelihood maximization for direct-coupling analysis of protein structure from many homologous amino-acid sequences." Journal of Computational Physics.

[5] Figliuzzi, Matteo, Pierre Barrat-Charlaix, and Martin Weigt. "How pairwise coevolutionary models capture the collective residue variability in proteins?." Molecular biology and evolution 35.4 (2018): 1018-1027.

[6] Grathwohl, Will, et al. "Oops i took a gradient: Scalable sampling for discrete distributions." International Conference on Machine Learning. PMLR, 2021.

> The authors should add related work on protein sequence design such as SPDesign, ChromaDesign, CarbonDesign, InstructPLM, etc.

We apologize for not providing a more thorough discussion of these related works in our initial rebuttal due to word limit constraints. Note that we have discussed Potts-based ChromaDesign and CarbonDesign in the previous response.

- SPDesign emphasizes protein-specific physicochemical features. Specifically, it employs ultrafast shape recognition vectors to identify similar protein structures within an in-house database and extracts sequence profiles via structural alignment to enhance sequence prediction. In contrast, our Bridge-IF prioritizes advancements in probabilistic modeling.

- InstructPLM aligns autoregressive protein language models, such as ProGen2, with pre-trained structural features to tackle the inverse folding problem. Conversely, our Bridge-IF utilizes a non-autoregressive iterative refinement approach. We contend that the evolutionary acquisition of proteins implies that each amino acid largely depends on its spatially nearest neighbors; thus, the left-to-right generation method may be sub-optimal for protein generation.

We will incorporate these discussions into our final version.

**We sincerely thank you once again for your help in improving the quality of our paper. We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**
We deeply appreciate the reviewer for the detailed feedback which greatly helps us to improve our work.

> "Note that ChromaDesign is trained on the same training structure data as Chroma, it is not surprising that ChromaDesign achieves excellent performance in predicting sequences for structures generated by Chroma." is not accurate, ChromaDesign is trained separately, and I think the main reason for this is the noise added for the training data, not the adaptive nature of Chroma and ChromaDesign. I have tried ChromaDesign which is very robust for data generated by different models. The authors could try using one of RFDiffusion, FrameDiff, MultiFlow or Proteus to evaluate ChromaDesign and Bridge-IF.
> 
> Notion: ChromaDesign makes a lot of improvements in de novo protein sequences, I think a fair comparison and illustration of potential enhancements is meaningful to the community, beyond ProteinMPNN is good enough, Bridge-IF doesn't have to achieve SOTA on all datasets.

We appreciate the opportunity to further dive into the application of the inverse folding model for de novo protein design. Below, we present our findings:

- A trade-off exists between diversity and sample quality. We have tested ChromaDesign at varying temperature values (temperature_S). Note that the performance reported in our previous rebuttal is obtained with a default temperature value of 0.01. We observed that as temperature_S increases, diversity rises, while both scTM and designability decrease. Remarkably, when the diversity of the generated samples is comparable, the sample quality of Bridge-IF (scTM: 0.73, designability: 0.85, diversity: 0.56) surpasses that of ChromaDesign (temperature_S = 0.25).

| temperature_S | scTM | designability (scTM > 0.5) | diversity |
| -| - | - | - |
| 0.01     | 0.86     | 0.98 | 0.34 |
| 0.1      | 0.85     | 0.98 | 0.38 |
| **0.25** | **0.70** | **0.83** | **0.53** |
| 0.5      | 0.64     | 0.74 | 0.72 |
| 1        | 0.42     | 0.18 | 0.84 |

- Inspired by these results, we performed further comparisons between Bridge-IF with a lower sampling temperature and ChromaDesign using backbone structures generated by FrameDiff. As anticipated, we found that Bridge-IF at a lower temperature significantly improved its performance, yielding results comparable to ChromaDesign (temperature_S = 0.01) regarding scTM and designability (scTM > 0.5). Following suggestions from RFdiffusion and FrameDiff, we employed designability (scRMSD < 2Å) as a stricter metric, revealing that Bridge-IF achieves superior performance.

|  | scTM | designability (scTM > 0.5) | designability (scRMSD < 2Å)
| - | - | - | - |
| ChromaDesign  | 0.86  | 0.99  | 0.41 |
| Bridge-IF     | 0.86     | 0.99  | **0.44** |

- We agree that the addition of noise to the training data is crucial for robust de novo protein design. For instance, ProteinMPNN systematically examines the impact of noise levels on sequence recovery and robust design. ChromaDesign utilizes diffusion augmentation, where the sequence prediction is based on a noisy structure and a specific time t, Given that ChromaDesign is conditioned on the noise level, it may provide enhanced flexibility in selecting the optimal noise level for particular design tasks. Due to time constraints during the rebuttal process, we could not conduct exhaustive attempts but achieved promising preliminary results. We will undertake further investigations in de novo protein design in future work.

> I observed that the performance of Bridge-IF on long sequences is not as good as ChromaDesign, how to explain this phenomenon? Does it stem from a large crop size in ChromaDesign?

Thanks for the careful review. While we also suspect that the large crop size in ChromaDesign may be a potential reason, we also conducted additional analyses. As noted in our previous response, a trade-off exists between diversity and sample quality. We argue that the suboptimal performance of our Bridge-IF on long sequences primarily results from insufficient control of the sampling temperature. This implementation leads to the generation of more diverse samples, which compromises performance on the more challenging long sequences. We found that ChromaDesign (temperature_S = 0.25), which produces samples demonstrating diversity comparable to that generated by Bridge-IF, also experiences a significant decline in performance (the highest TM-score) on length sequences, thereby validating our explanation.

|  Length | 200-300 | 300-400 | 400-500 |
| - | - | - | - |
| ChromaDesign (temperature_S = 0.01) | 0.88 | 0.87 | 0.81 |
| ChromaDesign (temperature_S = 0.25) | 0.69 | 0.62 | 0.46 |
| Bridge-IF | 0.76 | 0.68 | 0.53 |
I'm glad to see that the author's response has addressed most of my questions and I changed the score.

Bridge-IF surpasses ProteinMPNN in de novo protein sequence design, suggesting Bridge-IF has more potential for future exploration.

However, I still have some questions, and if the authors can continue to address them, I would be willing to continue to raise my score.
1. "Note that ChromaDesign is trained on the same training structure data as Chroma, it is not surprising that ChromaDesign achieves excellent performance in predicting sequences for structures generated by Chroma." is not accurate, ChromaDesign is trained separately, and I think the main reason for this is the noise added for the training data, not the adaptive nature of Chroma and ChromaDesign. I have tried ChromaDesign which is very robust for data generated by different models.
The authors could try using one of RFDiffusion (https://github.com/RosettaCommons/RFdiffusion/tree/main), FrameDiff (https://github.com/Immortals-33/Framediff/ tree/master), MultiFlow (https://github.com/jasonkyuyim/multiflow) or Proteus (https://github.com/Wangchentong/Proteus) to evaluate ChromaDesign and Bridge-IF.

Notion: ChromaDesign makes a lot of improvements in de novo protein sequences, I think a fair comparison and illustration of potential enhancements is meaningful to the community, beyond ProteinMPNN is good enough, Bridge-IF doesn't have to achieve SOTA on all datasets.

2.The authors state that the Potts-model based approach has no error accumulation problems. Then the Potts-model approach is still a potential baseline as CarbonDesign (default) (https://github.com/zhanghaicang/carbonmatrix_public), and I would like the authors to add a relevant comparison in the CATH dataset. I would like to see more discussion from the author about the Potts model.

3.The authors should add related work on protein sequence design such as SPDesign, ChromaDesign, CarbonDesign, InstructPLM, etc. 

4.I observed that the performance of Bridge-IF on long sequences is not as good as ChromaDesign, how to explain this phenomenon? Does it stem from a large crop size in ChromaDesign?