Title: Protein Inverse Folding From Structure Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: Protein design

DPO

Protein Language Model
Abstract: The inverse folding problem, aiming to design amino acid sequences that fold into desired three-dimensional structures, is pivotal for various biotechnological applications. 
Here, we introduce a novel approach leveraging Direct Preference Optimization (DPO) to fine-tune an inverse folding model using feedback from a protein folding model.
Given a target protein structure, we begin by sampling candidate sequences from the inverse‐folding model, then predict the three‐dimensional structure of each sequence with the folding model to generate pairwise structural‐preference labels. 
These labels are used to fine‐tune the inverse‐folding model under the DPO objective.  
Our results on the CATH 4.2 test set demonstrate that DPO fine-tuning not only improves sequence recovery of baseline models but also leads to a significant improvement in average TM-Score from 0.77 to 0.81, indicating enhanced structure similarity. 
Furthermore, iterative application of our DPO-based method on challenging protein structures yields substantial gains, with an average TM-Score increase of 79.5\% with regard to the baseline model.
This work establishes a promising direction for enhancing protein sequence design ability from structure feedback by effectively utilizing preference optimization.

Protein Inverse Folding From Structure Feedback

Abstract

The inverse folding problem, aiming to design amino acid sequences that fold into desired three-dimensional structures, is pivotal for various biotechnological applications. Here, we introduce a novel approach leveraging Direct Preference Optimization (DPO) to fine-tune an inverse folding model using feedback from a protein folding model. Given a target protein structure, we begin by sampling candidate sequences from the inverse-folding model, then predict the three-dimensional structure of each sequence with the folding model to generate pairwise structuralpreference labels. These labels are used to fine-tune the inverse-folding model under the DPO objective. Our results on the CATH 4.2 test set demonstrate that DPO fine-tuning not only improves sequence recovery of baseline models but also leads to a significant improvement in average TM-Score from 0.77 to 0.81, indicating enhanced structure similarity. Furthermore, iterative application of our DPO-based method on challenging protein structures yields substantial gains, with an average TM-Score increase of 79.5% with regard to the baseline model. This work establishes a promising direction for enhancing protein sequence design ability from structure feedback by effectively utilizing preference optimization † .

1 Introduction

Just as language serves as the foundation of human communication and social organization, proteins constitute the fundamental molecular machinery governing life's essential processes across all biological systems. The emerging task of protein sequence design (inverse folding) aims to find amino acid sequences that reliably fold into target three-dimensional architectures while exhibiting predetermined functional capabilities. This paradigm has catalyzed transformative applications spanning next-generation therapeutics development [31] to the creation of engineered biocatalysts revolutionizing industrial processes [43].

Recent advances in protein inverse folding have shifted from physical methods to deep-learning-based methods [3]. Traditional physics-based methods, such as those implemented in Rosetta [41], rely on energy-based modeling to identify sequences compatible with a given structure. In contrast, deep learning-based models have demonstrated significant progress by leveraging geometric structure encoders and massive sequence datasets. Methods like ProteinMPNN [10], ESM-IF [17], and LigandMPNN [11] employ SE(3)-equivariant networks [9] to capture the spatial properties of protein backbones, enabling accurate and efficient sequence design. More recently, the availability of largescale protein sequence databases has spurred the development of protein language models such as ESM-3 [16], ProGen2 [30], which learn rich sequence representations and show promise in generating structurally compatible and functionally diverse protein sequences [36, 39].

Model

P

S

V

K

I

G I

I

M S V K G G Q

I

Input Structure

Designed Sequences

Inverse Folding Process

Folding Process

Predicted Structures

…

TM-Align

P

S

K

V

I

G I

I

Chosen

M S V K G G Q

I

Rejected

DPO

Optimization

Sort and split by structure similarity

P

S

V

K

I

G I

I

M S V K G G Q

I

…

0.93

0.71

N sequences

N // 2

N // 2

TM-Score

Construct Training Pairs

Iterative Design

Inverse Folding

Model

Generate

Optimize

Multi-Round DPO

A

B

C

…

𝑇 ∈ 𝒯

𝑓: 𝒮 → 𝒯

𝑔: 𝒯 → 𝒮

𝑆 ∈ 𝒮

Current inverse folding models are typically formalized as generative models, which are trained to maximize the conditional probability of protein sequences given their corresponding structures. The underlying assumption is that proteins with similar sequences will also have similar structures. This allows for the design of new proteins that fold into desired structures by sampling from the learned generative model. However, previous studies [46, 22] have identified proteins with similar sequences but significant differences in structure and function. This suggests that sequence similarity (which is the learning objective of many inverse folding models) is not always a reliable indicator of structural similarity. Moreover, generative models for sequences - whether for protein sequences or text - may experience hallucinations and produce low-quality or repetitive responses. They can also sometimes fail to follow instructions accurately [27, 5, 51, 49]. These challenges underscore the need for further improvements to enhance the reliability and accuracy of these models. Due to these challenges, practical protein design workflows [38, 40] often involve generating a large number of sequences, followed by a sophisticated selection process to prioritize the most promising candidates. Only then can costly experimental validation be conducted. While necessary, this approach introduces additional steps and complexity into the protein design process, potentially limiting the broader application of inverse folding algorithms.

Fortunately, recent advances in preference-based learning, represented by Direct Preference Optimization (DPO) [37], offer an opportunity to mitigate those problems. These methods train models on a pre-generated dataset, capture the preference information by constructing chosen and rejected training pairs, and teach the model to classify good or bad responses. Some efforts have been made to apply DPO on protein language models [28, 44] and even inverse folding models [47, 35]. However, these existing efforts primarily rely on datasets derived from wet-lab experiments, where the function or structure preferences are manually curated through costly biochemical assays. As a result, the application of DPO and other preference-based training strategies has been restricted to small-scale datasets, limiting the scope and generalizability of the resulting models. In contrast, recent breakthroughs in protein folding methods, such as AlphaFold [21], ESMFold [25], and RoseTTAFold [23] have enabled fast, accurate, and in silico assessment of sequence-structure compatibility at scale.

Motivated by this, we explore a fully in silico training scheme that uses folding model feedback to enhance the sequence design capabilities of inverse folding models. Specifically, we utilize an inverse folding model to generate a diverse set of protein sequences. These sequences are then fed

into a folding model to predict their corresponding structures. We evaluate these structures w.r.t the ground-truth structure using TM-Align [52] and classify the sequences into 'chosen' and 'rejected' based on the TM-Score. Finally, we employ Direct Preference Optimization (DPO) to fine-tune the inverse folding model. Building on this, we extended our investigation to explore the effects of multi-round DPO. Through extensive experiments, we demonstrate the effectiveness of using folding model feedback within a DPO framework to improve inverse protein folding. Our DPO models consistently achieve higher sequence recovery across all 3 datasets (CATH 4.2 [32], TS50 and TS500 [12]) compared to their baselines. Notably, a detailed study on the CATH 4.2 dataset shows a steady improvement in structure similarity, suggesting that our method learns to prioritize backbone and fold-related features. Furthermore, scaling experiments reveal that the quality of contrastive samples is more crucial than sheer quantity for achieving high structural fidelity. Finally, experiments of multi-round DPO showcase substantial gains in TM-Score on challenging protein structures, evidencing the iterative refinement ability of our methods.

2 Preliminaries

Protein Folding and Protein Inverse Folding. The protein folding problem seeks to predict a protein's 3D structure given its sequence. The goal is to learn a mapping f : S → T , where S and T are the spaces of protein sequences and structures, respectively. Inverse folding (aka 'protein sequence design') predicts a protein's sequence given its structure, which learns an inverse mapping g : T → S from structure space T to sequence space S . In particular, denote D ⊂ T × S to be the dataset of known structure-sequence pairs (e.g., from PDB or CATH). The inverse folding model (parameterized by θ ) is trained to maximize the conditional likelihood as follows:

where T ∈ T denote the protein structure, and S = { s 1 , s 2 , . . . , s L } denote the sequence of length L . A common choice for factorizing the sequence probability is to adopt an autoregressive approach, leading to what is often referred to as the language model loss:

which encourages the model to recover ground truth residues, conditioned on both the target structure and previously predicted residues.

Reinforcement Learning from Human Feedback. Let π θ be the language model parameterized by θ . The goal of Reinforcement Learning from Human Feedback (RLHF) [8, 33] is to fine-tune large language models (LLMs) to better align with human preferences. Typically, given the prompt x , the language model π θ ( y | x ) generates a response y , and human feedback in the form of pairwise preferences between completions is used to train a reward function r ϕ ( x, y ) . This reward model helps the LLM prioritize preferred completions y w over less preferred ones y l , with the probability of preference modeled by:

where σ is the sigmoid function defined as σ ( z ) = 1 1+ e -z . The objective is to maximize this probability, and the reward function r ϕ is learned by minimizing the negative log-likelihood over the preference dataset D pair :

To prevent the fine-tuned model π θ from deviating too much from the initial supervised model π ref , a KL-divergence constraint is added, preserving linguistic quality while aligning with human preferences. However, RLHF can be complex and unstable. A more recent approach, Direct Preference Optimization (DPO) [37], simplifies the process by directly optimizing for human preferences without needing a reward model or reinforcement learning [42], offering a more streamlined alternative. The DPO loss directly minimizes the negative log-likelihood of the preference model:

Here, β is a parameter that determines the level of information retention from the reference model, helping to balance novelty and adherence to the reference behavior. This setup allows for a more straightforward optimization process that is inherently more stable.

3 Methods

Constructing Preferences Datasets. Given a structure T drawn from dataset D , we first apply the inverse-folding model π θ to sample N initial predictions,

Next, structures corresponding to the sampled sequence S n are predicted using a folding model, T n = g ( S n ) , where g is a protein folding model (e.g., ESMfold). The TM-Score c n ∈ [0 , 1] was then calculated with TMAlign [52] for each predicted structure in relation to its corresponding ground-truth structure c n = TM-Score ( T n , T ) . To create balanced training pairs, the top-ranked 50% of sequences are labeled as chosen, while the bottom 50% are labeled as rejected (Fig. 1). Formally, for N generated sequences with TM-Scores { ( S n , c n ) } N n =1 , we sort the sequences by the TM-Score in a descending order and split into chosen S w and rejected S l as follows:

Then for j = 1 , . . . , N 2 we have

Finally, our new preference dataset can be written as D pair = { ( T, S w j , S l j ) } N/ 2 j =1 . This equal split ensures a balanced dataset for training.

Finetuning with DPO and SFT Loss. Unlike traditional NLP datasets, designed protein sequences have a much smaller vocabulary and can share similar residues even when the TM-Score is low. As the original DPO loss will minimize the probability of rejected responses, the probability of similar parts will also be minimized, eventually causing model degeneration (similar to the alignment of mathematical problems in NLP tasks, where the responses are similar despite the answer being wrong [34]). A simple method to tackle this problem is to add an SFT loss as a regularization term. Given the constructed preference dataset D pair , the DPO loss on the inverse folding model π θ is:

The final loss can be written as:

where the λ is a super-parameter to control the contribution of rejected responses during training.

Iterative Training. To investigate whether our methods can consistently improve the performance in a multi-round setting, we also employ an interactive training framework where the inverse folding models are successively refined using preference data generated by their predecessors. For each iteration t , the model π t θ will first generate and construct preference data D t pair as described in Sec. 3, the model is then updated to π t +1 θ with the DPO and SFT loss (Eq. 10) on generated dataset D t pair . Crucially, the reference model π ref is reinitialized with the weight of π t θ at the start of each iteration, ensuring consistency in preference alignment while avoiding catastrophic forgetting.

4 Experiments

4.1 Benchmark Performance

To assess the broad applicability of our method, we apply it to two representative protein design models, ProteinMPNN [10] and InstructPLM [36], containing around 1 million and 7 billion parameters. This setting helps verify the performance of our method across both large and compact model architectures. Specifically, we first generate sequences and train models on the CATH 4.2 training set and investigate the performance on the test set. To further probe generalization, we also evaluate on two additional benchmarks, TS50 and TS500 [12], which comprise 50 and 470 diverse proteins, and are often employed as additional benchmarks to further test generalization capability[53, 14, 15]

beyond the CATH dataset. Details of hyperparameters are shown in Tab. 7. An interesting observation is that ProteinMPNN fails to converge on self-generated datasets (Sec. 3). We infer that the learning capacity of ProteinMPNN is too small to learn from its own predictions. To tackle this problem, we construct a simpler task, rather than fine-tune ProteinMPNN only on model predictions, we also add wild-type sequences as additional chosen samples, details are provided in the Appendix A.4.

As Tab. 1 shows, both DPO models yield consistent gains over their respective baselines. Specifically, InstructPLM-DPO raises recovery by 3.0% (from 53.58% to 55.21% ), and similar gains are also achieved in ProteinMPNN-DPO (from 40.27% to 41.29%). In terms of perplexity, it is no surprise that InstructPLM-DPO has a slight increase, as the model is trained exclusively on self-generated sequences, which inevitably cause a distribution shift from the reference model. However, despite the distribution shift, the perplexity remains within competitive ranges. On the other hand, as ProteinMPNN has an extra supervision of wild-type sequences, the perplexity of ProteinMPNN-DPO successfully decreased. On TS50, InstructPLM-DPO increases recovery from 60.20 % to 62.01 %, and on TS500 from 66.13 % to 66.37 %, while perplexity rises only 0.06 and 0.03, respectively. ProteinMPNN-DPO also exhibits similar performance improvement, recovery rate increases from 44.72 to 45.91 on TS50, and even more steady increases from 46.58 to 48.23 on the TS500 dataset. This confirms that preference optimization enhances design quality across both large and compact model architectures.

4.2 DPO Achieves Higher Structure Similarity

To evaluate whether our methods acquire transferable, structure-related features solely from selfgenerated training sequences, we investigate the structure similarity measured by TM-Score of InstructPLM-DPO on the CATH 4.2 test set, all structures are predicted using ESMfold. As Tab. 2 shows, despite never seeing native test-set examples during training, InstructPLM-DPO yields consistently higher TM-scores across all splits of the CATH 4.2 test set. Specifically, the overall TM-Score reaches 0.81 for our best model, which witnesses a steady performance gain compared with the baseline model (0.77). Moreover, for different splits of the test set, our methods improve the baseline from 0.54 to 0.58 on the Short split, which contains small proteins less than 100 amino acids, indicating a performance improvement of both short and long proteins. For Single-chain, the model achieves an improvement of 12 . 5% compared to the baseline model (from 0.56 to 0.63). Notably, all of the performance gains are acquired from the training set, the only supervision comes from the

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

-0.2

-0.1

0

0.1

0.2

Δ

mean Recovery

Δ

mean TM-Score

4a5p.A

1kvz.A

2ln7.A

3i5d.A

y = 0.79x+0.02

R

2

= 0.19

4a5p.A

TM-Score: 0.94

Recovery: 0.49

TM-Score: 0.37

Recovery: 0.38

1kvz.A

TM-Score: 0.87

Recovery: 0.35

TM-Score: 0.51

Recovery: 0.36

TM-Score: 0.90

Recovery: 0.54

TM-Score: 0.55

Recovery: 0.57

3i5d.A

TM-Score: 0.77

Recovery: 0.29

2ln7.A

TM-Score: 0.36

Recovery: 0.30

TM-Score

TM-Score

Recovery

Recovery

4a5p.A

1kvz.A

3i5d.A

2ln7.A

0

0.2

0.4

0.6

0.8

1

0.4 0.45 0.5 0.55

0

0.2

0.4

0.6

0.8

1

0.25

0.3

0.35

0.4

0

0.2

0.4

0.6

0.8

1

0.4 0.45 0.5 0.55

0

0.2

0.4

0.6

0.8

1

0.25

0.3

0.35

0.4

folding model. These improvements demonstrate the effectiveness and generalization ability of our methods.

To better understand the role of chosen and rejected samples, we conducted an ablation study on different weights of λ , where SFT indicates that only using chosen examples to train models (Eq. 10). We also divide the test set into a low-similarity group and a high-similarity group based on the baseline performance. As Tab. 2 shows, simply training models on the chosen sample will also benefit the overall structure similarity (from 0.77 to 0.78), but is still left behind by jointly using the chosen and rejected samples simultaneously. Introducing the rejected sample in optimization leads to a significant performance gain, even under a chosen example dominant setting ( λ = 10 ). Specifically, the rejected sample is more useful in low-similarity samples (from 0.37 to 0.45), showing that rejected samples help model lowering the probability of undesired generations. Increasing the weight of rejected samples will further improve the performance ( λ = 1 vs. λ = 10 ). However, the performance starts to drop when the rejected samples outweigh ( λ = 0 ) too much. These results show the importance of balancing the contribution of chosen and rejected samples during training.

Recovery Rate

TM-Score

RMSD

Steps

Steps

Steps

Recovery Rate

TM-Score

RMSD

Samples

Samples

Samples

56

82

2.2

55.5

81

2.15

55

80

2.1

54.5

79

2.05

54

78

2

53.5

77

1.95

53

76

1.9

0

10002000

4000

0

10002000

4000

0

10002000

4000

56

82

2.2

55.5

81

2.15

55

80

2.1

54.5

79

2.05

54

78

2

53.5

77

1.95

53

76

1.9

0

2040

80

0

2040

80

0

2040

80

4.3 A Closer Look at Proteins Generated by DPO Model

To evaluate the impact of our methods on the DPO model, we analyzed the relationship between the changes in structure similarity of sequences designed by InstructPLM-DPO, as measured by the TMScore ( ∆ mean TM-Score), and the corresponding changes in sequence recovery ( ∆ mean Recovery). These changes were calculated by taking the mean of 10 independently generated sequence samples for each protein structure in the CATH 4.2 test set. In general, a linear regression analysis reveals a positive correlation ( y = 0 . 79 x +0 . 02 ), suggesting that improvements in sequence recovery tend to be associated with enhanced structure similarity, which aligns with the intuitive expectation that better sequence recovery often leads to more accurate folds (top left in Fig. 2). However, the low coefficient of determination ( R 2 = 0 . 19 ) indicates a weak association. This implies that substantial increases in TM-Score can frequently occur with relatively small changes in sequence recovery. For instance, many exhibit large TM-score boosts ( ∆ TM-Score > 0.20) with small sequence recovery change ( ∆ Recovery < 0.10). This is reasonable as no wild-type sequences are involved during the training phase of InstructPLM-DPO. These observations suggest that our fine-tuning process prioritizes the exploration of sequences that fold into more accurate structures, even if deviating from the wild-type sequence.

To further illustrate this observation, we visualize 4 cases with significant structure similarity improvements after fine-tuning, protein 4a5p.A, 1kvz.A, 2ln7.A, and 3i5d.A.(Fig. 2, top right and bottom.) and provide sequence alignment information for these four cases in the Appendix B.4. The top right panel of Fig. 2 shows TM-Scores and recovery rates of all 10 predictions of the baseline and DPO models of 4 cases. The 2 cases in the top row (4a5p.A and 1kvz.A) show a clear trend of more robust predictions, that InstructPLM-DPO can eliminate the low-quality predictions and strengthen high-quality predictions, which aligns with our training target. Notably, the 2 cases of the lower row (3i5d.A and 2ln7.A) evidenced the behavior that the DPO model is capable of generating more structurally accurate sequences at the same level of sequence identity. For example, the prediction of protein 3i5d.A of InstructPLM-DPO has a TM-Score of 0.90 at a recovery rate level of 0.56, while the TM-Score of the baseline model at the same level of recovery rate (0.57) only has a TM-Score of 0.55. Predictions of protein 2ln7.A also supports this trend, the fine-tuned prediction achieves a TM-Score of 0.77 with a recovery rate of 0.29, showing notable increases in TM-Score at the same recovery level.

Baseline

Fine-tuned

Low TM-Score

High TM-Score

1.0

0.8

8

o

0.6

工

早

o

8

0.4

o

0.2

0.0

1mkn.A

1cyu.A

1fuw.A

2kxg.A

1u96.A

4qiw.K

1h31.B

3jcu.Z

2caz.C

1grx.A

4.4 DPO at Scale

Scaling has been shown to improve performance in large language models and, more recently, in protein language models [2, 39, 6]. Post-training scaling techniques also yield gains in general LLMs [24], but their impact on protein design remains unclear. Here, we systematically explore two axes of scale in our framework: the number of training steps and the number of contrastive samples per structure. To disentangle these factors, we ran two sets of experiments on the CATH 4.2 training split: Training steps, We fixed the number of generated sequences at 20 (10 chosen vs. 10 rejected) and evaluated performance at 0, 1 000, 2 000, and 4 000 update steps. Contrastive samples, We fixed the training steps at 4 000 and varied the number of generated sequences (contrastive pairs) from 0 up to 80 per structure. The baseline performance corresponds to step 0 and zero contrastive samples in each scenario.

As illustrated in Fig. 3, increasing the number of training steps yields consistent improvements in both sequence recovery and structural metrics (TM-score and RMSD), showing that the training process keeps improving the performance of the model. However, improvements saturate at the training step of 4000, indicating a performance bound that we can reach in this setting. In contrast, scaling up the number of contrastive samples boosts recovery rate but degrades structural fidelity-TM-scores decline and RMSD increases. A possible explanation is that larger contrastive sets dilute the useful training signals by introducing too many low-quality samples, for example, rejected samples with high TM-Scores, thus hindering the learning process eventually. These experiments highlight that data quality outweighs sheer quantity. Our experiments shed light that future work should therefore focus on enhanced sample selection or multi-objective scoring strategies to maintain high-quality contrastive examples while preserving fold accuracy.

4.5 Iterative Protein Design

To evaluate the capacity for iterative improvement, we selected 10 difficult structures from the CATH 4.2 test set to assess the performance limits of our approach. First, we divided all structures into "Low TM-Score" and "High TM-Score" groups based on the baseline performance. Within each group, 5 structures with the lowest TM-Scores were chosen. Implementation details, including the number of rounds, generations, training steps per round, sampling temperatures, and top-p values, are detailed in Tab. 7.

The final results, presented in Fig. 4, demonstrate a significant performance gain across all 10 structures. The mean TM-Score improved by 79.5%, rising from 0.39 to 0.70. The High TM-Score group showed the most substantial improvement. Since the design model was already capable of generating high-quality sequences for these structures, DPO effectively reduced the occurrence of lowquality predictions. For instance, in designing proteins 4qiw.K and 1h3l.B, the baseline model could produce sequences with TM-Scores exceeding 0.8. However, the overall performance was diminished by a high proportion of low-quality predictions (over half with TM-Scores below 0.5). After finetuning, the median TM-Score for 4 out of 5 structures in this group surpassed 0.8. Importantly, the upper bound of design performance also increased, indicating that DPO not only minimizes

low-quality predictions but also more accurately aligns the space of high-quality predictions, enabling the exploration of previously unseen, superior predictions. Structures in the Low TM-Score group also exhibited substantial gains. Specifically, proteins 1cyu.A, 1fuw.A, 2kxg.A, and 1u96.A now generate sequences with TM-Scores above 0.5, compared to a baseline TM-Score of approximately 0.3. However, the DPO model still faces challenges in designing protein 1mkn.A, suggesting that a stronger baseline model may be necessary for such difficult cases. Finally, considering the limitation of computational resources, we do not perform a larger scale of experiments in multi-round DPO. Given the notable performance of iterative training, we believe this is a promising topic and leave it for future work.

5 Related Works

Preference Optimization from AI Feedback. In response to the significant costs and limited accessibility associated with curating high-quality datasets, AI feedback has emerged as a critical component in preference fine-tuning. This approach leverages another LLM to predict the reward of a target model's response and subsequently fine-tunes the target model based on this AI-generated feedback [4, 24]. More recently, methods such as self-play [7, 48] and self-rewarding [50] have moved towards eliminating the reliance on external LLMs for feedback. For instance, SPIN [7] employs an iterative framework akin to DPO, utilizing model-generated data as rejected responses paired with human labels as the preferred responses, and iteratively refines the model. Self-rewarding [50] further abstracts away the need for chosen human labels by using the target model itself to determine preferred and dispreferred samples.

Preference Optimization in Protein Language Models. In contrast to the extensive exploration of preference optimization in natural language processing, protein language models (pLMs) have received comparatively less attention in this regard, primarily due to the scarcity of high-quality, labeled datasets. Nevertheless, initial efforts are being made in this direction. Widatalla et al. [47] proposed ProteinDPO to fine-tune ESM-IF for designing proteins with enhanced stability, while Mistani et al. [28] applied DPO on ProtGPT2 for binder design. Distinct from these prior studies that heavily depend on manually annotated datasets, our approach leverages synthetic data generated by the target model itself. Concurrently, Stocco et al. [44] introduced DPO_pLM to align pLMs on various desired properties, including structure similarity. However, their application of DPO focused on ZymCTRL [29], a model specifically designed for functional enzyme design, whereas our work explores a broader spectrum of protein design tasks. Similarly, Park et al. [35] explored the application of DPO on ProteinMPNN for peptide design, which operates at a smaller scale in terms of both data and model size compared to our methods.

6 Conclusion

Summary. We introduce a novel framework for optimizing inverse-folding models by the feedback of the folding model through preference optimization. Our primary finding is the significant improvement in both the sequence recovery and the structure similarity of designed protein sequences, as evidenced by extensive experiments on the CATH 4.2 dataset compared to the baseline model. Moreover, the case study shows that our methods not only preserve high-quality predictions of the baseline model while eliminating the low-quality predictions, but also generate more structurally accurate samples even at the same level as the baseline model. This indicates that the model learns to prioritize the underlying principles of protein folding.

Social impact. Our end-to-end, fully in silico optimization pipeline for protein inverse folding offers significant benefits across healthcare, industry, and academic research. Notably, our work validates the effectiveness of the optimization-from-feedback paradigm, suggesting that the pipeline is flexible and can readily incorporate alternative forms of feedback beyond structural metrics, thereby enabling the development of more powerful and versatile protein design models.

Limitations. While increasing the number of training steps generally leads to improved performance, simply scaling the number of contrastive samples per structure can be detrimental to structure similarity, even if it slightly boosts sequence recovery. This finding underscores a limitation of our current approach. More sophisticated preference dataset construction strategies may further improve the performance.

Acknowledgments and Disclosure of Funding

The work described in this paper was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N; and by the Hong Kong Innovation and Technology Fund, under Project ITS/241/21. Jiezhong Qiu was supported by NSFC 62306290.

References

Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature , 630(8016):493-500, 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Katherine I Albanese, Sophie Barbe, Shunsuke Tagami, Derek N Woolfson, and Thomas Schiex. Computational protein design. Nature Reviews Methods Primers , 5(1):13, 2025.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 , 2024.

Aadyot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel C Curran, Alexander M Hoffnagle, Kyle Ching, Michael Martyn, Stephen Nayfach, Jeffrey A Ruffolo, and Ali Madani. Scaling unlocks broader generation and deeper functional understanding of proteins. bioRxiv , pages 2025-04, 2025.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 , 2024.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science , 378(6615):49-56, 2022.

Justas Dauparas, Gyu Rie Lee, Robert Pecoraro, Linna An, Ivan Anishchenko, Cameron Glasscock, and David Baker. Atomic context-conditioned protein sequence design using ligandmpnn. Nature Methods , pages 1-7, 2025.

Nuno A Fonseca, Rui Camacho, and AL Magalhaes. Amino acid pairing at the n-and ctermini of helical segments in proteins. PROTEINS: Structure, Function, and Bioinformatics , 70(1):188-196, 2008.

Zhangyang Gao, Cheng Tan, Xingran Chen, Yijie Zhang, Jun Xia, Siyuan Li, and Stan Z. Li. KW-design: Pushing the limit of protein design via knowledge refinement. In The Twelfth International Conference on Learning Representations , 2024.

Zhangyang Gao, Cheng Tan, and Stan Z Li. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079 , 2022.

Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding, 2022.

Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. Science , page eads0018, 2025.

Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. International Conference on Machine Learning , pages 8946-8970, 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems , 32, 2019.

Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411 , 2020.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature , 596(7873):583-589, 2021.

Mickey Kosloff and Rachel Kolodny. Sequence-similar, structure-dissimilar protein pairs in the pdb. Proteins: Structure, Function, and Bioinformatics , 71(2):891-902, 2008.

Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S Morey-Burrows, Ivan Anishchenko, Ian R Humphreys, et al. Generalized biomolecular modeling and design with rosettafold all-atom. Science , 384(6693):eadl2528, 2024.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 , 2023.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science , 379(6637):1123-1130, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 , 2020.

Pouria Mistani and Venkatesh Mysore. Preference optimization of protein language models as a multi-objective binder design paradigm. arXiv preprint arXiv:2403.04187 , 2024.

Geraldene Munsamy, Ramiro Illanes-Vicioso, Silvia Funcillo, Ioanna T Nakou, Sebastian Lindner, Gavin Ayres, Lesley S Sheehan, Steven Moss, Ulrich Eckhard, Philipp Lorenz, et al. Conditional language models enable the efficient design of proficient enzymes. bioRxiv , pages 2024-05, 2024.

Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems , 14(11):968-978, 2023.

Pascal Notin, Nathan Rollins, Yarin Gal, Chris Sander, and Debora Marks. Machine learning for functional protein design. Nature biotechnology , 42(2):216-228, 2024.

Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton. Cath-a hierarchic classification of protein domain structures. Structure , 5(8):1093-1109, 1997.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228 , 2024.

Ryan Park, Darren J Hsu, C Brian Roland, Maria Korshunova, Chen Tessler, Shie Mannor, Olivia Viessmann, and Bruno Trentini. Improving inverse folding for peptide design with diversity-regularized direct preference optimization. arXiv preprint arXiv:2410.19471 , 2024.

Jiezhong Qiu, Junde Xu, Jie Hu, Hanqun Cao, Liya Hou, Zijun Gao, Xinyi Zhou, Anni Li, Xiujuan Li, Bin Cui, Fei Yang, Shuang Peng, Ning Sun, Fangyu Wang, Aimin Pan, Jie Tang, Jieping Ye, Junyang Lin, Jin Tang, Xingxu Huang, Pheng Ann Heng, and Guangyong Chen. Instructplm: Aligning protein language models to follow protein structure instructions. bioRxiv , 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Albert Ros-Lucas, Nieves Martinez-Peinado, Jaume Bastida, Joaquim Gascón, and Julio AlonsoPadilla. The use of alphafold for in silico exploration of drug targets in the parasite trypanosoma cruzi. Frontiers in Cellular and Infection Microbiology , 12:944748, 2022.

Jeffrey A. Ruffolo, Aadyot Bhatnagar, Joel Beazer, Stephen Nayfach, Jordan Russ, Emily Hill, Riffat Hussain, Joseph Gallagher, and Ali Madani. Adapting protein language models for structure-conditioned design. bioRxiv , 2024.

Valeria Scardino, Juan I Di Filippo, and Claudio N Cavasotto. How good are alphafold models for docking-based virtual screening? Iscience , 26(1), 2023.

Christophe Schmitz, Robert Vernon, Gottfried Otting, David Baker, and Thomas Huber. Protein structure determination from pseudocontact shifts using rosetta. Journal of molecular biology , 416(5):668-677, 2012.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Radka Snajdrova, Rebecca Buller, Stefan Lutz, Jeffrey Moore, Romas Kazlauskas, and Uwe Bornscheuer. Surfing the biocatalysis wave to new applications changed to: From nature to industry: Harnessing enzymes for biocatalysis, as per request of editor. Science , 382(6673), 2023.

Filippo Stocco, Maria Artigues-Lleixa, Andrea Hunklinger, Talal Widatalla, Marc Guell, and Noelia Ferruz. Guiding generative protein language models with reinforcement learning. arXiv preprint arXiv:2412.12979 , 2024.

Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Z Li. Global-context aware generative protein design. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023.

Annabel E Todd, Christine A Orengo, and Janet M Thornton. Evolution of function in protein superfamilies, from a structural perspective. Journal of molecular biology , 307(4):1113-1143, 2001.

Talal Widatalla, Rafael Rafailov, and Brian Hie. Aligning protein generative models with experimental fitness via direct preference optimization. bioRxiv , pages 2024-05, 2024.

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675 , 2024.

Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12944-12953, 2024.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 , 2024.

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641 , 2023.

Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research , 33(7):2302-2309, 2005.

Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structureinformed language models are protein designers. bioRxiv , pages 2023-02, 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We show how we use the folding model to optimize the inverse-folding through DPO, improving the structure similarity of designed sequences.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the main limitations in Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We build our methods on open-sourced algorithms and all hyperparameters are provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have released our code at https://github.com/Eikor/iplm-rl Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the training details in Section 4 and Appendix A Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our study used a fixed seed for all experiments, following standards in protein design tasks, and thus did not report statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are done with 8*NVIDIA A100 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of our work in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all datasets and models utilized in our experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our models and documents at https://github.com/Eikor/ iplm-rl .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This research does not involve LLMs as any important, original, or nonstandard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Implementation Details

A.1 Generation Configs for Evaluation

To better evaluate the performance of DPO models, we generate 10 samples at a temperature of 0.15 for each structure in all datasets used during evaluation. We report the mean value of 10 predictions for each metric.

A.2 Low-Rank adaptation

To keep the knowledge in the pre-trained model, we utilized LoRA [18] during the training process of InstructPLM-DPO. The key idea of LoRA (Low-Rank Adaptation) is to enhance the adaptability and efficiency of pre-trained models by integrating trainable low-rank matrices into existing model weights, allowing for targeted modifications without retraining the entire model.For each layer in the pre-trained model, let h n -1 ∈ R l × d be the inputs of n -th layer, l is the sequence length, and d is the hidden dimension. h n qkv be the output of QKV projection module of n th layer, the updated output h n ∗ qkv can be written as

where W A ∈ R r × d and W B ∈ R d × r are LoRA weights, which we initialized randomly from a Gaussian distribution with mean zero and standard deviation of 0.01. We set the rank r as 16 by default, resulting in a fraction of 0.1% of trainable total parameters. The strategy helps us balance model expressiveness with computational efficiency.

A.3 Hyperparameters

We use two different training configs for single-round training and multi-round training. The main difference lies in the generation setting and training steps. Specifically, for the construction of the single-round dataset, we generate 20 responses for each structure, with temperature at 1.0 and top p at 0.9. For multi-round, in order to encourage model exploration, we set the temperature at 1.1 and top p at 1 and generate 200 responses for each structure. For DPO training, we use AdamW [26] as optimizer. We set β to 0.5 for all experiments. We train our model on 8 * Nvidia A100 GPUs with a batch size of 128. Other parameters are summarized in Tab. 7.

A.4 Preference Optimization of ProteinMPNN

To train the ProteinMPNN model, we slightly changed our dataset curation process and constructed a simpler task. Specifically, we set a new threshold t r = 0 . 8 to control the process of classifying samples into chosen and rejected. Instead of training on all predictions like InstructPLM, we only trained ProteinMPNN on the predictions with a TM-Score lower than t r . Meanwhile, the wild-type sequence has also been introduced as a chosen sample during training. As Fig. 5 shows, when trained on the original task ( 3) the model failed to converge, whereas the model successfully trained on a simpler task.

B Additional Results

B.1 Compare with Other Baselines

We provide a more detailed comparison with other baselines in Tab. 6.

B.2 Evaluation using AlphaFold 3

To examine the robustness of our method across different structure predictors, we evaluated designs generated by our baseline and fine-tuned models using AlphaFold 3 [1]. Specifically, for each test structure where multi-round DPO was applied, we folded 10 designed sequences using AlphaFold 3, resulting in 200 sequences in total (100 from the baseline and 100 from the fine-tuned model). We report the average TM-Score and RMSD per structure, along with standard deviations, in the table 3 These results confirm that the improvements achieved by our method are consistent across

different folding models, suggesting that our approach captures structure-relevant features beyond model-specific feedback.

B.3 Performance on CATH 4.3

To further assess the robustness of our method, we additionally evaluate it on selected structures from the newer CATH 4.3 release (Table 4). Specifically, we selected protein domains from newly added Classes in CATH 4.3 that are not present in CATH 4.2, ensuring no overlap with the training data not just at the Fold level, but at the broader Class level (the Fold and Class level of protein structure classification can be find in [32]). After filtering out sequences longer than 512 residues, we obtained a test set comprising 750 diverse and challenging structures. Compared to the original CATH 4.2 test set, which differs from training data at the Topology (Fold) level, this new dataset represents a more stringent test of generalization. The results show that our fine-tuned model consistently outperforms the baseline across all metrics. This performance gap is especially notable given the increased structural diversity and novelty in the CATH 4.3 set.

B.4 Sequence Alignment of Visualized Samples

We performed sequence alignment (Fig. 7 to 10) of visualized samples in Fig. 2, illustrating each residue as a colored box to highlight matches (green), mismatches, and uncertain positions (gray). The results show that our fine-tuning process prioritizes the generation of sequences that fold into more accurate structures, even if it means deviating from the native sequence (low recovery rate).

B.5 Improvements on different fold types.

We conducted a detailed analysis where we grouped the test structures in CATH 4.2 according to their Class annotations (Mainly Alpha, Mainly Beta, Alpha-Beta, and Few Secondary Structures) . The average TM-scores before and after DPO fine-tuning are summarized in Table 5. As shown, DPO fine-tuning brings consistent improvements across all fold types, with especially notable gains in the Mainly Alpha and Mainly Beta categories. This demonstrates that our method not only improves structural quality overall but also robustly across a diverse range of protein topologies.

B.6 More Results on Iterative Design.

For additional clarity, we monitor the running TM-Scores and recoveries of all 10 structures during training in Fig. 6. The plot demonstrates a clear upward trend, with substantial gains in the early rounds, followed by a gradual saturation effect as the model converges towards higher structure similarity. Concurrently, the recovery rate remains at a relatively low level, further emphasizing our method's ability to prioritize structural fidelity over sequence similarity, leading to the discovery of alternative sequences that fold more accurately.

Hard Task

Simple Task

Training Steps

1K

3K

5K

1K

3K

5K

1K

3K

5K

Training Steps

Training Steps

Loss

Reward Accuracy

Margin

0.5

0.55

0.6

0.65

0.7

0.5

0.6

0.7

0.8

0.9

1

0

0.2

0.4

0.6

0.8

TM-Score

Recovery

0.2

0.3

0.4

0.5

0.6

0.7

0

5

10

15

20

Rounds

1-50

M Q D W A

T

F

K

K

K H

L

T

D T W D V D C D N

L

M P

T

S

L

F

D C

K D K N

T

F

I

Y

S

L

P

G P

V

K

A

L

C

R

Wild Type

G L

T

A D D

F

R

K

Y

H

L

A D S

E

T

I

D C D K

L

L

P

T D

L

F

K N

L

D S

V

V

Y

I

V

T

D P

A

P

F

A

A

E

A

K

baseline

G L

T

A

E

D F

R

K

Y

H

L

S

D S

E

T

I

D C D K

L

L

P

T

S

L

F

S N

L

D S

V

V

Y

I

V

T

D P

K

P

F

L

A

E

C

K

fine-tuned

51-100

G V

I

F

S

A D V

L

S

N S

E

F

Y

L

A

E

C N V

K

P

R

K

P

C

K

Y

K

L

K

K

S

S

N R

I

C

I

R

C

E

H E

L

P

V H

F

Wild Type

G V

E

K

E

K

V

I

V

T

S

K

E

Y

K

V

T

S

C

V

K

L

P

N K

P

C

V

F

T

L

K

K

S

E

D T

I

C

V

K

V

S

D N

K

P

V

E

L

G V

S

S

E

K

T

I

T

T

S

T

E

Y

T

V

T

S

C

T

K

L

P

N K

P

C

T

F

T

L

T

H S

T

S

T

I

C

V

S

C

S

N N

K

P

V

Y

L

101-107

A G V G

I

C

P

Wild Type

V G V G

S

C

A

V

G L

N

T

C

V

1kvz.A

1-50

G S

H M S

S

Q T

E

H K

E

G E

K

V

A M L

N

I

P

K

L

K

K

K

F

S

I

Y W G A D D A

T

L

K

K

G V G M F

V

S

D V

Wild Type

M A H H H H H H

E

H E

H G H

E

H A H

L

T

L

P D

L

E

L

T

L

D

L

V

A G A D W E

S

E

G E

G V G A

A

V

S D D

baseline

M A H H H H H H

E

H E

E

G A

S

L

L

D

L

T

L

P

K

T

E

E

T

Y D

L

V

Y

G A D

Y

E

S

E

G S

E

V

G L

A

V

S D D

fine-tuned

51-100

T

T

T

P

S

G G G H

T

V

L

S

G H R D

T

V

F

T D

L

G Q

L

K

E

K

D T

L

V

L

E

Y

D N

K

T

Y

T

Y

E

I

Q K

I

W I

Wild Type

V

A

L

P

E

G D A

A

T

V

L

V

A

R

E

A

E

E

E

D E

D G

L

F

S

A

D T

T

F

V

V

E

Y

D D E

E

Y

E

Y

E

V

I

E

V

F D

V

K

L

P

T

G D A

S

F

V

L

V

S

S

L

L

T

S

R

N E

S

G L

Y

S

S

D S

S

F

E

V

S

Y

N D K

T

Y

T

Y

K

V

T

K

V

F

T

101-147

T

H A D D R

T

V

I

I

K

K

E

E

P

I

L

T

L

T

T

C

Y

P

F

D Y

I

G D A

P

D R

Y

I

I

E

A

K

L

T

G S

Y

S

K

Wild Type

V

L

A D D E

E

E

L

E

E

L

E

E

P

E

L

V

V

I

G G E

P

E

E

E

E

G E

E

P

E

L

V

V

I

E

A

E

E

E

E

E

E

E

E

A D S

D D S

S

V

L

K

K

Y

D S

P

E

L

T

I

V

S

G S

P

S

S

V

K

G L

S

P

E

L

K

V

V

E

S

S

K

T

S

S

S

S G

2ln7.A

1-50

_

_

_

_

_

_

_

_

_

_

R

F

T

Q A

L

V

I

A

Y

V

I

G Y

V

C

V

Y

N K G Y Q D

T

D T

V

L

S

S

V

T

T

K

V

K G

I

A

Wild Type

M K

L

F

S

S

L

Y

D K

T

L

E W S

A

H R H A

V W G C

L

L

V

S

F

S

E

A

V

V

F

P

I

P

P

D V

L

L

I

P

V

V G T

C

baseline

M T

L

F

K

S

A

A

K

T

L

R

S

L

S

A

V

K

A

F

V

F

G N

L

F

V N

L

Q G N R

V

K

D E

V

K

S

E

T

K

T

E

V

K

G T

C

fine-tuned

51-100

L

T

N

T

S

E

L

G E

R

I

W D V

A

D Y

I

I

P

P

Q E

D G S

F

F

V

L

T

N M

I

I

T

T

N Q

T

Q S

K

C

A

E N

P

T

P

Wild Type

R

V

N W W R

L

G L

V

T

F

T

A

S

V

L

G G

P

L

A

Y

F

G S

F W L

L

T

N

L

I

V

P

L

S Q R

L

A

A

S

A

A

L

P

S

P

T

V N D

E

K

K

G K

K

T

Y

N A

K

N D G

I

P

L

K

E

D G D

F

F

I

A

T

N E

I

V

T

T

D Q T

A

S

T

G A

A

L

P

S

P

101-150

A

S

T

C

T

S

H R D C

K

R G F

N D A

R G D G V

R

T

G R

C

V

S

Y

S

A

S

V

K

T

C

E

V

L

S W C

P

L

E

K

I

V D

Wild Type

E

T

L

C

R D D S

D C

V

E

G N D A

R

L

G D G V

C

V G E

C

V

S

Y

S

S

S

E

S

T

C

E

V

V

A W C

P

L

E

K

E

V

L

E

T

L

C

E

S

D S

D C

I

E

G N D

S

I

I

G N G V

C

I

G E

C

V

S

Y

S

K

T W K

T

C Q V

E

S W C

P

R

T

V D V

K

151-200

P

P

N

P

P

L

L

A D A

E N

F

T

V

L

I

K

N N

I

R

Y

P

K

F

N

F

N K

R N

I

L

P

N

I

N S

S

Y

L

T

H C

V

F

S

R

K

Wild Type

P

P

R

P

P

L

L

A

E

A

E N

F

T

V

L

I

K

N N

I

E

F

P

K

F

N

Y

H K

R N

I

L

P

H

I

S

D S

Y

L

K

H C Q

F

S

R D

P

P

S

P

P

D F

S

D A

A

D F

T

V

F

I

K N

L

V

R

F

P

K

F

N V

S

K

R N

I

R

P

N

I

N E

T

Y

L

K

K

C

V

Y

D K

E

201-250

T

D P

D C

P

I

F

R

L

G D

I

V G E

A

E

E

D F Q

I

M A

V

R G G V M G V Q

I

R W D C D

L

D M P

Q S W C

V

P

Wild Type

T

D P

Y

C

P

I

F

R

L

G D

I

V G A

A

K

E N

F

Q E M A

V

E

G G V M G

I

Q

I

K W N C D

L

D M P

A

S

E

C

V

P

K H

P

Y

C

P

I

F

R

I

G D M V G A

A

K

E N

F

L

E M A

V

K

G G S

I

G

I

F

I

N W N C D

L

D

L

P

D S

E

C

K

P

251-300

R

Y

T

F

R

R

L

D N

K

D P

D N N V

A

P

G Y

N

F

R

F

A

K

Y

Y

K

N S D G

T

E

T

R

T

L

I

K G Y G

I

R

F

D V M

Wild Type

K

Y

S

F

R

R

L

D N

K

D P

E

N S

V

S

P

G Y

N

F

R

F

A

K

Y

Y

K

D S

S

G K

E

S

R

T

L

I

K

A

Y G

I

R

F D

I M

K

Y

S

F

R

R

L

D S

K N

P

E

N S

S

A

P

G Y

N

F

R

F

A

K

Y

Y

K

T

S

D G K

E

S

R

T

L

I

K

A

Y G

I

R

F D

I

I

301-350

V

F

G Q A G K

F

N

I

I

P

T

L

L

N

I

G A G

L

A

L

L

G L

V

N V

I

C D W

I

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

Wild Type

V

F

G K

A G K

F D

I

I

P

T

M I

N

I

G S

G L

A

L

L

G V

A

T

V

L

C D V

I

V

L

Y

C M K

K

R

Y

Y

Y

R

E

K

K

Y

V

F

G T

A G K

F

S

I

V

P

T

I

V N

L

G A G

L

A

L

L

G V

A

T

V

L

C D

I

I

V

L

Y

C M K

K

K

Y

Y

Y

R

E

K

K

Y

351-356

_

_

_

_

_

_

Wild Type

K

Y

V

E

D Y

K

Y

V

E

D Y

3i5d.A

1-50

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

_

Wild Type

M P

L

P

L

E

E

I

P

P

L

L

K

E

P

L

I

R

E

A

L

L

E D

L

G R

A G D

I

T

T

D A

I

I

S

S

S

V Q V

E

A

S

V

S

A

R

baseline

M P

L

P

L

E

L

I

P

P

E

L

K

E

P

L

L

L

K

F

K

N E

F

N K

D D E W N D

L

V

K W

I

S

S

K

F

S

I

V

E

S

V

Y

F

E

fine-tuned

51-100

_

_

_

S

S

E

T

V

P

L

I

L

L

F

A

E N

_

_

_

A

N D M E

G L

I

E

R

I

R

S

Q F

F

I

D Y

G V

R

L

P

T

I

L

Y

R

T

Wild Type

Q D

I

V

L

C G

I

P

L

I

L

E

L

A N

L

F

I

D D S

S W E D

L

V

E D

I

S

S

R

F

F

I

D Y

G V

K

L

P

E

I

K

L

S

F

E

I

K

P

L

E

K

I

P

L

I

L

K

L

S N

L

Y

I

D D E

K W N D

L

V

K W

I

S

S

K

F

S

I

D F

G V

K

L

P

E

I

K

L

S

Y

101-150

S

N E

L

K

V D D

I

V

L

L

I

N E

V

R

A D S

F

N

I

Y

F

D K

V

C

I

_

_

_

_

_

_

_

_

_

_

_

_

_

_

V

S

T

S

Y

N

Wild Type

S

D S

L

S

V

E

E

I

V

L

Y

I

K

E

V

L

S

S

S

F

L

I

D F

N S

I

C

I

S

S

S

L

F

Y

L

P

L

S

S

S

L

F

F

S

T

S

F D

S

D S

L

S

I

E

E

I

V

L

Y

I

E

E

V

E

S

S

S

F

E

I

D F

N A

I

C

I

S

E

E

L

V

Y

Y

P

L

E

I

S

E

F

V

S

T

S

F

E

151-200

E

R

V

I

S W V D V

_

_

_

_

_

_

_

_

_

_

_

_

I

K

S

A Q D E

F

Y

H Q

L

S

Q A

L

L

N N

I

N E

I

F G

I

Q E

T

Wild Type

E

S

S

F

S W V

S

S

P

I

S

L

F

L

S

S

V

S

S

S

S

S

S

S

S

S

S

F

Y

S M L

S

S

S

L

L

S

S

I

S

L

I

F

S

I

S

S

T

E

S

T

F

S W

I

S

E

P

I

Y

L

L

L

N N V

K

N S

S

F

S

S

E

E

F

F

Y

K M L

S

S

S

L

L

D N

I

D

L

I

F D

I

E

S

T

201-250

K

N M L

D Q

F

E

N R

Y

P D

L

L

K

E

V

F

R H V

T

I

Q R

I

S

E

V

L

Q R

L

L

G E N

I

S

V

R N

L

K

L

I

M E

S

Wild Type

L

S

L

L

S

S

F

S

L

F

F

P

S

L

L

S

S

V

F

S

S

V

S

I

L

S

I

S

S

V

L

S

L

L

L

S

S

S

I

S

I

S

S

L

K

S

I

L

S

S

L

K

L

L

S

K

F

K

L

N

Y

P

K

L

L

D K

V

Y

K

Y

V

S

I

K

E

I

S

Q V

L

Q M L

L

S

E N

I

S

I

K D

L

K

L

I

M E

S

251-300

L

A

L W A

P

R

E

K

D V

I

T

L

V

E

H V

R

A

S

L

S

R

Y

I

C

S

K

I

A

V

S

G E

I

K

V

V M L

S

G Y

I

E

D A

I

R

Wild Type

L

S

L

F

S

P

S

L

S

S

I

V

I

L

V

S

S

V

L

S

S

L

S

S

S

I

S

S

L

F

S

S

L

G S

I

S

L

V

S

L

S

S

L

I

F

S

S

I

L

L

S

L

Y

S

P

K

E

K D

I

I

I

L

V

E

H V

R M S

L

K

R

Y

I

C

S N

I

S

E

E

G K

L

N V

V M L

S

S

E

I

E

N S

I

R

301-350

K G

I

R Q T

_

_

_

_

_

_

N M D

I

E

V

S

D E

V M E

T

L

A

H A

L

R

E

L

R N A

K

K N

F

V

L

L

V

S

V D

I

R

R

Wild Type

S

S

I

S

L

L

S

L

S

S

S

F

S

S

S

I

S

L

S

S

S

V

L

S

L

L

S

I

S

S

L

L

S

L

L

L

S

S

S

F

S

L

L

S

L

S

S

I

S

S

N G

I

R Q T

S

L

S

S

I

Y

S

S

S

I

S

I

D S

L

I

L

E

K

L

Y

K

S

L

K

S

L

N

L

S

N S N

F

V

L

L

T

S

V D

I

R

R

351-383

F

V

K

R

L

I

D N R

F

K

S

I

L

V

I

S

Y

A

E

I

D E

A

Y

T

I

N V

L

K

T

I

Wild Type

F

V

S

S

L

I

S

S

S

S

S

S

I

S

S

I

S

L

V

S

L

S

S

L

S

S

I

S

I

L

S

L

S

Y

V

R

K

L

I

E

S

K

F

S

S

I

K

V

I

S

Y

Q E

I

S

S

D F

K

I

N V

L

G K

I

4ap5.A

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for optimizing protein inverse folding models using Direct Preference Optimization (DPO) guided by structural feedback from a folding model. Given a target structure, the model samples multiple candidate sequences, predicts their folded structures, and scores them via TM-Align. Based on the TM-Score, sequences are classified into \"chosen\" and \"rejected\" and used to fine-tune the inverse folding model under a DPO objective. Experiments are done on the CATH 4.2, TS50, and TS500 datasets demonstrate improvements in both sequence recovery and TM-Score. The authors further investigate the effect of training scale, the number of contrastive samples, and iterative refinement via multi-round DPO.\n\nSTRENGTHS AND WEAKNESSES:\n*Strength*\n\n-The authors evaluate their approach across multiple datasets (CATH 4.2, TS50, TS500), model architectures (ProteinMPNN, InstructPLM), and settings (single-round vs. multi-round, λ ablation), providing empirical evidence for robustness.\n\n-The pipeline is entirely in silico and can be extended iteratively, making it well-suited for real-world protein design where experimental structures may be limited or unavailable.\n\n-The use of equations, diagrams, and concrete implementation details make the method easy to follow and potentially reproducible.\n\n\n*Weakness*\n\n1. Several claims of \"significant\" improvement (e.g., Lines 251, 295) are made without statistical testing. In Figure 4, no significance tests are reported to validate the robustness of observed gains.\n2. The evaluation of contrastive sample sizes in Figure 3 is based on only four discrete points (0, 20, 40, 80), which obscures potential non-linear effects or turning points in TM-Score and RMSD. This makes it difficult to draw reliable conclusions about how contrastive sample quantity affects structural fidelity.\n3. TM-Score is used as the sole optimization and evaluation metric, yet structural similarity does not always correlate with functional correctness. Without assessing alternative measures (e.g., energy, function, or diversity), there is a risk of optimizing toward biologically irrelevant sequences.\n4. Unaddressed Risks in Multi-Round DPO Training\n The iterative nature of multi-round DPO could amplify early-stage model biases or propagate errors. However, the paper does not discuss any safeguards such as validation-based early stopping, diversity checks, or convergence criteria to mitigate this risk.\n5. While related work sections mention prior DPO-based efforts in enzyme and peptide design, the manuscript lacks a clear articulation of what makes structure-feedback-based protein design novel and technically distinct. The gap between this work and existing methods remains underexplored.\n6. Minor presentation issues:\n   - Table 1 lacks clarity on whether values are percentages.\n   - Formatting choices (bold for fine-tuned models, underline for baselines) deviate from conventions and may confuse readers.\n   - Line 169 uses \"ESMfold\" instead of the correct “ESMFold.”\n\nQUESTIONS:\n1. Figure 4 shows higher median TM-Scores for the fine-tuned model, but the heavy lower-tail suggests potential instability or skewness. Could the authors report statistical significance (e.g., Student's t-test), add distribution visualizations (e.g., violin/KDE plots), and clarify whether improvements are consistent across targets or driven by a few outliers? This would strengthen the claim of robust and meaningful improvement.\n\n2. Why did you choose CATH 4.2 instead of the latest CATH 4.3? Would the model's performance be consistent if evaluated on the newer dataset? Please clarify if CATH 4.3 was excluded for compatibility, coverage, or reproducibility reasons.\n\n3. Can you clarify the novelty of your method over prior DPO work? Are you the first to incorporate structural feedback into DPO for protein design? If not, how does your framework advance beyond existing works in terms of scale, architecture, or applicability?\n\nLIMITATIONS:\nYes. There is a paragraph titled “Limitations”.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use Direct Preference Optimization to fine tune existing models that perform protein design. That is, given a target structure, the idea is to hypothesize sequences that could fold into that structure using a pretrained model (they use proteinMPNN and InstructPLM). These sequences are folded (in silico) and then an alignment score is used to fine-tune the model used to generate hypothetical sequences. The effects of this on recovery rate, perplexity and structural similarity are studied.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength of the paper is the relatively simple application to existing models to achieve significant improvements in structural similarity. Indeed, as highlighted, recovery rates are less important when the task is to design de novo proteins. Yet, it is good to see that the method maintains relatively healthy recovery rates as well as perplexity, while improving structural similarity. However, is unclear what the associated computational cost is. It is suggested to include and discuss the following references:\n\n[1] Improving Protein Sequence Design through Designability Preference Optimization (https://arxiv.org/html/2506.00297v1)\n\n[2] PDBench: evaluating computational methods for protein-sequence design (https://academic.oup.com/bioinformatics/article/39/1/btad027/6986968)\n\n[3] Fine-tuning protein language models boosts predictions across diverse tasks\n(https://www.biorxiv.org/content/10.1101/2023.12.13.571462v1)\n\nQUESTIONS:\n1) How do the improvements look across types of folds (e.g.see R2)?\n\n2) How was the data for Fig 3 generated? Is that a specific protein, or averages? In general, it appears that as RMSD drops the recovery rate increases– that is de novo exploration reduces. \n\n3) What is the scale of computation involved? E.g. per DPO iteration of each model (for different numbers of samples).\n\nLIMITATIONS:\nYes, but the limitations are quite generic.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
This work applies Direct Preference Optimization (DPO) to the inverse folding task, leveraging TM-score as a feedback signal to enhance the foldability of designed protein sequences. The authors conducted comprehensive experiments to validate the effectiveness of the DPO strategy. In addition, they provide insightful analyses of the model, including its scaling behavior, an illustrative case study, and the impact of iterative training.

STRENGTHS AND WEAKNESSES:
The paper is well-written, with a comprehensive introduction, clear preliminaries, and detailed methodology. The authors also conducted in-depth analyses to evaluate and interpret the model. However, there are several minor concerns regarding the experimental section:

1. Lines 149–152 provide only a brief description of the application of DPO to ProteinMPNN. It would be helpful if the authors could elaborate further on this part for better clarity.

2. In line 189, it is unclear whether the authors explored other values for the hyperparameter λ, such as 0.1 or 0.01. Including such experiments would strengthen the analysis.

3. In Section 4.5, the results are reported only on ten test samples. It is recommended that the authors report performance metrics on the entire test set to provide a more comprehensive evaluation.

4. Since the main contribution of the paper is the application of DPO to inverse folding, demonstrating its effectiveness on additional baseline models would further substantiate the generalizability and robustness of the proposed approach.

QUESTIONS:
See the Strengths and Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for improving protein inverse folding by leveraging feedback from a folding model using Direct Preference Optimization (DPO). Given a target structure, the inverse folding model generates candidate sequences, which are scored by a folding model based on structural similarity (TM-score). These scores are used to create preference pairs (chosen vs. rejected), which guide DPO fine-tuning of the inverse folding model. The approach improves both sequence recovery and structural accuracy across datasets (CATH 4.2, TS50, TS500), and shows strong gains in multi-round iterative refinement. The method is fully in silico, scalable, and applicable to both small and large protein models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**Quality:**\n\n* Methodologically solid; applies Direct Preference Optimization (DPO) to inverse folding using feedback from folding models.\n* Demonstrates consistent improvements in structure similarity (TM-score) and sequence recovery across models and datasets.\n* Multi-round DPO refinement shows strong gains, particularly on challenging protein structures.\n* Fully in silico pipeline avoids reliance on experimental data, supporting scalability.\n\n**Clarity:**\n\n* The paper is well organized and the method is clearly described.\n* Figures are helpful, especially the pipeline diagram and TM-score visualizations.\n\n**Significance:**\n\n* Addresses a key gap: current inverse folding models over-optimize for sequence similarity.\n* Shows that structure feedback can improve sequence design quality in practice.\n* Results have practical implications for scalable, structure-aware protein design.\n\n---\n\n**Weaknesses**\n\n**Quality:**\n\n* Table 1 lacks standard deviations; no statistical testing is reported.\n\n  * Authors should use bootstrapped training set sampling and include critical difference diagrams to establish significance.\n* Figure 3 is too coarse (e.g., 0, 20, 40 samples); finer-grained evaluation would clarify where data scaling starts to hurt performance.\n* No learning curve analysis with respect to the number of unique training structures; it's unclear whether performance is limited by data volume or model capacity.\n\n**Clarity:**\n\n* Some figures (e.g., Fig. 3) lack clarity in axis labels and legends.\n* The role and tuning of λ in balancing DPO and SFT losses could be better explained.\n\n**Significance:**\n\n* Generalization to novel folds or unseen structural classes is not evaluated.\n* No integration of functional objectives (e.g., stability, binding) beyond structural similarity.\n\n**Originality:**\n\n* The use of DPO in protein language models is not new.\n\n  * Section 5 notes similar approaches (e.g., ProteinDPO, DPO\\_pLM), though those focus on other objectives (e.g., stability, function) or smaller scales.\n  * This work's main novelty is the specific application of **structure-based feedback** and fully synthetic training, which is incremental rather than fundamentally new.\n\nQUESTIONS:\nQuestions for the authors:\n\n1. Have you analyzed how performance scales with the number of unique training structures (i.e., a full learning curve)?\n2. How do you control for low-quality or ambiguous contrastive samples? Have you tried thresholding TM-Score or using clustering?\n3. Does structural diversity in the training set affect generalization?\n4. Have you tested sensitivity to the choice of folding model for feedback (e.g., ESMFold vs. AlphaFold)?\n\nSuggestions:\n\n1. Run a learning curve experiment varying the number of training structures to assess data efficiency and saturation.\n2. Filter training pairs using absolute TM-Score thresholds instead of a 50/50 split to improve signal quality.\n3. Combine structural feedback with additional objectives (e.g., stability, binding affinity) to enrich preference signals.\n4. Evaluate generalization to unseen folds or novel structural classes to test robustness.\n5. Visualize how DPO shifts sequence distributions (e.g., t-SNE of embeddings) to verify preference-driven learning dynamics.\n6. The results in Figure 3 on scaling contrastive samples are insightful, but the granularity is limited. The jumps from 0 to 20 to 40+ samples are too coarse to reveal meaningful trends or tipping points. A more fine-grained analysis (e.g., increments of 5 or 10) would help clarify how performance evolves and where degradation begins.\n7. Table 1 reports performance metrics without standard deviations, making it unclear whether improvements are statistically significant. To address this, multiple training runs with bootstrapped training set samples should be performed. Statistical significance can then be assessed using critical difference diagrams (e.g., via the `scikit-posthocs` package: [https://scikit-posthocs.readthedocs.io/en/latest/generated/scikit\\_posthocs.critical\\_difference\\_diagram.html](https://scikit-posthocs.readthedocs.io/en/latest/generated/scikit_posthocs.critical_difference_diagram.html)). This would provide a robust comparison across models and datasets.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}