Protein Inverse Folding From Structure Feedback
Authors: Junde Xu
Zijun Gao
Xinyi Zhou
hujie
Xingyi Cheng
Le Song
Guangyong Chen
Pheng-Ann Heng
Jiezhong Qiu
Venue: NeurIPS 2025 poster ()
Keywords: Protein design
DPO
Protein Language Model
Primary area: applications
TL;DR: This paper fine-tunes inverse-folding model with feedbacks from folding model through DPO.

Abstract: The inverse folding problem, aiming to design amino acid sequences that fold into desired three-dimensional structures, is pivotal for various biotechnological applications. 
Here, we introduce a novel approach leveraging Direct Preference Optimization (DPO) to fine-tune an inverse folding model using feedback from a protein folding model.
Given a target protein structure, we begin by sampling candidate sequences from the inverse‐folding model, then predict the three‐dimensional structure of each sequence with the folding model to generate pairwise structural‐preference labels. 
These labels are used to fine‐tune the inverse‐folding model under the DPO objective.  
Our results on the CATH 4.2 test set demonstrate that DPO fine-tuning not only improves sequence recovery of baseline models but also leads to a significant improvement in average TM-Score from 0.77 to 0.81, indicating enhanced structure similarity. 
Furthermore, iterative application of our DPO-based method on challenging protein structures yields substantial gains, with an average TM-Score increase of 79.5\% with regard to the baseline model.
This work establishes a promising direction for enhancing protein sequence design ability from structure feedback by effectively utilizing preference optimization.

Protein Inverse Folding From Structure Feedback
Junde XU
Zijun Gao'
Xinyi Zhou' Jie Hu
Xingyi Cheng 4
Le Song*
Guangyong Chen Pheng-Ann Heng'
Jiezhong Qiu
CUHK "Hangzhou Institute of Medicine, CAS Zhejiang Lab MBBZUAI
qiujiezhongohim.c cas.
Abstract
The inverse folding problem, aiming design amino acid sequences that fold into desired three-dimensional structures. is pivotal for various biotechnological applications. Here, we introduce novel approach leveraging Direct Preference Optimization (DPO) to fine-tune inverse folding model using feedback from a protein folding model. Given a target protein structure, we begin by sampling candi- date sequences from the inverse-folding model, then predict the ditre-dimensional structure each sequence with folding model to generate pairwise structural- preference labels. These labels are used to fine-tune the inverse-foldine model under the DPO objective. Our results on the CATH 4.2 test set demonstrate that DPO fine-tuning not only improves sequence recovery of baseline models but also leads to a significant improvement in average TM-Score from 0.77 to 0.81, indicating enhanced structure similarity. Furthermore, iterative application of our DPO-based method on challenging protein structures yields substantial gains, with average TM-Score increase of 79.5% with regard the baseline model. This work establishes a promising direction for enhancing protein sequence design ability from structure feedback by effectively utilizing preference optimizatior
Introduction
Just language serves as the foundation of human communication and social organization, proteins constitute the fundamental molecular machinery governing life's essential processes across all biological systems. The emerging task protein sequence design (inverse folding) aims to find amino acid sequences that reliably fold into target three-dimensional architectures while exhibiting predetermined functional capabilities. This paradigm catalyzed transformative applications spanning next-generation therapeutics development [3] to the creation of engineered biocatalysts revolutionizing industrial processes
Recent advances in protein inverse folding have shifted from physical methods to deep-lcarning- -base methods 3 Traditional physics-based based methods, such those implemented in Rosetta 44], rely on energy based modeling to identify sequences compatible with given structure. In contrast, deep Bearning-based models have demonstrated significant progress by leveraging geometric structure encoders and massive sequence datasets. Methods like ProteinMPNN
10 ESM-IF [77, and LigandMPNN [II] employ -quuvvaaunaaa networks ( to capture the spatial properties protein backbones, enabling accurate and efficient sequence design. More recently, the availability of large- scale protein sequence databases has spurred the development of protein language models such as ESM-3 [16), ProGen2 [30], which learn rich sequence representations and show promise in generating structurally compatible and functionally diverse protein sequences 33 39
"Corresponding author:
'Code available at htapss///ggihhbb..oo//kkoo///pp
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inverse Folding Process
Folding Process
9:
f:5
Inverse Folding
Folding Model
Model
M G
Input Structure
Designed Sequences 5
Predicted Structures
T E T
Optimization
Chosen
DPO
TM-Align
Rejected
B
Construct Training Pairs
Iterative Design
Generate
sequences
S
K
G
N1/2
Inverse Folding Model
Folding Model
0.71
M
V
G
Q
TM Score
Optimize
Sort and split by structure similarity
Multi-Round DPO
Figure 1: Overview of our methods. A. We connect the inverse-folding process and folding process with DPO, utilizing the structure similarity (TM-Scores) predicted by the folding model to guide the optimization of the inverse-folding model. B. We classify generated sequences into chosen and rejected based on their TM-Scores. C. Multi-round DPO for iterative refinement of designed sequences.
Current inverse folding models are typically formalized as generative models, which trained to maximize the conditional probability of protein sequences given their corresponding structures. The underlying assumption that proteins with similar sequences will also have similar structures. This allows for the design of new proteins that fold into desired structures by sampling from the learned generative model. However, previous studies 46 22 have identified proteins with similar sequences but significant differences in structure and function. This suggests that sequence similarity (which is the learning objective of many inverse folding models) not always reliable indicator of structural similarity. Moreover, generative models for sequences whether for protein sequences or text - may experience hallucinations and produce low-quality or repetitive responses. They can also sometimes fail to follow instructions accurately (27 5 [1 49], These challenges underscore the need for further improvements to enhance the reliability and accuracy of these models. Due to these challenges, practical protein design workflows [8 40) often involve generating a large number of sequences, followed by a sophisticated selection process to prioritize the most promising candidates. Only then can costly experimental validation be conducted. While necessary, this approach introduces additional steps and complexity into the protein design process, potentially limiting the broader application of inverse folding algorithms.
learning. represented by Direct Preference Opti- mization (DPO) [37], offer opportunity to mitigate those problems. These methods train models on a pre- ree-generated dataset, capture the preference information by constructing chosen and rejected training pairs, and teach the model to classify good or bad responses. Some efforts have been made to apply DPO on protein language models (28 44 and even inverse folding models [47] 35). However, these existing efforts primarily rely datasets derived from wet-lab experiments, where the function or structure preferences are manually curated through costly biochemical assays. As a result, the
Fortunately, recent advances in preference-bee
application of DPO and other preference-based training strategies has been restricted to small-scale datasets, limiting the scope and generalizability of the resulting models. In contrast, recent break- throughs in protein folding methods, such AlphaFold (21) ESMFold (25), and FoAFol 23] have enabled fast, accurate, and in silico assessment of sequence-structee structure compatibility at scale.
Motivated by this, we explore a fully in silico training scheme that uses folding model feedback to enhance the sequence design capabilities of inverse folding models. Specifically, we utilize an inverse folding model to generate diverse set of protein sequences. These sequences are then fed

---

into a folding model to predict their corresponding structures. We evaluate these structures w.r.t the ground-truth structure using TM-Align [52] and classify the sequences into 'chosen' and 'rejected' based on the TM-Score. Finally, we employ Direct Preference Optimization (DPO) to fine-tune the inverse folding model. Building on this, we extended our investigation to explore the effects of multi-round DPO. Through extensive experiments, we demonstrate the effectiveness of using folding model feedback within a DPO framework to improve inverse protein folding. Our DPO models consistently achieve higher sequence recovery across all 3 datasets (CATH 4.2 [32], TS50 and TS500 (22) compared to their baselines. Notably, a detailed study on the CATH 4.2 dataset shows a steady improvement in structure similarity, suggesting that our method learns prioritize backbonn and fold-related features. Furthermore, scaling experiments reveal that the quality of contrastive samples is more crucial than sheer quantity for achieving high structural fidelity. Finally, experiments of multi-round DPO showcase substantial gains in TM-Score on challenging protein structures, evidencing the iterative refinement ability of our methods.
2 Preliminaries
Protein Folding and Protein Inverse Folding. The protein folding problem seeks predict a protein's 's 3D structure given its sequence. The goal is to learn a mapping f S -> TT, where S and T are the spaces of protein sequences and structures, respectively. Inverse folding (aka "protein sequence design") predicts protein's is sequence given its structure, which learns an inverse mapping
g: T + from structure space T to sequence space S. In particular, denote D C T x S to be the dataset of known struecure-sequeecce pairs (e.g.. from PDB or CATH). The inverse folding model (parameterized by 0) is trained to maximize the conditional likelihood as follows:
max ETT,Sy> [log Po ,1,....,81. T));
where T denote the protein structure, and = {81, 82, ........... ) denote the sequence of length T. A common choice for factorizing the sequence probability is to adopt an autoregressive approach, leading to what is often referred to as the language model loss:
L
Linv -
Σ log Po (s: 8cx),
i=1
which encourages the model recover ground truth residues, conditioned on both the target structure and previously predicted residues.
Reinforcement Learning from Human Feedback. Let no be the language model parameterized by 0. The goal of Reinforcement Learning from Human Feedback (RL.F) [8] 33 is to fine-tune large language models (LLMs) to better align with human preferences. Typically, given the prompt x, the language model Th (y/zz generates a response y. and human feedback in the form of pairwise preferences between completions is used to train a reward function [z v). This reward model helps the LLM prioritize preferred completions yoe over less preferred ones yr. with the probability of
preference modeled by:
p(y= > y1|=) = (ro(2, Uw) - (2 U1))
(3)
where is the sigmoid function defined (2)) 1 probability, and the reward function ro is learned by minimizing the negative log-liiel over the preference dataset D ppair:
Ite-s- The objective is to maximize this
LR(T Dpair) -E (t)).......)>par.. [log (r vw) rx w)]
To prevent the fine-tuned model Ty from deviating too much from the initial supervised model Tea a KL-divergence constraint added, preserving linguistic quality while aligning with human prefer- ences. However, RLHF can be complex and unstable. A more recent approach, Direct Preference Optimization (DPO) B77 simplifies the process by directly optimizing for human preferences without needing a reward model or reinforcement learning [42), offering more streamlined alternative. The DPO loss directly minmmizes the negative log-likelihood of the preference model:
To x)
To (y1
x))
[Opo(TT; Wret) E Ez (1)
log (3log
Blog
(5)
Teet (B xr)
(yr xx)
Here, 33 is a parameter that determines the level of information retention from the reference model, helping to balance novelty and adherence to the reference behavior. This setup allows for a more straightforward optimization process that is inherently more stable.
3

---

3 Methods
Constructing Preferences Datasets. Given a structure T drawn from dataset D, we first apply the inverse-folding model Ty to sample N initial predictions,
N
(
II (T(- n=1
(6)
Next, structures corresponding to the sampled sequence Sn are predicted using folding model, Tn g(S.). where g is protein folding model (e.g., ESMfold). The TM-Score on E [0, was then calculated with TMAlign [52] for each predicted structure in relation to its corresponding ground-trute structure Cn TM-Score(Tn, I"). To create balanced training pairs, the top-ranked 50% of sequences are labeled as chosen, while the bottom 50% are labeled as rejected (Fig. D.. Formally, for N generated sequences with TM-Scores {(S,,, Ca) Je we sort the sequences by the TM-Score in descending order and split into chosen S and rejected St as follows:
permute indices so that Ck(1) 2 Ck(1) 2 ... 2 CK(N).
Then for j = ,...... we have
(T, ST. St)
(I, (T, SkO) Sk(++
(8)
Finally, our new preference dataset can be written as D Dpair {(T, ( ensures balanced dataset for training.
N/2 This equal split S
Finetuning with DPO and SFT Loss. Unlike traditional NLPP datasets, designed protein sequences have much smaller vocabulary and can share similar residues even when the TM-Score is low. As the original DPO loss will minimize the probability of rejected responses, the probability of similar parts will also be minimized, eventually causing model degeneration (similar the alignment of mathematical problems in NLP tasks, where the responses are similar despite the answer being wrong 34 A simple method to tackle this problem to add an SFT loss as regularization term. Given the constructed preference dataset D pair, the DPO loss on the inverse folding model Te is:
To(S" T))
To(S (T) (
(Tro(TTe Feet) E(T, s S - llgg (Blog
BBlog
Teet S (2))
(9)
Tre (S (T)
The final loss can be written as:
L Thet) E (Tt.S,.St~~ppiiii. loggo log [T)] + LDPO,
(10)
where the Å is a super-parameter to control the contribution of rejected responses during training.
Iterative Training. To investigate whether methods consistently improve the performance in
a multi-round setting, we also employ an interactive training framework where the inverse folding models are successively refined using preference data generated by their predecessors. For each iteration i, the model TÓ will first generate and construct preference data Dr as described in Sec. 3 the model is then updated to # with the DPO and SFT loss (Eq. 10) generated dataset D) Daair Crucially, the reference model Teet reinitialized with the weight of प. at the start of each iteration, ensuring consistency in preference alignment while avoiding catastrophic forgetting.
Experiments
4.1 Benchmark Performance
To assess the broad applicability of our method, we apply it to two representative protein design models, ProteinMPNN [10] and InstructPLM (36), containing around million and 7 billion parame- ters. This setting helps verify the performance of our method across both large and compact model architectures. Specifically, we first generate sequences and train models on the CATH 4.2 training set and investigate the performance on the test set. To further probe generalization, we also evaluate on two additional benchmarks, TS50 and TS500 [12, which comprise 50 and 470 diverse proteins, and are often employed additional benchmarks further test generalization capability [53 [5]

---

Table 1: Sequence design performance. We report the perplexity and recovery rate on the CATH 4.2, TS50, and TS500 datasets. The performance of our methods is shown bold, while baselines without fine-tuning are indicated with an underline.
CATH 4.2
TS50
TS500
Model
Perplexity
Recovery
Perplexity
Recovery
Perplexity
Recovery
ProteinMPNN 10
5.41
40.27
5.10
44.72
4.57
46.58
PiFold [5
4.55
51.66
3,86
58.72
3.44
60.42
LM-Design
53
4.52
55.65
3.50
57.89
3.19
67.78
KW-Design
13
3,46
60.77
3.10
62.79
2.86
69.19
InstructPLM 36
2.63
53.58
2,46
60.20
2.14
66.13
ProteinMPNN-DPO
5.09
41.29
4.85
45.91
4.26
48.23
InstructPLM-DPOO
2.71
55.21
2.52
62.01
2.17
66.37
Table Structure similarity of designed sequences. We report the TM-Scores on different splits of the CATH 4.2 test set, the best results are shown in bold.
Model
Short
Single-Chain
All
TM-Score< 0.5
TM-Score>> 0.5
InstructPLM
0.54
0.56
0.77
0.37
0.87
SFT
0,54
0.57
0.78
0.39
0,87
DPO
A
10
0.57
0.61
0.80
0.43
0.88
A
0.58
0.63
0.81
0.45
0.89
0.57
0.62
0.81
0.88
beyond the CATH dataset. Details of hyperparameters are shown in Tab. 7 An interesting observation that ProteinMPNN fails to converge on self-generated datasets (Sec. We infer that the learning capacity of ProteinMPNN is too small to learn from its predictions. To tackle this problem, construct a simpler task, rather than fine-tune ProteinMPNN only on model predictions, we also add wild-type sequences additional chosen samples, details provided in the Appendix A..4 A.4
As Tab. T shows, both DPO models yield consistent gains over their respective baselines. Specifically, PitructPL.M-DPP. raises recovery by 3.0% (from 53.58% to 55.21% % ), and similar gains are also achieved in ProteinMPNN-DPO (from 40.27 41. 11..%) In terms of perplexity, is no surprise that InstructPLM-DDRR has slight increase, as the model is trained exclusively sequences, which inevitably cause a distribution shift from the reference model. However, despite the distribution shift, the perplexity remains within competitive ranges. On the other hand, as ProteinMPNN has an extra supervision of wild-type sequences, the perplexity of ProteinMPNN -DO successfully decreased. On TS50, InstructPLMM MPM-PP increases recovery from 60.20 % to 62.01 %, and on TS500 from 66.13 13 % to 66.37 %, while perplexity rises only 0.06 and 0.03, respectively. ProteinMPNN-DPP also exhibits similar performance improvement, recovery rate increases from 44.72 to 45.91 on TS50, and even more steady increases from 46.58 to 48.23 on the TS500 dataset. This confirms that preference optimization enhances design quality across both large and compact model architectures.
self-generated
4.2 DPO Achieves Higher Structure Similarity
To evaluate whether our methods acquire transferable, structure-related features solely from self- generated training sequences, we investigate the structure similarity measured by TM-Score of InstrOc on the CATH 4.2 test set, all structures are predicted using ESMfold. As Tab, 2 shows, despite never seeing native test-set examples during training, InstrucIPLM-DDO yields consistently higher TM-scores across all splits of the CATH 4.2 test set. Specifically, the overall TM-Score reaches 0.81 for our best model, which witnesses a steady performance gain compared with the baseline model (0.77). Moreover, for different splits of the test set, our methods improve the baseline from 0.54 to 0.58 on the Short split, which contains small proteins less than 100 amino acids, indicating performance improvement of both short and long proteins. For Single-chain, the model achieves an improvement of 12.5% compared to the baseline model (from 0.56 0.63). Notably, all of the performance gains are acquired from the training set, the only supervision comes from the
5

---

4a5p.
1kvz. A
600000
0.5
4a5p.A-
0.8
0.8
1kvz.A
TM- S--Score
0.6
0.6
0.4
315d.A
à mean TM-Score
2107.A
0.4
0.4
0.3
0.2
0.2
0.2
0.45 0.5
0.25 0.3
0.35 0,4
315d.A
21n7.A
0.1
1
0.8
0.8
M--ccare
0.6
0.6
-0.1
-0.2
0.2
0.2
R/ 0.19
-0.2
0.2
0.4 0.45 0.5
0.25 0.3 0.35 0.4
Recovery
Recovery
mean Recovery
4a5p.A
1kvz. A
Recovery: TM-Score: 0.94
Recovery: TM-Score: 0.37
Recovery: 0.35
Recovery: 0.36 TN-Score: 0.51
TH-Score: 0.87
3
2ln7.A
Recovery: 0.54
Recovery: 0.57
Recovery: 0.29
Recovery: 0.30
TM-Score: 0.90
TM-Score: 0.55
TM-Score: 0.
TH-Score: 0.36
Figure 2: Per-structure changes in TM-score vs. changes sequence recovery on the CATH 4.2 test set. Top left: Change TM-Score (mean TM-Score 10 predictions) change in recovery (mean Recovery over 10 predictions) for each structure. Top right: TM-Score versus recovery of 10 sequences sampled from the baseline and the DPO model, samples selected for visualization marked by circle. Bottom: Examples structure predicted by ESMfold, ground-truth structures are shown in gray.
folding model. These improvements demonstrate the effectiveness and generalization ability of our methods.
To better understand the role of chosen and rejected samples, conducted ablation study different weights of X, where SFT indicates that only using chosen examples to train models (Eq. 10). We also divide the test set into low-similarity group and high-similarity group based on the bascline performance. As Tab. 2 shows, simply training models on the chosen sample will also benefit the overall structure similarity (from 0.77 0.78), but still left behind by jointly using the chosen and rejected samples simultaneously. Introducing the rejected sample in optimization leads significant performance gain, even under chosen example dominant setting (A 10). Specifically, the rejected sample is more useful in low-similarity samples (from 0.37 to 0,45), showing that rejected samples help model lowering the probability of undesired generations. Increasing weight of rejected samples will further improve the performance (A vs. X 10). However, the performance starts to drop when the rejected samples outweigh (A 0) too much. These results show the importance of balancing the contribution of chosen and rejected samples during training.

---

56
82
2.2
Recovery Rate
55.5
55
MM-Score
80
2,1
54.5
79
2.05
78
2
53.5
77
1.95
76
1.9
1000
2000
4000
1000
2000
4000
1000
2000
4000
Steps
Steps
Steps
82
2.2
Recovery Rate
55.5
81
2.15
TM-Score
80
2.1
RASD
54.5
79
54
78
2
53.5
77
1.95
53
76
1.9
0
20
40
80
20
40
80
0
20
40
80
Samp Les
Samp
Samples
Figure 3: Performance at different training scales. The upper row shows protein design perfor- mance changes at different training steps, and the lower row shows protein design performance at different numbers of contrastive samples (N in Eq. (6) of each structure.
4.3 A Closer Look at Proteins Generated by DPO Model
To evaluate the impact of our methods on the DPO model, we analyzed the relationship between the changes in structure similarity of sequences designed by InstructPLM-DDR Score (A mean TM-Score), and the corresponding changes in sequence recovery (A mean Recovery). These changes
measured by the TM-
calculated by taking the mean of 10 independently generated sequence samples for each protein structure in the CATH 4.2 test set. In general, a linear regression analysis reveals a positive correlation (y 0.79x 0.02), suggesting that improvements in sequence recovery tend to be associated with enhanced structure similarity, which aligns with the intuitive expectation that better sequence recovery often leads
accurate folds (top left in Fig. 2 However, the low coefficient of deternination (/²2 0.19) indicates a weak association. This implies that substantial increases in TM-Score can frequently occur with relatively small changes in sequence recovery. For instance, many exhibit large TM-score boosts (ATM-Score > 0.20) with small sequence recovery change (ARecovery < 00) This is reasonable as no wild-type sequences phase of InstructPLM-DPG These observations suggest that our fine-tuning process prioritizes the exploration of sequences that fold into more accurate structures, even if deviating from the wild-type sequence.
involved during the training
To further illustrate this observation, visualize 4 cases with significant structure similarity improve- ments after fine-tuning, protein 4a5p.A, Ikvz.A, 2ln7.A, and 3i5d.A.(Fii 2 top right and bottom.) and provide sequence alignment information for these four cases in the Appendix B.4 The top right panel of Fig. 2 shows TM-Scores and recovery rates of all 10 predictions of the baseline and DPO models of cases. The cases in the top row (4a5p.A and lkvz.A) show a clear trend of more robust predictions, that SitrructPLM-DPPS can eliminate the low-quality predictions and strengthen high-quality predictions, which aligns with our training target. Notably, the cases of the lower row ( and 2In7.A) evidenced the behavior that the DPO model is capable of generating more structurally accurate sequences at the same level of sequence identity. For example, the prediction of protein 315d.A of InstructPLM- M-DDO has TM-Score of 0.90 at a recovery rate level of 0.56, while TM-Score of the baseline model the same level of recovery rate (0.57) only has TM-Score of 0.55. Predictions of protein 2In7.A also supports this trend, the fine-tuned prediction achieves TM-Score of 0.77 with a recovery rate of 0.29, showing notable increases TM- TM--Score at the same recovery level.

---

Low TM-Score
High TM-Score
Beseline
Fine-tuned
0.4
0
0.2
0.0
1mkn.A
Icyu,A
1fuw.A
2kxg.A
1u96.A
4qtw:K
1h31.
3jcu.Z
2caz.C
1grx.
Figure 4: Multi-round result on 10 hard structures selected from CATH 4.2 test set. The plot shows the TM-Scores of predicted structures from the final round.
4.4 DPO at Scale
Scaling has been shown to improve performance in large language models and, recently, in Post-training scaling techniques also yield gains in general LLMs (24), but their impact protein design remains unclear. Here, systematically explore two axes of scale in our framework: the number of training steps and the number of contrastive samples per structure. To disentangle these factors, we ran two sets of experiments on the CATH 4.2 training split: Training steps, We fixed the number of generated sequences at 20 (10 chosen vs. 10 rejected) and evaluated performance the training steps at 000 and varied the number of generated sequences (contrastive pairs) from 0 up 80 per structure. The baseline performance corresponds step and zero contrastive samples in each scenario.
protein language models 22. 39 6
I 000,, 000, and 4 000 update steps. Contrastive samples, We fixed
As illustrated in Fig. increasing the number of training steps yields consistent improvements in both sequence recovery and structural metrics (TM-score and RMSD), showing that the training process keeps improving the performance of the model. However, improvements saturate at the training step of 4000, indicating a performance bound that we can reach in this setting. In contrast, scaling up the number of contrastive samples boosts recovery rate but degrades structural fidelityTMM-See decline and RMSD increases. A possible explanation is that larger contrastive sets dilute the useful many low-quality samples, for example, rejected samples with high TM-Scores, thus hindering the learning process eventually. These experiments highlight that data quality outweighs sheer quantity. Our experiments shed light that future work should therefore focus on enhanced sample selection or multi-objective scoring strategies to maintain high-qualij, contrastive examples while preserving fold accuracy.
training signals by introducing
4.5 Iterative Protein Design
To evaluate the capacity for iterative improvement, we selected 10 difficult structures from the CATH 4.2 test set assess the performance limits of our approach. First, we divided all structures into "Low TM-Score" and "High TM-Score" groups based on the baseline performance. Within each group, structures with the lowest TM-Scores were chosen. Implementation details, including the number of rounds, generations, training steps per round, sampling temperatures, and top-p values, are detailed in Tab.
The final results, presented in Fig. 4 demonstrate a significant performance gain across all 10 structures. The mean TM-Score improved by 79.5%, rising from 0.39 0.70. The High TM-Score group showed the most substantial improvement. Since the design model was already capable of generating high-quality sequences for these structures, DPO effectively reduced the occurrence of low- quality predictions. For instance, in designing proteins 4qiw. and 1h31,B, the baseline model could produce sequences with TM-Scores exceeding 0.8. However, the overall performance was diminished by high proportion of low-quality predictions (over half with T'-Scores below 0.5). After fine- tuning, the median TM-Score for out of structures this group surpassed 0.8. Importantly, the upper bound of design performance also increased, indicating that DPO not only minimizes

---

low-quality predictions but also more accurately aligns the space of high-quality predictions, enabling the exploration of previously unseen, superior predictions. Structures in the Low TM-Score group also exhibited substantial gains. Specifically, proteins Icyu.A, Ifuw.A, 2kxg.A, and 1u96.A now generate sequences with TM-Scorrs above 0.5, compared to baseline TM-Score of approximately 0.3. However, the DPO model still faces challenges in designing protein 1mkn,A, suggesting that stronger baseline model may be necessary for such difficult cases. Finally, considering the limitation of computational resources, we do not perform larger scale of experiments in multi-round DPO. Given the notable performance of iterative training, we believe this a promising topic and leave for future work.
5 Related Works
Preference Optimization from AI Feedback. In response to the significant costs and limited accessibility associated with curating high-quality datasets, Al feedback has emerged
critical component in preference fine-tuning. This approach leverages another LLM to predict the reward of
target model's response and subsequently fine-tunes the target model based on this Al Al-generated feedback [4] [24]. More recently. methods such as self-play [ 48) and self-rewarding [50] have moved towards eliminating the reliance on external LLMs for feedback. For instance, SPIN employs an iterative framework akin to DPO, utilizing model-generated data as rejected responses paired with human labels as the preferred responses, and iteratively refines the model. Self-rewarding [50] further abstracts away the need for chosen human labels by using the target model itself to determine preferred and dispreferred samples.
Preference Optimization in Protein Language Models. In contrast to the extensive exploration of preference optimization in natural language processing, protein language models (pI.Ms) have received comparatively less attention in this regard, primarily due to the scarcity of labeled datasets. Nevertheless, initial efforts are being made in this direction. Widatalla et al. [47] proposed ProtcinDPO to fine-tune ESM-IF for designing proteins with enhanced stability, while Mistani al. (8 applied DPO on ProtGPT2 for binder design. Distinct from these prior studies that heavily depend on manually annotated datasets, our approach leverages synthetic data generated by the target model itself. Concurrently, Stocco et al. 44 introduced DPO_pt _LM to align pLMs on various desired properties, including structure similarity. However, their application of DPO focused on ZymCTRL [29] model specifically designed for functional enzyme design, whereas our work explores broader spectrum of protein design tasks. Similarly, Park et al. [35] explored the application of DPO on ProteinMPNN for peptide design, which operates at a smaller scale in terms of both data and model size compared to our methods.
6 Conclusion
Summary. We introduce a novel framework for optimizing inverse-folding models by the feedback of the folding model through preference optimization. Our primary finding is the significant im- provement in both the sequence recovery and the structure similarity of designed protein sequences, as evidenced by extensive experiments on the CATH 4.2 dataset compared to the baseline model. Moreover, the case study shows that our methods not only preserve high-quality predictions of the baseline model while eliminating the low-quality predictions, but also generate more structurally accurate samples even at the same level as the baseline model. This indicates that the model learns to prioritize the underlying principles of protein folding.
Social impact. Our end-to-end, fully in silico optimization pipeline for protein inverse folding offers significant benefits across healthcare, industry, and academic research. Notably, work validates paradigm, suggesting that the pipeline is flexible
the effectiveness of the wwttmiiiatioo--o-feeeeiieee and can readily incorporate alternative forms of feedback beyond structural metrics, thereby enabling the development of more powerful and versatile protein design models.
While increasing the number of training steps generally leads to improved performance, simply scaling the number of contrastive samples per structure can be detrimental to structure similarity, even if it slightly boosts sequence recovery. This finding underscores limitation of our current approach. More sophisticated preference dataset construction strategies may further improve the performance.
Limitations.

---

Acknowledgments and Disclosure of Funding
The work described in this paper was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N, and by the Hong Kong Innovation and Technology Fund, under Project ITS/241/21. Jiezhong Qiu was supported by NSFC 62306290.
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):9935500 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Katherine I Albanese, Sophie Barbe, Shunsuke Tagami, Derek N Woolfson, and Thomas Schiex. Computational protein design. Nature Reviews Methods Primers, 5(1):13, 2025.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[5] Zechen Bai, Pichao Wang. Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
[6] Aadyot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel C Curran, Alexander M Hoffnagle, Kyle Ching, Michael Martyn, Stephen Nayfach, Jeffrey A Ruffolo, and Ali Madani. Scaling unlocks broader generation and deeper functional understanding of proteins. bioRxiv, pages 2025-04, 2025.
[7] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. arXiv preprint arXiv:2401 0..03333. 2024.
[8] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pages 2990-2999. PMLR, 2016.
[10] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science, 978(6615):49-56 2022.
[11] Justas Dauparas, Gyu Rie Lee, Robert Pecoraro, Linna An, Ivan Anishchenko, Cameron Glasscock, and David Baker. Atomic context-conditioned protein sequence design using ligandmpon. Nature Methods, pages 1-7, 2025.
[12] Nuno A Fonseca, Rui Camacho, and AL Magalhaes. Amino acid pairing at the n-and c- termini of helical segments in proteins. PROTEINS: Structure, Function, and Bioinformatics, 70(1):188-19) 2008
[13] Zhangyang Gao, Cheng Tan, Xingran Chen, Yijie Zhang, Jun Xia, Siyuan Li, and Stan 7. Li. K KW-design: Pushing the limit of protein design via knowledge refinement. In The Twelfth International Conference Learning Representations, 2024.
[14] Zhangyang Gao, Cheng Tan, and Stan Li. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01077 2022.
10

---

[15] Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding, 2022.
[16] Thomas Hayes, Roshan Rao, Halil Akin, Nicholas Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, al. Simulating 500 million years of evolution with language model. Science, page eads0018, 2025.
[17] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. International Conference on Machine Learning, pages 8946-8970, 2022.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2021.
[19] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[20] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
[21] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron-
neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583 5899, 2021.
[22] Mickey Kosloff and Rachel Kolodny, Sequence-simine simila,,,, structure-dissimilar protein pairs in the pdb. Proteins:
Structure, Function, and Bioinformatics, 71 2008
[23] Rohith Krishna, Jue Wang. Woody Ahem, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Ivan Anishchenko, Ian Humphreys, et al. Generalized
Gyu Rie Lee, Felix biomolecular modeling and design with rosettafold all-atom. Science, :ead1693)e 2024.
[24] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint vrXiv:2309.00266 2023.
[25] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science,
2023.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05101 2017.
[27] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497, 2020.
[28] Pouria Mistani and Venkatesh Mysore. Preference optimization of protein language models as a multi-objective binder design paradigm. arXiv preprint arXiv:2403.04/88 1877 2024.
[29] Geraldene Munsamy, Ramiro Illanes- Vicioso, Silvia Funcillo, Ioanna T Nakou, Sebastian Lindner, Gavin Ayres, Lesley S Sheehan, Steven Moss, Ulrich Eckhard, Philipp Lorenz, et al. Conditional language models enable the efficient design of proficient enzymes. bioRxiv, pages 2024-05, 2024.
[30] Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11):968- 979-9777, 2023.
[31] Pascal Notin, Nathan Rollins, Yarin Gal, Chris Sander, and Debora Marks. Machine learning for functional protein design. Nature biotechnology, 42(2):216 2024.

---

[32] Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thomton. Cath-a hierarchic classification of protein domain structures. Structure, 5(8):1093-1109, 1997.
[33] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[34] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
[35] Ryan Park, Darren J Hsu, C Brian Roland, Maria Korshunova, Chen Tessler, Shie Mannor, Olivia Viessmann, and Bruno Trentini. Improving inverse folding for peptide design with diversity-regularrzed direct preference optimization. arXiv preprint arXiv:2410. 19471, 2024.
[36] Jiezhong Qiu. Junde Xu, Jie Hu, Hanqun Cao, Liya Hou, Zijun Gao, Xinyi Zhou, Anni Li, Xiujuan Li, Bin Cui, Fei Yang, Shuang Peng, Ning Sun, Fangyu Wang, Aimin Pan, Jie Tang, Jieping Ye, Junyang Lin, Jin Tang, Xingxu Huang, Pheng Ann Heng, and Guangyong Chen. Instructpim: Aligning protein language models to follow protein structure instructions. bioRxiv, 2024.
[37] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[38] Albert Ros-Lucas, Nieves Martinez-Peinadee Jaume Bastida, Joaquim Gascón, and Julio Alonso- Padilla. The use of alphafold for in silico exploration of drug targets in the parasite trypanosoma cruzi. Frontiers in Cellular and Infection Microbiology, 12:944748, 2022.
[39] Jeffrey A. Ruffolo, Aadyot Bhatnagar, Joel Beazer, Stephen Nayfach, Jordan Russ, Emily Hill, Riffat Hussain, Joseph Gallagher, and Ali Madani. Adapting protein language models for structure-conditioned design. bioRxiv, 2024.
[40] Valeria Scardino, Juan Di Filippo, and Claudio N Cavasotto. How good are alphafold models for docking-based virtual screening? Iscience, 26(1), 2023.
[41] Christophe Schmitz, Robert Vernon, Gottfried Otting. David Baker, and Thomas Huber. Protein structure determination from pseudocontact shifts using rosetta. Journal of molecular biology, 416(5):668-677 677, 2012.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[43] Radka Snajdrova, Rebecca Buller, Stefan Lutz, Jeffrey Moore. Romas Kazlauskas, and Uwe Bornscheuer. Surfing the biocatalysis wave to new applications changed to: From nature to industry: Harnessing enzymes for biocatalysis. as per request of editor. Science, 382(6673). 2023.
[44] Filippo Stocco, Maria Prrigues-Lleixx Andrea Hunklinger, Talal Widatalla, Marc Guell, and Noelia Ferruz. Guiding generative protein language models with reinforcement learning. arXiv preprint axXiv:2412.12977 2024.
[45] Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Li. Global-context aware generative protein design. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[46] Annahel E Todd, Christine A Orengo, and Janet M Thornton. Evolution of function in protein superfamilies, from a structural perspective. Journal of molecular biology, 307(4): 113-1143, 2001.
[47] Talal Widatalla, Rafael Rafailov, and Brian Hie. Aligning protein generative models with experimental fitness via direct preference optimization. bioRxiv, pages 2024-05, 2024.
12

---

[48] Yue Wu, Zhiqing Sun, Huizhuo Yuan. Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self preference optimization for language model alignment. arXiv preprint arXiv:2405.0067,, 2024.
[49] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang. Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944-12953, 2024.
[50] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 2024.
[51] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
[52] Yang Zhang and Jeffrey Skolnick. Tm-align: protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-22 22009, 2005.
[53] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure- informed language models protein designers. bioRxiv, pages 2023-02, 2023,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show how we the folding model to optimize the inverse-folding through DPO, improving the structure similarity designed sequences.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the main limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We build our methods on open-sourced algorithms and all hyperparameters are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We have released our code at htttps::/githubbccom co
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com http///bbb///piiiiedoeee/eaaaaaa cwwsinnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide the training details in Section and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our study used fixed seed for all experiments, following standards in protein design tasks, and thus did not report statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments are done with &*NVIDIA A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/ss///.rrrc..cc///p//////t
Answer: [Yes]
Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss broader impacts of our work in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all datasets and models utilized in experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: release our models and documents at lattps://github.comm/ /ikor/// iplm-rl
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //errri////upp...// wwwparripppcccooeeeeeeeeeeeeeeee /c//oonfreennes/2225 for what should or should be described.
20

---

A Implementation Details
A.1 Generation Configs for Evaluation
To better evaluate the performance of DPO models, generate 10 samples at temperature of 0.15 for each structure in all datasets used during evaluation. We report the mean value of 10 predictions for each metric.
A.2 Low-Rank adaptation
To keep the knowledge in the pre-trained model, we utilized LoRA [8] during the training process of InstruciPLM-DPO. The key idea of LoRA (Low-Rank Adaptation) is to enhance the adaptability and efficiency of pre-trained models by integrating trainable low-rank matrices into existing model weights, allowing for targeted modifications without retraining the entire model.For each layer in the pre- tre-trained model, let h E Rixd be the inputs of n-th layer, is the sequence length, and d is the hidden dimension. 2e be the output of QKV projection module of nth layer, the updated output hi can be written
h ako likeo WB x WA X x
(11)
where WA E R'xd and WB E Rdxr are LoRA weights, which we initialized randomly from Gaussian distribution with mean zero and standard deviation of 0.01. We set the rank as 16 by default, resulting in fraction of 0. of trainable total parameters. The strategy helps us balance model expressiveness with computational efficiency.
A.3 Hyperparameters
We two different training configs for single-round training and multi-round training. The main difference lies in the generation setting and training steps. Specifically, for the construction of the single-round dataset, we generate 20 responses for each structure, with temperature 1.0 and top p at 0.9. For multi-round, in order to encourage model exploration, we set the temperature at 1.1 and top p at and generate 200 responses for each structure. For DPO training, we use AdamW (26) as optimizer. We set 8 to 0.5 for all experiments. We train our model on 8 Nvidia A100 GPUs with a batch size 128. Other parameters are summarized in Tab.
A.4 Preference Optimization ProteinMPNN
To train the ProteinMPNN model, we slightly changed our dataset curation process and constructed a simpler task. Specifically, we set a new threshold t, = 0.8 to control the process of classifying samples into chosen and rejected. Instead of training on all predictions like InstructPLM, we only trained ProteinMPNN on the predictions with a TM-Score lower than t,.. Meanwhile, the wild-type sequence has also been introduced as chosen sample during training. As Fig. Fisows, when trained on the original task simpler task.
3 the model failed to converge, whereas the model successfully trained on a
B
Additional Results
B.1 Compare with Other Baselines
We provide a more detailed comparison with other baselines in Tab.
B.2 Evaluation using AlphaFold 3
To examine the robustness of our method across different structure predictors, we evaluated designs generated by our baseline and fine-tuned models using AlphaFold 3 [.]. Specifically, for each test structure where multi-round DPO was applied, we folded 10 designed sequences using AlphaFold 3, resulting in 200 sequences in total (100 from the baseline and 100 from the fine-tuned model). We report the average TM-Score and RMSD per structure, along with standard deviations, in the table 3 These results confirm that the improvements achieved by our method are consistent across
21

---

different folding models, suggesting that our approach captures structure-relevant features beyond model-specific feedback.
B.3 Performance CATH 4.3
To further assess the robustness of our method, we additionally evaluate it on selected structures from the newer CATH 4.3 release (Table Specifically, we selected protein domains from newly added Classes in CATH 4.3 that are not present in CATH 4.2, ensuring no overlap with the training data not just at the Fold level, but at the broader Class level (the Fold and Class level of protein structure classification can be find in (32). After filtering out sequences longer than 512 residues, we obtained test set comprising 750 diverse and challenging structures. Compared test set, which differs from training data at the Topology (Fold) level, this new dataset represents a more stringent test generalization. The results show that our fine-tuned model consistently outperforms the baseline across all metrics. This performance gap is especially notable given the increased structural diversity and novelty the CATH 4.3 set.
the original CATH 4.2
B.4 Sequence Alignment of Visualized Samples
We performed sequence alignment (Fig. 7 to 10 of visualized samples in Fig. 2 illustrating each residue as a colored box highlight matches (green), mismatches, and uncertain positions (gray). The results show that our fine-tuning process prioritizes the generation of sequences that fold into more accurate structures, even if it means deviating from the native sequence (low recovery rate).
B.5 Improvements on different fold types.
We conducted detailed analysis where we grouped the test structures in CATH 4.2 according to their Class annotations (Mainly Alpha, Mainly Beta, Alpha-Beta, and Few Secondary Structures). The average TM-scores before and after DPO fine-tuning are summarized in Table As shown, DPO fine-tuning brings consistent improvements across all fold types, with especially notable gains in the Mainly Alpha and Mainly Beta categories. This demonstrates that our method not only improves structural quality overall but also robustly across diverse range of protein topologies.
B.6 More Results on Iterative Design.
For additional clarity, we monitor the running TM-Scores and recoveries of all 10 structures during training in Fig. 6 The plot demonstrates a clear upward trend, with substantial gains in the early rounds, followed by a gradual saturation effect as the model converges towards higher structure similarity. Concurrently, the recovery rate remains a relatively low level, further emphasizing our method's ability to prioritize structural fidelity over sequence similarity, leading to the discovery of alternative sequences that fold more accurately.
22

---

Table 3: Performance on the CATH 4.2 test split.
ESMFold TM-Score
AlphaFold3
Model
RMSD
TM-Score
RMSD
InstructPLM
0.40
3.42 2.25
0.43
3.09
InstructPLM-DDP
0.70
0.68
2.29
Table 4: Performance on newly introduced Classes of CATH 4.3 dataset with respect to CATH 4.2.
Model
Recovery
TM-Score
RMSD
InstructPLM
41.85
0.57 0.60
2.89
InstructPLM-DPO
43.62
2.79
Table 5: TM-Score comparison across different fold types.
Model
Mainly Alpha
Mainly Beta
Alpha Beta
Few Secondary Structures
InstructPLM
0.61
0.76 0.80
0.84
0.63
InStructPLM-DPO
0.66
0.86
0.65
23

---

Table 6: Sequence design performance on CATH 4.2 test set. We report the perplexity and recovery rate on the CATH 4.2 test set. The performance of our methods is shown in bold, while baselines without fine-tuning are indicated with an underline. We copied the results from [5] and reproduced the results of InstructPL. and ProteinMPNN in unified codebase.
Perplexity on CATH 4.2,
Recovery Rate on CATH 4.2
Model
Short Single-chain
All
Short
Single-chain
All
StrucIGNN
[9
8.29
8.74
6.40
29.44
28.26
35,91
GraghTrans
19
8.39
8.83
6.63
28.14
28.46
35.82
GCA 45
7.09
7,49
6.05
32.62
31.10
37.64
GVP 20
7.23
7.84
5.36
30.60
28.95
39.47
AlphaDesign
14
7.32
7.63
6.30
34.16
32.66
41.31
ProteinMPNN
10
8.82
9.17
5,41
28.83
27.20
40.27
ESM-IF 17
6.93
6.65
3.96
35.28
33.78
48,95
PiFold [15]
6.04
6.31
4.55
39.84
38.53
51.66
LM-Design
53
6.77
6,46
4.52
37.88
42.47
55.65
InstructPLM
36
3.22
3.17
2.63
40.88
40.87
53.58
KW-Design
13
5.48
5.16
3,46
44.66
45.45
60.77
ProteinMPNN-DPO
8.40
8.50
5.09
29.19
27.41
41.29
InstruciPL.M-DDR
3.54
3.43
2.71
43.34
43.44
55.21
Name
Single-round
Multi-round
Temperature
1.1
Top p
0.9
1.0
20
200
Number of rounds
20
DPO B
0.5
0.5
Learning rate
1e-5
1e-5
Optimizer
AdamW
Adam
Adam B
0.9
0.9
Adam 82
0.999
0.999
Adam
1c-8
1e-8
Batch size
128
128
LoRA r
16
16
LoRA a
16
16
Training steps
4000
200
Table 7: Training configs
Hard Task
Simple Task
0.7
0.8
Reward Accuracy
0.9
0.65
0.6
Margin
Loss
0.6
0.4
0.55
0.2
0.5
0.5
1K
3K
5K
1K
3K
5K
1K
3K
5K
Training Steps
Training Steps
Training Steps
Figure 5: Metrics of ProteinMPNN training process.

---

0.7
TM-Score
Recovery
0.6
0.5
0.4
0.3
0.2
5
10
15
20
Rounds
Figure 6: TM-Score and recovery rate changes across each round iterative refinement.
1kvz.A P
Type
S
baseline fine-tuned
Wid Type
F
CINVIIC
PREKKY
KILK
MI
E K
V VKL
51-100
5 5
Wild Type
V V
101-107
L
Figure 7: Sequence alignment of 1kvz.. Baseline recovery rate: 0.36, TM-Score: 0.51. Finetuned recovery rate: 0.35, TM-Score 0.87.
2ln7.A
Wid Type
G
baseline
WESEGT
AA
1.50
fine-tuned
M
H
M
VY
Widd Type
S
L
F
D
51-100
N
Wild Type
P
101-147
Figure 8: Sequence alignment of 2ln7.A. Baseline recovery rate: 0.30, TM-Score: 0.36. Finetuned recovery rate: 0.29, TM-Score 0.77.
25

---

315d.A
Wid Type
baseline
1-50
fine-tuned
Wild Type
FDGS
NO
6
19
LATE
WILL
AAL
51-100
Wil Type
101-150
Type
Wild Type
P
201-250
Wid Type
D
5
251-300
Wild Type
V
M
301-350
Type
351-356
Figure 9: Sequence alignment of 3.,d.A. Baseline recovery rate: 0.57, TM-Score: 0.55. Finetuned recovery rate: 0.54, TM-Score 0.90.
4ap5,A
Wid Type
baseline
1-50
fine-tuned
W
Wid Type
5
AAEN
N
MOMEE
F
Y
G
L
KLL
51-100
DF
Wild
V
101-150
Type
Wild Type
201-250
KLL K
KLL N
K
5
KI
1K
Wild Type
VVVI
LS
251-300
M
Wild Type
V V
301-350
Wild Type
351-383
Figure 10: Sequence alignment of 4ap3.A. Baseline recovery rate: 0.38, TM-Score: 0.37. Finetuned recovery rate: 0.49, TM-Score 0.94.
26


Reviews:
[{"review_id": "fwX9t5T438", "reviewer": "Reviewer_UQVk", "summary": "This paper presents a method for improving protein inverse folding by leveraging feedback from a folding model using Direct Preference Optimization (DPO). Given a target structure, the inverse folding model generates candidate sequences, which are scored by a folding model based on structural similarity (TM-score). These scores are used to create preference pairs (chosen vs. rejected), which guide DPO fine-tuning of the inverse folding model. The approach improves both sequence recovery and structural accuracy across datasets (CATH 4.2, TS50, TS500), and shows strong gains in multi-round iterative refinement. The method is fully in silico, scalable, and applicable to both small and large protein models.", "questions": "Questions for the authors:\n\n1. Have you analyzed how performance scales with the number of unique training structures (i.e., a full learning curve)?\n2. How do you control for low-quality or ambiguous contrastive samples? Have you tried thresholding TM-Score or using clustering?\n3. Does structural diversity in the training set affect generalization?\n4. Have you tested sensitivity to the choice of folding model for feedback (e.g., ESMFold vs. AlphaFold)?\n\nSuggestions:\n\n1. Run a learning curve experiment varying the number of training structures to assess data efficiency and saturation.\n2. Filter training pairs using absolute TM-Score thresholds instead of a 50/50 split to improve signal quality.\n3. Combine structural feedback with additional objectives (e.g., stability, binding affinity) to enrich preference signals.\n4. Evaluate generalization to unseen folds or novel structural classes to test robustness.\n5. Visualize how DPO shifts sequence distributions (e.g., t-SNE of embeddings) to verify preference-driven learning dynamics.\n6. The results in Figure 3 on scaling contrastive samples are insightful, but the granularity is limited. The jumps from 0 to 20 to 40+ samples are too coarse to reveal meaningful trends or tipping points. A more fine-grained analysis (e.g., increments of 5 or 10) would help clarify how performance evolves and where degradation begins.\n7. Table 1 reports performance metrics without standard deviations, making it unclear whether improvements are statistically significant. To address this, multiple training runs with bootstrapped training set samples should be performed. Statistical significance can then be assessed using critical difference diagrams (e.g., via the `scikit-posthocs` package: [https://scikit-posthocs.readthedocs.io/en/latest/generated/scikit\\_posthocs.critical\\_difference\\_diagram.html](https://scikit-posthocs.readthedocs.io/en/latest/generated/scikit_posthocs.critical_difference_diagram.html)). This would provide a robust comparison across models and datasets.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**Quality:**\n\n* Methodologically solid; applies Direct Preference Optimization (DPO) to inverse folding using feedback from folding models.\n* Demonstrates consistent improvements in structure similarity (TM-score) and sequence recovery across models and datasets.\n* Multi-round DPO refinement shows strong gains, particularly on challenging protein structures.\n* Fully in silico pipeline avoids reliance on experimental data, supporting scalability.\n\n**Clarity:**\n\n* The paper is well organized and the method is clearly described.\n* Figures are helpful, especially the pipeline diagram and TM-score visualizations.\n\n**Significance:**\n\n* Addresses a key gap: current inverse folding models over-optimize for sequence similarity.\n* Shows that structure feedback can improve sequence design quality in practice.\n* Results have practical implications for scalable, structure-aware protein design.\n\n---\n\n**Weaknesses**\n\n**Quality:**\n\n* Table 1 lacks standard deviations; no statistical testing is reported.\n\n  * Authors should use bootstrapped training set sampling and include critical difference diagrams to establish significance.\n* Figure 3 is too coarse (e.g., 0, 20, 40 samples); finer-grained evaluation would clarify where data scaling starts to hurt performance.\n* No learning curve analysis with respect to the number of unique training structures; it's unclear whether performance is limited by data volume or model capacity.\n\n**Clarity:**\n\n* Some figures (e.g., Fig. 3) lack clarity in axis labels and legends.\n* The role and tuning of λ in balancing DPO and SFT losses could be better explained.\n\n**Significance:**\n\n* Generalization to novel folds or unseen structural classes is not evaluated.\n* No integration of functional objectives (e.g., stability, binding) beyond structural similarity.\n\n**Originality:**\n\n* The use of DPO in protein language models is not new.\n\n  * Section 5 notes similar approaches (e.g., ProteinDPO, DPO\\_pLM), though those focus on other objectives (e.g., stability, function) or smaller scales.\n  * This work's main novelty is the specific application of **structure-based feedback** and fully synthetic training, which is incremental rather than fundamentally new.", "was_revised": true, "final_justification": "This paper presents a method for improving protein inverse folding by applying Direct Preference Optimization (DPO) guided by structure-based feedback from folding models. The rebuttal was thorough and addressed several of my concerns, strengthening the submission. However, some limitations remain, particularly in terms of evaluation depth and statistical rigor.\n\n### Resolved Issues:\n\n* **Learning curve and data efficiency:** The authors provided a new ablation showing graceful degradation with reduced structural diversity, demonstrating robustness to dataset size.\n* **Generalization to novel folds:** The authors clarified their original split ensured fold-level disjointness and added new results on CATH 4.3, confirming performance gains on structurally novel targets.\n* **Finer-grained sampling analysis (Fig. 3):** Additional experiments with intermediate numbers of contrastive samples clarified the performance plateau and supported their original design choices.\n* **Robustness to folding model choice:** Evaluation using AlphaFold3 confirmed that gains are not specific to ESMFold, increasing confidence in the generality of the method.\n\n### Remaining Issues:\n\n* **Statistical significance and variability:** The authors reported per-target standard deviations at inference time but did not perform training-level replication or statistical testing (e.g., bootstrapping, critical difference diagrams). This limits confidence in the reported improvements, particularly in Table 1.\n* **Lack of broader objective integration:** The framework remains narrowly focused on structural similarity. While extensibility was acknowledged, no preliminary results on alternative design objectives (e.g., stability or binding) were provided.\n* **No sequence space analysis:** The suggestion to visualize how DPO shifts sequence distributions was acknowledged but deferred, missing an opportunity to clarify model behavior.\n\n### Final Assessment:\n\nThis is a technically solid and clearly written paper that addresses a relevant problem with a practical, scalable method. The rebuttal meaningfully improved the submission, particularly in evaluating generalization and training dynamics. However, due to limited statistical rigor and a somewhat narrow scope of evaluation, I maintain a **Borderline Accept (4)** rating.", "rebuttal": "We thank the reviewer for their thorough and insightful comments. We appreciate the recognition of our method’s strengths in terms of clarity, methodological rigor, and the practical significance of integrating folding-based feedback via Direct Preference Optimization (DPO). Below we address each concern in detail.\n\n**Question 1, Suggestion 1, Question 3**: We thank the reviewer for raising the important question regarding the effect of structural diversity in the training set on generalization. To investigate this, we conducted an ablation study by subsampling the number of training structures from the full CATH 4.2 training set. Specifically, we randomly selected **50% (8,987 structures)** and **25% (4,515 structures)** of the full set, and re-ran the training pipeline under the same protocol, 0% indicates baseline performance before fine-tuning. The results on the held-out CATH 4.2 test set are summarized below:\n\n| Smaller Dataset  | 0%    | 25%   | 50%   | 100%  |\n| ---------------- | ----- | ----- | ----- | ----- |\n| Recovery &#8593; | 53.48 | 54.51 | 55.06 | 55.21 |\n| TM-Score &#8593; | 0.78  | 0.79  | 0.81  | 0.81  |\n| RMSD &#8595;     | 2.11  | 2.02  | 1.98  | 1.97  |\n\nThese results indicate that our method remains robust even when trained on a reduced number of structures, for example, it achieves comparable performance when using only 50% of the training set. However, as the number of training structures is further reduced, a noticeable drop in performance is observed. We will include these findings in the revised version to better illustrate the relationship between training set diversity and model generalization.\n\n**Question 2, Suggestion 2:** We use a 50:50 split instead of thresholding for the following reasons. First, the performance of protein design models varies significantly across different target structures. Applying a fixed TM-score threshold, for example, 0.8, would result in many structures lacking valid negative responses, as the model may generate only sequences above the threshold. This would reduce structural diversity in training. Second, for structures where the model performs poorly, it becomes difficult to find valid chosen responses, which leads to severe class imbalance. In contrast, the 50:50 split can be seen as a dynamic, per-structure threshold that ensures a balanced set of preference pairs for each structure. We acknowledge that this approach may introduce some noisy signals—for instance, cases where chosen responses have relatively low TM-scores or rejected responses have high TM-scores. However, since DPO is a contrastive learning objective, it only requires that the chosen sample be *relatively* better than the rejected one, not absolutely good. Given these considerations, we favored a simple but robust strategy over more complex selection mechanisms.\n\n**Question 4:** We agree with the reviewer that AlphaFold generally provides more accurate structure predictions than ESMFold. However, due to its significantly higher computational cost, AlphaFold is not well-suited for integration into the training loop. To examine the robustness of our method across different structure predictors, we evaluated designs generated by our baseline and fine-tuned models using AlphaFold 3.  Specifically, for each test structure where multi-round DPO was applied, we folded 10 designed sequences using AlphaFold 3, resulting in 200 sequences in total (100 from the baseline and 100 from the fine-tuned model). We report the average TM-Score and RMSD per structure, along with standard deviations, in the table below:\n\n|            | ESMFold-TMScore &#8593; | ESMFold-RMSD &#8595; | AlphaFold3-TMScore  &#8593; | AlphaFold3-RMSD &#8595; |\n| ---------- | ----------------------- | -------------------- | --------------------------- | ----------------------- |\n| Baseline   | 0.40 (0.09)             | 3.42 (0.69)          | 0.43 (0.12)                 | 3.09 (0.73)             |\n| Fine-tuned | 0.70 (0.18)             | 2.25 (0.89)          | 0.68 (0.17)                 | 2.29 (0.83)             |\n\nThese results confirm that the improvements achieved by our method are consistent across different folding models, suggesting that our approach captures structure-relevant features beyond model-specific feedback.\n\n**Suggestion 3:** Combining structural feedback with additional objectives such as stability or binding affinity is indeed a promising direction. One of the main goals of our work is to evaluate whether fully in silico simulation alone can improve protein design performance, without relying on wet-lab feedback. Importantly, our framework does not make strong assumptions about the scoring function and is compatible with a wide range of design objectives. In principle, predicted metrics such as stability or binding affinity can also be seamlessly incorporated into our training pipeline. However, such metrics often require costly simulations or are less mature compared to structure-based evaluation. We leave the integration and benchmarking of alternative objectives to future work, and we hope our results can inspire broader exploration of these directions in the community.\n\n**Suggestion 4:** The CATH 4.2 test set already contains folds that are disjoint from the training set, as we follow the data split protocol from [1], which ensures no overlap at the **CAT (Topology/Fold)** level [2]. Therefore, the results reported in Table 1 can be interpreted as evaluating generalization to unseen folds. To further assess the robustness of our method, we additionally evaluate it on selected structures from the newer CATH 4.3 release. Specifically, we selected protein domains from newly added **Classes** in CATH 4.3 that are not present in CATH 4.2, ensuring no overlap with the training data *not* just at the **Fold** level, but at the broader **Class** level. After filtering out sequences longer than 512 residues, we obtained a test set comprising 750 diverse and challenging structures. Compared to the original CATH 4.2 test set, which differs from training data at the Topology (Fold) level, this new dataset represents a more stringent test of generalization. The results show that our fine-tuned model consistently outperforms the baseline across all metrics. This performance gap is especially notable given the increased structural diversity and novelty in the CATH 4.3 set. \n\n| CATH 4.3 new Class | Recovery &#8593; | TM-Score &#8593; | RMSD &#8595; |\n| ------------------ | ---------------- | ---------------- | ------------ |\n| Baseline           | 41.85            | 0.57             | 2.89         |\n| Fine-tuned         | 43.62            | 0.60             | 2.79         |\n\n**Suggestion 5:** We appreciate the reviewer’s suggestion to visualize how DPO shifts the sequence distribution (e.g., using t-SNE on embeddings) to better understand the learning dynamics. Due to current constraints during the rebuttal phase, we are unable to include such analyses at this stage. However, we fully agree that improved visualizations would strengthen the work, and we plan to incorporate them in a future revision of the paper.\n\n**Suggestion 6:** We conducted additional experiments with smaller increments in the number of contrastive samples (0, 10, 20, 30, 40, 80). The results show that performance improves up to around 20 contrastive samples, after which it begins to plateau or slightly degrade. Specifically, both TM-score and RMSD indicate that using 20 samples yields the best balance between structural accuracy and stability, confirming the trend observed in our original submission. These findings support our original choice and reinforce the importance of moderate contrastive sampling in DPO training.\n\n| Steps            | 0     | 10    | 20    | 30    | 40    | 80    |\n| ---------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Recovery &#8593; | 53.48 | 55.09 | 55.21 | 55.04 | 55.15 | 55.18 |\n| TM-Score &#8593; | 0.78  | 0.80  | 0.81  | 0.81  | 0.78  | 0.78  |\n| RMSD &#8595;     | 2.11  | 1.98  | 1.97  | 1.98  | 2.08  | 2.08  |\n\n**Suggestion 7:** We appreciate the reviewer’s suggestion regarding statistical significance. Due to computational constraints, we were not able to retrain the model multiple times with different seeds or bootstrapped training sets. However, the results reported in Table 1 are already averaged over 10 independently sampled sequences per target structure, providing a robust estimate of performance variability at the inference level. Below, we report the standard deviations across these 10 samples for each metric:\n\n|            | Recovery (mean, std) &#8593; | TM-Score (mean, std) &#8593; | RMSD (mean, std) &#8595; |\n| ---------- | ---------------------------- | ---------------------------- | ------------------------ |\n| Baseline   | 53.58, 0.03                  | 0.78, 0.05                   | 2.11, 0.35               |\n| Fine-tuned | 55.21, 0.02                  | 0.81, 0.03                   | 1.97, 0.28               |\n\nWe believe these per-target statistics provide meaningful evidence of the improvements achieved by our method, even without full training-level replication. We agree that training-level variance is important and plan to include a more comprehensive statistical analysis in future work when resources permit.\n\n[1]: Ingraham, John, et al. \"Generative models for graph-based protein design.\" *Advances in neural information processing systems* 32 (2019).\n\n[2]: Orengo, Christine A., et al. \"CATH–a hierarchic classification of protein domain structures.\" *Structure* 5.8 (1997): 1093-1109."}, {"review_id": "CVQMMXQUCS", "reviewer": "Reviewer_tPSY", "summary": "This work applies Direct Preference Optimization (DPO) to the inverse folding task, leveraging TM-score as a feedback signal to enhance the foldability of designed protein sequences. The authors conducted comprehensive experiments to validate the effectiveness of the DPO strategy. In addition, they provide insightful analyses of the model, including its scaling behavior, an illustrative case study, and the impact of iterative training.", "questions": "See the Strengths and Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written, with a comprehensive introduction, clear preliminaries, and detailed methodology. The authors also conducted in-depth analyses to evaluate and interpret the model. However, there are several minor concerns regarding the experimental section:\n\n1. Lines 149–152 provide only a brief description of the application of DPO to ProteinMPNN. It would be helpful if the authors could elaborate further on this part for better clarity.\n\n2. In line 189, it is unclear whether the authors explored other values for the hyperparameter λ, such as 0.1 or 0.01. Including such experiments would strengthen the analysis.\n\n3. In Section 4.5, the results are reported only on ten test samples. It is recommended that the authors report performance metrics on the entire test set to provide a more comprehensive evaluation.\n\n4. Since the main contribution of the paper is the application of DPO to inverse folding, demonstrating its effectiveness on additional baseline models would further substantiate the generalizability and robustness of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and encouraging feedback. We are glad to hear that the paper is found to be well-written and the analyses insightful. We also appreciate the constructive suggestions for improving the clarity and completeness of our experiments. Below, we address the reviewer’s comments and concerns in detail.\n\n**Weakness 1:** To train the ProteinMPNN model, we slightly changed our dataset curation process and constructed a simpler task. Specifically, we set a new threshold t_r = 0.8 to control the process of classifying samples into chosen and rejected. Instead of training on all predictions like InstructPLM, we only trained ProteinMPNN on the predictions with a TM-Score lower than t_r . Meanwhile, in order to increase the diversity of the chosen sequences, the wild-type sequence has also been introduced as a chosen sample during training. Details can be found in the **Appendix A.1.4.** This result highlights an important insight: applying DPO effectively may require more powerful and expressive sequence design models. ProteinMPNN, while efficient and widely used, may lack sufficient capacity to fully benefit from DPO-style fine-tuning. We hope our findings can inspire the development of stronger structure-conditioned design algorithms that are better suited for preference-based optimization techniques like DPO.\n\n**Weakness 2:** We thank the reviewer for pointing out the importance of hyperparameter sensitivity analysis. We have conducted additional experiments varying the λ value and summarize the results (TM-Scores) below:\n\n|          | Short &#8593; | Single-Chain &#8593; | All &#8593; | TM-Score < 0.5 &#8593; | TM-Score > 0.5 &#8593; |\n| -------- | ------------- | -------------------- | ----------- | ---------------------- | ---------------------- |\n| λ = 10   | 0.57          | 0.61                 | 0.80        | 0.43                   | 0.88                   |\n| λ = 1    | 0.58          | 0.63                 | 0.81        | 0.45                   | 0.89                   |\n| λ = 0.1  | 0.58          | 0.62                 | 0.81        | 0.45                   | 0.88                   |\n| λ = 0.01 | 0.57          | 0.62                 | 0.81        | 0.44                   | 0.88                   |\n| λ = 0    | 0.57          | 0.62                 | 0.81        | 0.44                   | 0.88                   |\n\nThese results suggest that the model is relatively robust to λ values in a wide range. While λ = 1 yields slightly better performance overall, especially on challenging (TM-Score < 0.5) cases, smaller values like 0.1 or 0.01 also perform comparably well. We will include these findings in the revised manuscript to strengthen our analysis.\n\n**Weakness 3:** We thank the reviewer for this suggestion. In Section 4.5, we initially selected 10 samples to qualitatively examine whether multi-round DPO can help the model successfully design structures that it previously **failed** to recover, focusing on cases where improvement is most meaningful. These samples were chosen based on relatively low baseline performance to better illustrate the effect of iterative optimization. We agree that evaluating on a larger and randomly selected subset provides a more comprehensive view. To this end, we extended the multi-round DPO experiment to 100 randomly sampled proteins from the CATH 4.2 test set. The results are summarized below:\n\n|                  | Round 0 | Round 2 | Round 4 | Round 6 | Round 8 | Round 10 |\n| ---------------- | ------- | ------- | ------- | ------- | ------- | -------- |\n| TM-Score &#8593; | 0.72    | 0.75    | 0.77    | 0.78    | 0.77    | 0.77     |\n| RMSD &#8595;     | 2.56    | 2.39    | 2.23    | 2.19    | 2.21    | 2.23     |\n\nWe observe consistent improvement in both TM-Score and RMSD during the early rounds, indicating that multi-round DPO can generalize its benefits beyond the small and hard examples. However, the performance appears to saturate in the later rounds. This indicates that multi-round DPO is particularly effective in challenging cases with low initial performance, while for easier or already well-predicted samples, the benefit plateaus more quickly. We will incorporate these results into the revised version to strengthen our empirical analysis.\n\n**Weakness 4:** We agree that evaluating additional baseline models would further demonstrate the generalizability of our approach. However, due to time and resource constraints during the rebuttal period, we were unable to extend our experiments to new baseline architectures. Notably, our current results already cover a diverse range of datasets and evaluation settings, including challenging OOD structures from CATH 4.3, which collectively demonstrate the robustness and structural awareness gained through DPO training. We consider this a promising direction for future work."}, {"review_id": "ZItZkR5Dtm", "reviewer": "Reviewer_xi2h", "summary": "The paper proposes to use Direct Preference Optimization to fine tune existing models that perform protein design. That is, given a target structure, the idea is to hypothesize sequences that could fold into that structure using a pretrained model (they use proteinMPNN and InstructPLM). These sequences are folded (in silico) and then an alignment score is used to fine-tune the model used to generate hypothetical sequences. The effects of this on recovery rate, perplexity and structural similarity are studied.", "questions": "1) How do the improvements look across types of folds (e.g.see R2)?\n\n2) How was the data for Fig 3 generated? Is that a specific protein, or averages? In general, it appears that as RMSD drops the recovery rate increases– that is de novo exploration reduces. \n\n3) What is the scale of computation involved? E.g. per DPO iteration of each model (for different numbers of samples).", "limitations": "Yes, but the limitations are quite generic.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strength of the paper is the relatively simple application to existing models to achieve significant improvements in structural similarity. Indeed, as highlighted, recovery rates are less important when the task is to design de novo proteins. Yet, it is good to see that the method maintains relatively healthy recovery rates as well as perplexity, while improving structural similarity. However, is unclear what the associated computational cost is. It is suggested to include and discuss the following references:\n\n[1] Improving Protein Sequence Design through Designability Preference Optimization (https://arxiv.org/html/2506.00297v1)\n\n[2] PDBench: evaluating computational methods for protein-sequence design (https://academic.oup.com/bioinformatics/article/39/1/btad027/6986968)\n\n[3] Fine-tuning protein language models boosts predictions across diverse tasks\n(https://www.biorxiv.org/content/10.1101/2023.12.13.571462v1)", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and for highlighting the strengths of our work. We agree that recovery rate is not the most critical metric for de novo protein design, and we appreciate the recognition that our method improves structural similarity while maintaining reasonable recovery and perplexity levels. We also appreciate the suggested references. These works are indeed highly relevant to our study:\n\n[1] presents a concurrent approach applying DPO fine-tuning to protein language models, which aligns closely with our motivation and methodology.\n\n[2] emphasizes evaluating PLMs beyond recovery rate, which resonates with our view on the limitations of recovery as a sole metric.\n\n[3] explores parameter-efficient fine-tuning of PLMs across diverse tasks. \n\nWe will include and discuss these works in the revised manuscript to better situate our contributions within the context of recent advances. Below, we address the reviewer’s questions and suggestions in detail:\n\n**Question 1**: We appreciate the reviewer’s interest in understanding how our method performs across different fold types. To this end, we conducted an additional experiment where we grouped the test structures in CATH 4.2 according to their **Class** annotations (*Mainly Alpha*, *Mainly Beta*, *Alpha-Beta*, and *Few Secondary Structures*). The average TM-scores (mean and standard deviation) before and after DPO fine-tuning are summarized as follows:\n\n|            | Mainly Alpha &#8593; | Mainly Beta &#8593; | Alpha Beta &#8593; | Few Secondary Structures &#8593; |\n| ---------- | ------------ | ----------- | ----------- | ------------------------ |\n| Baseline   | 0.61 (0.08)  | 0.76 (0.05) | 0.84 (0.04) | 0.63 (0.05)              |\n| Fine-tuned | 0.66 (0.06)  | 0.80 (0.04) | 0.86 (0.02) | 0.65 (0.03)              |\n\nAs shown, DPO fine-tuning brings consistent improvements across all fold types, with especially notable gains in the *Mainly Alpha* and *Mainly Beta* categories. This demonstrates that our method not only improves structural quality overall, but does so robustly across a diverse range of protein topologies.\n\n**Question 2:** All the results in Fig. 3 are obtained from the CATH 4.2 test set, which contains 1,120 test structures that do not share the same Fold with the training set. For each structure, we sampled 10 sequences using both the baseline and fine-tuned models, and then computed the mean value of each metric (RMSD, TM-score, and recovery rate.) across the 10 sequences. Finally, the plotted points in Fig. 3 represent the averages over all 1,120 structures.  We would also like to clarify that the primary purpose of Fig. 3 is to analyze the scaling behavior of DPO with respect to different training configurations, including the number of training steps and training pairs. Our experiments reveal an interesting trend: while moderate amounts of preference pairs improve structural quality, excessive numbers of training pairs can actually degrade performance. This suggests that DPO benefits from high-quality, but not necessarily large-scale, feedback data, and highlights the importance of balancing signal strength with data quantity in preference-based optimization. For design sequences with high structure similarity but with low sequence recovery rate, we refer the reviewer to check our results on multi-round DPO in **Fig. 6**, which shows our methods achieve a higher TM-Score (0.7) but maintain low recovery (0.25).\n\n**Question 3:** The majority of the computational cost in our pipeline is concentrated in the **data construction phase**, particularly during sequence generation and structure prediction (folding). In contrast, the actual DPO fine-tuning stage is relatively lightweight. In our setting, a complete DPO training run takes approximately 16 hours on a server with 8×A100 GPUs. To promote future research, we will also release our full training dataset upon publication, which includes over 1.8 million sequences and their predicted structures generated by ESMFold."}, {"review_id": "nCTTsHk5OO", "reviewer": "Reviewer_k5nB", "summary": "This paper introduces a novel framework for optimizing protein inverse folding models using Direct Preference Optimization (DPO) guided by structural feedback from a folding model. Given a target structure, the model samples multiple candidate sequences, predicts their folded structures, and scores them via TM-Align. Based on the TM-Score, sequences are classified into \"chosen\" and \"rejected\" and used to fine-tune the inverse folding model under a DPO objective. Experiments are done on the CATH 4.2, TS50, and TS500 datasets demonstrate improvements in both sequence recovery and TM-Score. The authors further investigate the effect of training scale, the number of contrastive samples, and iterative refinement via multi-round DPO.", "questions": "1. Figure 4 shows higher median TM-Scores for the fine-tuned model, but the heavy lower-tail suggests potential instability or skewness. Could the authors report statistical significance (e.g., Student's t-test), add distribution visualizations (e.g., violin/KDE plots), and clarify whether improvements are consistent across targets or driven by a few outliers? This would strengthen the claim of robust and meaningful improvement.\n\n2. Why did you choose CATH 4.2 instead of the latest CATH 4.3? Would the model's performance be consistent if evaluated on the newer dataset? Please clarify if CATH 4.3 was excluded for compatibility, coverage, or reproducibility reasons.\n\n3. Can you clarify the novelty of your method over prior DPO work? Are you the first to incorporate structural feedback into DPO for protein design? If not, how does your framework advance beyond existing works in terms of scale, architecture, or applicability?", "limitations": "Yes. There is a paragraph titled “Limitations”.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "*Strength*\n\n-The authors evaluate their approach across multiple datasets (CATH 4.2, TS50, TS500), model architectures (ProteinMPNN, InstructPLM), and settings (single-round vs. multi-round, λ ablation), providing empirical evidence for robustness.\n\n-The pipeline is entirely in silico and can be extended iteratively, making it well-suited for real-world protein design where experimental structures may be limited or unavailable.\n\n-The use of equations, diagrams, and concrete implementation details make the method easy to follow and potentially reproducible.\n\n\n*Weakness*\n\n1. Several claims of \"significant\" improvement (e.g., Lines 251, 295) are made without statistical testing. In Figure 4, no significance tests are reported to validate the robustness of observed gains.\n2. The evaluation of contrastive sample sizes in Figure 3 is based on only four discrete points (0, 20, 40, 80), which obscures potential non-linear effects or turning points in TM-Score and RMSD. This makes it difficult to draw reliable conclusions about how contrastive sample quantity affects structural fidelity.\n3. TM-Score is used as the sole optimization and evaluation metric, yet structural similarity does not always correlate with functional correctness. Without assessing alternative measures (e.g., energy, function, or diversity), there is a risk of optimizing toward biologically irrelevant sequences.\n4. Unaddressed Risks in Multi-Round DPO Training\n The iterative nature of multi-round DPO could amplify early-stage model biases or propagate errors. However, the paper does not discuss any safeguards such as validation-based early stopping, diversity checks, or convergence criteria to mitigate this risk.\n5. While related work sections mention prior DPO-based efforts in enzyme and peptide design, the manuscript lacks a clear articulation of what makes structure-feedback-based protein design novel and technically distinct. The gap between this work and existing methods remains underexplored.\n6. Minor presentation issues:\n   - Table 1 lacks clarity on whether values are percentages.\n   - Formatting choices (bold for fine-tuned models, underline for baselines) deviate from conventions and may confuse readers.\n   - Line 169 uses \"ESMfold\" instead of the correct “ESMFold.”", "was_revised": true, "final_justification": "The rebuttal addressed several of my main concerns. Statistical testing and finer-grained sampling analyses improved the rigor of the results, while the additional CATH 4.3 evaluation strengthened claims of generalization. Clarifications on novelty over prior DPO work also made the contribution clearer. Some issues remain—such as per-target consistency, biological relevance of TM-score optimization, and risk handling in multi-round DPO—but overall the paper is now better supported. I am raising my score to 4: Weak accept.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback. We greatly appreciate the recognition of the strengths of our work, including its robustness, extensibility, and clarity of presentation. At the same time, we acknowledge the concerns raised regarding certain methodological and evaluative aspects. Below, we provide point-by-point responses to the weaknesses and questions.\n\n**Weakness 1, Question 1:** We thank the reviewer for the valuable suggestion regarding statistical rigor and visualization. As the reviewer noted, **Fig. 4** is presented as a **boxplot**, which includes the 25th percentile, median, and 75th percentile, providing a summary of the distribution and capturing potential skewness and variability in performance. To further address the reviewer’s concern, we perform t-test on the original data of Fig. 4:\n\n| t-stat | P-value  |\n| ------ | -------- |\n| -57.25 | < 1e-100 |\n\nWe also have conducted an additional evaluation using **AlphaFold 3** to assess the robustness of our improvements across different structure predictors. Specifically, for each test structure where multi-round DPO was applied, we folded 10 designed sequences using AlphaFold 3, resulting in 200 sequences in total (100 from the baseline and 100 from the fine-tuned model). We report the average TM-Score and RMSD per structure, along with standard deviations, in the table below:\n\n|            | ESMFold-TMScore &#8593; | ESMFold-RMSD &#8595; | AlphaFold3-TMScore &#8593; | AlphaFold3-RMSD &#8595; |\n| ---------- | --------------- | ------------ | ------------------ | --------------- |\n| Baseline   | 0.40 (0.09)     | 3.42 (0.69)  | 0.43 (0.12)        | 3.09 (0.73)     |\n| Fine-tuned | 0.70 (0.18)     | 2.25 (0.89)  | 0.68 (0.17)        | 2.29 (0.83)     |\n\nWe believe these results clearly reflect the significant performance improvements of our methods.\n\n**Weakness 2:** We conducted additional experiments with smaller increments in the number of contrastive samples (0, 10, 20, 30, 40, 80). The results show that performance improves up to around 20 contrastive samples, after which it begins to plateau or slightly degrade. Specifically, both TM-score and RMSD indicate that using 20 samples yields the best balance between structural accuracy and stability, confirming the trend observed in our original submission. These findings support our original choice and reinforce the importance of moderate contrastive sampling in DPO training.\n\n| Steps    | 0     | 10    | 20    | 30    | 40    | 80    |\n| -------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Recovery &#8593; | 53.48 | 55.09 | 55.21 | 55.04 | 55.15 | 55.18 |\n| TM-Score &#8593; | 0.78  | 0.80  | 0.81  | 0.81  | 0.78  | 0.78  |\n| RMSD &#8595;    | 2.11  | 1.98  | 1.97  | 1.98  | 2.08  | 2.08  |\n\n**Weakness 3:** We appreciate the reviewer’s concern regarding the biological relevance of the optimization target. We fully agree that incorporating functional and energetic criteria is crucial for real-world protein design. However, our primary goal in this work is to demonstrate that protein design algorithms can be effectively optimized *entirely in silico*, without relying on wild-type sequences or wet-lab supervision. To do so, we intentionally focus on TM-score as a well-established structural similarity metric, which provides a consistent and objective signal that can be computed automatically at scale. We view our approach as a proof of concept showing that feedback from structure prediction models alone can significantly improve inverse folding performance. We hope that our work can inspire future research to extend this framework by incorporating more biologically grounded objectives and stimulating the development of more accurate biological tools in the community.\n\n**Weakness 4:** We thank the reviewer for raising this important concern about potential risk amplification in multi-round DPO training. Indeed, recent work such as Ren et al. (2024) [1] has highlighted how iterative optimization can reinforce early-stage biases or drift from desired distributions. We will explicitly discuss this risk in the Limitations section of our revised manuscript. In our experiments, we report TM-Scores and Recovery throughout the multi-round training process to ensure that the model continues to learn meaningful structural features; please refer to **Fig. 6** for details. However, we would also like to emphasize that the primary contribution of our work is to demonstrate that fully in silico fine-tuning of protein language models using no external annotations or experimental data is feasible and can lead to significant performance gains. Our focus is thus on exploring and validating the possibility of DPO in protein design, rather than fully solving the challenge of safe or robust reinforcement-based learning in this domain. We leave the exploration of more robust and effective training frameworks for future work.\n\n**Weakness 5, Question 3:** We thank the reviewer for the thoughtful question regarding the novelty of our method compared to existing DPO-based approaches in protein design. First, we would like to emphasize that **all our results are obtained solely based on the model’s own predictions**, without relying on any external supervision such as wild-type sequences, evolutionary constraints, or wet-lab annotations. This contrasts with existing methods such as ProteinDPO[2] and ProtGPT2 (DPO)[3], which leverage annotated or curated datasets during training. Our framework instead demonstrates that protein language models can be improved entirely in silico by leveraging structure predictors as feedback, making the fine-tuning pipeline significantly more scalable and self-sufficient. Second, while there are a few concurrent works that also incorporate predicted structural feedback (e.g., using ESMFold) into the DPO process, these studies are typically limited in scope and scale. For instance, Stocco et al. focus only on a single enzyme [4], and Park et al. apply DPO to peptide sequences[5]. In contrast, our work performs DPO on over 1.8M predicted sequences and evaluates the fine-tuned model across diverse structural folds, demonstrating generalizable improvement beyond a narrow domain or sequence type. Third, our study also provides initial insights and foundations for designing larger-scale, fully in silico protein design frameworks by studying the scaling behavior of DPO. We hope this work can serve as a stepping stone toward more robust and autonomous pipelines in future research.\n\nWe will further clarify these points in the revised version and highlight them in the Introduction and Related Work sections.\n\n**Weakness 6:** We thank the reviewer for pointing out these presentation issues; we will clarify in the revised version. However, in terms of the Table. 1, since our framework is model-agnostic, we aimed to emphasize the performance gains achieved by our fine-tuned models relative to their respective baselines, rather than compare across architectures.\n\n**Question 2:** We chose CATH 4.2 primarily for two reasons. First, our baseline models were trained on CATH 4.2, and to avoid any potential data leakage, we did not include CATH 4.3 structures in the training set. Using the same version ensures a fair and controlled comparison. Second, to better evaluate the generalization ability of our method, we conducted experiments on an out-of-distribution (OOD) set constructed from the CATH 4.3 release. Specifically, we selected protein domains from newly added **Classes** in CATH 4.3 that are not present in CATH 4.2, ensuring no overlap with the training data *not* just at the **Fold** level, but at the broader **Class** level. After filtering out sequences longer than 512 residues, we obtained a test set comprising 750 diverse and challenging structures. Compared to the original CATH 4.2 test set, which differs from training data at the Topology (Fold) level, this new dataset represents a more stringent test of generalization. The results show that our fine-tuned model consistently outperforms the baseline across all metrics. This performance gap is especially notable given the increased structural diversity and novelty in the CATH 4.3 set. \n\n| CATH 4.3 new Classes | Recovery &#8593; | TM-Score &#8593; | RMSD &#8595; |\n| -------------------- | -------- | -------- | ---- |\n| Baseline             | 41.85    | 0.57     | 2.89 |\n| Fine-tuned           | 43.62    | 0.60     | 2.79 |\n\n[1]: Ren, Yi, et al. \"Bias amplification in language model evolution: An iterated learning perspective.\" *Advances in Neural Information Processing Systems* 37 (2024): 38629-38664.\n\n[2]: Widatalla, Talal, Rafael Rafailov, and Brian Hie. \"Aligning protein generative models with experimental fitness via direct preference optimization.\" bioRxiv (2024): 2024-05.\n\n[3]: Mistani, Pouria, and Venkatesh Mysore. \"Preference optimization of protein language models as a multi-objective binder design paradigm.\" arXiv preprint arXiv:2403.04187 (2024).\n\n[4]: Stocco, Filippo, et al. \"Guiding generative protein language models with reinforcement learning.\" arXiv preprint arXiv:2412.12979 (2024).\n\n[5]: Park, Ryan, et al. \"Improving inverse folding for peptide design with diversity-regularized direct preference optimization.\" arXiv preprint arXiv:2410.19471 (2024)."}]

Author rebuttal:

Meta-review: This paper develops an improved protein inverse folding model using DPO guided by structural feedback from a folding model (TM-Score).

Strength
-The authors evaluate their approach across multiple datasets (CATH 4.2, TS50, TS500) and show strong  empirical evidence for robustness.
- Multi-round DPO refinement shows strong gains, particularly on challenging protein structures.
- Fully in silico pipeline avoids reliance on experimental data, supporting scalability.

Weakness
- Concern about the use of TM-Score as reward and evaluation metric. 
- Potential bias introduced and amplified with the iterative training. 
- The method itself is limited in technical novelty.
Decision: Accept (poster)