ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning

## Abstract

Abstract Protein generative models have shown remarkable promise in protein design but still face limitations in success rate, due to the scarcity of high-quality protein datasets for supervised pretraining. We present ProteinZero, a novel framework that enables scalable, automated, and continuous self-improvement of the inverse folding model through online reinforcement learning. To achieve computationally tractable online feedback, we introduce efficient proxy reward models based on ESM-fold and a novel rapid ddG predictor that significantly accelerates evaluation speed. ProteinZero employs a general RL framework balancing multi-reward maximization, KL-divergence from a reference model, and a novel protein-embedding level diversity regularization that prevents mode collapse while promoting higher sequence diversity. Through extensive experiments, we demonstrate that ProteinZero substantially outperforms existing methods across every key metric in protein design, achieving significant improvements in structural accuracy, designability, thermodynamic stability, and sequence diversity. Most impressively, ProteinZero reduces design failure rates by approximately 36% - 48% compared to widely-used methods like ProteinMPNN, ESM-IF and InstructPLM, consistently achieving success rates exceeding 90% across diverse and complex protein folds. Notably, the entire RL run on CATH-4.3 can be done with a single 8 × \times × GPU node in under 3 days, including reward computation. Our work establishes a new paradigm for protein design where models evolve continuously from their own generated outputs, opening new possibilities for exploring the vast protein design space.

## 1 Introduction

Protein design and engineering represent one of the most promising frontiers in computational biology, with applications spanning from drug discovery to developing novel enzymes (proteinmpnn,; esm_if,; StructTrans,). A central challenge in this field is protein inverse folding—generating amino acid sequences that will fold into desired three-dimensional structures (GVP_GNN,; zhang2005tm,), which serves as the foundation for fixed backbone sequence design. This task is particularly crucial as protein backbone structure and side-chain conformation jointly determine functionalities such as binding and catalytic interactions.

Recent deep learning approaches such as ProteinMPNN (proteinmpnn,), ESM-IF (esm_if,), and graph-based methods (GVP_GNN,; StructTrans,) have significantly improved inverse folding accuracy. However, current state-of-the-art methods are trained with paired sequence-structure data from the Protein Data Bank (PDB) which, while valuable, represent only a tiny fraction of the vast protein sequence space (proteinmpnn,; esm_if,; instruct_plm,) and are subject to limited diversity and natural biases. This data scarcity creates a ceiling for model performance and limits the exploration of novel protein designs beyond what exists in nature or laboratory settings (offline_3,; ai_collapse_nature,). Moreover, there is a misalignment between the supervised learning task of inverse folding and the actual objectives in real-world protein design, where high designability, thermal stability, and diversity are desired (rfdiffusion,; chroma,). There is limited effort in addressing this alignment task, where existing work has primarily focused on RL-finetuning structural generative models (Multiflow,; foldflow2,; abdpo,; capempnn,; iclr_dpo_fail,) and has only managed to do single- or few-round alignment with carefully curated offline designability datasets, failing to unlock the potential in self-evolving protein design models with their own generated outputs.

We propose ProteinZero, a highly scalable online RL finetuning framework that, for the first time, enables fully automated, multi-round and stable self-improvement of the inverse folding model, and effectively optimizes multiple key objectives in protein sequence design, including designability, thermal stability, and sequence diversity while improving the inverse folding accuracy. The contributions of our work are as follows:

- 1.
We develop ProteinZero, the first successful and practical online reinforcement learning framework for automated self-improvement of protein sequence design models without reliance on carefully curated preference datasets.
- 2.
We propose a fast thermodynamic stability ddG estimator based on inverse folding sequence likelihoods and unconditional sequence priors. Combining it with lightweight designability rewards provided by ESMFold, we rapidly reduce the evaluation cost (Table [1](https://arxiv.org/html/2506.07459v2#S2.T1)), making online RL fine-tuning feasible for protein design for the first time.
- 3.
We develop a novel diversity-promoting regularizer based on protein embeddings that effectively prevents mode collapse (model_collapse,; ai_collapse_nature,; collapse2,; collapse3,) and improves the generated sequence diversity, a desired property for protein design.
- 4.
We elucidate the design space of online RL fine-tuning frameworks by examining different algorithms (GRPO, RAFT, DPO), rewards, divergence, and diversity regularization, and develop the best strategy that stably optimizes multiple objectives without mode collapse.
- 5.
Through extensive experiments on inverse folding tasks, we demonstrate that ProteinZero substantially outperforms existing methods across every key metric, achieving significant improvements in structural accuracy, thermodynamic stability, and sequence diversity while reducing design failure rates by approximately 36% - 48% compared to widely-used methods like ProteinMPNN (proteinmpnn,), ESM-IF (esm_if,) and InstructPLM (instruct_plm,). Notably, our model is able to design better sequences across diverse and complex protein folds, and significantly improves the success rate on long-chain proteins.

## 2 Related Work

Protein Inverse Folding Models.
Inverse protein folding has been transformed by deep learning approaches that generate amino acid sequences for target structures. ProteinMPNN (proteinmpnn,) pioneered message passing neural networks for this task, while ESM-IF (esm_if,) leveraged pretrained protein language models to enhance structure-conditioned sequence generation. Other notable approaches include GVP-GNN (GVP_GNN,), which incorporated geometric vector perceptrons, and StructTrans (StructTrans,), which employed specialized transformer architectures. Recently, InstructPLM (instruct_plm,) established a new state-of-the-art performance across diverse inverse folding benchmarks—we therefore adopt this model as our base architecture for fine-tuning. However, despite these advances, current models remain constrained by their reliance on fixed offline datasets, limiting exploration of novel sequence space. While they perform well on benchmark tasks, they lack mechanisms to learn from their own generations due to the prohibitive computational costs of quality assessment, creating a fundamental bottleneck for continuous improvement over multi-objectives.

**Table 1: Total wall-clock time (including MSA and template search) required to generate reward for eight inverse-folding sequences conditioned on the same structural backbone. Best results are highlighted in blue. The wall-clock time without MSA search can be found in Appx. Table [4](https://arxiv.org/html/2506.07459v2#A2.T4)**
| Length range | Structural and Designability Reward | Thermal Stability Reward (ddG) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | ESMFold | AlphaFold 2 | ColabFold | OpenFold | AlphaFold 3 | Pred-ddG(ours) | FoldX |
| 0–150 aa | 18.7 s | 1632.6 s ($\sim$87.3$\times$) | 576.2 s ($\sim$30.8$\times$) | 674.9 s ($\sim$36.1$\times$) | 705.4 s ($\sim$37.7$\times$) | $\sim$2 s (GPU) | 472.3 s ($\sim$236.2$\times$) |
| 150–300 aa | 47.5 s | 4112.5 s ($\sim$86.6$\times$) | 1272.8 s ($\sim$26.8$\times$) | 1424.7 s ($\sim$30.0$\times$) | 1920.5 s ($\sim$40.4$\times$) | $\sim$2 s (GPU) | 1520.6 s ($\sim$760.3$\times$) |

RLHF of Generative Models. Reinforcement Learning from Human Feedback (RLHF) has transformed generative models through two main approaches: online methods like PPO (ppo,), GRPO (grpo,), and RAFT (Raft,) that learn through direct reward optimization, and offline methods like DPO (dpo,) that learn from static preference datasets. While successful in LLMs, applying online RLHF to protein models has been limited due to computational costs (See Tab. [1](https://arxiv.org/html/2506.07459v2#S2.T1)). Existing protein RLHF work primarily relies on offline methods—Multiflow (Multiflow,) and Foldflow2 (foldflow2,) improve structural generation with ReFT but require curated datasets, while AbDPO (abdpo,) applies DPO for antibody design but is limited to single-round optimization. Previous work has mostly focused on improving structure generative models or co-design models, with limited effort on improving inverse folding for better alignment with design objectives. These approaches have critical limitations: they typically optimize a single objective, require extensive offline curated data, and cannot continuously improve from their own outputs—precisely the capabilities needed to explore protein design space while simultaneously optimizing stability, designability, and diversity.

Diversity Regularization. Promoting diversity in protein generative models is crucial for both increasing downstream success rate and maintaining exploration capability in online RL to avoid mode collapse and reward hacking (instructed_rlhf,; ORW-CFM-W2,; model_collapse,). (iclr_dpo_fail,) uses sequence-level Hamming distance as regularization (hanming,), which often compromises functional properties by encouraging arbitrary amino acid variations. Their DPO-based approach struggles to balance reward and diversity. Our embedding-level diversity regularization operates in the model’s representation space rather than on sequences directly, enabling functionally diverse proteins that maintain structural integrity without training instabilities.

## 3 Method

### 3.1 Problem Formulation

Protein Inverse Folding Models. Protein inverse folding aims to generate amino acid sequences that will fold into a specified three-dimensional protein structure. Formally, given a target structure $x$ represented as a set of 3D coordinates or backbone topology, the goal is to generate a sequence $y=(y_{1},y_{2},...,y_{L})$ where each $y_{i}$ is one of 20 possible amino acids and $L$ is the sequence length. Current SOTA protein inverse folding models (instruct_plm,; proteinmpnn,; esm_if,) typically formulate this as a conditional generation task, where a model $p_{\theta}(y|x)$ is trained to maximize the likelihood of ground-truth sequences given their corresponding structures: The conditional cross-entropy $\mathcal{L}_{\mathrm{CE}}(\theta;\mathcal{D})=-\mathbb{E}_{(x,y)\sim\mathcal{D
}}[\log p_{\theta}(y|x)]$, where $\mathcal{D}$ represents a dataset of paired structure-sequence examples, typically from the PDB. While supervised learning has shown promising results, these models face two critical limitations: 1) The reliance on limited supervised datasets restricts exploration of the vast protein sequence space, and 2) many tasks lack enough high quality data which requires massive time cost.

RL Fine-tuning Objective. To address these limitations, we propose ProteinZero, a framework that fine-tunes protein generative models through online reinforcement learning. Unlike standard supervised fine-tuning, our approach optimizes a reward-based objective $\mathcal{J}_{\mathrm{RL}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x},y\sim p_{
\theta}(\cdot\mid x)}[r(x,y)]-\alpha_{\mathrm{KL}}\cdot\mathrm{KL}\left(p_{
\theta}(\cdot\mid x)\|p_{\mathrm{ref}}(\cdot\mid x)\right)$, where $r(x,y)$ is a reward function evaluating the quality of generated sequences, $p_{\mathrm{ref}}$ is a reference model (typically the pre-trained model before fine-tuning), and $\alpha_{\mathrm{KL}}$ is a coefficient controlling the divergence from the reference model. This KL term prevents the fine-tuned model from deviating too far from the reference model, preserving its pre-trained knowledge while exploring new sequence spaces guided by rewards.

### 3.2 Fine-tuning Protein Generative Models via Online RL

Building upon our RL fine-tuning objective, the ProteinZero framework implements a comprehensive approach to protein design through online reinforcement learning. The general framework combines reward optimization with constraints to ensure stable and effective learning.

Reinforcement learning for protein design requires optimizing a model to generate sequences that maximize a reward function while maintaining reasonable proximity to a reference model. In practice, recent RL fine-tuning methods can be unified in this general objective through specialized algorithms that balance exploitation and exploration: $\mathcal{L}(\theta)=\mathcal{L}_{\mathrm{RL}}(\theta)+\mathcal{L}_{\mathrm{KL}
}(\theta)$. For instance, in the Group Relative Policy Optimization (GRPO) algorithm, this objective is realized as $\mathcal{L}_{\mathrm{GRPO}}(\theta)=-\mathbb{E}_{(x,y)\sim\mathcal{B}}\left[
\operatorname{min(A*\frac{p_{\theta}}{p_{\theta_{old}}},A*\operatorname{clip}(
\frac{p_{\theta}}{p_{\theta_{old}}},1-\epsilon,1+\epsilon))}\right]+\alpha_{
\mathrm{KL}}\cdot\mathrm{KL}\left(p_{\theta}\|p_{\mathrm{ref}}\right)$ (grpo,), where $A$ is the advantage function (grpo,), $p_{\theta_{old}}$ is learned policy of last iteration.

However, we observed that protein generative models suffer from mode collapse in online RL fine-tuning, converging to a narrow set of solutions that maximize rewards without diversity. To address this, we incorporate a diversity regularization term, resulting in a more comprehensive objective:

$$ $\mathcal{L}(\theta)=\mathcal{L}_{\mathrm{RL}}(\theta)+\mathcal{L}_{\mathrm{KL} }(\theta)+\mathcal{L}_{\text{Div }}(\theta)$ (1) $$

While diversity can be promoted by directly incorporating it into the reward function, our experiments showed that this approach often leads to training instability and performance degradation (See Tab. [3](https://arxiv.org/html/2506.07459v2#S4.T3)). Instead, our ProteinZero framework implements diversity regularization at the representation level through a separate loss term $\mathcal{L}_{\text{Div }}(\theta)$, which encourages diversity while preserving the integrity of the main reward optimization (See Tab. [2](https://arxiv.org/html/2506.07459v2#S4.T2), and Fig. [1](https://arxiv.org/html/2506.07459v2#S4.F1)).

However, no effective diversity regularization exists in protein generative model fine-tuning, and reward computation for each data pair $(x,y)$ typically takes minutes, potentially extending online training to months. Therefore, in this paper, we propose an embedding-level Diversity Regularization and a fast online reward modeling to enable online fine-tuning of inverse folding models.

#### 3.2.1 Diversity Regularization Based on Protein Embeddings

To address mode collapse in protein generative models during online RL fine-tuning, we propose a novel diversity regularization approach that operates at the protein embedding level. Unlike token-level diversity metrics which can compromise functional properties, our approach captures semantic diversity while maintaining structural integrity.

For each protein sequence in a batch, we compute a fixed-dimensional embedding vector by aggregating the last-layer decoder activations:

$$ $z_{i}(\theta)=\frac{\sum_{t}m_{i,t}h_{i,t}}{\sum_{t}m_{i,t}}\in\mathbb{R}^{d}, \quad 1\leq i\leq B$ (2) $$

where $h_{i,t}\in\mathbb{R}^{d}$ is the decoder activation at position $t$ for sample $i$, and $m_{i,t}\in\{0,1\}$ is an attention mask. These protein embeddings are $\ell_{2}$-normalized before computing a cosine-based diversity score: $D_{\cos}(\theta;\mathcal{B})=1-\overline{\cos}\in[0,1]$ , wherein $\overline{\cos}=\frac{2}{B(B-1)}\sum_{1\leq i<j\leq B}\cos\left(z_{i},z_{j}\right)$. The diversity regularization term is incorporated as:

$$ $\mathcal{L}_{\operatorname{Div}}(\theta)=-\alpha_{\operatorname{div}}\cdot D_{ \cos}(\theta;\mathcal{B})$ (3) $$

Since $z_{i}$ depends on $\theta$, the term supplies informative gradients that foster the generation of diverse, functionally plausible sequences. As shown in Table [3](https://arxiv.org/html/2506.07459v2#S4.T3), this embedding-level approach outperforms direct reward-based diversity methods, enabling effective online fine-tuning.

#### 3.2.2 Time-Efficient Reward Models

Traditional methods for evaluating protein designs require minutes to hours per evaluation, making online RL impractical. We solve this challenge with two efficient proxy reward models:

Designability Reward: We use ESM-fold (esm_if,) for structural inference instead of the slower AlphaFold2/3 (af2,; af3,). The TM-score reward $r_{\mathrm{TM}}(x,y)$ is computed by first folding the generated sequence $y$ using ESM-fold, then calculating the TM-score (tm_score,) between the predicted structure and the target structure $x$ with US-align (zhang2022us,), an updated version of TM-align (zhang2005tm,), which evaluates the length-normalized sum of distance-weighted Cα pair overlaps to quantify structural similarity.

Thermal Stability Reward:
We propose a novel stability reward $r_{\Delta\!\Delta G}(x,y)$ that serves as a backbone-specific folding-energy surrogate for single-chain proteins, referenced to the PDB wild-type sequence. Because our monomeric setting lacks an inter-chain interface, the unbound-state term required by the Boltzmann-aligned estimator (BA-DDG) (ddg_pre,) cannot be evaluated. Instead, drawing on evidence that backbone-conditioned likelihoods reflect folding stability (science_evo,; widatalla2024aligning,; cagiada2025predicting,; bennett2023improving,; zheng2023structure,; ingraham2019generative,), we normalize this likelihood with an unconditional sequence prior and anchor it to the wild-type baseline:

$$ $\Delta\!\Delta G(x,y)=-\,k_{B}T[(\log p_{\theta}(y\mid x)-\log p_{\varphi}(y)) -(\log p_{\theta}(y_{\text{wt}}\mid x)-\log p_{\varphi}(y_{\text{wt}}))],$ (4) $$

where $p_{\theta}(y\mid x)$ is the backbone-conditioned inverse-folding likelihood,
$p_{\varphi}(\cdot)$ the corresponding the unconditional sequence prior,
$y_{\mathrm{wt}}$ the PDB wild-type sequence, and $k_{B}T$ the thermal energy at 298 K (0.593 kcal mol^-1).
Specifically, the prior $p_{\varphi}(\cdot)$ is obtained by running the same inverse-folding network (e.g., ProteinMPNN or InstructPLM) with its coordinate channels masked, thus converting it into a sequence-only language model that captures the global residue-frequency and chain-length distributions of natural proteins.
Subtracting $\log p_{\varphi}(y)$ from $\log p_{\theta}(y\mid x)$ removes background amino-acid composition and chain-length preferences learned from natural proteins and isolates the backbone-specific excess compatibility of the candidate sequence $y$.
Meanwhile, evaluating the difference for $y_{\mathrm{wt}}$ anchors the estimate to a common base, and the resulting difference provides a local, computationally efficient surrogate for $\Delta\!\Delta G$ tailored to monomeric stability optimization.

Multi-objective reward: Our final reward combines both scores after min–max normalization across the candidate pool of inverse folding sequences generated for the same backbone within each reinforcement learning iteration: $\tilde{r}_{\mathrm{TM}}=(r_{\mathrm{TM}}-r_{\mathrm{TM}}^{\min})/(r_{\mathrm{
TM}}^{\max}-r_{\mathrm{TM}}^{\min})$ and $\tilde{r}_{\Delta\!\Delta G}$ analogously, giving $r(x,y)=\lambda_{\mathrm{TM}}\tilde{r}_{\mathrm{TM}}(x,y)+\lambda_{\Delta\!
\Delta G}\tilde{r}_{\Delta\!\Delta G}(x,y)$.
This reward model accelerates evaluation speed by at least 25x, reducing training time from months to days. Our experiments show that proteins designed with this reward achieve remarkable thermodynamic stability improvements (energy reductions of 500-800% compared to wild-type proteins) with high structural fidelity (See Tab. [2](https://arxiv.org/html/2506.07459v2#S4.T2)).

### 3.3 Online RL Fine-tuning of Inverse Folding Models via ProteinZero

Building upon our general framework, we implement two online RL algorithms for fine-tuning protein inverse folding models: RAFT and GRPO. Both approaches utilize our efficient reward modeling and diversity regularization while addressing the optimization from complementary perspectives.

#### 3.3.1 ProteinZero RAFT RAFT {}_{\text{RAFT}} start_FLOATSUBSCRIPT RAFT end_FLOATSUBSCRIPT : Reward-ranked Fine-tuning with Embedding Diversity

RAFT transforms RL into a supervised learning problem by filtering model outputs based on rewards. Our adaptation generates multiple candidate sequences for each target structure, evaluates them using our efficient reward function, and retains only the best sequences to form a filtered dataset. Unlike conventional RAFT implementations that incorporate KL-divergence into the reward, we separate the KL term and add our embedding-based diversity regularization ($\mathcal{L}_{\mathrm{CE}}$ is the cross entropy loss):

$$ $\mathcal{L}_{\text{ProteinZero}_{\text{RAFT}}}(\theta)=\mathcal{L}_{\mathrm{CE }}\left(\theta;\mathcal{D}_{\text{filtered }}\right)+\alpha_{\mathrm{KL}}\cdot \mathrm{KL}\left(p_{\theta}\|p_{\text{ref }}\right)-\alpha_{\text{div }}\cdot D _{\cos}\left(\theta;\mathcal{D}_{\text{filtered }}\right)$ (5) $$

#### 3.3.2 ProteinZero GRPO GRPO {}_{\text{GRPO}} start_FLOATSUBSCRIPT GRPO end_FLOATSUBSCRIPT : Embedding-Diversified Policy Optimization

GRPO (grpo,) directly optimizes the policy through a trust-region approach using the following objective:

$$ $\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)$ $\displaystyle=\mathbb{E}_{x\sim P(X),\left\{y_{i}\right\}_{i=1}^{G}\sim\pi_{ \theta_{\text{old }}}(Y\mid x)}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\left|y_{i} \right|}\sum_{t=1}^{\left|y_{i}\right|}\min\left[\frac{\pi_{\theta}\left(y_{i, t}\mid x,y_{i,<t}\right)}{\pi_{\theta_{\text{old }}}\left(y_{i,t}\mid x,y_{i,< t}\right)}\hat{A}_{i,t},\right.$ (6) $\displaystyle\left.\operatorname{clip}\left(\frac{\pi_{\theta}\left(y_{i,t} \mid x,y_{i,<t}\right)}{\pi_{\theta_{\text{old }}}\left(y_{i,t}\mid x,y_{i,<t} \right)},1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right]-\beta\mathbb{D }_{KL}\left[\pi_{\theta}||\pi_{ref}\right],$ $$

where $\varepsilon$ and $\beta$ are hyperparameters, and $\hat{A}_{i,t}$ is the advantage calculated from relative rewards within each group. The group relative advantage calculation aligns well with our reward models. Unlike methods that add KL penalty to rewards, GRPO directly adds KL divergence to the loss. We extend this formulation by incorporating our embedding-level diversity regularization ($\mathcal{L}_{\text{GRPO}}=-\mathcal{J}_{\text{GRPO}}$):

$$ $\mathcal{L}_{\text{ProteinZero}_{\text{GRPO}}}(\theta)=\mathcal{L}_{\text{GRPO }}(\theta)-\alpha_{\text{div }}\cdot D_{\cos}(\theta;\mathcal{B})$ (7) $$

Both algorithms effectively implement our ProteinZero framework but approach optimization differently. Our experiments show both methods significantly outperform baselines, with GRPO achieving slightly better results across most metrics.

## 4 Experiment

### 4.1 Experimental Setup

We evaluated ProteinZero on CATH-4.3 (orengo1997cath,), maintaining the train-test-validation split from (esm_if,). Our test set excluded structures with $>40\%$ sequence identity to training proteins of 0-150 residues and $>30\%$ identity to proteins of 150-300 residues, enabling assessment of out-of-distribution generalization. We trained and evaluated models separately for each length category (0-150, 150-300) using designability metrics (TM Score (tm_score,), PLDDT (esm_if,; af2,), scRMSD (instruct_plm,; proteinmpnn,)), stability measures (Pred-ddG (ours), FoldX ddG (foldx,)), and sequence properties (Recovery, Diversity).
Overall success was defined as achieving both scRMSD $<2\text{\AA}$ and FoldX ddG $<0$. We compared against state-of-the-art inverse folding models (ProteinMPNN (proteinmpnn,), ESM-IF (esm_if,), InstructPLM (instruct_plm,)) and an offline RL baseline (DPO) (dpo,). For online learning, we implemented two algorithms: $\text{ProteinZero}_{\text{RAFT}}$, which selects best rewarded sequences for fine-tuning, and $\text{ProteinZero}_{\text{GRPO}}$, which directly optimizes policy using relative rewards, both running for 20 iterations. Both methods employed embedding-level diversity regularization ($\alpha_{\text{div}}=0.05$) and KL constraints ($\alpha_{\mathrm{KL}}=0.1$). Our reward function combined ESM-fold-based TM scores and stability prediction ([4](https://arxiv.org/html/2506.07459v2#S3.E4)), reducing evaluation time by $\sim 2500\%$. Implementation details are provided in Appendix [B](https://arxiv.org/html/2506.07459v2#A2).

### 4.2 Main Results

**Table 2: Comparison of protein sequence design methods across different evaluation metrics for two protein size categories (0-150 and 150-300 residues). We report sequence recovery rate, sequence diversity, stability (Pred-ddG (ours) and FoldX ddG (foldx,)), design metrics (TM Score, PLDDT, scRMSD with percentage below $2\,\text{\AA}$), and overall Success Rate (i.e., $\text{scRMSD}<2\,\text{\AA}$ and $\text{FoldX ddG}<0$). Noting that we report our success rate measured by real ddG from FoldX instead of using predicted-ddG like (success_rate,), where the latter might deviate significantly from biological reality. Best scores are highlighted in blue, second-best in green.**
| Length | Method | InverseFold Acc. | Stability | Design Metrics | Overall |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recovery Rate $\uparrow$ | Pred-ddG $\downarrow$ | FoldX ddG $\downarrow$ | TM Score $\uparrow$ | PLDDT $\uparrow$ | Diversity $\uparrow$ | scRMSD $\downarrow$ (scRMSD <2Å% $\uparrow$) | Success (%) $\uparrow$ |  |
| 0-150 residues | Base Model |  |  |  |  |  |  |  |
| InstructPLM | $0.574_{\pm 0.009}$ | $-21.543_{\pm 1.330}$ | $-20.878_{\pm 1.445}$ | $0.812_{\pm 0.011}$ | $79.983_{\pm 0.614}$ | $0.281_{\pm 0.007}$ | $1.484_{\pm 0.044}$ ($85.71\%_{\pm 0.002}$) | $84.45\%_{\pm 0.0002}$ |
| SOTA Inverse Folding Models |  |  |  |  |  |  |  |  |
| ProteinMPNN (proteinmpnn,) | $0.426_{\pm 0.006}$ | $-21.509_{\pm 1.230}$ | $-20.792_{\pm 1.207}$ | $0.805_{\pm 0.009}$ | $79.883_{\pm 0.502}$ | $0.280_{\pm 0.005}$ | $1.500_{\pm 0.037}$ ($82.14\%_{\pm 0.002}$) | $81.95\%_{\pm 0.0002}$ |
| ESM-IF (esm_if,) | $0.377_{\pm 0.006}$ | $-17.900_{\pm 1.235}$ | $-14.328_{\pm 1.269}$ | $0.802_{\pm 0.009}$ | $78.918_{\pm 0.534}$ | $0.263_{\pm 0.005}$ | $1.515_{\pm 0.038}$ ($81.25\%_{\pm 0.002}$) | $80.71\%_{\pm 0.0002}$ |
| RL Baseline Method |  |  |  |  |  |  |  |  |
| DPO (dpo,) | $0.571_{\pm 0.008}$ | $-21.713_{\pm 1.260}$ | $-21.191_{\pm 1.332}$ | $0.820_{\pm 0.010}$ | $80.716_{\pm 0.571}$ | $0.274_{\pm 0.005}$ | $1.473_{\pm 0.041}$ ($87.58\%_{\pm 0.002}$) | $86.44\%_{\pm 0.0002}$ |
| Our Online RL Methods |  |  |  |  |  |  |  |  |
| $\text{ProteinZero}_{\text{RAFT}}$ (Ours) | $0.587_{\pm 0.008}$ | $-22.236_{\pm 1.272}$ | $-23.168_{\pm 1.356}$ | $0.849_{\pm 0.011}$ | $81.560_{\pm 0.613}$ | $0.296_{\pm 0.007}$ | $1.393_{\pm 0.044}$ ($92.86\%_{\pm 0.003}$) | $89.29\%_{\pm 0.0002}$ |
| $\text{ProteinZero}_{\text{GRPO}}$ (Ours) | $0.590_{\pm 0.008}$ | $-22.616_{\pm 1.327}$ | $-24.924_{\pm 1.382}$ | $0.867_{\pm 0.011}$ | $82.326_{\pm 0.612}$ | $0.306_{\pm 0.007}$ | $1.373_{\pm 0.044}$ ($93.55\%_{\pm 0.003}$) | $90.13\%_{\pm 0.0002}$ |
| 150-300 residues | Base Model |  |  |  |  |  |  |  |
| InstructPLM | $0.570_{\pm 0.009}$ | $-36.362_{\pm 2.451}$ | $-27.145_{\pm 1.797}$ | $0.824_{\pm 0.014}$ | $83.783_{\pm 0.568}$ | $0.305_{\pm 0.008}$ | $1.448_{\pm 0.048}$ ($88.24\%_{\pm 0.002}$) | $86.38\%_{\pm 0.0002}$ |
| SOTA Inverse Folding Models |  |  |  |  |  |  |  |  |
| ProteinMPNN (proteinmpnn,) | $0.405_{\pm 0.007}$ | $-35.778_{\pm 2.280}$ | $-27.057_{\pm 1.581}$ | $0.816_{\pm 0.012}$ | $82.361_{\pm 0.548}$ | $0.297_{\pm 0.006}$ | $1.469_{\pm 0.040}$ ($86.64\%_{\pm 0.002}$) | $84.67\%_{\pm 0.0002}$ |
| ESM-IF (esm_if,) | $0.446_{\pm 0.008}$ | $-32.125_{\pm 2.207}$ | $-24.816_{\pm 1.548}$ | $0.802_{\pm 0.013}$ | $82.042_{\pm 0.536}$ | $0.279_{\pm 0.006}$ | $1.487_{\pm 0.042}$ ($86.09\%_{\pm 0.002}$) | $82.81\%_{\pm 0.0002}$ |
| RL Baseline Method |  |  |  |  |  |  |  |  |
| DPO (dpo,) | $0.570_{\pm 0.009}$ | $-36.417_{\pm 2.325}$ | $-28.915_{\pm 1.571}$ | $0.830_{\pm 0.013}$ | $83.837_{\pm 0.506}$ | $0.296_{\pm 0.008}$ | $1.441_{\pm 0.042}$ ($88.97\%_{\pm 0.002}$) | $87.70\%_{\pm 0.0002}$ |
| Our Online RL Methods |  |  |  |  |  |  |  |  |
| $\text{ProteinZero}_{\text{RAFT}}$ (Ours) | $0.578_{\pm 0.009}$ | $-37.575_{\pm 2.391}$ | $-30.755_{\pm 1.661}$ | $0.841_{\pm 0.013}$ | $83.850_{\pm 0.542}$ | $0.324_{\pm 0.008}$ | $1.427_{\pm 0.046}$ ($89.17\%_{\pm 0.002}$) | $89.36\%_{\pm 0.0002}$ |
| $\text{ProteinZero}_{\text{GRPO}}$ (Ours) | $0.580_{\pm 0.009}$ | $-40.626_{\pm 2.422}$ | $-32.805_{\pm 1.694}$ | $0.862_{\pm 0.013}$ | $84.154_{\pm 0.539}$ | $0.331_{\pm 0.009}$ | $1.393_{\pm 0.045}$ ($90.43\%_{\pm 0.002}$) | $91.19\%_{\pm 0.0002}$ |

Overall Performance Analysis. Table [2](https://arxiv.org/html/2506.07459v2#S4.T2) shows our ProteinZero framework consistently outperforms existing protein design methods across all metrics. Both our online RL methods ($\text{ProteinZero}_{\text{GRPO}}$ and $\text{ProteinZero}_{\text{RAFT}}$) surpass the base model and established baselines, with $\text{ProteinZero}_{\text{GRPO}}$ achieving the best results in all metrics. Our approach effectively balances sequence recovery, structural accuracy, and stability while learning directly from model-generated outputs without additional human-labeled data. Most importantly, our methods reduce design failure rates by 36-48% compared to alternatives, achieving success rates exceeding 90% across diverse protein architectures. ProteinZero establishes a new Pareto frontier by simultaneously improving all metrics rather than trading them off, notably achieving both higher recovery rates and higher diversity—a traditionally difficult trade-off in protein design.

Comparison with SOTA Inverse Folding Models. Compared to state-of-the-art inverse folding models (ProteinMPNN (proteinmpnn,), ESM-IF (esm_if,), and InstructPLM (instruct_plm,)), $\text{ProteinZero}_{\text{GRPO}}$ demonstrates substantial improvements across all metrics, with TM Score gains of 0.05-0.06 while simultaneously achieving higher sequence recovery rates. This dual improvement in both structural and sequence metrics indicates our approach learns better fundamental sequence-structure relationships rather than overfitting. Notably, the performance gap widens for larger proteins, suggesting our online learning framework captures generalizable design principles beyond what’s possible with static datasets. Also see Fig. [1](https://arxiv.org/html/2506.07459v2#S4.F1) for qualitative comparison.

Figure: Figure 1: Qualitative results of protein structure designs from hold-out test set. Visual comparison between our ProteinZero method (cyan), native proteins (pink), and InstructPLM (lime green). Top panels show ProteinZero’s ability to transform unstable natural proteins into highly stable designs across diverse fold types, achieving 233%-858% energy improvements while maintaining high structural fidelity (TM-scores > 0.95). Bottom panels demonstrate ProteinZero’s superior performance compared to InstructPLM, particularly for challenging $\beta$-rich structures and complex architectures where our method generates stable designs with negative ddG values while InstructPLM often produces unstable ones with positive values.
Refer to caption: x1.png

Comparison with Widely Used RL Fine-tuning Methods. The comparison with DPO, a widely-used offline RL method, provides valuable insights into the benefits of online learning. While DPO (dpo,) shows modest improvements over InstructPLM, it falls significantly behind our online methods across all evaluated metrics. This substantial performance gap highlights a key limitation of offline RL approaches: without continuous feedback from newly self-generated outputs, these methods cannot effectively explore the vast protein sequence space beyond the preferences encoded in the initial dataset. In contrast, our online RL framework enables the model to continuously refine its policy based on the quality of its own generations, leading to significantly better outcomes across all metrics. The difference is particularly pronounced in success rate, where our methods achieve a significant relative reduction in failure rate compared to DPO, demonstrating the practical impact of our approach for real-world protein design applications.

Optimizing Energy-based Structural Stability via Online RL. Our methods achieve remarkable improvements in stability metrics, with $\text{ProteinZero}_{\text{GRPO}}$ enhancing FoldX ddG by 19-21% compared to InstructPLM across both protein size categories. These results confirm that our proxy reward models effectively guide policies toward thermodynamically favorable sequences—crucial for real-world applications. Unlike single-objective approaches that typically sacrifice other metrics when optimizing stability, our multi-objective RL framework successfully navigates complex trade-offs, delivering holistic improvements in stability, structural accuracy, and sequence diversity simultaneously.

### 4.3 Case Study on Diverse Protein Folds and Complex Protein Design Tasks

Transforming Unstable Natural Proteins into Therapeutically Valuable Designs: Our visual comparison in Figure [1](https://arxiv.org/html/2506.07459v2#S4.F1) demonstrates ProteinZero’s remarkable ability to convert naturally unstable proteins into highly stable designs while maintaining exceptional structural fidelity. For naturally unstable structures like membrane proteins and $\beta$-barrels—historically challenging targets for computational design—our method achieves dramatic stability enhancements of up to 800%. The $\beta$-barrel structure (4fd5A) undergoes a remarkable transformation from a highly unstable wild-type energy to a stable design with a 233% energy improvement, while the membrane protein shows a 186% stability enhancement. These results demonstrate ProteinZero’s sophisticated optimization of the sequence-structure relationship, generating stability profiles rarely found in nature but highly valuable for therapeutic applications, industrial enzymes, and research reagents. By consistently producing designs with near-native structural accuracy (TM-scores > 0.95) and dramatically enhanced thermodynamic stability across diverse fold types, our approach expands the accessible design space to include therapeutically relevant protein classes previously considered too difficult for protein design.

Mastery of Complex Architectures with Superior Performance Scaling: When compared directly with InstructPLM, ProteinZero demonstrates consistent superiority that becomes more pronounced as protein complexity increases. For challenging $\beta$-rich structures where conventional methods struggle, our approach achieves both higher structural accuracy and substantially better stability optimization. This pattern extends across diverse fold architectures, including $\beta$-sheets, $\alpha/\beta$ mixed domains, and $\alpha$-helical structures, where ProteinZero generates stable designs while InstructPLM often produces unstable ones. Most notably, ProteinZero exhibits a counterintuitive improvement in performance with increasing protein size—where traditional methods typically degrade—achieving better results for medium-sized proteins (150-300 Å) than smaller ones (0-150 Å). This positive scaling behavior demonstrates our reinforcement learning framework’s ability to effectively capture long-range dependencies and complex domain interactions, rather than simply memorizing patterns from training data. This capability enables more effective exploration of previously inaccessible regions of the protein design space, representing a fundamental advancement in computational protein engineering with profound implications for designing complex multi-domain proteins and enzymes.

### 4.4 Exploring the Design Space of Online RL for Fine-tuning Protein Generative Models

**Table 3: Systematic exploration of the online RL design space in ProteinZero. We conduct ablation studies across three critical design dimensions: reward models, learning objectives, and diversity regularization strategies. For each design dimension, best results are highlighted in blue.**
| Length | Design Configuration | InverseFold Acc. | Stability | Design Metrics | Overall |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recovery Rate $\uparrow$ | Pred-ddG $\downarrow$ | FoldX ddG $\downarrow$ | TM Score $\uparrow$ | PLDDT $\uparrow$ | Diversity $\uparrow$ | scRMSD $\downarrow$ (scRMSD <2Å% $\uparrow$) | Success (%) $\uparrow$ |  |
| 0-150 residues | Design Dimension 1: Reward Model Formulation |  |  |  |  |  |  |  |
| Only TM-score as Reward | 0.582 | -21.598 | -21.271 | 0.874 | 82.827 | 0.293 | 1.372 (93.62%) | 89.52% |
| Only ddG as Reward | 0.580 | -22.996 | -25.381 | 0.831 | 82.270 | 0.299 | 1.466 (87.75%) | 85.15% |
| Full ProteinZero (TM+ddG) | 0.590 | -22.616 | -24.924 | 0.867 | 82.326 | 0.306 | 1.373 (93.55%) | 90.13% |
| Design Dimension 2: Learning Objective Components |  |  |  |  |  |  |  |  |
| Without Diversity Term | 0.584 | -22.526 | -24.877 | 0.861 | 82.308 | 0.268 | 1.397 (92.75%) | 90.23% |
| Without KL Term | 0.564 | -22.352 | -24.264 | 0.841 | 80.979 | 0.316 | 1.429 (90.53%) | 86.41% |
| Full ProteinZero (All Terms) | 0.590 | -22.616 | -24.924 | 0.867 | 82.326 | 0.306 | 1.373 (93.55%) | 90.13% |
| Design Dimension 3: Diversity Regularization Strategies |  |  |  |  |  |  |  |  |
| Diversity as Reward | 0.579 | -19.738 | -18.681 | 0.836 | 81.107 | 0.284 | 1.439 (87.77%) | 78.65% |
| Hamming Distance as Reward | 0.565 | -14.137 | -11.135 | 0.831 | 81.785 | 0.276 | 1.466 (88.70%) | 74.63% |
| Full ProteinZero (Embedding Diversity) | 0.590 | -22.616 | -24.924 | 0.867 | 82.326 | 0.306 | 1.373 (93.55%) | 90.13% |
| 150-300 residues | Design Dimension 1: Reward Model Formulation |  |  |  |  |  |  |  |
| Only TM-score as Reward | 0.577 | -35.793 | -25.905 | 0.870 | 84.237 | 0.333 | 1.384 (91.25%) | 89.76% |
| Only ddG as Reward | 0.574 | -42.769 | -35.927 | 0.831 | 83.540 | 0.327 | 1.447 (88.52%) | 87.38% |
| Full ProteinZero (TM+ddG) | 0.580 | -40.626 | -32.805 | 0.862 | 84.154 | 0.331 | 1.393 (90.43%) | 91.19% |
| Design Dimension 2: Learning Objective Components |  |  |  |  |  |  |  |  |
| Without Diversity Term | 0.580 | -37.905 | -31.185 | 0.860 | 84.065 | 0.281 | 1.401 (89.71%) | 91.32% |
| Without KL Term | 0.569 | -40.688 | -33.193 | 0.835 | 83.008 | 0.328 | 1.440 (89.08%) | 87.92% |
| Full ProteinZero (All Terms) | 0.580 | -40.626 | -32.805 | 0.862 | 84.154 | 0.331 | 1.393 (90.43%) | 91.19% |
| Design Dimension 3: Diversity Regularization Strategies |  |  |  |  |  |  |  |  |
| Diversity as Reward | 0.558 | -33.904 | -23.967 | 0.849 | 83.326 | 0.315 | 1.421 (89.32%) | 81.71% |
| Hamming Distance as Reward | 0.568 | -32.128 | -23.228 | 0.836 | 83.668 | 0.294 | 1.432 (89.14%) | 80.29% |
| Full ProteinZero (Embedding Diversity) | 0.580 | -40.626 | -32.805 | 0.862 | 84.154 | 0.331 | 1.393 (90.43%) | 91.19% |

To identify the contribution of each component, we systematically analyze the design space across three critical dimensions: reward model formulation, learning objective components, and diversity regularization strategies. Table [3](https://arxiv.org/html/2506.07459v2#S4.T3) presents results for both protein length categories (0-150 and 150-300 residues) using our comprehensive evaluation protocol.

Reward Model Designs. First, we investigate the impact of different reward signals on model performance. We compare three configurations: (1) using only TM-score as reward, (2) using only predicted stability (ddG) as reward, and (3) our full approach combining both signals. Results demonstrate that while TM-score alone achieves the highest structural accuracy metrics, and ddG alone yields the best stability scores, our combined reward formulation achieves the best overall success rate. This indicates that the multi-objective reward design effectively balances the trade-off between structural accuracy and stability.

Learning Objective Components. Second, we examine the contribution of different terms in our learning objective. We compare the full objective against variants with either the diversity regularization term or the KL divergence term removed. Interestingly, removing the diversity term slightly improves the overall success rate for both protein categories, suggesting a potential trade-off between diversity and other performance metrics. However, the full objective maintains better balance across all metrics. Removing the KL term significantly reduces performance, confirming the importance of constraining policy updates to prevent catastrophic forgetting during online learning.

Diversity Regularization Strategies. Finally, we explore alternative approaches to incorporating sequence diversity. We compare: (1) directly using diversity as an additional reward signal, (2) using Hamming distance between generated sequences as a diversity measure, and (3) our approach using embedding-based diversity regularization. The results clearly demonstrate the superiority of our embedding diversity approach, which maintains high sequence diversity without compromising structural accuracy or stability. The alternative approaches show significant performance degradation, particularly in terms of stability (FoldX ddG) and overall success rate.

These ablation studies validate our design choices and highlight the importance of carefully balancing multiple objectives in online RL for protein design. Our ProteinZero framework effectively navigates these trade-offs, resulting in a robust approach that generalizes well across different protein sizes.

## 5 Conclusion

In this paper, we presented ProteinZero, a framework that enables protein generative models to continuously improve through online RL. By developing efficient proxy reward models and a novel protein-embedding level diversity regularization, we’ve made online RL in protein design computationally tractable for the first time. Our approach demonstrates unprecedented multi-objective optimization, simultaneously enhancing structural accuracy, thermodynamic stability, and sequence diversity while significantly reducing failure rates compared to widely-used methods. Most notably, ProteinZero excels at designing historically challenging protein architectures like $\beta$-barrels and membrane proteins, transforming unstable natural proteins into highly stable designs with maintained structural fidelity. The framework’s ability to learn from its own generated outputs rather than static datasets represents a paradigm shift in computational protein engineering, opening new possibilities for exploring vast protein design spaces inaccessible to previous methods. This self-improving capability points toward a future where protein design systems evolve continuously, potentially accelerating discoveries across medicine and biotechnology.

## References

- [1]
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al.
Accurate structure prediction of biomolecular interactions with alphafold 3.
Nature, 630(8016):493–500, 2024.
- [2]
Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk.
Self-consuming generative models go MAD.
In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [3]
Nathaniel R Bennett, Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Po Peng, Justas Dauparas, Minkyung Baek, Lance Stewart, et al.
Improving de novo protein binder design with deep learning.
Nature Communications, 14(1):2625, 2023.
- [4]
Matteo Cagiada, Sergey Ovchinnikov, and Kresten Lindorff-Larsen.
Predicting absolute protein folding stability using generative models.
Protein Science, 34(1):e5233, 2025.
- [5]
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola.
Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.
arXiv preprint arXiv:2402.04997, 2024.
- [6]
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al.
Robust deep learning–based protein sequence design using proteinmpnn.
Science, 378(6615):49–56, 2022.
- [7]
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.
Raft: Reward ranked finetuning for generative foundation model alignment.
arXiv preprint arXiv:2304.06767, 2023.
- [8]
Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu.
Online reward-weighted fine-tuning of flow matching with wasserstein regularization.
In The Thirteenth International Conference on Learning Representations, 2025.
- [9]
Scott Fujimoto and Shixiang Shane Gu.
A minimalist approach to offline reinforcement learning.
Advances in neural information processing systems, 34:20132–20145, 2021.
- [10]
Hans-Christof Gasser, Diego A Oyarzún, Javier Alfaro, and Ajitha Rajan.
Integrating mhc class i visibility targets into the proteinmpnn protein design process.
bioRxiv, pages 2024–06, 2024.
- [11]
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.
The curious case of neural text degeneration.
In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
- [12]
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives.
Learning inverse folding from millions of predicted structures.
In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 8946–8970. PMLR, 2022.
- [13]
Guillaume Huguet, James Vuckovic, Kilian Fatras, Eric Thibodeau-Laufer, Pablo Lemos, Riashat Islam, Cheng-Hao Liu, Jarrid Rector-Brooks, Tara Akhound-Sadegh, Michael M. Bronstein, Alexander Tong, and Avishek Joey Bose.
Sequence-augmented se(3)-flow matching for conditional protein backbone generation.
CoRR, abs/2405.20313, 2024.
- [14]
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola.
Generative models for graph-based protein design.
Advances in neural information processing systems, 32, 2019.
- [15]
John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng-Thow-Hing, Erik R Van Vlack, et al.
Illuminating protein space with a programmable generative model.
Nature, 623(7989):1070–1078, 2023.
- [16]
Xiaoran Jiao, Weian Mao, Wengong Jin, Peiyuan Yang, Hao Chen, and Chunhua Shen.
Boltzmann-aligned inverse folding model as a predictor of mutational effects on protein-protein interactions.
In The Thirteenth International Conference on Learning Representations, 2025.
- [17]
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron O. Dror.
Learning from protein structure with geometric vector perceptrons.
In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
- [18]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.
Highly accurate protein structure prediction with alphafold.
nature, 596(7873):583–589, 2021.
- [19]
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton.
Cath–a hierarchic classification of protein domain structures.
Structure, 5(8):1093–1109, 1997.
- [20]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
- [21]
Ryan Park, Darren J. Hsu, C. Brian Roland, Maria Korshunova, Chen Tessler, Shie Mannor, Olivia Viessmann, and Bruno Trentini.
Improving inverse folding for peptide design with diversity-regularized direct preference optimization.
CoRR, abs/2410.19471, 2024.
- [22]
Ryan Park, Darren J. Hsu, Chen Tessler, Maria Korshunova, C. Brian Roland, Olivia Viessmann, and Bruno Trentini.
Improving inverse folding for peptide design with diversity-regularized direct preference optimization, 2025.
- [23]
Jiezhong Qiu, Junde Xu, Jie Hu, Hanqun Cao, Liya Hou, Zijun Gao, Xinyi Zhou, Anni Li, Xiujuan Li, Bin Cui, Fei Yang, Shuang Peng, Ning Sun, Fangyu Wang, Aimin Pan, Jie Tang, Jieping Ye, Junyang Lin, Jin Tang, Xingxu Huang, Pheng Ann Heng, and Guangyong Chen.
Instructplm: Aligning protein language models to follow protein structure instructions.
bioRxiv, 2024.
- [24]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
- [25]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
CoRR, abs/1707.06347, 2017.
- [26]
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Serrano.
The foldx web server: an online force field.
Nucleic acids research, 33(suppl_2):W382–W388, 2005.
- [27]
Varun R. Shanker, Theodora U. J. Bruun, Brian L. Hie, and Peter S. Kim.
Unsupervised evolution of protein and antibody complexes with a structure-informed language model.
Science, 385(6704):46–53, 2024.
- [28]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
CoRR, abs/2402.03300, 2024.
- [29]
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal.
Ai models collapse when trained on recursively generated data.
Nature, 631(8022):755–759, 2024.
- [30]
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross J. Anderson, and Yarin Gal.
AI models collapse when trained on recursively generated data.
Nat., 631(8022):755–759, 2024.
- [31]
Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Avantika Lal, Tommi Jaakkola, Sergey Levine, Aviv Regev, Hanchen, and Tommaso Biancalani.
Fine-tuning discrete diffusion models via reward optimization with applications to DNA and protein design.
In The Thirteenth International Conference on Learning Representations, 2025.
- [32]
Chuanrui Wang, Bozitao Zhong, Zuobai Zhang, Narendra Chaudhary, Sanchit Misra, and Jian Tang.
Pdb-struct: A comprehensive benchmark for structure-based protein design.
CoRR, abs/2312.00080, 2023.
- [33]
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al.
De novo design of protein structure and function with rfdiffusion.
Nature, 620(7976):1089–1100, 2023.
- [34]
Talal Widatalla, Rafael Rafailov, and Brian Hie.
Aligning protein generative models with experimental fitness via direct preference optimization.
bioRxiv, pages 2024–05, 2024.
- [35]
Chengxin Zhang, Morgan Shine, Anna Marie Pyle, and Yang Zhang.
Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes.
Nature methods, 19(9):1109–1115, 2022.
- [36]
Yang Zhang and Jeffrey Skolnick.
Scoring function for automated assessment of protein structure template quality.
Proteins: Structure, Function, and Bioinformatics, 57(4):702–710, 2004.
- [37]
Yang Zhang and Jeffrey Skolnick.
Tm-align: a protein structure alignment algorithm based on the tm-score.
Nucleic acids research, 33(7):2302–2309, 2005.
- [38]
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu.
Structure-informed language models are protein designers.
In International conference on machine learning, pages 42317–42338. PMLR, 2023.
- [39]
Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang, and Quanquan Gu.
Antigen-specific antibody design via direct energy-based preference optimization.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.

## Appendix A Discussion

### A.1 Broader Impact

Our proposed ProteinZero represents a fundamental paradigm shift in computational protein design with far-reaching implications for biological research and biotechnology. By enabling protein generative models to improve autonomously through online reinforcement learning without human supervision, our work addresses one of the most significant bottlenecks in the field: the dependence on laboriously collected and annotated datasets.

Traditional protein design approaches rely heavily on manually curated data from repositories like the Protein Data Bank, which capture only a minuscule fraction of the vast protein sequence space. The process of experimentally determining protein structures and properties is time-consuming, resource-intensive, and represents a fundamental limiting factor in the advancement of protein engineering. ProteinZero liberates protein design from these constraints by creating a self-improving system that learns continuously from its own generated outputs.

This zero-supervision learning framework has profound implications for multiple domains. In therapeutic development, by dramatically reducing the time required to design stable, functional proteins, ProteinZero could significantly accelerate the development of novel biologics, vaccines, and enzyme therapies. The ability to rapidly explore previously inaccessible regions of protein sequence space may uncover therapeutic solutions that would be practically impossible to discover through traditional methods.

The reduced computational requirements of our approach (2500% faster than traditional methods while improve 36-50% design success rate) helps lower the barrier to entry for protein design, potentially democratizing access to this technology for researchers with limited computational resources. More efficient exploration of the protein design space could lead to breakthroughs in developing enzymes for bioremediation, sustainable manufacturing processes, and green chemistry applications.

By eliminating the dependency on human-labeled data, ProteinZero not only accelerates current research directions but potentially opens entirely new avenues of investigation that were previously impractical due to data constraints. This represents a new generation of AI systems for biology that can continuously improve and adapt through their own experiences, mirroring the evolutionary process that shaped natural proteins but operating at dramatically accelerated timescales. The long-term impact of such self-improving systems could fundamentally transform how we approach protein engineering challenges across medicine, industry, and basic science.

## Appendix B Experimental Details

### B.1 Prompt/Task Datasets

To evaluate ProteinZero’s ability to improve protein design through online reinforcement learning, we used the CATH-4.3 dataset, which contains protein domains classified according to Class, Architecture, Topology, and Homology. This dataset was specifically chosen to avoid overlap with the pre-training data used in baseline models, allowing us to better assess ProteinZero’s ability to generalize to out-of-distribution (OOD) protein structures. We split the dataset into two categories based on protein size: small proteins (0-150 residues) and medium-sized proteins (150-300 residues) to evaluate performance across different structural complexity levels.

Importantly, our online reinforcement learning approach does not require labeled sequence data during the fine-tuning process, as the model learns entirely from its own generated outputs via the reward function. This represents a significant departure from previous supervised approaches that rely heavily on labeled data.

### B.2 Implementation Details

1. Hyperparameter Settings:

$\textbf{ProteinZero}_{\textbf{RAFT}}$: We optimize our model with AdamW using an initial learning rate of $3\times 10^{-5}$ ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1\times 10^{-8}$, weight decay $=0.01$) over all RAFT iterations.
For each RAFT iteration, we apply a linear learning-rate decay (with zero warm-up) over the epochs.
We apply rank-$5$ LoRA adapters ($\alpha_{\text{LoRA}}=16$, dropout $=0.05$) to all self-attention and feed-forward projections. During each iteration, we partition the CATH 4.3 training set across GPUs, generating $K=8$ candidate sequences per backbone via nucleus sampling (temperature $=0.8$, $p=0.9$), and retain only the highest-reward sequence for fine-tuning.
Our policy updates incorporate a KL regularizer with a coefficient of $0.1$ against a frozen reference policy, whereas the original RAFT implementation used a grid search to explore different KL term weights (0 (disabled), 0.005, 0.01, 0.1). We conduct extensive ablation studies on the KL weight used in the original RAFT in Table [5](https://arxiv.org/html/2506.07459v2#A3.T5) within Section [C.2](https://arxiv.org/html/2506.07459v2#A3.SS2). Our empirical analysis reveals that this specific KL weight parameterization of $0.1$ is critical for achieving superior performance within the $\textbf{ProteinZero}_{\textbf{RAFT}}$ framework. We additionally employ an embedding-space diversity penalty with a coefficient of $0.05$, which was not included in the original RAFT. The reward function equally weights TM-score and predicted $\Delta\Delta G$. All experiments utilize mixed-precision FP16 (or BF16 where available) with two-step gradient accumulation per update. Our results suggest that stronger KL regularization helps mitigate instability in pretrained protein language models during fine-tuning.

$\textbf{ProteinZero}_{\textbf{GRPO}}$: We optimize our model using AdamW with an initial learning rate of $1\times 10^{-6}$ ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1\times 10^{-8}$, weight decay $=0$), and employ a linear learning-rate scheduler (no warm-up) over all 20 GRPO iterations.
We apply LoRA adapters with rank $r=5$ (scaling factor $\alpha_{\text{LoRA}}=16$, dropout $=0.05$) to all self-attention and feed-forward projections. Each episode samples from the CATH 4.3 training set (distributed across GPUs) and generates $K=8$ candidate sequences per backbone via nucleus sampling (temperature $=0.8$, $p=0.9$).
For policy optimization, we employ a GRPO clipping coefficient $\varepsilon=0.1$ with KL regularization against a frozen reference policy with a coefficient of $0.1$, complemented by an embedding-space diversity penalty with a coefficient of $0.05$. The reward function equally weights TM-score and predicted $\Delta\Delta G$. All experiments use mixed-precision FP16, with no gradient accumulation to ensure each episode constitutes a complete policy update.
We note that our KL regularization weight of $0.1$ differs from the original GRPO implementation [28], which uses $0.04$. We conduct extensive ablation studies on the KL weight used in the original GRPO in Table [5](https://arxiv.org/html/2506.07459v2#A3.T5) within Section [C.2](https://arxiv.org/html/2506.07459v2#A3.SS2). These experiments demonstrate that the KL weight configuration is essential for optimal performance in our $\textbf{ProteinZero}_{\textbf{GRPO}}$ setting, which establishes our configuration as the optimal solution. Our ablation studies reveal that decreasing KL regularization strength leads to performance degradation across multiple metrics, including sequence recovery, Pred-DDG, FoldX DDG, TM-score, pLDDT, scRMSD, and success rate. These findings indicate that stronger KL regularization may help stabilize pretrained protein language models during fine-tuning.

Direct Preference Optimization (Baseline): For each target structure, we sample $K=8$ candidate sequences at a temperature of $T=0.1$ to form chosen–rejected pairs according to our reward model, and we optimize the DPO loss over 20 epochs using AdamW with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay of 0.01. We apply a KL divergence regularization term against a frozen reference policy with a coefficient of $0.1$, and we incorporate an embedding-space diversity penalty with a coefficient of $0.05$. All experiments are conducted in mixed-precision FP16.

2. Hardware Usage: All experiments are conducted using eight NVIDIA A100 GPUs. We fine-tune the pretrained model by stratifying protein sequences into two length categories: 0-150 amino acids and 150-300 amino acids. The total wall-clock runtime for a complete training run through 20 iterations is $20.83$ hours for the 0-150 amino acid category and $274.33$ hours for the 150-300 amino acid category, corresponding to $1.04$ hours and $13.72$ hours per iteration, respectively.

### B.3 Evaluation Metrics

We evaluated ProteinZero using a comprehensive set of metrics across three key dimensions:

1. Structural Accuracy:

- •
TM Score: Measures the topological similarity between predicted and target structures, with values ranging from 0 to 1 (higher is better) [36].
- •
PLDDT (Predicted Local Distance Difference Test): Assesses the confidence in local structure prediction [18, 1].
- •
scRMSD (Self-consistency RMSD of structures): Measures the deviation of side chain positions, with percentage below $2\text{\AA}$ reported as an additional quality indicator [23, 21].

2. Stability Metrics:

- •
Pred-ddG [16]: Predicted change in Gibbs free energy, estimated directly from the model.
- •
FoldX ddG [26]: A more rigorous physics-based calculation of stability using the FoldX force field, which better correlates with experimental measurements.

3. Sequence Properties:

- •
Recovery: The percentage of amino acids matching reference sequences, indicating how well the model captures natural sequence preferences [21].
- •
Diversity: A measure of variation among generated sequences, calculated as the mean normalized Hamming distance between every pair of sequences conditioned on the same backbone (score ranges from 0 for identical sequences to 1 for sequences that differ at every position):
$D_{\text{Hamming}}(\mathcal{B})\;=\;\frac{2}{B(B-1)}\sum_{1\leq i<j\leq B}
\left[\frac{1}{L}\sum_{t=1}^{L}\mathbf{1}\!\bigl{[}y_{i,t}\neq y_{j,t}\bigr{]}
\right].$

### B.4 Baseline Methods

We compared ProteinZero against several state-of-the-art methods:
1. Supervised Inverse Folding Models:

- 1.
ProteinMPNN: A graph-based model that directly predicts amino acid sequences from backbone structures.
- 2.
ESM-IF: A transformer-based inverse folding model trained on substantial structural data.
- 3.
InstructPLM (our base model): A recently developed protein language model fine-tuned to follow structural design instructions.

2. Offline RL Baseline: DPO (Direct Preference Optimization): A widely used offline reinforcement learning method that learns from preference data without online interaction.

For fair comparison, all baseline methods used the same evaluation protocol and metrics. InstructPLM served as our starting model for ProteinZero fine-tuning, establishing a direct comparison between supervised learning and our online RL approach.

### B.5 Reward Model

Traditional methods for evaluating protein designs require minutes to hours per evaluation, making online reinforcement learning impractical. We solve this challenge with two efficient reward models:

1. Structural Alignment Reward: We use ESM-fold for structural inference instead of the slower AlphaFold2/3 [18, 1]. The TM-score reward $r_{\mathrm{TM}}(x,y)$ is computed by first folding the generated sequence $y$ using ESM-fold, then calculating the TM-score [36] between the predicted structure and the target structure $x$ with US-align [35], an updated implementation from the original TM-align [37].

2. Design Stability Reward: We calculate $r_{{\Delta\Delta\mathrm{G}}}(x,y)$, the estimation of $\Delta\Delta G$ by comparing the backbone-conditioned likelihood of each generated sequence with an unconditional sequence prior, $p_{\varphi}(y)$, provided by pretrained inverse folding models such as ProteinMPNN and InstructPLM, as proposed in [16, 27, 34, 4, 3]:
$\Delta\!\Delta G(x,y)=-\,k_{B}T[(\log p_{\theta}(y\mid x)-\log p_{\varphi}(y))
-(\log p_{\theta}(y_{\text{wt}}\mid x)-\log p_{\varphi}(y_{\text{wt}}))]$, where $y_{\text{wt}}$ represents the PDB wild-type sequence and $k_{B}T$ represents the thermal energy at 298 K ($0.593\,\mathrm{kcal\,mol^{-1}}$).

Our reward combines both scores after min–max normalization across the candidate pool of inverse folding sequences generated for the same backbone within each reinforcement learning iteration: $\tilde{r}_{\mathrm{TM}}=(r_{\mathrm{TM}}-r_{\mathrm{TM}}^{\min})/(r_{\mathrm{
TM}}^{\max}-r_{\mathrm{TM}}^{\min})$ and $\tilde{r}_{\Delta\!\Delta G}$ analogously, giving $r(x,y)=\lambda_{\mathrm{TM}}\tilde{r}_{\mathrm{TM}}(x,y)+\lambda_{\Delta\!
\Delta G}\tilde{r}_{\Delta\!\Delta G}(x,y)$.
This reward model accelerates evaluation speed by at least 2500% compared to traditional methods, reducing training time from months to days. Our experiments show that proteins designed with this reward achieve remarkable thermodynamic stability improvements (energy reductions of 500-800% compared to natural proteins) with high structural fidelity (See Tab. [2](https://arxiv.org/html/2506.07459v2#S4.T2)).

**Table 4: Total wall-clock time (excluding Multiple Sequence Alignment) required to generate eight inverse-folding sequences conditioned on the same structural backbone for each reward component in our fine-tuning pipeline (often used for De novo design tasks, but we focus on inverse folding tasks.). Best results are highlighted in blue.**
| Length range | Structural Alignment Reward (Prediction) | Design Stability Reward (ddG) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | ESMFold | AlphaFold 2 | ColabFold | OpenFold | AlphaFold 3 | Predicted-ddG | FoldX |
| 0–150 aa | 18.7 s | 197.6 s ($\sim$10.6$\times$) | 193.6 s ($\sim$10.4$\times$) | 189.6 s ($\sim$10.1$\times$) | 199.2 s ($\sim$10.7$\times$) | $\sim$2 s (GPU) | 472.3 s ($\sim$236.2$\times$) |
| 150–300 aa | 47.5 s | 223.2 s ($\sim$4.7$\times$) | 217.6 s ($\sim$4.6$\times$) | 200.8 s ($\sim$4.2$\times$) | 237.6 s ($\sim$5.0$\times$) | $\sim$2 s (GPU) | 1520.6 s ($\sim$760.3$\times$) |

### B.6 Online RL Algorithms

We implemented and evaluated two online reinforcement learning algorithms for ProteinZero:

1. $\text{ProteinZero}_{\text{RAFT}}$: Our adaptation of Reward-rAnked Fine-Tuning, which generates multiple candidate sequences, evaluates them using our reward models, and retains only the best sequences for supervised fine-tuning. We extended RAFT with our embedding level diversity regularization term.

2. $\text{ProteinZero}_{\text{GRPO}}$: Our adaptation of Group Relative Policy Optimization, which directly optimizes the policy using relative rewards within each batch. This was further enhanced with our embedding-level diversity regularization.

### B.7 More on Future Implications

A critical computational challenge in protein structure-conditioned generation stems from the runtime requirements of structural inference during reward computation. As shown in Tables [1](https://arxiv.org/html/2506.07459v2#S2.T1) and [4](https://arxiv.org/html/2506.07459v2#A2.T4), we comprehensively evaluate the wall-clock time necessary for reward generation across multiple structural prediction frameworks. For our inverse folding framework, which operates with predetermined backbone structures, ESMFold demonstrates substantial efficiency advantages, requiring only 18.7s and 47.5s for proteins in the 0-150 and 150-300 amino acid ranges, respectively. This represents a 26-87$\times$ acceleration compared to AlphaFold2, ColabFold, OpenFold, and AlphaFold3. The computational gap widens significantly when considering Multiple Sequence Alignment (MSA), which constitutes essential but time-intensive preprocessing for the AlphaFold family models. For thermal stability prediction, our Pred-ddG approach ($\sim$2s on GPU) achieves a 236-760$\times$ speedup over physics-based methods like FoldX. While our current implementation focuses on inverse folding with fixed backbones, these benchmarks establish important computational baselines for future extensions to de novo protein design tasks, where simultaneous optimization of sequence and structure would introduce additional complexity. Notably, as Table [4](https://arxiv.org/html/2506.07459v2#A2.T4) demonstrates, our framework’s reliance on ESMFold eliminates the computational burden of MSA search, a critical advantage for potential de novo applications where rapid structural evaluation is essential. De novo design presents different challenges, requiring not only the generation of applicable sequences but also the exploration of the vast conformational landscape to discover novel protein folds with targeted functional properties. This expanded search space would require efficient sampling strategies across both sequence and structural domains, while maintaining physically realistic conformations with proper hydrophobic packing, secondary structure formation, and domain-level architectural coherence. The computational efficiency gains demonstrated in our proxy reward models suggest that integrating lightweight structural prediction methods that avoid MSA requirements within a reinforcement learning framework could make online learning feasible even for these more complex design scenarios. The dramatic reduction in evaluation time enabled by our approach makes online reinforcement learning computationally tractable for current inverse folding tasks, while providing insights into the feasibility of extending this paradigm to full de novo design in future work.

## Appendix C Additional Experimental Results

### C.1 Qualitative Eevaluation of ProteinZero using AlphaFold3 and Rosetta Energy

To further validate the robustness and generalizability of our ProteinZero framework beyond the primary evaluation tools used in our main experiments, we conduct additional qualitative assessments using state-of-the-art alternative evaluation methods. Specifically, we employ AlphaFold3 for structural prediction and Rosetta Energy functions for thermodynamic stability assessment, which provide alternative validation of our approach’s effectiveness across different computational frameworks. This additional validation demonstrates that ProteinZero achieves consistent performance improvements regardless of the specific evaluation tools employed, which strengthens our conclusions and contributions.

Figure: Figure 2: Qualitative evaluation of ProteinZero using AlphaFold3 and Rosetta Energy. This figure complements our main results by demonstrating ProteinZero’s performance when evaluated with alternative protein structure prediction (AlphaFold3) and stability assessment (Rosetta Energy) tools. Across eight diverse and complex protein architectures (150-300 residues), our designed sequences (cyan) maintain exceptional structural alignment with native proteins (pink) as indicated by high AF3 PTM scores (0.93-0.96) and PLDDT values (89.62-96.92). The substantial improvements in Rosetta ddG values (-4.091 to -92.5) further validate our approach’s ability to simultaneously optimize structural accuracy and thermodynamic stability. These results reinforce the conclusions from our FoldX and ESMFold analyses, confirming that ProteinZero’s online reinforcement learning framework effectively balances multiple design objectives across various protein classes including membrane proteins, $\alpha/\beta$-mix mixed domains, $\alpha$-helical structures, and $\beta$-barrels.
Refer to caption: x3.png

### C.2 Additional Qualitative Results

**Table 5: Supplementary experimental results exploring different hyperparameter configurations for ProteinZero. We evaluate the impact of KL divergence coefficients ($\alpha_{\mathrm{KL}}$) and diversity regularization ($\alpha_{\mathrm{div}}$) on both GRPO and RAFT algorithms across two protein size categories. Best results within each algorithm and size category are highlighted in blue.**
| Length | Configuration | InverseFold Acc. | Stability | Design Metrics | Overall |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recovery Rate $\uparrow$ | Pred-ddG $\downarrow$ | FoldX ddG $\downarrow$ | TM Score $\uparrow$ | PLDDT $\uparrow$ | Diversity $\uparrow$ | scRMSD $\downarrow$ (scRMSD <2Å% $\uparrow$) | Success (%) $\uparrow$ |  |  |
| 0-150 residues | Additional GRPO Results |  |  |  |  |  |  |  |  |
| GRPO ($\alpha_{\mathrm{KL}}=0.04$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -22.06 | -22.71 | 0.86 | 82.13 | 0.31 | 1.39 (93%) | 89% |  |
| GRPO ($\alpha_{\mathrm{KL}}=0.04$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -22.50 | -24.55 | 0.85 | 82.23 | 0.27 | 1.41 (90%) | 90% |  |
| Additional RAFT Results |  |  |  |  |  |  |  |  |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.005$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -21.63 | -21.18 | 0.84 | 80.93 | 0.30 | 1.41 (92%) | 88% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.005$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -21.81 | -21.72 | 0.84 | 80.97 | 0.28 | 1.42 (92%) | 88% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.01$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -22.12 | -22.95 | 0.85 | 81.14 | 0.30 | 1.40 (92%) | 89% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.01$, $\alpha_{\mathrm{div}}=0.00$) | 0.59 | -22.18 | -22.98 | 0.84 | 81.28 | 0.28 | 1.42 (92%) | 89% |  |
|  | RAFT ($\alpha_{\mathrm{KL}}=0.0$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -21.70 | -21.50 | 0.85 | 81.08 | 0.30 | 1.40 (92%) | 87% |
|  | RAFT ($\alpha_{\mathrm{KL}}=0.0$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -22.03 | -22.73 | 0.84 | 81.23 | 0.28 | 1.41 (92%) | 87% |
| 150-300 residues | Additional GRPO Results |  |  |  |  |  |  |  |  |
| GRPO ($\alpha_{\mathrm{KL}}=0.04$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -39.53 | -31.95 | 0.86 | 83.98 | 0.33 | 1.42 (89%) | 90% |  |
| GRPO ($\alpha_{\mathrm{KL}}=0.04$, $\alpha_{\mathrm{div}}=0.00$) | 0.57 | -40.40 | -32.15 | 0.85 | 84.05 | 0.29 | 1.43 (89%) | 90% |  |
| Additional RAFT Results |  |  |  |  |  |  |  |  |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.005$, $\alpha_{\mathrm{div}}=0.05$) | 0.57 | -36.61 | -28.26 | 0.84 | 83.24 | 0.33 | 1.43 (89%) | 88% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.005$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -36.79 | -28.86 | 0.83 | 83.53 | 0.30 | 1.44 (88%) | 88% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.01$, $\alpha_{\mathrm{div}}=0.05$) | 0.58 | -37.23 | -30.08 | 0.84 | 83.57 | 0.33 | 1.43 (89%) | 89% |  |
| RAFT ($\alpha_{\mathrm{KL}}=0.01$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -37.48 | -30.47 | 0.84 | 83.67 | 0.31 | 1.44 (88%) | 89% |  |
|  | RAFT ($\alpha_{\mathrm{KL}}=0.0$, $\alpha_{\mathrm{div}}=0.05$) | 0.57 | -36.53 | -27.65 | 0.84 | 83.46 | 0.33 | 1.43 (89%) | 87% |
|  | RAFT ($\alpha_{\mathrm{KL}}=0.0$, $\alpha_{\mathrm{div}}=0.00$) | 0.58 | -36.95 | -29.47 | 0.84 | 83.52 | 0.31 | 1.44 (88%) | 87% |

Table [5](https://arxiv.org/html/2506.07459v2#A3.T5) presents additional experimental results exploring different hyperparameter configurations for ProteinZero, specifically evaluating the impact of KL divergence coefficients ($\alpha_{\mathrm{KL}}$) and diversity regularization ($\alpha_{\mathrm{div}}$) on both ProteinZero${}_{\text{GRPO}}$ and ProteinZero${}_{\text{RAFT}}$ algorithms across two protein size categories (0-150 and 150-300 residues).

For ProteinZero${}_{\text{GRPO}}$, we test configurations with $\alpha_{\mathrm{KL}}=0.04$ (the original GRPO setting) and varying diversity regularization ($\alpha_{\mathrm{div}}\in\{0.00,0.05\}$). In the 0-150 residue category, the configuration with $\alpha_{\mathrm{KL}}=0.04,\alpha_{\mathrm{div}}=0.05$ achieves recovery rate of 0.58, TM Score of 0.86, sequence diversity of 0.31, and overall success rate of 89%, while removing diversity regularization ($\alpha_{\mathrm{div}}=0.00$) yields enhanced thermal stability (Pred-ddG: -22.50 vs -22.06, FoldX ddG: -24.55 vs -22.71) but significantly degraded sequence diversity (0.27 vs 0.31) and structural accuracy (TM Score: 0.85 vs 0.86), achieving 90% overall success rate. For 150-300 residues, both configurations reach 90% success rates, with $\alpha_{\mathrm{div}}=0.05$ providing superior sequence diversity (0.33 vs 0.29) and designability metrics (TM Score: 0.86 vs 0.85).

For ProteinZero${}_{\text{RAFT}}$, we examine configurations with $\alpha_{\mathrm{KL}}\in\{0.0,0.005,0.01\}$ and $\alpha_{\mathrm{div}}\in\{0.00,0.05\}$. In the 0-150 residue category, the best performing configuration ($\alpha_{\mathrm{KL}}=0.01,\alpha_{\mathrm{div}}=0.00$) achieves recovery rate of 0.59, thermal stability of Pred-ddG: -22.18 and FoldX ddG: -22.98, and 89% overall success rate. Weaker KL regularization with $\alpha_{\mathrm{KL}}=0.005$ consistently underperforms (88% success rate), while completely removing KL constraints ($\alpha_{\mathrm{KL}}=0.0$) further degrades performance to 87% success rate. For 150-300 residues, similar patterns emerge with $\alpha_{\mathrm{KL}}=0.01$ configurations achieving 89% success rates compared to 88% for $\alpha_{\mathrm{KL}}=0.005$ and 87% for $\alpha_{\mathrm{KL}}=0.0$. Importantly, removing diversity regularization consistently reduces sequence diversity across all configurations.

Despite these extensive explorations, all configurations in Table [5](https://arxiv.org/html/2506.07459v2#A3.T5) underperform our optimal settings reported in Table [2](https://arxiv.org/html/2506.07459v2#S4.T2), where $\alpha_{\mathrm{KL}}=0.1$ and $\alpha_{\mathrm{div}}=0.05$ achieve superior results: ProteinZero${}_{\text{GRPO}}$ reaches 90.13% and 91.19% overall success rates for 0-150 and 150-300 residues respectively, while ProteinZero${}_{\text{RAFT}}$ achieves 89.29% and 89.36%. These results demonstrate that stronger KL regularization and our embedding-level diversity regularization are essential for optimal protein design performance.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: The content in this section is identical to the points raised in Weaknesses. I would be very willing to increase my score if the authors successfully address these concerns.
soundness: 2
strengths: ProteinZero's primary strength lies in its innovative concept of linking protein generation with self-improvement via online Reinforcement Learning. This paradigm enables continuous model iteration, effectively overcoming the limitations of static, supervised training data. The methodology demonstrates diverse and practical attempts within the online RL framework itself, including a novel embedding-level diversity regularizer, and features a well-designed approach to multi-objective optimization effectively balancing designability, stability, KL-regularization, and diversity. Furthermore, the framework exhibits efficient execution and scalability; specifically, the use of the ESMFold + TM-score proxy reward makes the RL training loop computationally feasible while demonstrating strong performance gains on the computational benchmark environment.
summary: ProteinZero introduces a novel online Reinforcement Learning (RL) framework designed for the continuous self-improvement of protein inverse folding models. To achieve scalable optimization, the methodology employs efficient proxy reward models, including an ESMFold-based TM-score for fast structural fidelity assessment and a fast ΔΔG (ddG) predictor for thermodynamic stability. The authors demonstrate a technically sound approach to multi-objective optimization, effectively balancing conflicting goals such as designability, stability, KL-regularization, and sequence diversity. The framework shows compelling computational improvements on the CATH-4.3 benchmark.
weaknesses: 1. Computational predictors like Pred-ddG and FoldX ΔΔG inherently have limited correlation with ground-truth wet-lab experimental stability metrics. Whether used as a reward model for optimization or as a metric for evaluation, reliance on these tools introduces an unavoidable systemic bias, meaning the model may optimize for computational artifacts rather than true biophysical stability.

2. The paper's review of related work is incomplete, which complicates the assessment of the necessity and unique contributions of ProteinZero. The application of online RL for optimizing protein generation has been explored previously [1].The methodology and formal approach of these prior RL works appear conceptually very similar. Therefore, a comprehensive review and comparison to such existing RL-based efforts are required.

[1] Wang C, Uehara M, He Y, et al. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design[J]. arXiv preprint arXiv:2410.13643, 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. Many prior protein foundation model + RL post-training works focus on offline RL, but online RL makes more sense for tasks with reliable reward functions (e.g., self-consistency). I strongly support exploring online RL.
2. Strong results in designability metrics. Experiments are thorough and well-executed. Extensive ablation on reward functions and RL algorithms (offline vs. online).
summary: This paper introduces ProteinZero, an online RL framework for improving inverse folding using ESMFold self-consistency and a ddG predictor as rewards. Employs embedding-level diversity regularization to prevent mode collapse and analyzes RL design space.
weaknesses: 1. Thermal stability reward is interesting but a bit unclear to me in both formulation and interpretation.
    1. Validation: There’s no analysis showing how well the proposed reward correlates with true ddG values. 
    2. Formulation: Why introduce two models $p_\theta$ and $p_\varphi$ (prior) instead of using same model (like $\log p_\theta(y|x) - \log p_\theta(y)$ or $\log p_\varphi(y|x) - \log p_\varphi(y)$? Having both $p_\theta$ and $p_\varphi$ seems to make the optimization more complex (or even inaccurate) without clear justification. The paper mentions (line 227) that "backbone-conditioned likelihoods reflect folding stability," implying that $\log p_\varphi(y|x) - \log p_\varphi(y)$ can do the same job? 
    3. Interpretation: The ddG improvement from the base InstructPLM (Table 1) looks minimal. Given that the objective somewhat resembles DPO, I wonder if this formulation is truly improving stability or just adding some kind of a regularization effect.
2. I have a more fundamental question—Do we actually want to minimize ddG, or do we want to match the target (wild-type) ddG? Accurate inverse folding and stabilization feel like slightly different objectives. Both matter: finding stable, well-expressing sequences is important since all-atom generation and sequence co-design are still imperfect. But if the model is biased toward minimizing ddG too aggressively, it might fail for naturally flexible proteins where some instability is functional.
    1. What is the ddG of wild-type structures? Could you add that as a reference row in Table 1?
3. Can you provide some related works or context for the diversity regularization part for the readers? I like the embedding-level diversity idea, but I’m curious what similar approaches have been explored in the broader RLHF literature (not protein, I saw authors mentioned hamming distance idea in Line 143-144) for preventing mode collapse or promoting output diversity.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - On the first point of the weaknesses, while it is unrealistic to ask the authors to add multiple oracles during the review period, it would be great to see some discussion about this in the paper, and even better to get an estimate of the computational cost of such an addition.
- What is the correlation between your Fast-ddG predictions and actual experimental stability measurements?
- Why specifically use cosine similarity in embedding space rather than other distance metrics?
- How sensitive is performance to the choice of $\alpha_{div}$?
- You report means $\pm$ standard deviations, but over how many runs?
- Why are other RL methods such as ResiDPO not include in the experiments?
soundness: 3
strengths: **Originality**: I like the idea of an online RL Framework for this problem. Furthermore, the authors incorporate multiple novel elements to the architecture, such as the ddG predicted, and the regulizer

**Quality**: The results in the experiment section are very impressive, and carefully checked, through extra steps like the AlphaFold3 validation

**Clarity**: The paper is very well explained. The text is well organized and clear.

**Significance**: It is important to improve inverse folding, as the community has been reliant on PMPNN for a very long time. The insights into RLHF in general are also very interesting.
summary: This paper introduces ProteinZero, an online Reinforcement Learning (RL) fine-tuning framework for inverse protein folding models. The goal is to optimize multi-objective protein design metrics. Experiments on the CATH-4.3 benchmark demonstrate significant improvements over state-of-the-art baselines like ProteinMPNN.
weaknesses: - The structural designability reward relies solely on ESMFold. Although this is briefly discussed in the paper, and external validation is provided, over-optimization toward the biases or error modes of a single folding model remains a subtle risk. It would be interesting to add multiple oracles, to prevent such biases.
- While the model is very interesting, the limitation of being used only for monomers, does impact its usefulness for drug design. 
- There is a growing trend in the field towards fully atomistic 3D design, as opposed to backbone design, which overpasses the need for inverse folding. Of course, this does not mean inverse folding is not valuable (SOTA protein design models still design backbones) but the existence of full-atom design models is not even mentioned in the paper. 
- It would have been great to see a comparison of quality metrics for a 3D backbone design model (like RFDiffusion or FoldFlow), doing inverse folding with this method, or PMPNN; then folding and comparing true and predicted structures. This is, as far as I know, the main usage of inverse folding models, so it would have been good to see this model put to use in that way.


Decision: Reject