GLID$^2$E: A Gradient-Free Lightweight Fine-tune Approach for Discrete Biological Sequence Design
Authors: Hanqun Cao
Haosen Shi
Chenyu Wang
Sinno Jialin Pan
Pheng-Ann Heng
Venue: NeurIPS 2025 poster ()
Keywords: Deep learning
Protein design
Diffusion Model
Reinforcement learning
DNA design
Protein inverse folding
Primary area: machine_learning_for_sciences

Abstract: The design of biological sequences is essential for engineering functional biomolecules that contribute to advancements in human health and biotechnology. Recent advances in diffusion models, with their generative power and efficient conditional sampling, have made them a promising approach for sequence generation. To enhance model performance on limited data and enable multi-objective design and optimization, reinforcement learning (RL)-based fine-tuning has shown great potential. However, existing post-sampling and fine-tuning methods either lack stability in discrete optimization when avoiding gradients or incur high computational costs when employing gradient-based approaches, creating significant challenges for achieving both control and stability in the tuning process.
To address these limitations, we propose GLID$^2$E, a gradient-free RL-based tuning approach for discrete diffusion models. Our method introduces a clipped likelihood constraint to regulate the exploration space and implements reward shaping to better align the generative process with design objectives, ensuring a more stable and efficient tuning process. 
By integrating these techniques, GLID$^2$E mitigates training instabilities commonly encountered in RL and diffusion-based frameworks, enabling robust optimization even in challenging biological design tasks. In the DNA sequence and protein sequence design systems, GLID$^2$E achieves competitive performance in function-based design while maintaining computational efficiency and a flexible tuning mechanism.

GLID'E: Lightweight Policy-Based Fine-Tuning for Discrete Diffusion in Biological Sequence Design
Hanqun Cao Haosen ;hi'; Chenyu Wang?, Sinno Jialin Pan', Pheng-Ann Heng'
"hhe Chinese University of Hong Kong Massachusetts Institute of Technology
Abstract
Diffusion models have emerged as powerful tools for biological sequence design, offering flexible conditional generation for engineering functional biomolecules. While reinforcement learning (RL)-based fine-tuning enables multi-objective opti- mization on limited data, existing methods face a critical trade-off: gradient-free approaches suffer from training instability in discrete spaces, whereas gradient- based methods incur prohibitive computational costs. This trade-off severely limits their practical applicability in biological design tasks. We propose GLID'E, a light-weight gradient RI framework that achieves stable and efficient fine-tuning of discrete diffusion models. Our key insight to constrain the exploration space through clipped likelihood mechanism while employing reward shaping to align generation with design objectives. This combination mitigates the inherent insta- bilities in RL-guided diffusion while maintaining computational efficiency. We demonstrate GLID'E's effectiveness on DNA and protein sequence design bench- marks, where matches exceeds the performance of gradient-busee methods while requiring significantly lower computational resources. Our approach provides practical solution for diucctin-drivee biological sequence optimization. The code is available at: lhtps: / github. com/ chq1155/CLIIDE
Introduction
Designing biological sequences with desired functional properties is fundamental to protein engi- neering and synthetic biology [ []. Recent diffusion [ S 6. 7 and flow-matching models
8 10 [1] have shown impressive capability in modeling sequence distributions. However, adapt- ing these pretrained models for controllable, task-specific design remains challenging, particularly when limited experimental data inadequately captures sequence-funce funcciinn relationships and design objectives involve multiple competing criteria.
Two primary paradigms have emerged for functional sequence design: conditional sampling and fine-tuning. Conditional sampling methods [2, [3] guide generation online by steering the diffusion process toward desired properties. While conceptually straightforward, they incur additional inference costs and struggle to balance multiple objectives effectively. Fine-tuning methods [4 15] 16] [7
18] offer complementary approach by embedding functional knowledge into model parameters. DRAKES [14, representative gradient-based method, employs Gumbel-Softman to enable gradient flow through discrete trajectories and uses KI. regularization for distribution alignment. Although fine-tuned models generate functional sequences efficiently at inference without additional costs, gradient-basee training poses significant challenges: backpropagation through entire generation trajectories requires storing multiple intermediate states, leading to substantial memory overhead and computational burden. Moreover, terminal-only rewards provide limited guidance, missing opportunities to leverage intermediate information for finer generation control.
'HC and HS contributed equally. "Correspondence to: pheng@ www.ecccckc..c.e
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A)
(B)
<M-
<M+
<M+
6
Binding Activity
Sample
<M-
DNA
Update
Folding
Protrained
Diffusion
Reward Oracle
Stability
renard
GLID'E
Protein
reward
roward
rOwaRd
roward
Figure 1: Overview of the GLID2E framework. (A) GLID'E employs a lightweight reinforcement learning approach to fine-tune pretrained discrete diffusion models for generating functional biological sequences, specifically regulatory DNA elements and thermostable proteins. (B) The framework incorporates two key innovations: Clipped Likelihood Constraint that preserves sample rationality while allowing exploration, and Reward Shaping mechanism that provides informative signals throughout the diffusion process, enabling stable and lightweight fine-tuning.
We propose GLID"E (Gradient LIghtweight fine-tuning for Discrete sequence DEsign), a reinforce- ment learning framework that reformulates fine-tuning as policy optimization. Unlike gradient-baser based methods that backpropagate through full trajectories, GLID?E computes lightweight gradients only on policy parameters while treating the generation process as environment interactions. Our approach introduces two key mechanisms for stable and efficient optimization. First, we employ clipped like- lihood constraint that leverages the pretrained model's likelihood estimates to assess sample quality, avoiding explicit KI. divergence computation while preventing unreasonable sequence exploration. This constraint enables effective high-reward region exploration while maintaining distributional validity. Second, our reward shaping mechanism provides informative signals at intermediate gener- ation steps by evaluating partial sequences, guiding the policy toward promising regions earlier in the process. This contrasts with terminal-oni only reward methods, enabling more efficient learning and finer-grained control over generation trajectories.
We validate GLIDºE on DNA and protein sequence design benchmarks, demonstrating comparable or superior performance state-of of-thett- methods while achieving significantly faster training and inference with reduced computational requirements. Ablation studies confirm that both the clipped likelihood constraint and reward shaping contribute substantially to stability and performance. Our results suggest that GLIDDE provides a practical and scalable solution for function-driven biological sequence optimization.
Related Work
Discrete Diffusion Models for Biological Sequences Diffusion models have been successfully extended from continuous to discrete domains, enabling powerful generative modeling for biological
sequences 19. 20, 5 6 Recent advances have demonstrated strong performance in protein structure
generation 21 22), RNA design [23], and DNA sequence modeling (4].
Key technical challenges distinguish discrete diffusion from its continuous counterpart. First, the corruption process requires carefully designed categorical transition matrices rather than Gaussian noise 19 20), though auto-regressive variants (25 can sidestep explicit transition design. Second,
iterative categorical sampling incurs significant computational costs, particularly for long sequences. While accelerated sampling techniques (26) and consistency models 277 have improved efficiency, the trade-off between fidelity and computational cost remains when training data is limited common scenario in biological applications where experimental characterization is expensive.
Conditional Generation and RL Fine-Tuning Adapting pretrained diffusion models for task- specific objectives follows two primary paradigms. Training-free guidance methods steer the sampling process online through classifier guidance [28 29), Sequential Monte Carlo [3] or classifier-fres guidance [30]. These approaches offer flexibility without retraining but incur additional inference costs and struggle balance multiple competing objectives
31
3
limiting their effectiveness
for complex biological design tasks.

---

RL Fine-tuming methods provide an alternative by embedding functional knowledge directly into model parameters [8 [6]. DRAKES [14] employs gradient-based optimization with Gumbel- Softmax reparameterization and KI. regularization, achieving strong performance in biological sequence design. However, backpropagating through entire generation trajectories requires substan- tial memory and computation. More broadly. reinforcement learning has emerged as a powerful framework for optimizing generative models with non non-differentiable objectives [34] 35 36 33 37], particularly in text generation where reward-driven fine-tuning improves fluency and alignment. Re- cent work applies policy gradient methods to molecular and protein design 14 38
but efficiency
and stability challenges persist in dimennionn discrete spaces.
Our Approach GLID'E bridges these paradigms by reformulating fine-tuning as lightweight policy optimization. Unlike gradient-base -based methods that backpropagate through generation trajectories, compute gradients only on policy parameters while treating sampling as environment interactions. Our clipped likelihood constraint replaces expensive KL divergence computation with efficient likelihood-based filtering, while reward shaping addresses sparse reward signals without requiring intermediate gradient flow. This design achieves the parameter-embeeddd knowledge benefits of fine-tuning with computational efficiency approaching training-free methods, providing practical solution for funetion-driven biological sequence optimization.
Preliminary
3.1 Problem Formulation
We consider the task of adapting a pretrained generative model to produce sequences with high functional value. Formally, let X S fl...., .N"" denote the discrete sequence space, where N is the vocabulary size and n the sequence length. Our goal is to transform a pretrained diffusion model Ppeior (z) into optimized policy (2( that assigns higher probability sequences with high reward r(z). where r X + R is a reward function evaluating functional properties.
This setting is motivated by fundamental tension in biological sequence design: pretrained diffusion models, trained on large-scale real-world data, generate sequences that are structurally plausible but not necessarily functionally optimized for specific tasks. Conversely, reward models can evaluate task-specific properties but may assign high scores to invalid or unrealistic sequences when used directly for optimization. Our objective is to leverage both the distributional knowledge of the pretrained model and the task-specific guidance from the reward function.
3.2 Discrete Diffusion Models
We briefly review discrete diffusion models based on continuous tiriiuuouu-time Markov chains (CTMCs) [5 6) The forward process gradually corrupts sequence zo E X' into fully masked sequence FT over time &. E [0. 7] via a time-dependent transition rate matrix (t).. The transition matrix is typically handcrafted to ensure that at T, the sequence consists entirely of special MASK tokens, representing maximum corruption.
The generative model learns reverse this corruption process. A neural network parameterized by 0 is trained to approximate the reverse-time transition rates (!), enabling ancestral sampling from the masked state ør back to realistic sequences To. The reverse process follows the time-reversed CTMC:
drr-t Qo( (T f)xr-t
di
where the model learns predict appropriate tokens to replace MASK symbols each timestep.
Throughout this work, we assume the pretrained diffusion model operates on fixed-length sequences and is unconditional, generating samples x Pp)io((:) without additional conditioning inputs. Our method fine-tunes this pretrained model to incorporate functional objectives while preserving its ability to generate valid sequences.
3

---

Table 1: Comparative analysis of biological sequence design methods, evaluating key features and computational requirements. GLIDEE uniquely combines light-weight policy optimization with preserved sequence naturalness, while achieving lower training complexity (UN-BBT--LA vs. O.N.B.T- for comparable methods). Notation: N = training iterations, B = batch size, = diffusion steps, L sequence length, d = model dimension, P = number of particles.
Feature Comparison
Computational Cost Comparison
Method
Requires Gradients
Preserves Naturalness
Theoretical Guarantees
Training
Inference
Memory
CG
O(1) 0(1)
O(d)) O(P-L-d ) O(P-I-d+ O(d)) O d) +(
SMC
O(P.T-L-d)
TDS
J
X
X
0(1)
O(P-T-I-d')
CFG
X
X
O(N-B-T-L-dP)
O(T-L-d)
DRAKES
O(N-B-T-L-d))
O(T-L-d)
GLID
X
X
O(N-B-T-L
O(T-L-)
Method
We present GLIDEE (Gradient LIghtweight fine-tuning for Discrete sequence DEsign), reinforce- ment learning framework that efficiently adapts pretrained discrete diffusion models for functional sequence design. Unlike pradient-based methods that backpropagate through entire generation tra- jectories, GLID"E treats the diffusion sampling process as an RL environment and optimizes only the policy parameters with lightweight gradients. Our framework addresses two fundamental chal- lenges: (1) preventing policy collapse while maximizing rewards, and (2) overcoming sparse terminal rewards through intermediate guidance. We introduce three core components: clipped likelihood constraint for rationality preservation (Section [41]), reward shaping for informative intermediate signals (Section 4.2). and PPO-based optimization that integrates these mechanisms (Section 4.3)
4.1 Clipped Likelihood Constraint
The Standard KL-Regularized Objective. Directly maximizing expected rewards in RL often leads
to training instabilities and policy collapse [8 40) A standard approach employs KL divergence regularization to constrain the optimized policy pa near a reference policy Pprior:
maxE [r(=)] - KL(Poll (Pprior))))))
(2)
where B > 0 controls the regularization strength. The optimal solution to Equation form:
2 takes the
((x)
pe- (x) x Perior (1) exp
This formulation presents a fundamental trade-off. Large 3 yields conservative policies that closely follow the prior but achieve limited reward trained on diverse task-agnostic data (c.g., entire proteomes or multi-species genomes), lacks task- specific inductive biases. Conversely, small permits aggressive exploration but risks generating invalid sequences when reward models exhibit pathologies, such as assigning artificially high scores sequences with excessive hydrophobic regions while ignoring critical functional constraints.
when the prior distribution,
Rethinking the Constraint. The optimal policy in Equation (3) uniformly mixes the prior and neward-induced distributions across all sequences. However, argue that rationality should be enforced as a hard constraint rather than softly blended with reward optimization. Pretrained diffusion models, trained on extensive real-world datasets, inherently capture distributional validity and can effectively assess sequence plausibility through likelihood estimates. Meanwhile, over-reliance on the prior may unnecessarily restrict exploration of high-reward regions, particularly for task-specific objectives where the prior provides limited guidance.
This motivates reformulating the optimization as:
max #
Eg-po [r(z)] - a Hpo)
(4)
subject to po(x) > only if Pprior(x) 2 e,

---

where > 0 is a likelihood threshold defining the rationality boundary, and the entropy term (po) -r [log Po(x)] encourages policy diversity. This formulation restricts the policy's support to sequences deemed plausible by the pretrained model while allowing the reward function to dominate within this constrained space.
Practical Implementation via Likelihood Clipping. Solving Equation (4) directly is intractable. We propose a practical approximation by modifying the reward function to penalize sequences with low prior likelihood. We first generate a calibration set of Real samples from the pretrained model and compute the empirical mean μ and standard deviation a of their oogliiklihoods. The threshold is set as c = μ - ko, where k 2 0 controls tolerance (we use k = 1 in experiments). This calibration leverages the pretrained model's confidence: samples within one standard deviation are considered plausible.
The modified reward function incorporates a likelihood-based penalty:
log Pprior (z) + kr 0)
F(x) r(x) + min
(5)
where the min(;, 0) operator activates penalties only for sequences below the threshold (i.e., log (x) < p=-ko, and B > 0 controls penalty strength. The standardization (log Ppesor (z)- ensures robustness across different models and tasks by normalizing likelihood scales.
Using f(x) as the reward, we optimize the policy via standard RI without explicit KI. regularization. This approach provides a computationally efficient alternative that avoids expensive KI. divergence calculations while effectively constraining the policy to the high-likelihood region of the pretrained distribution.
4.2 Reward Shaping
Motivation. Standard RL formulations for diffusion models provide rewards only upon complete sequence generation, resulting in sparse signals that slow learning. To accelerate optimization and provide finer-grained guidance, we introduce reward shaping [1), a technique that assigns intermediate rewards while preserving the optimal policy.
RL Formulation. Following [4], we formulate diffusion sampling as a Markov Decision Process (MDP). The state space comprises partially denoised sequences s, Iy at timestep t E {O...., Th, .7). where Zp is fully masked and øy is the final sequence. At each step, the diffusion model performs a denoising action by sampling from To T0(-(2(-1, t). transitioning from 8t-1 to st. The standard reward structure assigns zero rewards to all intermediate transitions (r(s,) = for t < T) and provides the final reward ((TT) F(xT) only at termination. The discount factor is 7 1.0.
Potential-Baze -ased Shaping. We employ potential-based reward shaping which modifies rewards via a potential function क S + RR as:
r r((8-1,81) r(s st) + ((t) - 4 (s:
This formulation guarantees that the cumulative return remains unchanged: Σt= (1 sp) r(8T) + ((7)) ((0).. By setting $(sn) (ST) 0, the optimal policies under and r' coincide.
Handling Masked Tokens. A key challenge is defining (s;)) for intermediate states containing MASK tokens, which the reward model cannot evaluate directly since it was trained only on complete sequences. We address this by completing partial sequences through Monte Carlo sampling: for each intermediate state &; with Amask remaining masks, we sample Nème completions by independently replacing each MASK token according the current policy vo(-(z,,t), with the MASK token probability set to zero. The potential function is defined as:
E cap complee (I
1st<
((x))
=
1= 0 0
where x complete denotes the completion distribution. In practice, we approximate the expectation using Tmme samples.
This design provides increasingly accurate reward estimates as denoising progresses: early states with many masks yield coarse estimates, while near-complete sequences provide precise signals. The
5

---

shaped rewards guide the policy toward promising regions throughout the trajectory, accelerating learning compared terminal-only feedback.
4.3 PPO-Based Policy Optimization
We integrate the clipped likelihood constraint and reward shaping within Proximal Policy Opti- mization (PPO) framework 42 a widely adopted policy gradient method known for stable training. PPO employs a clipped surrogate objective prevent excessively large policy updates that could destabilize training.
Mixed Reference Policy. To maintain proximity to the pretrained diffusion model while allowing reward-driven exploration, we define a mixed reference policy that interpolates between the pretrained policy peior and the current policy nea
log (x = v log (8e + - v) log (8c
(8)
where 77 E [0. 1] controls the mixture weight. This design creates a "soft anchor" to the pretrained distribution: higher enforces stronger adherence to the prior, while lower grants more exploration freedom. During online sampling, we record logits from both policies to compute nee efficiently.
Clipped Surrogate Objective. The PPO objective clips importance ratios to bound policy updates:
CLIP (0) = Er,.i min (pcAr. clip(pe, 1 e,
(11() (s the importance weight, Åg is the advantage estimate com- puted via Generalized Advantage Estimation (GAE) (43), and > 0 is the clipping threshold (set to 0.5 in our experiments, corresponding to clipping ratio of 1.5). The clipping mechanism prevents large policy shifts while avoiding expensive KL divergence computations.
where P: (o((s)s(-1)) Teer
Combined with entropy bonus to encourage exploration, the final objective becomes:
[(0)- = I.CLI(()
(0) + (HTo).
(10)
where >> O controls the entropy coefficient. Together with the clipped likelihood constraint (Equation 5) and reward shaping (Equation (6) this framework achieves efficient and stable fine- tuning of discrete diffusion models. The complete algorithm is provided Appendix A.4 with hyperparameter details in Appendix A.I
5 Experiments
We evaluate GLID'E on two biological sequence design benchmarks: regulatory DNA enhancer design and thermostable protein design. Our experiments demonstrate that GLID'E achieves state- of-the-art or competitive performance while maintaining computational efficiency. We conduct comprehensive ablation studies to validate the contribution of each component and analyze the impact of key hyperparameters
5.1 Experimental Setup
Datasets and Models
Following DRAKES 14. we use established benchmarks for both tasks.
DNA Design. We use a comprehensive enhancer dataset containing approximately 700,000 DNA sequences of 200 base pairs 44. characterized for activity in human cells via massively parallel reporter assays. The discrete diffusion model follows 45), while the reward oracle adopts the Enformer architecture 46 trained to predict activity in the HepG2 cell line.
Protein Design. The discrete diffusion model is pretrained on 19,700 high-resolution single-chain structures from PDB, following ProteinMPNN (2). The reward oracle is trained on the Megascale dataset (1.8M sequences across 983 designed domains) to predict thermodynamic stability measured by Gibbs free energy change (AAG). Both models use the ProteinMPNN architecture.

---

Evaluation Metrics
We use metrics for both functional performance and naturalness preservation.
DNA Design. Functionality is measured by Pred-Activity (predicted enhancer activity) and ATAC-Ace (chromatin accessibilityy [47 48 Naturalness quantified via 3-mer Correlation (similarity to natural k-mer distributions) and Log-Likelihood under the pretrained model
44
Protein Design. Stability is assessed by Pred-ddG (predicted AAG) and %(ddG>> (percentage of stabilizing sequences). Structural naturalness is evaluated using scRMSD (self-consistency RMSD between ESMFold [49] predictions from sequence and .ock-translation. We define Success Rate as the percentage of sequences satisfying both ddG>>0 and scRMSD
Baselines We compare against the pretrained diffusion model, conditional sampling methods (Classifier Guidance [29), SMC [13], TDS [50), and Classifier-Frss Free Guidance (30)), the MCTS-based tree method PepTune [1]] and the gradient-base based fine-tuning method DRAKES [4].
Implementation Details All experiments are conducted on single NVIDIA A40 GPU with 20GB memory. We use multiple GPUs for parallel runs across different random seeds. Hyperparameters detailed in Appendix A.I
5.2 DNA Sequence Design
Table 2:
General performance for DNA sequence design models. State-of-the-arl performance is bold, and the second-highest performance is underlined. KL, M1, and M2 denote KL regularization, reward shaping, and likelihood penalty, respectively.
Method
Pred-Activity (median) 1 ATAC-Ace (%)
3-mer Corr
Log-Lik (median)
Pretrained
0.17(0.04)
1.5(0.2)
-0.061(0.034)
-2.1(0.6)
CG
3.30(0.00)
0.0(0.0)
-0.65(0..00))
-266(0.6)
SMC
4.15(0.33)
39.9(8.7)
0.840(0.045)
-259(2.5)
TDS
4.64(0.21)
45.3(16.4)
0.848(0.008)
-257(1.5)
CFG
5.04(0.06)
92.1(0.9)
0.746(0.001)
-265(0.6)
DRAKES
5.61(0.07)
92.5(0.6)
0.887(0.002)
226((0.6)
GLIDÉE
7.29(0.162)
98.4(0.67)
0.49(0.074)
-240.933(3.7)
GLID'E w/o MI
2.57(0.60)
0.63(0.3)
0.473(0.078)
-239.1((10.07)
GLID'E w/o M2
6.62(0.42)
67.3(39.4)
0.458(0.009)
-244.65(21.5)
Main Results Table presents the DNA design results. GLID'E achieves the highest median Pred-Activity (7.29). substantially outperforming DRAKES by 30% and demonstrating superior functional optimization. Our method also attains the best ATAC-Acc (98.4%), indicating that generated sequences exhibit high chromatin accessibility, key indicator of functional enhancer activity in biological contexts.
Regarding naturalness metrics, GLID'E achieves competitive Log-Likelihood (-240.9), second only to the ablation variant without MI, confirming that our clipped likelihood constraint effectively preserves distributional validity. However, GLID"E exhibits lower 3-mer correlation (0.49) compared to DRAKES (0.887) and conditional sampling methods. This discrepancy reveals an important finding: the pretrained model's 's learned distribution diverges from conventional k-mer statistics of natural enhancers. Rather than indicating lower quality, this suggests GLID'E discovers functionally equivalent but compositionally distinct sequence variants that satisfy the model's 's likelihood con- straints while achieving superior predicted activity. This exploration beyond traditional motif patterns potentially expands the design space for functional enhancers.
2 exhibits lower variance in activity metrics (Pred-Activity std: 0.162) compared to natu- ralness metrics (Log-Lik std: 3.7), suggesting convergence toward reward-optimized regions while maintaining diversity in sequence-level characteristics.
The ablation experiments validate our design choices. Removing reward shaping (w/o MI) causes dramatic performance drops: Pred-Activity decreases 2.57 (65% reduction) and ATAC-Ace collapses to 0.63%, demonstrating that intermediate reward signals are critical for guiding
Ablation Study
7

---

the policy toward functional regions during early generation steps. Notably, Log- Likelihoedhd remains comparable (-239.1), indicating that the model still generates valid sequences but fails to optimize for functionality without shaped rewards.
Removing the likelihood constraint (w/o M2) maintains reasonable activity (Pred-Activity: 6.62) but reduces ATAC-Acc to 67.3% and degrades Log- Log-Likelihood to -244.7, confirming that the clipped likelihood mechanism preserves both distributional validity and biologically relevant sequence characteristics. This validates our hypothesis that the pretrained model's likelihood estimates capture essential naturalness constraints beyond simple k-mer statistics.
5.3 Protein Sequence Design
Table 3:
General performance for protein sequence design models. State-of-the-- performance is bold, and the second-highes performance is underlined. KI, M1, and M2 denote KL regul reward shaping, and likelihood penalty, respectively.
Pred-ddG
%ddGG > 0)
scRMSD
4(scRMSD <
Success Rate (%) f
Method
(median) f
(%) f
(median) 4
(%) f
Pretrained
-0.544(0.037)
36.6(1.0)
0.849(0.013)
90.9(0.6)
34.4(0.5)
CG
(0.561
36.9(1.1)
0.839(0.012)
90.9(0.6)
34.7(0.9)
SMC
0.659(0.044)
68.5(3.1)
0,841(0.006)
93.8(0.4)
63.6(4.0)
TDS
0.674(0.086)
68.2(2.4)
0.804(0.001)
94.4(1.2)
62.9(2.8)
CFG
-1.186(0.035)
11.0(0.4)
3. 3.1460).062)
29.4(1.))
1.3(0.4)
DRAKES
1.095(0.026)
86.4(0.2)
0.918(0.006)
91.8(0.5)
78.6(0.7)
PepTune
0.432(0.003)
94.0(2.0)
1.04((0.057)
87.3(4.0)
70.1(1.2)
GLID
1.012(0.094)
86.7(2.4)
0.961(0.047)
89.2(1.2)
76.7(1.2)
GLID'E w/o M1
0.843(0.099)
75.3(2.5)
0.950(0.074)
85.0(30)
62.0(3.4)
GLID"E w/o M2
0.893(0.170)
80.6(3.9)
0.970X0.049)
85.8(5.1)
67.5(4.4)
Main Results Table 3 shows protein design results. GLID'E achieves competitive stability perfor- mance with Pred-ddG of 1.012 (second to DRAKES' 1.095) and the highest %(ddG>0) at 86.7%, demonstrating that RL-based fine-tuning can match gradient-based methods for thermodynamic opti- mization. Peptune exhibits more concentrated ddG distribution, and RMSD distribution shows similar performance to DRAKES and GLID'E. The success rate of 76.7% approaches DRAKES' 78.6%, confirming the viability of our lightweight gradient approach. The
GLID exhibits slightly higher scRMSD (0.961) compared to conditional sampling methods like TDS (0.834), reflecting broader exploration of sequence space. We note that scRMSD, while informative, provides a less direct naturalness measure than DNA's Log- Likg-Liielioood because ESMFold- predicted structural similarity may not fully capture sequence- level naturalness a minor limitation of the evaluation framework. The higher variance across GLIDdE's metries (e.g.. Pred-ddG std: 0.094 vs. DRAKES' 0.026) further confirms this exploratory behavior, consistent with RL methods' tendency to sample more diverse regions of the design space.
Ablation Study Removing reward shaping (w/o M1) reduces Pred-ddG to 0.843 and success rate to 62.0%, though performance still exceeds all conditional sampling methods. This confirms that iterative RI.bbased optimization accumulates more effective learning compared to one-shot conditional generation, and that shaped rewards effectively address sparse reward signals by providing informative gradients throughout the generation trajectory.
Removing the likelihood constraint (w/o M2) causes consistent degradation across all metrics (Pred- ddG: 0.893, success rate: 67.5%) Since the pretrained model was trained on natural, thermostable PDB structures, sequences closer this distribution inherently achieve better stability and lower scRMSD. This validates that the clipped likelihood constraint serves dual purposes: preventing invalid sequence generation while implicitly guiding toward stable, natural structures. Unlike DRAKES' KL regularization, which uniformly constrains the entire distribution, our clipping mechanism permits controlled deviation (within one standard deviation), effectively filtering irrational outliers while allowing exploration of high-rewatt regions.
8

---

5.4
Hyperparameter Analysis
Table 4:
Ablation study for clipping ratio in protein sequence design models. State-of-the-ae
performance bold, and the second-highed performance underlined.
Pred-ddG (median)
%
ScRMSD
%(scRMSD 2)
Success Rate
Clipping Ratio
(%) f
(median) +
(%)
(%) f
0.5
0,402(0.147)
63.6(0.5)
1002((0.083)
87.0(1.3)
53.9(1.3)
1.0
(.824(0.221)
75.4(11.2)
0.988(0.061)
81.7(5.1)
60.5(1.8)
1.5
1.012(0.094)
86.7(2.4)
0.961(0.047)
89.2(1.2)
76.7(1.2)
2.0
0.932(0.144)
76.4(2.7)
0.902(0.041)
78.2(3.5)
64.7(2.5)
Clipping Ratio Table 4 shows the effect the clipping ratio (recall that the likelihood threshold 4-ko where & is the ratio). Stability metrics (Pred-ddG) initially increase then decrease: performance peaks at ratio 1.5, then declines at 2.0. This non-monotonic trend reveals a fundamental trade-off. Lower ratios (0.5-1.0) impose stricter likelihood constraints, limiting exploration of stable sequences beyond the pretrained distribution. Higher ratios (2.0) relax constraints excessively, allowing the policy venture into regions where the pretrained model's likelihood estimates become less reliable.
Interestingly, as the clipping ratio increases, median scRMSD decreases (s<cMM)D<<2) also decreases (87.0%~>78.2%) indicating emergence of bimodal distribution: tighter likelihood constraints compress the policy toward the prior, yielding more sequences with excellent structural consistency, but also producing outliers with degraded structures. The high variance at ratio 1.0 (Pred-ddG std: 0.281, (ddG>>)) std: 11.2) reflects this transition point where the model struggles to balance reward optimization and distributional validity.
while
Table 5: Ablation study for mixture ratio in protein sequence design models. State-of-the of-the-art performance bold, and the second-highest performance underlined.
Pred-ddG (median) f
% > 0)
scRMSD
(scRMMS 2)
Success Rate
Mixture Ratio
(%)
(median) $
(%) f
(%) f
0.01
1.012(0.094)
86.7(2.4)
(0,000))
89.2(1.2)
76.7(1.2)
0.1
0.255(0.151)
61.7(3.5)
0.898(0.035)
87.5(2.4)
51.7(1.2)
1.0
-0.302(0.023)
(0.6(
0.844(0.008)
93.4(1.2)
41.3(0.6)
Mixture Ratio Table examines the mixture ratio in the reference policy (Equation 8). Higher values enforce stronger adherence to the pretrained policy, resulting in progressively conservative behavior: 1 1.0, performance degrades Pred-ddG of -0.302 and success rate of 41.3%, barely improving over the pretrained baseline. This confirms that excessive regularization prevents effective reward optimization.
Conversely, low y (0.01) permits aggressive exploration, achieving the best stability metries (Pred- ddG: 1.012, success rate: 76.7%). The intermediate ratio (1) = 0.1) exhibits substantially elevated variance across all metrics (c.g., Pred-ddG std: 0.151). similar to the clipping ratio analysis at 1.0. This suggests that moderate regularization creates ambiguity in the optimization landscape: the policy oscillates between prioritizing reward signals and adhering to the prior, leading to bimodal behavior that generates either highrdeward or nathraaarrlnsss sequences without effectively balancing both objectives. These hyperparameter studies reveal that GLS performance is robust within reasonable ranges (clipping ratio: 1.5-2.0, mixture ratio: 0.01-0.1), with clear indicators (elevated variance) when hyperparameters approach suboptimal regimes.
6 Conclusion
We present GLID"E, a reinforcement learning framework that adapts discrete diffusion models for functional biological sequence design through two key innovations: a clipped likelihood constraint that preserves distributional validity without expensive KI. computation, and reward shaping that provides intermediate guidance throughout generation. These mechanisms enable stable and efficient

---

fine-tuning while maintaining parameter- knowledge embedding. Experiments demonstrate that GLID"E matches or exceeds ofate-of-the-ar performance on DNA enhancer and protein design benchmarks. Our method achieves 30% activity improvement on DNA tasks and competitive stability (Pred-ddG: 1.012, success rate: 76.7%) on protein tasks, while requiring significantly lower computational resources. Ablation studies confirm both mechanisms contribute substantially to performance. GLID'E's computational efficiency and modular design make it well-suited for multi- objective optimization, longer sequences, and data-scarce domains, providing practical foundation for function-driven biological sequence design.
Acknowledgements The work described in this paper was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N; and in part by Guangdong-How Hong Kong-Macao Joint Laboratory of Hluman-Maching Systems. Haosen Shi and Sinno J. Pan thank the support from the JC STEM Lab of Machine Learning and Symbolic Reasoning funded by The Hong Kong Jockey Club Charities Trust
Mielligence-Syneryy
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science, 778(6615):49-33) 2022.
[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, (116840-851)) 2020.
[4] Yang Song, Jascha Sohl-Diekstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.134555 2020.
[5] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 15:28266-28273 2022.
[6] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.
[7] Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprini arXiv:2410.21357 2024.
[8] Fang Wu, Tinson Xu, Shuting Jin, Xiangru Tang, Zerui Xu, James Zou, and Brian Hie. D-flow: Multi-modality flow matching for d d-peptide design. arXiv preprint arXiv:2411.
2024.
[9] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. In International Conference on Machine Learning, pages 46495-46513 PMLR, 2024.
[10] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference on Machine Learning, pages 5453-5511 PMLR, 2024.
[11] Chaoran Cheng, Jiahan Li, Jiajun Fan, and Ge Liu. a-flow: A unified framework for continuous- state discrete flow matching models. arXiv preprint arXiv: 10283, 2025.
[12] John Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng-Thow-Hing, Erik R Van Vlack, et al. Illumi- nating protein space with a programmable generative model. Nature, 0770-777) 2023.
10

---

[13] Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations.
[14] Chenyu Wang, Masatoshi Uehara. Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. arXiv preprint arXiv:24J0. 13643, 2024.
[15] Jarrid Rector-Brooke Mohsin Hasan, Zhangzhi Peng. Zachary Quinn, Chenghao Liu, Sarthak Mittal, Nouha Dziri, Michael Bronstein, Yoshua Bengio, Pranam Chatterjee, et al. Steering masked discrete diffusion models via discrete denoising posterior prediction. arXiv preprint arXiv:2410.08134, 2024.
[16] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885, 2023.
[17] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[18] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17000 2023.
[19] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 19:1798--17993 2021.
[20] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:12454- 2021.
[21] Kevin E Wu, Kevin Yang, Rianne van den Berg, Sarah Alamdari, James Zou, Alex Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature communications, 15(1):1059, 2024.
[22] Xinyou Wang, Zaixiang Zheng, YE Fei, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. In Forty-firsi International Conference on Machine Learning.
[23] Han Huang, Ziqian Lin, Dongchen He, Liang Hong, and Yu Li. Ribodiffusion: ter- tiary structure-based rna inverse folding with generative diffusion models. Bioinformatics, 2024.
[24] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328-43444 2022.
[25] Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 96:39957-39977 2023.
[26] Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusson models with self-conditioning. In The Eleventh International Conference on Learning Representations.
[27] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference Machine Learning, pages 32211-3222 1-32252. PMLR, 2023.
[28] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
11

---

[29] Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. CoRR, 2024.
[30] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
[31] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing 2023.
Systems,
[32] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[33] Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine Fine- tuning of continuous-ting diffusion models as retropy-regularized control. arXiv preprint WxXi::2402.15194 2024.
[34] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[35] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rXiv:24/2.15115. 2024.
[36] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12444 2025.
[37] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[38] Masatoshi Uchara, Xingyu Su, Yulai Zhao, Xiner Li, Aviv Regev, Shuiwang Ji, Sergey Levine, and Tommaso Biancalani. Reward-guided iterative refinement diffusion models at test-time with applications to protein and dna design. arXiv preprint arXiv:2502.1 11944,444444 2025.
[39] Andrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi S Jaakkola, Regina Barzilay, and lla R Fiete. Improving protein optimization with smoothed fitness landscapes. In ICLR, 2024.
[40] Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine Understanding reinforcement learning-basee based fine-tuning of diffusion models: A tutorial and review. arXiv preprint 23771:2407.13734, 2024.
[41] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In Icml, volume 99, pages 278-287, 1999.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.v:1777.0633 2017.
[43] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02448,, 2015.
[44] fachine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023.
[45] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, Advances in Neural Information Processing Systems, 17:113116-1330 130184, 2024.
12

---

[46] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam. Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 196-1203, 2021.
[47] Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. Designing realistic regulatory dna with autoregressive language models. Genome Research, 34(9):1411-1420, 2024.
[48] ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414):57, 2012.
[49] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary scale prediction of atomic-level protein structure with language model. Science, 379(6637)11123 1130, 2023.
[50] Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John Cunningham. Practi- cal and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36:31372-31403, 2023.
[51] Sophia Tang, Yinuo Zhang, and Pranam Chatterjee. Peptune: De novo generation of therapeutic peptides with multi-objeecive--uidee discrete diffusion. ArXiv, pages arXiv-2412, 2025.
13

---

A Appendix
A.1 Implementation details
Common dyperparameters: In our implementation, we utilized several key hyperparameters that were common across experiments. These parameters control various aspects of our reinforcement learning approach, particularly focusing on reward handling and likelihood clipping mechanisms.
For reward processing, employed parameters such likelihood penalty scale adjust the strength of likelihood constraint / in Section 4.1 and multiple heipleeeeeard__.._eeeppppp _ew.arddsaappllnng. to determine the number of multiple reward sampling for reward shaping in Section 4.2 For our clipped policy optimization, we use clip clip_mix__ Section The complete set of common hyperparameters and their values is presented in the table
and clip_ threshold to control the mixture, described in
below:
Table 6: Common hyperparameter configuration utilized across experiments.
Parameter
Setting
likelihood penalty scale
0.1
multiple_reward__e reward_sampling thr_shhrdd @lip_mix__ factor
1.5
0.01
Hyperparameters in DNA experiment: For our DNA experiments, we configured reinforcement learning parameters including GAE's À = 0.95, discount factor (- = 0.99), and learning rate (Ie 4).
We employed gradient norm clipping (1.0) and exponential moving average decay (0.999) to enhance training stability.
Table Hyperparameter configuration utilized in our DNA experiments.
Parameter
Setting
batch_size
decay
0.999
learning_rat rate
le-4
À in GAE
0.95
y
0.99
gradient_ norm norm_clip
1.0
cumbeel_temeerturre
1.0
entropy_scii scale
1e-3
Hyperparameters in Protein experiment: Our protein experiments used 3 encoder and decoder layers with hidden dimension 128 and 30 neighbors for graph representation following [14. We set the learning rate to 3e-5 with weight decay le-4 and used diffusion process with 50 timesteps.
A.2 Training time comparison
We compared the training times of DRAKES and GLID"E, shown in 9 The light-weight scheme enhances the algorithm's ' efficiency. We achieved consistent results with the training cost in Table |
A.3 Sequence diversity analysis
We further tested the sequence diversity of different baselines based entropy (Table TOP. The results showed that all baselines exhibited a decline, with GLID'E and CG achieving performance closest that of the pretrained model.
14

---

Table 8: Hyperparameter configuration utilized in our protein experiments.
Parameter
Setting
batch_size
16
hidden_din
128
cuin_encoder_aape
layers
3
num Lum_decoder layers num neighbors
3
30
dropout
0.0
backbone_noiie noise
0.1
gradient_norm_cli
1.0
learning_rate rate weight_decay
3e-5
le-4
temperature
0.1
X in GAE
0.95
sii__ttmestes
50
gumbel_softmax softmax temperature
0.5
entropy_ sccle
le-3
Table 9: Training time per epoch for different methods.
Method
DRAKES
GLID'E (Ours)
Time
23.28 + 0.14
13.54 = 0.04
A.4 Detailed Training Algorithm of GLID
15

---

Algorithm 1 E Training Algorithm based on PPO algorithm
1: Input: Policy network ps, Value network V&. Training epochs K, Advantage estimate Å.,
Clipping parameter
2: Initialization: Policy network parameters and value network parameters
3: while Termination condition not met do
4: Collect N trajectories т; (St,e, as,e, log p(g,((s where i 1, ,....,, N
5: Calculate indicate states 2,,t
6: Calculate reward ri, (x%,,111) $(x7e)) where 1. .,...,
7: and ri, r(a,,r (x + 3 min (og + ki 0)
8:
Compute advantage estimates As,e for each trajectory via GAE for 1 to K do
9:
for Each mini-batch B containing M samples do Compute the policy loss LCLIP(!)
11:
CCLIP(0) -E, min Poo(alss))
A., clip( Po(ar)st)
Pooe (up(s)) - + +))..)
PPoa (at st)
12:
Compute the value loss Cvr(Ø)
Lvr(Φ) (vr(Ø)=Ba Re [(o(6t)- - -v:)
where Vr is the estimated value Compute the entropy bonus S(0)
13:
S(0) = Eg [H (TT(11s)))
where H. is the entropy function 14:
Compute the total loss L(B, 6)
L(0, Φ) = (CLIr(0) + (vr(0) c2S(
where ej is hyperparameter Update policy network parameters 0 and value network parameters 6:
15:
0 - 0 og V (f)
o as C(0,
where og and ag are learning rates end for
16:
end for
18: end while
16

---

Table 10: Diversity results based on sequence entropy.
Method
Sequence Entropy 1
Pretrained
34.7
CG
34.6
SMC
24.9
TDS
24.9
CFG
8.4
DRAKES
33.3
GLID&E
34.6
B Discussions
B.1
Limitations
While GLID"E offers lightweight yet competitive approach for DNA and protein sequence design, several limitations warrant acknowledgment. Algorithmically, reinforcement learning enables broader design space exploration but lacks the theoretical guarantées of methods like DRAKES, potentially yielding less stable fine-tuning and requiring more extensive hyperparameter tuning. Additionally, though presented as a comprehensive framework, GLID'E requires validation across broader bio- logical systems, including Higand-binding proteins, enzymes, antibodies, and RNA sequences, to fully demonstrate its robustness and versatility. Finally, our reliance on in silico validation cannot defiiitielly establish real-world efficacy. Future wet lab experiments are essential address this limitation and advance practical biological sequence design.
B.2 Broader Impact
Our framework presents a versatile algorithm with broad applications in drug discovery systems, potentially accelerating therapeutic development and expanding discovery frontiers. However, this technology also poses misuse risks in designing harmful biological entities (proteins, RNA. DNA). Furthermore, Al-generated biological constructs raise important questions regarding intellectual property rights, patents, and ethical considerations that must be addressed the field advances.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract section and the introduction section clearly state the range of application of this research. Also, it addresses the current challenges as well as the innovative designs. Also, experiments demonstrate the contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The related limitations are well stated in Appendix B.1
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: Our proposed method contains full set of assumptions based on mathematics. However, the proposed method does not have theoretical guarantee, leading to proof in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All the experimental results can be reproduced. We have released the details for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: In current stage, we decide not to release the code and data due to anonymity preservation. Nevertheless, all the related materials will be released when the research work is accepted by this conference.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mniissccc/public//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the corresponding details introduced in Section 5 and Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We have multiple runs with different seeds valid the robustness of our methods. However, we have not prepared statistical signifance analysis. We acknowledge it as minor limitation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources are well introduced in Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeuriip..pe cc.p
Answer: [Yes]
Justification:
The Code of Ethics are all followed in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The related positive and negative impacts are well presented in the Appendix B.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All related models and data pose minor risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets properly cited, and with correct attribution to their creators. Also, all assets follow licensing requirements well.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All related new assets are well documented, including the pretrained model and evaluation scipts.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not utilized for any core component of this research work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttppr////euups..... http///rips.cccoooffeereeeee/e// Cc/Conferrecces/2255//L. for what should or should be described.
24


Reviews:
[{"review_id": "nep44ERgFF", "reviewer": "Reviewer_TFcp", "summary": "In this work, the authors present a method to fine-tune discrete diffusion models to output biological sequences that are more desirable according to an oracle model using reinforcement learning (RL). Their method, called GLID2E, uses PPO-based policy optimization to shape the distribution of the generated sequences to maximize a reward function. The two main innovations of this method are in designing the reward function. First, to avoid generating sequences that are too different from the pre-training data (i.e. retaining “rationality”), the authors propose a clipped likelihood constraint that disincentivizes the generation of sequences that have low log-likelihood according to the pre-trained model. Second, since discrete diffusion yields a full sequence only at the end of the generation process, the oracle can only be queried with a valid input at the end, leading to sparse reward signals. To alleviate this problem, the authors perform reward shaping that provides signals at intermediate steps by completing intermediate sequences using the current policy model and querying the oracle using these sequences. The authors evaluate GLID2E for DNA and protein sequence design and compare it to the current state-of-the-art DRAKES, and other baselines. They show that GLID2E performs comparably to DRAKES and even outperforms it on certain metrics while being more computationally efficient. Their ablation analyses also underscore the importance of their main innovations.", "questions": "Apart from the questions mentioned above, another minor question is mentioned below:\n- How is the GLID2E method gradient-free? Aren’t the authors fine-tuning the diffusion model using gradient-based methods? I am not too familiar with RL literature so I might be confused with the terminology.", "limitations": "Yes. But the limitations are only provided in the appendix and are unreferenced in the main text. They need to be moved to the main text in the final version.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I think the strengths of this submission slightly outweigh its weaknesses which could be fixed during the rebuttal period. I provide a more detailed evaluation below.\n\nStrengths:\n- Quality: The authors benchmark their method against relevant baselines for fine-tuning diffusion models for biological sequence design and most claims are supported by their results. Motivations for proposed innovations are clearly explained and the work is generally technically sound.\n- Clarity: The paper is generally well-written and is easy to understand.\n- Significance: Although I am unfamiliar with the most recent literature on fine-tuning diffusion models, the proposed method seems to make a clear advance over the state-of-the-art DRAKES in terms of computational efficiency while achieving comparable metrics. This work could be of interest to computational biologists and the wider community of researchers working on discrete diffusion models.\n- Originality: Again, given my unfamiliarity with the most recent literature on fine-tuning diffusion models, I cannot fully ascertain the novelty of the proposed method. However, the proposed innovations are well-motivated and build on existing works in insightful ways.\n\nWeaknesses:\n- Quality: \n\n1. In the DNA design task, GLID2E has much lower 3-mer correlation compared to other methods and the authors claim that this might be due to the model learning novel motifs that are different from those observed in natural sequences (lines 276-278). In my opinion, there isn’t enough evidence to support such statements - the sequences have not been experimentally validated and their predicted activities come from models that are naturally susceptible to incorrect predictions due to distribution shift. The authors could have more measured conclusions.\n\n2. This work also does not address the diversity of designed sequences. The authors make multiple mentions of exploring diverse sequence spaces but never provide concrete results to illustrate this important property. From lines 278-281, I am concerned that fine-tuning significantly reduces the diversity of generated sequences leading to some form of mode collapse. It would be helpful if the authors can provide metrics such as average pairwise edit distance between generated sequences, metrics to measure k-mer diversity, etc.\n\n3. On line 166, the authors mention that only 4 sequences are generated to compute the mean and standard deviation of log-likelihoods. This does not seem like a large enough number. Is this a possible typo?\n\n- Clarity:\n1. I think some sections of this paper can be improved to better inform a reader. For one of the major intended audiences – researchers working on biological sequence design – it could be helpful to provide more high-level details about the optimization framework. For example, it might not be immediately clear how the PPO-based optimization fits into the fine-tuning setup. This could probably be done by improving Figure 1 (which is never referenced in the main text) and using it to explain the overall framework being used.\n\n2. Another major area for improving clarity is providing more details about the benchmarking. For example, basic details such as the number of sequences being generated for benchmarking are missing. It is also unclear why only HepG2 data is used in the DNA design task when the referenced dataset contains data from 5 different cell lines.\n\n3. There is also a large body of work that performs biological sequence design using offline model-based optimization. This work largely fits into that framework but the authors do not cite that body of literature.", "was_revised": true, "final_justification": "My concerns regarding sequence diversity were addressed by the authors. They also promised to improve the clarity of the paper in the final version.", "rebuttal": "We greatly appreciate the time and effort you put into the review. The concerns you raised are very targeted and constructive. In response, we attempt to clarify the issues one by one and clarify the advantages and contributions of our method through addressing these questions.\n\n1. In the DNA design task, ... more measured conclusions.\n> We appreciate the insightful question. We believe it is necessary here to distinguish between two different \"naturalness\" measurement standards: 3-mer correlation reflects the similarity of traditional statistical patterns, while Log-Likelihood embodies the distributional cognition learned by pretrained diffusion models from large-scale real data. Our experimental results show that GLID2E achieved optimal performance on the Log-Likelihood metric (-239.889), while maintaining a high level of 90.6% on ATAC-Acc, comparable to the current best method DRAKES (92.5%). This combination of \"excellent Log-Likelihood + good biological metrics\" effectively rules out the possibility of adversarial samples, proving that the generated sequences both conform to the model's cognitive naturalness standards and maintain key biological properties.\n> \n>In fact, the decrease in 3-mer correlation combined with significant improvement in activity precisely demonstrates GLID2E's unique value: traditional methods can only mimic the n-gram patterns of known enhancers, limiting them to existing sequence feature spaces, while our Clipped Likelihood Constraint mechanism, while fundamentally preventing adversarial attacks, successfully discovered functionally equivalent but structurally different new sequence variants, achieving principled exploration of new functional sequence spaces. However, the current results are only based on model predictions, and we apologize for being unable to provide biological experimental validation.\n\n2. This work also does not address... generated sequences, metrics to measure k-mer diversity, etc.\n> Thanks for pointing out. We acknowledge that the paper indeed lacks quantitative analysis of the diversity of generated sequences, which is an important omission. We will supplement comprehensive diversity analysis in the revised version, including key metrics such as average pairwise edit distance between generated sequences, k-mer diversity statistics, sequence entropy, and unique sequence ratios, and compare with baseline methods to verify whether GLID2E maintains sufficient sequence diversity while optimizing functionality.\n\n3. On line 166, the authors mention ... number. Is this a possible typo?\n> Yes, this is a typo. We didn't count the samples within the batch. It should be 4*16(batch size)*10(scaling coefficient)*4(gradient accumulation step)=1,280.\n\n4. I think some sections of ... explain the overall framework being used.\n> Thank you very much for pointing this out. We will add the reference to Figure 1 in the original text and redesign Figure 1 to highlight our advantage of not needing to store entire trajectories.\n\n5. Another major area for improving clarity ... data from 5 different cell lines.\n> Thank you very much for pointing this out. \n>\n> First, regarding details, we completely followed all evaluation settings of DRAKES, including generating 640 sequences for each method (batch size of 64, with 10 total batches). We also conducted experiments with three random seeds and reported means and standard deviations. \n>\n> Then, there are two reasons for choosing HepG2: 1) HepG2 is a liver cancer cell line that is widely used in enhancer activity research and has been used by multiple baselines; 2) Although the dataset contains five cell lines, we believe focusing on a single cell line can avoid noise that might come from mixing multiple cell lines and can more precisely evaluate the effectiveness of the method.\n\n6. There is also a large body ... authors do not cite that body of literature.\n> This is a very key suggestion. Our initial idea was to focus on diffusion+biological work, but offline RL-based biological sequence design still has high relevance. We promise to supplement this in subsequent versions.\n\n7. How is the GLID2E method gradient-free?... confused with the terminology.\n> \"Gradient-free\" refers to avoiding backpropagating gradients through the entire diffusion trajectory, rather than not using gradients at all. PPO still requires policy gradients, but the computational complexity is reduced from O(T·L·d²) to O(L·d). We are considering changing it to \"lightweight gradient.\" Thank you for the suggestion!"}, {"review_id": "gMGxVOVBCz", "reviewer": "Reviewer_8abn", "summary": "The paper proposes GLID²E, a gradient‑free reinforcement‑learning (RL) fine‑tuning framework for discrete diffusion models aimed at biological‑sequence design. Two methodological contributions are central:\n\nClipped Likelihood Constraint (CLC) – instead of the customary KL penalty, the authors use a one‑sided penalty on sequences whose pre‑trained diffusion likelihood falls more than k standard deviations below the mean, thereby enforcing naturalness while permitting exploration.\n\nReward Shaping via Potential Functions – partial sequences are completed on‑the‑fly (MASK tokens replaced by the current policy) to yield dense potentials that propagate functional feedback throughout the denoising trajectory.\n\nCoupled with a PPO‑style clipped objective that mixes the current and prior policies (η‑mixture), the framework claims O(N B T L d) training complexity vs. O(N B T L d²) for gradient‑based counterparts.\n\nEmpirical evaluation on DNA enhancer design and thermostable protein design shows (i) a 31 % lift over DRAKES in median predicted activity for DNA and (ii) parity with DRAKES on protein stability with ∼40 % shorter training time.\n Ablations confirm that both CLC (M2) and reward shaping (M1) are necessary.", "questions": "1. Statistical Significance. Could you report 95 % CIs or perform Wilcoxon signed‑rank tests to substantiate the median improvements in Tables 2‑3? A statistically significant win would strengthen the claim of superiority.\n\n2. Reward‑Model Robustness. How sensitive is GLID²E to reward‑model mis‑specification? E.g., if an oracle is slightly noisy or domain‑shifted, does CLC prevent reward hacking?\n\n3. Hyper‑parameter Burden. The method introduces k, β, η, clip threshold, clip mix, etc. What guidance can you give for practitioners, and which parameter is most critical?\n\n4. 3‑mer Correlation Drop. In DNA design, activity increases yet 3‑mer correlation plummets (0.49). Does this suggest non‑natural yet high‑scoring sequences? Any risk of adversarial samples undetected by CLC?\n\n5. Generalisation Beyond Training Domains. Have you tried transferring a GLID²E fine‑tuned model between cell types or protein families to assess over‑fitting vs. functional generality?\n\n6. Missing baselines. From the RL related methods, the closest one to GLIDE seems to be CTRL by Uehara et al. 2024, why didn't you compare to it? Also, there is a number of non-diffusion baselines that have been explored for discrete biologcal sequence design, the current score of baselines is very limited (only diffusion models).\n\n7. In the beginning you mention GLIDE due to reliance on RL,  is suitable for limited data and can enable multi-objective design and optimization. However, both the Diffusiion Model and the Reward Oracle in GLIDE are than on over million sequences which is quite substantial amount. I would suggest refraining from statements like this one.\n\n8. Why didn't you use standard benchmarks and splits like AAV and GFP? \n\nAndrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi S Jaakkola, Regina Barzilay, and Ila R Fiete. Improving protein optimization with smoothed fitness landscapes. In ICLR, 2024.", "limitations": "The authors provide a candid discussion in Appendix B.1, acknowledging lack of theoretical guarantees and wet‑lab validation. I agree with their assessment and encourage adding statistical significance and broader biological tasks to future work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "| Dimension        | Strengths                                                                                                                                         | Weaknesses                                                                                                                                                                   |\n| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **Quality**      | *Solid engineering & ablations*: CLC, reward shaping and PPO ablations isolate each factor. *Efficiency*: 1.7× speed‑up vs. DRAKES . | *Statistical rigor*: no CIs or hypothesis tests; results rely on medians over three seeds. *Limited biological validation*: entirely in silico; no wet‑lab confirmation. |\n| **Clarity**      | Clear high‑level motivation; algorithm pseudocode is helpful; hyper‑parameters tabulated.                                         | Dense notation in Sec. 4; CLC derivation could use intuition; some figures referenced but not shown      |\n| **Significance** | Removes gradient dependence—important for long discrete sequences. May generalise to other discrete design tasks (RNA, small molecules).      | Improvement on proteins is marginal; DNA 3‑mer correlation drops markedly (0.49 vs. 0.89 for SMC), indicating potential drift from known biophysical motifs.                 |\n| **Originality**  | First demonstration of *gradient‑free* RL fine‑tuning for **discrete diffusion** in biology; CLC is novel.                                        | Reward‑shaping idea builds on prior potential‑based shaping; mixture‑policy PPO resembles existing “RL as control” work (e.g., Uehara et al. 2024) was only mentioned but not compared to.                          |", "was_revised": true, "final_justification": "The authors have replied to most of my concerns. After reading through the reviews and responses for the others, I believe the authors   did a great job in the rebuttal adding detailed derivation and benchmarks. I am now more convinced the manuscript is worthy of acceptance especially after incorporating all the additional results form the rebuttal. I raise my score accordingly.", "rebuttal": "We appreciate your time and effort for reviewing our submission. Your key insight and valuable question help us boost our paper's quality. Regarding the questions, we provide the following clarification: \n\n1. Statistical Significance. Could you report ... strengthen the claim of superiority.\n> Thank you for the suggestion. We will conduct Wilcoxon signed-rank tests, and the results are as follows:\nIn Table 2, comparing GLID²E and DRAKES, except for 3-mer, all other p-values are 0.03125 (<0.05), meeting statistical significance characteristics. In Table 3, comparing GLID2E and DRAKES, the results are shown in the following table, and we can see that the performance of DRAKES and GLID²E is basically similar.\n>\n> |   Metric       | Pred-ddG (Median) | %(ddG>0) | scRMSD (Median) | %(scRMSD<2) | Success Rate |\n> |----------|-------------------|----------|-----------------|-------------|--------------|\n> | p-value  | 0.3125           | 0.2187   | 0.21875         | 0.03125     | 0.0625       |\n\n2. Reward‑Model Robustness. How ... CLC prevent reward hacking?\n> We added three scales of noise to the ddG predicted by the diffusion model. Compared to before adding noise, the success rate was generally affected, mainly impacting Pred-ddG, and the larger the noise scale, the greater the impact.\n> | Noise Scale | Pred-ddG (Median) | %(ddG>0) | scRMSD (Median) | %(scRMSD<2) | Success Rate |\n> |-------------|-------------------|----------|-----------------|-------------|--------------|\n> | 0.01        | 0.81             | 0.73     | 0.94            | 0.91        | 0.66         |\n> | 0.1         | 0.71             | 0.74     | 1.00            | 0.84        | 0.60         |\n> | 0.5         | 0.21             | 0.53     | 0.88            | 0.95        | 0.49         |\n\n3. Hyper‑parameter Burden. The method ... is most critical?\n> Very practical perspective. The PPO algorithm involves many hyperparameters, but most of our model's related parameters adopt commonly used ones. For specifics, please refer to Table 6-Table 8. The two hyperparameters that have significant impact on results have already been proposed in the ablation study. The clip ratio directly controls the exploration-exploitation tradeoff - too strict will prevent beneficial exploration, while too loose will cause excessive distribution shift; the mixture ratio also directly controls the model's exploration of modes, similar to the principle of KL penalty. Being overly conservative will prevent effective exploration, while maintaining smaller values can provide training stability while serving as regularization.\n\n4. 3‑mer Correlation Drop. In DNA ... adversarial samples undetected by CLC?\n> Thank you fory your very insightful observation. Regarding the question about the decrease in 3-mer correlation, we believe this precisely demonstrates GLID2E's core advantage. We need to distinguish between two types of \"naturalness\" measures: 3-mer correlation reflects traditional statistical patterns, while Log-Likelihood embodies the learned distribution of pretrained models. GLID2E performs best on Log-Likelihood (-239.889) and achieves ATAC-Acc as high as 90.6%, indicating that generated sequences both conform to the model's perceived naturalness and maintain key biological properties, effectively ruling out the possibility of adversarial samples. In fact, this pattern demonstrates GLID2E's unique value: traditional methods can only mimic known enhancer patterns, while we discovered functionally equivalent but structurally different new sequence variants, expanding the feasible design space. The Clipped Likelihood Constraint mechanism fundamentally prevents adversarial attacks. Therefore, the decrease in 3-mer correlation combined with improved activity and good biological metrics indicates that GLID2E achieved principled exploration of new functional sequence spaces, which is precisely our method's core contribution.\n\n5. Generalisation Beyond Training Domains. ... vs. functional generality?\n> Good point! However, different cell types have different scales and biological experimental conditions compared to protein families, which leads to rewards not being able to accurately express the functionality of samples. Therefore, we believe they cannot be put together for fine-tuning.\n\n6. Missing baselines. From the RL ... is very limited (only diffusion models).\n> Thank you for pointing out the CTRL method. We believe adding this baseline is necessary and will subsequently include it in the evaluation. However, we need to claim that our paper aims to propose a reinforcement learning paradigm based on Discrete Diffusion Models, targeting discrete guided sampling. Therefore, we believe it is not necessary to include other non-diffusion baselines. Hope for your understanding.\n\n7. In the beginning you mention ... statements like this one.\n> Thank you for the correction. This statement is indeed not accurate enough. Our method's advantage lies in data efficiency during the fine-tuning stage, not the pre-training stage. We will address this in the paper.\n\n8. Why didn't you use standard benchmarks and splits like AAV and GFP?\n> We are not particularly clear what you refer to as the \"Standard\" benchmark. We are following DRAKES's setting in both experimental systems, and we believe this is a reasonable split with relevant baselines. We look forward to your insights on the current benchmark."}, {"review_id": "xGpaU5gi8c", "reviewer": "Reviewer_h7mh", "summary": "The paper introduces **GLID2E**, a gradient-free, reinforcement-learning-based fine-tuning framework specifically designed for discrete diffusion models in biological sequence generation tasks. GLID2E combines two key innovations: (1) a clipped likelihood constraint that maintains the rationality and naturalness of generated sequences by constraining deviations from a pretrained diffusion model's likelihood distribution, and (2) a reward shaping mechanism that provides intermediate rewards, effectively guiding generation throughout the diffusion steps rather than relying solely on terminal rewards. These approaches are integrated within a Proximal Policy Optimization (PPO) framework, enhancing training stability and computational efficiency compared to gradient-based fine-tuning methods. Empirical evaluations on DNA regulatory elements and protein stability design tasks demonstrate decent competitive performance against state-of-the-art methods like DRAKES, highlighting GLID2E’s relative efficacy in achieving multi-objective optimization with lower computational overhead.", "questions": "1. The authors introduce the clipped likelihood constraint (Eq. 1, page 5), which uses a fixed standard deviation multiplier \\( k = 1 \\) to determine the rationality threshold. Can the authors explicitly justify this specific choice mathematically or empirically for me? In particular, I wonder how sensitive GLID2E’s performance is to changes in \\( k \\). Could the authors provide a detailed sensitivity analysis across multiple values (e.g., \\( k \\in \\{0.5, 1, 2, 3\\} \\))? \n\n2. Regarding the PPO-based surrogate objective (Eq. on page 6), the authors did define a mixed reference policy using a mixture ratio parameter \\( \\eta \\). However, the exact impact of varying \\( \\eta \\) mathematically and empirically is a bit unclear to me. Could the authors explicitly derive or demonstrate how varying \\( \\eta \\) affects policy updates mathematically? Additionally, a detailed empirical ablation across several distinct values of \\( \\eta \\) (beyond those presented in Table 5, page 9) would help me understand its role. \n\n3. The authors position GLID2E as a method suitable for multi-objective optimization. Yet, the current benchmarks are somewhat limited, and no comparisons are made to existing advanced discrete diffusion multi-objective methods, notably PepTune (which employs MCTS-based sampling guidance). Would the authors consider performing comparisons against an MCTS strategy, clearly outlining differences in both theoretical formulations (such as their likelihood constraints vs. MCTS guidance mechanism) and empirical performance metrics (reward convergence, sequence diversity)? \n\n4. The authors utilize reward shaping (Eq. 2, page 6) to provide intermediate signals during generation. However, I am confused by the mathematical stability and bias introduced by intermediate reward signals. Could the authors mathematically derive or experimentally evaluate potential biases or instabilities introduced by intermediate sampling of partial sequences (especially given the stochastic nature of replacing MASK tokens)? \n\n5. The work extensively rely on reward oracles, which inherently contain inaccuracies or biases. I am wondering if the authors woulkd be able to explicitly assess the robustness of GLID2E to reward oracle misspecification or noisy reward signals? For example, the authors could artificially inject noise into the oracle outputs or bias the oracle predictions systematically and rigorously quantify the sensitivity of GLID2E’s policy optimization outcomes.", "limitations": "Just a few thoughts: the authors briefly address some limitations and negative societal impacts in Appendix B. However, I believe this discussion could be significantly improved by explicitly clarifying a few critical points. In particular, I think the authors should consider explicitly discussing the implications of GLID2E-generated sequences potentially diverging significantly from biologically verified motifs. Could this lead to biologically plausible yet functionally uncertain or risky designs? Maybe the authors should also consider explicitly addressing whether reliance on potentially inaccurate reward oracles might inadvertently amplify biological or experimental biases. Clearly outlining these issues and suggesting potential safeguards (with robustness checks or validation steps) would strengthen the paper’s responsible innovation perspective.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- I find two compelling methodological innovations in this paper: the clipped likelihood constraint (Eq. 1, page 5) and the reward shaping mechanism (Eq. 2, page 6). Both techniques are mathematically rigorous and intuitively address well-known challenges in gradient-free RL fine-tuning of discrete diffusion models, specifically issues related to training stability and sparse rewards.\n- The computational efficiency of the proposed GLID2E framework is quite impressive! The authors provide explicit complexity analysis (Table 1, page 4) and runtime benchmarks (Table 9, page 15), demonstrating meaningful efficiency improvements over gradient-based approaches such as DRAKES.\n- The empirical results for DNA (Table 2, page 7) and protein (Table 3, page 8) sequence design benchmarks are strong, with competitive or superior performance metrics on predicted activity and stability. The clear ablations also show the practical benefit of each proposed component.\n- The authors provide detailed pseudo-code (Algorithm 1, page 16) clearly demonstrating the PPO-based optimization steps, which greatly enhances reproducibility and helps me to evaluate the model training strategy too.\n\n**Weaknesses:**\n- Although the paper claims multi-objective optimization capability, from my understandng of the field, the authors overlook direct comparisons with key existing multi-objective discrete diffusion models. Given other models' explicit focus on multi-objective generation via MCTS-guided diffusion, explicit benchmarking against these would considerably clarify the claimed advantages of GLID²E.\n- Hyperparameter choices, such as the standard deviation multiplier \\( k = 1 \\) in the clipped likelihood constraint (Eq. 1, page 5) and the specific values of the clipping and mixture ratios (Tables 4 and 5, page 9), are not mathematically justified sufficiently or have thorough sensitivity analyses. If the authors could provide empirical sensitivity studies on these parameters it would help the study's overall methodological robustness and transparency.\n- The absence of rigorous statistical significance analyses (like, hypothesis testing, confidence intervals) for experimental results, to me, gives me less confidecne in the robustness and reproducibility of performance claims. \n- While computational efficiency claims are supported by preliminary runtime data, explicit complexity analyses or computational benchmarks across varying sequence lengths, or more thorough memory usage analyses, are limited. \n- Given the heavy dependence on reward oracle correctness, I would recommend that the authors explicitly address potential oracle bias or misspecification issues?", "was_revised": true, "final_justification": "I am increasing my score to a 5. The authors clarified all of my points and I am convinced the paper, with these new explanations and results, will be a valuable addition to NeurIPS.", "rebuttal": "Thank you for your time and effort reading and reviewing our paper. Also, thank you for the valuable questions. We provide detailed answers below, hope it may clarify ourselves as well as enhance your understanding.\n\n1. The authors position GLID2E ... metrics (reward convergence, sequence diversity)?\n> Thank you for your suggestion. We acknowledge that the current work indeed has limitations in multi-objective optimization evaluation. However, we claim that GLID2E can explore space through reward shaping and likelihood constraints from pretrained diffusion models. Compared to MTCS, it can achieve higher computational efficiency. We will subsequently update PepTune to the current benchmark.\n\n2. Hyperparameter choices, ... methodological robustness and transparency.\n> We will supplement ablation experiments regarding k. According to experimental results, we can see that k=1 achieves the best trade-off. When k is less than 1.0, it will lead to conservative exploration, generating samples closer to the original distribution, but functionality will be limited. When k is greater than 1.0, the model will have more aggressive exploration, and lower quality samples will also lead to decreased model functionality.\n>| K | Pred-Activity (Median) | ATAC-Acc | 3-mer Corr | Log-Lik (Median) |\n>|---|------------------------|----------|------------|------------------|\n>| 0.5 | 7.12 | 0.99 | 0.53 | -252.9 |\n>| 1.0 | 7.45 | 0.99 | 0.49 | -244.7 |\n>| 2.0 | 7.06 | 0.97 | 0.48 | -232.5 |\n>| 3.0 | 7.21 | 0.83 | 0.50 | -262.1 |\n\n3. The absence of rigorous statistical ... robustness and reproducibility of performance claims.\n> Thank you for the suggestion. We will conduct Wilcoxon signed-rank tests, and the results are as follows:\nIn Table 2, comparing GLID2E and DRAKES, except for 3-mer, all other p-values are 0.03125 (<0.05), meeting statistical significance characteristics. In Table 3, comparing GLID2E and DRAKES, the results are shown in the following table, and we can see that the performance of DRAKES and GLID2E is basically similar.\n> |   Metric       | Pred-ddG (Median) | %(ddG>0) | scRMSD (Median) | %(scRMSD<2) | Success Rate |\n> |----------|-------------------|----------|-----------------|-------------|--------------|\n> | p-value  | 0.3125           | 0.2187   | 0.21875         | 0.03125     | 0.0625       |\n\n4. While computational efficiency ... memory usage analyses, are limited.\nThank you for the detailed suggestion. We will provide runtime comparisons for different lengths and memory usage comparisons:\nWe have collected inference metrics for sequences of different length ranges. Since we set the generation site amount fixed as 25 (following the same setting as DRAKES), the differences in these metrics are not significant.\n> | seq_length | batch_size | num_runs | mean_time | std_time | mean_gpu_memory |\n> |------------|------------|----------|-----------|----------|-----------------|\n> | 50 | 8 | 50 | 0.011207 | 0.002391 | 1048.058 |\n> | 100 | 8 | 50 | 0.009823 | 0.000122 | 1050.813 |\n> | 150 | 8 | 50 | 0.010457 | 0.000191 | 1053.742 |\n> | 200 | 8 | 50 | 0.010417 | 6.69E-05 | 1057.576 |\n> | 250 | 8 | 50 | 0.010685 | 8.98E-05 | 1055.277 |\n> | 300 | 8 | 50 | 0.010617 | 5.87E-05 | 1057.729 |\n> | 350 | 8 | 50 | 0.010712 | 8.75E-05 | 1053.25 |\n> | 400 | 8 | 50 | 0.010695 | 0.000147 | 1071.35 |\n\n5. Regarding the PPO-based surrogate... would help me understand its role.\n> We appreciate your detialed review of our method, the derivation corresponding to eta is as follows. This ratio is r_t, which directly participates in the calculation of L_CLIP and gradient updates. The larger eta is, the more the updates are constrained by priors and the more conservative it becomes; conversely, it allows for large-scale exploration.\n$$\\frac{\\pi_\\theta(a|s)}{\\pi_{\\text{ref}}(a|s)} = \\frac{\\pi_\\theta(a|s)}{\\pi_{\\text{prior}}(a|s)^\\eta \\pi_\\theta(a|s)^{1-\\eta}} = \\left(\\frac{\\pi_\\theta(a|s)}{\\pi_{\\text{prior}}(a|s)}\\right)^\\eta.$$\n\n6. The authors utilize reward shaping ... (especially given the stochastic nature of replacing MASK tokens)?\n> Thank you for pointing out. Regarding the questions on both aspects, our claims are as follows:\n>\n> Before claims, we would like to build a mathematical formulation for the sampling process to demonstrate the unbiased property. To obtain the reward $\\Phi$, we need to score along a *complete* sequence, so every MASK token in $s_t$ is independently replaced by a sample from the current policy $p_\\theta(\\cdot | s_{t-1}, t)$ with the MASK token set to zero probability. Let $B$ denote the number of completions drawn. Then we have \n>\n>$$\\hat{\\Phi}(s_t) = \\frac{1}{B} \\sum_{b=1}^{B} r(x_t^{(b)}), \\quad \\mathbb{E}[\\hat{\\Phi}(s_t) | s_t] = \\Phi(s_t),$$\n>\n> where x_t is the sampled sequence at time t. Therefore, $\\hat{\\Phi}$ is an **unbiased estimator** of Φ. Its variance is\n>\n> $$\\text{Var}[\\hat{\\Phi}(s_t) | s_t] = \\frac{\\sigma_r^2(s_t)}{B},$$\n>\n> where $\\sigma_r^2(s_t)$ is the conditional variance of the reward. During the diffusion process, the number of MASKs decreases, reducing both $\\sigma_r^2$ and gradient noise, making the estimator progressively stable.\n>\n> Then, for the two perspective being mentioned: \n> \n> 1) Since only one token is sampled at each step, the incremental rewards at each step will cancel each other out across the entire sequence, so that the cumulative reward and original reward are always on the same order of magnitude, and gradient explosion will not occur. Meanwhile, in the latter half of generation, the number of MASKs gradually decreases, and the corresponding variance of sigma also gradually decreases, making the values more stable. \n> \n> 2) Lines 205-206 state that the expected return and original return are independent of the trajectory. Meanwhile, the potential function of Monte Carlo estimation is unbiased for the true potential function, so the corresponding gradient is also unbiased and will not introduce systematic bias.\n\n7. The work extensively rely on ... policy optimization outcomes.\n> We appreciate your insight to the effect of reward oracle. \n>\n> 1) We supplemented Oracle noise experiments on protein systems:\n> We added three scales of noise to the ddG predicted by the diffusion model. Compared to before adding noise, the success rate was generally affected, mainly impacting Pred-ddG, and the larger the noise scale, the greater the impact. \n>\n> 2) From the reward model performance of DRAKES paper, we can conclude that protein oracle has higher oracle noise than DNA, which leads to the unstable training and high sensitivity to oracle noise. However, we would like to claim that the proposal of reward shaping and mixture is to address Oracle bias, and the ablation results w/o M1 have demonstrated it. \n>\n> | Noise Scale | Pred-ddG (Median) | %(ddG>0) | scRMSD (Median) | %(scRMSD<2) | Success Rate |\n> |-------------|-------------------|----------|-----------------|-------------|--------------|\n> | 0.01        | 0.81             | 0.73     | 0.94            | 0.91        | 0.66         |\n> | 0.1         | 0.71             | 0.74     | 1.00            | 0.84        | 0.60         |\n> | 0.5         | 0.21             | 0.53     | 0.88            | 0.95        | 0.49         |"}]

Author rebuttal:

Meta-review: This paper presents GLID2E, a RL framework for fine-tuning discrete diffusion models for biological sequence design. Reviewers noted that the core technical contributions (a clipped likelihood constraint to ensure stability and a reward shaping mechanism to handle sparse rewards) are novel, well-motivated, and technically sound. The paper demonstrates that the proposed approach achieves competitive or superior performance on DNA and protein design tasks compared to state-of-the-art methods while being more computationally efficient. The initial reviews raised several concerns regarding the experimental validation. These included a lack of analysis on sequence diversity (Reviewer TFcp), insufficient testing of the method's robustness to noisy reward oracles (Reviewers 8abn, h7mh), missing comparisons to key baselines like PepTune (Reviewer h7mh), and a need for more rigorous statistical analysis and hyperparameter ablations (Reviewer h7mh), but the authors' rebuttal and discussion with the reviewers resolved these concerns, leading all three reviewers to raise their scores and recommend acceptance. I agree with the reviewers and believe this work is a valuable contribution to the field of generative modeling for biological design. I recommend acceptance.
Decision: Accept (poster)