Improved Off-policy Reinforcement Learning in Biological Sequence Design
Authors: Hyeonah Kim
Minsu Kim
Taeyoung Yun
Sanghyeok Choi
Emmanuel Bengio
Alex Hernández-García
Jinkyoo Park
Venue: ICML 2025 poster (2025)
Keywords: Biological sequence design
GFlowNets
offline RL
active learning
Primary area: reinforcement_learning

Abstract: Designing biological sequences with desired properties is challenging due to vast search spaces and limited evaluation budgets. Although reinforcement learning methods use proxy models for rapid reward evaluation, insufficient training data can cause proxy misspecification on out-of-distribution inputs. To address this, we propose a novel off-policy search, $\delta$-Conservative Search, that enhances robustness by restricting policy exploration to reliable regions. Starting from high-score offline sequences, we inject noise by randomly masking tokens with probability $\delta$, then denoise them using our policy. We further adapt $\delta$ based on proxy uncertainty on each data point, aligning the level of conservativeness with model confidence. Experimental results show that our conservative search consistently enhances the off-policy training, outperforming existing machine learning methods in discovering high-score sequences across diverse tasks, including DNA, RNA, protein, and peptide design.

Improved Off-policy Reinforcement Learning in Biological Sequence Design
Hyeonah Kim 12
Minsu Kim
Taeyoung Yun Sanghyeok Choi Emmanuel Bengio
Alex Harrández-Garria 12 Jinkyoo Park2
Abstract
has emerged one of the successful paradigms for au- tomatic biological sequence design (Angermueller et al., 2020; Jain et al., 2022). Particularly, they employ deep neu- ral networks as inexpensive proxy models for rapid reward evaluation by approximating expensive oracle score func- tions. There are two approaches for training the policy in this context: on-policy and off off-policy.
Designing biological sequences with desired prop- erties is challenging due to vast search spaces and limited evaluation budgets. Although rein- forcement learning methods use proxy models for rapid reward evaluation, insufficient training data can cause proxy misspecification of-distribution inputs. To address this, we pro- pose a novel off-policc search, 5-Conservative Search, that enhances robustness by restricting policy exploration to reliable regions. Starting from high-score offline sequences, we inject noise by randomly masking tokens with probability 8. then denoise them using our policy. We further adapt 5 based on proxy uncertainty on each data point, aligning the level of conservativeness with model confidence. Experimental results show that our conservative search consistently enhances the off- off-policy training, outperforming existing ma- chine learning methods discovering high-score sequences across diverse tasks, including DNA, RNA, protein, and peptide design.
out-
DyNA PPO (Angermueller et al., 2020), representative on-policy RL method for biological sequence design, em- ploys Proximal Policy Optimization (PPO; Schulman et al., 2017) within a proxy-based active training loop. While DyNA PPO has demonstrated effectiveness various bi- ological sequence design tasks, its major limitation the limited search flexibility inherent to on-policy methods. cannot effectively leverage offline data points, even like data collected from previous rounds.
On the other hand, Generative Flow Networks (GFlowNets; Bengio et al., 2021), off-policy RL methods akin to max- imum entropy policies (Tiapkin et al., 2024; Deleu et al., 2024), offer diversity capabilities and flexible ex- ploration strategies. Jain et al. (2022) applied GFlowNets to biological sequence design with additional Bayesian active learning schemes. They leveraged the off-polic nature of GelowNets by mixing offline datasets with on-policy data during training. This approach provided more stable training compared to DyNA PPO, resulting in better performance.
1. Introduction
Designing biological sequences with desired properties is crucial in therapeutics and biotechnology (Zimmer, 2002;
However, recent studies have consistently reported that GFlowNets perform poorly in large-scale settings, such as green fluorescent protein design (Kim et al., 2023; Surana et al., 2024). We hypothesize that this bounded performance stems from the insufficient quality of the proxy model in the early rounds. Although GFlowNets can generate novel sequences beyond given data points, the resulting out-of- distribution inputs lead to unreliable rewards from the un- dertrained proxy (Trabucco al., 2021; Yu al., 2021). This motivates to introduce conservative search strat- egy restrict the search space to the neighborhoods of the data points observed during RL training when generating sequences to query for the next round.
Lorenz al., 2011; Barrera et al., 2016; Sample et al., 2019; Ogden et al., 2019). However, this task is challenging due to the combinatorially large search space and the expensive and black-box nature of objective functions. Recent advances in deep learning methods for biological sequence design have shown significant promise at overcoming these chal- lenges (Brookes & Listgarten, 2018: Brookes et al., 2019; Angermueller et al., 2020; Jain et al., 2022).
Among various approaches, reinforcement learning (RL)
'illa Quebec AI Institute Universsité de Montréal " KAISS *Valence Labs. Correspondence to: Hyeonah Kim <hyconah. kim @ qubbe...
Contribution. In this paper, we propose novel off-policy search method called ò-Conservative Search (8-CS), which enables trade-off between sequence novelty and robustness
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Improved Off-policy RL in Biological Sequence Design
to proxy misspecification with conservativeness parame- ter 8. Specifically, we iteratively train GFlowNet using 6-CS as follows: (1) we inject noise by independently mask- ing tokens in high-score offline sequences with Bernoulli distribution with parameter 5; (2) the GFlowNet policy se- quentially denoises the masked tokens; (3) we use these denoised sequences to train the policy. Figure illustrates the overall procedure of 8-CS. We adjust the conservative level with adaptive (a;; a) using the proxy model's uncer- tainty estimates (x) for each data point
Step A (Proxy Training): We train a proxy model fa(x) using the offline dataset D 1 at round t.
Step B (Policy Training with 6-CS): We train a generative policy p(a; /) using the proxy model fo and the dataset Dz-1 with 6-CS.
Step C (Offline Dataset Augmentation with ô-CS); We apply 8-CS query batched data {a to the oracle V: f1;)). Then the offline dataset is augmented as: Dr +-
D (-r,, b:))
Our extensive experiments demonstrate that 6-CS sig- nificantly improves GFlowNets, successfully discovering higher-score sequences compared to existing model-based optimization methods on diverse tasks, including DNA, RNA, protein, and peptide design. This suggests that ö-CS is robust and scalable framework for advancing research and applications in biotechnology and synthetic biology.
The overall algorithm is described in Algorithm 1. In the following subsections, we describe the details of Step A and Step B.
3.1. Step A: Proxy Training
Following Jain et al. (2022), we train the proxy model fø using the dataset D.-1 by minimizing the mean squared error loss:
2. Problem Formulation
[ E (5) (U(x))f((())
(I)
We aim to discover sequences z E V² that exhibit desired properties, where V denotes the vocabulary, such as amino acids or nucleotides, and represents the sequence length, which is usually fixed. The desired properties, such as binding affinity or enzymatic activity, are evaluated by a black-box oracle function f Vr. + R. Evaluating f is often both expensive and time-consuming since it typically involves wet-lab experiments or high-fidelity simulations.
where Dj is the dataset at active round augmented with oracle queries. In the initial round (!. = 1), we use the given initial dataset Do. See Appendix A.1 for detailed implementation.
3.2. Step B: Policy Training with 8-CS
For policy training, we employ GFlowNets, ! which aim to produce samples from generative policy where the prob- ability of generating a sequence æ is proportional to its reward, i.e.,
Advancements in experimental techniques have enabled the parallel synthesis and evaluation of sequences in batches. Therefore, lab-in-the-loop processes emerging as prac- tical settings that enable active learning. Following this paradigm, we perform I' rounds of batch optimization, where in each round, we have the opportunity to query B batched sequences to the (assumed) oracle objective func- tion f. Following Angermueller et al. (2020) and Jain al. (2022), we assume the availability of an initial offline dataset Do ((x("), (y())) ber of data points No is typically many orders of magnitude smaller than the size of the search space, as mentioned in the introduction. The goal is to discover, after rounds, a set of sequences that are novel, diverse, and have high oracle function values.
p(x;0) x R(x;)) F(x;o)
Here, F(x; d) is an acquisition function. The proxy fo(x) can be directly used as rewards, but mainly employ the upper confidence bound (UCB; Srinivas et al., 2010) acquisition function following Jain et al. (2022).
where y = fz). The initial num-
Policy parameterization. ates state transitions sequentially within trajectories т = (8o > -> 8t, = x)), where 8q = () represents the empty sequence, and each state transition involves adding se- quence token. The full sequence sr. is obtained after L steps, where L is the sequence length. The forward policy (p(T; 0) is compositional policy defined as
The forward policy PF gener-
3. Active Learning for Biological Sequence Design
Following Jain et al. (2022), we formulate an active learning process constrained by budget T' rounds with query size B. The active learning is conducted through an iterative pro- cedure consisting of three standard stages, two of which are modified by our proposed method, à-Conservative Search (8-CS), which will be detailed in Section 4.
PF(T: = II Pr(s:(si-1:0)
(2)
OOur focus is primarily the use of GFlowNets an off- policy reinforcement learning framework; Section 5.2 and Appendix B.2 for further discussion.
2

---

Improved Off-policy RL in Biological Sequence Design
Proxy training
Policy training with b-Conservative Search
Evaluation & update
Oracle f
with prob. d
D
Proxy fo
Policy Pp
Trained Proxy fe
Dr
Noise injection
Denoising with Pr
Figure 1. The active learning process for biological sequence design with 5-Conservative Search (6-CS), Starting with high reward sequences from the offline dataset, we inject token-level noise with probability 6, which determines the conservativeness the search. Then, the GFlowNet policy denoises the masked sequences. Lastly, the GFlowNet policy is trained with new sequences. After policy training, we query a new batch of sequences and update the dataset for the next round.
GFlowNets have a backward policy PB(T(xr) that mod- els the probability of backtracking from the terminal state I. The sequence x = eL) can be uniquely con- verted into state transition trajectory т, where each in- termediate state represents subsequence. In the case of sequences, there is only single way to backtrack, (B(T(z) = 1. This makes these types of GFlowNets equiv- alent to soft off-policy RL algorithms. For example, the trajectory balance (TB) objective of GFlowNets (Malkin et al., 2022) becomes equivalent to path consistency learn- ing (PCL) (Nachum et al., 2017), an putropy-maximizing value-based RL method according Deleu al. (2024).
the rank-based reweighted prior.
Inject noise by masking tokens into x using the noise injection policy Puoise (ī x 8).
Denoise the masked tokens using the denoising policy Paenoiso ( 2 0).
These trajectories are used update the GFlowNet parame- ters 0 by minimizing the loss function LIB (TB(T; 0). For more details on the algorithmic components of 5-CS and its inte- gration with active learning GFlowNets, see Algorithm 1.
Rank-based reweighted prior. First, we sample reference sequence æ from the prior distribution PD. To exploit nigh-scoring sequences, we employ rank- prioritiza- tion (Tripp et al., 2020):
Learning objective and training trajectories. The policy is trained to minimize TB loss follows.
Cra (T; = log
Zo PT; R(I; 6)
2
(3)
w(x; kr-,,)) x KN + rank (x)
Usually, GFlowNets training is employed to minimize TB loss with training trajectories on full supports, asymp- totically guaranteeing optimality for the distribution of p(2; 0) x R(x; d). A key challenge in prior works Jain al. (2022) that the proxy model (x) often produces highly unreliable rewards R(x; Φ) for out-of-distribution inputs. In our approach, mitigate this by providing off-policy trajectories within more reliable regions by in- jecting conservativeness into off-policy search. Therefore, we minimize TB loss with 8-CS, which offers controllable conservativeness.
Here, rank, (z) is a relative rank of the value of f(x) in the dataset Dz-1 with a weight-shifting factor l;; we fix k = 0.01. This assigns greater weight to sequences with higher ranks. Note that this reweighted prior can also be used during proxy training.
Noise injection policy. Let = (E1, C2,.... EL) denote the original sequence of length L. We define noise in- jection policy where each position E 21, L} is independently masked according to a Bernoulli distribu- tion with parameter E [0, 1]. resulting in the masked sequence (ē1, ē,. The noise injection policy Phoise (Ø ax, )) defined
4. Controllable Conservativeness in Off-Policy Search
4.1. d-Conservative Search
P Pooiso (ā
xx, (x,6)
This section introduces 5-Conservative Search (6-CS), an off-policy search method that enables controllable explo- ration through conservative parameter 6. Here, ò defines the Bernoulli distribution governing the masking of tokens in sequence. Our algorithm proceeds follows:
L
II
I( [MASK]] + -I{é = e]],
where I(-) is the indicator function.
Denoising policy. We employ the GFlowNet forward pol-
Sample high-score offline sequences x PD._, (a:) from
icy Pp to sequentially reconstruct the masked sequence
3

---

Improved Off-policy RL in Biological Sequence Design
i= =- (2..., ēz) by predicting tokens from left to right. The probability of denoising next token é from previously denoised subsquence 8, is:
in high-dimensional tasks by using trust-region-based search restrictions (Wan et al., 2021; Eriksson al., 2019; Biswas et al., 2021; Khan et al., 2023) and by combining it with deep generative models (Stanton et al., 2022; Gruver et al., 2024). However, these methods usually suffer from scalabil- ity issues due to the complexity of the Gaussian process (GP) kernel (Belanger et al., 2019) or the difficulty of sampling from an intractable posterior (Zhang et al., 2022).
Paenoise (ee T;;
14e2 Ex),
if é + [MASK],
(5 = (8)-1, 6e) 8)-1; if é = [MASK].
The fully reconstructed sequence = 8r. is obtained by sampling from:
Offline model-based optimization (MBO) (Kumar & Levine, 2020; Trabucco et al., 2021; Yu et al., 2021; Chen et al., 2022; Kim et al., 2023; Chen et al., 2023a; Yun et al., 2024) also addresses the design of biological sequences using of- fline datasets only, which can be highly efficient because they do not require oracle queries. These approaches have re- ported meaningful findings, such as the conservative require- ments proxy models since proxy models tend to give high rewards on unseen samples (Trabucco et al., 2021; Yu et al., 2021; Yuan et al., 2023; Chen et al., 2023b). This supports our approach of adaptive conservatism in the search process. Surana et al. (2024) recently noted that offline design and existing benchmarks are insufficient to reflect biological reliability, indicating that settings without additional oracle queries might be too idealistic. For a more comprehensive discussion, see the work by Kim et al. (2025).
Paenoise (3c 0):0= II Paencise (ēx 1, R;; O):
By denoising the masked tokens with the GFlowNet policy, which infers each token sequentially from left to right, we generate new sequences â that balance novelty and conser- vativeness through the parameter 8.
4.2. Adjusting Conservativeness
Determining the conservativeness parameter 8 is crucial. This work proposes a simple but effective way to adjust ó based on the uncertainty of the proxy on each sequence r. The key insight is that even though the proxy predicts the score inaccurately, its uncertainty, which measures how the proxy gives predictions inconsistently, can guide how conservatively we search. Specifically, we define a function that assigns lower values for highly uncertain samples and vice versa: 5(2; o) = gomms Xa (z), where X a scaling factor. We clamp the result to ensure it fall in [0, 1].
Reinforcement learning methods, such as DyNA PPO (Angermueller et al., 2020) and GFlowNets (Bengio et al., 2021; Jain et al., 2022; 2023b; Garfán et al., 2024), and sampling with generative models (Brookes & Listgarten, 2018; Brookes et al., 2019; Das et al., 2021; Song & Li, 2023) aim to search the biological sequence space using a sequential decision-making process with a policy, starting from scratch. Similarly, sampling with gen- erative models (Brookes & Listgarten, 2018; Brookes et al., 2019; Song & Li, 2023) searches the sequence space using generative models like VAE (Kingma & Welling, 2014). While these approaches allow for the creation of novel se- quences, as sequences are generated from scratch, they are relatively prone to incomplete proxy models, particularly in regions where the proxy is misspecified due to being out-of-distribution.
In this adaptive å control, the reduced å leads to more conservative search if the proxy gives inconsistent predic- tions. We estimate (I)) the standard deviation of the proxy model fo (z),, via MC dropout (Gal& & Ghahramani, 2016) or an ensemble method (Lakshminarayanan et al., 2017).
Note that we measure the uncertainty on the observed data points. In practice, deenst is set as 0.5 for short sequences and 0.05 for longer ones, such as proteins, öL tokens are masked on average. While more tailored choices of dennst. could be made with task-specific knowledge. our empirical findings indicate that these simple defaults work well across a variety of settings; see Section 6.
An alternative line of research is evolutionary search (Arnold, 1998; Bloom & Arnold, 2009; Schreiber et al., 2020; Sinai al., 2020; Ren et al., 2022; Ghari et al., 2023: Kirjner et al., 2024), a popular method in biological se- quence design. These methods iteratively edit sequences and constrain new sequences so as not to deviate too far from the seed sequence; usually the wild-type, which occurs in nature. This can be viewed as constrained optimiza- tion, where diitrii-iitiiiiiinn inputs to the proxy model get avoided, as they can lead to unrealistic and low-score biological sequences. As such, they do not aim to produce highly novel sequences.
5. Related Work
5.1. Biological Sequence Design
Designing biological sequences using machine learning methods is widely studied. Bayesian optimization (BO) methods (Mockus, 2005; Belanger et al., 2019; Zhang et al., 2022) exploit posterior inference over newly acquired data points to update a Bayesian proxy model that can measure useful uncertainty. The BO method can be greatly improved
4

---

Improved Off-policy RL in Biological Sequence Design
5.2. Nets
1.0
XED;
y=x
GFlowNets were introduced by Bengio et al. (2021) and unified by Bengio et al. (2023), demonstrating effectiveness across various domains, including language modeling (Hu et al., 2023), diffusion models (Sendera et al., 2024; Venka- traman et al., 2024), and scientific discovery (Jain et al., 2022; 2023a; A14Science et al., 2023). Several works have aimed to improve their training methods for better credit as- signment (Malkin et al., 2022; Madan et al., 2023; Pan et al., 2023; Jang et al., 2024) and extensions to multi-objective settings (Jain et al., 2023b; Chen & Mauch, 2024). Comple- mentary to these, several studies have investigated advanced off-policy exploration strategies (Rector-Broww et al., 2023; Shen et al., 2023; Kim et al., ;0;4d;c;a;b; Madan et al., 2025). Our method is particularly related to these explo- ration methods, yet the major difference is that they are designed under the assumption that the reward model is accurate, which does not hold in active learning and thus requires conservativeness.
Predicted Score
0.8
0.4
0.2
0.0
0.6 True Score
Dist. from Do
<
S 8
Spearman p(f. fo)
0.893
0.799
0.278
0.202
Figure 2. Proxy failure on Hard "F--Bin-8& 8. 's 0' and 's 8' de- note the imitial dataset and the whole sequence space, respectively.
tion 6.2, we evaluate the proposed method on tasks from the FLEXS benehmark (Sinai al., 2020). Section 6.3 provides deeper comparison with GFNSeqEditor and GFN-AL, highlighting 8-CS's balanced use of proxy models. Finally, we investigate noise injection and denoising policies by comparing with local search GflowNets (Kim et al., 2024d) in Section 6.4. All results are reported over five independent runs.? 2
Nets for biological sequence design. GFlowNets were first introduced for active learning in biological se- quence design by Jain et al. (2022), showing that off-policy updates can improve training stability and efficiency by leveraging static annotated datasets. They also incorpo- rated epistemic uncertainty encourage exploration in curiosity- manner. Separately, Ghari et al. (2023) pro- posed GFNSeqEditor, which uses GelowNets as priors for evolutionary editing of biological sequences, rather than for de novo generation. Their flow model is trained offline (i.c., without a proxy model) and used to generate diverse edits for a given input sequence. Our method can be seen as hybrid of off- ooflcyl RL and evolutionary search, capitaliz- ing on both the high novelty offered by GFlowNets and the high rewards with -iit-ii-iitiittionnn robustness provided by constrained search, when they are properly balanced by Our experimental results in Section 6.3 demonstrate that 5-CS achieves balanced exploration.
6.1. Study on proxy failure and the effect of 8-CS
Task: Hard TF-Bind-8. We aim to generate DNA se- quences (length 8) that maximize the binding affinity to the target transcription factor. Comprehensive analysis allowed since the full sequence space is characterized by ex- periments (Barrera et al., 2016). However, the original task, especially due to the large initial dataset, is easy to optimize (Sinai et al., 2020), making performance differences among methods less clear. To clearly verify the effect of 8-CS, we introduce a hard variant by modifying both the initial dataset distribution and the scoring landscape. Specifically, we collect the initial dataset near a certain sequence while ensuring that the initial sequences have lower scores than the given sequence to form dataset Do of size 1,024. We further modify the landscape to assign zero to any sequence scoring below 0.3. These modifications better reflect real- world design scenarios where the search space is vast, data is limited, and many sequences can have near-zero scores.
In this paper, focus on applying 8-CS within GelowNets. Recent work has established theoretical equivalence be- tween GFlowNet training objectives, such as detailed bal- ance and trajectory balance, and well-known off off-policy max- imum entropy RL algorithms, including Soft Q- Learning (Haarnoja et al., 2017) and Path Consistency Learning (Nachum et al., 2017), when appropriate reward correc- tions are applied (Tiapkin et al., 2024; Deleu et al., 2024). The experimental results are also provided in Appendix B.2.
Proxy failure. Figure 2 illustrates the proxy values and true scores for all z X' in the first round. While the proxy provides accurate predictions for the initial data points x E 'Du (represented by the red dots), it produces unreliable predictions for points outside Dp- The correlation between ff and fo significantly decreases as data points move farther from the observed initial dataset (denoted as '< 0' in the
6. Experiments
We first investigate proxy failures and the impact of our conservative search by directly applying 8-CS to GFN-AL (Jain et al., 2022) in Section 6. Subsequently, in Sec-
Available at http://gitaah.ommtyomaaakmmmaa es.
5

---

Improved Off-policy RL in Biological Sequence Design
1.0
region.
0.9
CMA-ES (Hansen, 2006) is a well-known evolutionary algorithm that optimizes a continuous relaxation of one- hot vectors encoding sequence with the covariance matrix.
Mean
0.6
GFN-L
ChAS (Brookes et al., 2019) and DbAS (Brookes & List- garten, 2018) are probabilistic frameworks that use model- based adaptive sampling with a variational autoencoder (VAE; Kingma & Welling, 2014). Notably, CbAS re- stricts the search space with a trust-region search similar to the proposed method.
0.5
6-0.5 Ad
0
10
Rounds
Figure 3. Mcan score over rounds on Hard TF-Bind-8.
table). This supports our hypothesis that the proxy model performs poorly on out-of-distribtioon data.
DyNA PPO (Angermueller et al., 2020) uses proximal policy optimization (PPO; Schulman et al., 2017), an on-policy training method.
Effect of 8 conservativeness. As shown in Figure 3, 6-CS with 8 < significantly improves GFN-AL by focusing on sequences that maintain higher correlation with the ground- truth scores. By restricting the search space to a smaller edit distance, 8-CS better aligns with ground-truie scores. By limiting the search within these constrained edit distances, 5-CS enhances the correlation with the oracle. Even with a fixed 8, Figure in Appendix B.6 shows consistent gains. We also confirm similar improvements on the anti-microbial peptide design (AMP) task from Jain et al. (2022); detailed results in Appendix B.1 show that 5-CS consistently outper- forms the original GFN-AL.
GFN-AL (Jain et al., 2022) is our main baseline that uses GFN with Bayesian active learning.
Experiment setup. For each task, we conduct 10 active learning rounds starting from the initial dataset Do- The query batch size is all set as 128. Further details of each task are provided in the following subsections. To evaluate the performance, we measure the maximum, median, and mean scores of Top-K sequences.
6.2.1. RNA SEQUENCE DESIGN
Task. The goal is to design an RNA sequence that binds to the target with the lowest binding energy, which is measured by ViennaRNA (Lorenz et al., 201 1). The length (L) of RNA is set to 14, with 4 tokens. In this paper, we have three RNA binding tasks, RNA-A, RNA-B, and RNA-C, whose initial datasets consist of 5,000 randomly generated sequences with certain thresholds; adopt the offline dataset provided in Kim al. (2023). We use 5 0.5 and X 5.
6.2. Experiments on FLEXS Benchmark
Implementation details. For proxy models, we employ a convolutional neural network with one-dimensional con- volutions according to (Sinai et al., 2020). We use a UCB acquisition function and measure the uncertainty with an ensemble of three network instances. We use &cunst 0.5
for DNA (L- 8) and RNA (L 14) sequence design and 5 = 0.05 for protein design (L = 238, 90). Lastly, we set A to satisfy (EDu(z)(r((z) ss I based on the observations from
Results. As shown in Figure 4, our method outperforms all baseline approaches. The curve in Figure 4 increases significantly faster than the other methods, indicating that 8-CS effectively trains the policy and generates appropriate queries in each active round. See Appendix C.I in details.
the initial round. More details are provided Appendix A.1 and Appendix A.2.
Baselines. As our baseline methods, we employ representa- tive exploration algorithms. We use the same architecture to implement proxy models for all baselines except for GFN- AL, where we adopt the original implementation. Further details are provided in Appendix A.3.
6.2.2. DNA SEQUENCE DESIGN
Task. In this task, we follow the setup from the previous studies (Sinai et al., 2020: Jain et al., 2022: Trabucco et al., 2022; Kim et al., 2023). The initial dataset 2p is the bottom 50% in terms of the score, which results in 32, 898 samples, with a maximum score of 0.439. Similar to RNA, we use 8 0.5 and X 5.
AdaLead (Sinai et al., 2020) is model-guides evaluation method with a hill-climbing algorithm.
Bayesian optimization (BO; Snoek et al., 2012) is a clas- sical algorithm for black-box optimization. We employ the BO algorithm with a sparse sampling of the mutation space implemented by Sinai et al. (2020).
Results. In TF-Bind-8, every method successively discovers DNA sequences with high binding affinity scores in the early rounds. Nevertheless, the results show that 8-CS further improves GEN-AL by restricting search space; see the max, median, diversity, and novelty in Appendix C.2. In addition, 5-CS demonstrates the best median scores, as well.
TuRBO (Eriksson et al., 2019) is a Bayesian optimization algorithm that partitions search space into promising local regions and performs the optimization process within the
6

---

Improved Off-policy RL in Biological Sequence Design
Ours
GFN-AL
AdaLead
CbAS
DbAS
DynaPPO
CMAES
B0
TuRBO
RNA-A
RNA-B
RNA-C
1.0
0.8
0.8
Score
0.6
0.4
0.4
0.2
10
TF-Bind-8
GFP
AAV
1.0
0.65
Score
0.8
0.60
0.55
3.4
2
10
2
10
2
10
Rounds
Rounds
Rounds
Figure 4. Mean scores of Top- 128 over active rounds. Ours (GFN-AL 8-CS) consistently outperforms baseline in RNA, DNA
(TF-Bind-8), 8), and protein (GFP and AAV) design tasks.
6.2.3. PROTEIN SEQUENCE DESIGN
ity of our search strategy in practical biological sequence design tasks. Full results are provided in Appendix C.3.
We consider two protein sequence design tasks: the green fluorescent protein (GFP; Sarkisyan al., 2016) and addi- tive aedeno-associatee virus (AAV; Ogden al., 2019).
6.3. Achieving Pareto Frontier with Balanced Conservativeness
GFP. The objective is identify protein sequences with high log- olgeeluoreeeeece intensity values. 3 The vocabulary is defined as 20 standard amino acids, i.e.. |V| = 20, and the sequence length L is 238; thus, we set 8 as 0.05 and as 0.1, according to our guideline. The initial datasets are generated by randomly mutating the provided wild-type sequence for each task while filtering out sequences that have higher scores than the wild-type; obtain the initial dataset with (Dul 10 200 with maximum score of 3.572.
In this analysis, we demonstrate that 8-CS achieves a bal- anced search using 8, producing Pareto frontiers or com- parable results to the baseline methods: GFN-AL (Jain et al., 2022) and GFNSeqEditor (Ghari et al., 2023). No- tably, GFN-AL can be seen as variant of our method with 8 = 1, which fully utilizes the entire trajectory search. This approach is expected to yield high novelty and diversity, but it is also prone to generating low rewards due to the increased risk of out-of-distributioo samples affecting the proxy model. GFNSeqEditor, on the other hand, leverages GFlowNets prior, editing from wild-type sequence. It is designed to deliver reliable performance and be more ro- bust to to sequences similar to the wild type. However, unlike 8-CS, GFNSeqEditor does not utilize such obtained samples for training GelowNets full trajectory level; GFNSeqEditor is expected to have lower diversity and novelty compared GFN-AL and 8-CS.
AAV. The aim is to discover sequences that lead to higher gene therapeutic efficiency. The sequences are composed of the 20 standard amino acids with length of 90, resulting in the search space of 20 In the same way as GFP, we collect an initial dataset of 15,307 sequences with 0.05 and 1.
issues by constraining the search
maximum score of 0.500. We
Figure 4 shows the results of all methods in pro- tein sequence design tasks. Given the combinatorially vast
Results.
design space with sequence lengths = 238 and 90, most baselines fail to discover new sequences whose score is higher than the maximum of the dataset. In contrast, our method finds high-score sequences beyond the dataset, even with single active round. This underscores the superior-
As shown in Figure 5, GFN-AL generally produces higher diversity and novelty in the RNA and GFP tasks compared to GFNSeqEditor: However, GFNSeqEditor performs better in terms of reward on the large-scale GFP task, whereas GFN-AL struggles due to the lack of constrained search procedure in such large combinatorial space. In contrast, 8- CS achieves Pareto-optimal performance compared to both
*FLEXS uses TAPE (Rao al., 2019), whereas Jain al. (2022) employs the Design-Bi Design-Bench Transformer (Trabucco et al., 2022). Additional results are provided in Appendix B.5.

---

Improved Off-policy RL in Biological Sequence Design
Oure
GFN-AL
GFNSeqEdllor
RNA-A (L = 14)
GFP (L 238)
AAV (L 90)
0.95
0.66
3.56
Avg. Score
Avg. Score
0.80
0.60
3.54
Avg.
0.85
3.52
0.80
3.50
0.48
60
5.0
7.5
10.0
12.5
Diversity
Diversity
Diversity
0.95
0.88
3.56
Avg. Score
Avg. Score
Avg. Score
0.90
0.60
3.54
0.85
3.52
0.54
0.80
3.50
0.48
7.4
10
2.5
5.0
10.0
Novelty
Novelty
Novelty
Figure 5. Average score and diversity/novelty with five independent runs. Our method (GFN-AL + 5-CS) consistently approaches Pareto frontier performance. We set = 0.5 for short sequences (L & 50) and set 0.05 for long length sequences (L > 50).
Score
does not inherently use proxy models, we adapt it for a con- servative search comparison by setting its backtracking step K to 8L, aligning its level of conservativeness with ours.
As illustrated in Figure 6, our method achieves more bal- anced trade-off's among high scores, diversity, and novelty than LS-GFN, likely due to its flexibilityy in selecting which tokens to mask. In an auto-regressive sequence genera- tion setting, the backtracking in LS-GFN destroys only the last tokens, limiting the local search region, whereas our approach can randomly mask any token in a sequence. Moreover, as shown in Table 3 and Figure 8, our approach demonstrates it remains more robust when the initial dataset is restricted to lower-quality sequences; see Appendix B.4.
Novelly
Diversity
Novelty
Diversity
Back-ans-forth
Back-ond-forth
aa) RNA-A
(b) GFP
Figure Comparison 8-CS with back and- forth search LS-GEN. Further results are provided in Appendix B.3
methods, clearly outperforming GFNSegEditor across six tasks, with higher rewards, diversity, and novelty. For the RNA and GFP tasks, we achieve higher scores than GFN- AL while maintaining similar novelty but slightly lower diversity. the AAV task, 5-CS shows a distinct Pareto improvement. These results demonstrate that 8-CS pro- vides beneficial balance by combining conservative search with amortized inference on full trajectories using off-policy GelowNets training, effectively capturing the strengths of both GFN-AL and GFNSeqEditor. The results with various 6 are provided in Appendix B.8.
6.5. Further Studies
Studies the effect of 5-CS with various off-policy RL.
Studies on 6. We study the choice of dconst and the ef- fectiveness of the adaptive 8(x, In Appendix B.7 and Appendix B.8, we investigate how varying &canst affects per- formance. While 8-CS generally outperforms GFN-AL, its gains diminish as domnst grows, because å 1 reverts to on-policy exploration without any conservative constraint. Furthermore, the results in Appendix B.6 verify the ben- efit of adaptive 8(I, o).. While even a fixed ò improves GFN-AL on Hard TF-Bind-8, adaptive 8 yields additional performance gains in overall.
6.4. Comparison with Back-and-Forte Search
The noise injection and denoising policy shares similarities with the back-and-fortt strategy suggested in local search GFlowNets (LS-GFN; Kim et al., 2024d). LS-GFN uses a backward policy to partially destroy a solution and a for- ward policy to reconstruct a new solution. While L.S-GNN
Studies on the varying proxy qualities. We further vali- date 8-CS under degraded proxy models by truncating the initial dataset its 50%. 25%, and 10% score percentiles.
8

---

Improved Off-policy RL in Biological Sequence Design
As shown in Figure 8, across all truncation levels, 8-CS con- sistently outperforms GFN-AL and LS-GFN, confirming that incorporating conservative search is crucial for robust- ness against proxy misspecification; the details are provided in Appendix B.4.
References
AI4Science, M., Hernandez-Gareia, A., Duval, A., Volokhova, A., Bengio, Y., Sharma, D., Carrier, P. L., Benabed, Y., Koziarski, M., and Schmidt, V. Crystal- GFN: sampling crystals with desirable properties and constraints. arXiv preprint arXiv:2310.0492.. 2023.
7. Conclusions and discussion
Angermueller, C., Dohan, D., Belanger, D., Deshpande, R., Murphy, K., and Colwell, L. Model-based reinforcement learning for biological sequence design. International Conference on Learning Representations (ICLR), 2020.
In this paper, we introduced a novel off-policy sampling method for GFlowNets, called 8-CS, which provides con- trollable conservativeness through the use of a á parameter. Additionally, we proposed an adaptive conservativeness ap- proach by adjusting for each data point based on prediction uncertainty. We demonstrated the effectiveness of 6-CS in active learning GFlowNets, achieving strong performance across various biological sequence design tasks, including DNA, RNA. protein, and peptide design, consistently out- performing existing baselines.
Arnold, F. H. Design by directed evolution. Accounts of chemical research, 31(3):125-133 1998.
Barrera, L. A., Vedenko, A., Kurland. J. V., Rogers, J. M., Gisselbrecht, S. S., Rossin, E. J., Woodard, J., Mariani, L., Kock, K. H., Inukai, S., et al. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science, (5166280):4450-1455 2016.
Limitations and future works. The main limitation of our method is that it does not fundamentally resolve the draw- backs of active learning: it serves as a useful tool within the existing framework. Investigating robust proxy training and uncertainty measurement techniques remains necessary. These improvements are orthogonal to our approach and can enhance 5-CS when integrated. Future work includes combining 8-CS with existing GelowNet methods. For ex- ample, applying improved credit assignment for larger-scale tasks (Jang et al., 2024) and extending to multi-objectivv settings (Jain et al., 2023b; Chen & Mauch, 2024) could significantly boost its applicability and effectiveness.
Belanger, D., Vora, S., Marict, Z., Deshpande, R., Dohan, D., Angermueller, C., Murphy, K., Chapelle, O., and Colwell, L. Biological sequence design using batched Bayesian optimization. In NeurIPS 2019 Workshop Machine Learning and the Physical Sciences, 2019.
Bengio, E., Jain, M., Korablyov, M., Precup, D., and Ben- gio, Y. Flow network based generative models for non- iterative diverse candidate generation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Bengio, Y., Lahlou, S., Deleu, T., Hu, J., Tiwari, M., and Bengio, F. GFlowNet foundations. Journal of Machine Learning Research, 24(210):1555, 2023.
Impact Statement
This work tackles a common challenge in scientific discov- ery: generating candidates with limited data and unreliable proxy models. In biological sequence design, we intro- duce a conservative search strategy that adapts exploration range based on proxy uncertainty to reduce reward hacking. While developed for this domain, the principle of align- ing exploration with model confidence may generalize to other domains where generative models and active learning pipelines face similar constraints.
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., and Church, G. M. Low-N protein engineering with data- efficient deep learning. Nature methods, 18(4):389-396 2021.
Bloom, J. D. and Arnold, F. H. In the light of directed evolu- tion: pathways of adaptive protein evolution. Proceedings of the National Academy of Sciences, 106:9995-10000, 2009.
Brookes, D., Park, H., and Listgarten, J. Conditioning by adaptive sampling for robust design. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Acknowledgements
Minsu Kim is supported by KAIST Jang Yeong Sil Fellow- ship and the Canadian AI Safety Institute Research Program at CIFAR through a Catalyst award. This work was sup- ported by the National Research Foundation of Korea (NRF)
Brookes, D. H. and Listgarten, J. Design by adaptive sam- pling. arXiv preprint arXiv: 1ix.:vv1100007144 2018.
grant funded by the Korean govermment (MSIT) (RS-2025- 00563763).
Chen, C., Zhang, Y., Fu, J., Liu, X. S., and Coates, M. Bidirectional learning for offline infinite-width model- based optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

---

Improved Off-policy RL in Biological Sequence Design
Chen, C., Zhang, Y., Liu, X., and Coates, M. Bidirectional learning for offline model- biological sequence de- sign. In International Conference on Machine Learning (ICML), 2023a.
Hansen, N.
The CMA evolution strategy: comparing review. Towards New Evolutionary Computation: Ad- vances in the Estimation of Distribution Algorithms, pp. 75-102, 2006.
Chen, C. S., Beckham, C., Liu, Z., Liu, S., and Pal, C. martorng-mentoring for offline model-based optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 76619-76634 2023b.
Hernández-Garcia. A., Saxena, N., Jain, M., Liu, C.- H., and Bengio, Y. Multi-fidelity active learning with GetowNets. Transactions on Machine Learning Re- search, 2024. ISSN 2835-8856. URL https://
openreview net httt/forummidddlaaaee pert Certification.
Ex-
Chen, Y. and Mauch, L. Order-preserving GFlowNets. Inter- national Conference on Learning Representations (ICLR), 2024.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8): 1997. doi: 10. 1162/ncco.1997.93
Chow, Y., Nachum, O., and Ghavamzadeh, M. Path con- sistency learning in Tsallis entropy regularized MDPs. In Proceedings of the 35th International Conference Machine Learning (ICML), pp. 979 979-988.10 PMLR, 2018. 8.
Hu, E. J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., and Malkin, N. Amortizing intractable infer- ence in large language models, 2023.
Das, P. Sercu, T., Wadhawan, K., Padhi, I., Gehrmann, S., Cipeigan, F., Chenthamarakshan, V., Strobelt, H., Dos Santos, C., Chen, P.-Y., et al. Accelerated antimicro- bial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering, 5(6):613-623, 2021.
Jain, M., Bengio, E.,, Hernandez-Garei... A., Rector-Brrodee J., Dossou, B. F., Ekbote, A., Fu, Zhang, T., Kilgour, M.. Zhang, D., et al. Biological sequence design with GFlowNets. In International Conference on Machine Learning (ICML), 2022.
Jain, M., Deleu, T., Hartford, J., Liu, C.-H., Hernandez- Garcia, A., and Bengio, Y. GFlowNets for Al-driven scientific discovery. Digital Discovery, 233)557-5577, 2023a.
Deleu, T., Nouri, P., Malkin, N., Precup. D., and Bengio, Y. Discrete probabilistic inference as control in multi-path environments. In Uncertainty in Artificial Intelligence (UAI), 2024.
Jain, M., Raparthy, S. C., Hernández-Garcaa, A., Rector- Brooks, J., Bengio, Y., Miret, S., and Bengio, E. Multi- objective gflownets. In International Conference on Ma- chine Learning (ICML), 2023b.
Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable global optimization via local bayesian optimization. Neural Information Processing Systems (NeurIPS), 2019.
Jang, H., Kim, M., and Ahn, S. Learning energy decomposi- tions for partial inference of GFlowNets. In International Conference on Learning Representations (ICLR), 2024.
Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approxi- mation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning (ICML), 2016.
Khan, A., Cowen-Rivers, A. I., Grosnit, A., Robert, P. A., Greiff, V., Smorodina, E., Rawat, P., Akbar, R., Dreczkowski, K., Tutunov, R., et al. Toward real-world automated antibody design with combinatorial bayesian optimization. Cell Reports Methods, 3(1), 2023.
Ghari, M., Tseng, A., Eraslan, G., Lopez, R., Biancalani, T., Scalia, G., and Hajiramezanali, E. Generative flow networks assisted biological sequence editing. In NeurIPS 2023 Generative Al and Biology (GenBio) Workshop, 2023. URL https:://///..... /ttp:////peeeeeieeeee net. forum? id- (-0-9-001110
Kim, H., Kim, M., Choi, S., and Park, Genetic-guided GFlowNets for sample efficient molecular optimization. In Neural Information Processing Systems (NeurIPS), 2024a.
Gruver, N., Stanton, S., Frey, N., Rudner, T. G., Hotzel, I,, Lafrance-Vanasse, J., Rajpal, A., Cho, K., and Wil- son, A. G. Protein design with guided discrete diffusion. Neural Information Processing Systems (NeurlPS), 2024.
Kim, M., Berto, F., Ahn, S., and Park, J. Bootstrapped train- ing of score-conditioned generator for offline design of biological sequences. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep mmergy-based policies. In International Conference on Machine Learning (ICML), 2017.
Kim, M., Choi, S., Son, J., Kim, H., Park, J., and Bengio, Y. Ant colony sampling with GFlowNets for combinatorial optimization. arXiv preprint urXiv:2403.070.. 2024b..
2024b.
10

---

Improved Off-policy RL in Biological Sequence Design
Kim, M., Ko, J., Zhang, D., Pan, L., Yun, T., Kim, W., Park, J., and Bengio, Y. Learning scale logits for temperature- conditional GFlowNets. International Conference Machine Learning (ICML), 2024c.
Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y. Trajectory balance: Improved credit assignment in GFlowNets. In Advances in Neural Information Process- ing Systems (NeurlPS), 2022.
Kim, M., Yun, T., Bengio, E., Zhang, D., Bengio, Y., Ahn, S., and Park, J. Local search GFlowNets. In International Conference on Learning Representations (ICLR), 2024d.
Mockus. J. The bayesian approach global optimization. In System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31 31-September 4, 1981, pp. 473 481. Springer, 2005.
Kim, M., Gu, J., Yuan, Y., Yun, T., Liu, Z., Bengio, Y., and Chen, C. Offline model-based optimization: Comprehen- sive review. arXiv preprint 2503: 17286, 2025.
Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Bridging the gap between value and policy based rein- forcement learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Kingma, D. P. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Ogden, P. J., Kelsic, D., Sinai, S., and Church, G. M. Comprehensive AAV capsid fitness landscape reveals viral gene and enables muchine-guides design. Science, 366(6469): 1111-11911111 143, 2019.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference Learning Repre- sentations (ICLR), 2014.
Pan, I.., Malkin, N., Zhang, D., and Bengio, Y. Better training of GFlowNets with local credit and incomplete trajectories. International Conference Machine Learn- ing (ICML), 2023.
Kirjner, A., Yim, J., Samusevich, R., Bracha, S., Jaakkola, T. S., Barzilay, R., and Fiete, I. R. Improving protein optimization with smoothed fitness landscapes. In Inter- national Conference on Learning Representations (ICLR), 2024.
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song, Y. Evaluating protein transfer learning with TAPE. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Kumar, A. and Levine, S. Model inversion networks for model-based optimization. Advances in neural informa- tion processing systems (NeurIPS), 2020.
Rector-Brooks, J., Madan, K., Jain, M., Korablyov, M., Liu, C.-H., Chandar, S., Malkin, N., and Bengio, Y. Thomp- sampling for improved exploration in GFlowNets. In ICML 2023 Structured Probabilistic Inference & Genera- live Modeling (SPIGM) Workshop. 2023.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Process- ing Systems (NIPS), 2017.
Ren, Z., Li, J., Ding, F., Zhou, Y., Ma, J., and Peng, J. Proximal exploration for model-guided protein sequence design. In Proceedings of the 39th International Confer- ence Machine Learning (ICML), 2022.
Lee, M., Vecchietti, L. F., Jung, H., Ro, H. J., Cha, M., and Kim, H. M. Robust optimization in protein fitness land- scapes using reinforcement learning in latent space. In International Conference on Machine Learning (ICML), 2024.
Richter, L., Boustati, A., Nüsken, N., Ruiz, F., and Akyildiz, O. D. VarGrad: A low-variance gradient estimator for variational inference. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Lorenz, R., Bernhart, S. H., Höner Siederdissen, C., Tafer, H., Flamm, C.. Stadler, P. F., and Hofacker, I. L. Vien- naRNA package 2.0. Algorithms for molecular biology, 6:1-14, 2011.
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z., al. A tutorial on Thompson sampling. Foundations and Trends@ in Machine Learning, 11(1):1-9, 2018.
Madan, K., Bector-Brooos, J.. Korablyov, M., Bengio, E., Jain, M., Nica, A. C., Bosc, T., Bengio, Y., and Malkin, N. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning (ICML), 2023.
Sample, P. J., Wang, B., Reid, D. W., Presnyak, V., Mc- Fadyen, J., Morris, D. R., and Seelig, G. Human UTR design and variant effect prediction from a mas- sively parallel translation assay. Nature Biotechnology. 37(7):803 809, 2019.
Madan, K., Lamb, A., Bengio, E., Berseth, G., and Ben- gio, Y. Towards improving exploration through sibling augmented GelowNets. In International Conference on Learning Representations (ICLR), 2025.
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N.,

---

Improved Off-policy RL in Biological Sequence Design
Bozhanova, N. G., Baranov, M. S., Soylemez, O., al. Local fitness landscape of the green fluorescent protein. Nature, 533(77033:397-441,, 2016.
Trabucco, B., Kumar, A., Geng, X., and Levine, S. Conser- vative objective models for effective offline model-based optimization. In International Conference Machine Learning (ICML), 2021.
Schreiber, J., Lu, Y. Y., and Noble, W. S. Ledidi: Designing genomic edits that induce functional activity. BioRxiv, pp. 2020-05, 2020.
Trabucco, B., Geng, X., Kumar, A., and Levine, S. Design- Bench: Benchmarks for data-driven offiin model-based optimization. In International Conference Machine Learning (ICML), 2022.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXhv:1007.06347, 2017.
Tripp. A., Daxberger, E., and HemánneezLobatto, J. M. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sendera, M.. Kim, M., Mittal, S., Lemos, P., Scimeca, L., Rector-Brooks, J., Adam, A., Bengio, Y., and Malkin, N. Improved off off-policy training of diffusion samplers. Neural Information Processing Systems (NeurIPS), 2024.
Venkatraman, S., Jain, M., Seimeca, L.,, Kim, M.. Sendera, M., Hasan, M., Rowe, L., Mittal, S., Lemos, P., Bengio, E., et al. Amortizing intractable inference in diffusion models for vision, language, and control. Neural Infor- mation Processing Systems (NeurIPS), 2024.
Shen, M. W., Bengio, E., Hajiramezanali, E.,, Loukas, A., Cho, K., and Biancalani, T. Towards understanding and improving GFlowNet training. In International Confer- ence on Machine Learning (ICML), 2023.
Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisa- tion over high-dimensional categorical and mixed search spaces. International Conference on Machine Learning (CCL),20211
Sinai, S., Wang, R., Whatley, A., Slocum, S., Locane, E., and Kelsic, E. D. AdaLead: A simple and robust adap- tive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141, 2020.
Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NIPS), 2012.
Yu, S., Ahn, S., Song, L., and Shin, J. RoMA: Ro- bust model adaptation for offline model-based optimiza- tion. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yuan, Y., Chen, C. S., Liu, Z.., Neiswanger, W., and Liu, X. S. Importance-aware co-teaching for offline model- based optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 55718- 55733, 2023.
Song, Z. and Li, L.. Importance weighted expectation- maximization for protein sequence design. In Proceed- ings of the 40th International Conference on Machine Learning, 2023.
Srinivas, N., Krause, A., Kakade, S., and Seeger, M. Gaus- sian process optimization in the bandit setting: no regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
Yun, T., Yun, S., Lee, J., and Park, J. Guided trajectory generation with diffusion models for offline model-based optimization. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Stanton, S., Maddox, W., Gruver, N., Maffettone, P., De- laney, E., Greenside, P., and Wilson, A. G. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning (CCML),20))
Zhang, D., Fu, J., Bengio, Y., and Courville, A. Unifying likelihood free inference with black-box sequence design and beyond. In International Conference on Learning Representations (ICLR), 2022.
Zimmer, M. Green fluorescent protein (GFP): applications, structure, and related photophysical behavior. Chemical reviews, 102(3):75--782, 2002. 2002.
Surana, S., Grinsztajn, N., Atkinson, T., Duckworth, P., and Barrett, T. D. Overconfident oracles: Limitations of in silico sequence design benchmarking. In ICML 2024 Al for Science Workshop, 2024. URL https: //operrview net. www/ PP-
Tiapkin, D., Morozov, N., Naumov, A., and Vetrov, D. Gen- erative flow networks as reguaay-re RL. Artifi- cial Intelligence and Statistics (AISTATS), 2024.
12

---

Improved Off-policy RL in Biological Sequence Design
A. Implementation detail
Algorithm Active Learning GFlowNets with 8-CS
Input: oracle f. initial dataset Do, active rounds T, query size B, training batch size M.
2: for = 1 to 7 do
3: STEP A: PROXY TRAINING
4: for k = 1 to numProxyTrall in do
5: train proxy (a(x) with current round dataset Dz-1: L(Ø) Pa (0) (U(x)-
6: end for
7:
STEP B: POLICY TRAINING >
8:
for k = to numP commooliyyrrern do
9:
obtain off-policy trajectories fI,..., FM from f given by d-CS (D,-1,
10:
obtain offline trajectory ......, TM from 21...., 2M ~ (T).
train 0 with TB loss over A ,...., fM and ;
M
Z ZoPF 2
M
ZgP= (f:F(f;:)) 2
((
Σ 1=1
log
log
2M
R(E;0)
2M 1=1
R(0;;))
end for
STEP C: DATASET AUGMENTATION WITH ORACLE f QUERY 14: obtain query samples ê1, 21...., fg from 8-CS (De-1, B. 8).
15: 'D= < 1 U ((i,, ((:)))
16: end for
Algorithm Sampling with 8-CS
1: Input: Dataset 1. batch size M, conservative parameter
sample high reward data .i......2. .M with rank-based reweighed prior PD..I (-).
3: obtain masked data ø1,-.. fas with noise injection policy Proise (F 5) from 21, 01,--.:: RMM
4:
obtain denoised f with denoising policy P (? 1; 7) from T1.-. E1..... 5M.
5:
return LT. , LM.
A.1. Proxy training
For training proxy models, follow the procedure of (Jain et al., 2022). We Adam (Kingma, 2015) optimizer with learning rate x 10 and batch size of 256. The maximum proxy update is set as 3000. To prevent over-fitting, early stopping using the 10% of the dataset validation set and terminate the training procedure if validation loss does not improve for five consecutive iterations.
A.2. Policy training
As described in Section 6, we employ two-layer long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) with 512 hidden dimensions. The policy is trained with learning rate of 5 x 10-4 with batch size of 256. The learning rate of is set as 10-3 The coefficient x in Section 3.2 is set as 0.1 for TF-Bind-S and AMP with MC dropout, according to Jain al. (2022), and 1.0 for RNA and protein design with Ensemble following Ren et (2022).
A.3. Implementation details of baselines
We adopt open-source code from FLEXS benchmark (Sinai et al., 2020).
AdaLead (Sinai et al., 2020): We use a default settings of hyperparmeters for AdaLead. Specifically, we use a recombination rate of 0.2, mutation rate 1/L, where I. is sequence length, and threshold 0.05.
13

---

Improved Off-policy RL in Biological Sequence Design
DbAS (Brookes & Listgarten, 2018): We implement DbAS with variational autoencoder (VAE; Kingma & Welling, 2014) the generator. The input one-hot encoding vector, and the output latent dimension is 2. In each cycle, DbAS starts by training the VAE with the top 20% sequences in terms of the score.
CbAS (Brookes al., 2019): Similar DbAS, we implement CbAS with VAE. The main difference from DbAS is that select top 20% sequences with the weights p(ær ( //(=) g where p(:; ((0) is trained with the ground-truth samples and (()) is trained on the generated sequences over training rounds.
DyNA PPO (Angermueller et al., 2020): We closely follow the algorithm presented in (Angermueller et al., 2020). For a fair comparison, we use CNN ensembles parameterize the proxy model instead of suggested architectures.
CMA-ES (Hansen, 2006): We implement covariance matrix adaptation evolution strategy (CMA-ES) for sequence generation. As the generated samples from CMA-ES are continuous, we convert the continuous vectors into one-hot representation by computing the argmax at each sequence position.
BO (Snock et al., 2012): We use classical GP-BO algorithm for all tasks. For Gaussian Process Regressor (GPR), default setting from the sklears library. For the acquisition function, they use Thompson sampling (Russo et al., 2018).
TuRBO (Eriksson et al., 2019): We closely follow the algorithm presented in (Eriksson al., 2019). As it supports continuous search space, convert the continuous vectors into one-hot representation by computing the argmax each sequence position.
Furthermore, we employ GFN-AL and GFNSeqEditor. We adopt the original implementation and setup for TF-Bind-8 and AMP. For newly added tasks, report better results among the original MLP policy and the LSTM policy. Note that GFP in FLEXS is different from the one employed in GFA-AL; we treat this as new task based on the observation in the work from Surana et al. (2024).
GFN-AL (Jain et al., 2022): We strictly follow hyperparameters the original code in they conduct experiments on TF-Bind-8 and AMP. The proxy is parameterized using an MI.P with two layers 2,048 hidden. For the policy, a 2-layer MLP with 2,048 hidden dimensions is used, but we also test it with 2-layer LSTM policy.
EFiter (Ghari et al., 2023): We implemented the editing procedure on top of the GFN-AL. Note that GFNSe- qEditor does not utilize the proxy model, so the GFlowNets policy is trained using offine data only with the same policy training procedure of GFN-AL. GFNSeqEditor can also implicitly control the edit percentage with its hyperparameters, which are set d 0.01, 0.0001, À 0.1 in this study. Note that ö the conservativeness parameter.
14

---

Improved Off-policy RL in Biological Sequence Design
B. Further studies
B.I. Anti-microbial peptide design
Task setup. The goal is to generate protein sequences with anti anti-microbial properties (AMP). The vocabulary size [V] = 20, and the sequence length (L) varies across sequences, and we consider sequences of length 50 or lower. For the AMP task, consider much larger query batch size for each active round because they be efficiently synthesized and evaluated (Jain al., 2022). We set ó as 0.5 with À 1.
Table 1. Results on AMP with different acquisition functions (UCB, EI), The mean and standard deviation from five runs are reported. Improved results with 8-CS over GFN-AL are marked in bold.
Max
Mean
Diversity
Novelty
COMs
0.930 + 0.001
0.920
0.000
0.000
0.000
11.869 = 0.298
DyNA PPO
0.953 + 0.005
0.941
0.012
18. + 5.109
16.556 + 3.653
GFN-AL (UCB)
0.936 +: 0.004
0.919 + 0.005
28.504 + 2.691
19.220 = 1.369
GFN-AL + 6-CS (UCB)
0.948 0.015
0.938
0.016
25.379 + 3.735
23.551 + 1.290
GEN-AL (EI)
0.950 + 0.002
0.940
0.003
15.576 + 7.896
21,810 = 4.165
GFN-AL + 6-CS (EI)
0.962 0.003
0.958 0.004
16.631
2.135
24.946 + 4.246
Results. The results in Table illustrate that ours consistently gives improved performance over GFN-AL regardless of acquisition function. According to the work from Jain et al. (2022), also adopt conservative model-based optimization method, (COMs; Trabucco et al., 2021) and on-policy reinforcement learning, DyNA PPO (Angermueller et al., 2020) as baselines. Our method demonstrated significantly higher performance in terms of mean, diversity, and novelty compared to the baselines.
15

---

Improved Off-policy RL in Biological Sequence Design
B.2. 8-CS with various off-policy RL
We extend our experiments on protein design tasks to include three soft off-policy RL algorithms: Soft Q-Learning (SQL) (Haarnoja et al., 2017), Log-Partitte Variance Gradient (VarGrad: Richter al., 2020), and Soft Path Consistency Learning (Soft PCL) (Nachum et al., 2017; Chow al., 2018; Madan et al., 2023), yielding four baselines in total (TB, SQL, VarGrad, and Soft PCL). Trajectory Balance (TB) and VarGrad apply constraints over full trajectories, resulting in higher variance but lower bias. TB explicitly estimates state flows or terminal partition functions, whereas VarGrad does so implicitly via batch averaging. Soft PCL operates over sub-trajectories (akin to TD-A), balancing bias and variance through trajectory length. SQL relies on one-step transitions, reducing variance but increasing bias. For broader discussion on connections between off off-policy RL in discrete sampling, see the works by Tiapkin al. (2024) and Deleu et (2024).
Table 2. Mean of Top-128 after ten active rounds
GFP
AAV
SQL
3.331 +: 0.000
0.480
0.000
SQL+ 8-CS
3.573 + 0.003
0.495 0.001
VarGrad
3.331 + 0.000
0.480 + 0.000
VarGrad + 5-CS
3.567 + 0.003
0.668 0.011
Soft PCL (A=0.9)
3.331 + 0.000
0.480 1 0.000
Soft PCL. (A=0.9) 6-CS
3.578 + 0.004
0.622 + 0.011
As shown in Table 2, 5-CS consistently improves performance across all methods, demonstrating its plug-and-pla versatility. Since the global sequence structure is crucial in protein design, full-trajectory methods (TB and VarGrad with 5-CS) tend to outperform shorter-horizon approaches.
16

---

Improved Off-policy RL in Biological Sequence Design
B.3. Comparison with back-and-forth search in LS-GEN
Our search with noise injection and denoising policies shares similar insight with back-and-forth search in local search GFlowNets (LS-GFN) Kim et al., 2024d). They both induce new sequences by partially revising given sequences. In particular, LS-GFN incorporates back-and-forth search strategy that partially backtracks trajectories using backward policy and reconstructs them using forward policy of the GFN. This section compares the search strategies under the conservative search; we control the conservativeness of LS-GFN by setting the backtracking step as dL.
Score
Score
Score
Novelty
Diversity
Novelty
Diversity
Novelty
Diversity
Ours
Back-and-forth
(a) RNA-B
(b) RNA-C
(c) AAV
Figure 7. Comparison d-CS with back-and-forth search in LS-GEN on RNA-B,C and AAV
17

---

Improved Off-policy RL in Biological Sequence Design
B.4. Studies with smaller initial dataset protein designs
In this section, we extend our ablation study by comparing our method, 8-CS, against two baselines: GFN-AL and an additional off off-po search method, LS-GFN (Kim al., 2024d). This study evaluates the performance of 6-CS under varying initial dataset sizes, Poo 1, 000, and compares the results to the original ablation study setup.
As shown in Table 3, our 6-CS demonstrates a substantial advantage over both GFN-AL and its improved variant, LS-GFN, which leverages back-and-forth search. The results highlight the effectiveness of 6-CS in enhancing GFN training by enabling a more robust and conservative off-policy search, which is critical for improving proxy-based active learning.
Table Ablation study results with 1,000 initial datapoints for GFP and AAV tasks, showing maximum values achieved after active learning.
Method
GFP
AAV
Adalead
3.568 + 0.005
0.557 + 0.023
GFN-AL
3,586 + 0.006
0.560 + 0.008
GFN-AL + LS (Kim et al., 2024d)
3.580 + 0.003
0.493 0.006
GFN-AL + 6-CS
3.591 + 0.007
0.704 = 0.024
To further verify that 8-CS is robust to proxy misspecification compared to other GFN methods, conducted additional experiments to test whether this hypothesis holds at different levels of proxy model quality.
To degrade the proxy model quality, we truncated the initial dataset at different evels-50%, 25%. and 10% percentiles based on reward values. Proxy models trained on datasets with lower percentile cutoffs are more misspecified for higher-reward data points, making GFN training and search more challenging. Under these circumstances, we compared our method with CFN-AL and LS-GFN (Kim al., 2024d) GFN baselines.
GFN-AL 6-CS
GFN-AL LS
GFN-AL
0.4
Maximum Values
0.3
0.2
0.1
0.0
50th
25th
10th
Initial dataset
Figure 8. Maximum values achieved after active learning with varying initial dataset quality (AAV task).
As shown in Figure 8, the performance decreases as the percentile decreases, which is expected because the proxy quality deteriorates significantly. Among the baselines, our method consistently provides substantially better performance than the others. This demonstrates that our hypothesis- that a conservative search with 6-CS is necessary holds across different levels of proxy model quality.
18

---

Improved Off-policy RL in Biological Sequence Design
B.5. Experiments different GFP benchmark
The GFP task has multiple benchmark variants, primarily due differences in oracle models (e.g., TAPE (Rao et al., 2019). Design- Benchh Transformer (Trabucco et al., 2022)) and scoring normalization strategies. Some studies normalized oracle scores (Kirjner et al., 2024; Lee et al., 2024), while others adopt raw scores (Sinai et al., 2020; Jain al., 2022; Ren et al., 2022); see the work by Surana et al. (2024) for comprehensive comparison.
To enable consistent comparison across benchmarks, we follow the experimental setup of LatProtRL (Lee et al., 2024). Specifically, we run 15 active learning rounds, evaluating 256 sequences per round, using the oracle model and initial dataset split as in the work by Kirjner al. (2024). Following LatProtRL, we report the median top- score across independent runs. Results for AdaLead and LatProtRL directly obtained from the work by Lee al. (2024).
Table 4. GFP optimization results the benchmark from Kirjner et al. (2024) and Lee et al. (2024)
GFP Medium
GFP Hard
AdaLead LatProtRL
0.93 + 0.0
0.75 0.1
0.93 + 0.0
0.85 0.0
GFN-AL (6 1)
0,21 +: 0.0
0.09 0.0
GFN-AL 8-CS (6 0.05)
0.57 + 0.0
0.60 0.1
GFN-AL + &-CS (6 0.01)
1.06 0.1 0.86 0.0
As shown in Table 9, 8-CS consistently outperforms baseline methods across all benchmark variants. In particular, significantly improves upon GFN-AL (equivalent to 1, i.e., without conservative search), achieving the best results with 8 0.01. This benchmark is intentionally designed to be more challenging compared to Beiign-Bench, as the initial datasets in GFP-medium and GFP-hard consist of low-scoring and distant sequences to the 99th percentile sequences; see Appendix A in the work by Kirjner et al. (2024). This results in greater proxy uncertainty, which in turn necessitates a lower to achieve effective exploration.
19

---

Improved Off-policy RL in Biological Sequence Design
B.6. Studies effect of adaptive 8
This section provides the experimental results verify the effectiveness of considering uncertainty on each data point for adjusting the conservativeness.
B.6.1. HARD TF-BIND-8
1.0
Mean Score
GFN-AL
6 0.75
5-22
0 Ao
6 0.5
5 0.25 Ad
0.4
0
4
Rounds
Figure Median score over rounds on Hard TF-Bind-8.
To verify its effectiveness and give intuition about how to set ō, we conduct experiments with various in Hard TF-Bind-8. The results show that 8-CS with < 1 can significantly outperform GFN-AL by searching for data points that correlate better with the oracle. This holds even when we use constant values for 8. However, as depicted in Figure 9, using adaptive ((x, σ) mostly gives the improved scores.
B.6.2. RNA SEQUENCE DESIGN
RNA-A
RNA-B
RNA-C
0.9
0.8
0.8
Median Score
Median
Median
0.6
0.6
6 0.5
6 = 0.5 0.5
6= 0.5
0.5
0.4
0.5 Ao
6= 0.5
Rounds
Rounds
Rounds
Figure 10. Effect of adaptive control RNA
We examine the effects of proxy uncertainty-based 8. In RNA, the average proxy standard deviation at the initial round is observed as 0.005 to 0.012. Therefore, we set À = 5 to roughly make λø z= 1/L, where L = 14. As illustrated in Figure 10, ((r; consistently gives the higher score. However, the constant 5 = still outperforms all baselines, exhibiting the robustness of 8-CS.
20

---

Improved Off-policy RL in Biological Sequence Design
B.7. Sensitivity analysis various const.
This section provides the experimental results with varying consst on RNA and protein designs. Note that we use dconst 0.5 in the main experiments, meaning that approximately 7 tokens masked in RNA designs, while use &const 0.05 for protein desiggs-aaproximig masking 4- tokens for GFP (L 238) and AAV (L 90).
Median Soore
Median Score
0.6
Median
0.8
0.4
0.5
0.4
0=0.5
5 0.1
0-0.1
0 0.1
0
2
10
Rounds
Rounds
Rounde
(a) RNA-A
(b) RNA-B
(c) RNA-C
Figure 11. Adaptive delta with various donnst On RNA
0.90
0.077
Median Score
0.85
5
0.025
Median
0.60
00005
8=0.05
0.55
0=0.01
0.50
Rounds
Rounds
(a) GFP
(b) AAV
Figure 12. Adaptive delta with various &onna protein designs
The results in Figure and Figure 12 show that GFN-AL with 8-CS consistently gives improved results compared to GFN-AL. Even in AAV, &canst 0.075 outperforms other baseline, including GFN-AL whose score is 0.509. Moreover, we obtain even better results by setting deuss 0.01 in GFP and & 0.025 in AAV, indicating that domain-specifer tuning of &onnst can yield further gains.
21

---

Improved Off-policy RL in Biological Sequence Design
B.8. Balancing capability with various 6
Similar Section 6.3, we evaluate 8-CS on RNA-B and RNA-C by varying dconst from 0.1 0.5. As illustrated in Figure 13, 6-CS consistently reaches the Pareto frontier. In the protein designs, vary Óconst between 0.01 and 0.05. Figure 14 demonstrates that GFN-AL with 8-CS forms the Pareto curve over both GFN-AL and EFNttr,ditor, further confirming the balancing capability of 8-CS.
Ours (6 = 0.1)
Ours (6 = 0.25)
Ours (6 0.5)
GFN-AL
GFNSoqEddlor
RNA-A
RNA-B
RNA-C
0.96
0.95
Avg Score
Avg. Score
0.92
Avg. Score
0.96
0.90
0.88
0.68
0.85
0.84
0.80
0.50
Diversity
Diversity
Diversity
0.96
Avg. Score
Avg. Score
0.92
Avg. Score
0.96
0.90
0.88
0.88
0.85
0.84
0.80
0.80
7.6
6.00
6.75
Novelty
Novelty
Novelty
Figure 13. Average score and diversity/novelty with on RNA designs with various 6.
Ours (6 0.01)
Ours (5 0.025)
Oure (5 = 0.05)
GFN-AL
GFNSeqEdltr
GFP (L 238)
AAV (L 90)
0.72
3.575
3.550
0.84
Avg.
3.525
0.56
0.48
20
30
40 Diversity
50
60
10
12
Diversity
0.72
Avg. Soore
3.550
0.84
3.525
3.500
0.48
0.0
2.5
5.0
7.5
10.0
Novelty
Novelty
Figure 14. Average and diversity/novelty protein designs with various 8.
22

---

Improved Off-policy RL in Biological Sequence Design
C. Full results of main results
C.1. Full results RNA sequence design
Max Scores
Median Scores
Mean Scores
Ours
1.00
GFN-AL
0.8
Adalead
Score
0.75
0.6
CbAS
DBAS
0.50
0.4
0.25
0.2
80
CMA-ES
0
6
10
6
10
6
10
Rounds
Rounds
Rounds
Figure 15. The max, median, and mean curve over rounds in RNA-A
Table The results RNA-A ten rounds.
Max
Median
Mean
Diversity
Novelty
AdaLead
0.968 0.070
0.808
0.049
0.817 0.048
3.518 0.446
6.888 0.426
BO
0.722 = 0.025
0.510 + 0.008
0.528 + 0.004
9.531 + 0.062
5.842 = 0.083
TuRBO
0.875 + 0.078
0.670 1 0.093
0.682 1 0.096
3.695 1 0.166
6.464 1 0.759
CMA-ES
0.816 0.030
+
0.599 + 0.020
5.
6.373 + 0.159
CbAS
0,678 = 0.020
0.467 0.009
0.481 = 0.008
9.457
5,428 + 0.078
DbAS
0.670 0.041
0.472 0.016
0.485 0.015
9.483 0.100
5.450 + 0.132
DyNA PPO
0.737 = 0.022
0.507 0.007
0.521 = 0.009
8.889 0.034
5.828 = 0.095
GFN-AL
1.030 + 0.024
0.838 1 0.013
0.849 1 0.013
6.983 1 0.159
7.398 1 0.024
GFN-AL + 5-CS
1.055 = 0.000
0.939 + 0.008
0.947 + 0.009
6.442 + 0.525
7.406 + 0.066
Max Scores
Median Scores
Mean Scores
Ours
GFN-AL
0.8
CbAS
DGAS
0.4
0.4
DyNA
CMA-ES
0.2
60
Rounds
Rounds
Rounds
Figure 16. The max, median, and mean curve over rounds in RNA-B
Table The results of RNA-B after ten rounds.
Max
Median
Mean
Diversity
Novelty
AdaLead
0.965 = 0.033
0.817 + 0.036
0.828 : 0.032
3.334 + 0.423
7.441 + 0.135
BO
0.720 + 0.032
0.502 + 0.013
0.517 + 0.014
9.495 + 0.103
5.903 + 0.116
TuRBO
0,869 = 0.086
0.705 + 0.073
0.715 + 0.073
3.638 + 0.173
6.713 = 0.560
CMA-ES
0.850 0.063
0.581 0.028
0.602 0.032
5.568 0.365
6.480 1 0.200
CbAS
0.668 = 0.021
0.465 0.005
0.077
9.234 + 0.356
5.523 + 0.083
DbAS
0.652 = 0.021
0.463 + 0.019
0.475 0.019
9.019 1 0.648
5.537 1 0.150
DyNA PPO
0.730 0.088
0.481 + 0.028
0.499 0.029
8.978 + 0.196
5.839 + 0.198
GEN-AL
1.001 = 0.016
0.858 + 0.004
0,870 + 0.006
6.599 + 0.384
7.673 + 0.043
GFN-AL 8-CS
1.014 + 0.001
0.929 + 0.004
0.934 + 0.003
5.644 4 0.307
7.661 + 0.064
23

---

Improved Off-policy RL in Biological Sequence Design
Max Scores
Median Scores
Mean Scores
1.0
Ours
AdaLead
Score
0.75
CbAS
0.6
0.6
0.50
0.4
0.4
DyNA
CMA-ES
0
6
6
10
0
10
0
6
10
TURBO
Figure 17. The max, median, and mean curve over rounds in RNA-C
Table
The results of RNA-C after rounds.
Max
Median
Mean
Diversity
Novelty
AdaL.cad
0.867 + 0.081
0.723 + 0.057
0.735 + 0.057
3.893 + 0.444
5.856 + 0.515
BO
0,694 = 0.034
0.506 + 0.003
0.519 : 0.003
9.714 + 0.054
5,430 = 0.043
TuRBO
0.773 0.222
0.621
0.221
0.632 + 0.215
4.403 + 1.542
4.906 + 1.701
CMA-ES
0.753 = 0.062
0.496
0.521 0.037
5.581 + 0.399
5.019 = 0.294
CbAS
0.696 1 0.041
0.4921
0.507 0.017
9.518 1 0.310
5.033 1 0.086
DbAS
0.678 0.025
0.495
0.508 0.011
9.249 + 0.414
5.128 + 0.153
DyNA PPO
0.728 = 0.060
0.478 0.015
0.489 0.015
9,246 0.086
5.306 1 0.124
GNE-AL
0.951 0.034
0.774 0.004
0.786
7.072 4.16
6.661 + 0.071
GEN-AL + 6-CS
1.094 = 0.045
0.972 + 0.043
0.983 - 0.043
6,493 + 1.751
6.494 + 0.084
24

---

Improved Off-policy RL in Biological Sequence Design
C.2. Full results of TF-Bind-8
Max Scores
Median Scores
Mean Scores
1.0
Ours
GFN-AL
AdaLead
0.8
0.3
CbAS
DbAS
0.6
DyNA
CMA-ES
80
0.4
Rounds
Rounds
Rounds
Figure 18. The max, median, and mean curve over rounds in TF-Bind-8
Table The results TF- [F-Bind-8 after ten rounds.
Max
Median
Mean
Diversity
Novelty
AdaLead
0.995 0.004
0.937 + 0.008
0.939 + 0.007
3,506 + 0.267
1.194 + 0.035
BO
0.977 0.008
0.806 0.007
0.815 0.005
4.824 + 0.074
1.144 + 0.029
TuRBO
0.944 + 0.039
0.719 1 0.091
0.725 1 0.097
4.276 1 0.680
1.005 1 0.182
CMA-ES
0.986 = 0.008
0.843 + 0.032
0.843 + 0.030
3.617 + 0.321
1.130 + 0.083
CbAS
0.988 0.004
0.835 + 0.011
0.845 = 0.009
4.662 0.079
1.134 + 0.021
DbAS
0.987 + 0.004
0.831 + 0.005
0.845 + 0.005
4.694 + 0.056
1.141 + 0.047
DyNA PPO
0.977 0.013
0.746 0.010
0.761 0.006
4.400
1.120 + 0.021
GFN-AL
0.976 + 0.002
0.947 1 0.004
0.947 1 0.009
3.158 1 0.166
2.409 1 0.071
GEN-AL 5-CS
0.981 = 0.002
0.971 + 0.006
0.972 + 0.005
1.277 + 0.182
2.237 + 0.356
25

---

Improved Off-policy RL in Biological Sequence Design
C.3. Full results of protein design
Max Scores
Median Scores
Mean Scores
Ours
3.55
GFN-AL
3.5
Adalnad
CbAS
DBAS
3.58
3.45
3.4
DyNA PPO
CMA-ES
3.40
3.57
4
10
2
6
10
2
10
TURBO
Rounds
Rounds
Figure 19. The max, median, and
rounds in GFP
Table 9. The results of GFP after ten rounds.
Max
Median
Mean
Diversity
Novelty
AdaLead
3.581 +: 0.004
3.549
0.002
3.552 + 0.002
47.237 +: 1.213
1.467 0.094
BO
3.572 + 0.000
3.378 + 0.000
3.331 0.000
62.955 ::t 0.000
0.000 0.000
TuRBO
3,581 = 0.006
3.466 0.049
3.438 + 0.057
51.112 = 6.802
0.584 0.379
CMA-ES
3.572 0.000
3.410 0.000
3.384 0.000
58.299 = 0.000
0.000 0.000
CbAS
3.572 = 0.000
3.378 = 0.000
3.002
62.926 = 0.139
0.009 0.012
DbAS
= 0.000
3.334
62.926 = 0.139
0.009 0.012
DyNA PPO
3.572 + 0.000
3.378
0.000
62.955 = 0.000
0.000 0.000
GEN-AL
3,578 = 0.003
3.511 0.006
3.508 = 0.004
60.278 = 0.819
20.837 0.916
GFN-AL & 8-CS
3.592 + 0.003
3.567 + 0.003
3,569 + 0.003
46.255 + 10.534
17.459 + 5.538
Max Scores
Median Scores
Mean Scores
Ours
0.70
0.65
0.65
GFN-AL
0.65
0.60
0.60
CbAS
0.60
DBAS
0.55
DyNA
0.55
CMA-ES
0.50
80
Rounds
Rounds
Rounds
Figure 20. The max, median, and mean curve over rounds in AAV
Table 10. The results of AAV after rounds.
Max
Median
Mean
Diversity
Novelty
AdaLead
0.565 0.027
0.505 + 0.016
0.509 0.017
5.693 0,946
2.133 + 1.266
BO
0.500 0.000
0.478 + 0.000
0.480 0.000
4.536 0.000
0.000 + 0.000
TuRBO
0.500 0.000
0.478 + 0.000
0.480 0.000
4.533 0.006
0.002 0.003
CMA-ES
0.500 0.000
0.481 1 0.000
0.482 0.000
4.148 1 0.000
0.000 1 0.000
CbAS
0.500 0.000
0.478 + 0.000
0.480 0.000
4.545 0.018
0.002 0.003
DbAS
0.500 + 0.000
0.478 + 0.000
0.480 + 0.000
4.545 + 0.018
0.002 + 0.003
DYNA PPO
0.500 0.000
0.478 + 0.000
0.480 0.000
4.536 0.000
0.000 0.000
GEN-AL.
0.560 0.008
0.509 + 0.002
0.513 0.002
4.044 0.303
1.966 + 0.157
GFN-AL. 8-CS
0.708 0.010
0.663 + 0.007
0.665 0.006
11.296
0.865
10.233 + 0.822
26


Reviews:
[{"review_id": "57jvyLcAu2", "reviewer": "Reviewer_6J1t", "summary": "This manuscript proposes a novel off-policy search strategy, δ-Conservative Search (δ-CS), that improves the reliability of GFlowNets for biological sequence design by controlling exploration according to proxy model confidence. The method randomly masks high-scoring offline sequences with probability δ, then relies on GFlowNets to denoise the masked tokens, keeping generated sequences closer to well-understood regions while still encouraging diversity. Experiments on DNA, RNA, and protein design tasks show that δ-CS consistently outperforms existing methods, including simpler off-policy baselines, by striking a better balance between exploration and exploitation when proxy models are uncertain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. The initial value and adjustment strategy of $\\delta$ may depend on task experience, and there is a lack of general guidelines.\n\nIn this study, we simply set delta=0.5 for DNA/RNA (masking 4 to 7 tokens on average) and delta=0.05 for protein design (masking approximately 4 to 12 tokens). Our intention is to show that even with this simple setting without task-specific tunning, $\\delta$-CS gives consistent improvements. Moreover, the results in Fig 12-14 in the appendix show that setting delta to mask approximately 1~12 tokens (i.e., 0.1 ≤ $\\delta$ ≤ 0.5 in DNA/RNA, 0.01 ≤ $\\delta$ ≤ 0.05 for protein design) is consistently beneficial compared to exploration without $\\delta$-CS.\n\nAs the reviewer mentioned, carefully setting delta based on experience and domain knowledge can bring even more improvements.\n\n---\n\n > W2. The effectiveness of adaptive $\\delta$ is highly dependent on the uncertainty estimation accuracy. If the estimation deviation is large, it may affect the performance.\n\n\nWe measure the uncertainty as the inconsistent proxy predictions using MC dropout or Ensemble. The key insight of the adaptive $\\delta$ is to search the space more conservatively when the proxy gives inconsistent predictions on a certain datapoint compared to other datapoints. In addition, thanks to the scale parameter $\\lambda$, we can adjust $\\delta$ according to each point’s *relative* uncertainty. So, even if the estimation deviation (i.e., the discrepancy between predicted uncertainty and true uncertainty) is large, it does not overly degrade performance.\n\n---\n\n> W3. While $\\delta$-CS is designed for GFlowNets, its applicability to other types of generative models or RL algorithms in biological sequence design is not extensively explored.\n\n\nThanks for the valuable feedback. As suggested, we conducted additional experiments using Soft Q-Learning (SQL) on the Hard TFBind-8 benchmark to demonstrate the broader applicability of our approach. As shown in the table below, $\\delta$-CS significantly improves both exploration and performance in SQL, confirming that our method extends beyond the GFlowNet setting.\n\n\n| | t=1 | t=5 | t=10 |\n| - | - | - | - |\n| SQL | 0.531 (0.011) | 0.879 (0.014) | 0.936 (0.008) |\n| SQL + $\\delta$-CS ($\\delta$=0.5) |0.546 (0.013) | 0.975 (0.011) | 0.993 (0.007) | \n\n\n**The rationale for the experimental focus:** $\\delta$-CS introduces a new off-policy exploration strategy under the unreliable proxy by conservatively leveraging known high-reward sequences. We initially focused on applying $\\delta$-CS within GFlowNets because they offer a natural framework for structured sequence generation in biological domains. Moreover, there is a theoretical equivalence between GFlowNet training objectives (e.g., detailed balance, trajectory balance) and established off-policy max-entropy RL methods (e.g., Soft Q-Learning, Path Consistency Learning) when reward corrections are applied (Tiapkin et al., 2024; Deleu et al., 2024). This connection supports that the insights from our GFlowNet experiments can be extended to improvements in general off-policy RL settings.\n\n\n\n> Q1. Does the oracle $f$ in the paper come from ground truth or a proxy evaluation model? If it comes from ground truth, how to evaluate the noised sequence that does not have ground truth?\n\n\nIn the case of TFBind-8, we use ground truth since all possible sequences are experimentally evaluated (Barrera et al., 2016) as the search space is relatively small ($4^8$). For other tasks, we assume a simulator or trained model with a larger dataset as our ground truth oracle, like various previous works (Sinai et al., 2020; Kirjner et al., 2023), and evaluate perturbed sequences using these oracle functions. \n\nSpecifically, we use ViennaRNA (Lorenz et al., 2011) to evaluate the newly proposed RNA sequences. For GFP, the TAPE transformer model trained with 52,000, which is much larger than our initial dataset, is used as an oracle (Rao et al., 2019). In AAV, the oracle is built using comprehensive single-mutation data from AAV2 capsids, modeled additively (summing individual mutation effects with some noise), and applied to multiple target tissues across varying design lengths (Ogden et al., 2019).\n\n- Tiapkin et al. (2024) “Generative flow networks as entropy-regularized RL.”\n- Deleu et al. (2024) “Discrete probabilistic inference as control in multi-path environments.”\n- Sinai et al. (2020) “Adalead: A simple and robust adaptive greedy search algorithm for sequence design.”\n- Kirjner et al. (2023) “Improving protein optimization with smoothed fitness landscapes.”\n- Barrera et al. (2016) “Survey of variation in human transcription factors reveals prevalent DNA binding changes.”\n- Lorenz et al. (2011) “ViennaRNA Package 2.0.”\n- Rao et al. (2019) “Evaluating protein transfer learning with TAPE.”\n- Ogden et al. (2019) “Comprehensive AAV capsid fitness landscape reveals a viral gene and enables machine-guided design.”\n\n**Thanks for your valuable comments**", "claims_and_evidence": "I have no comments on this topic.", "methods_and_evaluation_criteria": "Please see the Part of  Strengths and Weaknesses.", "theoretical_claims": "Please see the Part of  Strengths and Weaknesses.", "experimental_designs_or_analyses": "Please see the Part of  Strengths and Weaknesses.", "relation_to_broader_scientific_literature": "This work will benefit protein design and drug design.", "essential_references_not_discussed": "I have no comments on this topic.", "other_strengths_and_weaknesses": "**Strengths**\\\n1.\tThe paper is generally well-written.\n\n2.\tThe proposed method combines the diversity exploration of GFlowNets and the conservatism of evolutionary search. Through masking and adaptive delta mechanism, it effectively alleviates the problem of proxy model on out-of-distribution samples.\n\n3.\tExperimental evaluations across diverse biological sequence design tasks were performed to demonstrate both the generalizability and superior performance of the proposed method.\n\n\n**Weakness**\\\n1.\tThe initial value and adjustment strategy of δ may depend on task experience (e.g., longer sequences require smaller δ), and there is a lack of general guidelines.\n\n2.\tThe effectiveness of adaptive δ is highly dependent on the uncertainty estimation accuracy. If the estimation deviation is large, it may affect the performance.\n\n3.\tWhile δ-CS is designed for GFlowNets, its applicability to other types of generative models or RL algorithms in biological sequence design is not extensively explored.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "1.\tDoes the oracle f in the paper come from ground truth or a proxy evaluation model? If it comes from ground truth, how to evaluate the noised sequence that does not have ground truth?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UxBLqwnjPs", "reviewer": "Reviewer_JojN", "summary": "The paper proposes δ-CS, a novel off-policy RL  approach for biological sequence design. It addresses the challenge of proxy misspecification, where proxy models used for sequence evaluation are unreliable on ood inputs. The method is integrated into GFlowNets, and works by injecting and denoising noise into high-score sequences with dynamically adjusted conservativeness. Experiments show that δ-CS significantly improves GFlowNets by balancing sequence exploration and robustness, leading to improved DNA, RNA, protein, and peptide design outcomes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (Theoretical Claims) No theoretical guarantees on convergence or stability with increasing rounds, would be interesting to see some theoretical or experimental insights.\n\nThanks for highlighting the importance of theoretical guarantees and analysis. We acknowledge that rigorous theoretical guarantees, such as formal convergence rates, are indeed valuable. However, deriving such guarantees for deep-learning-based active learning is exceptionally challenging—moreover, the proxy model and the generative policy interplay to explore the black-box landscape. Our primary goal is to provide a methodological and empirical contribution. We demonstrate the effectiveness of $\\delta$-CS by presenting performance gains on DNA, RNA, and protein design tasks. These results clearly illustrate the practical benefits of our proposed approach across various domains. We will include this in our limitations of Section 7.\n\n---\n\n> W1. Does training on $D_{t-1}$ (containing more synthetic sequences) drift proxy model accuracy?\n\nIn the active learning setting, we assume that $D_{t-1}$ consists of annotated data with true oracle functions with limited accessiblity; e.g., we can query sequences with batch size of B = 128 at each round. Note that the predicted data $(x, f_{\\phi}(x))$ is added to $D_{t-1}$ during the policy training.\n\n---\n\n> W2. Any intuition on how extreme $\\delta$ values (near 0 or 1) might affect performance? No major experiments needed, just some textual discussion would suffice.\n\nWhen $\\delta$=0, no noise is injected into the offline data, so the generative policy is trained purely on the existing data without any additional exploration. In contrast, when $\\delta$=1 the original offline data is completely replaced with noise, and the model fully relies on its own policy to denoise, meaning fully on-policy learning with no conservatism. \n\n---\n> Q1. Unclear if growing synthetic data proportion (from $\\delta$-CS) leads to accuracy drift in later rounds (potential model bias).\n\nAt the start of each round, we reinitialize the generator model. During training, the model uses a proxy (acquisition function) to evaluate sequences, but we do not add them to our dataset. After training, we propose new sequences to be evaluated with oracle functions and then add those annotated results to the dataset, so the dataset always contains only real (annotated) data.\n\n---\n\n**Thanks for your valuable comments**", "claims_and_evidence": "Yes. Fig3 and table 1 supports the claim that δ-CS improves robustness by restricting policy exploration to reliable regions; Figs 9&10 demonstrates that adapting δ based on proxy uncertainty improves robustness.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical guarantees on convergence or stability with increasing rounds, would be interesting to see some theoretical or experimental insights.", "experimental_designs_or_analyses": "Yes, esp. fig 2,3,5. The proxy model failure analysis confirms low correlation between proxy and oracle for OOD sequences, and the ablation studies on δ values confirm that conservative search outperforms unconstrained GFlowNet training.", "relation_to_broader_scientific_literature": "This paper has broad connection to offline-RL, active learning, and biological sequence design.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "Strengths\n- Intuitive adaptive δ mechanism balances novelty and conservativeness.\n- Writing is clear and concise.\n- Strong experimental validation across diverse sequence design tasks.\n\nWeaknesses\n- Does training on Dₜ₋₁ (containing more synthetic sequences) drift proxy model accuracy?\n- Any intuition  on how extreme δ values (near 0 or 1) might affect performance? No major experiments needed, just some textual discussion would suffice.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above, Unclear if growing synthetic data proportion (from δ-CS) leads to accuracy drift in later rounds (potential model bias).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GY2pYT7NuN", "reviewer": "Reviewer_GtXk", "summary": "The manuscript proposes a conservative search method applied in GFlowNets for Biological Sequence Design. The proposed method restricts the number of mutations based on the length of the sequence and the prediction uncertainty. Experiments show that this sampling methodology stabilizes the training of GFlowNets, improving performance over GFlowNet-AL and other traditional baselines. These results highlight that simple methodologies for restricting the search space are effective when the functionality landscape is usually located around the initial sequence of interest.\n\n## update after rebuttal\n\nThe authors have addressed many of my concerns. I updated my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Active learning setting\n\nWe follow the existing generative active learning settings (GFN-AL [1], FLEXS [2]), which are standard in this field. As noted in our main text and appendix, we perform active learning as follows:\n\nStarting with an initial dataset $D_0$, each active learning round t consists of three steps:\n- (Step A) train the proxy model on the current dataset $D_{t-1}$;\n- (Step B) train the generative policy using the proxy and proposing a batch of B new sequences;\n- (Step C) evaluate those sequences using Oracle and add them to the dataset. \n\nWe run 10 active learning rounds with a batch size of 128. We have the same limited oracle calls for all methods. At each round, we assess performance using the top-K sequences (with K=128).\nWe adopt the initial dataset of DNA and RNA from [1] and [3], respectively. For GFP and AAV, we generated the initial dataset using wild-type, but as discussed in the following questions, we have re-conducted experiments using the dataset from [REF1, 2].\n\n### Experimental setting for GFP\n\nFirst of all, our GFP benchmark is quite standard in this field; we use TAPE [4], one of the widely adopted benchmarks in representative studies (e.g., [2], [7]). The benchmarking methods for the GFP task are diverse (e.g., TAPE [4], Design-Bench [5]; see [6] for detailed comparisons among them). Additionally, scoring normalization varies across studies, with some methods employing normalized scores [REF1, 2] and others using unnormalized scores [1, 2, 7].\n\nAs suggested, we have conducted experiments by following the experimental setting of LatProtRL [REF2]: we evaluate 256 sequences in 15 active learning rounds by adopting the oracle function with the given initial dataset that is split following [REF1]. As with [REF2], we report the median value of the top-128 sequences over 5 independent runs. The results for AdaLead and LatProtRL are directly obtained from [REF2]. Our approach achieved better performance than LatProtRL and substantially improved GFN-AL.\n\n| | GFP Medium |  GFP Hard |\n| - | - | - |\n| AdaLead | 0.93 (0.0) | 0.75 (0.1) |\n| LatProtRL | 0.93 (0.0) | 0.85 (0.0) |\n| GFN-AL (𝛿=1) | 0.21 (0.0) | 0.09 (0.0) |\n| GFN-AL + 𝛿-CS (𝛿=0.05) | 0.57 (0.0) | 0.60 (0.1) | \n| GFN-AL + 𝛿-CS (𝛿=0.01) | 1.06 (0.1) | 0.86 (0.0) | \n\nIn the revised manuscript, we will discuss the challenges in GFP with additional experimental results.\n\n### Distance to the wild-type and restricted exploration in GFP\n\nWe assume the oracle to be perfect, as with previous works. The performance of proxies typically degrades when predicting samples far from the offline data distribution, which is particularly evident in our constrained search.\n\nWe agree that incorporating distance from wild-type sequences leads to more realistic benchmarking. However, we focus on proposing a broadly applicable off-policy search method, not benchmarking itself. Importantly, our approach is compatible with distance-based constraints, as they are orthogonal to our method. To demonstrate this, we applied δ-CS to Proximal Exploration (PEX) [7], which explicitly encourages proximity to the wild type—see results below.\n\n| GFP Hard  | 𝛿-CS | 𝛿-CS + PEX |\n|-|-|-|\n| Fitness | 0.86 (0.0) | 1.08 (0.0) |\n| Distances to wt | 21.31 (1.7) | 7.47 (0.6) |\n\n### Validation with other off-policy RL algorithms\nAs suggested, we applied our method to other RL methods: soft Q learning. Please refer to the answer to reviewer 6J1t (W3).\n\n### Other baselines\nAs suggested, we compare with random mutation and LS-GFN. LS-GFN is our special case where the masking positions are fixed as the last 𝛿L tokens. \n\n| | GFP Hard |\n| - | - | \n| 𝛿-CS (𝛿=0.01) | 0.86 (0.0) | \n| Random (𝛿=0.01) | 0.27 (0.0) | \n| LS-GFN (𝛿=0.01)  | 0.34 (0.0) | \n\nThe table shows that fixing masking positions can overly restrict the search space. However, specifying more significant positions in sequences based on domain knowledge can be more efficient than randomly choosing the positions. \n\n### Related works\nAs suggested, we'll include the missing references and re-organize Sec 5.\n\n### Other comments\n> DyNa PPO in GFP \n\nIn Fig 4, the mean score of Dyna PPO remains unchanged over active rounds, which means it fails to generate sequences with higher scores than the initial dataset. Note that GFP scores are not normalized. \n\n> \"Large-scale\"\n\nWe use \"large-scale\" to denote longer sequences, especially with large action space, like protein design tasks. \n\n> line 212\n\nWe'll revise it.\n\n\n[1] Jain et al. \"Biological sequence design with GFlowNets.\" ICML (2022)\n\n[2] Sinai et al. \"Adalead\" (2020)\n\n[3] Kim et al. \"Bootstrapped training of score-conditioned generator for offline design of biological sequences.\" NIPS (2023)\n\n[4] Rao et al. \"Evaluating protein transfer learning with TAPE.\" NIPS (2019)\n\n[5] Trabucco et al. \"Design-bench\" ICML, 2022.\n\n[6] Surana et al. \"Overconfident oracles.\" (2025).\n\n[7] Ren et al. \"Proximal exploration for model-guided protein sequence design.\" ICML, 2022.", "claims_and_evidence": "The claims made by the manuscript are supported by clear evidence.", "methods_and_evaluation_criteria": "The evaluation criteria make sense for the problem investigated.", "theoretical_claims": "The mathematical of the proposed method seems correct.", "experimental_designs_or_analyses": "The reviewer has clarification questions regarding the active learning setting and the oracles used for evaluation. Specifically,\n\n1. How was the active learning setting set for the proposed method and the baseline methods?\n2. How is the oracle performing for sequences that are far in terms of mutations from the wild type? The values for the GFP performance graph seem high even for methods that have problems generating long sequences like DynaPPO.", "relation_to_broader_scientific_literature": "The key contributions of the paper related to the broader scientific literature are:\n\n1. Proposing a conservative search approach that can be combined to solve sampling issues in the training of GFlowNets. \n2. It applies an adaptive conservativeness approach that is dependent on the oracle prediction uncertainty.", "essential_references_not_discussed": "Additional references related to the application of RL to Protein Engineering like [REF1] and [REF2] seem to be missing.\n\n[REF1] Wang, Yi, et al. \"Self-play reinforcement learning guides protein engineering.\" Nature Machine Intelligence 5.8 (2023): 845-860.\n[REF2] Lee, Minji, et al. \"Robust optimization in protein fitness landscapes using reinforcement learning in latent space.\" ICML (2024).", "other_strengths_and_weaknesses": "Strengths:\n1. The manuscript adapts GFlowNets by proposing a conservative search framework, i.e., sampling a limited number of mutations from initial state sequences, to address the issue of generating out-of-distribution samples when handling large-length sequences.\n2. The idea of using restricted exploration to improve the training efficiency of GFlowNets is an interesting direction.\n3. The proposed adaptive conservativeness approach is adjusted for each data point based on prediction uncertainty.\n\nWeaknesses:\n1. The main concern of the reviewer is the reliability of the results presented in the manuscript. The oracle and active learning settings are not well defined. Additionally, there are some concerns regarding dataset splits used for some of the tasks presented.\n2. As mentioned by the authors as one of the main limitations, the restricted exploration strategy does not solve different drawbacks from active learning. Restricting the number of mutations is likely to improve the results for landscapes in which most of the functional sequences are close to the main wild-type sequence.\n3. Even though the title suggests a general method analysis for different RL algorithms, the experiments are only extending a GFlowNet as the main policy.", "other_comments_or_suggestions": "1. (Related Work) GFlowNets are introduced on Page 5 after there are other follow-up works to GFlowNets introduced in Page 4. The Related Work section should be re-organized.\n2. The sentence in line 212: “capitalizing on both the high novelty offered by GFlowNets” is counterintuitive as your method proposes restricting the novelty of the GFlowNet generation given its tendency to generate out-of-distribution samples.", "questions_for_authors": "1. What is the meaning of “large-scale” settings in line 36? From my understanding, it means that GFlowNets achieve poor performance for long sequences? The authors should clarify this definition.\n2. (Title) The title suggests that the proposed conservative search can be applied to other RL-based algorithms but only test to GFlowNet. For other algorithms, the proposed policy is only applied for a setting with discrete actions and with a sampling order that gives enough context to propose mutations to the masked positions? If this is the case, the reviewer wonders about the advantage or improvement in performance over sampling mutations using a masked protein language model, for example.\n3. The methodology for generating the datasets seems critical. How were the initial sequences for GFP generated? With random mutations from the wild-type? Given the functional landscape of GFP this might lead to data leakage. A split like the one proposed by [REF1] might be needed.\n[REF1] Kirjner, Andrew, et al. \"Improving protein optimization with smoothed fitness landscapes.\" ICLR (2023).\n4. All baselines achieve very high performance on the GFP dataset. Given that sequences with more than 10 mutations are very unlikely to be functional, the reviewer suggests showing the number of mutations from the wild-type for each of the methods. A metric showing the distance from the wild-type for the top-scoring mutants might be needed, as shown in [REF2]. Mutants far from the wild type with high predicted functionality might suggest a problem with the oracle's reliability.\n[REF2] Lee, Minji, et al. \"Robust optimization in protein fitness landscapes using reinforcement learning in latent space.\" ICML (2024).\n5.  It is not clear how the active learning setting is defined for the proposed method and baselines. Additionally, it is also unclear if the oracle is used only for evaluation for all the methods.\n6. Given that the positions to mutate are pre-sampled, it would be interesting to have baselines mutating these same positions randomly or using a masked protein language model to compare the performance. \n7. The manuscript needs a better explanation and intuition of how fixing positions changes the training stability of GFlowNets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWT7JqzR0j", "reviewer": "Reviewer_GMEp", "summary": "This work focuses on the problem of biological sequence design, recognizing that trained proxy scoring models often produce unreliable predictions. To address this challenge, they introduce restrictions into GFlowNet exploration by implementing a masking mechanism that increases the probability of exploring reliable data points. Specifically, adding masks to certain tokens in the offline dataset allows for exploration at those positions, while unmasked positions directly utilize offline data through teacher forcing. Through this approach, the authors' proposed δ-CS method supports more reliable exploration, achieving excellent results across multiple datasets (DNA, RNA, peptide design) and attaining Pareto optimality by balancing diversity and fitness.\n\n## update after rebuttal\nThank the authors for their rebuttal, which addressed many of my concerns, especially by adding experiments on incorporating $\\delta$-CS results with additional off-policy methods. As a result, I have increased my score from 2 to 3. However, for a primarily empirical work like this, I still believe that the current depth of investigation into the effects of $\\delta$-CS is insufficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">W1, Q2: The term \"mask\" is confusing\n\nBecause our infilling sites are uniformly distributed across the entire sequence, we refer to them as “masks,” similar to BERT. However, we recognize that this term can be confusing. Alternatively, we could describe the algorithm as random teacher forcing: some tokens are predicted while others remain identical to the original sequence. This approach also removes the need for $\\tilde{x}$ in the equation, since it is not directly used as input by the neural network.\n\n> W2: 𝛿CS with other algorithms beyond GFlowNet\n\n𝛿-CS can be applied to any off-policy RL method; we chose GFlowNets because they are representative method in this domain. To demonstrate $\\delta$-CS as a plug-and-play component, we applied 𝛿-CS to Soft Q-Learning (SQL), which is a representative off-policy RL method in the general domain:\n\n\n| Hard TFBind8| t=1 | t=5 | t=10 |\n| - | - | - | - |\n| SQL | 0.531 (0.011) | 0.879 (0.014) | 0.936 (0.008) |\n| SQL + 𝛿-CS (𝛿=0.5) |0.546 (0.013) | 0.975 (0.011) | 0.993 (0.007) | \n\n---\n\n> W3: Conservative proxy\n\nWe acknowledge conservative model-based optimization (COMs) methods, which aim to construct robust proxy models [1, 2]. However, mere conservativeness is insufficient for active learning scenarios, where active querying for information gain from the oracle is essential. Although COMs perform well in offline model-based optimization (MBO), our results show they underperform in active learning settings. Specifically, COMs that impose conservativeness through adversarial smoothness in proxy training (Trabucco et al., 2021) achieve poorer outcomes than alternative methods.\n\nOur method shares the philosophy that conservatism is beneficial but differs significantly in implementation. Instead of embedding conservativeness directly into proxy training, we integrate it into the search process itself. This approach preserves high information gain through explicit uncertainty modeling while ensuring conservatively robust optimization, effectively balancing uncertainty-driven exploration with robustness. We will put this discussion at the main paper. Thanks.\n\nTrabucco et al. \"Conservative objective models for effective offline model-based optimization.\" ICML, 2021.\n\n---\n>W4: Validation in complex biological sequence task\n\nWe have already included longer sequence benchmarks (such as GFN and AAV), which are widely recognized and commonly used in the literature. While TFB tasks are indeed simpler and considered toy benchmarks, they remain popular for initial evaluations. We agree that incorporating additional real-world DNA benchmarks could further strengthen our evaluation.\n\nRegarding the suggested benchmarks, we noticed that the majority of existing literature [1,2,4] focuses on diffusion-based algorithms, which fall outside the scope of our research (therefore we focused to benchamrk [3]). Although we aimed to replicate results from TACO [3], which employs autoregressive models, their dataset and pre-trained checkpoints are unfortunately not publicly available. Consequently, extending our method to include these benchmarks within the rebuttal period is challenging. However, we commit to evaluating our method on longer DNA benchmarks by the camera-ready deadline, and we anticipate similar performance trends to those observed in GFP and AAV tasks.\n\n---\n\n>W5: $\\delta$-CS vs. conservative reward\n\nAs discussed in W3, our approach still can utilize acquisition function like UCB while introducing conservatism to avoiding overoptimization. Though we have different approach, we can utilize the uncertainty oracle as proposed in [1] since we do not any restriction for the way of measuring uncertainty; this will be an interesting future work.\n\nWe'll include this discussion into our revised manuscript\n\n---\n> Q1. Do we actually know the distribution $P_{D_{t-1}}$? \n\nWe define a discrete distribution by normalizing the weights in line 132. Specifically, $P_{D_{t-1}}(x) \\propto w(x; D_{t-1}, k),$ where $k$ is a hyperparameter controlling the distribution’s peakiness. \n\n---\n> Q3. Couldn't DyNA PPO directly leverage offline data through teacher-forcing methods?\n\nWe agree that teacher-forcing offers a way to incorporate offline data into DyNA-PPO. However, because PPO is inherently on-policy, directly leveraging offline data remains challenging without methods like importance sampling, which can introduce high variance. While teacher-forcing also can be used to pretrain the policy pretraining from offline sequences, it may lead to overfitting to the offline distribution since the reward information is not used during pretraining. To further verify this point, we conducted DyNA PPO experiments where we first pretrain the policy via teacher-forcing on offline data and then fine-tune it with PPO on TFBind8:\n\n||Top-128 Mean|\n|--|--|\n|Dyna PPO|0.761 ± 0.006|\n|Dyna PPO + teacher - forcing pretraining|0.777 ± 0.001 |\n|GFN-AL | 0.947 ± 0.009 |\n|GFN-AL + ours | 0.972 ± 0.005 |", "claims_and_evidence": "See Other Strengths And Weaknesses and Questions For Authors.", "methods_and_evaluation_criteria": "See Other Strengths And Weaknesses and Questions For Authors.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See Other Strengths And Weaknesses and Questions For Authors.", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses and Questions For Authors.", "essential_references_not_discussed": "See Other Strengths And Weaknesses and Questions For Authors.", "other_strengths_and_weaknesses": "## Strengths:\n1. The paper is well-written, provides a detailed literature review, and clearly explains both the motivation and methods.\n2. Experimental details are provided thoroughly, and source code is also provided, ensuring high reproducibility.\n3. The motivation makes sense, as considering the unreliability of proxies in biological sequence optimization is indeed an important problem.\n\n## Weaknesses:\n1. While the masking approach proposed here is conceptually novel, I find the term \"mask\" potentially misleading. In sequence modeling machine learning, mask usage typically refers to approaches like BERT, where certain tokens are masked within a sequence while content on both sides remains visible, enabling bidirectional modeling. After reading the abstract, I initially expected the work to employ this type of modeling. Upon further reading, it appears that the \"mask\" here merely serves as a \"selective exploration\" training method. Specifically, in GPT's next token prediction, training traditionally requires teacher forcing for every token through trajectories from dataset $D$. The masking here simply selects certain tokens that are allowed free exploration, while maintaining an autoregressive training approach at its core. This usage of \"mask\" may confuse readers. I would like the authors to clarify their rationale for using the \"mask\" concept within an autoregressive training framework. Without sufficient justification, I recommend the authors consider alternative terminology.\n\n2. The rationale for using GFlowNet remains unclear. As mentioned in Weakness 1, the essence of the method is conducting selective exploration on offline data to discover new trajectories, but this approach doesn't seem strongly coupled with GFlowNets. I believe the authors should include other methods capable of learning from offline data in their baselines, and then demonstrate the superiority of $\\delta$-CS as a plug-and-play component, proving its advantages over alternative approaches.\n\n3. The authors' motivation is premised on proxy models producing severe unreliable predictions (Figure 2), leading them to reduce exploration of unreliable data points. However, I note that the proxy used is very simple (a CNN with no additional regularization). The authors should further discuss whether more advanced proxy training approaches could mitigate this issue (which represents an alternative solution to unreliability, as in [2]).\n\n4. The DNA design experiments (TFBS binding sites <10bp) are overly simplistic (as mentioned on line 261). The authors should consider more complex experiments involving Cis-regulatory Element design as demonstrated in recent works ([1][2][3][4]).\n\n5. [1] shares a similar approach with this work, using prediction uncertainty to adjust the optimization process. The difference is that [1] incorporates uncertainty into the reward function, while the authors use it to restrict exploration probability from offline data. I would like the authors to discuss the differences between these two approaches. Additionally, [1] employs different uncertainty estimation methods, and I would like the authors to discuss how their uncertainty estimation approach differs from that in [1].\n\n## References\n\n[1] Uehara, Masatoshi, et al. \"Bridging model-based optimization and generative modeling via conservative fine-tuning of diffusion models.\" Advances in Neural Information Processing Systems 37 (2024): 127511-127535.\n\n[2] Reddy, Aniketh Janardhan, et al. \"Designing Cell-Type-Specific Promoter Sequences Using Conservative Model-Based Optimization.\" Advances in Neural Information Processing Systems 37 (2024): 93033-93059.\n\n[3] Yang, Zhao, et al. \"Regulatory DNA Sequence Design with Reinforcement Learning.\" The Thirteenth International Conference on Learning Representations.\n\n[4] Wang, Chenyu, et al. \"Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design.\" The Thirteenth International Conference on Learning Representations.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses and Questions For Authors.", "questions_for_authors": "1. Do we actually know the distribution $P_{D_{t-1}}$? I understand the authors may intend to construct a discrete distribution using the weight in line 132, but this requires a more rigorous definition of the distribution. \n\n2. What is the significance of introducing $\\tilde{x}$ on line 160? This notation is not used in the formula, which only requires $\\tilde{e}_t$. This aligns with my concern in Weakness 1, and actually the modeling here doesn't require a complete masked $\\tilde{x}$, so is using \"mask\" as terminology for the method appropriate?\n\n3. Line 22 mentions that the biggest drawback of DyNA PPO is its inability to utilize offline data, but I believe this might be solvable. Couldn't DyNA PPO directly leverage offline data through teacher-forcing methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript proposes a conservative search strategy for GFlowNets in the context of biological sequence design. The approach constrains the number of mutations during exploration based on sequence length and model uncertainty, effectively focusing the search around the initial sequence. Experimental results demonstrate that this strategy improves the stability of GFlowNet training and leads to better performance compared to GFlowNet-AL and traditional baselines. The results suggest that simple, principled restrictions on the search space can be highly beneficial, especially in domains where functionality tends to concentrate near a reference sequence.

The reviewers agree that the paper is clearly written, with well-motivated methodology and thorough experimental setup. The authors provided thoughtful clarifications and new experiments during the rebuttal, which addressed most concerns raised in the initial reviews.

I recommend acceptance and encourage the authors to incorporate the additional experimental results and clarify the rationale behind their experimental choices in the final version.
Decision: Accept (poster)