Improved Off-policy Reinforcement Learning in Biological Sequence Design

## Abstract

Abstract Designing biological sequences with desired properties is a significant challenge due to the combinatorially vast search space and the high cost of evaluating each candidate sequence.
To address these challenges, reinforcement learning (RL) methods, such as GFlowNets, utilize proxy models for rapid reward evaluation and annotated data for policy training. Although these approaches have shown promise in generating diverse and novel sequences, the limited training data relative to the vast search space often leads to the misspecification of proxy for out-of-distribution inputs.
We introduce δ 𝛿 \delta italic_δ -Conservative Search, a novel off-policy search method for training GFlowNets designed to improve robustness against proxy misspecification. The key idea is to incorporate conservativeness, controlled by parameter δ 𝛿 \delta italic_δ , to constrain the search to reliable regions. Specifically, we inject noise into high-score offline sequences by randomly masking tokens with a Bernoulli distribution of parameter δ 𝛿 \delta italic_δ and then denoise masked tokens using the GFlowNet policy. Additionally, δ 𝛿 \delta italic_δ is adaptively adjusted based on the uncertainty of the proxy model for each data point. This enables the reflection of proxy uncertainty to determine the level of conservativeness. Experimental results demonstrate that our method consistently outperforms existing machine learning methods in discovering high-score sequences across diverse tasks—including DNA, RNA, protein, and peptide design—especially in large-scale scenarios. The code is available at https://github.com/hyeonahkimm/delta_cs .

## 1 Introduction

Designing biological sequences with desired properties is crucial in therapeutics and biotechnology (Zimmer, ; Lorenz et al., ; Barrera et al., ; Sample et al., ; Ogden et al., ). However, this task is challenging due to the combinatorially large search space and the expensive and black-box nature of objective functions. Recent advances in deep learning methods for biological sequence design have shown significant promise at overcoming these challenges (Brookes & Listgarten, ; Brookes et al., ; Angermueller et al., ; ).

Among various approaches, reinforcement learning (RL), which leverages a proxy model as a reward function, has emerged as one of the successful paradigms for automatic biological sequence design (Angermueller et al., ). RL methods have the benefit of exploring diverse sequence spaces by generating sequences token-by-token from scratch, enabling the discovery of novel sequences. They employ deep neural networks as inexpensive proxy models to approximate costly oracle objective functions. The proxy model serves as a reward function for training deep RL algorithms, enabling the policy network to generate high-reward biological sequences. The model can be trained in an active learning manner (Gal et al., ), iteratively annotating new data by querying the oracle with points generated using the policy; these iterations are called query rounds. There are two approaches for training the policy in this context: on-policy and off-policy.

DyNA PPO (Angermueller et al., ), a representative on-policy RL method for biological sequence design, employs Proximal Policy Optimization (PPO; Schulman et al., ) within a proxy-based active training loop. While DyNA PPO has demonstrated effectiveness in various biological sequence design tasks, its major limitation is the limited search flexibility inherent to on-policy methods. It cannot effectively leverage off-policy data points, such as data collected from previous rounds.

Conversely, Generative Flow Networks (GFlowNets; Bengio et al., ), off-policy RL methods akin to maximum entropy policies (Tiapkin et al., ; Deleu et al., ), offer diversity-seeking capabilities and flexible exploration strategies. applied GFlowNets to biological sequence design with additional Bayesian active learning schemes. They leveraged the off-policy nature of GFlowNets by mixing offline datasets with on-policy data during training. This approach provided more stable training compared to DyNA PPO and resulted in better performance.

However, recent studies have consistently reported that GFlowNets perform poorly on long-sequence tasks such as green fluorescent protein (GFP) design (Kim et al., ; Surana et al., ). We hypothesize that this poor performance stems from the insufficient quality of the proxy model in the early rounds. For example, in the typical benchmark, the proxy is trained with 5,000 sequences, while GFPs have a combinatorial search space of $20^{238}>10^{309}$. GFlowNets are capable of generating sequences from scratch and producing novel sequences beyond the data points. However, when the quality of the proxy model is unreliable, novel sequences that are out-of-distribution to the proxy model yield unreliable results (Trabucco et al., ; Yu et al., ).
This motivates us to introduce a conservative search strategy to restrict the trajectories to the neighborhood of the existing data point.

Contribution. In this paper, we propose a novel off-policy search method called $\delta$-Conservative Search ($\delta$-CS), which enables a trade-off between sequence novelty and robustness to proxy misspecification by using a conservativeness parameter $\delta$. Specifically, we iteratively train a GFlowNet using $\delta$-CS as follows: (1) we inject noise by independently masking tokens in high-score offline sequences with Bernoulli distribution with parameter $\delta$; (2) the GFlowNet policy sequentially denoises the masked tokens; (3) we use these denoised sequences to train the policy. When $\delta=1$ (full on-policy), this becomes equivalent to sampling full trajectories, and when $\delta=0$ (fully conservative), this reduces to only showing offline sequences in off-policy training. We adaptively adjust $\delta(x;\sigma)$ using the proxy model’s uncertainty estimates $\sigma(x)$ for each data point $x$. This approach allows the level of conservativeness to be adaptively adjusted based on the prediction uncertainty of each data point. Figure 1 illustrates the overall procedure of the $\delta$-CS algorithm.

Our extensive experiments demonstrate that $\delta$-CS significantly improves GFlowNets, successfully discovering higher-score sequences compared to existing model-based optimization methods on diverse tasks, including DNA, RNA, protein, and peptide design. This result offers a robust and scalable framework for advancing research and applications in biotechnology and synthetic biology.

Figure: Figure 1: The active learning process for biological sequence design with $\delta$-Conservative Search ($\delta$-CS). Starting with high reward sequences from the offline dataset, we inject token-level noise with probability $\delta$, which determines the conservativeness of the search. Then, the GFlowNet policy denoises the masked sequences. Lastly, the GFlowNet policy is trained with new sequences. After policy training, we query a new batch of sequences and update the dataset for the next round.
Refer to caption: x1.png

## 2 Problem Formulation

We aim to discover sequences $x\in\mathcal{V}^{L}$ that exhibit desired properties, where $\mathcal{V}$ denotes the vocabulary, such as amino acids or nucleotides, and $L$ represents the sequence length, which is usually fixed. The desired properties are evaluated by a black-box oracle function $f:\mathcal{V}^{L}\to\mathbb{R}$, which evaluates the desired property of a given sequence, such as binding affinity or enzymatic activity. Evaluating $f$ is often both expensive and time-consuming since it typically involves wet-lab experiments or high-fidelity simulations.

Advancements in experimental techniques have enabled the parallel synthesis and evaluation of sequences in batches. Therefore, lab-in-the-loop processes are emerging as practical settings that enable active learning. Following this paradigm, we perform $T$ rounds of batch optimization, where in each round, we have the opportunity to query $B$ batched sequences to the (assumed) oracle objective function $f$. Due to the labor-intensive nature of these experiments, $T$ is typically very small. Following Angermueller et al. () and , we assume the availability of an initial offline dataset $\mathcal{D}_{0}=\{(x^{(n)},y^{(n)})\}_{n=1}^{N_{0}}$, where $y=f(x)$. The initial number of data points $N_{0}$ is typically many orders of magnitude smaller than the size of the search space, as mentioned in the introduction. The goal is to discover, after $T$ rounds, a set of sequences that are novel, diverse, and have high oracle function values.

## 3 Active Learning for Biological Sequence Design

Following , we formulate an active learning process constrained by a budget of $T$ rounds with query size $B$, is executed through an iterative procedure consisting of three stages with a novel component of $\delta$-Conservative Search ($\delta$-CS) which will be detailed described in Section 4:

Step A (Proxy Training): We train a proxy model $f_{\phi}(x)$ using the offline dataset $\mathcal{D}_{t-1}$ at round $t$.

Step B (Policy Training with $\delta$-CS): We train a generative policy $p(x;\theta)$ using the proxy model $f_{\phi}(x)$ and the dataset $\mathcal{D}_{t-1}$ with $\delta$-CS.

Step C (Offline Dataset Augmentation with $\delta$-CS): We apply $\delta$-CS to query batched data $\{x_{i}\}_{i=1}^{B}$ to the oracle $y_{i}=f(x_{i})$. Then the offline dataset is augmented as: $\mathcal{D}_{t}\leftarrow\mathcal{D}_{t-1}\cup\{(x_{i},y_{i})\}_{i=1}^{B}$.

The overall algorithm is described in Algorithm 1. In the following subsections, we describe the details of Step A and Step B.

### 3.1 Step A : Proxy Training

Following , we train the proxy model $f_{\phi}$ using the dataset $\mathcal{D}_{t-1}$ by minimizing the mean squared error loss:

$$ $\mathcal{L}(\phi)=\mathbb{E}_{x\sim P_{\mathcal{D}_{t-1}}(x)}\left[\left(f(x)- f_{\phi}(x)\right)^{2}\right],$ (1) $$

where $\mathcal{D}_{t}$ is the dataset at active round $t$, augmented with oracle queries. In the initial round ($t=1$), we use the given initial dataset $\mathcal{D}_{0}$. See Section A.1 for detailed implementation.

### 3.2 Step B : Policy training with δ 𝛿 \delta italic_δ -CS

For policy training, we employ GFlowNets, which aim to produce samples from a generative policy where the probability of generating a sequence $x$ is proportional to its reward, i.e.,

$$ $p(x;\theta)\propto R(x;\phi)=f_{\phi}(x)+\kappa\sigma(x).$ (2) $$

Following , the reward $R(x;\phi)$ is defined as $f_{\phi}(x)+\kappa\sigma(x)$, which combines the proxy value $f_{\phi}(x)$ and the uncertainty $\sigma(x)$ in the form of the upper confidence bound (UCB; Srinivas et al., ) acquisition function. This approach prioritizes regions with higher uncertainty, enabling us to query them in the next active round. Here, $\kappa$ is a mixing hyperparameter.

Policy parameterization. The forward policy $P_{F}$ generates state transitions sequentially through trajectories $\tau=(s_{0}\rightarrow\ldots\rightarrow s_{L}=x)$, where $s_{0}=()$ represents the empty sequence, and each state transition involves adding a sequence token. The full sequence $s_{L}=x$ is obtained after $L$ steps, where $L$ is the length of the sequences.
The forward policy $P_{F}(\tau;\theta)$ is a compositional policy defined as:

$$ $P_{F}(\tau;\theta)=\prod_{i=1}^{L}P_{F}(s_{i}|s_{i-1};\theta).$ (3) $$

GFlowNets have a backward policy $P_{B}(\tau|x)$ that models the probability of backtracking from the terminal state $x$.
The sequence $x=(e_{1},\ldots,e_{L})$ can be uniquely converted into a state transition trajectory $\tau$, where each intermediate state represents a subsequence.
In the case of sequences, there is only a single way to backtrack, so $P_{B}(\tau|x)=1$. This makes these types of GFlowNets equivalent to soft off-policy RL algorithms. For example, the trajectory balance (TB) objective of GFlowNets (Malkin et al., ) becomes equivalent to path consistency learning (PCL) (Nachum et al., ), an entropy-maximizing value-based RL method according to Deleu et al. ().

Learning objective and training trajectories. The policy is trained to minimize TB loss: $\mathcal{L}_{\text{TB}}(\tau;\theta)$:

$$ $\mathcal{L}_{\text{TB}}(\tau;\theta)=\left(\log\frac{Z_{\theta}P_{F}(\tau; \theta)}{R(x;\phi)}\right)^{2}$ (4) $$

Usually, GFlowNets training is employed to minimize TB loss with training trajectories $\tau$ on full supports, asymptotically guaranteeing optimality for the distribution:

$$ $p(x;\theta)\propto R(x;\phi).$ $$

A key challenge in prior works is that the proxy model $f_{\phi}(x)$ often produces highly unreliable rewards $R(x;\phi)$ for out-of-distribution inputs. In our approach, we mitigate this by providing off-policy trajectories within more reliable regions by injecting conservativeness into off-policy search. Therefore, we minimize TB loss with $\delta$-CS, which offers controllable conservativeness.

## 4 δ 𝛿 \delta italic_δ -CS: controllable conservativeness in off-policy search

This section introduces $\delta$-Conservative Search ($\delta$-CS), an off-policy search method that enables controllable exploration through a conservative parameter $\delta$. The parameter $\delta$ defines the Bernoulli distribution governing the masking of tokens in a sequence. Our algorithm is iterations of the following:

- $\bullet$
Sample high-score offline sequences $x\sim P_{\mathcal{D}_{t-1}}(x)$ from the rank-based reweighted prior.
- $\bullet$
Inject noise by masking tokens into $x$ using the noise injection policy $P_{\text{noise}}(\tilde{x}\mid x,\delta)$.
- $\bullet$
Denoise the masked tokens using the denoising policy $P_{\text{denoise}}(\hat{x}\mid\tilde{x};\theta)$.

These trajectories are used to update the GFlowNet parameters $\theta$ by minimizing the loss function $\mathcal{L}_{\text{TB}}(\tau;\theta)$. For more details on the algorithmic components of $\delta$-CS and its integration with active learning GFlowNets, see Algorithm 1.

Rank-based reweighted prior. First, we sample a reference sequence $x$ from the prior distribution $P_{\mathcal{D}_{t-1}}$. To exploit high-scoring sequences, we employ rank-based prioritization (Tripp et al., ).

$$ $w(x;\mathcal{D}_{t-1},k)\propto\frac{1}{kN+\text{rank}_{f,\mathcal{D}_{t-1}}(x )}.$ $$

Here, $\text{rank}_{f,\mathcal{D}_{t-1}}(x)$ is a relative rank of the value of $f(x)$ in the dataset $\mathcal{D}_{t-1}$ with a weight-shifting factor $k$; we fix $k=0.01$.
This assigns greater weight to sequences with higher ranks. Note that this reweighted prior can also be used during proxy training.

Noise injection policy. Let $x=(e_{1},e_{2},\ldots,e_{L})$ denote the original sequence of length $L$. We define a noise injection policy where each position $i\in\{1,2,\ldots,L\}$ is independently masked according to a Bernoulli distribution with parameter $\delta\in[0,1]$, resulting in the masked sequence $\tilde{x}=(\tilde{e}_{1},\tilde{e}_{2},\ldots,\tilde{e}_{L})$.
The noise injection policy $P_{\text{noise}}(\tilde{x}\mid x,\delta)$ is defined as:

$$ $P_{\text{noise}}(\tilde{x}\mid x,\delta)=\prod_{i=1}^{L}\left[\delta\cdot \mathbb{I}\{\tilde{e}_{i}=\text{[MASK]}\}+(1-\delta)\cdot\mathbb{I}\{\tilde{e} _{i}=e_{i}\}\right],$ $$

where $\mathbb{I}\{\cdot\}$ is the indicator function.

Denoising policy.
We employ the GFlowNet forward policy $P_{F}$ to sequentially reconstruct the masked sequence $\tilde{x}=(\tilde{e}_{1},\tilde{e}_{2},\ldots,\tilde{e}_{L})$ by predicting tokens from left to right.
The probability of denoising next token $\tilde{e}_{t}$ from previously denoised subsquence $\hat{s}_{t-1}$ is:

$$ $P_{\text{denoise}}(\hat{e}_{t}\mid\hat{s}_{t-1},\tilde{x};\theta)=\begin{cases }\mathbb{I}\{\hat{e}_{t}=\tilde{e}_{t}\},&\text{if }\tilde{e}_{t}\neq\text{[ MASK]},\\ P_{F}(\hat{s}_{t}=(\hat{s}_{t-1},\hat{e}_{t})\mid\hat{s}_{t-1};\theta),&\text{ if }\tilde{e}_{t}=\text{[MASK]}.\end{cases}$ $$

The fully reconstructed sequence $\hat{x}=\hat{s}_{L}$ is obtained by sampling from:

$$ $P_{\text{denoise}}(\hat{x}\mid\tilde{x};\theta)=\prod_{t=1}^{L}P_{\text{ denoise}}(\hat{e}_{t}\mid\hat{s}_{t-1},\tilde{x};\theta).$ $$

By denoising the masked tokens with the GFlowNet policy, which infers each token sequentially from left to right, we generate new sequences $\hat{x}$ that balance novelty and conservativeness through the parameter $\delta$.

Figure: Algorithm 1 Active Learning GFlowNets with $\delta$-CS

### 4.1 Adjusting conservativeness parameter δ 𝛿 \delta italic_δ

Determining the conservative parameter $\delta$ is a crucial aspect of the algorithm. We propose and study two variants, constant and adaptive $\delta$.

#### Constant.

As a simple approach, we set $\delta$ as a constant, selecting it to have the noise policy mask 4–15 tokens per sequence. Despite its simplicity, this choice effectively enhances policy training and leads to the discovery of high-scoring sequences during active rounds; we provide further studies on $\delta$-CS with a constant $\delta$ in Appendix B.

#### Adaptive.

Another intuitive approach is to adjust $\delta$ based on the uncertainty of the proxy $\sigma$ on each sequence $x$, that is $\delta(x;\sigma)$. Specifically, we define a function that assigns lower $\delta$ values for highly uncertain samples and vice versa: $\delta(x;\sigma)=\delta_{\text{const}}-\lambda\sigma(x)$. We estimate $\sigma(x)$, the standard deviation of the proxy model $f_{\phi}(x)$, via MC dropout (Gal & Ghahramani, ) or an ensemble method (Lakshminarayanan et al., ). $\lambda$ is a scaling factor and related to the influence of the proxy uncertainty on $\delta$; we set it to satisfy $\lambda\sigma\approx\frac{1}{L}$ based on the observations from the initial round.

In our main experiments, we use adaptive $\delta(x,\sigma)$ as the default setup.

## 5 Related work

Designing biological sequences using machine learning methods is widely studied. Bayesian optimization methods (Mockus, ; Belanger et al., ; Zhang et al., ) exploit posterior inference over newly acquired data points to update a Bayesian proxy model that can measure useful uncertainty. However, these methods usually suffer from scalability issues due to the complexity of the Gaussian process (GP) kernel (Belanger et al., ) or the difficulty of sampling from an intractable posterior (Zhang et al., ).

Offline model-based optimization (MBO) (Kumar & Levine, ; Trabucco et al., ; Yu et al., ; Chen et al., ; Kim et al., ; Chen et al., ; Yun et al., ) also addresses the design of biological sequences using offline datasets only, which can be highly efficient because they do not require oracle queries. These approaches have reported meaningful findings, such as the conservative requirements on proxy models since proxy models tend to give high rewards on unseen samples (Trabucco et al., ; Yu et al., ; Yuan et al., ; Chen et al., ). This supports our approach of adaptive conservatism in the search process. Surana et al. () recently noted that offline design and existing benchmarks are insufficient to reflect biological reliability, indicating that settings without additional Oracle queries might be too idealistic.

Reinforcement learning methods, such as DyNA PPO (Angermueller et al., ) and GFlowNets (Bengio et al., ; ; Jain et al., ; Hernández-García et al., ), and sampling with generative models (Brookes & Listgarten, ; Brookes et al., ; Das et al., ; Song & Li, ) aim to search the biological sequence space using a sequential decision-making process with a policy, starting from scratch. Similarly, sampling with generative models (Brookes & Listgarten, ; Brookes et al., ; Song & Li, ) searches the sequence space using generative models like VAE (Kingma & Welling, ).
While these approaches allow for the creation of novel sequences, as sequences are generated from scratch, they are relatively prone to incomplete proxy models, particularly in regions where the proxy is misclassified due to being out-of-distribution.

An alternative line of research is evolutionary search (Arnold, ; Bloom & Arnold, ; Schreiber et al., ; Sinai et al., ; Ren et al., ; Ghari et al., ; Kirjner et al., ), a popular method in biological sequence design. Especially Ghari et al. () proposed GFNSeqEditor, which utilizes GFlowNets as prior distribution to edit biological sequences as an evolutionary search. Evolutionary search methods iteratively edit given sequences and constrain the new sequences so as not to deviate too far from the seed sequence; they usually start from the wild-type, which occurs in nature.
This can be viewed as constrained optimization, where out-of-distribution for the proxy model can lead to unrealistic and low-score biological sequences. Consequently, they do not aim to produce highly novel sequences. Our method can be seen as a hybrid of off-policy RL and evolutionary search, capitalizing on both the high novelty offered by GFlowNets and the high rewards with out-of-distribution robustness provided by constrained search where they are properly balanced by $\delta$. Our experimental comparison with GFNSeqEditor (Ghari et al., ) demonstrates this hybridization balanced by $\delta$ enables us to discover sequences with greater novelty and higher rewards rather than merely using the GFlowNet policy as editing priors.

## 6 Experiments

Following the FLEXS benchmark (Sinai et al., ),(^1^11FLEXS (Fitness Landscape EXploration Sandbox) is a widely-used open-source simulation environment for biological sequence design, which is available at [https://github.com/samsinai/FLEXS](https://github.com/samsinai/FLEXS).) we evaluate our proposed method on various biological sequence design tasks.
Furthermore, we analyze the effect of $\delta$-CS by directly comparing with GFN-AL on TF-Bind-8 and an anti-microbial peptide design in Section 6.6. For each experiment, we conduct five independent runs.

Implementation details.
For proxy models, we employ a convolutional neural network (CNN) with one-dimensional convolutions (Sinai et al., ) with a UCB acquisition function and an ensemble of three network instances to measure the uncertainty.
Note that we use the same architecture to implement proxy models for all baselines. For the GFlowNet policy, we use a simple two-layer long short-term memory (LSTM) network (Hochreiter & Schmidhuber, ) and train the policy with 1,000 inner-loop updates using a learning rate of $5\times 10^{-4}$ with a batch size of 256. However, in TF-Bind-8 and AMP, where we analyze the effectiveness of $\delta$-CS compared to GFN-AL, we directly implement $\delta$-CS on top of the GFN-AL implementation. Lastly, we set $\delta$ and $\lambda$ according to the description in Section 4; specifically, $\delta=0.5$ for tasks with $L\leq 50$ and $\delta=0.05$ for long-sequences. More details are provided in Section A.2.

Baselines. As our baseline methods, we employ representative exploration algorithms. Further details are provided in Section A.3.

- •
AdaLead (Sinai et al., ) is a well-implemented model-guided evaluation method with a hill-climbing algorithm.
- •
Bayesian optimization (BO; Snoek et al., ) is a classical algorithm for black-box optimization. We employ the BO algorithm with a sparse sampling of the mutation space implemented by Sinai et al. ().
- •
CMA-ES (Hansen, ) is another well-known evolutionary algorithm that optimizes a continuous relaxation of one-hot vectors encoding sequence with the covariance matrix.
- •
CbAS (Brookes et al., ) and DbAS (Brookes & Listgarten, ) are probabilistic frameworks that use model-based adaptive sampling with a variational autoencoder (VAE; Kingma & Welling, ). Notably, CbAS restricts the search space with a trust-region search similar to the proposed method.
- •
DyNA PPO (Angermueller et al., ) uses proximal policy optimization (PPO; Schulman et al., ), an on-policy training method.
- •
GFN-AL () is our main baseline that uses GFN with Bayesian active learning.

**Table 1: Maximum rewards achieved by each baseline method across six tasks, with the sequence length ($L$) for each task specified. The mean and standard deviation from five runs are reported. The highest values for each task are highlighted in bold.**
|  | RNA-A<br>($L=14$) | RNA-B<br>($L=14$) | RNA-C<br>($L=14$) | TF-Bind-8<br>($L=8$) | GFP<br>($L=238$) | AAV<br>($L=90$) |
| --- | --- | --- | --- | --- | --- | --- |
| AdaLead | 0.968 $\pm$ 0.070 | 0.965 $\pm$ 0.033 | 0.723 $\pm$ 0.057 | 0.995 $\pm$ 0.004 | 3.581 $\pm$ 0.004 | 0.565 $\pm$ 0.027 |
| BO | 0.722 $\pm$ 0.025 | 0.720 $\pm$ 0.032 | 0.506 $\pm$ 0.003 | 0.977 $\pm$ 0.008 | 3.572 $\pm$ 0.000 | 0.500 $\pm$ 0.000 |
| CMA-ES | 0.816 $\pm$ 0.030 | 0.850 $\pm$ 0.063 | 0.496 $\pm$ 0.041 | 0.986 $\pm$ 0.008 | 3.572 $\pm$ 0.000 | 0.500 $\pm$ 0.000 |
| CbAS | 0.678 $\pm$ 0.020 | 0.668 $\pm$ 0.021 | 0.492 $\pm$ 0.018 | 0.988 $\pm$ 0.004 | 3.572 $\pm$ 0.000 | 0.500 $\pm$ 0.000 |
| DbAS | 0.670 $\pm$ 0.041 | 0.652 $\pm$ 0.021 | 0.495 $\pm$ 0.010 | 0.987 $\pm$ 0.004 | 3.572 $\pm$ 0.000 | 0.500 $\pm$ 0.000 |
| DyNA PPO | 0.737 $\pm$ 0.022 | 0.730 $\pm$ 0.088 | 0.478 $\pm$ 0.015 | 0.977 $\pm$ 0.013 | 3.572 $\pm$ 0.000 | 0.500 $\pm$ 0.000 |
| GFN-AL | 1.030 $\pm$ 0.024 | 1.001 $\pm$ 0.016 | 0.774 $\pm$ 0.004 | 0.976 $\pm$ 0.002 | 3.578 $\pm$ 0.003 | 0.560 $\pm$ 0.008 |
| GFN-AL + $\delta$-CS | 1.055 $\pm$ 0.000 | 1.014 $\pm$ 0.001 | 0.972 $\pm$ 0.043 | 0.981 $\pm$ 0.002 | 3.592 $\pm$ 0.003 | 0.708 $\pm$ 0.010 |

Figure: Figure 2: Median scores of Top-128 over active rounds. Ours (GFN-AL + $\delta$-CS) consistently outperforms baseline in RNA, DNA (TF-Bind-8), and protein (GFP and AAV) design tasks.
Refer to caption: x2.png

For each task, we conduct 10 active learning rounds starting from the initial dataset $\mathcal{D}_{0}$. The query batch size is all set as 128 except for the AMP design, whose query size is 1,000. Further details of each task are provided in the following subsections. To evaluate the performance, we measure the maximum, median, and mean scores of Top-$K$ sequences.

### 6.1 RNA sequence design

Task setup.
The goal is to design an RNA sequence that binds to the target with the lowest binding energy, which is measured by ViennaRNA (Lorenz et al., ). The length ($L$) of RNA is set to 14, with 4 tokens. In this paper, we have three RNA binding tasks, RNA-A, RNA-B, and RNA-C, whose initial datasets consist of 5,000 randomly generated sequences with certain thresholds; we adopt the offline dataset provided in Kim et al. (). We use $\delta=0.5$ and $\lambda=5$, according to the guidelines in Section 4.

Results.
As shown in Figure 2 and Table 1, our method outperforms all baseline approaches. The curve in Figure 2 increases significantly faster than the other methods, indicating that $\delta$-CS effectively trains the policy and generates appropriate queries in each active round. More results are provided in Section C.1.

### 6.2 DNA sequence design

Task setup. In this task, we aim to generate diverse and novel DNA sequences that maximize the binding affinity to the target transcription factor. The length $(L)$ of the sequence is fixed with 8. The initial dataset $\mathcal{D}_{0}$ is the bottom 50% in terms of the reward, which results in $32,898$ samples, with the maximum score of 0.439. Though this has been widely used in many studies (Sinai et al., ; ; Trabucco et al., ; Kim et al., ), the TF-Bind-8 is easy to optimize, especially due to its size (Sinai et al., ). Similar to RNA, we use $\delta=0.5$ and $\lambda=5$.

Results. As shown in Table 1, AdaLead achieves the highest maximum performance, while $\delta$-CS still outperforms GFN-AL. We believe that AdaLead’s greedy evolutionary search capability is powerful, especially in the small search space of TF-bind-8. However, in Figure 2, $\delta$-CS demonstrates the best median performance compared to the other baselines; see the mean, diversity, and novelty in Section C.2.

### 6.3 Protein sequence design

We consider two protein sequence design tasks: the green fluorescent protein (GFP; Sarkisyan et al., ) and additive adeno-associated virus (AAV; Ogden et al., ).

GFP. The objective is to identify protein sequences with high log-fluorescence intensity values.(^2^22The score is evaluated by ML oracle models. FLEXS uses TAPE (Rao et al., ) for evaluation, while Design Bench Transformer (Trabucco et al., ) is employed in GFN-AL.) The vocabulary is defined as 20 standard amino acids, i.e., $|\mathcal{V}|=20$, and the sequence length $L$ is 238; thus, we set $\delta$ as 0.05 and $\lambda$ as 0.1, according to our guideline. The initial datasets are generated by randomly mutating the provided wild-type sequence for each task while filtering out sequences that have higher scores than the wild-type; we obtain the initial dataset with $|\mathcal{D}_{0}|=10\;200$ with a maximum score value of 3.572.

AAV. The aim is to discover sequences that lead to higher gene therapeutic efficiency. The sequences are composed of the 20 standard amino acids with a length of 90, resulting in the search space of $20^{90}$. In the same way as in GFP, we collect an initial dataset of 15,307 sequences with a maximum score of 0.500. We use $\delta=0.05$ and $\lambda=1$.

Results.
Table 1 shows the results of all methods in protein sequence design tasks. Given the combinatorially vast design space with sequence lengths $L=238$ and $90$, most baselines fail to discover new sequences whose score is higher than the maximum of the dataset.
In contrast, as depicted in Figure 2, our method finds high-score sequences beyond the dataset, even with a single active round. This underscores the superiority of our search strategy in practical biological sequence design tasks. Full results are provided in Section C.3.

### 6.4 Anti-microbial peptide design

**Table 2: Results on AMP with different acquisition functions (UCB, EI). The mean and standard deviation from five runs are reported. Improved results with $\delta$-CS over GFN-AL are marked in bold.**
|  | Max | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- |
| COMs | 0.930 $\pm$ 0.001 | 0.920 $\pm$ 0.000 | 0.000 $\pm$ 0.000 | 11.869 $\pm$ 0.298 |
| DyNA PPO | 0.953 $\pm$ 0.005 | 0.941 $\pm$ 0.012 | 15.186 $\pm$ 5.109 | 16.556 $\pm$ 3.653 |
| GFN-AL (UCB) | 0.936 $\pm$ 0.004 | 0.919 $\pm$ 0.005 | 28.504 $\pm$ 2.691 | 19.220 $\pm$ 1.369 |
| GFN-AL + $\delta$-CS (UCB) | 0.948 $\pm$ 0.015 | 0.938 $\pm$ 0.016 | 25.379 $\pm$ 3.735 | 23.551 $\pm$ 1.290 |
| GFN-AL (EI) | 0.950 $\pm$ 0.002 | 0.940 $\pm$ 0.003 | 15.576 $\pm$ 7.896 | 21.810 $\pm$ 4.165 |
| GFN-AL + $\delta$-CS (EI) | 0.962 $\pm$ 0.003 | 0.958 $\pm$ 0.004 | 16.631 $\pm$ 2.135 | 24.946 $\pm$ 4.246 |

Task setup.
The goal is to generate protein sequences with anti-microbial properties (AMP). The vocabulary size $|\mathcal{V}|=20$, and the sequence length ($L$) varies across sequences, and we consider sequences of length 50 or lower. For the AMP task, we consider a much larger query batch size for each active round because they can be efficiently synthesized and evaluated ().
We set $\delta$ as 0.5 with $\lambda=1$.

Figure: Figure 3: Average score and diversity/novelty with five independent runs. Our method (GFN-AL + $\delta$-CS) consistently approaches Pareto frontier performance. We set $\delta=0.5$ for short sequences ($L\leq 50$) and set $\delta=0.05$ for long length sequences ($L>50$).
Refer to caption: x3.png

Figure: (a) Proxy vs. true scores at $t=1$
Refer to caption: x4.png

Results. The results in Table 2 illustrate that ours consistently gives improved performance over GFN-AL regardless of acquisition function.
According to the work from , we also adopt conservative model-based optimization method, (COMs; Trabucco et al., ) and on-policy reinforcement learning, DyNA PPO (Angermueller et al., ) as baselines. Our method demonstrated significantly higher performance in terms of mean, diversity, and novelty compared to the baselines.

### 6.5 Achieving Pareto frontier with balancing capability of δ 𝛿 \delta italic_δ -CS

In this analysis, we demonstrate that $\delta$-CS achieves a balanced search using $\delta$, producing Pareto frontiers or comparable results to the baseline methods: GFN-AL () and GFNSeqEditor (Ghari et al., ). Notably, GFN-AL can be seen as a variant of our method with $\delta=1$, which fully utilizes the entire trajectory search. This approach is expected to yield high novelty and diversity, but it is also prone to generating high rewards due to the increased risk of out-of-distribution samples affecting the proxy model. GFNSeqEditor, on the other hand, leverages GFlowNets as a prior, editing from a wild-type sequence. It is designed to deliver reliable performance and be more robust to out-of-distribution issues by constraining the search to sequences similar to the wild type. However, unlike $\delta$-CS, GFNSeqEditor does not utilize such obtained samples for training GFlowNets in full trajectory level; GFNSeqEditor is expected to have lower diversity and novelty compared to GFN-AL and $\delta$-CS.

As shown in Fig. 3, GFN-AL generally produces higher diversity and novelty in the RNA and GFP tasks compared to GFNSeqEditor. However, GFNSeqEditor performs better in terms of reward on the large-scale GFP task, whereas GFN-AL struggles due to the lack of a constrained search procedure in such a large combinatorial space. In contrast, $\delta$-CS achieves Pareto-optimal performance compared to both methods, clearly outperforming GFNSeqEditor across six tasks, with higher rewards, diversity, and novelty. For the RNA and GFP tasks, we achieve higher scores than GFN-AL while maintaining similar novelty but slightly lower diversity. In the AAV task, $\delta$-CS shows a distinct Pareto improvement. These results demonstrate that $\delta$-CS provides a beneficial balance by combining conservative search with amortized inference on full trajectories using off-policy GFlowNets training, effectively capturing the strengths of both GFN-AL and GFNSeqEditor.

### 6.6 Study on proxy failure and conservativeness effect

Task: Hard TF-Bind-8. By modifying the initial dataset distribution and the landscape, we can make a harder version of TF-Bind-8. Specifically, we collect the initial dataset near a certain sequence (considered as a wild-type) while ensuring that the initial sequences have lower scores than the given sequence, which is 0.431. The size of $\mathcal{D}_{0}$ is 1,024. Furthermore, we modify the landscape to give 0 rewards for sequences with scores lower than 0.3. These features are often observed in protein design tasks, where the search space is extremely large, e.g., $20^{238}$ previously–with a limited real-world dataset, and the score often falls to 0.

Proxy failure. As shown in Figure 2, $\delta$-CS gives slightly better performance than GFN-AL, but only marginally. This is because the TF-Bind-8 task is relatively easy to optimize, leading to similar results across methods. To more clearly assess the effectiveness of $\delta$-CS, we conduct several studies on the harder TF-Bind-8 task, which is more difficult to optimize. Figure 4(a) illustrates the proxy values and true scores for all $x\in\mathcal{X}$ in the first round. While the proxy provides accurate predictions for the initial data points $x\in\mathcal{D}_{0}$ (represented by the red dots), it produces unreliable predictions for points outside $\mathcal{D}_{0}$. This supports our hypothesis that the proxy model performs poorly on out-of-distribution data.

Effect of $\delta$ conservativeness. Figure 4(b) illustrates that the correlation between the oracle $f$ and the proxy $f_{\phi}$ significantly decreases as data points move farther from the observed dataset. This strongly motivates the use of $\delta$-CS, which constrains the search bounds using $\delta$. By limiting the search to within these constrained edit distances, $\delta$-CS enhances the correlation with the oracle.

Studies on $\delta$. We study on the choice of $\delta$ and effectiveness of adaptive $\delta(x,\sigma)$, in Section B.1.

## 7 Conclusions

In this paper, we introduced a novel off-policy sampling method for GFlowNets, called $\delta$-CS, which provides controllable conservativeness through the use of a $\delta$ parameter. Additionally, we proposed an adaptive conservativeness approach by adjusting $\delta$ for each data point based on prediction uncertainty. We demonstrated the effectiveness of $\delta$-CS in active learning GFlowNets, achieving strong performance across various biological sequence design tasks, including DNA, RNA, protein, and peptide design, consistently outperforming existing baselines.

## Appendix A Implementation detail

### A.1 Proxy training

For training proxy models, we follow the procedure of (). We use Adam (Kingma, ) optimizer with learning rate $1\times 10^{-5}$ and batch size of 256. The maximum proxy update is set as 3000. To prevent over-fitting, we use early stopping using the $10\%$ of the dataset as a validation set and terminate the training procedure if validation loss does not improve for five consecutive iterations.

### A.2 Policy training

As described in Section 6, we employ a two-layer long short-term memory (LSTM; Hochreiter & Schmidhuber, ) with 512 hidden dimensions. The policy is trained with a learning rate of $5\times 10^{-4}$ with a batch size of 256. The learning rate of $Z$ is set as $10^{-3}$.
The coefficient $\kappa$ in Equation 2 is set as 0.1 for TF-Bind-8 and AMP with MC dropout, according to , and 1.0 for RNA and protein design with Ensemble following Ren et al. ().

### A.3 Implementation details of baselines

We adopt open-source code from FLEXS benchmark (Sinai et al., ).

- •
AdaLead (Sinai et al., ): We use a default settings of hyperparmeters for AdaLead. Specifically, we use a recombination rate of 0.2, mutation rate of $1/L$, where $L$ is sequence length, and threshold $\tau=0.05$.
- •
DbAS (Brookes & Listgarten, ): We implement DbAS with variational autoencoder (VAE; Kingma & Welling, ) as the generator. The input is a one-hot encoding vector, and the output latent dimension is 2. In each cycle, DbAS starts by training the VAE with the top 20% sequences in terms of the score.
- •
CbAS (Brookes et al., ): Similar to DbAS, we implement CbAS with VAE. The main difference from DbAS is that we select top 20% sequences with the weights $p(\bm{x}|\bm{z},\theta^{(0)})/q(\bm{x}|\bm{z},\phi^{(t)})$, where $p(\cdot;\theta^{(0)})$ is trained with the ground-truth samples and $q(\cdot;\phi^{(t)})$ is trained on the generated sequences over $t$ training rounds.
- •
DyNA PPO (Angermueller et al., ): We closely follow the algorithm presented in (Angermueller et al., ). For a fair comparison, we use CNN ensembles to parameterize the proxy model instead of suggested architectures.
- •
CMA-ES (Hansen, ): We implement a covariance matrix adaptation evolution strategy (CMA-ES) for sequence generation. As the generated samples from CMA-ES are continuous, we convert the continuous vectors into one-hot representation by computing the argmax at each sequence position.
- •
BO (Snoek et al., ): We use classical GP-BO algorithm for all tasks. For Gaussian Process Regressor (GPR), we use a default setting from the sklearn library. For the acquisition function, they use Thompson sampling (Russo et al., ).

Furthermore, we employ GFN-AL and GFNSeqEditor. We adopt the original implementation and setup for TF-Bind-8 and AMP. For newly added tasks, we report better results among the original MLP policy and the LSTM policy. Note that GFP in FLEXS is different from the one employed in GFA-AL; we treat this as a new task based on the observation in the work from Surana et al. ().

- •
GFN-AL (): We strictly follow hyperparameters of the original code in they conduct experiments on TF-Bind-8 and AMP. The proxy is parameterized using an MLP with two layers of 2,048 hidden. For the policy, a 2-layer MLP with 2,048 hidden dimensions is used, but we also test it with a 2-layer LSTM policy.
- •
GFNSeqEditor (Ghari et al., ): We implemented the editing procedure on top of the GFN-AL. Note that GFNSeqEditor does not utilize the proxy model, so the GFlowNets policy is trained using offline data only with the same policy training procedure of GFN-AL. GFNSeqEditor can also implicitly control the edit percentage with its hyperparameters, which are set $\delta=0.01,\sigma=0.0001,\lambda=0.1$ in this study. Note that $\delta$ is not the conservativeness parameter.

## Appendix B Further studies

### B.1 Studies on effect of δ 𝛿 \delta italic_δ on Hard TF-Bind-8

Figure: (a) Median scores with various fixed $\delta$
Refer to caption: x6.png

To verify its effectiveness and give intuition about how to set $\delta$, we conduct experiments with various $\delta$ in Hard TF-Bind-8. The results show that $\delta$-CS with $\delta<1$ can significantly outperform GFN-AL by searching for data points that correlate better with the oracle. In the Hard TF-Bind-8 task, a more conservative search with $\delta=0.25$ is beneficial since the proxy is unreliable in the early rounds.
In particular, ours with $\delta=1$ means the full on-policy search (no conservativenss). The performance differences between $\delta=1$ and GFN-AL came from ${\epsilon}$-noisy behavior policy, which selects random actions with a probability of 0.001, in GFN-AL.
Furthermore, using adaptive $\delta(x,\sigma)$ mostly gives the improved scores as depicted in Figure 5(b).

We further examine the effect of adaptive $\delta$ adjustment on RNA in the following section.

### B.2 Studies on the effect of adaptive δ 𝛿 \delta italic_δ on RNA-A

Figure: Figure 6: Effect of adaptive control on RNA
Refer to caption: x8.png

We examine the effects of proxy uncertainty-based $\delta$. In RNA, the average proxy standard deviation $\bar{\sigma}$ at the initial round is observed as 0.005 to 0.012. Therefore, we set $\lambda=5$ to roughly make $\lambda\bar{\sigma}\approx 1/L$, where $L=14$. As illustrated in Figure 6, $\delta(x;\sigma)$ consistently gives the higher score. However, the constant $\delta=5$ still outperforms all baselines, exhibiting the robustness of $\delta$-CS.

### B.3 Balancing capability of δ 𝛿 \delta italic_δ on RNA

Similar to Section 6.5, we also verify the balancing capability of $\delta$-CS on RNA-B and RNA-C. The $\delta$ is set as 0.5.

Figure: Figure 7: Average score and diversity/novelty with five independent runs on RNA design tasks.
Refer to caption: x11.png

### B.4 Studies on rank-based reweighted sampling in proxy training

The rank-based reweighing also can be used in proxy training, i.e., $x\sim P_{\mathcal{D}_{t-1}}(x;k)$, where $k$ is a reweighting factor and fixed as 0.01 in this work. The results show that rank-based reweighted proxy training improves performance mostly. However, the gap is small, and $\delta$-CS still works well even without reweighting.

**Table 3: Ablation studies of rank-based reweighted proxy training**
|  |  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- | --- |
| RNA-A | with rank-based | 1.055 $\pm$ 0.000 | 0.939 $\pm$ 0.008 | 0.947 $\pm$ 0.009 | 6.442 $\pm$ 0.525 | 7.406 $\pm$ 0.066 |
| without rank-based | 1.049 $\pm$ 0.010 | 0.936 $\pm$ 0.016 | 0.944 $\pm$ 0.015 | 5.782 $\pm$ 0.697 | 7.397 $\pm$ 0.098 |  |
| RNA-B | with rank-based | 1.014 $\pm$ 0.001 | 0.929 $\pm$ 0.004 | 0.934 $\pm$ 0.003 | 5.644 $\pm$ 0.307 | 7.661 $\pm$ 0.064 |
| without rank-based | 1.009 $\pm$ 0.008 | 0.932 $\pm$ 0.012 | 0.938 $\pm$ 0.012 | 6.252 $\pm$ 0.291 | 7.673 $\pm$ 0.033 |  |
| RNA-C | with rank-based | 1.094 $\pm$ 0.045 | 0.972 $\pm$ 0.043 | 0.983 $\pm$ 0.043 | 6.493 $\pm$ 1.751 | 6.494 $\pm$ 0.084 |
| without rank-based | 1.097 $\pm$ 0.022 | 0.958 $\pm$ 0.029 | 0.965 $\pm$ 0.031 | 5.472 $\pm$ 1.921 | 6.464 $\pm$ 0.192 |  |

## Appendix C Full results of main results

### C.1 Full results of RNA sequence design

Figure: Figure 8: The max, median, and mean curve over rounds in RNA-A
Refer to caption: x12.png

**Table 4: The results of RNA-A after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 0.968 $\pm$ 0.070 | 0.808 $\pm$ 0.049 | 0.817 $\pm$ 0.048 | 3.518 $\pm$ 0.446 | 6.888 $\pm$ 0.426 |
| BO | 0.722 $\pm$ 0.025 | 0.510 $\pm$ 0.008 | 0.528 $\pm$ 0.004 | 9.531 $\pm$ 0.062 | 5.842 $\pm$ 0.083 |
| CMA-ES | 0.816 $\pm$ 0.030 | 0.585 $\pm$ 0.016 | 0.599 $\pm$ 0.020 | 5.747 $\pm$ 0.110 | 6.373 $\pm$ 0.159 |
| CbAS | 0.678 $\pm$ 0.020 | 0.467 $\pm$ 0.009 | 0.481 $\pm$ 0.008 | 9.457 $\pm$ 0.189 | 5.428 $\pm$ 0.078 |
| DbAS | 0.670 $\pm$ 0.041 | 0.472 $\pm$ 0.016 | 0.485 $\pm$ 0.015 | 9.483 $\pm$ 0.100 | 5.450 $\pm$ 0.132 |
| DyNA PPO | 0.737 $\pm$ 0.022 | 0.507 $\pm$ 0.007 | 0.521 $\pm$ 0.009 | 8.889 $\pm$ 0.034 | 5.828 $\pm$ 0.095 |
| GFN-AL | 1.030 $\pm$ 0.024 | 0.838 $\pm$ 0.013 | 0.849 $\pm$ 0.013 | 6.983 $\pm$ 0.159 | 7.398 $\pm$ 0.024 |
| GFN-AL + $\delta$-CS | 1.055 $\pm$ 0.000 | 0.939 $\pm$ 0.008 | 0.947 $\pm$ 0.009 | 6.442 $\pm$ 0.525 | 7.406 $\pm$ 0.066 |

Figure: Figure 9: The max, median, and mean curve over rounds in RNA-B
Refer to caption: x13.png

**Table 5: The results of RNA-B after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 0.965 $\pm$ 0.033 | 0.817 $\pm$ 0.036 | 0.828 $\pm$ 0.032 | 3.334 $\pm$ 0.423 | 7.441 $\pm$ 0.135 |
| BO | 0.720 $\pm$ 0.032 | 0.502 $\pm$ 0.013 | 0.517 $\pm$ 0.014 | 9.495 $\pm$ 0.103 | 5.903 $\pm$ 0.116 |
| CMA-ES | 0.850 $\pm$ 0.063 | 0.581 $\pm$ 0.028 | 0.602 $\pm$ 0.032 | 5.568 $\pm$ 0.365 | 6.480 $\pm$ 0.200 |
| CbAS | 0.668 $\pm$ 0.021 | 0.465 $\pm$ 0.005 | 0.477 $\pm$ 0.004 | 9.234 $\pm$ 0.356 | 5.523 $\pm$ 0.083 |
| DbAS | 0.652 $\pm$ 0.021 | 0.463 $\pm$ 0.019 | 0.475 $\pm$ 0.019 | 9.019 $\pm$ 0.648 | 5.537 $\pm$ 0.150 |
| DyNA PPO | 0.730 $\pm$ 0.088 | 0.481 $\pm$ 0.028 | 0.499 $\pm$ 0.029 | 8.978 $\pm$ 0.196 | 5.839 $\pm$ 0.198 |
| GFN-AL | 1.001 $\pm$ 0.016 | 0.858 $\pm$ 0.004 | 0.870 $\pm$ 0.006 | 6.599 $\pm$ 0.384 | 7.673 $\pm$ 0.043 |
| GFN-AL + $\delta$-CS | 1.014 $\pm$ 0.001 | 0.929 $\pm$ 0.004 | 0.934 $\pm$ 0.003 | 5.644 $\pm$ 0.307 | 7.661 $\pm$ 0.064 |

Figure: Figure 10: The max, median, and mean curve over rounds in RNA-C
Refer to caption: x14.png

**Table 6: The results of RNA-C after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 0.867 $\pm$ 0.081 | 0.723 $\pm$ 0.057 | 0.735 $\pm$ 0.057 | 3.893 $\pm$ 0.444 | 5.856 $\pm$ 0.515 |
| BO | 0.694 $\pm$ 0.034 | 0.506 $\pm$ 0.003 | 0.519 $\pm$ 0.003 | 9.714 $\pm$ 0.054 | 5.430 $\pm$ 0.043 |
| CMA-ES | 0.753 $\pm$ 0.062 | 0.496 $\pm$ 0.041 | 0.521 $\pm$ 0.037 | 5.581 $\pm$ 0.399 | 5.019 $\pm$ 0.294 |
| CbAS | 0.696 $\pm$ 0.041 | 0.492 $\pm$ 0.018 | 0.507 $\pm$ 0.017 | 9.518 $\pm$ 0.310 | 5.033 $\pm$ 0.086 |
| DbAS | 0.678 $\pm$ 0.025 | 0.495 $\pm$ 0.010 | 0.508 $\pm$ 0.011 | 9.249 $\pm$ 0.414 | 5.128 $\pm$ 0.153 |
| DyNA PPO | 0.728 $\pm$ 0.060 | 0.478 $\pm$ 0.015 | 0.489 $\pm$ 0.015 | 9.246 $\pm$ 0.086 | 5.306 $\pm$ 0.124 |
| GNF-AL | 0.951 $\pm$ 0.034 | 0.774 $\pm$ 0.004 | 0.786 $\pm$ 0.004 | 7.072 $\pm$ 0.163 | 6.661 $\pm$ 0.071 |
| GFN-AL + $\delta$-CS | 1.094 $\pm$ 0.045 | 0.972 $\pm$ 0.043 | 0.983 $\pm$ 0.043 | 6.493 $\pm$ 1.751 | 6.494 $\pm$ 0.084 |

### C.2 Full results of TF-Bind-8

Figure: Figure 11: The max, median, and mean curve over rounds in TF-Bind-8
Refer to caption: x15.png

**Table 7: The results of TF-Bind-8 after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 0.995 $\pm$ 0.004 | 0.937 $\pm$ 0.008 | 0.939 $\pm$ 0.007 | 3.506 $\pm$ 0.267 | 1.194 $\pm$ 0.035 |
| BO | 0.977 $\pm$ 0.008 | 0.806 $\pm$ 0.007 | 0.815 $\pm$ 0.005 | 4.824 $\pm$ 0.074 | 1.144 $\pm$ 0.029 |
| CMA-ES | 0.986 $\pm$ 0.008 | 0.843 $\pm$ 0.032 | 0.843 $\pm$ 0.030 | 3.617 $\pm$ 0.321 | 1.130 $\pm$ 0.083 |
| CbAS | 0.988 $\pm$ 0.004 | 0.835 $\pm$ 0.011 | 0.845 $\pm$ 0.009 | 4.662 $\pm$ 0.079 | 1.134 $\pm$ 0.021 |
| DbAS | 0.987 $\pm$ 0.004 | 0.831 $\pm$ 0.005 | 0.845 $\pm$ 0.005 | 4.694 $\pm$ 0.056 | 1.141 $\pm$ 0.047 |
| DyNA PPO | 0.977 $\pm$ 0.013 | 0.746 $\pm$ 0.010 | 0.761 $\pm$ 0.006 | 4.430 $\pm$ 0.030 | 1.120 $\pm$ 0.021 |
| GFN-AL | 0.976 $\pm$ 0.002 | 0.947 $\pm$ 0.004 | 0.947 $\pm$ 0.009 | 3.158 $\pm$ 0.166 | 2.409 $\pm$ 0.071 |
| GFN-AL + $\delta$-CS | 0.981 $\pm$ 0.002 | 0.971 $\pm$ 0.006 | 0.972 $\pm$ 0.005 | 1.277 $\pm$ 0.182 | 2.237 $\pm$ 0.356 |

### C.3 Full results of Protein design

Figure: Figure 12: The max, median, and mean curve over rounds in GFP
Refer to caption: x16.png

**Table 8: The results of GFP after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 3.581 $\pm$ 0.004 | 3.549 $\pm$ 0.002 | 3.552 $\pm$ 0.002 | 47.237 $\pm$ 1.213 | 01.467 $\pm$ 0.094 |
| BO | 3.572 $\pm$ 0.000 | 3.378 $\pm$ 0.000 | 3.331 $\pm$ 0.000 | 62.955 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| CMA-ES | 3.572 $\pm$ 0.000 | 3.410 $\pm$ 0.000 | 3.384 $\pm$ 0.000 | 58.299 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| CbAS | 3.572 $\pm$ 0.000 | 3.378 $\pm$ 0.000 | 3.334 $\pm$ 0.002 | 62.926 $\pm$ 0.139 | 00.009 $\pm$ 0.012 |
| DbAS | 3.572 $\pm$ 0.000 | 3.378 $\pm$ 0.000 | 3.334 $\pm$ 0.002 | 62.926 $\pm$ 0.139 | 00.009 $\pm$ 0.012 |
| DyNA PPO | 3.572 $\pm$ 0.000 | 3.378 $\pm$ 0.000 | 3.331 $\pm$ 0.000 | 62.955 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| GFN-AL | 3.578 $\pm$ 0.003 | 3.511 $\pm$ 0.006 | 3.508 $\pm$ 0.004 | 60.278 $\pm$ 0.819 | 20.837 $\pm$ 0.916 |
| GFN-AL + $\delta$-CS | 3.592 $\pm$ 0.003 | 3.567 $\pm$ 0.003 | 3.569 $\pm$ 0.003 | 46.255 $\pm$ 10.534 | 17.459 $\pm$ 5.538 |

Figure: Figure 13: The max, median, and mean curve over rounds in AAV
Refer to caption: x17.png

**Table 9: The results of AAV after ten rounds.**
|  | Max | Median | Mean | Diversity | Novelty |
| --- | --- | --- | --- | --- | --- |
| AdaLead | 0.565 $\pm$ 0.027 | 0.505 $\pm$ 0.016 | 0.509 $\pm$ 0.017 | 05.693 $\pm$ 0.946 | 02.133 $\pm$ 1.266 |
| BO | 0.500 $\pm$ 0.000 | 0.478 $\pm$ 0.000 | 0.480 $\pm$ 0.000 | 04.536 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| CMA-ES | 0.500 $\pm$ 0.000 | 0.481 $\pm$ 0.000 | 0.482 $\pm$ 0.000 | 04.148 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| CbAS | 0.500 $\pm$ 0.000 | 0.478 $\pm$ 0.000 | 0.480 $\pm$ 0.000 | 04.545 $\pm$ 0.018 | 00.002 $\pm$ 0.003 |
| DbAS | 0.500 $\pm$ 0.000 | 0.478 $\pm$ 0.000 | 0.480 $\pm$ 0.000 | 04.545 $\pm$ 0.018 | 00.002 $\pm$ 0.003 |
| DyNA PPO | 0.500 $\pm$ 0.000 | 0.478 $\pm$ 0.000 | 0.480 $\pm$ 0.000 | 04.536 $\pm$ 0.000 | 00.000 $\pm$ 0.000 |
| GFN-AL | 0.560 $\pm$ 0.008 | 0.509 $\pm$ 0.002 | 0.513 $\pm$ 0.002 | 04.044 $\pm$ 0.303 | 01.966 $\pm$ 0.157 |
| GFN-AL + $\delta$-CS | 0.708 $\pm$ 0.010 | 0.663 $\pm$ 0.007 | 0.665 $\pm$ 0.006 | 11.296 $\pm$ 0.865 | 10.233 $\pm$ 0.822 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: * Can the authors elaborate how $\delta$ is chosen in the experiments? Is this a choice that can be reproduced in practice, i.e., does it require running $\delta$-CS with multiple (all?) values of $\delta$, or does it require determining $\delta$ from a (small) validation experiment, or something else?
soundness: 3
strengths: * The exposition of the paper is very clear.
* $\delta$-CS which works by injecting noise and learning to denoise the signal is a neat idea to constrain generative search.
* The method is evaluated against many strong baselines on relevant tasks, and shown to outperform all baselines most of the time.

Overall, this is a good contribution to the line of work on GFlowNets.
summary: The paper studies the problem of generative search, where we seek to generate novel and diverse particles from a large design space that yield high reward. A core practical challenge is that the proxy reward model used for the generative search is unreliable, particularly away from its supporting distribution. The paper proposes a new approach, $\delta$-conservative search, that controls "how much to trust" the proxy reward model during generative search. The approach is validated with extensive evaluation and shown to outperform other methods.
weaknesses: * In the adaptive method for setting $\delta$, why does setting $\lambda \sigma \approx 1/L$ not set $\delta \approx \delta_{const} - 1/L$ to a constant that does not depend on the uncertainty $\sigma$? Given this, I find the name "adaptive" and the exposition slightly misleading. The adaptivity seems to stem from the sequence length $L$ and not the uncertainty about the reward proxy $\sigma$. How are the experimental results affected if $\lambda$ was chosen a constant?

* It would be nice to see the evaluation of various $\delta$ from Appendix B.1 extended to multiple datasets. It seems that the choice of $\delta$ is critical for the achieved performance and it would be interesting what ranges of $\delta$-values outperform the strongest baseline in each of the tasks to develop an understanding for the robustness of $\delta$-CS.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Regarding the choice of $\delta$, although the authors reported a control experiment on $\delta$ in the appendix, are all the experiments in Table 1 based on the same $\delta$?
2. What is the initial dataset $D_0$? How is it initialized? What is the cost to acquire the reward for all sequences in $D_t$ in line 3 of Alg.1? 
3. It seems that model $f$ is fixed in training, and it provides reward information for $\delta$-CS; how can we ensure $f$ can provide safe and correct reward information?
soundness: 2
strengths: This paper leverages GFlowNets for sequential generation, allowing it to efficiently generate diverse and high-reward biological sequences., and this method’s emphasis on balancing novelty with conservativeness addresses a key challenge in RL for scientific domains where exploration can easily veer into unreliable regions. This paper includes comparisons with various baselines and established benchmarks in biological sequence design.
summary: This paper introduces a novel method called $\delta$-Conservative Search ($\delta$-CS) for designing biological sequences with desired properties. Experiments across DNA, RNA, protein, and peptide design tasks demonstrate that δ-CS outperforms traditional methods, finding higher-scoring sequences. The approach successfully combines the advantages of evolutionary and reinforcement learning methods, offering a scalable and robust framework for biotechnological applications​.
weaknesses: 1. Novelty: This paper draws from evolutionary search and existing GFlowNet frameworks, combining aspects of both rather than developing a unique algorithmic structure. Specifically, $\delta$-Conservative Search extends GFlowNets by adding a conservativeness parameter, δ, to manage exploration around known data. While useful, balancing exploration with reliability is a recurring theme in off-policy RL, particularly with techniques like uncertainty-based exploration in Bayesian optimization and upper confidence bounds (UCB).
2. Evaluating a sequence's performance is critical in the design of biological sequences. A fast, low-cost, and relatively accurate method can often significantly reduce the complexity of the problem. In this paper, the reward information provided to the model is Oracle f. Since the reward signal provided by $f$, $f$ determines the model's learning, the accuracy and generalizability of $f$ are crucial. I doubt the evaluation and experiments are testing out-of-distribution sequences, which rely on the accuracy of $f$.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Can we use $\delta$-Conservative Search with any other proxy model? 

2. For the baseline methods that require surrogate/proxy (e.g. BO), is the same proxy model of $\delta$-CS used, with the same initial dataset?

3. How diversity/novelty is calculated? Is it the diversity of the last top-k batches? Presenting this metric across active learning rounds would have enhanced clarity and provided a more comprehensive understanding of the diversity achieved by $\delta$-CS.

4. In Figure 5, AAV task, there is a substantial difference in diversity between the proposed method and baselines. Given that the search space is medium-sized—somewhere between RNA-A and GFP—what might be the reason for this significant disparity?

5. How does performance vary with respect to the initial dataset size, particularly in domains with large search spaces? The initial dataset sizes currently employed are very large (e.g. 50% of the whole search space for TF-Bind-8, or the value set for GFP is again too much) compared to what is used for active learning approaches in the literature.

6. How does noise injection/level of conservativeness differ throughout the active learning rounds? Have you analyzed the behavior with respect to the proxy uncertainty?

7. Line 478, Shouldn't the phrase in line 478 be "prone to generating low rewards"?

8. What is the main source of benefit of using an RL method that utilizes a proxy model (as a reward function) and trains a generative policy instead of using some other active learning frameworks that directly utilize inexpensive proxy/surrogate models that can work under very limited training data? Why train a reward model + policy network instead of directly using the proxy model for querying?
soundness: 2
strengths: - $\delta$-CS provides a balance between novelty and conservativeness.
- An adaptive variant of $\delta$-CS adjusts the level of conservativeness dynamically based on the difficulty of the search landscape, which may vary in practical applications.
- The paper provides empirical evidence by validating $\delta$-CS across various biological sequence design tasks (DNA, RNA, protein, peptide) against baseline methods.
- The paper is generally well-written, although there are some minor descriptive confusions, as listed in the weaknesses below.
summary: On-policy reinforcement learning (RL) methods for biological sequence design tasks exhibit limited search flexibility. In contrast, off-policy methods like GFlowNets provide diversity-seeking capabilities and flexible exploration strategies. However, GFlowNets struggle with long sequence tasks due to the (hypothesis (*)) poor quality of the proxy model which serves as a reward function for training the policy network.
This paper addresses the problem of GFlowNets' ineffectiveness in long biological sequence design tasks due to the proxy model misspecifications for out-of-distribution inputs and proposes $\delta$-CS, an off-policy search method for training GFlowNets to enhance robustness.
$\delta$-CS addresses this issue by incorporating a conservativeness parameter, $\delta$, which can adaptively control the search boundaries. This approach effectively restricts the search to more reliable regions, mitigating the impact of inaccurate proxy models for out-of-distribution data points.
weaknesses: 1. The experiments do not convincingly demonstrate the “actual benefit” of the proposed approach relative to the related methods. I think, specifically, the paper should have presented how $\delta$-CS enhances robustness against proxy model misspecification by empirically evaluating the performance across varying proxy model initializations (e.g., from poor to high quality). Although the hard version of the task with a small search space (TF-Bind-8) is given, the hypothesis (*) is not quantitatively validated well enough when the search space is scaled up.

2. Another major concern is the lack of proxy model ablation, i.e. how does the performance of $\delta$-CS change under different proxy models used as a reward function? (See also the question below) This could demonstrate the flexibility of the proposed search approach without depending on the proxy model type specification.

3. The paper presents $\delta$-CS as a novel off-policy search method for training GFlowNets under proxy misspecification, however, how does $\delta$-CS differ from existing off-policy search methods for GFlowNets such as [7,8]? The method [7] proposes to train GFlowNet with the exploration of local neighborhoods, similar to restricting the search space via conservativeness, and [8] incorporates Thompson sampling to improve exploration. I think these should have been direct baselines of $\delta$-CS, as it would show the effectiveness of $\delta$-conservativeness against local search and sampling starting from the same initial misspecified proxy model. ( which were first published in 2023, so I think it is a fair comparison to ask). How does $\delta$-CS differ from these methods?


4. Using BO with a generative model [5,6] is a well-established approach to biological sequence design, which would provide a more competitive baseline than classical BO. Additionally, it would have provided stronger evidence to compare against existing trust region methods for biological sequence design [1,2,3,4], as the search intuition is similar and would be appropriate baselines for the proposed method-- since $\delta$-CS constraints the search space to reliable regions, similar to the trust region concept. BO is selected as a classical baseline, however, there are state-of-the-art trust region BO methods [2,3] that apply to combinatorial search spaces such as in biological sequence design space.

5. A quantitative analysis showing how the level of conservativeness/noise injection amount differs throughout the active learning rounds, which would validate the balancing capability of $\delta$-CS.

6. A part that requires a bit of clear discussion is that, if the proxy model is not of good quality then exploration helps to recover from getting stuck at local optima and collect diverse points, whereas over-exploration yields unreliable results as explained for using GFlowNets case. I think this trade-off could have been argued more clearly while motivating the method.

7. How do convergence and diversity of sampled sequences change w.r.t the batch size? It would have been better to show batch size ablations in a problem domain.

8. The paper lacks a discussion or a theoretical analysis on how $\delta$ yields better exploration, sample complexity, and/or convergence throughout the active learning process.

9. There is no clear discussion on the limitations and potential drawbacks of the method.



**Minor:**

- Line 77: The explanation regarding the search space size is unclear. Also, why it is necessary to show that the search space size is greater than 10^309? There is no explanation of what those numbers correspond to. (It is only clear to a person who is familiar with protein design literature, that 20 represents amino acids and the exponent is the length of the GPF protein sequence.)
- Regarding writing, the transition between RL and active learning terminology could be smoother. Specifically, the term "trajectories" is used without giving an explanation. Do they refer to the trajectory of sequences generated throughout the active learning/query rounds? For instance, the sentence (lines 80-81) could have been expressed more clearly, e.g. searching sequences around the neighborhood of observed (or annotated) data points. Or sampling full trajectories (in line 88).
- Appendix B.1, line 832, misspelling in "conservatives"




> [1] Biswas, S., Khimulya, G., Alley, E.C. et al. Low-N protein engineering with data-efficient deep learning. Nature Methods 18, 389–396 (2021).

> [2] Wan, Xingchen & Nguyen, Vu & Ha, Huong & Ru, Binxin & Lu, Cong & Osborne, Michael. Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces. International Conference on Machine Learning (ICML), 2021.

> [3] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 5496–5507, 2019.

> [4] Khan A, Cowen-Rivers AI, Grosnit A, Deik DG, Robert PA, Greiff V, Smorodina E, Rawat P, Akbar R, Dreczkowski K, Tutunov R, Bou-Ammar D, Wang J, Storkey A, Bou-Ammar H. Toward real-world automated antibody design with combinatorial Bayesian optimization. Cell Rep Methods. 2023 Jan 3;3(1):100374. 

> [5] Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. 
International Conference on Machine Learning (ICML), 2022.

> [6] Nate Gruver, Samuel Stanton, Nathan Frey, Tim G. J. Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. 2024. Protein design with guided discrete diffusion. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), 2023.

> [7] Kim, M., Yun, T., Bengio, E., Zhang, D., Bengio, Y., Ahn, S., & Park, J. (2023). Local search gflownets. arXiv preprint arXiv:2310.02710.

> [8] Rector-Brooks, Jarrid, et al. "Thompson sampling for improved exploration in gflownets." arXiv preprint arXiv:2306.17693 (2023).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: **Novel Contribution**: A key challenge in previous active learning framework is that, the proxy model often produces
highly unreliable rewards $R(x; \phi)$ for out-of-distribution inputs. This paper mitigates this by providing off-policy trajectories denoised from noise-injected high-reward sequences. This paper uses $\delta$ to control the how "novel" the denoised sequences are compared with existing ones. For example, $\delta=1$ means the full on-policy search.  

**Comprehensive Experiments**: The experiments include a range of biological sequence design tasks, including DNA, RNA, protein, and peptide design. The comparison with several baselines, including DyNA PPO and other Bayesian optimization methods, further strengthens the claims.
summary: The authors propose an off-policy search method, δ-Conservative Search (δ-CS), a novel off-policy search method for training GFlowNets designed to improve robustness against proxy misspecification. The proxy misspecification is critical in the biological sequence design problem due to its vastness of the search space and high cost of evaluating candidate sequences. 

The key of the δ-CS method is its dynamic control of search conservativeness through a parameter δ, which balances the trade-off between exploring new sequences and staying within the reliable regions of the proxy model. The method injects noise into high-scoring offline sequences by masking tokens randomly and then denoises them using a GFlowNet policy. This approach ensures that the search process is guided by the uncertainty of the proxy model, making it robust against proxy mis-specification, particularly for out-of-distribution sequences.

In summary, δ-CS offers a scalable, adaptive, and effective framework for generating novel and high-reward biological sequences, providing valuable contributions to the fields of synthetic biology and biotechnology.
weaknesses: **Complexity in Implementation**: Although the proposed method shows strong performance, its implementation appears to be complex. The exploration in the active learning process is controlled by $\delta$, making its adaptation crucial. The current adaptation is controlled by the hyperparameters $\lambda, \delta_{\rm const}$. Additionally, the selection of high-score offline sequences is based on a rank-based reweighted prior, with hyperparameter ($k$). There are also potential alternatives for selecting high-reward offline sequences and adapting $\delta$. As a result, it remains unclear whether the "three steps" outlined in Lines 187-189 offer a general improvement over standard active learning methods or if the paper has cherry-picked one favorable combination.

**Uncertainty Estimate**: The method assumes that the proxy model can reliably estimate uncertainty. However, as noted in the paper, the proxy model is unreliable for out-of-distribution (OOD) inputs. This raises a concern: while the paper does not trust the proxy's output for OOD inputs during search, it still relies on the proxy to compute uncertainty for all inputs. This assumption seems questionable. I know that the experiments in Appendix B demonstrate that this adaptation is effective, but I want to hear the opinions from the authors on that. 


**Choice of GFlowNets** The paper employs GFlowNets for the policy, which are equivalent to soft off-policy reinforcement learning algorithms. However, it does not explore whether the $\delta$-conservative search would be effective with other search methods. Furthermore, in Tables 1 and 2, the performance gain of GFN-AL + $\delta$-C over GFN-AL does not appear to be substantial compared to the performance gains of GFN-AL itself. This suggests that the main improvement might stem from the use of GFlowNets rather than from the search method. Many studies (to name a few, [1,2,3,4,5]) claim significant improvements over the standard GFlowNet (used in this paper), so it might be worth investigating whether replacing the GFlowNet in GFN-AL with a modified version could lead to even better performance than modifying the search.

[1] Local Search GFlowNets. Minsu Kim, et al. ICLR 2024 spotlight.
[2] Learning Energy Decompositions for Partial Inference of GFlowNets. Hyosoon Jang, et al. ICLR 2024 oral.
[3] Order-Preserving GFlowNets. Yihang Chen, et al. ICLR 2024.
[4] Learning to Scale Logits for Temperature-Conditional GFlowNets. Minsu Kim, et al. ICML 2024.
[5] QGFN: Controllable Greediness with Action Values. Elaine Lau, et al. NeurIPS 2024.

Meta-review: This paper propose a new robust RL method to combat reward misspecification focusing on the biological sequence design task. There are a lot of back and forth discussion between the author and reviewers. 

The main drawback of the paper lies in its inadequate experimental evaluation and a lack of competitive baselines. These are partially improved during the rebuttal phase with new experiments. However, what remains to be a problem, as raised by both reviewer WGsn and reviewer 6RU1, is that because GFN+$\delta$-CS is the only baseline that incorporate the proposed $\delta$-CS method, it is unclear whether the benefit comes from $\delta$-CS or from the unique combination of GFN+$\delta$-CS. It could be that $\delta$-CS only work well with GFN but not with any other base algorithm. Given that this is a pure empirical paper, the current experimental evaluation is inadequate to support its claim.

I suggest that the author follow the suggestion of reviewer WGsn and 6RU1 in designing informative experiments.

Decision: Reject