Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets

## Abstract

Abstract Generative Flow Networks (GFlowNets) are amortized sampling methods for learning a stochastic policy to sequentially generate compositional objects with probabilities proportional to their rewards. GFlowNets exhibit a remarkable ability to generate diverse sets of high-reward objects, in contrast to standard return maximization reinforcement learning approaches, which often converge to a single optimal solution. Recent works have arisen for learning goal-conditioned GFlowNets to acquire various useful properties, aiming to train a single GFlowNet capable of achieving different goals as the task specifies.
However, training a goal-conditioned GFlowNet poses critical challenges due to extremely sparse rewards, which is further exacerbated in large state spaces. In this work, we propose a novel method named R etrospective B ackward S ynthesis (RBS) to address these challenges.
Specifically, RBS synthesizes a new backward trajectory based on the backward policy in GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby efficiently solving the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks.

## 1 Introduction

Generative Flow Networks (GFlowNets; Bengio et al. ()) are a new class of probabilistic models
designed for sampling compositional
objects from high-dimensional unnormalized
distributions, and share similarities with Markov Chain Monte Carlo (MCMC) (Metropolis et al., , Hastings, , Andrieu et al., ).
Unlike MCMC methods, GFlowNets do not suffer from the mixing problem (Salakhutdinov, , Bengio et al., , ), as they generate each sample independently and amortize the sampling cost.
Indeed, GFlowNets transform sampling into a sequential decision-making problem: the agent learns a stochastic policy for sampling proportionally to the rewards, wherein each sequence of actions
yields a unique object.
In this regard, GFlowNets resemble reinforcement learning (RL), although RL agents typically seek an optimal policy maximizing
the cumulative reward.
There have been recent successes of GFlowNets in challenging problems due to their promising ability to generate high-quality and diverse candidates, including molecule discovery  (Bengio et al., , Li et al., , Jain et al., ), biological sequence design  (Jain et al., ), and causal modeling (Deleu et al., , , Atanackovic et al., ).

Humans often excel in solving compositional decision-making problems efficiently and generalizing knowledge to novel tasks due to the goal-directed nature of human behavior. This kind of goal-directed behavior is characterized by the ability to set a goal, develop a plan or strategy to achieve it and persistently take the actions that lead toward the goal. The vital role of goal-achieving has been a cornerstone not only in human development (Molinaro and Collins, ) but also in machine intelligence (Andrychowicz et al., , Liu et al., , Park et al., , Niemueller et al., , Addison, ).
To pave the way towards more human-like intelligence systems for GFlowNets that exhibit improved efficiency and generalization, it is promising to incorporate goal-conditioned policies within them, which aims to achieve arbitrary goals as the task specifies.

As recently studied in OC-GAFlowNets (Pan et al., ), training a goal-conditioned GFlowNet that learns to reach any goal can facilitate rapid adaptation to novel tasks with unseen rewards without the need for learning from scratch.
However, it is very challenging to efficiently train GFlowNets conditioning on goals due to the sparse and binary nature of rewards, which are exacerbated in a higher-dimensional goal and state space.

To tackle these critical challenges, we first formulate the problem of goal-conditioned GFlowNets (GC-GFlowNets) as learning a unified forward policy to reach any desired goals.
Then, we propose a novel approach, Retrospective Backward Synthesis (RBS), which is a simple yet effective method for efficiently training GC-GFlowNets. Concretely, the insight of RBS is sourced from the data perspective, augmenting the training data with high quality and diversity. It not only transforms a sequence of actions with a negative reward into successful experiences with a positive reward, thus increasing the quality of training experiences, but also generates a totally novel trajectory to increase the diversity of data. We conduct a comprehensive evaluation of our method across various tasks from different benchmarks, where a binary bonus is given only if the agent reaches the desired goal. In comparison with a thorough set of baselines, our method largely improves sample efficiency during training and boosts the generalization ability of GFlowNets. To conclude, our contributions are three-fold:

- •
We first formulate the problem of goal-conditioned GFlowNet as training universal forward policies, which take as input not only the current state but also an outcome state (goal) to achieve.
- •
We present a novel method named Retrospective Backward Synthesis, which imagines a new trajectory from a terminal state (goal), enhancing the quality and diversity of the training data.
- •
We successfully showcase the effectiveness of our method through extensive experiments. In the bit sequence generation task, our method achieves about a 100% success rate while all of the baselines completely fail. The results serve as a testament to its capability and underscore its potential for further GC-GFlowNets research.

## 2 Background

### 2.1 GFlowNets Preliminaries

Let $\mathcal{X}$ denote the space of compositional objects and $R$ denote a reward function that assigns non-negative values to objects $x\in\mathcal{X}$. The non-negative reward function is denoted by $R(x)$. GFlowNets work by learning a sequential,
constructive sampling policy $\pi$ that samples objects $x$ according to the distribution defined by the reward function ($\pi(x)\propto R(x)$). At each timestep, GFlowNets choose to add a building block $a\in\mathcal{A}$ (action space) to the partially constructed object $s\in\mathcal{S}$ (state space). This can be described by
a directed acyclic graph (DAG) $\mathcal{G}=(\mathcal{S},\mathcal{A})$, where $\mathcal{S}$ is a finite set of all possible states, and $\mathcal{A}$ is a
subset of $\mathcal{S}\times\mathcal{S}$, representing directed edges. The generation of an object $x\in\mathcal{X}$ corresponds to a complete trajectory $\tau=(s_{0}\to\cdots\to s_{n})\in\mathcal{T}$ in the DAG starting from the initial state $s_{0}$ and terminating in a terminal state $s_{n}\in\mathcal{X}$. We define state flow $F(s)$ as a non-negative weight assigned to each state $s\in\mathcal{S}$. The forward policy $P_{F}(s^{\prime}|s)$ is the forward transition probability over the children of each state, and the backward policy $P_{B}(s|s^{\prime})$ is the backward transition probability over the parents of each state. The marginal likelihood of sampling $x\in\mathcal{X}$ can be derived as $P_{F}^{\top}(x)=\sum_{\tau=(s_{0}\to\cdots\to x)}P_{F}(\tau)$. The primary objective of GFlowNets is to train a parameterized policy $P_{F}(\cdot|s,\theta)$ such that $P^{\top}_{F}(x)\propto R(x)$ (Bengio et al., , ).

#### 2.1.1 Training Criterion of GFlowNets

Detailed Balance. The detailed balance (DB) objective realizes the flow consistency constraint on the edge level, i.e., the forward flow for an edge $s\to s^{\prime}$ matches the backward flow, as defined in Eq. (1). For terminal states $x$, it pushes $F(x)$ to match terminal rewards $R(x)$. DB learns to predict state flows $F_{\theta}(s)$, forward policy $P_{F}(\cdot|s;\theta)$ and backward policy $P_{B}(\cdot|s;\theta)$.

$$ $\forall s\to s^{\prime}\in\mathcal{A},\ \ \ F_{\theta}(s)P_{F}(s^{\prime}|s; \theta)=F_{\theta}(s^{\prime})P_{B}(s|s^{\prime};\theta).$ (1) $$

Sub-Trajectory Balance.
Trajectory Balance (TB) extends DB from the edge level to the trajectory level based on a telescoping calculation of Eq. (1),
which parameterized the normalizing constant $Z_{\theta}$, forward policy $P_{F}(\cdot|s;\theta)$ and backward policy $P_{B}(\cdot|s;\theta)$, whose learning objective is defined as
$Z_{\theta}\prod_{t=1}^{n}P_{F}(s_{t}|s_{t-1};\theta)=R(x)\prod_{t=1}^{n}P_{B}(
s_{t-1}|s_{t};\theta)$.
However, TB can incur large variance due to only optimizing the trajectory-level constraint (Madan et al., ).
Sub-trajectory Balance (SubTB) (Madan et al., ) aims to mitigate the variance of TB, which considers the flow consistency criterion in the sub-trajectory level ($\tau_{i:j}=\{s_{i}\to\cdots\to s_{j}$}), where
$s_{i}$ and $s_{j}$ are not necessarily the initial and terminal state.
The learning objective of SubTB for each sub-trajectory is defined as in Eq. (2).
It is converted into a training objective for optimization by defining the loss function as the squared difference between the left and right-hand sides of Eq. (2) (Madan et al., ) in the log-scale, considering a weighted combination of all possible $O(n^{2})$ sub-trajectories.

$$ $F_{\theta}(s_{i})\prod_{t=i+1}^{j}P_{F}\left(s_{t}|s_{t-1};\theta\right)=F_{ \theta}(s_{j})\prod_{t=i+1}^{j}P_{B}\left(s_{t-1}|s_{t};\theta\right)$ (2) $$

## 3 Related Work

Generative Flow Networks (GFlowNets).
Recently, there have been a number of efforts applying GFlowNets to different important cases, e.g., biological sequence design (Jain et al., ), molecule generation (Bengio et al., ), combinatorial optimization (Zhang et al., , ), Bayesian structure learning (Deleu et al., ).
There have also been many works investigating how to improve the training of GFlowNets, enabling them to achieve more efficient credit assignment (Pan et al., ), better exploration (Pan et al., , Lau et al., ), or more effective learning objectives (Bengio et al., , Madan et al., ) that can better handle computational complexity (Bengio et al., ) and large variance (Malkin et al., ), and generalize to stochastic environments (Pan et al., , Zhang et al., ).
GC-GFlowNets learn flows and policies conditioning on outcomes (goals) for reaching any targeted outcomes (Pan et al., ).
However, little attention has been given to this topic, leaving this promising direction largely unexplored. Meanwhile, it is challenging to train goal-conditioned policies due to sparse rewards. Our work not only provides a formal definition of GC-GFlowNets but also proposes a novel method called retrospective backward synthesis to significantly improve their training efficiency and success rates.

Goal-Conditioned Reinforcement Learning. Our formulation of goal-conditioned GFlowNets is heavily inspired by the works of goal-conditioned RL. Standard Reinforcement Learning (RL) only requires the agent to finish one specific
task defined by the reward function (Schaul et al., ), while goal-conditioned RL trains an agent to achieve arbitrary goals as the task specifies (Andrychowicz et al., ). Goal-Conditioned RL augments the observation with an additional goal that
the agent is required to achieve (Liu et al., ). The reward function is usually defined as a binary bonus of reaching the goal.
To overcome the challenge of the sparsity
of reward function, prior work in goal-conditioned RL has introduced algorithms based on a variety of techniques, such as hindsight
relabeling (Andrychowicz et al., , Fang et al., , Yang et al., , Fang et al., , Ding et al., ), contrastive learning (Eysenbach et al., , ), state-occupancy
matching (Durugkar et al., , Ma et al., ) and hierarchical sub-goal planning (Chane-Sane et al., , Kim et al., , Nasiriany et al., ). Our work is closely related to hindsight relabeling, denoted as HER (Andrychowicz et al., ), which relabels any experience with some commanded goal to the goal that was
actually achieved in order to learn from failures. HER can generate non-negative rewards to alleviate the negative
sparse reward problem, even if the agent did not complete the task. However, the
agent using HER still suffers from low sample efficiency on large-scale problems, due to its limitation in only operating on the seen trajectories, while our method can imagine new trajectories with positive rewards for policy training.

## 4 Retrospective Backward Synthesis

In this section, we first formulate the problem of GC-GFlowNets in §4.1, followed by a motivating example in §4.2 to demonstrate the insights and efficacy of our method intuitively. Subsequently, we introduce our approach and discuss the techniques we developed for improved efficiency in §4.3.

### 4.1 Problem Formulation

Inspired by the literature on goal-conditioned RL  (Liu et al., , Park et al., , Veeriah et al., ), we extend the idea of flow functions and policies in GFlowNets to goal-conditioned GFowNets (GC-GFlowNets) that can generalize to different goals $y$ in the goal space (Pan et al., ). To tackle this problem, we augment the DAG with goals. Formally, GC-GFlowNets are formulated as a goal-augmented DAG $\mathcal{G}=(\mathcal{S},\mathcal{A},\mathcal{Y},\phi)$, where $\mathcal{Y}$ denotes the goal space describing the tasks, and $\phi:\mathcal{S}\to\mathcal{Y}$ is a tractable mapping function that maps the state to a specific goal. In this paper, we consider an identity function for $\phi$ following Pan et al. (). In the goal-augmented DAG, the reward function $R(x,y):\mathcal{S}\times\mathcal{Y}\to\mathbb{R}$ is goal-conditioned, determining whether the goal object is reached:

$$ $R(x,y)=\begin{cases}1,&\|\phi(x)-y\|\leq\epsilon\\ 0,&\text{otherwise }\end{cases}.$ (3) $$

Therefore, the primary objective of GC-GFlowNets is to train a parameterized goal-conditioned forward policy $P_{F}(\cdot|s,y,\theta)$ such that $P^{\top}_{F}(x|y)\propto R(x,y)$, where $P^{\top}_{F}(x|y)$ is the marginal likelihood of sampling $x\in\mathcal{X}$ given $y$. Meanwhile, the
flow function $F_{\theta}(s)$ and backward policy $P_{B}(s|s^{\prime},\theta)$ can be extended to goal-conditioned flow and policy $F_{\theta}(s|y)$ and $P_{B}(s|s^{\prime},y,\theta)$. Different from Eq. (1), the resulting learning objective for GC-GFlowNets for intermediate states is as follows:

$$ $\forall s\to s^{\prime}\in\mathcal{A},\ \ \ F_{\theta}(s|y)P_{F}(s^{\prime}|s, y,\theta)=F_{\theta}(s^{\prime}|y)P_{B}(s|s^{\prime},y,\theta).$ (4) $$

In practice, we train GC-GFlowNets by minimizing the following loss function $\mathcal{L}$ in the log-scale shown in Eq. (5).

$$ $\mathcal{L}_{\rm GC-GFN}=\left(\log\frac{F_{\theta}(s|y)P_{F}(s^{\prime}|s,y, \theta)}{F_{\theta}(s^{\prime}|y)P_{B}(s|s^{\prime},y,\theta)}\right)^{2},$ (5) $$

where $F_{\theta}(s^{\prime}|y)$ is substituted with $R(s^{\prime},y)$ if $s^{\prime}$ is a terminal state.

### 4.2 A Motivating Example

Figure: Figure 1: Succee rates with different set sizes in set generation.
Refer to caption: x1.png

Training GC-GFlowNets according to Eq. (4) can be challenging due to the sparsity of reward signals – the agent receives a non-zero reward only when it reaches the desired goal state, while all other states yield zero reward. The high-dimensional space presents a further challenge, as the agent may spend a significant amount of time exploring unproductive regions of the state space without receiving any meaningful feedback.

Reward relabeling (Andrychowicz et al., , Fang et al., , Pan et al., ) can alleviate sparse reward issues in goal-conditioned tasks by relabeling the achieved states as goals, but struggles to generalize to larger-scale scenarios with a large state space.
This challenge is exacerbated in the context of GFlowNets, where multiple trajectories can lead to the same goal, and their number increases exponentially in dimensionality. Agents need to explore and remember all possible paths to gain comprehensive knowledge to understand and complete the task.

We demonstrate this inefficiency problem in a goal-conditioned set generation task (Pan et al., ), where the
agent generates a set of size $|S|$ from $|U|$ distinct elements
sequentially starting from an empty set. At each timestep, the agent chooses to add an
element from $U$ to the current set $s$ (the GFlowNets state)
without repeating elements. We randomly sample a target state of size $|S|$ from $U$ ($|U|=30$) for each episode, and the GC-GFlowNets agent receives a negative
reward of 0 0 as long as the final generated object is not the target state.
Standard reward relabeling methods (HER (Andrychowicz et al., )) struggle in the task for $|S|\geq 12$. These methods solely rely on experiences collected from interactions with environments, potentially trapping the agent in local optima and hindering the discovery of an effective goal-achieving strategy for problems with increasing scales.

### 4.3 Proposed Method

In this section, we propose a novel approach, Retrospective Backward Synthesis (RBS), which is a simple yet effective method to efficiently tackle this challenging problem in GC-GFlowNets.

Figure: Figure 2: Overview of the Retrospective Backward Synthesis (RBS) approach.
Refer to caption: x2.png

Consider a trajectory $\tau=\{s_{0}\to\cdots\to s_{i}\to\cdots\to s_{n}\}$ collected by the forward policy $P_{F}$ of GC-GFlowNets which fails to reach the goal ($s_{n}\neq y$) with a zero reward.
As illustrated in
Fig. 2,
RBS utilizes the potential of the backward policy $P_{B}$ to synthesize a reverse trajectory $\tau^{\prime}=\{y\to\cdots\to s_{i}^{\prime}\to\cdots\to s_{0}\}$ from the commanded goal.
Therefore, $\tau^{\prime}$ provides a successful training experience as it achieves the goal state, thus enriching training data with positive feedback for GC-GFlowNets to mitigate the sparse reward problem. Unlike previous reward relabeling techniques, such as HER (Andrychowicz et al., ), which simply replaces the original goal $y$ with the achieved state $s_{n}$, RBS provides more diverse and informative new training data by imaging a new trajectory $\tau^{\prime}\neq\tau$, thus leading to more sample-efficient learning.
In practice, we store both collected trajectories $\{\tau_{i}\}_{i=1}^{m}$ and experiences from RBS $\{\tau^{{}^{\prime}}_{i}\}_{i=1}^{m}$ in the replay buffer, and jointly replay the two types of trajectories to optimize GC-GFlowNets.

Next, we introduce the techniques we develop to guarantee the efficiency and effectiveness of training GC-GFlowNets with RBS, where the overall algorithm is summarized in Alg. 1.

Figure: Algorithm 1 Retrospective Backward Synthesis GFlowNets

Age-Based Sampling. To optimize data utilization, we leverage a replay buffer (Lin, , Schaul et al., ) to store both the trajectories collected by GC-GFlowNets and the retrospective backward synthesized experiences.
To guarantee that all experiences are fully considered during training, we introduce an age-based sampling technique. Specifically, age-based sampling assigns the highest priority $p_{\rm max}>0$ to newly added experiences and updates their priority to zero after being learned. Consequently, newly added experiences can be replayed first, while learned experiences are randomly sampled. This prioritization scheme ensures that experiences are leveraged more thoroughly, which balances the exploration of fresh experiences and the exploitation of acquired knowledge.

Backward Policy Regularization.
The backward policy can be chosen freely as studied in (Bengio et al., ).
In the extreme case where $P_{B}$ is set to be a uniform policy, the optimization of GC-GFlowNets becomes challenging as it is not learnable and the data is excessively diverse.
On the other hand, specifying $P_{B}$ to be a deterministic policy can limit the data diversity.
We therefore learn the backward policy $P_{B}(\cdot|s^{\prime},\theta)$ within GC-GFlowNets based on the flow consistency criterion in §4.1, instead of specifying it to be a fixed policy, which offers smooth sampling aligned with the current model’s capacity.
Yet, when $P_{B}$ degenerates into a deterministic policy, it fails to provide diverse backward trajectories, which can limit the potential of GC-GFlowNets to generalize well.
To strike a balance between these two extremes and further enhance the diversity of the imagined trajectories, we introduce a backward policy regularization for $P_{B}$.
This regularization term penalizes the Kullback-Leibler (KL) divergence between $P_{B}(\cdot|s^{\prime},\theta)$ and a uniform distribution $\mathcal{U}$, thus encouraging $P_{B}$ to resemble a uniform distribution, while allowing for learning and adaptation.
Our training objective can be written as in Eq. (6), where $\gamma$ is the regularization coefficient.

$$ $\mathcal{L}_{\rm RBS-GFN}=\mathcal{L}_{\rm GC-GFN}+\gamma\times D_{\rm KL}(P_{ B}(\cdot|s^{\prime},y,\theta)\|\mathcal{U}).$ (6) $$

To avoid interference with the original training objective $\mathcal{L}_{\rm GC-GFN}$, we employ a linearly decaying hyperparameter $\beta$ to regulate the coefficient $\gamma$ (i.e., $\gamma\leftarrow\beta\times\gamma$). Consequently, the KL penalty gradually diminishes towards zero over the course of training.

Intensified Reward Feedback.
For long-horizon and high-dimensional tasks, a critical factor that affects learning effectiveness is the efficient propagation of the reward signal, which may require a number of steps and affect the learning of intermediate steps.
We propose an efficient technique that intensifies the learning signal, defined as
$\mathcal{L}_{\rm GC-GFN}=\left(\log\left[{F_{\theta}(s|y)P_{F}(s^{\prime}|s,y,
\theta)}\right]-\log\left[{CR(x,y)P_{B}(s|s^{\prime},y,\theta)}\right]\right)^
{2}$ for terminal states $s^{\prime}$,
where $C$ is a intensification coefficient to scale the effect of $R(x,y)$.
The mechanism behind this technique is that a larger value of $C$ indeed
amplifies the gradient of $P_{B}$ by $\log(C\times R(x,y))$ for terminal states (the detailed derivation can be found in Appendix A).
By the intensified reward feedback with a large value of $C$, we effectively strengthen the learning signal propagated backward through the trajectory, enabling more efficient learning and faster convergence in complex environments.

While our proposed method can effectively improve the training of GC-GFlowNets, it may still face challenges when dealing with extremely large-scale state spaces, e.g., antimicrobial peptides generation (Malkin et al., ) with $20^{50}$ possible states. To further improve its scalability, we propose a hierarchical approach for RBS that decomposes the task into low-level sub-tasks that are easier to complete. Detailed descriptions for how we realize hierarchical decomposition for RBS can be found in Appendix B due to space limitation.

Empirical Validation Fig. 1 compares RBS and GC-GFN with HER (Andrychowicz et al., ) in the set generation task with increasing set sizes.
The result demonstrates that our RBS approach can scale up to problems with large set sizes, while GC-GFlowNet with HER fails to maintain good performance as the problem complexity increases.

## 5 Experiments

In this section, we conduct extensive experiments to investigate our Retrospective Backward Synthesis (RBS) method to answer the following key questions:

- •
How does RBS-GFN compare with baselines in terms of sample efficiency and success rates?
- •
Can RBS-GFN scale to more complex environments?
- •
Can RBS-GFN effectively generalize to unseen goals and varying environments?
- •
Can RBS-GFN be built upon different GFlowNets training objectives?
- •
What’s the influence of employing different backward policies for backward trajectory synthesis?

### 5.1 GridWorld

In the GridWorld environment (Bengio et al., ),
the model learns to achieve any given goals starting
in a $H\times H$ grid. Specifically, we investigate mazes with increasing horizons $H$ ($32$, $64$, and $128$), respectively, resulting in different levels of difficulty categorized as small, medium, and large.

We compare our proposed RBS-GFN approach with the following state-of-the-art baselines.
(i) GFN w/ HER (Andrychowicz et al., ) is a GC-GFlowNets that relabels the negative reward in a failed trajectory with a positive reward.
(ii) OC-GAFN (Pan et al., ) is a recent method that utilizes contrastive learning to complement successful experiences,
and employs a trained Generative Augmented Flow Network (GAFN; Pan et al. ()) as an exploratory component to generate diverse outcomes $y$, which are subsequently provided to sample goal-conditioned trajectories.
(iii) DQN w/ HER leverages both deep Q-learning algorithm (Mnih et al., , Jang et al., ) and HER technique (Andrychowicz et al., ) to learn a near-optimal policy. This baseline is used to ablate the effects of GFlowNet-based training compared with RL style methods.
To ensure fairness, each baseline has the same model architecture and training steps as RBS-GFN, and we follow the experimental setup for hyperparameters as in (Pan et al., ).
We run each algorithm with three different seeds and report their performance in mean and standard deviation.
A more detailed description of the experimental setup can be found in Appendix C.

#### 5.1.1 Performance Comparison

The success rates for different methods for increasing sizes of the GridWorld environment (including small, medium, and large) are summarized in Fig. 4. We obtain the following observations based on the results. (i) GFN-based goal-conditioned approaches consistently outperform RL-based goal-conditioned methods (DQN w/ HER), as the latter can easily get trapped in local optima due to its greedy policy. The results validate the promise of training goal-conditioned policies using GFlowNets and pave the way for further advancements in goal-conditioned learning with GFlowNets.
(ii) Moreover, our proposed RBS-GFN method significantly outperforms GFN w/ HER and also the stronger OC-GAFN method in terms of sample efficiency and outcome-reaching ability, particularly in larger spaces. In contrast, the performance of GFN w/ HER deteriorates as the complexity of the environment increases, which highlights its limitations in handling large state spaces. (iii) The inferior performance of our method without RBS highlights the significance of our proposed approach. We remark that the superior performance of RBS is attributed to its enhancement of training data with higher quality and diversity.

Figure: Figure 3: Performance comparison in GridWorld. *Left*: Small. *Middle*: Medium. *Right*: Large.
Refer to caption: x3.png

#### 5.1.2 Generalization

We now evaluate the generalization ability of our RBS-GFN method to unseen goals and environments, which is important for real-world applications where the agent can encounter novel situations. To evaluate its ability to generalize to unseen goals, we mask $n$ goals from various locations in the map and test the success rates of reaching these unseen goals after the training process, as illustrated in Fig. 6(a) ($n=20$). As shown in Fig. 6(b), RBS-GFN obtains an almost $100$% success rate demonstrating its capacity to effectively determine the required actions to reach novel goals. Moreover, it outperforms our strongest baseline OC-GAFN by an approximately $15\%$ success rate.

Figure: (a) Testing map
Refer to caption: x5.png

We further investigate its generalization capability to unseen environments. We introduce unseen obstacles during the testing phase following (Kumar et al., ), which creates novel environments that the agent has not encountered during training. As shown in Fig. 6(c), RBS-GFN maintains a success rate of almost $100\%$, while OC-GFN obtains a success rate of $9\%$ and the RL-based method DQN completely fails. More unseen maps and corresponding results are provided in Appendix D.
The superior performance of RBS-GFN in unseen environments can be attributed to its ability to efficiently discover diverse paths to reach the goal as shown in Fig. 6(b).
Although OC-GAFN also has the potential to discover diver paths, its performance is limited by the available training budget.
On the other hand, DQN is limited to discovering a single trajectory to reach the goal as shown in Fig. 6(a), making it highly susceptible to failure when the learned path is blocked by unseen obstacles.

#### 5.1.3 Versatility

In this section, we demonstrate the generality of our approach by integrating it with another recent GFlowNets method based on SubTB (Madan et al., ), whose learning objective is based on $\mathcal{L}_{\rm SubTB}$ as introduced in Eq. (2). We evaluate the goal-reaching performance of RBS-GFN (SubTB) in terms of the success rate in the GridWord task (with $H=10$).
As shown in Fig 4, RBS-GFN can also be successfully built upon SubTB with a success rate of $100\%$, and achieves consistent performance gains.

#### 5.1.4 Ablation Study

In this section, we conduct an in-depth analysis of the key components of RBS-GFN to better understand their effect with a focus on two critical techniques including backward policy regularization and age-based sampling, while we defer the discussion of intensified reward feedback which is essential for scaling up to high-dimensional problems in Appendix A.1.

The backward sampling policy $P_{B}$ plays an important role in synthesizing helpful trajectories for training GC-GFlowNets. We evaluate the effect of different choices of $P_{B}$, including the regularized $P_{B}$ (based on Eq. (6)), a learned backward policy without constraints, and a fixed and uniform one. We compute the entropy of the forward policy $P_{F}$ to measure the ability to generate diverse trajectories of GC-GFlowNets.
We further visualize the trajectory distribution in the replay buffer for different choices of $P_{B}$ with t-SNE (Van der Maaten and Hinton, ) in Fig. 7(b).

As shown in Fig 7(a), the proposed regularized $P_{B}$ converges faster than other variants in terms of success rate while maintaining a satisfactory level of entropy. Furthermore, Fig 7(b) illustrates that the synthesized trajectory distribution of regularized $P_{B}$ and uniform $P_{B}$ cover a wide range, which effectively compensates for the limited coverage of the original data distribution, while learned $P_{B}$ struggles to synthesis trajectories that significantly differs from the original data distribution.

Fig. 9 demonstrates the effect of our proposed age-based sampling technique (with horizon $H=128$), which highlights its importance in improving learning efficiency and stability, as the model struggles to efficiently achieve a high success rate without age-based sampling (which fails to fully utilize and learn from newly-generated samples).

Figure: (a) Quantitative results
Refer to caption: x15.png

### 5.2 Bit Sequence Generation

In this section, we investigate the performance of RBS-GFN in the bit sequence generation task (Malkin et al., ).
Unlike previous approaches that generate these sequences in a left-to-right manner (Malkin et al., , Madan et al., ), we adopt a non-autoregressive prepend/append Markov decision process following Shen et al. (). The action space includes pretending or appending a $k$-bit word from the vocabulary $V$ to the current state, which increases the difficulty of the task (as the underlying structure of the problem is a directed acyclic graph rather than a simple tree (Malkin et al., )).
We consider bit sequence generation with small, medium, and large sizes with increasing lengths and vocabulary sizes following Pan et al. ().

Figure: Figure 8: Performance comparison in bit sequence generation.
Refer to caption: x17.png

As shown in Fig. 9, even the strongest OC-GAFN method struggles to learn efficiently given a limited training budget, while all other baselines completely fail. In contrast, RBS-GFN achieves high success rates of approximately 100% with fast convergence across different scales of the tasks. It is worth noting that RBS-GFN without either age-based sampling (denoted as RBS-GFN-w/o-AgeSample) or intensified reward feedback (see detailed discussion in Appendix A.1) both fail to generalize to more complex tasks, including *medium* and *large*, demonstrating the importance of our proposed techniques in enabling RBS-GFN to efficiently learn across various levels of complexity.

Figure: (a) TF Bind.
Refer to caption: x19.png

### 5.3 TF Bind Generation

In this section, we study a more practical task of generating DNA sequences with high binding activity with targeted transcription factors (Jain et al., ). Similar to the bit sequence generation task, the agent prepends or appends a symbol from the vocabulary to the current state at each step.
As shown in Fig. 10(a), RBS-GFN archives a success rate of $100\%$ and learns much more efficiently thanks to its retrospective backward synthesis mechanism, and outperforms other baselines, which illustrates its effectiveness for DNA sequence generation. We provide additional experimental analysis about this task in Appendix D.

### 5.4 AMP Generation

In this section, we study the antimicrobial peptides (AMP) (Jain et al., ) generation task for investigating the scalability of our proposed method. The task involves generating a sequence with a length of 50 from a vocabulary with a size of 20.
We follow the same experimental setup as in §5.3, considering an action space with prepend and append operations following Shen et al. (). The state space contains $20^{50}$ possible AMP sequences, which poses a significant challenge for efficient exploration and optimization. Moreover, the task is extremely difficult due to the vast sequence space and complex structure-function relationships compared to the case studied in  Pan et al. ().

To tackle this large-scale problem, we employ the goal decomposition method proposed in §4.3 (see details in Appendix B. By breaking down the target goal into simpler sub-goals, i.e., generating a shorter sub-sequence,
we can effectively reduce the complexity of the search space, enabling more efficient learning. We refer to this approach as Hier-RBS-GFN.
As demonstrated in Fig. 10(b), Hier-RBS-GFN significantly improves the learning efficiency and outperforms all baseline methods, which shows the scalability of our approach for tackling complex tasks with vast search spaces.

## 6 Conclusion

In this paper, we address the critical challenges of realizing goal-directed behavior and learning in GFlowNets. To overcome the training challenge due to extremely sparse rewards, we propose a novel method called Retrospective Backward Synthesis, which significantly improves the training of goal-conditioned GFlowNets by synthesizing backward trajectories. Our extensive experiments demonstrate state-of-the-art performance in terms of both success rate and generalization ability, which outperforms strong baselines. For future work, it is promising to further improve our method, e.g., sampling method considering alternative priorities (Sujit et al., ).

## Appendix A Intensified Reward Feedback

We re-write the loss function of GC-GFlowNets in the case if $s^{\prime}$ is terminal as follows:

$$ $\mathcal{L}_{\rm GC-GFN}=\left(\log{F_{\theta}(s|y)P_{F}(s^{\prime}|s,y,\theta )}-\log{\left[CR(x,y)P_{B}(s|s^{\prime},y,\theta)\right]}\right)^{2},$ (7) $$

which can be degenerated to Eq. (5) when we set $C=1$. In order to enable effective reward propagation, we set C as a large value in real implementation. Below we show a large value of $C$ can scale the gradient w.r.t. $P_{B}$ without influencing $P_{F}$ and $F$.

$$ $\displaystyle\frac{\partial\mathcal{L}_{\rm GC-GFN}}{\partial\theta}$ $\displaystyle=2\left(\log F_{\theta}(s|y)+\log P_{F}(s^{\prime}|s,y,\theta)- \log\left[C\times R(x,y)P_{B}(s|s^{\prime},y,\theta)\right]\right)\times$ (8) $\displaystyle\Bigl{(}\frac{1}{F_{\theta}(s|y)}\frac{\partial F_{\theta}(s|y)}{ \partial\theta}+\frac{1}{P_{F}(s^{\prime}|s,y,\theta)}\frac{\partial P_{F}(s^{ \prime}|s,y,\theta)}{\partial\theta}$ $\displaystyle-\frac{1}{C\times R(x,y)P_{B}(s|s^{\prime},y,\theta)}\frac{ \partial(C\times R(x,y)P_{B}(s|s^{\prime},y,\theta))}{\partial\theta}\Bigr{)}$ $\displaystyle=2\left(\log F_{\theta}(s|y)+\log P_{F}(s^{\prime}|s,y,\theta)- \log\left[C\times R(x,y)P_{B}(s|s^{\prime},y,\theta)\right]\right)\times$ $\displaystyle\Bigl{(}\frac{1}{F_{\theta}(s|y)}\frac{\partial F_{\theta}(s|y)}{ \partial\theta}+\frac{1}{P_{F}(s^{\prime}|s,y,\theta)}\frac{\partial P_{F}(s^{ \prime}|s,y,\theta)}{\partial\theta}$ $\displaystyle-\frac{1}{P_{B}(s|s^{\prime},y,\theta)}\frac{\partial(P_{B}(s|s^{ \prime},y,\theta))}{\partial\theta}\Bigr{)}$ $$

### A.1 Empirical Validation

To validate the effects of our proposed technique, i.e., intensified reward feedback, we conduct the ablation study in the bit sequence generation tasks, which are more complex and high-dimensional than the GridWorld tasks. In practice, we set $C=1e^{7}$ for *small* task, $C=1e^{25}$ for *medium* task, and $C=1e^{40}$ for *large* task. From the results shown in Fig 11, we observe that RBS-GFN completely fails in the task without intensified reward feedback, obtaining only a $0\%$ success rate.

Figure: Figure 11: Performance comparison with RBS-GFN without using intensified reward feedback.
Refer to caption: x21.png

## Appendix B Hierarchical Goal Decomposition

It is still challenging to tackle problems with extremely large-scale state spaces, and even our method can fail in these scenarios. To address this problem, we propose a hierarchical method to decompose the task into several low-level tasks that are easier to complete. Leveraging the consistent sequential structure in compositional GFlowNets tasks, we can set sub-goals manually, eliminating the need to learn a sub-goal generation policy additionally (Chane-Sane et al., , Kim et al., , Nasiriany et al., ). After training sub-level policies, all the generated sub-goals can be combined together to obtain the final goal.

Considering a sequence generation problem (Jain et al., ) as an example, wherein an agent is tasked with generating a sequence of length $l$ from a vocabulary of size $|\mathcal{V}|$, we can decompose this task into $k$ sub-level tasks. Consequently, we can train $k$ models, each capable of generating a sequence of length $l/k$. Subsequently, the $k$ generated sub-sequences can be concatenated to form a sequence of length $l$.

## Appendix C Experimental Setup

We build our implementation for all baselines and environments upon publicly available open-source repositories.(^1^11[https://github.com/GFNOrg/gflownet](https://github.com/GFNOrg/gflownet))

##### GridWorld.

The GridWorld (Bengio et al., ) is conceptualized as a 2-dimensional hypercube with side length $H:\{(s^{1},s^{2})|s^{i}\in\{0,1,\cdots,H-1\}$,
where the model learns to achieve any given goals (outcomes) starting from a fixed initial state $(0,0)$. We examine grids with $H$ set to $32$, $64$, and $128$, respectively, resulting in different levels of difficulty categorized as small, medium, and large. The agent receives a positive reward of $1$ only if it reaches the desired goal state. We use the Adam (Kingma and Ba, ) optimizer with a learning rate of $1e^{-3}$ for $2e^{4}$ training steps.

##### Bit Sequence Generation.

This task requires the model to generate sequences of length $n$ by pretending or appending a $k$-bit word to the current state. We consider $k=2,n=40$ for *small* task, $k=3,n=60$ for *medium* task, and $k=5,n=100$ for *large* one. We use the Adam (Kingma and Ba, ) optimizer with a learning rate of $5e^{-4}$ for $1e^{5}$ training steps.

##### TF Bind Generation.

Similar to the bit sequence generation task, the agent prepends or appends a symbol from the vocabulary with a size of $4$ to the current state at each step to generate a sequence of length $8$. We use the Adam (Kingma and Ba, ) optimizer with a learning rate of $5e^{-4}$ for $5e^{3}$ training steps.

##### AMP Generation.

This biological task requires the agent to generate antimicrobial peptides (AMP) with lengths of $50$ (Jain et al., ) from a vocabulary with size of $20$. For both RBS-GFN, Hier-RBS-GFN and all the baselines, we use the Adam (Kingma and Ba, ) optimizer with a learning rate of $5e^{-4}$ for $1e^{5}$ training steps.

### C.1 Implementation Details

We describe the implementation details of our method as follows:

- •
We use an MLP network that consists of 2 hidden layers with 2048 hidden units and ReLU activation (Xu et al., ).
- •
The trajectories are sampled from a parallel of 16 rollouts in the environment at each training step.
- •
We set the replay buffer size as $1e6$ and use a batch size of $128$ for sampling data and computing loss function.
- •
We combine the current state and goal state together as the input of our model. The input is transformed as one-hot embedding followed by our MLP model.
- •
We run all the experiments in this paper on an RTX 3090 machine.

### C.2 Baselines

We describe the implementation details of the baselines we use throughout this paper as follows:

- •
The only difference between GFN w/ HER and our method is that GFN w/ HER leverages HER (Andrychowicz et al., ) technique to enhance training experiences, while we utilize our proposed retrospective backward synthesis to augment the data with new reverse trajectories.
- •
For OC-GAFN, we follow the same experimental setup described in (Pan et al., ). This method not only leverages goal relabeling (Andrychowicz et al., ) but also uses GAFN (Pan et al., ) to generate diverse outcomes $y$, which are subsequently provided to sample outcome-conditioned trajectories. OC-GAFN requires training an additional GAFN model, which would be computationally expensive. At each training step, OC-GAFN takes 2 times gradient update, where one is for the negative samples and the other is for the relabeled samples. OC-GAFN does not maintain a replay buffer and uses newly sampled data to train its model.
- •
Following the implementation in (Andrychowicz et al., ), DQN w/ HER leverages both deep Q-learning algorithm (Mnih et al., , Jang et al., ) and HER technique (Andrychowicz et al., ) to learn a near-optimal policy.
- •
SAC w/ HER leverages both SAC algorithm (Haarnoja et al., ) and HER technique (Andrychowicz et al., ) to learn a near-optimal entropy-regularized policy. We follow the hyperparameters used in (Huang et al., ).

Figure: Figure 12: Performance of GFN with SubTB in TF Bind tasks.
Refer to caption: x22.png

## Appendix D Additional Experimental Results

We provide more experimental results that demonstrate the superior ability of our method.

##### Generalization.

We investigate the generalization ability of our method in more unseen maps. We show the three designed maps that consider different locations of goals and obstacles in Fig. 13(a-c). We also compare our method with baselines in terms of the success rate on these unseen maps. The experimental results shown in Fig. 13(d-e) demonstrate our proposed RBS method significantly enhances the generalization ability of GC-GFlowNets.

##### Versatility

To demonstrate that our method can also be applied to SubTB (Madan et al., ) learning objective, we provide additional experimental results on the TF Bind sequence generation task. We observe that both RBS-GFN trained with DB (denoted as RBS-GFN(DB) ) and RBS-GFDN trained with SubTB (denoted as RBS-GFN (SubTB)) achieve a $100\%$ success rate. notably, our method RBS gets consistent performance improvement in this task, while RBS-GFN-w/o-RBS almost fails to succeed.

Figure: (a) Map 1
Refer to caption: x23.png

##### SAC performance in TFBind sequence generation task.

We investigate the performance of SAC, known as soft actor-critic algorithm (Haarnoja et al., ), which is an entropy-regularized RL method rather than standard $\arg\max$ DQN. We evaluate the performance of SAC in the TF Bind

Figure: Figure 14: Performance of SAC in TF Bind tasks.
Refer to caption: x29.png

sequence generation task. As the action space is this task is discrete, we implement a discrete SAC algorithm based on the codes from CleanRL (Huang et al., ). From the results shown in Fig 14, we observe that SAC (w/ HER) even performs worse than the DQN algorithm. We hypothesize that it is because SAC prefers new states to maximize the entropy rather than high-reward states to complete the task. Although HER can provide abundant of successful experiences, it is still not enough for SAC to succeed.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How does RBS compare with diffusion policies, and in what scenarios does RBS offer distinct advantages?
2. Does RBS assume deterministic or stochastic environments, and how does it handle situations where the backward dynamics are not straightforward?
3. How can the authors ensure that the synthesized backward trajectories are meaningful and do not lead to false positives in the learning process?
4. Could the authors include a comparison with goal-conditioned RL methods to highlight the specific benefits of using GC-GFlowNets?
soundness: 3
strengths: 1. The paper identifies and targets a significant issue in the training of goal-conditioned GFlowNets, offering a practical and innovative solution. Augmenting backward trajectories for training is interesting.
2. Comprehensive empirical results are provided, demonstrating the effectiveness of RBS over existing methods on multiple benchmarks.
summary: The authors introduces a novel method, Retrospective Backward Synthesis (RBS), aimed at enhancing the training of goal-conditioned Generative Flow Networks (GFlowNets) by synthesizing new backward trajectories. RBS augments "virtual" backward trajectories in
goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity. RBS improves the sample efficiency and performance of GFlowNets across a range of tasks, including sequence generation and biological sequence design.
weaknesses: 1. Limited Discussion on Potential Drawbacks: The paper does not sufficiently address the potential limitations of RBS. For instance, there is no discussion about the computational overhead of synthesizing backward trajectories, nor is there a mention of whether the method is robust to different types of reward structures or environment dynamics.
2. Relevance and Scope of Application: The improvements are made specifically within the context of GC-GFlowNets, which may limit the applicability of the method.
3. Comparison with Diffusion Policies: Given the similarities between the proposed RBS and diffusion policies, a direct comparison would be valuable to understand the unique contributions and differences of RBS.
4. Assumptions on Environment Dynamics: It is unclear whether the proposed RBS method assumes or requires any particular properties of the environment, such as determinism or stochasticity. If the backward dynamics are infeasible or the environment is highly stochastic, the performance of RBS may be affected, and this should be addressed.
5. Quality of Synthetic Trajectories: The paper should include a discussion on how to ensure the quality of the synthesized backward trajectories, especially in cases where such trajectories may not correspond to realistic or feasible paths in the actual environment.
6. Lack of Comparison with Goal-Conditioned RL: Without a comparison to goal-conditioned RL, it is difficult for readers to fully appreciate the relative strengths and weaknesses of GC-GFlowNets. Including such a comparison would provide a more complete picture of the method's positioning within the broader field of goal-directed learning.
7. The authors may further investigate existing literature on augmenting backward trajectories for sample-efficient RL or backward learning in goal-conditioned RL, which makes the paper more comprehensive.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: - In algorithm 1) line 6, how do we guarantee that the backward policy could reach $s_0$ from $y$ each time?
- Are tuning for reward intensification and backward policy regularization difficult? What the the effect of hyper-parameters on the performance?
soundness: 3
strengths: - Backward-Looking Strategy for Enhanced Training: The proposed Retrospective Backward Synthesis (RBS) method utilizes a backward-looking strategy to synthesize trajectories from the goal state, significantly enriching training data. This approach effectively improves sample efficiency by converting failed experiences into successful learning signals, addressing the sparse reward problem.
- Empirical Validation of Sample Efficiency: The paper presents strong empirical results across a range of benchmarks, demonstrating that RBS markedly improves sample efficiency. The method achieves nearly 100% success rates in complex tasks where state-of-the-art baselines fall short, underscoring its practical impact.
- Clear Writing and Presentation: The paper is well-written and presented, with clear explanations, structured methodology, and comprehensive experimental results. The clarity facilitates a strong understanding of both the theoretical and practical aspects of the proposed approach.
summary: The paper proposes Retrospective Backward Synthesis (RBS), a novel method to enhance the training of goal-conditioned Generative Flow Networks (GC-GFlowNets). GC-GFlowNets have shown potential in generating diverse sets of high-reward candidates but face challenges due to sparse reward structures and limited coverage of explored trajectories, especially when using offline data. To address these limitations, RBS synthesizes backward trajectories that originate from a desired goal, enriching the training data with high-quality, diverse samples. This approach helps transform unsuccessful action sequences into positive learning experiences, thereby improving sample efficiency and generalizability.

The authors introduce additional techniques, such as reward signal intensification and backward policy regularization, to stabilize training and prevent mode collapse. Empirical results across various benchmarks, including GridWorld and bit sequence generation, demonstrate that RBS outperforms state-of-the-art methods in terms of success rates, sample efficiency, and scalability. Notably, RBS achieves nearly 100% success in large-scale tasks where competing approaches fail, highlighting its robustness and potential for further advancements in GC-GFlowNets.
weaknesses: - Scalability and Continuous Environments: The paper’s experiments focus on relatively simple and discrete environments, raising concerns about how well the Retrospective Backward Synthesis (RBS) method would scale to more complex, continuous, real-world tasks. The absence of testing in high-dimensional or continuous state-action spaces limits insights into its broader applicability.
- Tuning Challenges: The method's reliance on hyperparameters, such as reward scaling and backward policy regularization, introduces tuning challenges. While these components are beneficial for stabilizing training, they require careful adjustment, potentially impacting the ease of replication and practical deployment in varied scenarios.
- Lack of Comparison with Model-Based RL: Despite the inherent use of backward trajectory synthesis, which resembles model-based planning, the paper does not compare RBS with established model-based RL approaches such as MBPO or Dreamer. This omission makes it difficult to assess how RBS performs relative to other methods that also utilize environment models for planning and sample efficiency.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please address my concerns in the weakness
soundness: 3
strengths: * The paper is well-written and straightforward to understand.
* Retrospective Backward Synthesis (RBS) is introduced with clear motivation, and the paper also presents training techniques such as backward policy regularization. 
* Empirical results demonstrate that the proposed method outperforms baselines, showing improved performance and sample efficiency.
summary: This paper addresses key challenges in goal-conditioned Generative Flow Networks (GFlowNets), specifically the problems of sparse rewards and limited trajectory coverage. The authors introduce Retrospective Backward Synthesis (RBS), a method that generates additional backward trajectories to expand the training data. Their approach aims to improve both the quality and diversity of training trajectories, providing more learning signals in scenarios with sparse rewards. Empirical evaluations demonstrate improved sample efficiency and performance compared to baseline methods across multiple benchmarks.
weaknesses: * The evaluation tasks do not include key benchmarks like RNA Generation from Pan et al. (2023a), which limits direct comparison.
* The differences between the proposed RBS method and OC-GAFN are not clearly articulated. A more comprehensive discussion is needed to clarify the specific advantages of the RBS method. 
* It remains unclear how goals are defined across the evaluated tasks, which could impact generalizability and reproducibility.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1.	Age-Based Sampling is a very straightforward technique. How does it compare to previous methods like Prioritized Experience Replay (PER)? Did the authors attempt using PER as well?

2.	Regarding the experimental setup, since you’ve compared your method with reinforcement learning approaches, I assume that these experiments share the same tasks as those in reinforcement learning. If that’s the case, why weren’t newer RL methods selected as baselines? Additionally, for tasks like bit sequence generation, TF binding generation, and AMP generation, is DQN an appropriate baseline?
soundness: 2
strengths: This paper proposes a novel method called Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories in goal-conditioned GFlowNets to improve the quality and diversity of training trajectories. This approach introduces rich learnable signals, effectively addressing the sparse reward problem.
summary: The paper tackles the challenge of training goal-conditioned Generative Flow Networks (GC-GFlowNets) in environments with sparse rewards and limited offline data. It introduces Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories to enrich training data, improving sample efficiency and diversity. Experiments demonstrate that RBS significantly improves performance and generalization in various benchmarks.
weaknesses: 1.The experimental tasks are relatively simple and insufficiently comprehensive.

2.The latest goal-conditioned reinforcement learning algorithms are not selected for comparison.

Meta-review: This paper introduces Retrospective Backward Synthesis (RBS), a novel method that addresses challenges in training goal-conditioned Generative Flow Networks (GC-GFlowNets) by enriching training data with synthesized high-quality, diverse backward trajectories. Additional techniques are introduced, such as reward signal intensification and backward policy regularization, to prevent mode collapse. Empirical results across various benchmarks demonstrating its superior performance in discrete environments.

The reviewers generally acknowledge the paper's contribution in its potential for advancing GC-GFlowNet applications, considering it both novel and solid. The paper is well-written, with a clear motivation, structured methodology, and comprehensive experimental results. Empirical evaluations demonstrate that RBS effectively addresses the sparse reward problem and significantly outperforms state-of-the-art methods in terms of sample efficiency, success rates, and scalability.

Despite these strengths, some weaknesses and limitations are noticeable in the original manuscript. First, the experiments are limited to discrete environments, raising concerns about the scalability of RBS to continuous real-world tasks. The evaluation tasks also omit some key benchmarks, such as RNA generation. Comparisons with alternative approaches, such as diffusion policies, goal-conditioned RL, and model-based RL methods, are absent, leaving the unique advantages and positioning of RBS unclear. Additionally, the paper does not adequately address the potential computational overhead of synthesizing backward trajectories or its robustness to different reward structures and environment dynamics. Furthermore, questions about the quality of synthesized trajectories, the feasibility of backward dynamics in stochastic environments, and the tuning challenges associated with hyperparameters like reward intensification remain unanswered.

The authors’ responses offer detailed clarifications to these questions and provide additional experimental results. Notably, they address the inherent difficulty of the benchmarks used in the paper, compare their approach with goal-conditioned RL, present experiments on RNA generation, demonstrate generalization to different environments and reward structures, and explain hyperparameter tuning along with its robustness.

Three out of four reviewers recommend acceptance. While the remaining reviewer did not provide follow-up feedback on the authors' response, it seems that the concerns raised have been addressed. Additionally, although two positive reviewers did not respond to the authors’ rebuttal, their initial assessments were favorable. I support the reviewers' recommendation and suggest that the authors carefully address the main points raised in the reviews, particularly focusing on additional analyses and experiments, when preparing the final version.

Decision: Accept (Poster)