Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets
Authors: Haoran He
Can Chang
Huazhe Xu
Ling Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Backward Sampling
Goal-Conditioned Learning
Generative Flow Networks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: A simple yet efficient technique named retrospective backward synthesis for sample-efficient goal-conditioned GFlowNets training.

Abstract: Generative Flow Networks (GFlowNets), a new family of probabilistic samplers, have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast to standard return maximization approaches (e.g., reinforcement learning) which often converge to a single optimal solution. Recent works have focused on developing goal-conditioned GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly in high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data is available. In this work, we propose a novel method called \textbf{R}etrospective \textbf{B}ackward \textbf{S}ynthesis (\textbf{RBS}) to address these critical problems. Specifically, RBS synthesizes new backward trajectories in goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks. Our codes are available at \url{https://github.com/tinnerhrhe/Goal-Conditioned-GFN}.

Published as a conference paper at ICLR 2025
LOOKING BACKWARD: RETROSPECTIVE BACKWARD SYNTHESIS FOR GOAL-CONDITIONED GFLOWNETS
Haoran He', Can Chang", Huazhe Xu², Ling Pan
Hong Kong University of Science and Technology
Tsinghua University
ABSTRACT
Generative Flow Networks (GesowNets), have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast standard return maximization approaches (c.g., reinforce- ment learning) which often converge to a single optimal solution. Recent works have focused on developing condicondi GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data available. In this work, we propose a novel method called Retrospective Backward Synthesis (RBS) to address these critical problems. Specifically, RBS synthesizes new backward trajec- tories in goal-con coal-conditooned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by large margin and outperforms strong baselines on various standard evaluation benchmarks. Our codes are available https:: : // / thub. com/t. onneh Goa l Condi inaediiiiinnnn onndd-GFN
family of probabilistic samplers,
INTRODUCTION
Generative Flow Networks (etsowNets: Bengio al. (2021) new class of probabilistic models
designed for sampling compositional objects from high-dimensional unnormalized distributions.
GFlowNets generate each sample independently and amortize the sampling cost, and therefore do
not suffer from the mixing problem (Salakhutdinov 2009 Bengio et al., 2013 202T) in Markov
Chain Monte Carlo (MCMC) (Metropolis et al.] 1953: Hastings 1970 Andrieu et al. (2003). Indeed,
1970]Andriiu
problem: the agent learns stochastic policy for sampling proportionally to the rewards, wherein each sequence of actions yields
GFlowNets transform sampling into a sequential
unique object. In this regard, GFlowNets resemble reinforcement learning (RL) (Tiapiin al.
2024 Deleu et al. 2024b He et al., 2024), although standard RI. typically focuses on optimizing policies for single reward-maximizing objective. Due to the promising ability of (FlowNets to generate high-quality and diverse candidates, they have achieved great success in challenging
problems, including molecule discovery (Bengio et al. 2021 Li et al. 2022, Jain et al. 2023
biological sequence design (Jain et al. 2022), and causal modeling (Deleu et al. 2022 2024a 4a
Atanackovic al. 2024).
Goal-directed learning has been well conceptualized and studied across various fields, particularly in reinforcement learning (Andrychowicz et al., 2017 Park et al. [2024] Niemueller et al. 2019 Addison 2024), where is known as condiitode RL (GCRL) (Liu al.] (022). GCRL trains a single model and learns general policies capable of reaching arbitrary target states, showcasing significant benefits and performance improvements. However, it remains largely unexplored in the context of GFlowNets with only a few prior studies. One such study proposes training goal-conditioned GFlowNets (GC-GFlowNets)) (Pan et al.) 2023a) that can reach any goal within the object space (Roy (ct al. 2023). It further demonstrates that GC-GFlowNNets facilitate rapid adaptation to novel tasks with unseen rewards, eliminating the need to learn from scratch, unlike traditional GCRL.
1 "Corrsponndence Ling Pan (lingpan @uutt ust.hk)

---

Published as a conference paper at ICLR 2025
However, it is challenging to train GFlowNets conditioned on goals due to the sparse and binary nature of rewards, as the agent only receives positive rewards upon reaching the specified goals. GC-GFlow/Ntss collect data by interacting with the environments in restricted steps, leading to risk of getting trapped in constrained distributions (Yarats et al. 2021). Furthermore, previous approaches highly rely on the diversity and coverage of training trajectories, which poses a critical challenge when limited to offline data. This is particularly important in offline goal-conditioned learning, which enables training general goal-reaching policies from purely offfine interaction trajectories without any further environment interaction (Ma al. 2022a).
To tackle these challenges, we propose a novel approach called Retrospective Backward Synthesis (RBS), simple yet effective method for efficiently training GC-GFlowNets that learns unified forward policy capable of reaching any desired goals. Different from existing approaches inspired by GCRL litraatree (Pan et al. 2023a Roy et al., 2023), RBS augments the forward trajectories collected by the forward policies with synthesized backward trajectories by looking backward, guided by the inherent backward policies. It is noteworthy that the synthesized backward trajectories represent successful experiences as they consistently reach the desired goals. The key insight of RBS lies in its data-driven approach, which enriches the training data with high quality and diversity. It not only transforms sequences of actions with failure rewards into successful experiences with meaningful rewards, thereby increasing the quality of training experiences, but also generates entirely novel trajectories to increase the diversity of data. Nevertheless, it is still challenging to scale it up to more complex and long-howe long-horizon problems, where GC-GFlowNets are prone to instabilities and mode collapse. We hypothesize this is due to ineffective reward gradient propagation caused by severe credit assignment issues, which results in poor utilization of valuable learning signals. To address these issues, we propose to intensify reward signals in the training objective to strengthen gradient backpropagation and regularize the backward policies to enhance the diversity of synthesized backward trajectories. We conduct a comprehensive evaluation of our method across various tasks from different benchmarks, where a binary bonus is awarded only if the agent reaches the desired goal. In comparison with a thorough set of baselines, our method largely improves sample efficiency during training and enhances the generalizability of GC-GFlowNets. Our contributions are three-fold:
We present a novel method named Retrospective Backward Synthesis, which imagines new trajectory from a desired goal, enhancing the quality and diversity of the training data.
We introduce effective techniques, e.g., reward intensification and backward policy regularization, stabilize and improve the training process.
We showcase the effectiveness of our method through extensive experiments. A noteworthy result is that our method achieves about a 100% success rate in large-scale sequence generation tasks while all of the baselines completely fail. The results serve as a testament to its capability and underscore its potential for further GC-GFlowNets research.
PRELIMINARIES
2.1 GELOWNETS
Let X' denote the space of compositional objects and R denote reward function that assigns non- negative values to objects x E X. The non-negative reward function is denoted by R(r). GFlowNets work by learning a sequential, construetive sampling policy that samples objects according to the distribution defined by the reward function (I(x) x R(x)). At each timestep, GFlowNets choose to add a building block E AA (action space) to the partially constructed object E S (state space). This can be described by directed acyclic graph (DAG) G (S,A), where S is a finite set of all possible states, and A is a subset of S x S, representing directed edges. The generation of an object x E X' corresponds to a complete trajectory T = (sg + + s,)) T in the DAG starting from the initial state so and terminating in a terminal state s, E X. We define state flow each state
F(s) as non-negative weight assigned forward transition probability over the children of each state, and the backward policy Ps(s)s') is the backward transition probability over the parents of each state. The marginal likelihood of sampling
S. The forward policy (s(''(s) is the
x E X can be derived as PT (x) = Er= -Er=(eo -x)) (T) The primary objective of GelowNets is
to train parameterized policy Pr((|s,0) 0) such that PT. (x) x R(x) (Bengio et al.
2021
2023
2

---

Published as a conference paper at ICLR 2025
2.1.1 TRAINING CRITERIA OF GFLOWNETS
Detailed Balance. The detailed balance (DB) objective realizes the flow consistency constraint on the edge level, i.e., the forward flow for an edge - s' matches the backward flow, as defined in Eq. 41). For terminal states z, pushes F(x)) match terminal rewards R(z). DB learns predict state Hlws (s))), forward policy (-(11s; 0) and backward policy Pa (-|8;;;000 0):
V E A, Fo(s))P(s'(ss:)) = (ssss))
(I)
(Sub-) Trajectory Balance. Trajectory Balance (TB) extends DB from the edge level to the trajectory
level based telescoping calculation of Eq. ①.. which parameterized the normalizing constant
Zo. forward policy P= 8) and backward policy P 0). whose learning objective is defined
R(x) IT== (s 1s; 0). Sub-trajectory Balance (SubTB) (Madan et al. 2023b) aims mitigate the variance TB and bias of DB, which considers the flow consistency criterion in the sub-trajectory level (Tiy {s;
as Ze Pf($e|s(_1;:)
8j) )) where and 8j are not necessarily the initial and terminal state. The learning objective of SubTB for each sub-trajectory defined as in Eq. (2) Its loss function is the squared difference between the left and right-hand sides of Eq. (2) (Madan al. 2023a) in the log-scale, considering a weighted combination of all possible ((n") sub-trajectories.
(s(s:) II PF (8)(s-110)) Fo(s,) 1m1+1
=o(ss)
PB (81-1186; 1=++1
II
(2)
2.2 PROBLEM FORMULATION OF GC-GFLOWNETS
Inspired by the literature on goal-condiine conditiond RL (Liu al. 2022 Park al. 2024 Veeriah et
2018), the idea of flow functions and policies GFlowNets can be generalized to different goals y in the goal space (Pan et al. 2023a), leading the formulation of goal-conditioned GFowNets (GC- Gelsw/eets.. GC-GFlowNer ets can be formulated as goal-augmented DAG g = (S, A, V. Φ), where ) denotes the goal space describing the tasks, and の S + y is tractable mapping function that maps the state to a specific goal. this paper, we consider an identity function for & following Pan
cet al. (2023a). In the goal-augmented DAG, the reward function R(x,y) S x y R is also conditioned on goals, determining whether the goal object is reached:
-f1 [1, [1(()) vll <
R(x, v) = 10. otherwise
Therefore, the primary objective of GC-GFlowNets is to train a parameterized coal-conditioned forward policy (-1s,, 8) such that P (æly) x R(x, y). where P (xly) the marginal likelihood of sampling E X' given Mcanwhile, the flow function (s(s) and backward policy PB(s|s', 0) can be extended goal-conditioned flow and policy Fø(sly) and (s(s|'', y: 0). Different from Eq. (T), the resulting learning objective for GC-GFlowNert Nets for intermediate states is as follows:
Vs - 8' E A. Pa(s)))PPss 8, 0) = Fa(s' Pn(sss' 0).
In practice, Ge-sFlowNets can be trained by minimizing the following loss function L in the log- scale (for numerical stability as discussed in Bengio et al. (202)) h as shown in Eq. (5) where Fo lv) is substituted with R(s', y) if s' is terminal state.
Iog (o(1)))
LGC-GEN
(5)
(o(ssl))
3 PROPOSED METHOD: RETROSPECTIVE BACKWARD SYNTHESIS
In this section, we first introduce a motivating example in $3.1 to demonstrate the insights and efficacy of our method intuitively. Subsequently, we introduce our novel approach and discuss the techniques we developed for improved efficiency in $3.2 which improves the training of GC-GFlowNets in a simple yet effective manner.

---

Published as a conference paper at ICLR 2025
3.1 A MOTIVATING EXAMPLE
Training GC-tFlowNets according to Eq. (4) can be challenging due to the sparsity of reward signals the agent receives non-zers reward only when it reaches the desired goal state, while all other states yield zero reward. The high-dimensional space presents a further challenge, as the agent may spend significant amount of time exploring unproductive, restricted regions of the state space without receiving any meaningful feedback. Reward relabeling (Andrychowiez et al. 2017| Fang et al.) 2018] Pan et al. 2023a) aims to alleviate sparse reward issues in goal-conditioned tasks by relabeling the achieved states as goals, which have demonstrated success in the goal-conditioned RI. literature, but still struggles to generalize to larger-scale scenarios with large state space. This challenge is exacer- bated in the context of GFlowNets, where agents can discover different trajectories leading the same goal, and their number increases exponentially in dimensionality. Consequently, these reward labeling techniques may struggle to generalize across these different goal-trajectory relationships and prone to get stuck in local optima, as they rely solely on observed data without expanding their data coverage.
We demonstrate this inefficiency problem a goal- conditioned set generation task (Pan al. 2023b), where the agent generates a set of size [S] from [0| distinct el- ements sequentially starting from an empty set. At each timestep, the agent chooses to add an element from U the current set s (the GFlowNets state) without repeating ele- ments. We randomly sample target state of size [S] from U (UU)= = 30) for each episode, and the GC-GFlowNets agent receives a negative reward of 0 as long the final gener- ated object is not the target state. Previous state-of-the-an thr method (Pan et al. (2023a) based
100
Success
70
60
50-
OC-GARN
10 Set Size
11
12
standard reward rela-
Figure 1: Succee rates with increasing set sizes in set generation.
beling (HER (Andrychowicz et al. 2017)), 7) struggles in this nigh-dimensional task for [S] 12 with large state space. These methods solely rely on experiences collected from interactions with environments, potentially trapping the agent in local optima and hindering the discovery of an effective goal-achieving strategy for problems with increasing scales.
3.2 PROPOSED METHOD
In this section, we propose a novel approach, Retrospective Backward Synthesis (RBS), which is a simple yet effective method efficiently tackle these challenges mentioned above in GC-GFlowNets.
8; 8n). collected by the forward policy Pr of GC-GFlowNer that fails to reach the goal (sn, # y) and receives a zero reward. As illus- trated in Fig. 2, RBS utilizes the potential of the backward policy P3 to synthesize a backward trajectory 7' = {y ... + S - + su} from the commanded goal. When employing r' for training, we reverse to guarantee r' starts from the initial state sp. similar to r. Therefore, r' provides successful training experience as it achieves the goal state, thus enriching training data with positive feedback for GC-GFlowNets to mitigate the sparse reward problem. Unlike previous reward relabeling techniques, such as HER (Andrychowicz et al. 2017). which simply replaces the original goal y with the achieved state 8n, RBS provides more diverse and informative new training data in the trajectory level. RBS imagines a totally new trajectory r' * r, thus leading to significant data augmentation and more sample-efficient learning. practice, we store both collected trajectories {Tr (m and experiences from RBS (7 |m a replay buffer, and jointly replay the two types of trajectories to optimize GC-GFlowNers
Consider a trajectory т = (50 +
Update
collected
trajactory
reached
goal
Replay buffer
GFlowNets
target goal
Po
target
goal
RB5
trajectory r
Figure Overview of the Retrospective Backward Synthesis (RBS) approach.
However, training GC-GFlow.ets with RBS may still face the risks of learning instabilities and mode collapse when sealing to longer-horizon and more complex tasks, which may be caused by ineffective reward gradient backpropagation and inefficient experience replay. To address these challenges and
4

---

8o) with retrospective backward Published as a conference paper at ICLR 2025
further enhance the diversity of generated backward trajectories, we introduce the techniques we have developed the next paragraphs, where the overall algorithm is summarized Alg. |
Algorithm 1 Retrospeccive Backward Synthesis GFlowNets
1: Initialize: GC-GFlowNets Fo(sly), (s'''s,, 0) and Pa(s|s', y, with parameters 0, replay
buffer B, max priority Pmaa.
2: = {0, 1,... .N- - 1 do
3: Sample goal y ~ y randomly
4: Collect forward trajectory = {so + $1 + + tp) with Pp, and obtain reward
R(s,, y) < (s,, y/
5:
Store 7 - (T. y, Rt with priority Pmax > 0 0n in B
6:
Collect backward trajectory T' {y
8
synthesis using Pn. and obtain reward R(y, v) + [[( y)
7:
Store T' = (T's y. R) with priority Pmax > in B
8:
Sample a batch {Tr);= (T:)) proportionally to their priorities from B.
9:
Update GC- IC-GFFloNeet towards minimizing Ey. (6)
10: Update priorities of and T', and the weighting coefficient y
11:
end for
Age-Based Sampling. Trajectories collected by GC-GFlowNNts and the retrospective backward synthesized experiences are stored in a reply buffer for better data utilization. However, uniform sampling of these goal-reaching trajectories often fails to expose the agent to sufficiently diverse experiences that match its evolving learning ability (Fang et al. 2019), which highlights the im- portance of sampling strategy (Kloek & van Dijk 1976 (Schaul et al. 2016 To guarantee that all experiences are fully considered during training, we introduce an age-based sampling technique. Specifically, age-based sampling assigns the highest priority Pmax > 0 to newly added experiences and updates their priority to zero after being learned. Consequently, newly added experiences can be replayed first, while learned experiences are randomly sampled. This prioritization scheme ensures that experiences are leveraged more thoroughly, which balances the exploration of fresh experiences and the exploitation of acquired knowledge.
Backward Policy Regularization. The backward policy can be chosen freely as studied in (Bengio et al. 2023). In the extreme case where PB is set to be a uniform policy, the optimization of GC- GFlowNets becomes challenging
it is not learnable and the data is excessively diverse. On the other hand, specifying PB as a deterministic policy can limit data diversity. We therefore learn the backward policy (- 0) within GC-GFlowNe based on the flow consistency criterion in 22. instead of specifying it to be a fixed policy, which offers smooth sampling aligned with the current model's i capacity. Yet, when PB degenerates into deterministic policy. it fails to provide diverse backward trajectories, which can limit the potential of GC-GFlowNets to generalize well. To strike a balance between these two extremes and further enhance the diversity of the imagined trajectories, we introduce a backward policy regularization for PB- This regularization term penalizes the Kullback- Leibler (KL) divergence between [=[-|s', 0) and uniform distribution 24, thus encouraging P3 to resemble uniform distribution, while allowing for learning and adaptation. Our training objective can be written as in Eq. (6. where 7 is the regularization coefficient.
LABS GGC-GFN + 7 (Pa((P' y. 0)[(2)).
(6)
To avoid interference with the original training objective LOC-GFN, we employ linearly decaying hyperparameter 3 to regulate the coefficient 7 (i.e., y + x 7). Consequently, the KL penalty
gradually diminishes towards zero over the course of training.
Intensified Reward Feedback. For long-horizon and high-dimensional tasks, a critical factor that affects learning effectiveness is the efficient propagation of the reward signal, which may require number of steps and affect the learning of intermediate steps. The recent OC-GAFN (Pan et al. 2023a) approach considers the terminal reward at each step (due to the binary nature of rewards), but may introduce stochasticity of the learning signal particularly in the case of the more challenging graph-structured DAG. We propose an efficient technique that intensifies the learning signal, defined
as LGC-GFN GFN = (log [a (s(''s ss y, 0)] log [CR(s,, PB(sss' V. B)1) for terminal states s', where C' is intensification coefficient scale the effect of R(x, y). The mechanism behind this technique is that a larger value of C' indeed amplifies the gradient of PB by log (C R(x,, y)) for terminal states (the detailed derivation can be found in Appendix Ap. By the intensified reward feedback with
5

---

Published as a conference paper at ICLR 2025
large value of C, effectively strengthen the learning signal propagated backward through the trajectory, enabling more efficient learning and faster convergence complex environments.
While our proposed method can effectively improve the training of GetooFlo it may still face challenges when dealing with extremely large-scale state spaces, c.g., antimicrobial peptides generation (Malkin et al. 2022) with 2000 possible states. To further improve its scalability, we propose a hierarchical approach for RBS that decomposes the task into low-level sub-tasks that are easier to complete. Detailed descriptions for how we realize hierarchical decomposition for RBS can be found in Appendix B due to space limitation.
Empirical Validation Fig. compares RBS and OC-GAFN (with HER (Andrychowicz et al., 2017) in the set generation task with increasing set sizes. The result demonstrates that our RBS approach can scale up to problems with large set sizes, while the previous SOTA method OC-GAFN fails to maintain good performance as the problem complexity increases.
4 RELATED WORK
Generative Flow Networks (GFlowNets). Recently, there have been a number of efforts applying
GFlowNets to different important cases, e.g., biological sequence design (Jain et al. 2022), molecule
generation (Bengio et al., (02T). combinatorial optimization (Zhang et al.] 2023a] 2024). Bayesian structure learning (Deleu et al. 2022). There have also been many works investigating how to improve the training of GFlowNets, enabling them to achieve more efficient credit assignment (Pan) et al., 2023b), better exploration (Pan et al. 2023c Lau et al. 2024). or more effective learning objectives (Bengio al. 2023] Madan al., 2023a) that can better handle computational complex- ity (Bengio et al. 2021] and generalize to stochastic environments (Pan et al.] [203d]] Zhhng et al.) 2023b). GC-GFlowNett learn flows and policies conditioning on outcomes (goals) for reaching any targeted outcomes (Pan et al. 2023a). However, little attention has been given to this topic, leaving this promising direction largely unexplored. Mcanwhile, it is challenging to train conl-conditioned policies due to sparse rewards. Our work not only provides a formal definition of GC-GFlowNets but also proposes novel method called retrospective backward synthesis to significantly improve their training efficiency and success rates.
Goal-Conditioned Reinforcement Learning, Our formulation of goal-conditioned GFlowNets is heavily inspired by the works of goal-conditioned RL. Standard Reinforcement Learning (RL) only requires the agent to finish one specific task defined by the reward function (Schaul et al. 2015)). while goal-conditioned RL trains an agent achieve arbitrary goals as the task specifies (Andrychowiez jet al. (017). Coul-Conditioned RL augments the observation with an additional goal that the agent required to achieve (Liu et al. (022). The reward function is usually defined as binary bonus of reaching the goal. To overcome the challenge of the sparsity of reward function, prior work in goal-conditioned RT. has introduced algorithms based on a variety of techniques. such as hindsight relabeling (Andrychowicz et al., 2017 7, Fang et al. 2018 Yang et al., 2022, Fang et al./ 2019 Ding
et al.
2019), contrastive learning (Eysenbach et al. 2020 2022). state-occupancy matching ( (Durug kar al. 2021: Ma et al., (022b) and hicrarchical sub-goal planning (Chane-Sane et al., 2021 Kim
et al.
2021 Nasiriany et al.] 2019). Our work is closely related to hindsight relabeling, denoted as al. 2017), which relabels any experience with some commanded goal to the goal that was actually ac achieved in order to learn from failures. HER can generate non-negative rewards to alleviate the negative sparse reward problem, even if the agent did not complete the task. However, the agent using HER still suffers from low sample efficiency on large-scale problems due to its limitation in operating only on the observed trajectories, while our method can imagine new trajectories with positive rewards for policy training.
HER (Andrychowicz
5 EXPERIMENTS
In this section, we conduct extensive experiments to investigate our Retrospective Backward Synthesis (RBS) method to answer the following key questions: i) How does RBS-GFN compare against previous baselines in terms of sample efficiency and success rates? ii) Can RBS-GFN scale to complex and nigh-dimensional environments? ii) Can RBS-GFN effectively generalize to unseen goals and unseen environments? iv) Is RBS-GFN general and can be built upon different GFlowNets training objectives? v) What are the effects of important components in RBS-GFN?

---

Published as a conference paper at ICLR 2025
5.1 GRIDWORLD
We first conduct a series of experiments based on the GridWorld environment (Bengio et al., 2021), in which the model learns to achieve any given goals starting in a H x H grid. Specifically, we investigate mazes with increasing horizons H (32, 64, and 128), respectively, resulting in different levels of difficulty categorized small, medium, and large.
We compare our proposed RBS- GFN approach with the following state-of-the-art baselines. (i) GFN w/ HER (Andrychowicz et al. 2017) is a GC-GFlowNets that relabels the negative reward in failed trajectory with positive reward. (ii) OC-GAFN (Pan al. (023a) is recent method that utilizes contrastive learning to complement successful experiences, and employs a trained Generative Augmented Flow Network (GAFN; Pan et al. (2023c)
exploratory component to generate diverse outcomes y, which are subsequently provided to sample goal-conditioned trajectories. (iii) DQN w/ HER leverages both deep Q-learning algorithm (Mnih et al. 2013 Jang et al. 2019)) and HER technique (Andrychowicz et al., 2017) to learn a near-optimal policy. This baseline is used to ablate the effects of
training compared with RL-style methods. To ensure fairness, each baseline has the same model architecture and training steps as RBS-GEN, and we follow the experimental setup for hyperparameters as in (Pan et al. (023a). We run each algorithm with three different seeds and report their performance mean and standard deviation, more detailed description of the experimental setup can be found in Appendix C
5.1.1 PERFORMANCE COMPARISON
The success rates for different methods for increasing sizes of the GridWorld environment (including small, medium, and large) are summarized in Fig. 3 We obtain the following observations based on the results. (i) CFN-basee conditioned methods (DQN w/ HER), as the latter can easily get trapped in local optima due to its greedy policy. The results validate the promise of training goal-conditioned policies using GFlowNets and pave the way for further advancements in goal-conditioned learning with GFlowNets. (ii) Moreover, our proposed RBS-GFN method significantly outperforms GFN w/ HER and is stronger than the OC-GAFN method in terms of sample efficiency and outcome-reaching ability, particularly in larger spaces. In contrast, the performance of GFN w/ HER deteriorates as the complexity of the environment increases, which highlights its limitations in handling large state spaces. (iii) The inferior performance of our method without RBS highlights the significance of our proposed approach. We remark that the superior performance of RBS is attributed to its enhancement of training data with higher quality and diversity.
conditindt approaches consistently outperform RL-based goal-
Hoe Leon 32
Harizon 64
Horizon 128
100
100
Surcess Bate
Scccss Rate
Succoss
Success
8
20
Ait-a Domi
Stops
1M
Stops
Staps
Figure 4: Results of RBS-GFN (SubTB).
Figure 3: Performance comparison in GridWorld. Left: Small. Middle: Medium, Right: Large.
5.1.2 SPECIAL CASE: OFFLINE GC-GFLOWWNESS
Given the potential of learning general goal-reaching policies solely from given offline datasets Ma et al. (2022a). we investigate the offline goal-conditioned scenario where GC-GFlowNets learn from fixed offline data without interacting with the environments. As result, all the baselines are restricted in limited training datasets, while RBS- GFN can synthesize a number of new trajectories to enhance the learning process. We compare our method with the two strongest baselines, OC-GAFN and vanilla goal-conditioned GFN, and evaluate them on three different sizes of GridWorld. As demonstrated in Fig. [5] our method achieves nearly 100% success rates across all scenarios, while the performance of the baselines declines significantly as the problem size increases. The experimental details and learning curves can be found in Appendix C.3
5.1.3 GENERALIZATION
7

---

Published as a conference paper at ICLR 2025
BRAG-GGN (Our)) BOC-GAFN
We evaluate the generalization ability of our RBS-GFN
method to unseen goals and environments, which is important for real-world applications where the agent can encounter novel situations. To evaluate ability to generalize to unseen goals, we mask n goals from various locations in the map and test the success rates of reaching these unseen goals after the train- ing process, as illustrated in Fig. ([)) (n 20). As shown in Fig. (()), RBS-GEN obtains almost 100% success rate, demonstrating its capacity to effectively determine the required actions to reach novel goals. Moreover, it outperforms strongest baseline OC-GAFN by an approximately 15% suc-
Success (%)
small
medium
large
Figure 5: Success rates Grid- World tasks with different sizes.
cess rate. We further investigate its generalization capability unseen environments. We introduce
(a) Testing map
(b) Success rate
(a) DQN
(b) RBS-GFN
(c) Success rate
Figure 6: (a) Visualization of GridWorld. the start point, and 1/2 is the unseen goal. (b) The average success rate of reach- ing these unseen goals for 100 trials per goal.
Figure 7: (a) DQN fails to generalize unseen maps with obstacles. (b) RBS-GFN can find diverse trajec-
tories. (c) The average success rate over 200 trials of reaching the goal
unseen obstacles during the testing phase following (Kumar al. 2020), which creates novel environ- ments that the agent has not encountered during training. As shown in Fig. 7c), RBS-GFN maintains success rate of almost 100%, while OC-GFN obtains success rate of 9% and the RL-based method DQN completely fails. More unseen maps and corresponding results are provided in Appendix [ The superior performance of RBS-GFN in unseen environments efficiently discover diverse paths to reach the goal as shown Fig. bb). Although OC-GAFN also has the potential to discover diver paths, its performance is limited by the available training budget (i.e., fewer pre-training iterations). On the other hand, DQN is limited to discovering single trajectory to reach the goal as shown in Fig. 7a), making it highly susceptible to failure when the learned path is blocked by unseen obstacles.
be attributed to its ability to
5.1.4 VERSATILITY
In this section, demonstrate the generality of our approach by integrating it with another recent GFlowNets method based on SubTB (Madan et al., (0236), whose learning objective is based on LSubTB introduced in Eq. (2). We evaluate the goal-reaching performance of RBS-GFN (SubTB) terms of the success rate in the GridWord task (with H 10).
As shown in Fig RBS-GEN can also be successfully built upon SubTB with a success rate of 100%, and achieves consistent performance gains.
5.1.5 ABLATION STUDY
In this section, conduct in-depth analysis of the key components of RBS-GFN better under- stand their effect with a focus on two critical techniques, including backward policy regularization and age-based sampling, while defer the discussion of intensified reward feedback, which is essential for scaling up to high-dimensional problems Appendix A.I
The backward sampling policy Pn plays an important role in synthesizing helpful trajectories for training GC-GFloo.eets. We evaluate the effect of different choices of PB. including the regularized PB (based on Eq. (6)), a learned backward policy without constraints, and fixed and uniform one. We compute the entropy of the forward policy PF measure the ability to generate diverse trajectories of GC-GFlow Nets. We further visualize the trajectory distribution in the replay buffer for different choices of P/ with t-SNE (Van der Maaten & Hinton 2008) in Fig. (b).
8

---

Published as a conference paper at ICLR 2025
As shown in Fig 8 a). the proposed regularized Pg converges faster than other variants in terms of success rate while maintaining a satisfactory level of entropy. Furthermore, Fig B illustrates that the synthesized trajectory distribution of regularized Pn and uniform Pn cover wide range, which effectively compensates for the limited coverage of the original data distribution, while learned Pn struggles to synthesis trajectories that significantly differs from the original data distribution.
Fig. I0 demonstrates the effect of our proposed age-based sampling technique (with horizon 128), which highlights its importance in improving learning efficiency and stability, as the model struggles to efficiently achieve high success rate without age-based sampling (which fails fully utilize and learn from newly-generater samples).
has
Torrec
(a) Quantitative results
(b) t-SNE visualization
Figure 8: Comparison results of using different PB to synthesize experiences.
5.2 BIT SEQUENCE GENERATION
In this section, we investigate the performance RBS-GFN in the bit sequence generation task (Malkin al. 2022). Unlike previous approaches that generate these sequences in a left-to-right manner (Malkin al. 2022; Madan al. 2023a), we adopt non-autoregressive prepend/append Markov decision process fol fowlonng Shen et al. (2023), which is more challenging task compared to the one studied Pan al. (2023a). The action space includes pretending or appending k-bit word from the vocabulary V to the current state, which increases the difficulty of the task (as the underlying structure of the problem is directed acyclic graph rather than a simple tree (Malkin et al.) 2022)). We consider bit sequence generation with small, medium, and large sizes with increasing lengths and vocabulary sizes following Pan et (2023))
medium
large
Success Rate
Success Rale
Rate
80
Succecs
Stops
0.2
Stops
Stops
320
Figure 10: Performance compared with RBS-GFN- wo-AAeSample.
Figure 9: Performance comparison in bit sequence generation.
As shown in Fig. 9 even the strongest OC-GAFN method struggles to learn efficiently given a limited training budget, while all other baselines completely fail. In contrast, RBS-GFN achieves high success rates of approximately 100% with fast convergence across different scales of the tasks. It is worth noting that RBS-GFN without either age-based sampling (denoted as intensified reward feedback (see detailed discussions Appendix A. both fail to generalize to more complex tasks, including medium and large, demonstrating the importance of our proposed techniques in enabling RBS-GFN to efficiently learn across various levels of complexity.
5.3 TF BIND GENERATION
In this section, we study a more practical task of generating DNA sequences with high binding activity with targeted transcription factors (Jain et al., (022). Similar to the bit sequence generation task, the agent prepends or appends a symbol from the vocabulary the current state at each step. As shown in Fig. a), RBS-GFN archives success rate of 100% and learns much more efficiently thanks to its retrospective backward synthesis mechanism, and outperforms other baselines. which illustrates its effectiveness for DNA sequence generation. We provide additional experimental analysis about this task in Appendix D

---

Published as a conference paper at ICLR 2025
5.4 AMP GENERATION
In this section, we study the antimicrobial pep- tides (AMP) (Jain et al. 2022) generation task for investigating the scalability of our proposed method. The task involves generating se- quence with a length of from a vocabulary with a size of 20. We follow the same exper- imental setup as $5.3 considering an ac- tion space with prepend and append operations
Sucoeer Sate
Steas
following Shen et al. (023). The state space contains 20 possible AMP sequences, which poses significant challenge for efficient explo- sequence generation tasks. ration and optimization. Moreover, the task is extremely difficult due to the vast sequence space and complex structure-function relationships compared to the case studied in Pan et al. (2023a).
(a) TF Bind.
(b) AMP.
Figure 11: Succee rates on the TF Bind and AMP
To tackle this large-scale problem, we employ the goal decomposition method proposed in $3.2 (see details in Appendix B By breaking down the target goal into simpler sub sub-goals, i.e., generating shorter sub-sequence, we can effectively reduce the complexity of the search space, enabling more efficient learning. We refer to this approach as Hier-RBS-GFN, As demonstrated in Fig. b)), Bier-RBB-GEN significantly improves the learning efficiency and outperforms all baseline methods, which shows the scalability of our approach for tackling complex tasks with vast search spaces.
5.5 APPLICATION: DOWNSTREAM FINETUNING
A notable advantage of GC-GFlowNett is that the pre-trained policy can be leveraged to handle down- stream tasks with unseen rewards, unlike the typical fine-tuning process of reinforcement learning as they generally learn reward-maxe soaedddmximizzaiion policies that may discard valuable information (Pan et al.) 2023a). this section, we study the application of GC- GC-GFlowNet and validate its effectiveness for adapting to downstream bit sequence generation tasks with unseen rewards in different scales. We generally follow the experimental design Pan al. (2023a), while we train GC-GFlowNets with fewer pre-training budgets, which better illustrates the efficiency of our method in the pre-training stage. From the results shown in Fig. 12 find that RBS-GFN outperforms OC-GAFN by large margin, as a more efficient and effective pre-trained goal-reaching strategy can significantly contribute the fine-tuning process, while OC-GAFN struggles to efficiently discover modes given limited pre- pre-training budgets in this challenging goal-conditioned training stage.
RB5-GFN (Ours)
Number of modes
25
OC-GAFN
Number modes
Number of modes
Steps
Steps
Steps
165
Figure 12: Fine-tuning GC-GFlowNets on the downstream sequence generation task with different scales. We report the number of modes discovered during training. The number of modes is calculated using a oppere--xxclusion procedure. candidate is added the list of modes if it is above a certain reward threshold and is further away than some distance threshold from all other modes.
6 CONCLUSION
In this paper, we address the critical challenges of realizing goal-directed behavior and learning in GFlowNets. To overcome the training challenge due to extremely sparse rewards, we propose novel method called Retrospective Backward Synthesis, which significantly improves the training of goal condiiondd GFlowNets by synthesizing backward trajectories. Our extensive experiments demonstrate tfte--of-hhe-ar performance in terms of both success rate and generalization ability, which outperforms strong baselines. For future work, is promising further improve our method, e.g., sampling method considering alternative priorities (Sujit et al. 2023). More discussions about work are provided in Appendix E
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work of Haoran He and Ling Pan is supported by National Natural Science Foundation of China 62406266. Huazhe Xu is supported by Tsinghua dushi program. We also thank the anonymous reviewers for their valuable suggestions.
ETHICS STATEMENT
This paper presents work whose goal advance the field of Machine Learning. Specifically, we propose a novel method called RBS to enhance the learning of GC-GFlowNets. Since this method is easy to reproduce (as we will release our code soon) and exhibits the SOTA performance, it encourages future research to further advance this field. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
REPRODUCIBILITY STATEMENT
All details of our experiments can be found in Appendix C which includes descriptions of the tasks, experimental setup, network architecture, and hyperparameters. The proof of intensified reward feedback is referred to in Appendix A The code will be open- open-sourced upon publication of this work.
REFERENCES
Ursula Addison. Sluman-inspire goal reasoning implementations: survey. Cognitive Systems Research, 83:101181, 2024.
Christophe Andrieu, Nando De Freitas, Amaud Doucet, and Michael I Jordan. An introduction to meme for machine learning. Machine learning, 40:5543, 2003.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Lazar Atanackovic, Alexander Tong, Bo Wang. Leo Lee, Yoshua Bengio, and Jason S Hartford. Dyngfn: Towards bayesian inference of gene regulatory networks with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: wttp:///www.eeee.e.e.o. net Eo rum d
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representa- tions. In International conference on machine learning, pp. 560. PMLR, 2013.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research. 24(210):1-55, 2023.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. International Conference on Machine Learning, pp. 1430-1444 1440. PMLR. 2021.
Henry Charlesworth and Giovanni Montana. Plangan: Model-based planning with sparse rewards and multiple goals. Advances in Neural Information Processing Systems, 33:8532-8542, 2020.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Tristan Deleu, Mizu Nishikawa- Toomey, Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint bayesian inference of graphical structure and parameters with a single generative flow network. Advances in Neural Information Processing Systems, 36, 2024a.

---

Published as a conference paper at ICLR 2025
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete proba- bilistic inference as control multi-path environments. arXiv preprint urXiv:2402. 10309, 2024b.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation
learning. Advances in neural information processing systems, 32, 2019.
Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances Neural Information Processing Systems, 34:8622-866 2021.
Ashley D Edwards, Laura Downs, and James C Davidson. bawward reinforcement learning. arXiv preprint 1803. 10227, 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint arXiv:201 1.08909, 2020.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-356200, 2022.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2018.
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculun-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019.
Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for efficient reinforcement learning. In International Conference on Learning Representations, 2019. URL https::/
net httee//ooommmmddyysssseee
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint 1oXiv:2301.04100 2023.
W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
Haoran He, Emmanuel Bengio. Qingpeng Cai, and Ling Pan. Rectifying reinforcement learning for reward matching. arXiv preprint arXiv:2406.00 02200,,, 2024.
Mare Höftmann, Jan Robine, and Stefan Harmeling. Backward learning for goal-conditioned policies. In NeurlPS 2023 Workshop on Goal-Condition Reinforcement Learning, 2023.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga. Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-fil implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022. URL http://
httppppsss/333/11a11aa /tt////3//1113/// html
Moksh Jain, Emmanuel Bengio, Alex Hernandez- Garcia, Jarrid Rector- Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786-9801. PMLR, 2022.
Rector-Brooks,
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garcia, and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2(3):577-57, 2023.
Beakcheol Jang, Myconghwi Kim, Gaspard Harerimana. and Jong Wook Kim. Q-learning algorithms: A comprehensive classification and applications. IEEE Access, 7:133533- 2019, doi: 10.1 1.11109//CESS22000
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchical reinforcement learning. Advances in neural information processing systems, 34:28336-28349 2021.
12

---

Published as a conference paper at ICLR 2025
Sungyoon Kim, Yunseon Choi, Daiki E Matsunaga, and Kee- Kee-Eung Kim. Stitching sub -abberajectories with conditional diffusion model for goal-conditioned offline rl. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 13160-13167, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Teun Kloek and Herman K. van Dijk. Bayesian estimates of equation system parameters, an application of integration by monte carlo. Econometrica, 46:1-19, 1976. http::/t::/pp
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need:
Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 18:8198-8222 2020.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, pp. 5618-5627. PMLR, 2020.
Elaine Lau, Stephen Zhewen Lu, Ling Pan, Doina Precup, and Emmanuel Bengio. Qgfn: Controllable greediness with action values. arXiv preprint arXiv:2402.05234. 2024.
Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35:995-1007, 2022.
Minghwan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08225 2022.
Ronny Lorenz, Stephan H. Bernhart, Christian Höner Siederdissen, Hakim Tafer, Christoph Flamm, Peter F. Stadler, and Ivo L. Hofacker. Viennarna package 2.0. Algorithms for Molecular Bi- ology: AMB, 6:26 26, 2011. URL httt:p::/aai.
1305927]
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in neural information processing systems, 15:310-323, 2022a.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. arXiv preprint a3X2v32206.03023, 2022b.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Kanika Madan, Jarrid Rector-Brocd Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica. Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 024467-24483 PMLR, 23 Jul 2023a.
Kanika Madan, Jarrid Rector-Brooke Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference Machine Learning, pp. 23467-23483. PMLR, 2023b.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlownets. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http:::s///peeeevvv
n forum? 5bdd=btttwwwwl
Nicholas Metropolis, Arianna Rosenbluth, Marshall N Rosenbluth, Augusta Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 1166:1087 1953.
13

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini arXiv:/312.5602, 2013.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies, Advances in Neural Information Processing Systems, 32, 2019.
Tim Niemueller, Till Hofmann, and Gerhard Lakemeyer. Goal reasoning in the clips executive for integrated planning and execution. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 754-763, 2019.
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-training and tne-tuning generative flow networks, 2023a.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference on Machine Learning, pp. PMLR, 2023b.
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. In The Eleventh International Conference on Learning Representations, 2023c. URL htatt:::///penevviiii
net inforummidd-sn -
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. In Uncertainty in Artificial Intelligence, pp. 1628-1638. PMLR, 2023d.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal-conditioned rl with latent states actions. Advances in Neural Information Processing Systems, 36, 2024.
Julien Roy, Pierre-Luc Bacon, Christopher Pal, and Emmanuel Bengio. Goal-conditioned gflownets for controllable multi-objective molecular design. arXiv preprint vrXiv:230.004620, 2023.
Russ R Salakhutdinov. Learning in markov random fields using tempered transitions. Advances in neural information processing systems, 22, 2009.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR (Poster), 2016.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International Conference Machine Learning, pp. 30956-30975. PMLR, 2023.
Shivakanth Sujit, Somjit Nath, Pedro Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 36:23237-23255 2023.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry P Vetrov. Generative flow networks as entropy-regullrizeed rl. In International Conference on Artificial Intelligence and Statistics, pp- 4213 4221. PMLR, 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (111),2 2008
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning, arXiv preprint arXiv:1806.0090 09605, 2018
Haoran Wang, Zeshen Tang, Yaoru Sun, Fang Wang, Siyu Zhang, and Yeming Chen. Guided cooperation in hierarchical reinforcement learning via model-based rollout. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Jianhao Wang, Wenzhe Li. Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 22292222224333 2021.
14

---

Published as a conference paper at ICLR 2025
Mianchu Wang, Rui Yang, Xi Chen, and Meng Fang. GOPlan: Goal-conditioned offline reinforcement learning by planning with learned models. In NewrIPS 2023 Workshop on Goal- Souul-Coditioonedd Rein- forcement Learning, 2023. URL https : wptp//vwwe......oomomem.. net/forum? d=qU6t ZmppN7
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1r0.v::505.00833 2015.
Rui Yang, Meng Fang, Lei Han, Yali Du, Feng Luo, and Xiu Li, MHER: Model-based hindsight experience replay. In Deep RI. Workshop NeurlPS 2021, 2021. URL https:::/ppeereeeeeeee ew
net/forum? 3dd=msxx---------------
jhnILM
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint 244404788 2022.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL httpsss//ppeeerreeee.eeette///es net wwwp::/ee.ttoottttmooeee..m.mmme
6sTvGaf
David W Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. arXiv preprint arXiv:2302.05446, 2023a.
Dinghuai Zhang, Ling Pan, Ricky TQ Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows. arXiv preprint arXiv:2302.055793 2023b.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A INTENSIFIED REWARD FEEDBACK
We re-write the loss function of GC-GFlowNets in the case s' is terminal as follows:
(s.ppsssssss. LGC-GFN (log Po(slu)P(($$ ss 0) - log [CR(x,, R) Pb 0)))
which can be degenerated Eq. ( when we set C 1. In practice, we set C to a large value to facilitate effective reward propagation. Below, we demonstrate that large C' scales the gradient with respect PB without affecting Pp or F. We show that
ÖLGC-GFX = 2 x Z
Flog Z
(8)
00
00
where
2 log Pa + log Pp (1(8'18,, y. log (CR(2,,)
(9)
log Z 00
Falall
a (elll ..0))
Fo(ely)
88
(10)
Since the scaling coefficient C' does not depend on the model parameters 9, the derivative w.r.t.
simplifies as follows:
a
CR(2, v)
(11)
00
00
Substituting this into Eq. 10 we obtain:
log 00
(Fly) 00
(s.90))).) 00
(Pp(s .y.0)
(aly)
(a
00
(12)
where C' is eliminated. Consequently, C only appears in the last term of Z. Therefore it only affects PB. leaving gradients w.r.t PF and F9 unchanged.
A.1 EMPIRICAL VALIDATION
To validate the effects of our proposed technique, i.e., intensified reward feedback, we conduct the ablation study in the bit sequence generation tasks, which are more complex and high- high-dimensional than the GridWorld tasks. In practice, we set C le for small task, C' = 2 for medium task, and C' = 10 for large task. From the results shown in Fig 13 we observe that RBS-GFT completely fails in the task without intensified reward feedback, obtaining only 0% success rate.
small
medium
large
100
100
80
Success Rate
Success Rate
50
60
Success
40
20
RBS GPN (OurS)
RB5 2005-GFN------000
0.4
0.0
0.2
0.6
0.0
0.2
0.4
0.6
0.8
1.0
Steps
le5
Steps
1e5
Steps
1e5
Figure 13: Performance comparison with RBS-GEN without using intensified reward feedback.
16

---

Published as a conference paper at ICLR 2025
B HIERARCHICAL GOAL DECOMPOSITION
It is still challenging to tackle problems with extremely large-scale state spaces, and even our method can fail in these scenarios. To address this problem, we propose a hierarchical method to decompose the task into several low-level tasks that are easier to complete. Leveraging the consistent sequential structure in compositional GFlowNets tasks,
set sub- gool-goass manually, eliminating the need to
learn a sub-goal generation policy additionally (Chane-Samie et al.] 2002:] Kim et al. [021]] Nasiriany al. 2019). After training sub-level policies, all the generated sub-goals can be combined together to obtain the final goal.
Considering a sequence generation problem (Jin et al., 2022) as an example, wherein an agent is tasked with generating sequence length from a vocabulary of size 1/, we can decompose this task into & sub-level tasks. Consequently, we can train is models, each capable of generating sequence of length l/k. Subsequently, the k generated sub-sequences can be concatenated to form sequence of length /.
C EXPERIMENTAL. SETUP
We build our implementation for all baselines and environments upon publicly available open-source repositories The code will be open-sourced upon publication of the work.
2-dimensional hypercube 1) where the model learns to achieve any given goals (outcomes) starting from fixed initial state (0. 0). We examine grids with H set to 32, 64, and 128, respectively, resulting in different levels of difficulty categorized as small, medium, and large. The agent receives a positive reward of only if it reaches the desired goal state. We use the Adam (Kingma & Ba 2014) optimizer with a learning rate of 1e- for 2e training steps.
GridWorld. The GridWorld (Bengio 2021 is conceptualized
with side length H {(s) { 1.
This task requires the model generate sequences of length n by pretending or appending a k-bit word to the current state. We consider k = 2, n 40 for small task, n, 60 for medium task, and & n 100 for large one. We use the Adam (Kingma Ba 2014 optimizer with a learning rate of 5e -4 for le training steps.
Bit Sequence Generation.
TF Bind Generation. Similar to the bit sequence generation task, the agent prepends or appends a symbol from the vocabulary with a size of 4 to the current state at each step generate a sequence of length 8. We use the Adam (Kingma Ba 2014) optimizer with a learning rate of 4 for 5e training steps.
AMP Generation. This biological task requires the agent to generate antimicrobial peptides (AMP) with lengths of 50 (Jain al. 2022) from vocabulary with size of 20. For both RBS-GFN, Hier- RBS-GFN and all the baselines, we use the Adam (Kingma & Ba 2014) optimizer with a learning rate of -4 for 1e training steps.
C.1 IMPLEMENTATION DETAILS
We describe the implementation details of our method as follows:
We use an MLP network that consists of hidden layers with 2048 hidden units and ReLU activation (uu et 2015).
The trajectories are sampled from parallel of 16 rollouts in the environment each training step.
We set the replay buffer size as le6 and use a batch size of 128 for sampling data and computing loss function.
We combine the current state and goal state together as the input of our model. The input is transformed as one-hot embedding followed by our MLP model.
We run all the experiments this paper on RTX 3090 machine.
17

---

Published as a conference paper at ICLR 2025
C.2 BASELINES
We describe the implementation details of the baselines we use throughout this paper as follows:
The only difference between GFN w/ HER and our method is that GFN w/ HER leverages HER (Andrychowicz al. 2017 technique to enhance training experiences, while we utilize our proposed retrospective backward synthesis to augment the data with new reverse trajectories.
For OC-GAFN, follow the same experimental setup described in (Pan et al., 2023a). This method not only leverages goal relabeling (Andrychowicz et al.) 2017) but also uses GAFN (Pan et al. 2023c) to generate diverse outcomes y. which are subsequently provided to sample outcome- conditioned trajectories. OC-GAFN requires training an additional GAFN model, which would be computationally expensive. At each training step, OC-GAFN takes 2 times gradient update, where one is for the negative samples and the other is for the relabeled samples. OC-GAFN does not maintain a replay buffer and uses newly sampled data to train its model.
Following the implementation in (Andrychowicz et al.) 2017). DQN w/ HER leverages both deep Q-learning algorithm (Mnih et al., [013] Jang et al. 2019) and HER technique (Andrychowicz et al. 2017 learn near-optimal policy.
SAC w/ HER leverages both SAC algorithm (Haanoja et al. 2018) and HER technique (Andrychowicz al.] 2017) learn a near-optimal entropy-regulane policy. We follow the
hyperparameters used in (Huang et al. (0022).
C.3 DETAILS OF OFFLINE EXPERIMENTS
For offline dataset collection, store the samples recorded in the replay buffer of vanilla GFN during training until convergence. This dataset includes 8,016 trajectories with varying levels of performance. Given that the training of GelowNets is off-policy, we reuse the learning objective in Eq. 66 without modification. The implementation details our offline algorithm (RBS-GFN) and the baselines are illustrated below:
(offline-) RBS-GFN (ours): Similar to the online version of RBS- RBS-GFN, the dataset is augmented by the reverse trajectories collected by PB at each training step.
(offline-) OC-GAFN: Unlike CBS-GFN, OC-GAFN fails generate new trajectories. Instead, it only augments the dataset by relabeling the outcomes of failed trajectories with their actual terminal states.
(offline-) GFN: This baseline utilizes the original dataset without any additional data generation or augmentation processes.
To ensure fair comparison, all other settings, including the network architecture and hyperparameters, are kept the same as those used in the online implementations. Offiine Horizon-16
Offiine -
Offline 32
1000
100
100
Success Rate
Succuss Bate
80
Succiss Rate
80
50
Ros cm 10uml
oC GA/h
00:5Am
0.5
1.3
Steps
Steps
Steps
164
small
(b) medium
(c)
large
Figure 14: Learning curves of offline experiments on three different scales of GridWorld. Success rates are averaged over three random seeds.
D ADDITIONAL EXPERIMENTAL RESULTS
We provide more experimental results that demonstrate the superior ability of our method.
D.1 COMPUTATION OVERHEAD
RBS-GEN is efficient since we only need to synthesize a single r' using backward policy PB from the goal, and rollout minibatch of data at each training step. It is also worth noting that the strongest
18

---

Published as a conference paper at ICLR 2025
100
100
100
Success Rabe
Success Rabe
80
60
Success
40
23
J-0.500
GIN CC-AANN
3 00009
0c 0MT
Steps
Steps
(a)
(b)
(c)
Figure 15: Additional experimental results. (a) Success rates on GridWorld with horizon of 32. The x-axis corresponds to the wall clock time. (b) Ablation study with different values of the decay hyperparameter 3. RBS-GFN is robust different 3. (c) Success rates RNA generation (Pan
et al. 2023a). RBS-GFN consistently outperforms the strongest baseline, OC-GAFN, on the task.
baseline OC-GAFN requires training two GFN models (i.e., an unconditioned GAFN model and conl-cond GFN model), while RBS-GFN only needs train a single noal-conditioned GFN agent, which largely reduces training compute requirements. We quantify the wall-clock time and corresponding achieved success rates on the GridWorld task with horizon of 32. From the results shown in Fig. 15)a), we find that RBS-GEN achieves a significantly higher success rate in less time compared to the previous strongest method, OC-GAFN.
D.2 RESULTS ON THE RNA GENERATION TASK
We further evaluate the performance of RBS-GFN on the RNA generation task (Lorenz et al. [001]] which involves constructing RNA sequences following Pan et al. (2023a). We compare method with the strongest baseline, OC-GAFN. From the results shown in Fig. [5[c), we observe that RBS-GFN consistently achieves the best performance.
D.3 ROBUSTNESS TO HYPERPARAMETER
Regarding the reward intensification technique, the scaling coefficient is the only task-dependent hyperparameter that requires tuning, it depends on the nature of specific tasks and also accom- modates different horizons. However, this can be easily done through standard techniques like grid search (similar to tuning conventional hyperparameters such as the learning rate (Malkin et al. 2022 Jain al. 2022)). As for backward policy regularization. we demonstrate that the RBS-GFN exhibits robust performance across wide range of values for the decay hyperparameter 3, consistently achieving 100% success rate with high sample efficiency, shown in Fig. b)).
D.4 COMPARISON WITH MODEL-BASED COAL-CONDITIONED RL
To further demonstrate the effectiveness of our proposed RBS-GFN, we carefully design a sophis- ticated model-based GC-RL method following MHER (Yang et al.) 2021) and Dreamerv3 (Hafner et al. 2023) for additional comparison. Specifically, we consider actor-critic learning introduced in Deamerv3, and employ the model-based imagination technique introduced in MHER to augment the training data. It is noteworthy that while previous model-basee methods like MHER augment datasets by forward imagination based on HER, RBS introduces novel way to sample backward trajectories to increase both data quality and diversity. The results shown in Fig. (T(a) demonstrate that RBS-GFN consistently outperforms this GC-RI method by large margin.
D.5 PERFORMANCE ON STOCHASTIC EXVIRONMENTS
To further demonstrate that RBS-GFN can generalize to stochastic dynamics. we build it upon stochastic GFN (Pan et al. 2023d and consider randomness in the environment following Machado al. (2018). Specifically,
environment transitions according the selected action with probability Ot, while with probability a the environment executes a randomly chosen action. Here we take the grid environments for evaluation and set 0.01 make the environments stochastic. The
19

---

Published as a conference paper at ICLR 2025
experimental results shown in Fig. 16 demonstrate that RBS- CBS-GFF can also generalize to stochastic environments and achieve higher success rates compared with the strongest baseline OC-GAFN. Given the inherent randomness in the environments, which can significantly influence goal-reaching strategies, is reasonable for the overall performance to decline compared to that in deterministic environments.
D.6 ROBUSTNESS TO DIFFERENT REWARD STRUCTURES
To demonstrate that RBS-GFN is robust to different reward structures, we add additional experiments on the GridWorld tasks, where the agent receives dense rewards (which corresponds an easier setting compared to the sparse reward case we studied in the main paper). Concretely, the reward is defined as the Manhattan distance between the current state and the desired goal. The results shown in Fig b)]b) demonstrate that RBS-GEN achieves even further performance improvements in the dense rewards setting.
Horizon
Hortzon
Horizon
Horizon
Succerss Rate
Successs
0/7
0.5
Stops
1.0
17
164
0.5
Steps
2.0
3.5
(a)
(b)
Hortzon 32
Hortzon
Stochastic (H-32)
Stochastic (H- 64)
100
100
Success Rate
Success Rate
80
Success Rata
Success Rate
10am
105-CFM (0LB)
0.2
0.5
1.9 Steps
0.0
3.5
3.2
Steps
164
Steps
Stsps
164
(c)
(d)
Figure 16: Additional experimental results the GridWorld benchmark. (a) Success rates compared with an advanced model-based goal-conditioned RL method. (b) RBS-GFN can generalize to the dense rewards structure and gain further performance improvement. (c) RBS-GFN with the age- based sampling technique, outperforms its variant with PER. (d) Success rates on stochastic GridWorld environments (Pan et al.) 2023d) compared with OC-GAFN (across 3 random seeds). RBS-GFN consistently outperforms the strongest baseline.
D.7 COMPARISON WITH PRIORITIZED EXPERIENCE REPLAY (PER)
Specifically, we follow the standard PER setting Schaul et al. 2016) with a = 0.7 and , = 0.4, and adopt the GC-GFlowNet loss instead of TD error priority to suit our scenario. We utilize the open-sourced codes in pothttss::. /ttpt/gtttt////e com/ Hwwhh wwwphhh/pwrrrrrieeeeeee eeperriieeeeeeeeeee replay to implement it. From the results in Fig c). we observe that our age-based sampling outperforms PER by large margin that learns more efficiently. We hypothesize that this because the GC-GFlowNet loss is not stable; for some data samples, the loss would remain irreducible and stay high throughout the training process. Consequently, PER restricts training data coverage. leading to reduced overall performance. In contrast, our age-based sampling technique ensures that experiences leveraged more thoroughly.
D.8 GENERALIZATION AND VERSATILITY
Generalization. We investigate the generalization ability of our method in more unseen maps. We show the three designed maps that consider different locations of goals and obstacles in Fig. [8]a-c). We also compare method with baselines in terms of the success rate these unseen maps. The
20

---

Published as a conference paper at ICLR 2025
experimental results shown in Fig. 18 demonstrate proposed RBS method significantly enhances the generalization ability of GC-GFlowNets.
Versatility To demonstrate that method also be applied to SubTB (Madan al. 2023b) learning objective, we provide additional experimental results on the TF Bind sequence generation task. We observe that both RBS-GEN trained with DB (denoted as RBS- RBS-GFN(DB) and BBS-GGDDN trained with SubTB (denoted as RBS-GFN GFN (SubTB)) achieve a 100% success rate. notably, our method RBS gets consistent performance improvement in this task, while RBS-GFN-Wo-RDS almost fails to succeed.
100
100
80
Success Rate
Success Rate
40
40
RBS-CFN [Dun)
GAFN
GIN (wi HER)
DGN (az HER)
SAC 1ax HER)
Steps
1c3
Steps
1e3
(a)
(b)
Figure 17: (a) Performance SAC in TF Bind tasks. (b) Performance of GFN with SubTB TF Bind tasks.
D.9 COMPARISON WITH SAC
We investigate the performance of SAC, known as soft actor-critic algorithm (Haarnoja et al. 2018) which is an entropy-regularized RL method rather than standard arg DQN. We evaluate the performance of SAC in the TF Bind sequence generation task. As the action space this task is discrete, we implement discrete SAC algorithm based the codes from CleanRL (Huang et al.) 2022). From the results shown in Fig 17]a) a), we observe that SAC (w/ HER) even performs worse than the DQN algorithm. We hypothesize that it is because SAC prefers new states to maximize the entropy rather than high-reward states to complete the task. Although HER can provide abundant of successful experiences, it is still not enough for SAC to succeed.
E LIMITATIONS AND DISCUSSIONS
E. LIMITATIONS AND FUTURE WORK
In this paper, we mainly address key training challenges GC-GFlowNets problems, and study standard evaluation benchmarks from the GFlowNets literature (Bengio et al.) 2021 2023) with structured tasks (e.g.. DNA/RNA generation) where the dynamies are known and well-defined, For some tasks where environment dynamics might be unknown or infeasible to directly model, we can learn backward dynamic model f to predict the previous state, e.g., 8t-1 [St, at-1), following Höftmann et al. )023): Pan et al. (2023d). With a sufficiently collected dataset, learning a dynamic model is feasible it be framed as regression problem. We hope work can inspire future research in this promising direction studying unknown dynamics in the environment.
To align with the established and commonly used benchmarks and evaluation protocols in both GFlowNets (Bengio al.) (202)) and GC-GFlloNetss (Pan et al.) (023a) literature, our work primarily focuses on deterministic and discrete environments, which have been well-established and studied.
While recent theoretical work (Bengio al. 2023) explore continuous GFlowNets, their practical implementations and applications remain limited, as highlighted in Jain et al. (2023), where training continuous GFlowNetss poses significant challenges and scaling them realistic tasks remains
21

---

Published as a conference paper at ICLR 2025
(a) Map
(b) Map
(c) Map
Success Rate (56)
Success Rate (96)
Success
0%
0%
0%
RBS-GFN OC-GAFN DQN
RBS OC GAFF (W/ HER)
RBS OC-GAPN HER)
(d) Success rate
(e) Success rate
(I) Success rate
Figure 18: (a)~(c): Additional designed unseen maps to evaluate the generalization ability. (d)~(e): Average success rate over 3 random seeds on these unseen maps for 200 trials.
open problem in the field. We leave the extension of method continuous environments for future work.
E.2 BACKWARD LEARNING IN REINFORCEMENT LEARNING (RL)
Previous works (Goyal et 2019 Edwards 2018 Lai al. 2020 Wang 2021 in model- based RL leverage backward world models to optimize policies for returning to high-value states, while they fail to address the reward sparsity challenges in coal-conditioned RL. Höftmann al. (2023) use a backward dynamic model to generate trajectories for learning goal-conditionas policies by imitation learning, which sidesteps the requirement of rewards for policy learning. However, its performance heavily relies on the quality the learned backward model and has only been evaluated in relatively simple maze environments. Model-based goal-conditions promising direction, which employs a learned world model to imagine future trajectories to improve policy learning in an online (Yang et al. [202]] Charlesworth & Montanal 2020: Wang et al., (2024) or offline manner (Kim al.] 2024 Wang al. (2023). However, these methods primarily leverage forward imagination through the learned dynamic model to improve policy learning, whereas RBS introduces novel approach by sampling backward trajectories, thereby enhancing both data quality and diversity.
RL has been emerging as
22


Reviews:
[{"review_id": "Hn8qn0XLR0", "reviewer": "Reviewer_fNHf", "summary": "The authors introduces a novel method, Retrospective Backward Synthesis (RBS), aimed at enhancing the training of goal-conditioned Generative Flow Networks (GFlowNets) by synthesizing new backward trajectories. RBS augments \"virtual\" backward trajectories in\ngoal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity. RBS improves the sample efficiency and performance of GFlowNets across a range of tasks, including sequence generation and biological sequence design.", "questions": "1. How does RBS compare with diffusion policies, and in what scenarios does RBS offer distinct advantages?\n2. Does RBS assume deterministic or stochastic environments, and how does it handle situations where the backward dynamics are not straightforward?\n3. How can the authors ensure that the synthesized backward trajectories are meaningful and do not lead to false positives in the learning process?\n4. Could the authors include a comparison with goal-conditioned RL methods to highlight the specific benefits of using GC-GFlowNets?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies and targets a significant issue in the training of goal-conditioned GFlowNets, offering a practical and innovative solution. Augmenting backward trajectories for training is interesting.\n2. Comprehensive empirical results are provided, demonstrating the effectiveness of RBS over existing methods on multiple benchmarks.", "weaknesses": "1. Limited Discussion on Potential Drawbacks: The paper does not sufficiently address the potential limitations of RBS. For instance, there is no discussion about the computational overhead of synthesizing backward trajectories, nor is there a mention of whether the method is robust to different types of reward structures or environment dynamics.\n2. Relevance and Scope of Application: The improvements are made specifically within the context of GC-GFlowNets, which may limit the applicability of the method.\n3. Comparison with Diffusion Policies: Given the similarities between the proposed RBS and diffusion policies, a direct comparison would be valuable to understand the unique contributions and differences of RBS.\n4. Assumptions on Environment Dynamics: It is unclear whether the proposed RBS method assumes or requires any particular properties of the environment, such as determinism or stochasticity. If the backward dynamics are infeasible or the environment is highly stochastic, the performance of RBS may be affected, and this should be addressed.\n5. Quality of Synthetic Trajectories: The paper should include a discussion on how to ensure the quality of the synthesized backward trajectories, especially in cases where such trajectories may not correspond to realistic or feasible paths in the actual environment.\n6. Lack of Comparison with Goal-Conditioned RL: Without a comparison to goal-conditioned RL, it is difficult for readers to fully appreciate the relative strengths and weaknesses of GC-GFlowNets. Including such a comparison would provide a more complete picture of the method's positioning within the broader field of goal-directed learning.\n7. The authors may further investigate existing literature on augmenting backward trajectories for sample-efficient RL or backward learning in goal-conditioned RL, which makes the paper more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MDBoaa9h1N", "reviewer": "Reviewer_HyCh", "summary": "The paper proposes Retrospective Backward Synthesis (RBS), a novel method to enhance the training of goal-conditioned Generative Flow Networks (GC-GFlowNets). GC-GFlowNets have shown potential in generating diverse sets of high-reward candidates but face challenges due to sparse reward structures and limited coverage of explored trajectories, especially when using offline data. To address these limitations, RBS synthesizes backward trajectories that originate from a desired goal, enriching the training data with high-quality, diverse samples. This approach helps transform unsuccessful action sequences into positive learning experiences, thereby improving sample efficiency and generalizability.\n\nThe authors introduce additional techniques, such as reward signal intensification and backward policy regularization, to stabilize training and prevent mode collapse. Empirical results across various benchmarks, including GridWorld and bit sequence generation, demonstrate that RBS outperforms state-of-the-art methods in terms of success rates, sample efficiency, and scalability. Notably, RBS achieves nearly 100% success in large-scale tasks where competing approaches fail, highlighting its robustness and potential for further advancements in GC-GFlowNets.", "questions": "- In algorithm 1) line 6, how do we guarantee that the backward policy could reach $s_0$ from $y$ each time?\n- Are tuning for reward intensification and backward policy regularization difficult? What the the effect of hyper-parameters on the performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Backward-Looking Strategy for Enhanced Training: The proposed Retrospective Backward Synthesis (RBS) method utilizes a backward-looking strategy to synthesize trajectories from the goal state, significantly enriching training data. This approach effectively improves sample efficiency by converting failed experiences into successful learning signals, addressing the sparse reward problem.\n- Empirical Validation of Sample Efficiency: The paper presents strong empirical results across a range of benchmarks, demonstrating that RBS markedly improves sample efficiency. The method achieves nearly 100% success rates in complex tasks where state-of-the-art baselines fall short, underscoring its practical impact.\n- Clear Writing and Presentation: The paper is well-written and presented, with clear explanations, structured methodology, and comprehensive experimental results. The clarity facilitates a strong understanding of both the theoretical and practical aspects of the proposed approach.", "weaknesses": "- Scalability and Continuous Environments: The paper’s experiments focus on relatively simple and discrete environments, raising concerns about how well the Retrospective Backward Synthesis (RBS) method would scale to more complex, continuous, real-world tasks. The absence of testing in high-dimensional or continuous state-action spaces limits insights into its broader applicability.\n- Tuning Challenges: The method's reliance on hyperparameters, such as reward scaling and backward policy regularization, introduces tuning challenges. While these components are beneficial for stabilizing training, they require careful adjustment, potentially impacting the ease of replication and practical deployment in varied scenarios.\n- Lack of Comparison with Model-Based RL: Despite the inherent use of backward trajectory synthesis, which resembles model-based planning, the paper does not compare RBS with established model-based RL approaches such as MBPO or Dreamer. This omission makes it difficult to assess how RBS performs relative to other methods that also utilize environment models for planning and sample efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5m5sGFAGLN", "reviewer": "Reviewer_xjJi", "summary": "This paper addresses key challenges in goal-conditioned Generative Flow Networks (GFlowNets), specifically the problems of sparse rewards and limited trajectory coverage. The authors introduce Retrospective Backward Synthesis (RBS), a method that generates additional backward trajectories to expand the training data. Their approach aims to improve both the quality and diversity of training trajectories, providing more learning signals in scenarios with sparse rewards. Empirical evaluations demonstrate improved sample efficiency and performance compared to baseline methods across multiple benchmarks.", "questions": "Please address my concerns in the weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and straightforward to understand.\n* Retrospective Backward Synthesis (RBS) is introduced with clear motivation, and the paper also presents training techniques such as backward policy regularization. \n* Empirical results demonstrate that the proposed method outperforms baselines, showing improved performance and sample efficiency.", "weaknesses": "* The evaluation tasks do not include key benchmarks like RNA Generation from Pan et al. (2023a), which limits direct comparison.\n* The differences between the proposed RBS method and OC-GAFN are not clearly articulated. A more comprehensive discussion is needed to clarify the specific advantages of the RBS method. \n* It remains unclear how goals are defined across the evaluated tasks, which could impact generalizability and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5oJkhlkVge", "reviewer": "Reviewer_iEkK", "summary": "The paper tackles the challenge of training goal-conditioned Generative Flow Networks (GC-GFlowNets) in environments with sparse rewards and limited offline data. It introduces Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories to enrich training data, improving sample efficiency and diversity. Experiments demonstrate that RBS significantly improves performance and generalization in various benchmarks.", "questions": "1.\tAge-Based Sampling is a very straightforward technique. How does it compare to previous methods like Prioritized Experience Replay (PER)? Did the authors attempt using PER as well?\n\n2.\tRegarding the experimental setup, since you’ve compared your method with reinforcement learning approaches, I assume that these experiments share the same tasks as those in reinforcement learning. If that’s the case, why weren’t newer RL methods selected as baselines? Additionally, for tasks like bit sequence generation, TF binding generation, and AMP generation, is DQN an appropriate baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a novel method called Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories in goal-conditioned GFlowNets to improve the quality and diversity of training trajectories. This approach introduces rich learnable signals, effectively addressing the sparse reward problem.", "weaknesses": "1.The experimental tasks are relatively simple and insufficiently comprehensive.\n\n2.The latest goal-conditioned reinforcement learning algorithms are not selected for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)