Towards Improving Exploration through Sibling Augmented GFlowNets
Authors: Kanika Madan
Alex Lamb
Emmanuel Bengio
Glen Berseth
Yoshua Bengio
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Generative Flow Networks
Exploration
Primary area: generative models

Abstract: Exploration is a key factor for the success of an active learning agent, especially when dealing with sparse extrinsic terminal rewards and long trajectories. We introduce Sibling Augmented Generative Flow Networks (SA-GFN), a novel framework designed to enhance exploration and training efficiency of Generative Flow Networks (GFlowNets). SA-GFN uses a decoupled dual network architecture, comprising of a main Behavior Network and an exploratory Sibling Network, to enable a diverse exploration of the underlying distribution using intrinsic rewards. Inspired by the ideas on exploration from reinforcement learning, SA-GFN provides a general-purpose exploration and learning paradigm that integrates with multiple GFlowNet training objectives and is especially helpful for exploration over a wide range of sparse or low reward distributions and task structures. An extensive set of experiments across a diverse range of tasks, reward structures and trajectory lengths, along with a thorough set of ablations, demonstrate the superior performance of SA-GFN in terms of exploration efficacy and convergence speed as compared to the existing methods. In addition, SA-GFN's versatility and compatibility with different GFlowNet training objectives and intrinsic reward methods underscores its broad applicability in various problem domains.

Published as a conference paper at ICLR 2025
TOWARDS IMPROVING EXPLORATION THROUGH SIBLING AUGMENTED GFLOWNETS
Kanika Madan', Alex Lamb?, Emmanuel Bengio', Glen Berseth', Yoshua Bengio!
ABSTRACT
Exploration is a key factor for the success of an active learning agent, especially when dealing with sparse extrinsic terminal rewards and long trajectories. We in- troduce Sibling Augmented Generative Flow Networks (SA-GFN), a novel frame- work designed to enhance exploration and training efficiency of Generative Flow Networks (GFlowNets). SA-GFN uses a decoupled dual network architecture, comprising of main Behavior Network and an exploratory Sibling Network, enable a diverse exploration of the underlying distribution using intrinsic rewards. Inspired by the ideas on exploration from reinforcement learning, SA-GFN pro- vides a general-purposs exploration and learning paradigm that integrates with multiple GFlowNet training objectives and is especially helpful for exploration over wide range of sparse or low reward distributions and task structures. An extensive set of experiments across diverse range of tasks, reward structures and trajectory lengths, along with a thorough set of ablations, demonstrate the superior performance of SA-GEN terms of exploration efficacy and convergence speed as compared to the existing methods. In addition, SA-GFN's versatility and com- patibility with different GFlowNet training objectives and intrinsic reward meth- ods underscores its broad applicability in various problem domains.
INTRODUCTION
Exploration a fundamental aspect of a learning agent that actively interacting with its environ- ment and learning from the experience collected. A sufficient amount of exploration also leads to a better generalization, enabling the agent to learn from a diverse set of experiences collected from different regions of the underlying state space. In reinforcement learning, this is commonly referred to as the doesn't 't explore can get stuck and is unable to collect novel experiences to further improve its policy. In the absence of any intermediate reward signals, the learning agent might need to traverse several low zero reward regions in order discover the ones with high-rewards, making exploration non-trivial task, especially when faced with a large state space, sparse terminal reward signals and complex reward distributions.
trade-off (Sutton & Barto 2018). An agent that only exploits and
Manually designing dense rewards has been successful in many areas of robotics and games (Mnih) al. 2015 Baker al.) 2019 Hatner al. 2019), but this approach is not easy to design and scale. Unstructured exploration based methods, such as e-greedy or randomized probability match- ing, are not able to effectively leverage any structure and hence are not efficient beyond simple settings. Intrinsic motivation methods have been successfully used (Pathak et al.) 2017; Burda jet al., 2018, [Badia al., 2020) /Zhang et al. (02T) to guide exploration over a range of problems, and are based generating intrinsic rewards, or rewards internal to agent, encourage visiting novel states. Random Network Distillation (RND) (Burda et al. 2018) is a commonly used intrinsic reward method that is easy use and scale, and generates intrinsic rewards defined in terms of an evolving distance metric based on the fixed and learnt representations of a given state.
At the same time, for many scenarios such as biological sequence design (Jain et al. 2023 and molecule generation (Bengio et al. 2021a), diversity can be an equally important aspect for the learnt policy. While reinforcement learning (RI.) methods are based on learning reward-max maximiin
Mila Québec AI Institute, Université de Montréal, Microsoft Research, 3 Valence Labs. Corresponding author; madankan's ka. s@@ma com

---

Published as a conference paper at ICLR 2025
policy that can sample from the highest mode of the distribution, generating diverse high-reward solutions might be what we actually care about.
Generative Flow Networks, (GFlowNetss; Bengio et al. 202 amortized variational inference algorithms that learn a stochastic forward policy ग [a|s) to generate an object æ in proportion to its (positive) reward R(x). Unlike RL. GFlowNets
generate diverse high-rewarding candidates. However, similar to RL, the training framework of GelowNets is based on actively sampling trajec- tories from the forward policy and learning from the corresponding terminal rewards. Therefore, the trade-off'
success of a GFlowNet Net policy also depends on achieving a good ssploration-exlloitaee ensure that the policy has been able discover high-rewarding modes of the target distribution.
Some previous works, such as Malkin et al. (2022) Deleu et al. (2022); Madan et al. (002); Pan al.] (2023) have addressed improving credit assignment GetowNets, and Pan et al. (2022); Rector-Brooks et al. (2023); Lau et al. (2023), have looked into exploration. However, the range of exploration tasks that have been considered far are limited to smaller objects to be generated,
relatively easier reward settings, and extremely sparse reward settings.
In this work, we aim to reduce this gap by proposing a general framework dubbed Sibling Aug- mented Generative Flow Networks or SA-GFN, that can not only explore better, but can also learn faster over wide range of tasks that are not just limited to easy reward or extremely sparse zero-reward settings previously considered. We also expand the set of previous exploration bench- marks to include non-zero, yet difficult to explore, sparse reward structures, as well as challenging sequence design problems for both short and long range trajectories. To our knowledge, this is the first extensive evaluation focused on the exploration aspect of GFlowNets.
The main contributions of our work are the following:
1. We propose a novel general-purpose framework, called Sibling Augmented Genera- tive Flow Networks SA-GFN, that achieves much better exploration and training of GelowNets using intrinsic rewards and a disentangled dual network architecture.
2. We show the generality of SA-GFN, which is in principle compatible with any GFlowNet objective and any intrinsic reward method, and hence is simple to work with.
Through an extensive set of experiments and ablations over a wide range of tasks and difficulty levels, we establish the exploration benefits of SA-GFN compared to the previously introduced methods.
RELATED WORK
2.1 INTRINSIC MOTIVATION BASED EXPLORATION METHODS
Intrinsic motivation based methods encourage exploration by providing additional intrinsic rewards, ic. rewards internal to the agent, for visiting unexplored regions of the state space.
Prediction Error based methods: Predictions from a learnt world dynamics model are compared against the ground truth, and higher prediction errors provide higher intrinsic rewards (Achiam & Li Li al. 2019), encouraging the policy explore unknown regions of the distribution.
Sastry 2017
Novelty based methods:
The intrinsic reward is defined in terms of an evolving distance metric between the learnt embedding and a target embedding of the visited states. Count-basee methods are early examples of this class of methods (Bellemare et al.) 2016]] Ostrovski et al. [017 Tang al. 2016). but these do not scale well. more scalable variation is Random Network Distillation or RND (Burda et al.) 2018) in which representations from a randomly initialized target network are distilled into learnt network, providing an evolving distance metric; several works (Pathak et al.
2017
Badia et al. 2020 Zhang et al. 2020 2021) have built on this framework.
2.2
EXPL.ORATION FOR TRAINING GFLOWNETS
Similar to reinforcement learning. GFlowNets also face the problem of exploration when learning over long trajectories, high-dimensional spaces, sparse rewards and non-uniform reward settings
2

---

Published as a conference paper at ICLR 2025
reward structures. Connections between RL and GFlowNets have been made in ?. Although intrinsic rewards were incorporated into the training objectives of GelowNets in Pan al. (2022). only extremely sparse settings with zero rewards were considered, and empirically found it to not be able to explore well to more general, difficult and low (not necessarily zero) reward setrings (see Section [5). flector-rrooks
with
al. (2023) used variation of Thompson Sampling using an ensemble of K forward policies to estimate uncertainty (Osband et al., [2166] 2018) to guide exploration. However, the size of such ensemble methods is sensitive parameter and these methods tend to not scale well. [au et al. (2023) proposed to use moving average copy of the online network to be used as a sampler stabilize training and improve exploration, but only considered settings where exploration is not as much of an issue.
BACKGROUND
3.1 GFLOWNETS
Consider directed acyclic graph (DAG), G (S, A),, such that represents the set of nodes and A. represents the set of edges (s + 8') A. Given an edge (s -> s'), is called the parent of s' and s' called the child of s. The DAG is called pointed because it has a unique root node with no parents, also called source node initial state so- Terminal states do not have any children and belong to the set X' of objects that the GFlowNet policy could constructively sample through a sequence of actions. trajectory
(Sm= Sm+1. ... Sn is formed by sequence of actions (s; - $1+1) and is called complete if sm = so and s, is a terminal state.
GFlowNets learn stochastic forward policy π, that can take sequence of actions to sequentially generate an object æ in proportion to its reward R(x)) Each such action taken by the GFlowNet modifies the state, for example by adding element to the partially constructed object generated so far, until the policy decides to stop and terminal object x is generated, after which corresponding reward R(x) is provided by the environment. We emphasize that there are only terminal rewards provided at the end of the trajectory.
3.2 GETS TRAINING
Many training objectives have been defined for GFlowNets, such as Flow Matching objective (Ben- gio et al., 202 Detailed Balance objective (Bengio et al. 20221b). Trajectory Balance objective (Malkin et al., 2022) and SubTB()) objective (Madan et al. 2023). and these operate on the level of the state, edge, full length (complete) trajectories and sub-trajectories of any lengths, respectively. These training objectives are obtained by setting up a set of flow-matching constraints with the property that when all these constraints are satisfied, the GFlowNet sampling policy has the desired property that generates terminal states with probability proportional to the given reward function. Each constraint of the right-hand side to the left-hand side the equality constraint. Each loss term thus corresponds an amount of constraint violation. Training consists in sampling trajectories and measuring these constraint violations (the loss) and its gradient on the parameters of interest. Furthermore, they all enable off-policy training, i.e., these training trajectories can be sampled from any full-support dis- tribution. However, some distributions will yield faster convergence because they focus on areas where the rewards are higher and where the current policy has not yet explored.
be turned into loss, typically by taking the square of the logarithm of the ratio
The Flow Matching (FM) (Bengio et al. 2021a) objective parameterizes GFlowNets through edge flows F(s - s':0) on states s. The Detailed Balance (DB) (Bengio et al. (2021b) and the SubTB(A) (Madan et al. 2023) objectives paramaterize the state flow F(s; f), forward policy Pr [s; 0), and backward policy P (s|s';0) on actions s + to define a GFlowNet. The Trajectory Balance (TB) (Malkin et al. 2022 objective works with complete trajectories, and parameterizes the GFlowNet through an initial state flow Zg. and forward and backward policies s= f), PB (s|ss B) respec- tively. The flow-mater wow-matching constraints represented by these parameterized quantities are converted into a loss function by equating the left and right hand sides of the constraint equations as a squared loss. The flow matching equation for the Trajectory Balance loss is shown in Eq. [
3

---

Published as a conference paper at ICLR 2025
Ze II
Pro (81+1ls;) R(s,)
II PBo (8.15++1).
4 SIBLING AUGMENTED GENERATIVE FLOW NETWORKS (SA-GFN)
We propose a flexible GFlowNet learning framework, dubbed Sibling Augmented Gen- erative Flow Networks or SA-GEN, that in- corporates intrinsic rewards in a simple manner and leverages the off-policy learning capabil- ities GFlowNets to learn from exploratory data and better match the underlying energy function. Specifically, SA-GFN uses Ran- dom Network Distillation (RND) (Burda al.] 2018) as intrinsic rewards, and has a disen- tangled dual network architecture in which exploratory sibling policy provides exploratory data to train the main behavior policy. By dis- entangling exploration from learning, SA-GEN enables simultaneous efficient exploration and effective learning.
Behavior Network Sibling Network
π BN
SN
Sample
Sample
Updatc SN
Update
BNN
Relabel SN
BN
BN U
BN
SN
RND-based intrinsic rewards are defined such that states with higher novelty reap larger in- trinsic rewards. However, directly incorporat- ing continuously evolving reward into the training objective of GFlowNets makes the tar- get reward distribution moving target; this can negatively impact learning. We empiri- cally show that by disentangling the exploration network (trained using the RND based intrin- sic rewards) from the main behavior network (trained using rewards from the true distribu- tion), the proposed SA-GFN achieves a better exploration and learning of the underlying dis- tribution.
.NN SN
After Training
BN π
Sample-
BN
BN
Netwoek
Sibling Network Polscy
SM from Sibling Network
Bebavicr Network Policy
Ralabokod of Sibling Network
Sibling Augmented Generative Flow Networks (SA-GFN) has a decoupled architec- ture, consisting of (a) a main Behavior Network with policy x BN and (b) an exploratory Sibling Network with policy x SN Exploratory trajecto- ries, SN,, sampled using intrinsic rewards from policy 7 sis are (a) used to update the Sibling Net- work and (b) relabeled with true rewards, TS and combined with SN to update the Behavior Network.
Figure 1:
4.1 SA-GFN ARCHITECTRRE
Sibling Augmented Generative Flow Networks (SA-GFN) adopts a decoupled architecture consisting of two separate networks: a Sibling Network and a Behavior Network. The Behav- ior Network is the main GFlowNet that aims to learn the true target reward distribution, while the Sibling Network is an exploratory policy network that uses intrinsic rewards to explore the space. The exploratory data collected by the Sibling Network is relabeled with the true wards (Andrychowicz et al. (017)) and is com- bined with the on-policy data collected by the Behavior Network train its forward policy. The off-polll off-policy learning capabilities of GFlowNets and relabeling of trajectories frees the Behavior Net- work from the task of modeling the continuously evolving intrinsic rewards, enabling SA-GFN to achieve an efficient exploration.
The main components of the proposed SA-GFN architecture are follows.
4

---

Published as a conference paper at ICLR 2025
Intrinsic Reward Network: This module generates intrinsic rewards for a given of sampled trajectories. In SA-GFN, we use Random Network Distillation (RND) (Burda et al. 2018 to com- pute these intrinsic rewards in which a randomly generated target network is distilled into leart predictor network. As consequence, novel states, or those that are difficult to predict, tend to reap higher intrinsic rewards. In principle, any other intrinsic reward as well as extensions of RND can be used and we provide the evidence for this in Section I0.3
The Sibling Network uses intrinsic reward based exploration sample tra- jectories SS according to the reward r. defined in Eq. 2 where frgx: |3; and AsN are the hyperparameters for the reward exponents, and ri, rf represent the intrinsic and extrinsic rewards, respectively, at time f. Any of the GFlowNet training objectives discussed in Section 3.ctionf.2jcan be used to train the Sibling Network using (rSN, rrN). ). We use the TB loss in our experiments, and provide extension to the other GFlowNet objectives in Section 10.4
Sibling Network:
SN = ((r2) + E r())
(2)
Behavior Network: The Behavior Net- work is the main GFlowNet that learns to sample according to a given extrinsic re- ward function. The Behavior GFlowNet
Algorithm 1: Sibling Augmented Generative
Flow Networks (SA-GFN)
learns from two sets of trajectories: (a) on-
Require:
Sibling Network SN:
policy trajectories and corresponding re-
P (). P5 (s|s'), ZSN; -BBhavior
wards RRN, . generated by the for-
Network or BN:
ward policy of the Behavior Network, and (b) exploratory trajectories generated by the Sibling Network with relabeled rewards, denoted by (T&N rr)) Since GFlowNets can be trained off-policy, the two sets of trajectories are combined ((TBNUTHN ), BN Urt3) and are used UTSN
PEN (s' [s, P (s|s"), ZBN; random target
network: Φ; predictor network: d;
1: Input
2: BenN Begy: reward exponents for extrinsic rewards, rū. for Behavior Network and Sibling Network;
3: 35: reward exponent for intrinsic rewards ri:
together to train the Behavior Network.
4: 8SN. SEN: reward exponents for final rewards of Sibling Network and Behavior
By decoupling the training of the main Behavior Network from that of the ex- ploratory Sibling Network, any optimiza- tion instabilities arising from a continu- ously changing intrinsic rewards do not have a negative impact on the training of the main Behavior Network. Instead, learning from the exploratory data gen- erated by the Sibling Network allows a very efficient exploration of the target dis- tribution, and an extensive set of exper- iments over a diverse set of reward set- tings corroborate this in Section [] More- over, this decoupling makes the SA-GEN framework general enough that any of the GFlowNeet training objectives and any of the variants of intrinsic rewards can be seamlessly incorporated, as empirically shown in Sections [1044] and [0.3]
Network;
5: for each training iteration do
6: Collect trajectories SN using the forward policy P (s'(s) of the Sibling Network Compute reward r.SN for Sibling Network
7:
using Eq. 2
8:
II Update the Sibling Network Compute C (rSN, .r. using Eq. I and
update the Sibling Network
P (s' (s). PS
(s)s)))) (s|s'), ZSN
9: Collect trajectories T IRNN & extrinsic
rewards r using the forward policy
P (s' (8) of the Behavior Network.
10:
BN S t (7 SN r() Relabel: (T
11:
Update: ,BN +- BN UTS
BNY
12:
Update: ,.N +- Br U SMS
13:
II Update the Behavior Network Compute CBN (BN, ,r using Eq. |
and update the Behavior Network PF (s' 1)),PBB (s|s'), BN
4.2 SA-GFN TRAINING OBJECTIVES
14: 11 Update the Intrinsic Reward Network
Any of the GFlowNet training objectives from Section 3.2 can be used to train the Behavior and Sibling networks of the SA- GFN architecture. Trajectories generated
Update the intrinsic reward network &
using RND loss (|Ø()) d(s))12
15: end for
5

---

Published as a conference paper at ICLR 2025
8 x
32 x.22
64 64
128 128
10-2
10-3
10"
10"
SA-GIN
GAFN
erter
10
10-1
armor
GFN
10-1
10
10"4
10-1
10
10000
20000
10000
20000
10000
20000
10000
20000
Figure 2: For a Sparse HyperGrid with zero-rewards, SA-GFN outperforms all other baselines, high- lighting faster convergence, efficient exploration, and better learning of the true reward distribution.
by the Sibling Network are relabeled with true extrinsic rewards (Andrychowicz et al., 2017) to train the Behavior Network using off- polipo learning. To encourage visiting novel states, the Intrinsic Reward Network is based on, but not limited to (as shown in 10.3). RND (Burda et al. 20188). Full algorithm is laid out Algo and Figure
4.3 HYPOTHESIZED BENEFITS OF SA-GEN
In addition improving exploration, we hypothesize the following other benefits of SA-GFN.
Better Training Learning only from a terminal rewards can severely limit the training efficiency efficient
of agent. Having intermediate rewards, such intrinsic rewards, induce exploration. However, since GFlowNets learn to match fixed target reward distribution, using non- stationary rewards during training creates moving targets, leading to inefficiencies during training. The proposed SA-GFN decouples exploration from training by using separate Sibling Network generate exploratory trajectories that are relabeled and used to train the main Behavior Network. The target reward distribution of the Behavior Network thus stays unchanged, while experience from unexplored regions
distribution is being constantly fed, allowing SAGEN to explore better as compared to the single network variants, as evident through results shown in Section
SA-GFN allows using any of the GFlowNet training objectives and architectures to train the models, shown in [0.4. Moreover, other intrinsic reward methods, including extensions of RND, can be used for the Intrinsic Reward Module of the Sibling Network, shown in 10.3 SA-GFN also allows to other techniques improve training, such as replay buffer, multiple heads or ensemble (Rector- Brooor et al.) 2023), other exploration focused variants of GFlowNets, reward exponents and tempered policies (Kim al. 2024).
Flexible & Expandable Architecture
EXPERIMENTS
To evaluate SA-GFN against other baselines, we address the following research questions:
1. Number of discovered modes: We track the number of modes learnt by each method over a diverse range of task structures and reward settings; Section 5.1 and 5.2
2. Learning of the true reward distribution: We measure the L1 error between the true reward distribution and the learnt empirical distribution for each method, Section 5.1 We also visualize the learnt empirical distributions at the end of the training to compare against the true reward; Section T0.7
3. Efficient exploration under variety of tasks and reward structures: We extend the range tasks to cover a wide range difficult exploration settings and difficult to explore reward structures; Section 5.1 5.2 and 5.3
4. Robustmess to trajectory length and size of the action space: We test all methods over a large range of trajectory lengths and varying dimensionalities the action space for multiple domains and task structures; Section 5.1 5.2
To evaluate wide range of exploration tasks, conducted experiments the following four domains, which to our knowledge covers the widest range of exploration settings considered so far: (a) Sparse Zero-Ren Reward HyperGrid: Hypergrid with zero-reward regions; (Section 5.1.1) T), (b) Sparse

---

Published as a conference paper at ICLR 2025
Figure 3: Sparse Rewards: [Left]] zero rewards [Right] very low non-zero rewards, and high-reward corners.
x
16
32 x
48
10-1
L1 ororr
10-1
4
3000
10
10
10-3
64 64
80
112 1112
10-4
L1 arear
L1 creer
10
10
10-4
10-3
1000
10a
10f
100
10a
10a
109
102
10a
109
10
10*
136
136
144 144
152 152
10-4
10-4
L1 enor
oner
SA-GFN
GAFN
10
RND
10
10a
10a
108
100
8 8 x
16 16
16
32 32
32 32
64 x 64 64 64
2xx10
x0-0
40-1
1.15
crerr
30-1
eror
ernr
10-1
1.1x
20
x100-1
SA-GFN
6x10--
10-
AND
Figure 4: For the Sparse HyperGrid with low-reward, yet difficult to explore, setting, SA-GFN outperforms all other baselines methods, highlighting its benefits in terms of faster convergence, efficient exploration, and a better learning of the underlying true reward distribution.
HyperGrid: HyperGrid with regions of very low, non-zero, difficult to explore rewards (Section 5.1.2p. (c) Bit Sequence Task: from Malkin al. (2022) involving auto-regressive gener- ation of sequences of varying lengths and vocabulary sizes (Section 5.2). and (d) Small Molecule Generation: from (Bengio al. (021a), involving generation sEH protein binders (Section 5.3
5.1 HYPERGRID
The HyperGrid environment is d-dimensional grid of size H x H X Sp: (0.0.....00) 0). To ensure DAG structure, each action increments one of the coordinates by without leaving the grid, with additional stop action end the trajectory.
H with initial state

---

Published as a conference paper at ICLR 2025
SA-GFN
GAFN
RND
GFN TB
PPO
0
50
50
50
50-
50
100
100
100
100
100
0
100
100
100
100
0
100
Figure 5: SA-GFN learns all the four modes of the distribution while other methods cannot, showing their poor exploration properties in a Sparse HyperGrid with difficult to explore rewards setting.
In order evaluate over a broad spectrum of hard exploration problems, we control exploration difficulty through the following:
Grid size: Larger grids necessitate traversing longer trajectories through very low reward or sparse reward spaces order find high rewarding modes. Thus, larger the grid, harder the exploration.
Grid Dimensionality: A HyperGrii spanning multiple dimensions even for shorter trajectories. For example, a 2-dimensional grid with H = 100 and 4-dimensional grid with H- 10 have the same number of terminal states, but pose different exploration challenges.
be difficult explore,
Reward Density: Extremely spare reward structures, where the reward is zero unless mode is reached, pose an important exploration problem. At the same time, low-reward non-zero sparse reward structures can also be very difficult to explore and are more common find.
5.1.1 SPARSE HYPERGRID WITH ZERO REWARDS
We first consider the Sparse HyperGrid setting from (Pan et al. 2022), ) Figure Left], where the reward is +1 only at the three corners and 0 otherwise. Exploration difficulty is controlled through the length of horizon: larger values of H make discovering the modes harder. We compare SA- GFN with number of strong baselines such as GAFN (Pan al., 2022). GFN with Trajectory Balance (Malkin et al, 2022). and RND with a single GFN architecture (Burda et al. 2018). We find that across all grid sizes, SA-GFN substantially outperforms all baseline methods in terms of faster convergence, efficient exploration, and an overall better learning of the underlying reward distribution measured by the L1 error between the empirical distribution and the distribution, see Figure 2
5.1.2 SPARSE HYPERGRID WITH HARD TO EXPLORE REWARDS
We now consider a low-reward sparse HyperGrid with difficult to explore reward structure. In this HyperGrid, high-rewarding modes are placed at the 2d corners of d-dimensional grid and relatively low (but non-zero) reward regions are located through the rest of the grid, see Figure Right]. These low reward regions separating the high reward modes make mode discovery challenging.
The structure of the reward is the similar to the experiments in Malkin et al. 2022), and we consider the following two hard to explore reward settings: (Rg = 10 - RR R, = 1.0, R₂ 3.0) and (Ro = 10-3 Ri 1.0, Rg- 3.0). The difficulty of exploration is controlled via reward density, grid size and action-space dimensionality. Compared to the strong baselines of GAFN (Pan et al. 2022). GFN with Trajectory Balance (Malkin et al. 2022). and RND policy with single network architecture (Burda et al.] 2018), we find that SA-GFN performs a much better exploration and a better matching of the underfying distribution, see Figure and Section 10.2 Additional results in Section 10.5
We further visualize the distributions learnt by SA-GEN and other methods, and find that across all grid lengths, SA-GEN discovers all four modes of the distribution, while other methods able to discover only subset of the modes, corroborating the benefits provided by SA-GFN, Figure 5
Additional Experiments on figher-dimensional HyperGrids (d = 6, 8) are detailed in 10.1 and Figure 9 and for rewards setting (Ro 10-5 Rı 1.0, Ra 3.0) in 10.2 Moreover, visual
8

---

Published as a conference paper at ICLR 2025
tions of the learnt distributions corresponding to Figure are provided in Figure 18 and Figure 19 with a further discussion in Section 10.7 to highlight the better exploration properties of SA-GFN
5.2 BIT SEQUENCE GENERATION
Taken from from| Malkin et al. (2022), this is auto-regressive sequence generation problem. At each time step, the policy adds kib token to the end of the partial sequence generated far, and the final fixed length (n = 120) sequence (A E {0, 1)") is auto-regressively generated from left to right. For fixed sequence length, by varying the values of k, the actual length of the trajectory ($) and the size of the action space or vocabulary ([V) = 2*) can be efficiently controlled without changing the domain. The form of the reward function is defined in terms of a pre-defined fixed set of modes M X' and completely unknown to the learning agent until visits a terminal state and obtains the value of the reward function that terminal state. For given sequence X, the reward is defined in terms of the edit distance d as: R(x) = exp( - minyeM d(x, y)).
We evaluate the following three metrics to
test the effectiveness of exploration: (a) num-
Number of Modes lound
50
ber of modes discovered for all values of
SA-GEN
k = speed of mode discovery for ing the longest trajectories, Figure Center),
6, 8, 10), Figure Top], (b)
GAFN
1 hav-
40
TB GFN
and (c) the Spearman correlation between the
30-
probability of generating a sequence p(x)
F(z)/Z and its reward R(x) on uniformly sampled test set, Figure Bottom.] SA-GFN compared with the baseline methods of GFlowNets with FM (Bengio et al. (021aa and TB (Malkin et al. 2022). A2C reinforcement learning with Entropy Regularization (Willaams
20
2
10
Number Bits Action Space
Number of Modes Modes
40
SA-GEN
GAPN
30-
GFN
FM GEN
20
Botropy
& Peng 1991; Mnih & Gregor 2014). Soft Actor-Crite (Christodoulou 2019: HHarnoja jet al. (017). and MARS (Xie et al. 2021). an MCMC method. For all of these three metries,
Seft Amnn-Cnnn
10
MCMC
0
10000
20000
30000
40000
50000
we find that SA-GFN (a) discovers more modes for all values of k (b) maintains a high corre- lation between the probability of generating sequence x and its reward R(x), and (c) discov- ers more modes much faster when compared to other strong methods, see Figure [ proving its robustness to the size of the action space and its effectiveness to do a better exploration.
Training Steps
Spermmnn Correlation (p.1)
0.6
0.4
TE GPN
FM GIN
0.2
All:- Dnvepp
MFM:
4
10
Number of Bits Action Space
5.3 SMALL MOLECULE GENERATION
Figure 6: [Top] For all values of k (1, (1,2,4,6, 8, 10), SA-GEN finds the highest num- ber of modes, [Center] discovers more modes faster (shown for = 1), [Bottom] has the high- est Spearman correlation between the reward test set and the sampling probability for all k.
We further consider a more practical task of Molecule Generation from (Bengio al.
2021a) that involves generation of binders of sEH protein (soluble epoxide hydrolase), and has a large state space of the order of 10 and between 100 and 2000 actions depending on the state. We compare SA-GFN, with five other strong baselines: (a) GAFN (Pan et al. 2022), (b) GFN: GelowNets trained with FM objective (Bengio et al., a (c) PPO-RND: PPO with RND rewards (Burda et al.] 2018), (d) PPO (Schulman et al. 2017 and (e) MARS (Xie al. 2021). We evaluate the (a) aver- age reward generated by the top-10 unique molecules, (b) average Tanimoto similarity for top- 10 samples, see Figure Left] and Figure Center]. and find that SA-GFN, achieves the highest reward and has the lowest Tanimoto similarity, emphasizing high quality diverse generated can- didates. We also include results for Tanimoto similarity for top-k generated molecules, where

---

Published as a conference paper at ICLR 2025
8
TJainooo Strdannee
205
Okey
40
MAIS PPU INU GEN
GAIN SA-GIN
MAR5 190 RO GIN
G47 $A-ORN
Top
Figure 7: We find that SAGEN [Left] generates the highest reward, [Center] achieves the highest diversity in terms of Tanimoto Similarity (lower is better) of the generated molecules, and [Right] generates most diverse top-k molecules for = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
[10, 20, 30, 40, 50, 60, 70, 80, 90, 100] and find that SA-GEN consistently generates molecules with higher diversity, showing its better exploration properties, Figure 7 Right].
ABLATION STUDIES
3.0
10
2.5
10
SA-GFN
Numbar
4.5
SA-GFN
10-
GARN
GATN
2dgt
10"4
mperred
3000
10000
15000
20000
200
400
600
300
1000
5000
10000
15000
20090
Updale
Update
Update
Figure 8: [Left] SA-GFN explores more efficiently & [Center] discovers modes faster compared to state-only and egde-only intrinsic reward based single network architectures. [Right] Simple exploration methods, such tempered policy logits, not sufficient ensure good exploration.
SA-GEN vs state and edge based augmentations:
We evaluated SA-GFN against a single- network variant with state & edge-level intrinsic reward augmentations for sparse reward setting and found the SA-GFN's decoupled architecture allows faster exploration and better learning of the distribution, Figure 3 Left] and Figure (Center))
SA-GFN vs simpler exploration: Simple exploration, such tempering logits, and entropy based methods such SAC (Haarnoja al. 2017). do not explore well, Figure [8 Right], 10.6 and
10.5
7 DISCUSSION & CONCLUSION
We introduce Sibling Augmented Generative Flow Networks, SA-GFN, that adopts dual net- work architecture to disentangle exploration from learning, and leverages off-policy learning and trajectory relabeling to learn behavior policy from exploratory data. SA-GFN provides effi- cient exploration strategy and an easy integration of intrinsic rewards with the existing GFlowNet objectives, outperforming all other strong baselines over wide range of exploration tasks.
Limitations and Future Work: SA-GEN maintains two separate networks, and future works could explore single network variant by using two-headed architecture, for example. Multiple sibling networks (or heads) could also be incorporated to further improve exploration.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Joshua Achiam and Shankar Sastry, Surprise-based intrinsic motivation for deep reinforcement learning. CoRR, abs/1703.1 2017.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR, abs/1707.01495, 2017.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. CoRR, abs/2002.06038, 2020.
Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. CoRR, htts/1909.07555 2019.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. CoRR, abs/1606.00 01868, 2016.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. CoRR, abs/2106.04399 04399, 2021a.
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. CoRR, abs/211 0tte/2111.099 1bb//211.00966, 1.00266, 9b.lb.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. CoRR, abs//8100.120 2018.
Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, abs/1910.07207, 2019.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks, 2022. URL https://erxiie or.. harg/abs//200 13903
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. CoRR, abs/1702.08165, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, ftss/1801.0125 2018.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learn- ing behaviors by latent imagination. CoRR, abs/ 0bbs/192..0100 2019.
Moksh Jain, Emmanuel Bengio, Heee-Heennddzz Garcia, Jarrid Rector- cector-Brooks, Bonaventure F. P. Dossou, Chanakya Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with gflownets, 2023. URL
fitts::/aarxxr org/ abs ..200.001115
Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for tempperatue-conndiionnl gflownets, 2024.
02823
Elaine Lau, Nikhil Vemgal, Doina Precup, and Emmanuel Bengio. DGFN: double generative flow networks. CoRR, 0bs/2310.116855 2023.
Boyao Li, Tao Lu, Jiayi Li. Ning Lu, Yinghao Cai, and Shuo Wang. Curiosity-driven exploration for off-policy reinforcement learning methods. In 2019 IEEE International Conference on Robotics and Biomimetics, ROBIO 2019, Dali, China, December 6-8, 2019, pp. 100--1144, IEFF, 2019.
11

---

Published as a conference paper at ICLR 2025
Kanika Madan, Jarrid Rector-Brooks Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability, 2023. URL https://arxiv. /bs/ 2209. 12782
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. CoRR, abs/2201.13259) 2022.
Andriy Mnih and Karol Gregor. Neural variational inference and learning belief networks, 2014.
URL https: http:///axivv..oo org. awt.aassss00 1402. 0030
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-55555 533-533-333,32 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. CoRR, abs/1 (12111002046111 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement abs 1806. 03335
learning, 2018. URL htthtttt////xiii/
Georg Ostrovski, Mare G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based explo- ration with neural density models. CoRR, abs/1703.01310 2017.
Ling Pan, Dinghuai Zhang, Aaron C. Courville, Longho Huang, and Yoshua Bengio. Generative augmented flow networks. CoRR, abs/2210.03308, 2022.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflowness with local credit and incomplete trajectories. In Proceedings of the 40th International Conference Machine Learning. ICML'22 '2. JMLR.org. 2023.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. CoRR, abs/1705.05363, 2017.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets, 2023. URL
org abs /
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018 URL
wtt.://oo//teeetooooooooooo 2nd.
html
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. CoRR, 0bb//6611.0471,, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, 45s/1706.03366, 2017.
Ronald Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3:241-, 09 1991. doi: 0000000954409991999443333
Yutong Xie. Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: Markov molecular sampling for multi-objective drug discovery. International Conference Learning Representations (ICLR), 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. Bebold: Exploration beyond the boundary of explored regions. CoRR,
abs/2012.08621,, 2020.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Noveld: A simple yet effective exploration criterion. Curran Associates, Inc., 2021.
12

---

Published as a conference paper at ICLR 2025
8 APPENDIX
EXPERIMENTAL DETAILS
9.1 EXPERIMENTAL DETAILS: SPARSE REWARD HYPERGRID WITH ZERO REWARDS
The Sparse reward HyperGrid setting consists of reward of +1 the three corners of the grid (except the starting corner), and a reward of 0 everywhere else, as visualized in Figure Left]. We use increasing grid sizes with H E {8, 32, 64, 128). The reward function parameters and architecture choices are based on the published codebase of Malkin et al. (2022) and Pan et al. (2022). For both Sibling Network and Behavior Network, a shared network architecture with multiple heads to parameterize the GFlowNet parameters Pp, PB. F is used.
All models are trained with Adam optimizer with a batch size of 16 for total of 20000 up- dates and 3 seeds. The learning rate is chosen from {0.001, 0.005, 0.01, 0.03) } for the forward and backward policies Pr and PB with the trajectory balance objective (Maikin et al.] 2022). and the learning rate of the Zo is 10x learning rates of Pr and PB. Reward temperatures values of {BenN = 1.0, BesN = 0.25, S5N intrinsic rewards, we choose RND rewards with the intrinsic reward coefficient chosen from {0.00005, 0.00001, 0.0005, 0.0001, 0.005, 0.001, 0.05, 0.01)
L0,,BB 1.0, 3 1.0] are used. For
9.2 EXPERIMENTAL DETAILS: SPARSE HYPERGRID WITH DIFFICULT TO EXPLORE REWARDS
This version of the Sparse HyperGrid has a reward distribution as shown in Figure Right], such that high reward "regions" are located at the 2" corners of d-dimensional grid, with low-reward regions everywhere else. We run all experiments on horizon sizes from H {8. 16, 32, 48, 64, 80, 96, 112, 128, 136, 144, 152}. The reward structure is defined as in Malkin et al. (2022) and reward function parameters of {Ro 1e-4, R Ri 1.0, R₂ = 3.0). For both Sibling Network and Behavior Network, a shared network architecture with multiple heads to parameterize GFlowNet parameters Pr, PB, F is used, and is based on the published codebase of (Malkin) al. 2022).
All models are trained with Adam optimizer with a batch size of 16 for a total of 10 trajec- tories (62500 batches) and 3 seeds. The learning rate is chosen from {0.001, 0.005, 0.01, 0.03) for the forward and backward policies PF and PB with the trajectory balance objective (Malkin al., 2022]. and the learning rate of the Zg is 10x learning rates of Pp and PB. Reward tem- peratures values of {Benx = 1.0, Begx = 0.25, 8sN For intrinsic rewards, we choose RND rewards with the intrinsic reward coefficient chosen from (0.00005, 0.00001, 0.0005, 0.0001, 0.005, 0.001, 0.05, 0.01).
8D 1.0, B, = 1.0} are used.
9.3 EXPERIMENTAL DETAILS: BIT SEQUENCES
For this task, a maximum sequence length of 120 is considered for E 21,2,4,6, 6.8. 10) and the respective vocabulary size of 2h All methods a Transformer based architecture (Vaswani et al, 2017]. with 3 layers, 64 dimension, and 8 attention heads and the definition modes M, set of test sequences, distance metric the same as in MMakin et al. 2022). Reward temperatures values of {8.aN = 1.0, BeSN = 1.0, 35N = 3.0, BAN 3.0, = 1.0} are used. Adam optimizer is used for all methods with batch size of 16 and 50000 iterations
3 seeds. The learning rate is chosen from (0.001, 0.005, 0.01, 0.03) for all model parameters (except for 7g), and the learning rate for
Ze is chosen to be 10x the learning rate of all other parameters.
9.4 EXPERIMENTAL DETAILS: SMALL MOLECULE
All the experiments expand on the published code of Bengio et al. (2021a) and Malkin et al. (0022) All models are trained for a maximum of 50000 batches of 4 trajectories each. The proxy model giving the reward, the held-out of molecules used to compute the correlation metric, and the GFlowNet model architecture and its hyperparameters are taken from Bengio et al. (2021a) and
Malkin al. (2022).
13

---

Published as a conference paper at ICLR 2025
x
x 8
x
16 16
16 X
16
16 x 16
10
6 10
10"7
L1 crrr
elerrr
10
9x000
5 10
9 10
L1
8x10
SA-GFN
GAFN
x 10-1
7x
AND
8x 10
10a
10a
10a
102
Figure 9: SA-GFN performs a better exploration and learns the underlying distribution much faster when compared to other methods even in high-dimensional HyperGriids (d 6, 8) with high- dimension action spaces.
10
ADDITIONAL EXPERIMENTS
10.1
HYPERGRIDS SPANNING HIGHER DIMENSIONS
In order to test exploration capabilities of SA-GFN in the case of high- -ighesiiinsiinall action spaces and multi-dimensional HyperGrids, we conducted experiments on the Sparse HyperGrid from Section 5.1.2 and use the same baselines as reported in Sections 5.1.1 and 5.1.2
We found that SA-GFN outperforms all other methods even when the HyerGrid is extended to multiple dimensions and the dimensionality of the action space is increased, Figure(9 We can see that SA-GFN explores and matches the underlying distribution much better than the other baselines.
10.2 ADDITIONAL REWARD SETTINGS (HYPERGRID)
8x
32
54
10-1
18
107
207
317
100
107
201
26.x 95
x52
16-1
CAFN
Figure 10: Harder-to-Explor HyperGrid Setting: For harder explore grid configu- ration {Ro le - 5, R 1.0, R2 3.0). the proposed method, SA-GEN, outperforms all other strong basclines. highlighting its efficient exploration properties over a wide range hard-to-explore tasks and reward structures.
To evaluate exploration over wide range reward structures, we include an additional HyperGrid reward setting with (Ro = 1, R = 1.0, R2 3.0) that presents more challenging exploration problem. We find that the proposed method, SA-GFN, again outperforms all other strong baselines, see 10.2) and achieves much better match to the underlying true distribution.
10.3 EASY ADAPTABILITY TO OTHER INTRINSIC REWARDS
In order to highlight the versatility and case adaptability the proposed method over intrinsic rewards other than regular RND (Burda al. 2018), add Noveld (Zhang al.) 2021)
14

---

Published as a conference paper at ICLR 2025
intrinsic bonus to encourage exploration in the Sibling Network. The proposed method SA-GFN outperforms all other baseline methods, for both non-zero sparse reward setting, see Figure II and zero sparse reward settings, Figure T2 showcasing its inclusiveness in terms of different types of intrinsic rewards.
8
16 16
32
48 48
10
10
emer
10
1X
L1 anror
3 10
10
10
10-3
10P
10
10a
10a
10a
10'
64 x 64
80 x 80
96 96
112 x 112
10-4
10"1
eitor
1. eirrrrr
11 croo
10-1
10
128 x 128
136 136
144 144
152 x 152
10-4
10-4
eroor
11 ernor
L3 ortor
SA/GFNIR)
GAFN
10
10-1
10a0
10
RND
GIN
10P
10
10a
10a
10a
10a
Figure 11: Adaptability to Different Intrinsic Rewards: (non-zero sparse reward) The proposed method, SA-GFN, is adaptable to wide variety of intrinsic rewards. For example, SA-GFN when used with Noveld, labeled as SA-GFN(IR), outperforms all other baseline methods, achieving much better exploration of the underlying spare non-zero reward distribution.
128
SA-GEN
10-4
SA-GFNOR)
entor
eroo
10
11 eror
GAIN
10-2
GFN
10
10-6
10
10-3
10000
Figure 12: Adaptability to Different Intrinsic Rewards: (zero sparse reward) The proposed method, SA-GFN, is adaptable wide variety of intrinsic rewards. For example, SA-GFN when used with Noveld, labeled as SA-GFN(IR), outperforms all other baseline methods, achieving much better exploration of the underlying sparse zero-reward distribution.
10.4
EXTENSION TO OTHER GFLOWNET TRAINING OBJECTIVES
The proposed method, SA-GEN can used with any of the training objectives introduced in Section
3.2
Results using Trajectory Balance (Malkin et al. 2022)
provided in Section 5.1 and Section
5.2
while Flow Matching objective (Bengio et al. (02 la) is used Section 5.3 Here, we include
results from the Detailed Balance objective (Bengio et 2021b) for completeness, Figure
15

---

Published as a conference paper at ICLR 2025
10-4
L1 cnur
L3 emor
GA-GITTBI
10
10
AND
GTM.
15" 10'
10
10a
107
10
107
107
10
10a
10a
10
10
107
10
Figure 13: Extension to other GFlowNet Training Objectives:
The proposed method, SA-GFN,
can be trained using any the GelowNet training objectives. Here we include the Detailed Balance objective for sparse non-zero reward setting for completeness.
10.5 COMPARISON WITH SAC (HYPERGRID)
We add the SAC (Haarojj al. 2018 baseline for HyperGrid from Section 5.1 for completeness, see Figure [4] and Figure 13 We find that without adding large replay buffer, SAC not able to explore over difficult and sparse reward settings, while the proposed method SA-GFN performs stronger than any of the previous baselines across all reward configurations of the HyperGrid.
10
10-3
4x
eroor
10-1
wrter
10-1
13 emtr
10-1
30
1.0
10a
1.04
10P
107
10a
10a
1.0
10"
10a
1.0
30*
128
128
136 x 136
152% 152
10-1
10
SA-
GAPN
RND
GFN
10-5
100
302
10a
10"
107
10f
10a
107
10
30a
107
20
Figure 14: Soft Actor Critic (SAC) method is not able explore well in difficult sparse non-zert reward settings, while the proposed method SA-GFN explores much better as compared to all other baseline methods.
x
32
64
128 x 128
10-3
SA-GFN
10-1
10
10-1
eme
10ml
GIN
10
SAC
10:4
10
Figure 15: Soft Actor Critic (SAC) method is not able to explore well in difficult sparse zero- reward settings, while the proposed method SA-GFN explores much better as compared all other baseline methods.
16

---

Published as a conference paper at ICLR 2025
10.6 EXPLORATION USING TEMPERED LOGITS
We expand on our ablation study fom Section and provide additional results to show that just using tempered logits for exploration is sufficient for sparse and difficult explore reward settings, Figure T6 for sparse reward structures and Figure 17 for low-reward structures. Better exploration strategies, such as that in the proposed method, SA-GFN, become essential to ensure an efficient exploration of complex spaces and reward structures.
x
32
64
128 128
10-1
SA-GEN GAPN AND GFN
10-
10a5
10
19ta
10
10
10-7
4x
10000
20000
10000
20000
10000
20000
10000
20000
Figure 16: Tempered logits not sufficient effectively explore difficult zero-reward structures
8
16 x 16
32
48
10
4x10--
11 erroe
2x100
10-
10m3
107
1.04
10P
1.0
102
1.0
30*
1.07
30*
64 x 64
80 x 8D
96
112 112
30-4
wror
L1 emnr
enor
10
10-1
1.04
10a
1.2
128 128
136 x 136
144 144
152 152
10
10-1
1. ener
SAV
GAFN
10
10-4
10:2
30
GFN temp
Figure 17: Tempered logits are not sufficient to effectively explore difficult to explore non-zero reward structures
10.7 VISUALIZING LEARNT DISTRIBUTIONS FOR HYPERGRID
In Figure 18 and Figure 19 we further visualize the learnt empirical distributions by SA-GEN and all other baseline methods presented in Section 5.1.2 and Figure
We find that SA-GFN discovers all four modes ing very large grid sizes, while other baseline methods suffer and are able to discover only limited number modes as grid size increases and the exploration problem gets harder. Moreover, these vi- sualizations confirm that SA-GFN not only discovers all the modes, but also learns and matches the true distribution well.
distribution
HyperGrid sizes, includ-
17

---

Published as a conference paper at ICLR 2025
8 (SA-GFN)
8 x B (GAFN)
x 8: (AND)
8 (GFN TB)
x 8 (PPO)
4-
2
16 16: (FA)GFN)
16 16: (GAFN)
16 16: (RND)
16 16 (GPN TB)
16 (PPO)
5.0
5.0
5.0
5.0
5.0
7.5
7.5
7.5
7.5
7.5
10.0
15.0
15.0
15.0
15.0
15.0
10
15
10
15
100
15
10
15
1.0
15
32 x 32, (SA-GFN)
32 32 (GAFN)
32 32: (RND)
32 x32 (GFN TB)
32 32 (PPO)
0-
0
0+
5-
15
15-
15
20
20
20
20-
20
1.0
x 48 (SA-GFN)
48 (GAFN)
48 (RND)
48 48: (GFN TB)
48 (PPO)
20
20
20-
20
30
30
(SA-GFN)
64 (GAFN)
64: (RND)
64 (GFN TB)
64 64: (PPO))
20-
20
30
20
30-
30
60
60
60
Figure 18: Visualization of the learnt empirical distributions for SA-GFN and all other methods corresponding to Figure We can see that as the size of the HyperGrid increases, SA-GFN is able discover all the four modes and learns the underlying distribution well, while other methods tend to suffer, especially as the exploration problem gets harder with increasing grid sizes.
18

---

Published as a conference paper at ICLR 2025
80 x 80 (FSA-GN)
80 (GAFN)
80 x 80: (RND)
80 x [GFN TB)
x 80 (PPO)
20
20
20
20-
40
40
40
40-
20
60
60
20
40
60
20
40
60
20
40
60
20
40
60
96 x 96 (SA-GFN)
96 96 (GAFN)
96 96: (RND)
96 96 (GFN TB)
96 96: (PPO)
20
20
20
50
60
60
60
60
25
50
25
75
50
25
112 112: (5A-GG)))
112 (GAFN)
112 112 (AND)
112 112: (GFN
112 (PPO)
40
40
60
100
100
100
100
100
50
(SA-GFN)
(AND)
25
25
25
25
25
50
50-
50-
50-
50-
75-
75
75
100
100
100
100
100
125
125
125
125
125
50
100
50
50
100
50
100
50
100
152 152 (SA-GFN)
152 152 (GAFN)
152 x 152 (RND)
152 x 15:1: (GIN TB)
152 x : (PPO)
50
50
50-
50-
50-
150
150
150
150
150
50
100
150
50
100
150
100
150
100
150
50
1.00
150
Figure 19: Visualization of the learnt empirical distributions for SA-GFN and all other methods corresponding to Figure g We can see that as the size of the HyperGrid increases, SA-GFN is able discover all the four modes and learns the underlying distribution well, while other methods tend to suffer, especially as the exploration problem gets harder with increasing grid sizes.
19


Reviews:
[{"review_id": "TfS4h9lomh", "reviewer": "Reviewer_XUfJ", "summary": "This paper is about exploration when learning generative flow networks. This is particularly relevant in a setting with sparse extrinsic terminal rewards. The proposed methods uses second, sibling, network along the main behavior network and use an intrinsic reward scheme (random network destination) for exploration. Using the two networks, it is possible to share re-labeled training data from the sibling network which is trained with the non-stationary intrinsic reward to the main behavior network with learns with the true reward. A set of abstract experiment evaluate the claims of better learning and the results show success of the proposed method. Ablation studies are also provided.", "questions": "Given that RL and generative flow networks are not exactly the same, is there a way to improve RND for the generative flow network setting?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper addresses a relevant and interesting problem. The choice of an intrinsic reward methods and the parallel sibling network is interesting and novel for generative flow networks. Most of the experiments are fairly abstract but relevant.", "weaknesses": "The related work is short and makes it hard to assess novelty and relevance. The description of generative flow networks in 3.1 is short it does not become clear what the policy does and how it interacts with states and objects. RND is far from the only intrinsic reward scheme in reinforcement learning and it would strengthen the contribution to not compare against several alternative schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BoaLkhfEdG", "reviewer": "Reviewer_GhQ4", "summary": "This work proposes a decoupled GFlowNet (Sibling) parallel with the main GFlowNet (Behavior) to improve the exploration of the learning process with sparse rewards. The auxiliary GFlowNet is trained on the original rewards combined with novelty-based intrinsic rewards, while the main network is trained only on the original rewards. Due to GFlowNet's off-policy property, the main network can use both its on-policy data and the data sampled using the sibling network. The proposed method achieves strong results in the tasks considered.", "questions": "- How is the intrinsic reward calculated? According to equation 2, the intrinsic reward depends on the terminal states and all intermediate states. Would it be different if it is only calculated for the terminal states?\n- Why do some L1 error curves of baselines increase in Fig. 4, such as 8x8x8x8? I have not observed this in other related work. Please explain.\n- Out of my own curiosity, do the authors see potential methods for using the decoupled intrinsic reward network in RL tasks, such as navigation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and written clearly. The storyline from the original GFlowNet through GAFN with additional intrinsic reward to SA-GFN with a decoupled network is clear and makes a lot of sense. The decoupled networks and relabeled rewards are simple yet effective methods. It is interesting to see the performance improvement by using the trajectories sampled with novelty exploration while only using the original rewards. Although intrinsic rewards are not a new idea for reinforcement learning, decoupling its samples and rewards for training has rarely been explored. The experiments are comprehensive.", "weaknesses": "- In the introduction, the authors claim to “expand the set of previous exploration benchmarks to include non-zero…” I assume they mean the experiment environment in section 5.1.2. It seems unfair to say so because the HyperGrid with non-zero rewards has already been used in previous work [1].\n- Many references are incomplete, without information like the publisher.\n- For the experiment results in Fig. 4, it would be better to plot the curves for more steps so that we can see the plateaued values of baselines, such as in 32x32, 64x64, 96x96, and so on.\n- Last paragraph of 5.2. The indices of the three metrics and findings for the metrics are the same (a,b,c), which makes reading difficult.\n\n[1] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS), 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EbBraDMnJ8", "reviewer": "Reviewer_kk92", "summary": "This paper introduces a novel architecture named Sibling Augmented Generative Flow Networks (SA-GFN), to improve the exploration capabilities in Generative Flow Networks (GFlowNets). The authors propose using a dual-network architecture, which consists of a Behaviour Network and a exploratory network (Sibling Network), which uses intrinsic rewards. By decoupling the two networks, it allows the Sibling Network to perform exploratory tasks, providing diverse data samples by relabelling the rewards with the true ones to the Behavior Network. With this design, the training objective of the behaviour network is stationary. Results shown in the selected datasets that SA-GFN outperforms baselines.", "questions": "1. How does SA-GFN compared with other SOTA intrinsic methods, such as NovelD  then?\n2. Is the sibling network also periodically updated using the data collected by the behaviour policy or directly using the behaviour policy's weight? If not, does this mean, by only trianing on the data collected by the RND, the sibling network is already sufficiently exploring the environment?\n3. Currently, the training frequency of the sibling network and behaviour network seems to be 1:1. How does this ratio influence the training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and the core idea is presented in a very clear way.\n2. The idea itself is simple yet helpful for combining various intrinsic rewards methods and GFlowNets.\n3. In the selected dataset, the proposed methods achieve the best performance compared to all baselines.\n4. A detailed ablation study has also been provided.", "weaknesses": "1. Intrinsic reward method used as the baseline covers only RND. Other intrinsic reward methods such as NovelD, DEIR and others, which are shown better performance compared to RND are not included. In the appendix, NovelD is used as intrinsic reward for the sibling network but itself is still missing as a baseline\n2. The tested environment is relatively simple. Testing in more complicated/hard-exploration tasks would be desirable.\n3. The training of the Sibling network is unclear to me. See question section\n4. An ablation study of the hyperparameter tuning of SN-GFN would be great. See question 3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E5x3Gvyjhu", "reviewer": "Reviewer_t2LW", "summary": "Sibling Augmented Generative Flow Networks are proposed to solve the exploration problem in sparse reward reinforcement learning. This approach decouples the exploration policy from the exploitation policy: the exploitation policy is responsible for maximizing the cumulative extrinsic reward (enlarging the trajectory of the generation of the labeled exploration policy at the data level), and the exploration policy is responsible for maximizing the combination of the cumulative intrinsic and extrinsic reward. The authors claim that this can enhance the performance and stability of policy learning. A series of experiments verify the proposed method.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The sibling policy and the algorithm is new but can be quite straightforward to come up with.\n3. The experiments part provides a wide range of comparisons with existing methods.", "weaknesses": "1. The experimental environment is too monotonous. There is only a grid world, and there are many other environments in the sparse rewards-related research, such as Montezuma's Revenge and some robot manipulation simulation tasks in Mujoco. I am curious to see if SA-GFN would be as effective for other tasks. If the author thinks it is difficult to provide corresponding experiments, I hope there will be a qualitative analysis or discussion.\n2. If I understand correctly, the author seems to think that the GFlowNets presented in this paper do not fall under the category of reinforcement learning, which I have reservations about. The goal of GFlowNets is also to maximize the cumulative discount reward, and the training paradigm of off-policy RL is also used. This is the category of reinforcement learning in my opinion. I hope the author can further explain the positioning of the method proposed in this paper.\n3. In the training of SA-GFN, the critic network is not used. In general, without the critic network, policy training will be unstable because of the high variance. This paper does not provide related discussion or experiments on this aspect. I would like to hear the author's thoughts on this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)