Pre-Training and Fine-Tuning Generative Flow Networks
Authors: Ling Pan
Moksh Jain
Kanika Madan
Yoshua Bengio
Venue: ICLR 2024 spotlight (2024)
Keywords: Generative Flow Network (GFlowNets)
Pre-train
Goal-conditioned
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a fundamental paradigm for leveraging the potential of pre-training in GFlowNets for efficient adaptation to downstream tasks.

Abstract: Generative Flow Networks (GFlowNets) are amortized samplers that learn stochastic policies to sequentially generate compositional objects from a given unnormalized reward distribution.
They can generate diverse sets of high-reward objects, which is an important consideration in scientific discovery tasks. However, as they are typically trained from a given extrinsic reward function, it remains an important open challenge about how to leverage the power of pre-training and train GFlowNets in an unsupervised fashion for efficient adaptation to downstream tasks.
Inspired by recent successes of unsupervised pre-training in various domains, we introduce a novel approach for reward-free pre-training of GFlowNets. By framing the training as a self-supervised problem, we propose an outcome-conditioned GFlowNet (OC-GFN) that learns to explore the candidate space. Specifically, OC-GFN learns to reach any targeted outcomes, akin to goal-conditioned policies in reinforcement learning. 
We show that the pre-trained OC-GFN model can allow for a direct extraction of a policy capable of sampling from any new reward functions in downstream tasks.
Nonetheless, adapting OC-GFN on a downstream task-specific reward involves an intractable marginalization over possible outcomes. We propose a novel way to approximate this marginalization by learning an amortized predictor enabling efficient fine-tuning.
Extensive experimental results validate the efficacy of our approach, demonstrating the effectiveness of pre-training the OC-GFN, and its ability to swiftly adapt to downstream tasks and discover modes more efficiently.
This work may serve as a foundation for further exploration of pre-training strategies in the context of GFlowNets.

Published as a conference paper at ICLR 2024
PRE-TRAININ AND FINE-TUNING GENERATIVE FLOW NETWORKS
Ling Pan Moksh Jain Kanika Madan Yoshua Bengio 2.34 Hong Kong University of Science and Technology Mila Québec AI Institute Université de Montréal *CIFAR Al Chair {penny. ling. pan) gmail. com
ABSTRACT
Generative Flow Networks (GFlowNets) are amortized samplers that learn stochastic policies sequentially generate compositional objects from a given unnormalized reward distribution. They can generate diverse sets of high-reward objects, which is important consideration in scientific discovery tasks. How- ever, as they are typically trained from a given extrinsic reward function, it re- mains important open challenge about how to leverage the power of pre- training and train GFlowNets in an unsupervised fashion for efficient adaptation to downstream tasks. Inspired by recent successes of unsupervised pre-training in various domains, we introduce a novel approach for reward-free pre-training of GelowNets. By framing the training as a self-supervised problem, we propose an condite-cndddtoonedd GFlowNet (OC-GFN) that learns to explore the candidate space. Specifically, OC-GFN learns to reach any targeted outcomes, akin to goal- conditioned policies in reinforcement learning. We show that the pre- pre-trained OC- GFN model can allow for direct extraction of policy capable of sampling from any new reward functions in downstream tasks. Nonetheless, adapting OC-GFN on a downstream task-specific reward involves an intractable marginalization over possible outcomes. We propose novel way to approximate this marginalization by learning an amortized predictor enabling efficient fine-tuning. Extensive exper- imental results validate the efficacy of our approach, demonstrating the effective- ness of pre-training the OC-GFN, and its ability to swiftly adapt to downstream tasks and discover modes more efficiently. This work may serve foundation for further exploration of pre-training strategies in the context of GFlowNets.
INTRODUCTION
Unsupervised learning on large stores of data on the internet has resulted in significant advances in variety of domains (Howard & Ruder, 2018; Devlin et al., 2018; Radford et al., 2019; Henaff, 2020). Pre-training with unsupervised objectives, such as next-token prediction in auto-regressive language models (Radford et al., 2019), on large-scale unlabelled data enables the development of models that can be effectively fine-tuned for novel tasks using few samples (Brown et al., 2020). Unsupervised learning at scale allows models to learn good representations, which enables data- effticeeffiient adaptation to novel tasks, and is central to the recent development towards larger models.
On the other hand, in the context of amortized inference. Generative Flow Networks (GFlowNets; Bengio al., 2021) enable learning generative models for sampling from high-dimensional distribu- tions over discrete compositional objects. Inspired by reinforcement learning (RL), GFlowNets learn stochastic policy sequentially generate compositional objects with probability proportional to a given reward, instead of reward maximization. Therefore, GFlowNets have found success in ap- plications to scientific discovery problems generate high-quality and diverse candidates (Bengio et al., 2021; Jain et al., 2023a;b) as well as alternatives to Monte-Carke Markov chains and variational inference for modeling Bayesian posteriors (Deleu et al., 2022; van Krieken et al., 2022).
As a motivating example, consider the drug discovery pipeline. A GFlowNet can be trained to generate candidate RNA sequences that bind to a target using as reward the binding affinity of the RNA sequence with the target (Lorenz al., 2011; Sinai al., 2020) (which can be uncertain and imperfect based on the current understanding of the biological system and available experimental data). However, there is way efficiently adapt the GelowNet to sample RNA sequences binding

---

Published as a conference paper at ICLR 2024
to different target of interest that reflects new properties. Unlike human intelligence, GelowNets currently lack the ability to leverage previously learned knowledge to efficiently adapt to new tasks with unseen reward functions, and need to be trained from scratch to learn a policy for matching the given extrinsic reward functions for different tasks. Inspired by the success of the unsupervised pre-training and fine-tuning paradigm in vision, language, and RI. (Jaderberg et al., 2016; Sekar et al., 2020) domains, it natural to ask how can this paradigm benefit GelowNets. As a step in this direction, we propose a fundamental approach to realize this paradigm for GFlowNets.
In this paper, we propose novel method for freer unsupervised pre-training of GFlowNets. We formulate the problem of pre-training GFlowNets as a self-supervised problem of learning an outcome-condittoneed GFlowNet (OC-GFN) which learns to reach any outcome (goal) tional understanding of the environment (akin to goal-conditioned RL (Chebotar et al., 2021)). The reward for training this OC-GFN is defined as the success of reaching the outcome. Due to the inher- ent sparse nature of this task-agnostic reward, it introduces critical challenges for efficient training OC-GFN in complex domains, particularly with nigher-dimensional outcomes and aonghorizon problems, since it is difficult to reach the outcome to get meaningful reward signal. To tackle these challenges, we introduce a novel contrastive learning procedure to train OC-GFN to effectively han- dle such sparse rewards, which induces an implicit curriculum for efficient learning that resembles goal relabeling (Andrychowicz et al., 2017). To enable efficient learning in long-horizon tasks, we further propose goal teleportation scheme to effectively propagate the learning signal to each step.
a func-
remarkable result is that one can directly convert this pre-trained OC-GFN to sample proportion- ally to new reward function for downstream tasks (Bengio et al., 2023). It is worth noting that in principle, this can be achieved even without re-training the policy, which is usually required for fine-tuning in RL, as it only learns a reward-maximizg formation. Adapting the pre-trained OC-GFN model to a new reward function, however, involves an intractable marginalization over possible outcomes. We propose a novel alternative by learn- ing a predictor that amortizes this marginalization, allowing efficient fine-tuning of the OC-GFN to downstream tasks. Our key contributions can be summarized as follows:
policy that may discard many useful in-
We propose reward-free free pre-training for GFlowNets as training cuncome-conditionel GFlowNet (OC-GFN) that learns to sample trajectories to reach any outcome.
We investigate how to leverage the pre-trained OC-GFN model to adapt to downstream tasks with new rewards, and we also introduce efficient method to learn an amortized predictor to approximate an intractable marginal required for fine-tuning the pre-trained OC-GFN model.
Through extensive experiments on the GridWorld domain, we empirically validate the efficacy of the proposed pre-trane training and fine-tuning paradigm. We also demonstrate its scalability to larger- scale and challenging biological sequence design tasks, which achieves consistent and substantial improvements over strong baselines, especially in terms of diversity of the generated samples.
2 PRELIMINARIES
Given a space of compositional objects X. and non-negative reward function R X' + R+, the GFlowNet policy и trained towards sampling objects x E X' from the distribution defined by R(z), i.c., Tz)) x R(r). The compositional objects are each sampled sequentially, with each step involving the addition of a building block a E AA (action space) to the current partially constructed object s E S (state space). We can define a directed acyclic graph (DAG) g = (S.A} with the partially constructed objects forming nodes of G. including special empty state so. The edges are s + s where s' is obtained by applying an action a E AA to s. The complete objects X' are the terminal (childless) nodes in the DAG. The generation of an object æ E X' corresponds trajectories in the DAG starting from 8o and terminating in a terminal state 8, = J. i.e., 7 7 (so ->
complete
> I). We assign non-negative weight, called state flow F(s) to each state S. The forward policy (ssss)) [s) is a collection of distributions over the children of each state and the backward policy (s(()ss) is collection of distributions over the parents of each state. The forward policy induces distribution over trajectories Pp (T), and the marginal likelihood of sampling a terminal state is given by marginalizing over trajectories terminating in x. PJ (x) = Er= -x-=(()-----))) Pr(7). Gelo solve the problems of learning a parameterized policy Pr (- 0; such that Pf (1) x R(x).
Training GFlowNets. We use the detailed balance learning objective (DB; Bengio al., 2023) to learn the parameterized policies and flows based on Eq. (1), which considers the flow consistency constraint in the edge level (i.e., the incoming flow for edge s' matches the outgoing flow).
2

---

Published as a conference paper at ICLR 2024
When it is trained to completion, the objective yields the desired policy.
(I) As in reinforcement learning, exploration is key challenge in GFlowNets. Generative Augmented Flow Networks (ets,owNets;; Pan et al., 2023b) incorporate intrinsic intermediate rewards repre- sented as augmented flows in the flow network to drive exploration, where ri s') is specified by intrinsic motivation (Burda et al., 2018), yielding the following variant of detailed balance.
- s' E A,
Pss)ss = F(s') P(s'))
vs + s' E A,
((s))=(('(s)) = (B))s))))) r(s 8).
RELATED WORK
GFlowNets. While several learning objectives have been proposed for improving credit assignment and sample efficiency in GFlowNets (Bengio et al., 2021; 2023) such as detailed balance (Bengio et al., 2023), sub- uub-trajectory balance (Madan et al., 2022) and forward-looking objectives (Pan al., 2023a), they need to be trained from scratch with a given reward function, which may limit its ap- plicability to more practical problems. Owing to their flexibility, GFlowNets have been applied to wide range of problems where diverse high-quality candidates are need, such as molecule genera- tion (Bengio et al., 2021), biological sequence design (Jain et al., 2022), combinatorial (Zhang al., 2023a;b), and multi-objective optimization (Jain et al., 2023b). There have also been recent efforts in generalizing GFlowNets to handle stochastic environments including Stochastic GFlowNets (Pan al., 2023c) and Distributional GFlowNets (Zhang et al., 2023c), which are effective in han- dling stochasticity in transition dynamics and rewards; and also improving training efficiency of GFlowNets inspired by reinforcement learning (Pan al., 2020; Lau al., 2024) and evolutionary algorithms (Ikram et al., 2024).
Unsupervised Pre-Training in Reinforcement Learning. Following the progress in language modeling and computer vision, there has been growing interest in pre-training reinforcement learn- ing (RI.) agents in an unsupervised stage without access to task-specific rewards for learning rep- resentations. Agents typically learn set of different skills (Eysenbach et al., 2018; Hansen et al., 2019; Zhao et al., 2021; Liu & Abbeel, 2021), and then fine-tune the learned policy to downstream GFlowNets learn a stochastic policy match the reward
tasks. Contrary to reward-maximizann distribution. As we show in the next section, such a learned policy can be adapted to a new reward function even without re-training (Bengio et al., 2023).
Goal-Conditioned Reinforcement Learning. Different from standard reinforcement learning (RL) methods that learn policies or value functions based observations, goal-conditioned RL (Kael- bling, 1993) also take goals into consideration by augmenting the observations with an additional input of the goal and have been studied in number of prior works (Schaul et al., 2015; Nair et al., 2018; Veeriah et al., 2018; Eysenbach et al., 2020). Goal-conditioned RI. is trained to greedily achieve different goals specified explicitly as input, making it possible for agents to generalize their learned abilities across different environments.
4 PRE-TRAINING AND FINE-TUNING ATIVE FLOWW NETWORKS
In the original formulation, a GFlowNet need to be trained from scratch whenever it is faced with a previously unseen reward function (with consis- tent state and action spaces, as motivated in Sec- tion 1). In this section, we aim to leverage the power of pre-training in GelowNets for efficient adaptation to downstream tasks. To tackle this im- portant challenge, we propose a novel approach to frame the problem of pre-training GFlowNets as a self celf-supervised problem, by training an outcome- conditioned GFlowNet that learns to reach any input terminal state (outcome) without task-specific rewards. Then, we propose to leverage the power of the pre-trained GFlowNet model for efficiently fine-tuning downstream tasks with a new reward function.
action
outcomes
action
intrinsic reward
Outcome-
positive samples
Unconditional
conditioned GFN
GAFN
state
conditional reward
Figure 1: The unsupervised pre-training
phase of outcton-conditionee Get.
4.1 UNSUPERVISED PRE-TRAINING STAGE
As GFlowNets are typically learned with a given reward function, it remains an open challenge for how to pre-train them in reward-free fashion. We propose novel method for unsupervised
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Reward-free Pre-training of Unsupervised GFlowNets.
Require: GAFlowNet F (s); P (s']s,, Pg (sls'); fixed target network oi predictor network d; Outcome-conditioned GFlowNet (OC-GFN) FC (sly), ( (s'(s.y), ( (s]s',
1: for each training step t = 1 to I"' do
2: Collect a trajectory -+ = { + ..... } with PV
3: I/ Update the outcome-conditione GFlowNet model
4: yt f(s+), -. 11)=tt()===)==) < ((((s)) = y }+}= =1
5: Update OC-GFN towards minimizing Eq. (5) with r+ and l(*
6:
Collect a trajectory T = { + .... + + with Pf conditioned on y
7:
y f(s ). R(s ly + [[f(s)) y
8:
Update OC-GFN towards minimizing Eq. (5) with T and Ry)
9:
If Update the unconditional GAFlowNe model
10: Update GAFlowNet towards optimizing Eq. (2) based on r+ and r; + l1x(s) - e()]l
Update the predictor network @ towards minimizing [[o(s) 21>]l22
pre-training of GFlowNets without task-specific rewards. We formulate the problem of pre-training GFlowNets as a self-supervised problem of learning an citcome--conditions which learns to reach any input target outcomes, inspired by the success of goal-conditioned rein- forcement learning in generalizing to a variety of tasks when it is tasked with different goals (Fang terminal
GFlowNett (OC-GFN)
al., 2022). We denote the outcome as y fs)), where f is an identity function and state, so the space of outcomes is the same as the state space X. What makes OC-GFN special is that, function of the outcome, i.e., R(s) r(v),
when fully trained, given reward R a posterior one can adapt the OC-GFN to sample from this reward, which can generalize to tasks with different rewards.
4.1.1 WUTCOME-CONDITTONN
GFLOWWETS (OC-GFN)
We extend the idea of flow functions and policies in GFlowNets to OC-GFN that can generalize to different outcomes y in the outcome space, which is trained to achieve specified y. OC-GFN can be realized by providing an additional input y to the flows and policies, resulting in outcome- conditioned forward and backward policies (F [s'|ss))) y) and PB(s|s', yy, and flows F(sly). The resulting learning objective for OC-GFN for intermediate states is shown in Eq. (3).
F( (s,(' y) ((s (s'ly)) PP) v).
Outcome generation We can train OC-GFN by conditioning outcomes-conditioned flows and policies on a specified outcome y, and we study how to generate them autotelically. is worth noting that we need to train it with full-support over We propose to leverage GAFN (Pan et al., 2023b) with augmented flows that enable efficient reward-free exploration purely by intrinsic moti- vation (Burda et al., 2018). In practice, we generate diverse outcomes y with GAFN, and provide them to OC-GFN to sample an outcome-conditioned trajectory
(5o.... Sn.). The effect of the GAFN is studied in Appendix C.2, which is critical in large and high-dimensional problems as it affects the efficiency of generating diverse outcomes. The resulting conditional reward is given as R(s,ly) = 1 [s,,) = y). Thus, OC-GFN receives a zero reward it fails to reach the target outcome, and positive reward otherwise, which results in learning outcome-att outcome-achieving
achieving policy.
Contrastive training However, can be challenging to efficiently train OC-GFN in problems with large outcome spaces. This is because it can be hard to actually achieve the outcome and obtain meaningful learning signal owing to the sparse nature of the rewards most of the conditional rewards R(sn (2) will be zero if f(x)) # when OC-GFN fails to reach the target.
To alleviate this we propose a contrastive learning objective for training the OC-GFN. After sam- pling a trajectory -+ = {s] + + ** } from unconditional GAFN, we first train an OC-GFN based on this off-policy trajectory by assuming it has the ability to achieve y+ = = when condi- tioned on yt Note that the resulting conditional reward Iv*) = [{f(s = -
this case, as all correspond to successful experiences that provide meaningful learning signals to OC- GFN. We then sample another on-policy trajectory -- = {sp .... S, from OC-GFN by con- ditioning it on y, and evaluate the conditional reward by R() = ({ = . Although most of R(=, l1rt) can be zero in large outcome spaces during early learning, we provide sufficient successful experiences in the initial phase, which is also related to goal relabeling (Andrychowicz et al., 2017). This can be viewed implicit curriculum for improving the training of OC-GFN.
4

---

Published as a conference paper at ICLR 2024
Algorithm 2 Supervised Fine-Tuning of Outcome-Conditione GFlowNets.
1: Initialize the numerator network (s'lss) and the GFlowNee-like predictor network Q(yls'
2: Obtain the pre- tre-trained outcome-con connommmod state F(s and forward policy Pp (s'(s.
3: for each training step t to T do
4: Collect a trajectory т = {so - ... sn with (s'sss
.. 5: Sample outcomes from a tempered/e-gee ergeddy version of ((+ss'ss)
6: Update N and Q towards minimizing Eq. (9)
7: Compute the policy (s*(ss ((slss)//)) N(s)(s)
Outcome teleportation The contrastive training paradigm can significantly improve learning ef- ficiency by providing bunch of successful trajectories with meaningful learning signals, which tackles the particular challenge of sparse rewards during learning. However, the agent may still suffer from poor learning efficiency in long-horizer tasks, as it cannot effectively propagate the success/failure signal back to each step. We propose novel technique, outcome teleportation, for further improving the learning efficiency of OC-GFN as in Eq. (4), which considers the terminal reward R(xly) for every transition (noting the binary nature of the rewards R).
PP s. s. F(s' () PB(s)s's R(zly)
This formulation can efficiently propagate the guidance signal to each transition in the trajec- tory, which can significantly improve learning efficiency, particularly in high-dimensionall outcome spaces, as investigated in Section 5.2. can be interpreted as form of reward decomposition in outcome-conditioned tasks with binary rewards. In practice, we train OC-GFN by minimizing the following loss function CoC-GFN (T, y) log-scale obtained from Eq. (4), i...
Σ (logF(ssv)) + log Pr (s'(s, y) - log F Fs'll) - log PB(s|s y) log R(zly))² (5)
Theoretical justification. We now justify that when OC-GFN trained to completion, it can suc- cessfully reach any specified outcome. The proof can be found in Appendix B.1.
Proposition 4.1. If LoC-GEN (T, y) for all trajectories T and outcomes y, then the outcome- conditioned forward policy Pr(s s, y) can successfully reach the target outcome y.
The resulting procedure for the reward-free unsupervised pre-training of OC-GFN is summarized in Algorithm and illustrated Figure 1.
4.2 SUPERVISED FINE-TUNING STAGE
In this section, we study how to leverage the pre-trained OC-GFN model and adapt it for downstream tasks with new reward functions.
remarkable aspect of GFlowNets is the ability to demonstrate the adaptability in generating task-specific policy. By utilizing the pre-trained OC-GFN model with sutcome-conditionel flows F(sly) and policies (s'(''ss s, 8), we can directly obtain a policy that samples according to new task-specific reward function Ii(s) = r(y) according to Eq. (6), which is based on (Bengio et al., 2023). A detailed analysis for this can be found in Appendix B.2.
PF(S'(s) Eg ((sy) Pr (s'(s.
(xr()))(sll))
Eq. (6) serves as the foundation for converting a pre-trained OC-GFN model to handle downstream tasks with new and even out-of-distribution rewards. Intriguingly, this conversion can be achieved without any re-training for OC-GFN on downstream tasks. This sets it apart from the typical fine- tuning process in reinforcement learning, which typically requires re-training to adapt a policy, since they generally learn reward-maximizing policies that may discard valuable information.
Directly estimating the above summation often necessitates the use of Monte-Carke averaging for making each decision. Yet, this can be computationally expensive in high-dimensional outcome spaces if we need to calculate this marginalization at each decision-making step. which leads to slow thinking. To improve its efficiency complex scenarios, essential to develop strategies that are both efficient while maintaining accurate estimations.
5

---

Published as a conference paper at ICLR 2024
action
extrinsic
reward
Amortized
Prodictor
extrinsic reward
Outcome conditioned GFN
Policy
state
Outcome conditioned GFN
Figure 2: Left: Convert the outcome- condiiiiond GFlowNet to downstream tasks without re- re-training the networks. Right: An efficient amortized predictor in the supervised fine-tuning phase.
Learning Amortized Predictor In this section, we propose novel approach to approximate this marginal by learning an amortized predictor.
Concretely, we propose to estimate the intractable sums in the numerator in Eq. (6) by numerator network (s'(s)
This would allow us to efficiently estimate the intractable sum with the help of (s'ls) that directly estimates N(-ss) for any state 8, which could benefit from the generalizable structure of outcomes with neural networks. We can also obtain the corresponding policy by PF (s'(s) = N(*'ss)/E+ Ns*lss). For learning numerator network (+1ss,, we need to have sampling policy for sampling outcomes y given states and next states s', which can be achieved by
E, r(y)
Q(uis' (sy) s.y)
((s'ss)
Therefore, we have the following constraint according to Eq. (7)
(ss ()) = (s)))
(8)
based on which we derive the corresponding loss function by minimizing their difference by training in the log domain, i.e.,
Lamortized = (log (ssl(s) (s) + log Q(vls'ss) log rogr(v) log F(sly) log (1(((1,,)))
Theoretical justification. In Proposition 4.2, show that N(s'(s) correctly estimate the summation when it is trained to completion and the distribution outcomes has full support. Proposition 4.2. Suppose that V(s, s', y), Lamortized 8', 0, then the amortized predictor N(s') s) estimates Ey ((sy)) (,y))
The proof can be found Appendix B.3. Proposition 4.2 justifies the use of the numerator network as efficient alternative for estimating the computationally intractable summation in Eq. (6).
Empirical validation. We investigate the converted/learned sampling policy in the standard GridWorld domain (Bengio et al., 2021), which has a multi-modal reward function. More details about the setup is in Appendix C. due to space limitation. We visualize the last x 10 samples from different basclines including training GFN from scratch, OC-GFN with the Monte Carlo-based estimation with Eq. (6) and the amortized marginalization approach with Eq. (9).
As shown in Figure 3(b), directly training GFN with trajectory balance (Malkin et al., 2022) can suffer from the mode collapse problem and fail to discover all modes of the target distribution in Figure 3(a). On the contrary, we directly obtain policy from the pre-trained OC-GFN model that samples proportionally to the target rewards as shown in Figure 3(c), while fine-tuning OC- GFN with the amortized inference could also match the target distribution in Figure 3(d), which validates its effectiveness in estimating the marginal.
Practical implementation. In practice, we can train the amortized predictor N--ss) and the sam- pling policy Q(-Is',s) in GFlowNet-like procedure. We sample (s, s') with N by interacting with the environment. This can also be realized by sampling from the large and diverse dataset D ob- tained in the pre-training stage. We then sample outcomes y from the sampling policy Q.11s/, which can incorporate its tempered version or with e-greedy exploration (Bengio et al., 2021) for ob- taining rich distributions of y. We then learn the amortized predictor and the sampling policy by

---

Published as a conference paper at ICLR 2024
(b)
Figure 3: Distribution of 2 10 samples from different baselines. (a) Target distribution. (b) GFN (from scratch). (c) OC-GFN (Monte Carlo-based). (d) OC-GEN (Amortizer-based)
one of these sampled experiences. Finally, we can derive the policy that will be converted to down- stream tasks by (s)(s = N s/)/ (s'ss). The resulting training algorithm is summarized Algorithm and the right part Figure 2.
EXPERIMENTS
In this section, we conduct extensive experiments to better understand the effectiveness of our ap- proach and aim answer the following key questions: (i) How do outcome-conditioned GelowNets (OC-GFN) perform in the reward-free unsupervised pre-training stage? (ii) What is the effect of key modules? (iii) Can OC-GEN transfer downstream tasks efficiently? (iv) Can they scale to complex and practical scenarios like biological sequence design?
5.1 GRIDWORLD
We first conduct series of experiments on GridWorld (Bengio et al., 2021) to understand the ef- fectiveness of the proposed approach. In the reward-free ffree unsupervised pre-training phase, train a (unconditional) GAFN (Pan et al., 2023b) and an outcome-condition without task task-specific rewards, where GAFN is trained purely from self-supervisce intrinsic rewards according to Algorithm We investigate how well OC-GFN learns in the unsupervised pre-training stage, while the supervised fine-tuning stage has been discussed in Section 4.2. Fach algorithm is run for 3 different seeds and the mean and standard deviation are reported. detailed description of the setup and hyperparameters can be found in Appendix C.I.
GFN (OC-GFN) on a map
Outcome distribution. We first evaluate the quality of the exploratory data collected by GAFN, as is essential for training OC-GFN with a rich distribution of outcomes. We demonstrate the sample distribution from GAFN in Figure 4(a), which has great diversity and coverage, and validates its effectiveness in collecting unlabeled exploratory trajectories providing diverse target outcomes for OC-GFN to learn from.
000M AX07
(b)
(c)
(d)
Figure 4: Results in GridWorld in different with different scales of the task. (a) Outcome distribu- tion. (b)-(d) Success rate of OC-GFN and its variants small, medium, and large maps, respectively.
Outcome reaching performance. We then investigate the key designs in pre-training OC-GFN by analyzing its success rate for achieving target outcomes. We also ablate key components to investigate the effect of contrastive training (CT) and outcome teleportation (OT). The success rates of OC-GFN and its variants are summarized in Figures 4(b)-(d) in different sizes of the map from small large. As shown, OC-GFN can successfully learn reach specified outcomes with success rate of nearly 100% at the end of training in maps with different sizes. Disabling the outcome teleportation component (OC-GFN w/o OT) leads to lower sample efficiency, particular in larger maps. Further deactivating the contrastive training process (OC-GFN w/o ÔT & CT) fails to learn well as the size of the outcome space grows, since the agent can hardly collect successful trajectories for meaningful updates. This indicates that both contrastive training and outcome teleportation are important for successfully training OC-GFN in large outcome spaces. It also worth noting that

---

Published as a conference paper at ICLR 2024
outcome teleportation can significantly boost learning efficiency in problems with combinatorial explosion of outcome spaces (e.g., sequence generation in the following sections).
Outcome-conditioned behaviors. We visualize the learned behaviors of OC-GFN in Figure 5, where the red and green squares denote the starting point and the goal, respectively. We find that OC-GFN can not only reach different outcomes with specified
high success rate, but also discover diverse trajectories leading outcome with the forward policy Pr. OC-GFN able to generate diverse tra- jectories, which is different from typical goal-conditioned RL approaches (Schaul et al., 2015) that usually only learn a single solution to the goal state. This ability can be very helpful for generalizing to downstream tasks with similar structure but subtle changes (e.g., obstacles in the maze) (Kumar et al., 2020).
Figure 5: Be-
haviors of OC- GFN.
Applicability to other objectives. We now investigate the versatility of our approach by building it upon another recent GelowNets learning objective with SubTB (Madan et al., 2022). We investigate the cutcome-reachiinn in Figure 6(a), while the learned sample distribution by fin-ttuiigg the outcome-conditione flows and policies of OC-SubTB with our amortized predic- tor is shown in Figure 6(b). As shown, OC-SubTB can successfully learn to reach target outcomes, where the proposed contrastive training and outcome teleportation method achieve consistent gains efficiency. Fine-tuning OC-SubTB can also discover all the modes, as opposed to training GFN (Malkin al., 2022) from scratch (Fig- ure 3). A more detailed discussion is in Appendix B.2.
performance of OC-SubTB the pre- vee-traning stage with a medium map as
(a) Figure 6: OC-SubTB in GridWorld.
(b)
5.2 BIT SEQUENCE GENERATION
We study the bit sequence generation task (Malkin et al., 2022), where the agent generates sequences of length n. We follow the same procedure for pre- pre-training OC-GFN without task-specifie rewards as in Section 5.1 with more details in Appendix C.I.
Analysis of the unsupervised pre-training stage. We first analyze how well OC-GFN learns in the unsupervised pre-training stage by investigating its success rate for achieving specified outcomes in bit sequence generation problems with different scales including small, medium, and large. As shown in Figures 7(a)-(c), OC-GFN can successfully reach outcomes, leading to a high success rate as summarized in Figure 7(d). It is also worth noting that it fails to learn when the outcome telepor- tation module disabled due to the particularly large outcome spaces. In such highly challenging scenarios, the contrastive training paradigm alone does not lead to efficient learning.
Size
Success rate
Small
98.37 + 0.33%
Medium
97.22 + 1.70%
Sccrume of
Large
93.75 4: 2.04%
(a) Small.
(b) Medium.
(c) Large.
(d) Summary.
Figure 7: Success rates in the bit sequence generation task with different scales of the task.
Analysis of the Supervised Fine-Tuning Stage. After we justified the effectiveness of training OC-GFN in the unsupervised pre-training stage,
CC-SM Marc
we study whether fine-tuning the model with the amortized approach can enable faster mode discovery when adapting downstream tasks (as described in Appendix C.I). We compare the proposed approach against strong baselines including training a GFN from scratch (Bengio -aatppllss--atinn----
et al., 2023), Q-Networks (DQN) (Mnih et al., 2015). We evaluate each method in terms of the number of modes discovered during the course of training as in previous works (Malkin et al., 2022). We summarize the normalized (between 0 and to facilitate comparison across tasks) number of modes averaged over each downstream task Figure 8, while results for each individual downstream task can be found in Appendix C.5 due space limitation.
(Dai et al., 2020), and Deep
Figure 8: Results bit sequence generation.
8

---

Published as a conference paper at ICLR 2024
We find that OC-GFN significantly outperforms baselines in mode discovery in both efficiency and performance, while DQN gets stuck and does not discover diverse modes due to the reward- maximizing nature, and MCMC fails to perform well in problems with larger state spaces.
5.3 TF BIND GENERATION
We study more practical task of generating DNA sequences with high binding activity with tar- geted transcription factors (Jain et al., 2022). We consider 30 different downstream tasks studied in (Barrera et al., 2016), which conducted biological experiments to determine the binding proper- ties between a range of transcription factors and every conceivable DNA sequence.
Analysis of the unsupervised pre-training stage. We analyze how well OC-GFN learns by evalu- ating its success rate. Figure shows that OC-GFN can successfully achieve the targeted goals with a success rate of nearly 100% in the practical task of generating DNA sequences.
Analysis of the supervised fine-tuning stage. We investigate how well OC-GFN can transfer to downstream tasks for generating TF Bind 8 se- quences. We tune the hyperparameters on the task PAX3_R270CC and evaluate the baselines on the other 29 downstream tasks. We inves- tigate the learning efficiency and performance in terms of the number of modes and the top-K scores during the course of training. The results in 2 downstream tasks are shown in Figure 10, with the full results in Ap- pendix C.6 due space limitation. Figures 10(a)-(b) illustrate the number of discovered modes while Figures 10(c)-(d) shows the top- K scores. We find that the proposed approach discovers more diverse modes faster and achieves higher top-K (K = 100) scores compared to baselines. We summarize the rank of each baseline in all 30 down- stream tasks (Zhang et al., 2021) in Appendix C.6, where OC-GFN ranks highest compared with other methods. We further visualize Figure 10(e) the 1-SNE plots of the TF Bind sequences discovered by transferring OC-GFN with the amortized approach and the more expensive but com- petitive training of GFN from scratch. As shown, training a GFN from scratch only focuses on limited regions while the OC-GFN has a greater coverage.
Figure 9: Success rate in TF Bind generation.
(a)
(b)
(d)
(e)
Figure 10: Results in the TF Bind sequence generation task in different downstream tasks.
5.4 RNA GENERATION
We study larger task of generating RNA sequences that bind to a given target introduced in (Lorenz et al., 2011). We follow the as Section 5.3, where details are in Appendix C.I. We consider four different downstream tasks from Vi- ennaRNA (Lorenz et al., 2011), I). each considering the binding energy with a different target as a reward. The left part in Figure 11 shows that OC-GFN achieves a success rate of almost 100% at the end of the pre- training stage. The right part in Figure 11 summarizes the averaged normalized (between and to facilitate comparison across tasks) performance in terms of the number of modes averaged over four downstream tasks, where the performance for each individual task is shown in Appendix C.7. We observe that OC-GFN is able to achieve much higher diversity than baselines, indicating that the pre-training phase enables the OC-GFN to explore the state space much more efficiently.
procedure
Figure 11: Results in RNA generation. Left: success rate. Right: number of modes (normalized).
5.5 ANTIMICROBIAL PEPTIDE GENERATION
To demonstrate the scalability of our approach even more challenging and complex scenarios, we also evaluate it on the biological task of generating antimicrobial peptides (AMP) with lengths 50 (Jain al., 2022) (resulting in an outcome space of 2050 candidates). Figure 12(a) demonstrates

---

Published as a conference paper at ICLR 2024
the success rate of OC-GFN for achieving target out- comes, which can still reach a high success rate in particularly large outcome spaces with our novel train- ing paradigm according to contrastive learning and fast goal propagation. Figure 12(b) shows the number of modes discovered, where OC-GFN provides consistent improvements, which validates the efficacy of OC-GFN successfully scale to much larger-scale tasks.
(a) Success rate. (b) Number of modes.
Figure 12: Results in AMP generation.
CONCLUSION
We propose a novel method for unsupervised pre-training GFlowNets through an outcome- conditioned GFlowNet, coupled with approach to efficiently fine-tune the pre-trained model downstream tasks. Our work opens the door for GFlowNets to pre-trained for fine-tuning for downstream tasks. Empirical results on the standard GridWorld domain validate the effective- ness of the proposed approach successfully achieving targeted outcomes and the efficiency amortized predictor. We also conduct extensive experiments in the more complex and challenging biological sequence design tasks to demonstrate proves learning performance compared with strong baselines including training a GFlowNet from scratch, particularly in tasks which are challenging for GelowNets to learn.
practical scalability. Our method greatly im-
ACKNOWLEDGGENTS
The authors acknowledge funding from CIFAR, Genentech, Samsung, and IBM.
ETHICS STATEMENT
We do not foresee any immediate negative societal impact of our work. Our work is motivated by the need for ways to accelerate scientific problems. However, we do note that there potential risk of dual use of the technology by nefarious actors (Urbina et al., 2022), as with all work around this topic.
REPRODUCIBILITY
All details of our experiments are discussed Appendix with a detailed description of the task, network architectures, hyper-parameters for baselines, and setup. We implement all baselines and environments based on open-source repositories described in Appendix C. The proofs proposi- tions can be found in Appendix B. Limitations are discussed in Appendix D.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. Advances in neural information processing systems, 30, 2017.
Luis A Barrera, Anastasia Vedenko, Jesse V Kurland, Julia M Rogers, Stephen S Gisselbrecht, Elizabeth Rossin, Jaie Woodard, Luca Mariani, Kian Hong Kock, Sachi Inukai, al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351 (6280): 1550-145,, 2016.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow net- work based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS), 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023. URL httt: html.
// wwwp....rr//ppprrs//22///// 22-0364 wttp:///////222222//2222
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894 2018.
10

---

Published as a conference paper at ICLR 2024
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint crXiv:2104.07749 2021.
Hanjun Dai, Rishabh Singh, Bo Dai, Charles Sutton, and Dale Schuurmans. Learning discrete energy-based based models via auxiliary-variin variiblee local exploration. Advances in Neural Information Processing Systems, 33:10443-10055. 2020.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2022.
Jacob Devlin, Ming-Way Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810... /010.0888000 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need:
Learning skills without a reward function. International Conference on Learning Representations (ICLR), 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint arXiv:2011. 1,09999 2020.
Kuan Fang, Patrick Yin, Ashvin Nair, and Sergey Levine. Planning to practice: Efficient online fine-tuning by composing goals in latent space. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4076-4083 IEEE, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor. International Conference Machine Learning (ICML), 2018.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv: arXiv:I906.000 05030, 2019.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, pp. 3192- PMLR, 2020.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-339 2018.
Zarif Ikram, Ling Pan, and Dianbo Liu. Evolution guided generative flow networks, arXiv preprint arXiv:2402.02 186, 2024.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2016.
Moksh Jain, Emmanuel Bengio, Alex Harrena Jarrid fector-Brooks. Bonaventure F.P. Dossou, Chanakya Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. Inter- national Conference on Machine Learning (ICML), 2022.
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Has Hao Liu, Alex Sonnandz--Garcin and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2023a.
Moksh Jain, Sharath Chandra Raparthy, Alex Hermandez- Kernannez-Garrii,, Jarrid gio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Confer- ence on Machine Learning, pp. 14631-14651 2023b.
Yoshua Ben-
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pp. 1094-8. Citeseer, 1993.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), 2015.

---

Published as a conference paper at ICLR 2024
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198-82118 2020.
Elaine Lau, Stephen Zhewen Lu, Ling Pan, Doina Precup, and Emmanuel Bengio. Qgfi: Control- lable greediness with action values. arXiv preprint arXiv:2402.05534, 2024.
Hao Liu and Picter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-6747 PMLR, 2021.
Ronny Lorenz, Stephan H Bernhart, Christian Höner Zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo Hofacker. ViennaRNA package 2.0. Algorithms for molecular biology, 6(1):26, 2011.
Kanika Madan, Jarrid Rector- Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bose. Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. arXiv preprini 2209. 12782, 2022.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. Neural Information Processing Systems (NeurIPS), 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature. 51 518(7540):529-533 -332299-33,,33 2015.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
Ling Pan. Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradients. Advances in Neural Information Processing Systems, 33:1 15:11177-11777, 2020.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. arXiv preprint arXiv:2302.0168) 2023a.
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative aug- mented flow networks. International Conference on Learning Representations (ICLR), 2023b.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. arXiv preprint arXiv:2302.09443 2023c.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprini 1707.06347, 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Daniildiis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint, 2020.
Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artifcial-intelligneee- powered drug discovery. Nature Machine Intelligence, 4(3): 2022.
Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomezak, Frank van Harmelen, and An- nette ten Teije. A-nesi: A scalable approximate method for probabilistic neurosymbolic inference.
arXiv preprint arXiv:2212. /2393, 2022.
12

---

Published as a conference paper at ICLR 2024
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-gen Many-goals reinforcement learning. arXiv preprint arXiv:186..0990 09605, 2018.
David Zhang. Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. arXiv preprint arXiv:2302.05444 2023a.
Dinghuai Zhang, Jie Fu, Yoshua Bengio, and Aaron Courville. Unifying likelihood-free inference with black-box optimization and beyond. arXiv preprint arXiv:2110.0337.. 2021.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint 2rX. 2023b.
Dinghuai Zhang, Ling Pan, Ricky TQ Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows. arXiv preprint arXiv:2302.05793 2023c.
Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.08107, 2021.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL DETAILS OF GFLOWNETS
A.1 DETAILS FOR THE TRAINING LOSSES
As introduced Section 2, the flow consistency constraint for detailed balance (DB) (Bengio al., 2023) is P = PB for - E A. In practice, we train DB in the log-scale for stability (Bengio al., 2021), i.e.,
)
Ys + A, (DB(s, log F(s')PP(s(s'))
(10)
and terminal flows F(z) match the corresponding rewards R(z). The learnalec objects here the state flow F(s), forward policy Pp (s'(s)) and backward policy PB(s|s'), which are parameter-
ized by neural networks. For GAFlowNets (Pan et al., 2023b) that consider intrinsic intermediate rewards ri + s') into the flow network, we also train the model in log-scale according Eq. (2), i.e.
(log F(*')P(((s') (
F(s)PP(s'(s)
Vs s' (DB
(11)
The learnable objects are the same as in DB, except that r: + s') is also learnable, which is represented by intrinsic motivation by random network distillation (Burda et al., 2018)).
B PROOFS
B. PROOF OF PROPOSITION 4.1
Proposition 4.1 If LOC-GEN(T, y) for all trajectories and outcomes y, then the outcome-
conditioned forward policy Pp(s'ss (,, y) can successfully reach any target outcome y.
Proof. As Loc-Or 8) is satisfied for all trajectories T {so- ... Sn = x} and outcomes y, we have that
F(so wy) II
PF v) II
(( t=
(12)
t=0
Since R(xly) is either or 0 in outcome-condition tasks, get that
1-1
F(soly)
II
8) = R(xly) II 1=0
Pa Po(5e1524155)
(13)
t=
Then, the probability of reaching the target outcome y is
P(x ylv) Σ P(r T.e==W
(14)
By definition, we have that
P(T = II
Pr (8
(15)
t=
Therefore, we get that Combining Eq. (13) with Eq. (16), and due to the law total probability, we obtain that
R-1
P(x = vly) Σ
II Pr Pr(S
(16)
F(soly) vlv)
Σ R(zly)
II
TWOn=
t=
R(yly)
II
PB (
(17)
T, = 1=0
Σ
II
Pa pp( =
14

---

Published as a conference paper at ICLR 2024
where we note that R(yly) = according the definition of the reward function (which is binary).
With the same analysis for the where the agent fails reach the target outcome y, i.e., x + y and R(aly) = O, we have that
Yx f y, F(soly) P(xly) = 0.
(18)
Combing Eq. (17) with Eq. (18), we have that P(x yly) i.e. the outcome-conditioned forward policy (r(s'18, (8,8) can successfully reach any target outcome y.
B.2 ANALYSIS OF THE CONVERSION POLICY
We now elaborate more details about the effect of Eq. (6) in the text based on (Bengio et al., 2023).
When the outcome-conditione GFlowNet (OC-GFN) is trained to completion, the following flow consistency constraint in the edge level is satisfied for intermediate states.
F(sly) Pr ( y) Pisll))PP(ss)
(19)
We define the state flow function as F (5) = E,, (())) (s)b),
and the backward policy as
Ps(()s') Σy r(y) (s' y/) (s(s)s',))
(20)
Ey F(y')
while the forward policy is defined in Eq. i.e,
(d = En r(y) (r E, (())((s)))))
(21)
Then, we have
(s(s) 8) Σ (FF
(22)
and
('(()))(()1))) Σ Er
(23)
y
Combining the above equations, we have that
(1)())))) F(d')
(24)
which corresponds new flow consistency constraint in the edge level. A more detailed proof can be found in (Bengio et al., 2023).
is also worth noting that OC-GFN should be built upon the detailed balance objective (Bengio et al., 2023) diseussed in Section 2 or other variants which learn flows. Instead, the trajectory balance objective (Malkin et al., 2022), does not learn a state flow function necessary for converting the pre-trained OC-GFN model to the new policy n..
Discussion about applicability
We obtain the learning objective for OC-GFN when built upon sub-trajectory balance (SubTB) (Madan et al., 2022) for a sub-trajectory Taj [s,,"-,,3) 353 as in Eq. (25) following Section 1.
j-1 ((sly) II PP v) = F(s,lu) II Pa(
(25)
Further incorporating proposed outcome teleportation technique for OC-GFN (SubTB) results the training objective as Eq. (26), which trained in the log domain:
LogF())))))) log F(s:l))) +
(TC
Σ wiy
log PF(sc+1/su.0
Er Et
(26)
j-1
2
log F(syly)
Σ
log Ps( -log R(xly))
15

---

Published as a conference paper at ICLR 2024
where Waj Logressa A denotes the weight for the sub-trajectory Tegj. and À is hyperparam- eter for controlling the weights. Besides the performance of OC-SubTB studied in the World task in Section 5.1, we also demonstrate its effectiveness in two TF Bind generation tasks by study- ing the success rate of OC-SubTB in the unsupervised pre-training stage, and the number of modes and top-K scores for fine-tuning OC-SubTB. Results are summarized in Figure 13, where ob- serve consistently high success rate (nearly 100% at the end the pre-training stage) and improved performance in the fine-tuning stage.
X--
(a)
(b)
(c)
(d)
(c)
Figure 13: Results of OC-GFN when built upon SubTb in the TF Bind sequence generation task different downstream tasks MAX3_R270C_RI and training OC-SubTB. (b)-(c) The number of modes discovered. (d)-(e) Top- K scores.
(a) Success rate of pre-
B.3 PROOF OF PROPOSITION 4.2
Proposition 4.2 Suppose that V(8, 8' y), Lamortised 8, 0, then the amortized predictor N(s' 8) estimates Ey r() F() ( (.)
Proof. As Laamonized (8, 8 y) 0 for all states 8, next states s', and outcomes y, have that Vs, 1s.s', N(s'ls)Q(uls'. s) (ss
(27)
Therefore, by summing over possible outcomes y on both sides, we obtain that
Vs,8', Σ (() Σ (r
(28)
As Σ, (() 1, we get that
Vs N(s'(s) Σ (F0
(29)
Based on the above analysis, we get that the amortized predictor N(s'(s) estimates the marginal Σ, (s)) (s())s,
C EXPERIMENTAL DETAILS
All baseline methods are implemented based on the open-source implementation, where we follow the default hyperparameters and setup in (Bengio et al., 2021). The code will be released upon publication of the paper.
C.1 EXPERIMENTAL SETUP
GridWorld We employ the same standard reward function for GridWorld from (Bengio et al., 2021) in Eq. (30), where the target reward distribution is shown in Figure 3(a) with 4 modes located near the corners of the maze.
R(x) I (0.25 < |m/H - 0.5l) +
II T (0.3 Jr/H 0.5| <..4) 10
(30)
httpps://gn wttphh......ooooo//eoe
16

---

Published as a conference paper at ICLR 2024
The GFlowNet model is feedforward network consisting of two hidden layers with 256 hidden units per layer using Lea activation. We use a same network structure for the outcome- conditioned GFlowNet model, except that we concatenate the state and outcome as input to the network. For the unconditional GAFlowNet model, we follow (Pan et al., 2023b) and leverage random network distillation as the intrinsic reward mechanism. The coefficient for intrinsic rewards set to be 1.0 (which purely learns from intrinsic motivation without task-specific rewards). We also use same network structure for the amortized predictor network. We train all models with the Adam (Kingma & Ba, 2015) optimizer (learning rate is 0.001) based on samples from parallel of 16 rollouts in the environment.
Bit Sequence Generation We consider generating bit sequences with fixed lengths based on (Malkin et al., 2022). The GFlowNet model is a feedforward network that consists of 2 hid- den layers with 2048 hidden units with ReLU activation. The exploration strategy is e-greedy with e = 0.0005, while we set the sampling temperature to 1.0, and use a reward exponent of 3. The learning rate for training the GFlowNet model is 5 x 10-3 with the Adam optimizer, with a batch size of 16. We use a same network structure for the cutcome-conditionn amortized predictor network. We train all models for 50000 iterations, using parallel of 16 rollouts in the environment.
GFlowNet model and the
TF Bind and RNA Generation For the TFBind-8 generation task, we follow the same setup as in (Jain et al., 2022). The vocabulary consists of 4 nucleobases, and the trajectory length is 8 and 14. The GFlowNet model is a feedforward network that consists of 2 hidden layers with 2048 hidden units and ReLU activation. The exploration strategy is c-greedy with c = 0.001, while the reward exponent is 3. The learning rate for training the GFlowNet model le 4, with a batch size of 32. We train all models for 5000 iterations.
AMP Generation We basically follow the same setup for the antimicrobial peptide generation task as in Malkin et al. (202)); Jain et al. (2022), where we consider generating AMP sequence with length 50. The GFlowNet model is an MLP that consists of 2048 hidden layers with 2 hidden units with ReLU activation. The exploration strategy e-gredy with = 0.01, while the sampling temperature is set to 1, and uses reward exponent of 3. The Ir for training the GFlowNet model is 0.001, and the batch size is 16.
Metrics We use the same evaluation metrics as in the literature in GFlowNets (Bengio et al., 2021; Malkin et al., 2022; Madan al., 2022; Pan et al., 2023b;a). The number of modes is calculated using a sphere-exclusion procedure. Specifically, the generated candidates are sorted by reward, and then scanning down this sorted list, a candidate is added to the list of modes if it is above certain reward threshold and is further away than a distance threshold ó from all other modes. For the normalized number of modes, we normalize the number of modes discovered in each task by min-max normalization, and average this value across different tasks.
C.2 ADDITIONAL ABLATION STUDY OF OC-GFN IN THE UNSUPERVISED PRE-TRAINING STAGE
We now further investigate the impact of the GAFN (Pan et al., 2023b) in the unsupervised pre- training stage as discussed in Section 1. based on the learning objective for OC-GFN presented in Eq. (3). We investigate the influence of GAFN by ablating the model design and replacing it with a random agent, and GFlowNet agent that is trained with task-irrelevant rewards (i.e., R(x)= = 1) in the unsupervised pre-training stage, while the GAFN is learned by pure intrinsic motivation with random network distillation (Burda et al., 2018). We conduct this ablation study based on the GridWorld environment as in Section 5.1. Please note that the experiments are run for about 2500 updates only due to time limitation in the rebuttal period (and therefore the success rate is not around 100% yet). It is also worth noting that the effect of completely removing the GAFN part has been studied in Section 5.1, where OC-GFN w/ OT & CT in Figure 4 in the text examines the significance of the contrastive training technique.
We compare the success rate of each method for reaching given outcomes following Section 5.1. As shown in Figure 14, OC-GFN (GAFN) learns much more efficiently than the other two variants, where the GAFN model is replaced with a GFN model trained with task-irrelevant rewards (OC- GFN (GFN)) or a random agent (OC-GEN (random)), with a more significant margin in larger maps.
We also study the generated outcome distribution by ablating this model design to gain better un- derstanding during the learning process. Figure 15 demonstrates the outcome distribution generated
17

---

Published as a conference paper at ICLR 2024
(UR) OE-09, (54P1)
CC GPN (AATR)
(a) Small.
(b) Medium.
(c)
Large.
Figure 14: Ablation study of OC-GFN in the unsupervised pre-training stage in GridWorld in suc- cess rate.
by each method at the same iteration (with 2500 updates) in GridWorld with a large map. As shown, OC-GFN (GAFN) motivates the agent explore unfamiliar regions, resulting in more diverse out- come distribution for training the OC-GFN and improving its learning efficiency. This observation consistent with Figure 14, demonstrating the importance of the GAFN model.
(a) OC- OC-GFN (Random).
(b) OC-GFN (GFN).
(c) OC-GEN (GAFN).
Figure 15: Outcome distribution for variants of OC-GFN in the unsupervised pre-training stage in GridWorld with large map.
C.3 ADDITIONAL RESULTS A DIRECT PRE-TRAINING SCHEME
One can also pre-train a GelowNet on task A and then fine-tune the model on task B. However, this simple scheme for pre-training GFlowNets does not lead universally applicable strategy for efficient transfer. This is because the GFlowNet learns to sample proportionally from the reward function of task A during the pre-training stage, and thus, if there is not enough shared structure between the rewards for task A and task B (or the goals are unrelated), the GFlowNet model pre- trained on task A may not be beneficial for the fine-tuning stage task B, and could potentially even have a negative impact.
DEDN 29EEIA
CEOS
(a)
(b)
(c)
(d)
Figure 16: Comparison results of GFN (from pre-trained) in TF Bind generation tasks. (a)-(b) Mode discovery and top- scores SIX6 T165A RI. (c)-(d) Normalized mode discovery and normalized top-K K scores averaged over 10 downstream tasks.
We now compare our proposed OC-GFN with fine-tuning this schema (which areitrained on PAX3 R270C_R2) on 10 TF Bind generation tasks (as described in Section 5.3). Figures 16(a)- (b) demonstrate the results in terms of mode discovery and top-K scores on one downstream task SIX66TI65A_RR for demonstration, while Figures 16 summarize the normalized mode discovery and top-K performance averaged over each downstream task. As shown, this GFN (from pre- trained) schema does not provide universal effectiveness (which can lead to lower efficiency in mode discovery and top-K performance), as is sensitive the reward structures of the downstream
18

---

Published as a conference paper at ICLR 2024
tasks. It has also been highlighted in previous works (Bengio al., 2021) that GFlowNets prone to overfitting on certain tasks, which could further result worse performance compared training GelowNet from scratch on task B. On the contrary, we cast the problem of unsupervised pre- training of GelowNet with self-suuervised learning, which learns reach any oulcome for learning functional understanding of the task. Therefore, it provides a universally effective strategy to different downstream tasks, and agnostic to the reward structures.
C.4 ADDITIONAL RESULTS OF COMPARISON WITH RL METHODS
We include an additional comparison of OC-GFN with other two strong RL methods including PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) with automatic tuning of the entropy coefficient. in the same two downstream tasks of TF Bind generation in Figure 10 in Section 5.3 (PAX3 -AXX3_270C__RI and (XXB7_M190L.RI) Figure 17 demonstrates the comparison results in terms of the number of discovered modes ((a)-(b)) during learning and the top- scores ((c)-(d)). As shown, the reward- maraaddaaimizztioon nature of RL approaches limits the resulting solution diver- sity (Bengio et al., 2021; 2023), which underperforms GFN (from scratch) in the number of discov- ered modes and top- scores, and OC-GEN consistently outperforms baselines in the mode-seeking ability.
ecars
6oong
6O0TH
6G0TH
MON:
Mone
Monc
(a)
(b)
(c)
(d)
Figure 17: Additional comparison results with other RL methods (PPO and SAC) in different down- stream tasks in TF Bind generation. (a)-(b): Number modes. (c)-(d): Top- K scores.
Computation cost We now compare the computation cost of each baseline method in Figure 17 terms of memory and time (for collecting experiences and for training the model), As shown in Figure 18, training OC-GFN only leads to marginal computation overhead.
(a)
Figure 18: Comparison of computation costs. (a) GPU Memory. (b) Collect experiences. (c) Train models.
C.5 ADDITIONAL RESULTS ON THE BIT SEQUENCE GENERATION TASK
shown in Figures 19(a)(c)) OC-GFN more efficient discovering modes and discovers more modes, compared with baselines in different scales of the task, from small to large. DQN gets stuck and does not discover diverse modes due to the reward maximizing nature of regular reinforcement learning algorithms, while MCMC fails to perform well in problems with larger state spaces. Besides different scales of the problem, we also consider different downstream tasks as in Figures 19(d)-(e), which further demonstrate the effectiveness of OC-GFN in the supervised fine-tuning stage.
C.6 ADDITIONAL RESULTS ON TF BIND GENERATION TASK
The full results in the TF Bind generation task are demonstrated in Figure 21 for the 30 downstream from (Lorenz et al., 201 1). The mean rank averaged all tasks summarized in Table where OC-GFN significantly outperforms baselines and able to discover more modes in a more efficient way.
19

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
(c)
Figure 19: Results in the bit sequence generation task in different with different scales of the task.
Table I: Mean rank of each baseline in all downstream tasks (the lower the better).
DQN
MCMC
GFN (from scratch) OC-GFN
Number modes
4.00
2.77
1.97
1.27
Top-K reward
4.00
2.80
2.00
1.20
C.7 ADDITIONAL RESULTS ON THE RNA GENERATION TASK
The full results in the RNA sequence generation task are demonstrated in Figure 20 for the four downstream from (Lorenz et al., 1), where OC-GFN significantly outperforms baseline methods achieving better diversity in more efficient manner.
22344
O028
(a)
(b)
(c)
(d)
Figure 20: Full Results in the RNA generation task.
D LIMITATIONS AND FUTURE DIRECTIONS
We address the potential challenge of sparse rewards in training outcome-condiiioned GFlowNet in the unsupervised pre-training stage with contrastive learning procedure based on the concept of goal relabeling. This work lays the foundation for further exploration of pre-training and fine- tuning strategies in GFlowNets, enabling the transfer of pre-trained models to downstream tasks with different reward functions. However, it is also worth noting that Eq. (6) assumes that the state and action spaces remain consistent across many relevant problems. Extending our results to problems with different state and action spaces would be exciting direction for future research. It is an interesting future direction for considering smoother rewards or other techniques for addressing this problem. In addition, it is also promising consider continuous outcomes in future works (which has the potential to tackle even larger-scale problems), while our work mainly focuses on discrete outcomes.
20

---

Published as a conference paper at ICLR 2024
Figure 21: Full Results in the TF Bind generation task.
21


Reviews:
[{"review_id": "PPvxDenmKA", "reviewer": "Reviewer_5Pd3", "summary": "The paper  proposes the outcome-conditioned GFlowNet (OC-GFN) for reward-free pre-training and fine-tuning of GFlowNets in order for efficient adaptation to downstream tasks. OC-GFN is learnt to reach any specified outcome, and an amortized predictor is learnt to approximate an intractable marginal required for fine-tuning. The paper provides extensive experimental results to validate the effectiveness of their proposed approach.", "questions": "1. How does the proposed approach perform compared to existing methods for pre-trained models or RL methods besides DQN? and what about the computation cost of these methods?\n2. Is the trained GAFlowNet necessary? What about its performance and how does it influence the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach for reward-free pre-training and fine-tuning of GFlowNets, which can serve as a foundation for further research of GFlowNet pretraining.\n2. The paper provides a thorough description of the proposed approach, including the formulation of the problem, the training procedures, and the evaluation metrics. The experiments are well-designed and conducted, and the results are presented clearly.", "weaknesses": "1. The paper lacks a comparison with existing approaches for pre-trained models or goal-conditioned RL methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5mc95q5Npq", "reviewer": "Reviewer_Deqj", "summary": "The paper introduces a novel approach to pretrain generative flow networks (GFlowNet) in a self-supervised manner, focusing on aligning input with target outcomes. When adapting to a downstream task, there's no need to re-train the GFlowNet; instead, outcomes are integrated using Monte Carlo sampling. The authors cleverly introduce an amortized predictor to overcome sampling challenges.", "questions": "In the 'Discussion about applicability' section, the trajectory balance's inability to learn the stateflow function, and its subsequent inapplicability for converting a pre-trained GFlowNet on a new reward, is mentioned. Have the authors evaluated the sub-trajectory balance objective (as per Pan et al.) which does incorporate the state-flow function?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The concept presented in this paper is both simple and elegant. The unsupervised fine-tuning approach offers a significant contribution, adeptly addressing the training challenges associated with GFlowNet. Overall, the paper is well-structured and easy to follow, making it a valuable addition to the literature.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBMfECyix3", "reviewer": "Reviewer_ja74", "summary": "- This paper tackles the problem of pretraining Generative Flow Networks (GFNs) and fine-tuning them to quickly approximate new sampling distributions.\n- The authors take a reinforcement learning (RL) perspective, and observe that for GFNs the state-space, action-space, and transition probabilities remain unchanged for many problems of interests.\n- This lets them apply goal-conditioned RL methods as a generic strategy to pretrain GFNs, which they dub outcome-conditioned GFNs (OC-GFNs).\n- For fine-tuning, they show how to immediately adapt OC-GFNs when given the reward function of a downstream task (see Eq. 6). Since this involves the computation of an intractable sum, they amortize it with a learned predictor.\n- The authors demonstrate the efficacy of their fine-tuned OC-GFNs on toy (GridWorld & BitSequence) and real-world biology problems (DNA binding, RNA generation, AMP generation), with some ablations on the toy problems.", "questions": "- Please comment on the weaknesses outlined above.\n- Figures 10 and 11, right: Why is adaptation slower for OC-GFN than GFN in the first few thousand iterations? This is surprising since one would hope pretraining helps bootstrap downstream performance as in vision / language / RL. If it’s an exploration phase, did you validate it and is there a way to side-step it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The exposition is generally clear, and I enjoyed reading the paper. The authors first present the goal-conditioning idea and how it applies to GFNs, then walk the reader through their derivation and assumptions for amortized adaptation. I especially appreciated Section 2 which gave a clear and concise background.\n- The paper tackles an impactful problem for GFNs. While the pretraining solution is not particularly novel, it’s a neat application of goal-condition RL to an amortized sampling problem. The authors also figured out how to make it work on a wide range of problems, and provide several ablations in the main text and the appendix.\n- The insight that a new sampling policy can be readily obtained from an outcome-conditioned flow is neat and, as far as I can tell, novel. This could spawn interest in outcome-conditioned flows and different ways to amortize Eq. 6.", "weaknesses": "- There should be a discussions of assumptions behind the OC-GFNs pretraining. Namely, that transfer is only possible when the reward function changes but not if the action-space or the state-space change. Moreover, the goal-conditioning requires a well specified set of outcomes Y — presumably not all states s are terminal states — which makes the proposed method not truly unsupervised. These limitations (together with the applicability mentioned at the end of A.2) could be stated explicitly in the main text, and left to future work.\n- While there are enough benchmarks, I believe none include continuous action/state spaces. Moreover, the experiments only one GFN variant — the detailed-balance one, which is also used for OC-GFN. It would help validate the generality of OC if we had experiments showing it worked on these different settings. Moreover, I’d be curious to know how other pretrained amortized sampling baselines (eg, VAEs, normalizing flows) fare against OC-GFN — and what about pretraining a GFN on task A (without OC) and fine-tuning it on task B?\n- (minor) The second and fourth paragraphs of Section 4.2 mention the “reasoning potential” of GFNs, and that intractable marginalization leads to “slow thinking”. Are these anthropomorphisms really needed for this paper?\n- (minor) I wished the preliminaries (Section 2) included a training objective like Eq. 5 & 9, and that these more clearly specified which are the optimization variables.\n- Some typos, there maybe more:\n    - p. 3: multi-objective what?\n    - p. 4: “given a reward R a posterior as a function”\n    - p. 4: autotelicly → autotelically?\n    - p. 5: “in log-scale obtained from Eq. (5)” should be Eq. 4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0yOFjJK4U", "reviewer": "Reviewer_gG5W", "summary": "The paper describes a methodology for self-supervised reward-free pretraining of generative flow networks (GFlowNets). The authors propose a novel scheme for pretraining of GFlowNets and show its efficiency on a number of tasks. They also compare it with the baselines on a number of datasets. The authors include a number of improvements over the standard GFlowNets: \n\n- Self-supervised pretraining\n\n- Target task finetuning (or should it be called, e.g., task transfer as it does not finetune the coefficients of the original model to the new task?)\n\n- Amortisation procedure for the target task finetuning to alleviate the problem of high cost of estimation of the task-specific reward function (Eq 6)", "questions": "Q1: \nWhile the experimental results show clear improvement when using the outcome teleportation module, the theoretical  motivation of Eq 4 still remains not entirely clear. The  original detailed balance equation represents the reversibility of the Markov chain; the proposed method, in contrast, does not satisfy such condition as it is seemingly assymetrical with backwards flow. One possible way would be to consider the right hand side a factorisation of the transition function $\\tilde{P}_B(s | s’, y) = P_B (s | s’, y) R(x|y)$ but that won’t give $\\int \\tilde{P}_B(s | s’, y) ds = 1$. One can interpret it that it works as a regularisation of the loss function in Eq (5). Another related question relates to the transition between Eq 14 and Eq 15 in the Appendix related to the proof of this statement. It is not clear where did $R(x|y)$ disappear in between  Eq. (14) and (15). In the standard case of the detailed balance equation, that would have been a valid transition, but why is it valid for the non-1 R(x|y)?\n\nQ2: Not sure I completely understand how the number of discovered modes (normalised and unnormalised) is calculated  (see Figures 10 and 11)?\n\nQ3: “A remarkable aspect of GFlowNets is the ability to demonstrate the reasoning potential in generating a task-specific policy.“ Not sure the word reasoning would be the right way to describe it as I am not sure it is objective; despite not necessarily agreeing with the wording, I do think the meaning behind it, i.e. transferring to the task without re-training (Section 4.2), is a valuable aspect.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (Originality, Significance) A multifold methodological contribution (see above), which helps define new ways to train and use the GFlowNet models, most importantly, including the insight about transferring the model to the downstream task without re-training (Section 4.2)\n\n- (Quality) Well-written paper\n\n- (Quality) Thorough analysis of the method on a number of tasks\n\n- (Reproducibility/Clarity) The paper appears to provide good explanation of the experimental conditions and therefore, addresses reproducibility well (apart from Q2)", "weaknesses": "- (Soundness aspects) There are some questions to resolve about the motivation of the outcome teleportation module (see Q1), hence the current score. \n\n- Clarifications on the experimental setting (see Q2)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)