COFlowNet: Conservative Constraints on Flows Enable High-Quality Candidate Generation
Authors: Yudong Zhang
Xuan Yu
Xu Wang
Zhaoyang Sun
Chen Zhang
Pengkun Wang
Yang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: generative flow network
offline RL
molecule design
Primary area: reinforcement learning
TL;DR: We propose an offline version of generative flow network.

Abstract: Generative flow networks (GFlowNets) have been considered as powerful tools for generating candidates with desired properties. Given that evaluating the property of candidates can be complex and time-consuming, existing GFlowNets train proxy models for efficient online evaluation. However, the performance of proxy models is heavily dependent on the amount of data and is of considerable uncertainty. Therefore, it is of great interest that how to develop an offline GFlowNet that does not rely on online evaluation. Under the offline setting, the limited data results in an insufficient exploration of state space. The insufficient exploration means that offline GFlowNets can hardly generate satisfying candidates out of the distribution of training data. Therefore, it is critical to restrict the offline model to act in the distribution of training data. The distinctive training goal of GFlownets poses a unique challenge for making such restrictions. Tackling the challenge, we propose Conservative Offline GFlowNet (COFlowNet) in this paper. We define unsupported flow, edges containing unseen states in training data. Models can learn extremely little knowledge about unsupported flow from training data. By constraining the model from exploring unsupported flows, we restrict COFlowNet to explore as optimal trajectories on the training set as possible, thus generating better candidates. In order to improve the diversity of candidates, we further introduce a quantile version of unsupported flow restriction. Experimental results on several widely-used datasets validate the effectiveness of COFlowNet in generating high-scored and diverse candidates. All implementations are available at https://github.com/yuxuan9982/COflownet.

Published as a conference paper at ICLR 2025
COFLOWNET: CONSERVATIVE CONSTRAINTS ON FLOWS ENABLE HIGH-QUALITY CANDIDATE GENERATION
Yudong Zhang', Xuan Yu', Xu Wang
Zhaoyang Sun Chen Zhang', Pengkun Wang Yang Wang1.2.3
1. University of Science and Technology of China (USTC), Hefei, China
Suzhou Institute for Advanced Research, USTC, Suzhou, China
3. State Key Laboratory of Precision and Intelligent Chemistry, USTC, Hefei, China
mzyd2020email... yx202 4 @0mall., wx3090 ustc.edu..cm cn lsunzhacyangemail. zhangchenze@mall. pengkun@, angyang jstc. edu. cn
ABSTRACT
Generative flow networks (GFlowNet) have been considered powerful tools for generating candidates with desired properties. Given that evaluating the property of candidates be complex and time-consuming, existing GFlowNets train proxy models for efficient online evaluation. However, the performance of proxy models is heavily dependent on the amount of data and is of considerable uncertainty. Therefore, it is of great interest that how to develop an offline GFlowNet that does not rely on online evaluation. Under the offline setting, the limited data results in an insufficient exploration of state space. The insufficient exploration means that offline GFlowNets can hardly generate satisfying candidates out of the distribution of training data. Therefore, it is critical to restrict the offline model to act in the distribution of training data. The distinctive training goal of GFlownets poses unique challenge for making such restrictions. Tackling the challenge, we propose Conservative Offline GFlowNet (COFlowNet) in this paper. We define unsupported flow, edges containing unseen states in training data. Models can learn extremely little knowledge about unsupported flow from training data. By constraining the model from exploring unsupported flows, we restrict COFlowNet to explore as optimal trajectories on the training set as possible, thus generating better candidates. In order improve the diversity of candidates, we further introduce a quantile version of unsupported flow restriction. Experimental results on several widely-used datasets validate the effectiveness of COFlowNet generating high-scored and diverse candidates. All implementations are available at hhttps://githul com/ wtt.m//ywwxun999//0f0llooo
INTRODUCTION
Reinforcement learning (RL) is typically about finding an optimal solution to given target (Mnih et al. 2015; Sutton, 2018). RL models are required to generate the single highest-reward sequence of actions. However, has become increasingly apparent that the ability to produce a variety of candidate solutions, not just the optimal one, is highly valuable for numerous real-world applications, including molecule design (Huang et al. [016) et al. 202 Bengio et al. (022]) and exploration in RL (Hazan et al. 2019). For example, in the scenario of molecule design, rather than generating high-scoring molecule that cannot be synthesized, the model should generate a series of molecules with suboptimal scores, so that chemists can pick molecules that are easier to synthesize.
Generative Flow Networks (GFlowNets) (Bengio et al. 202T Jain et al. 2022, Bengio et al. 2023
Gao et al. 2022 have emerged potent tool for generating diverse candidates. The key insight of GFlow is to ensure that the probability of generating a candidate proportional to the positive reward associated with that candidate. Therefore, GFlowNets is able sample series of possible
DD.. Xu Wang and Prof. Yang Wang the corresponding authors. Email: wx.309@uste.edie

---

Published as a conference paper at ICLR 2025
candidates with the reward distribution. Taking advantage of such ability, GFlowNets have expressed promising potential in many object generation application areas. Jain et al. (2022) embeds GFlowNet into active learning framework for biological sequence design, which iteratively generates diverse candidates and screens the candidates to enhance the training of GelowNet. [eleuu et al. (022); Sahhikawa-Tooomee et al. (2022) leverage GFlowNetts general framework for generative modeling of discrete and composite objects, which approximates the posterior distribution over the structure of Bayesian networks. Liu al. (2023a) leverages GFlowNets for sampling structured sub-network modules, thereby enhancing predictive robustness. Zhang et al. (2023a c) apply GFlowNets to address combinatorial optimization challenges. Additionally, Zhou et al. (2023) explores the use of GelowNets for phylogenetic inference, demonstrating the model's versatility in diverse applications.
In many scenarios, evaluating generated candidates could be expensive and time-consuming, making it impractical to calculate the accurate score of candidates. For example, in the realm of drug design, evaluating a potential molecule often requires conducting biological experiments or performing complex chemical calculations. Given that these assessment procedures can span from several minutes to multiple days, they are impractical to integrate directly into the training phase of an RL model. To this point, existing GFlowNets follow the method proposed in |Angermueller et al. (2019), which suggests training a proxy model based on evaluated candidates to approximate the accurate scores (rewards) of candidates. Specifically, we have a set of candidates X {2rx) and set of corresponding scores Y = {yx y = oracal (x4)). where oracal denotes the expensive but accurate evaluation of candidates. Based on the dataset (X, Y), proxy model f x + y can be trained to approximate oracal. The proxy model is then employed to calculate the rewards of online sampled candidates, with which the GelowNets can be trained.
While GetswNets have achieved notable success across various domains of object generation, their effectiveness is significantly contingent upon the quality of the proxy models they rely on. Typically, GFlowNets are trained to align with the candidate-reward However, the scarcity of data can introduce considerable variability in the proxy model's 's accuracy. the proxy model fails to accurately mirror the true quality of the candidates, the resulting performance of the GFlowNets will be suboptimal. Given that proxy model training requires a comprehensive dataset of candidates along with their actual scores, alternative approach involves the development of offline reinforcement learning methods. These methods could potentially sidestep the pitfalls associated with proxy model dependency, offering more reliable framework for GFlowNets training.
distribution as estimated by proxy model.
Unlike conventional RL models, GFlowNets are trained with the specific objective of generating candidates with probabilities that are directly proportional the positive rewards linked to those candidates. This distinctive training goal presents unique challenges for the development of offline GFlowNets, making existing offline RL techniques policy in GFlowNet frameworks is determined by inflows and outflows of states, which complicates the application of actor-critic methods (Nair et al., 2020 Tarasov et al. 2024). In Q-learning frameworks (Kostrikov et al. 202I). Q-values are maximized iteratively by the Bellman operation, which actually against the flow match constraint in GFlowNet. Policy constraint or matching methods (Wang et al., (023) may assimilate less desirable state space, such as some low-reward areas, making them unsuitable for offline GFlowNets. In essence, the distinctive training goal of GFlowNets calls for the development of new offline techniques not found in traditional RI. approaches.
not be directly applied. Specifically, the
In this paper, we propose a novel offline training strategy for GFlowNets to make full utilization of collected data, called Conservative Offline GFlowNet (COFlowNet), To avoid the generation of highly uncertain candidates, we define unsupported flows and propose to regularize the unsupported flows, so that the model can learn informative knowledge from training data. To enhance the diversity of generated candidates, we introduce a quantile matching algorithm, and modify the regularization of unsupported flows into quantile style. We evaluate the proposed offline training strategy following the experimental setting of [Begiio et al. (202T). By applying the proposed training objective, the offline version of GFlowNet shows great potential for generating diverse and high-score candidates.
The main contributions of this paper are as follows:
Our research endeavors to adapt GFlowNets for offline scenarios. A pioneering offline training strategy named Conservative Offline GFlowNet (COFlowNet) is introduced. Central to our approach is the concept of unsupported flows. By regularizing unsupported flows, the model learns informative knowledge from training data and generates high-score candidates.
2

---

Published as a conference paper at ICLR 2025
To enhance the diversity of generated candidates, we introduce quantile matching algorithm. By modifying the regularization of unsupported flows into quantile style, we achieve the final training objective of COFlowNet, termed conservative quantile matching (CQM).
We evaluate the proposed offline training strategy in alignment with the experimental setting in Bengio et al. (20TT) The proposed COFlowNet, equipped with the novel offline training objective, exhibits significant promise in producing a spectrum of diverse and high-performing candidates.
RELATED WORK
GFlowNet. Since the introduction of GFlowNets by [Bengio et al. (2021). there has been surge of research in this domain, covering various aspects of the technology. Malkin et al. (022));
Zimmermann et al. (2022) have explored the relationship with variational methods, demonstrating that GFlowNets surpass variational inference when utilizing off- off-polliy training data. Pan et al.
(202 2023b) has established frameworks to enhance credit assignment efficiency by incorporating intermediate signals within GFlowNets. mJain et al. (0222) has delved into multi-objective generation capabilities, while Pan et al. (2023c) integrated world modeling. An unsupervised learning approach for GFlowNets was suggested by Pan et al. (2023a). and Ma et al. (2024) examined the of isomorphism tests to mitigate flow bias in training. From a probabilistic modeling perspective, Zhang et al. (202bb has concurrently trained an energy-based model alongside a GFlowNet, validating its effectiveness on discrete data modeling tasks and proposing a bidirectional proposal mechanism later adopted by Kim et al. (202)) for local search algorithms. [Zhang al. (2022a); Lahlou al. (2023);
Zhang et al. (2023b) provided a theoretical analysis and bridged the gap between diffusion modeling and GelowNets. GelowNets have also shown promise in numerous object generation applications, including biological sequence design by Jain et al. (2022) and causal structure learning by Deleu al.) (2022). Considering the complexity and time intensity associated with candidate property evaluation, current GFlowNets utilize proxy models to facilitate efficient online assessments. Yet, these proxy models' effectiveness is highly contingent upon data volume, introducing significant margin of uncertainty. Hence, there is considerable interest in developing an offline GFlowNet that is independent of real-time evaluation mechanisms. However, GFlowNets train to generate candidates with probabilities proportional to their rewards, presenting challenges for offline adaptation that don't 't align with standard offline RL techniques. In the following, we analyze why existing offline RL methods are not applicable.
Offline RL. Most existing offline RL methods are based on an actor-critic framework or Q-learning framework. In flow-based framework, policy is directly given by (als) Fss, a)/ (s). where & denotes a state with actions a E (s) and ((s,, a) denotes the flow of state s taking action a. The policy is too fixed, making it difficult to apply techniques of actors from actor-critic-based methods of offline reinforcement learning, such as AWAC (Nair et al. 2020) and ReBRAC (Tarasoy al. 2024). However, Q-learning-baan based frameworks are also not applicable. The concept of flows is analogous to Q-values in Q-learning and the critic in the actor-critic method. However, there are still some sigmificant differences between them. Q-values are iterated by the Bellman operation. estimating how good an action is. But they focus on the maximum value, while the values of flows denote the total sum of rewards passing the state, making it impractical to apply methods such as Kostrikov et al. 2)22I) to flow matching objective. For the flow matching objective to learn the behavior policy of offline datasets better, an intuitive way is to use policy constraint methods (Wang et al. 2023). Policy constraint methods will force the behavior strategy to learn the bad parts of the dataset, such as some low-reward areas, thus is not ideal for offline GFlowNets. Our proposed method avoids this problem by not directly forcing trained policy stay close to behavior policy.
3 METHOD
3.1 PROBLEM FORMULATION
We describe the problem of interest here. We aim at training a policy to generate candidate objects x e
X' with probability proportional to a reward function R(x) + R We generate candidate object
x from an initial state sg, and make a series of actions to transfer the state finally into x. The procedure
can be described by trajectory of state transformation, denoted 7 (80; 81, 82, ... Sn, x)).
3

---

Published as a conference paper at ICLR 2025
We denote the set of states S, and the set of actions A {(s s')sss E S}. Note that here assume the relationship between action and future state a one-to-one correspondence, i.e., there is only one action (s - s') that transfers to s'. We say is parent of s' and s' is child of when we have (s + s') E A. Specially, we have A(s) denoting the set of actions between & and all its children, and thus have A(x) 2 for any terminal states
Generative Flow Networks (Bengio et al. 2021) (GFlowNets) are developed for the target that generates candidate objects æ with probabil- ity proportional R(x). Such an objective is achieved in GFlowNets by casting the set of ac- tion trajectories as flow and converting the flow consistency equations into a learning objective.
$14
11
$12
As in Fig. [] the state transformation can be illustrated as a directed acyclic graph (DAG), In the flow network, each node represents a state, and each edge represents flow (action and its corresponding probability). The source (or root) node only generates outflows. Interior nodes have both inflows and outflows, with inflows equal to outflows. Leaf nodes (or terminal states) only receive inflows and store them as sinks. At specific state s,, we have several input flows from its prior states and output flows to its successive states. The flow consistency equations require the inflow and outflow of an interior state (node) to be matched, and the inflow of a terminal state is the reward of the state. We define F(s. s') (s, s') +
Figure I: Illustration of the flow network. The triangle means initial state, the circles denote in- terior states, and the squares denote the terminal states. At each interior state, have its inflow and outflow matched. For a terminal state, the inflow of it equal the reward at it.
as flow between state s and s'. By setting R(s) = O for interior states and A(s) = Ø for terminal states, the flow match constraint of state is given as,
(FM(s)=
Σ F(s,8') F(s,s')-R(() R(8') s:(s->))A
Σ
F(s',8")
s"s(s'=>s)EA
The above equation constrains that the inflows of a state are equal to the outflows of the state plus the reward of the state. Considering whole trajectory T, we have,
(FM(T)= = Σ (s)
SET
Note that square (s) to ensure the value is positive. The training objective of GFlowNets to learn F(s,, s') to minimize (TM( Bengio et al. (2021) proves that a global optimum of the expected loss provides the correct flows. And the training objective can be achieved by setting the probability of action (s + 8') as,
F(s,8')
P((s + +8))))) = - (=(((=)))))((s))) F(s,
This equation suggests that the probability of taking action (s 5') is the ratio of the flow of this action to the outflow of s.
3.2 CONSERVATIVE GELOWNET FOR OFFLINE RL
Due to the innovative design of GFlowNets, i.e., the flow matching, existing offline RL frameworks are not applicable. To this end, this paper proposes a conservative offline GFlowNet (COFlowNet), which learns informative knowledge from training data and shows the ability to generate diverse candidates.
In the offline setting, the model is constrained from extensively exploring the state space and must rely solely on the provided training data. This limitation can lead to inadequate coverage of the state space within the data. Consequently, the performance of candidates whose trajectories include many states not present in the training data may be highly uncertain, as limited clues of their performance can be learned from training data.
4

---

Published as a conference paper at ICLR 2025
To avoid the generation of highly uncertain candi- dates, COFlowNet makes constraints on the flows. The offline dataset D used is composed by a list of transitions (St. at, 8t+1, rt, de). ) where indexes transition sampled from trajectory т. Specifically, we call flow (s + 8') supported if there exists a trajectory (5g + 91 + $2 + + S,)) in the dataset D such that = and St+1 = s'. Otherwise, it is unsupported. These supported flows compose our action set Ap of training data and serve as the basis for imposing node-specific constraints within the proposed COFlowNet framework.
States and Actions present in dataset
States Actions not present in dataset
Sg
Figure 2: Illustration of supported and un- supported flows. Edges not present in the dataset are defined unsupported flows. For instance, (82 + 83) is unsupported.
For inflows in state 8, we constrain the unsupported inflows in the dataset by adding a regularization term to constrain them into small values. The unseen actions are thus constrained to help COFlowNet better learn the behavior policy of training data. The regularization term of unsupported inflows of state s can be formulated as,
(s)
Σ
F(s'.s)-
F(s...) l AD
Σ
For outflows node E S, our strategy similar, and the regularization term outflows is defined as,
Rome (s) =
Σ s"=(s=>s")EA
F(s,s")
Σ
F(s,
(5)
Let define two trade-off factors an and 02 2 0 for (s) and Rou We turn Equation into our constrained flow matching (CFM) objective for interior and terminal states to optimize the parameter 0.
CCFM(s) (s) + a, R2, + RzRo
(6)
We here square Rin and Rout for the same reason as Eq. 2 We next prove that applying the regularization will exactly decrease the unsupported flow and will not hurt the supported flow.
Theorem 1. Given two flow estimation function, á trained with regularization and F trained without regularization, we have F(s, I F(s,s') obtains for unsupported flow (s, i.e., (s, A and (s, 8') $ Ap.
Proof. With trainable parameters in F denoted as 0, the derivative of constrained flow matching objective to 9 is,
dLFFM(() (s) 00
aF(s'
aF(s,
Σ
Σ
00
08
(ss(
aF(s',s)
aF(sD.
22 Rs)
Σ
(7)
00
(s
0F 88)
RazRou((s)
Σ
Σ
00
00
("((s=)))EA
And we have the derivative flow matching objective without regularization as,
dLrM(s)
OF(s's)
OF(s,8")
= 2LFM
Σ
Σ
(8)
00
00
00
5

---

Published as a conference paper at ICLR 2025
The second and third terms of the derivative in Eq. 7 only minimize flows not appear the offline dataset. We get 5', 5() only when = 0 and 02 = 0.
all children of Sa
Sc
Theorem 2.
Given any state, minimizing its inflow from unsupported states will enlarge the inflow from supported states.
S4
Proof. For any state s, for example, $4 in Figure 3, we have that the total inflows equal the total outflows:
F() + F(ss, + F(53, )- = Sc)
S1
S
S3
where S. denotes all the children of 84 and F SS.) is the sum of outflows from 94 to its children. The total reward (flow) in the training data is determined, as all terminal states are given by the data, and the reward can only be obtained at terminal states. Therefore, the outflow F(s4, Se) is constant and denoted as F.. Furthermore, the inflow from supported states F(11, 84) + F(22, 84) is equal to F. (sa,, s4). Since F, is constant, minimizing the inflow from unsupported states F(s3. $1) will lead to an increase in the inflow from supported states F(51.8.) $1) + F(s2, s1). This reasoning process can be casily extended to general conditions (with more supported and unsupported states).
Figure 3: Illustration of The- orem Due to the balance of inflows and outflows, re- ducing the flow between 83 and 84 will increase the other two inflows.
Directly using Eq. will result in large flow states near Sp. To solve the numerical issue, proposed the log sum exp form of our objective similar to [engio et al. (2021).
(CFM(+)= = (logle +
Σ
exp(F)o (s' (8', [))] logle + R(s)) +
exp (8, 8)))))²
8 EA
+ alogle +
Σ
(xp(rrou (s', logle +
Σ
exp dog 8)]]}
<bi(t'p S))EAD
+ a22 +
Σ
exp(Eilu (8, 8))]] - -logle +
Σ
exp(Pe (5)))))))
3.3 BETTER DIVERSITY WITH QUANTILE MATCHING FLOWS
We here introduce and modify the quantile matching algorithm to consider the uncertainty of reward and thus enhance the diversity of generated candidates of our framework. Quantile matching algorithm is originally used to handle situations where the reward function is stochastic and outperforms deterministic flow matching algorithms even on deterministic datasets (Zhang a 2023d).
Follow the definition in Zhang et al. (2023d), we use Zs s') to represent the quantile flow between s and s'. And the equality of inflows and outflows is expanded to the distribution between two random variables, i.e., the quantile flow,
3 (s) log
Σ -SEEA
2 s)) log
exp (2/0 (8,
(10)
(S-
We thus have the regularization term of unsupported inflows in Eq. Fas,
8 log Σ
exp
Z ( (s',
log (o6 As
Σ
exp Z (sb:
(11)
in
(s' (s'-
Similarly, the regularization term of unsupported outflows of is formed as this, 8 L.
(s) = llog
exp LoR (0, -
logg
Σ
exp ton (s, (s,
(12)
(
We deploy the pinball error (²( = =13 - 1(6 < ()) to both 83 and 50t where (1()) is smooth E loss:
if < otherwise
(6)
(13)
1181 -

---

Published as a conference paper at ICLR 2025
Finally, we propose the conservative quantile matching (CQM) objective for COFlowNet,
N
N
Loqu(s) Pa. (2) +a1
Ps. (8 (s))) + 02
Pg. (6 ma (s))) (14)
EXPERIMENT
In this section, evaluate the proposed COFlowNet During the evaluation, we mainly focus on two research questions: 1) How the performance of candidates generated by COFlowNet? 2) How is the diversity of the generated candidates? To facilitate a more comprehensive evaluation, we select various metrics tailored to different tasks. These metrics are chosen to better evaluate the performance and diversity. Besides the questions, will also investigate the impact of different components of COFlowNet. Additionally, we deploy the proposed COFlowNet additional tasks in B All the experiments are conducted on an NVIDIA Tesla A 100 80GB. The offline dataset is formatted as D = {s. s', .r (ss. s') E AD, R(s') {) When training the model, we sample batched data from and calculate the loss function. We can thus align the samples used for training in our offline model with states visited in online models for fair comparison. detail the experimental settings and results on specific tasks in the following.
tasks, Hypergrid and molecule design.
4.1 HYPERGRID
4.1.1 TASK DEFINITION
We first evaluate the proposed COFlowNet the hypergrid task from Bengio al. (002)) The state space is D-dimensional hypercube grid of size HD, where represents the dimension of the grid. The agent is tasked with formulating long-term plans and learning from sparse reward signals. It begins at the origin of the grid, i.e., at coordinate (0, 0, 0), and must navigate by incrementing one of the coordinates by with each move. Additionally, the agent has the option to execute special termination action from any state. Upon deciding to stop, the agent is awarded reward as specified by the following reward function,
R(x)= Ro + R1 II
Fd H-1
0.5| E (..25.055)) + R2 II T(
H-1 0.5| E (0.3, 0.4)))
Fd
(15)
Where I is the indicator function, which returns 1 when the input condition is true otherwise 0, we set Ru 0. 0.00001, R1 0.5, R2 2, H 8 and D 4. The formula reveals that there are 2" = 16
distinct modes for this task, where a mode is defined as a local region (potentially encompassing one more states) that yields the highest reward value.
4.1.2 OFFLINE DATA CONSTRUCTION
Three strategies are applied to construct the offline dataset for training COFlowNet on Hypergrid:
1) Expert: employ an online GFlowNet generate an offline dataset with 2 10 trajectories. 2) Random: randomly generate an offline dataset with 2 x 10t trajectories. 3) Mixed: take 10 trajectories from Expert and 10 trajectories from Random.
4.1.3 RESULT
We report two metries of COFlowNet, number of modes and & error. Number of modes denotes how many modes the model finds, which reflects the diversity of the model. As mentioned, the goal of GFlowNets is to generate candidates with probabilities proportional to their rewards. In this case, we can enumerate all states and accurately give the ground truth the probability that a candidate is generated p(x)
R(x)) /(x)/ R(x). And can approximate the probability of generating x, denoted n [z), by repeated sampling and counting frequencies for error is defined as E[pp(z) - ग [z)[]. which estimates whether the model generates candidates with probabilities proportional their rewards.
This task is rather simple, we employ vanilla GFlowNet (Bengio et al. 2021 for comparison. Since the optimal diversity of candidates is rather small, we apply CFM loss in Eq. 9 on this task rather

---

Published as a conference paper at ICLR 2025
6.0B-04
16
Eipert
distribuistn ertor
5.0F-04
14
ron tapen wio
4.0B-04
num of morder
12
10
Mixol
3,0F-04
Misal
2.0E:-04
Random
1.0B-04
Random wio
OFlowNa
0.0E+00
150000
300000
450000
600000
10000 20000
30000
40000 50000
states visited
states visited
(a) & error
(h) number of modes
Figure 4: Experimental result on Hypergrid. Expert, Random and Mixed correspond to different
settings of constructing offiine datasets as described in Sec. 4.1.2 w/o means removing the regular- ization term from the training loss.
than CQM loss in Eq. 14 The result on Hypergrid is reported in Fig. 4 where w/o means removing regularization term from the training loss, i.e., using only flow matching loss and setting a; to O.
As reported in Fig. 4b most models find all of the 16 modes within 40000 times state visiting. When trained Mixed data, COFlowNet can generate all the modes with the least visit of states. Applying the proposed regularization method will reduce the number of state visits by 10%. It is worth noting that the model performs badly on the Random dataset matter regularization is applied not. Such a result indicates that the performance of offline models relies on the quality of training data.
Fig. 4a reports the é error of different models. We can find that vanilla GelowNet achieves the goal of generating candidates with probabilities proportional to their rewards with the least times of state visiting. can be found that when trained on Expert and Mixed, COFlowNet can also get close the goal of GFlowNets, but takes more times of state visiting. Actually, due to the limited coverage of state space of training data, COFlowNet can hardly really achieve the goal. is also worth noting that the & error is high when the regularization is removed. The reason can be found in Eq. 7 When the regularization is removed, the model can not find any clue to reduce the unsupported flow, resulting in randomly exploring those flows, which is against the goal. For example, suppose that we have supported flow ($1, 2) and an unsupported flow ($2, 2) flow into the same terminal state æ. Then, the reward of is divided randomly into the two flows (only related to the initialization of F). Without the regularization, the model will never know whether the flow should be divided into (#2, z)). Worse still, the model even needs allocate inflow for the state 82 balance the inflow and outflow of 82-
Overall, through the result of this simple task, we find the potential of COFlowNet and the effective- ness of the proposed conservative regularization strategy. Next, we introduce complex and realistic task: molecule design.
4.2
MOLECULE DESIGN
4.2.1
TASK DEFINITION
In recent years, artificial intelligence methods have flourished in the field of chemistry, especially for molecular-related tasks (Du et al.) 2024, Li et al. 2024, Zhang et al., 2023e). Among those tasks, molecule design is typical application scenario for Getow where both the diversity and quality of candidates are required. In this task, the objective is to design a variety of molecules that exhibit specific chemical properties. The emphasis on diversity in this task is crucial, as it allows chemists to discover molecules that are not only characterized by their desired chemical properties but are also easy to synthesize. In this section, we interested in designing molecules with large binding energy to particular protein target.
In this task, we have states molecule graphs or SMILESS and actions adding components to a molecule. Therefore, the molecule design task becomes a decision process. We have a vocabulary
wi http
fied Maedddddd____eeeeee leoleclll Input_ Line Kntry_
System
8

---

Published as a conference paper at ICLR 2025
Data scarcity
Fully trained
Models
avg top 10
avg top 100
avg top 1000
avg top 10
avg top 100
avg top 1000
MARS
8.0778
7.833
7.5992
PPO
8.4249
8.3387
8,2555
Get
8.4381
8.2909
8.0580
8,5283
8.3539
8.1440
QM-GFlowNet
8.4979
8,3272
8.1014
8,5552
8.4019
8.1886
COFlowNet w/o
8.3400
8,2046
8.0996
8,4859
8.3278
8.1083
COFlowNet
8.4638
8.3034
8.1088
8.5029
8.3730
8.1693
Table 1:
The average reward of the top candidates. Darker blue denotes the best result and lighter blue denotes the second best.
of building blocks specified by junction tree modeling (Jin et al. 2018), which inherit from vanilla GFlowNet (Bengio et al. (021). At each step, the action space is determined by two factors: selecting an which building block will be attached, and deciding which block attach. In case, the size of the vocabulary of building blocks is 105. Given a molecule, a building block can be added the molecule at different positions. And the number of applicable actions of a state is greater than 105, leading to a larger state space.
4.2.2 DATASET
As computing binding energy is computationally expensive, existing online RL models train proxy models approximate it. Interestingly, the proxy models are also employed evaluate their molecule models for computational convenience. If the target of an RL model is to fit the distribution of reward function, there is nothing wrong with evaluating the model with proxy, in which case the model is to fit the proxy. But for molecule design, we should not ignore the gap between the proxy model and oracle, ie., the expensive computation. To this end, we propose two kinds of evaluation. Specifically, we first construct dataset D with 300k molecules as [engio al. (2021). proxy model PD is trained on D, to serve as oracle, since we are unable to access the real oracle due to the expensive computation. We design two settings for evaluation: Fully trained: In this case, we have D as an offline dataset and train our model on it. Online models are trained with Pp. Pp is also employed evaluate all molecule design models. Data scarcity: In this case, small dataset D:s containing about 14A: molecules is generated by well-trained generative model. COFlow Net is trained on the small dataset Ds. Also, we have weak proxy model Pw trained on Ds, and online models are trained with Pw. Similarly, PD is employed to evaluate all molecule design models.
1800
number of mmodes (R-7.5)
GFlowNet
FM-COFlowNer
number of modes (R~8)
1600
GFlowNet
FM-COFlowNet
15000
NM
1400
QM-GFlewNa
COFlowmet
1200
COFlawnet
PPO
1000
PPO
10000
MARS
800
MARS
600
200
0
0
5
3
5.
state visited (IOF)
state visited (10*)
(a) number of modes with reward >
7.5
(b) number of modes with reward > 8
Figure 5; Experimental result on Molecule Design with large dataset.
4.2.3 RESULT
We compare COFlowNet with two popular flow-based baselines, vanilla GFlowNet (Bengio et al. 2021) and QM-GFlowNet (Zhang et al. 2023d). Two more baselines, MARS (Xie et al. and PPO (Schulman al. 2017), involved here compare Cet with now-llow methods.

---

Published as a conference paper at ICLR 2025
Noting that MARS and PPO require fully-trained proxy model to provide accurate reward, only report their performance under the setting of Fully trained. Additionally, we introduce COFlowNer w/o offline bascline, which removes the proposed conservative regularization term and is employed to substantiate our assertion that the regularization term enhances our model's performance. For the effectiveness comparison, consider the average reward of the top k {10, 100, 1000) candidates. Such a metric indicates the ability of models to generate high-score candidates.
As reported in Table. [ MARS performs significantly worse than flow-based models, including our COflowNet. While PPO generates candidates with scores comparable to those of flow-based models, its performance in terms of diversity is markedly inferior, which will be shown in the following. The performance gap between COFlowNet and COFlowNet w/o validates the effectiveness of the proposed conservative regularization term on unsupported flows. QM-GFlowNet achieves the best performance in generating high-score candidates. Online models, QM- QM-GFlloNeet and GFlowNet, possess better exploration capabilities through interaction with proxy models, while Offline models can solely learn from collected data, where the state space is limited. Consequently, given good proxy model, offline models can hardly outperform online models. The performance gap between GelowNet and COFlowNer w/o also supports the analysis. Surprisingly, COFlowNet can beat vanilla GFlowNet on most the metrics, especially when data is scarce to train a strong proxy model. Considering the above analysis between online and offline models, such results further validate the effectiveness of the proposed method. Meanwhile, as demonstrated in Table. [ model is suboptimal, with limited available data, the performance of QM-GFlowNet deteriorates more rapidly compared to COFlowNet.
when the proxy
number of modes (R>7. 3.9
18000
1600
166000
GelowNet
FM-COFlawNer
mumber of modees (Rt-8)
1400
GFlowNet
PM-COFlowNet
14000
OM-GFlowNet
1200
OM-GFlowNNt
12000
COFlownet
COFlowndt
1000
10000
800
8000
600
6000
4000
400
2000
200
5
3
state visited (10f)
state visited (10*)
(a) number of modes with reward 7.5
(b) number of modes with reward >
Figure 6: Experimental result on Molecule Design with small dataset.
We further compare the diversity of candidates generated by different models. Considering that we introduce quantile matching loss for improving diversity, we here replace COFlowNer w/o with FM-COFlowNer, which utilizes flow matching loss rather than quantile matching loss. As shown in Fig. 5 and Fig. 6 COFlowNet achieves the best diversity. MARS and PPO show poor diversity performance. Notably, COFlowNet generates nearly 20 times many candidate modes MARS, demonstrating its superior ability to explore diverse candidates. The inferior diversity performance of MARS and PPO denotes that they generating similar candidates and overfit the proxy model. The diversity gaps between FM-COFlowNet and COFlowNer, GFlowNet and QM-GFlowNet indicate the effectiveness of quantile matching loss in improving the diversity of candidates. Specifically, applying quantile matching loss leads to 10% to 20% improvement in diversity.
CONCLUSION
This paper extends GFlowNets to offline scenarios, especially for applications where evaluating candidate is quite expensive and historical data has been collected. To take full utilization of the offline data, we define unsupported flows. By regularizing the unsupported flows, we prevent the model from making uncertain exploration state space, thus generating candidates with higher scores. Additionally, to improve the character of GFlowNet in generating diverse candidates, introduce a quantile matching algorithm. By modifying the regularization of unsupported flows into quantile version, we finally propose the conservative offline GFlowNet, called COFlowNet.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This paper is partially supported by the National Natural Science Foundation of China (No. 12227901), the Natural Science Foundation of China Youth Project (No. 62402472), and the Natural Science Foundation of Jiangsu Province of China Youth Project (No.BK20240461, No.BK20240460),
REFERENCES
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations. 2019.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. The Journal of Machine Learning Research, 24(1): 1 1000::00001)0060
2023.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Wenjie Du, Shuai Zhang, Jun Xia Wu, Ziyuan Zhao, Junfeng Fang, and Yang Wang. Mmgnn: A molecular merged graph neural network for explainable solvation free energy prediction. In Proceedings of the Thirty-hird International Joint Conference on Artificial Intelligence, pp. 5808 5816. 2024.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: A benchmark for practical molecular optimization. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Pro- cessing Systems, volume 35, pp. 21342-21357. Curran Associates, Inc., 2022. URL neurips cttessccc//ee cc/paper wwtppppeerillleepppeeeeeeeeeeeee fttps//aapeeeee///2///... 00445577800a220c4eee00000-000-00
_nd Benchmarks.
pdf
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pp. 2681-2691. PMLR, 2019.
Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):300-327, 2016.
Moksh Jain, Emmanuel Bengio, Alex Hermandez- Jarrid Rector- Brotor-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786-9801. PMLR, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. arXiv preprint arXiv:2310.02710, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 11X1v:2110.06169, 2021.
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández- Garcia, Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pp. 18269-18300 PMLR, 2023.
11

---

Published as a conference paper at ICLR 2025
Jiahe Li, Wenjie Du, and Yang Wang. Molclw: Molecular contrastive learning with learnable weighted substructures. 2024 IEEE Interaational Conference on Bioinformatics and Biomedicine (BIBM), pp. 828 831. IEEE, 2024.
Dianbo Liu, Moksh Jain, Bonaventure FP Dossou, Qianli Shen, Salem Lahlou, Anirudh Goyal, Nikolay Malkin, Chris Chinenye Emezue, Dinghuai Zhang, Nadhir Hassen, et al. Gflowout: Dropout with generative flow networks. In International Conference Machine Learning, pp. 21715-21729. PMLR, 2023a.
Shuchang Liu, Qingpeng Cai, Zhankui He, Bowen Sun, Julian McAuley, Dong Zheng. Peng Jiang, and Kun Gai. Generative flow network for listwise recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1524-1534, 2023b.
George Ma, Emmanuel Bengio, Yoshua Bengio, and Dinghuai Zhang. Baking symmetry into gflownets. arXiv preprint arXiv:2406.05426, 2024.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35: 5955-5967, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint TrXiv:2006.0935) 2020.
Mizu Nishikawa-TTomme, Tristan Deleu, Jithendaraa Subramanian, Yoshua Bengio, and Laurent Charlin. Bayesian learning of causal structure and mechanisms with gflownets and variational bayes. arXiv preprint arXiv:2211.022 02763,, 2022.
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. arXiv preprint arXiv:2210.03008 2022.
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-training and fine-tuning generative flow networks. arXiv preprint arXiv:2310.03-/1 2023a.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gfoowets with local credit and incomplete trajectories. In International Conference on Machine Learning, pp. 26878-26890. PMLR, 2023b.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. In Uncertainty in Artificial Intelligence, pp. 1628-1638. PMLR, 2023c.
Malak Pirtskhalava, Anthony A Amstrong, Maia Grigolava, Mindia Chubinidze, Evgenia Alim- barashvili, Boris Vishnepolsky, Andrei Gabrielian, Alex Rosenthal, Darrell E Hurt, and Michael Tartakovsky, Dbaasp v3: database of antimicrobiallcyttooiii activity and structure of peptides as resource for development of new therapeutics. Nucleic acids research, 49(D1):D288-D22 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17710:1707.06347. 2017.
Richard Sutton.
Reinforcement learning: An introduction. A Bradford Book, 2018.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Haozhi Wang, HAO Jianye, Yinchuan Li, et al. Regularized offline gflownets. 2023.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang. Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. arXiv preprint arXiv:2103 (0t.:/033//44. 10432, 2021.
David W Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. arXiv preprint arXiv:2302.05444 2023a.
12

---

Published as a conference paper at ICLR 2025
Dinghuai Zhang, Jie Fu, Yoshua Bengio, and Aaron Courville. Unifying likelihood-free inference with black-box optimization and beyond. arXiv preprint arXiv:2110.03372, 2021.
Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets and beyond. arXiv preprint vrXiv:2209.02666 2022a.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning, pp. 26412-26428 PMLR, 2022b.
Dinghuai Zhang, Ricky Tian Qi Chen, Cheng-Hao Liu, Aaron Courville, and Yoshua Bengio. Diffu- sion generative flow samplers: Improving learning signals through partial trajectory optimization. arXiv preprint arXiv:2310.0267> 2023b.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. Advances in neural information processing systems, 36:1 1952-11966 2023c.
Dinghuai Zhang, Ling Pan, Ricky TQ Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows. arXiv preprint arXiv:2302.05799 2023d.
Jiahui Zhang, Wenjie Du Di Wu, Jiahe Li, Shuai Zhang, and Yang Wang. Improving efficiency in rationale discovery for out-of-distribution molecular representations. In 2023 IEEE International Conference Bioinformatics and Biomedicine (B/BM), pp. 401-407. IEEE, 2023e.
Mingyang Zhou, Zichao Yan, Elliot Layne, Nikolay Malkin, Dinghuai Zhang, Moksh Jain, Mathieu Blanchette, and Yoshua Bengio. Phylogfn: Phylogenetic inference with generative flow networks. arXiv preprint arXiv:2310.0877 2023.
Heiko Zimmermann, Fredrik Lindsten, Jan- Millee van de Meent, and Christian A Naesseth. A variational perspective on generative flow networks. arXiv preprint arXiv:2210.07992, 2022.
A TIME CONSUMPTION
In this part, we compare the training cost between the proposed method with online baselines. Compared to online models, COFlowNet is sampling-free during training, resulting in a reduction in training time. In Table. 2 we report the average training time per epoch for both COFlowNet and the baseline models. Since online models sample batches during training while offline models trained on batched states, report the time of online and offline models training number of states (i.e,, the number of states the offline dataset), which we term as an epoch. The reported times reflect the average duration for 100 epochs. It is worth noting that exclude the online evaluation time consumption of the online model (called oracle or proxy model) and the time consumption of training proxy models, As shown in Table. 2 the offline version of vanilla GetowNet. i.e., MlM-Cotl takes approximately 1/3 less time to train than GFlowNet. Even with a more complex matching method, COFlowNet remains faster than GFlowNet and is approximately twice as fast as DMM-FlooNet. This demonstrates that COFlowNet offers significant efficiency advantages in terms of time cost.
the same
Model
GFlowNet
QM-GFlowNet
FM-COFlowNNt
COFlowNet
Training time (s/epoch)
1.594
2.904
1.055
1.521
Table 2: Comparison of training time consumption. Darker blue denotes the best result and lighter blue denotes the second best.
B EXPERIMENTS ON MORE TASKS
In Sec. we have evaluated COFlowNet on two distinct tasks: the Hypergrid task and the Molecule Design task. Furthermore, method is inherently generalizable to any domain where GelowNets are applicable. To further validate the efficiency of COFlowNet beyond these tasks, we have extended experiments other tasks.
13

---

Published as a conference paper at ICLR 2025
B.1 ANTI-MICROBIAL PEPTIDE DESIGN
To further validate the efficiency of COFlowNet beyond these tasks, we have extended our experiments include the Anti-Microbial Peptide Design task (Pirtskhalava et al., 202T). In this task, the objective is to generate peptides (short protein sequences) with anti- nnti-microbial properties, where actions involve selecting amino acids from predefined with 20 elements and the longest sequence is with 50 elements. For given model, we have D denoting the set of generated candidates with top 100 scores, and evaluate the methods using the following three metrics,
Performance: The average score/reward of the top 100 generated candidates.
Performance(D) EEED reward(2)
(16)
[D]
Diversity: The average pairwise distance among the top 100 candidates.
Diversity (D) = Ex,,, ED d(z
(17)
D -
where -) is the Levenshtein distance between two sequences. Novelty: The average distance between the top 100 candidates and known peptides, indicating the ability to generate new peptides.
Novelty(D) Exeen mins,eDo d(z;, 83)
(18)
D
where Du is the dataset used for training the proxy model.
Model
Performance
Novelty
Diversity
QM-GFlowNet
0.895
29.12
12.14
GelowNet
0.868
15.72
11.32
COFlowNet w/o
0.788
25.68
10.43
FM-COFlowNet
0.853
28.53
13.44
COFlowNet
0.878
28.88
12.45
Table 3: The average reward of the top candidates. Darker blue denotes the best result and lighter blue denotes the second best.
The experimental results are summarized in Table. showing that COFlowNet achieves superior performance across all three metrics, comparable advanced online models despite being trained offline. These results further substantiate COFlowNet's capability to generate high-quality and diverse candidates across various tasks.
In summary, our COFlowNet demonstrates consistent and significant improvements over other methods across a diverse range of tasks, highlighting its broad applicability and efficiency.
B.2 ITEM RECOMMENDATION
To further demonstrate the effectiveness of COFlowNet in diverse domains, have extended our experiments to another classic and practical task in the domain of online businesses, item recommendation, inspired by the literature Liu al. (023b)). In this task, for given user u, models are required to sample trajectories, where each state corresponds to an item. Notably, instead of calculating rewards on the terminal states, we calculate rewards based on the whole trajectories in this task, which correspond to lists of items. This task aims to provide diverse recommendations with high quality, which is exactly good application of the problem studied this paper.
Experimental setting. The experiment is conducted on MLIM dataset, subset of the Movielens datase We report three key metries for evaluation: average reward (Avg.R), max reward (Max.R), and Coverage. We employ the online user environment proposed in Liu et al. (2023b) to calculate
"The dataset available http:///////oooaaa.oooooaaaaaaaa
14

---

Published as a conference paper at ICLR 2025
the preference of users to items, denoted as s(u, a), which returns the user u's response of item a. Given a list of items A and a user u, the reward of calculated as
R(u, A)
Σ aEA
s(u, a).
A
In order to evaluate the diversity of candidates generated by the model, we employ the Coverage metric, which describes the number distinct items exposed in a batch of generated lists items.
Experimental results and analysis. The results summarized in Table 4 According to the results, COFlowNet achieves similar performance to QM-GFlowNet generating high-score candidates. Considering the inherent distinctions between online and offline models, the small performance gap between COFlowNet and QM-GFlowNer is predictable and acceptable. However, COFlowNet is able to generate candidates with more diversity than QM-GFlowNet, evidenced by the Coverage metric. The Avg.R and Max.R gaps between COFlowNet w/o and COFlowNet further demonstrate the effectiveness of the proposed conservative constraint. The poor performance of COFlowNet w/o indicates that it is difficult to develop an effective offline model. Our proposed conservative constraint enables effective offline training. Additionally, COFlowNet, equipped with the quantile matching mechanism, outperforms FM-COFloWNer across all three metrics, validating the effectiveness of the proposed conservative quantile matching (CQM) objective introduced Eq.
Model
Avg.R
Max.R
Coverage
GelowNet
1.996
2.832
47.95
QM-GFlowNe
2.016
2.865
65.61
COFlowNet w/o
1.280
2.692
115.1
Net
1.664
2.800
62.15
COFlowNet 1.998
2.850
109.35
Table 4: Experimental results of item recommendation task.
Therefore, these results collectively demonstrate that COFlowNet exhibits excellent offline learning capabilities and consistent effectiveness across various tasks, and contributes to the research community in diverse valuable and practical domains.
15


Reviews:
[{"review_id": "WjabiGruxE", "reviewer": "Reviewer_z4PM", "summary": "This paper tackles the generation task, which aims to search for state transitions with high rewards. The fundamental task strongly impacts generating or searching for plausible candidates of sequences, where its possible applications include molecule design or path generation.\nTo this end, the authors have proposed COFlowNet, which focuses on finding high-reward state transitions by simply regularizing unsupported flows. The efficacy of the method is confirmed in Hypergrid and Molecule Design tasks.", "questions": "**Questions 1:** Regarding the weakness in literature, is there any other task to demonstrate the efficiency of the proposed method? If so, it would be much better to add one or two more results of COFlowNet.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "**Strength 1:** The main strength of this work is the broad impact on any candidate-generation tasks.\n- I feel that the method can be applied to any candidate-generation task, such as molecule design. Also, as pointed out in the paper, flow generation can be further used to search plausible flows to find the optimal decision-making in the RL study.", "weaknesses": "**Weakness 1:** The comparison with prior works and the evaluations seem limited.\n- By referring the baseline paper of GFlowNet, it is compared with other non-flow-based approaches such as MCMC (MARS) and PPO. However, COFlowNet is only compared with GFlowNet, which makes it hard to figure out the quantitative gains of COFlowNet over MCMC and PPO. It is better to add these two baselines in the experiments.\n- Moreover, only the main experiment considered in this paper, which includes the comparisons with others, is Molecule Design. (I have not published works for this topic) In literature, is there any other task to demonstrate the efficiency of the proposed method? A single demonstration seems to be limited to say the consistent and meaningful gains of COFlowNet over others.\n\n**Minor Comment 1:** For consistency, please choose the one among COFlowNet or COFlownet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gIBrJo1lG9", "reviewer": "Reviewer_9yMW", "summary": "The paper present a method to the problem of offline GFlowNet training. Many conventional approaches require approximating training scores/rewards of candidate model outputs to provide a variety of generated candidate solutions, however, this approach is sensitive to out-of-distribution data. In this work the authors remove any reliance on proxy models for estimating the candidate to reward optimization objective for GFlowNet training. The authors propose to constrain the actions that a GFlowNet model may take to generate an output sample such that only actions supported by the training data are permissible. The authors show that their flow regularization only positively affects training on the supported flow paths, and that their approach is truly offline and can improve candidate model diversity.", "questions": "1. No references are provided for vanilla GFlowNet and QM-GFlowNet. There is no justification that these are the state-of-the-art methods.\n2. Although COFlowNet w/o shows that the regularization term enhances performance, but can not beat the competing method QM-GFlowNet. The experiment results are weak. If the experiment results can not be improved, please provide a more in-depth analysis of where COFlowNet falls short compared to QM-GFlowNet.\n3. Are there other potential ways of increasing diversity during GFlowNet regularization that were explored aside from the quantile matching algorithm? \n4. Can COFlowNet be easily extended to other domains where RL is popular, like game playing and autonomous vehicle driving for example?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors demonstrate originality in their work by clearly outlining their research problems in the context of other popular works and systematically laying out solutions for them with corresponding theorems and proofs. While maintaining an easy-to-read and high-level dialogue in addressing their approach’s significance, aim, and scope, the authors also manage to spare no mathematical details when necessary. Theorem 1 and the writing in sections 3.3 and 4 show formally the details required to both implement and understand the proposed approach.", "weaknesses": "In theorem 2, the use of the phrase “be more close to policy of offline dataset” seems too imprecise to be in the theorem statement. Since the proof is rather short this could perhaps be moved to the end of the proof, so that reader’s still get the high-level point that the author is trying to convey. Additionally, the conclusion of the proof for theorem 2 seems to be absent, although the technical/algebraic details seem sound. \n\nAlthough the authors effectively demonstrate the efficacy of COFlowNet on the Molecule Design problem, providing experimental results on datasets from other domains would help show the success of their method beyond a shadow of a doubt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhZPVJah6m", "reviewer": "Reviewer_7QzZ", "summary": "In this paper, the author introduces COFlowNet, an offline Generative Flow Network (GFlowNet) designed to generate high-quality and diverse candidates without relying on online evaluations. The effectiveness of COFlowNet is validated through experiments on commonly used datasets.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Adapting GFlowNets for offline scenarios is a valuable idea.\n+ The proposed method shows superiority over baseline models like QM-GFlowNet and FM-GFlowNet.", "weaknesses": "1. The authors conduct experiments on Hypergrid and molecule design tasks. Could the proposed method be generalized to more tasks and real applications? It is recommended to conduct experiments on additional tasks.\n\n2. How much time and computational cost are saved by training in the offline setting? Could the authors compare the training cost between the proposed method and traditional methods?\n\n3. It would be better to use \\textbf{} for the first sentence in figure and table captions to indicate their targets clearly.\n\n4. It would be  better to indicate the meaning of 'mixed', 'expert', and 'random' in Figure4.\n\n5. Do number of modes and ℓ1 error are commonly used in evaluation of GFLowNets? Are there more metrics to measure the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)