Order-Preserving GFlowNets
Authors: Yihang Chen
Lukas Mauch
Venue: ICLR 2024 poster (2024)
Keywords: probabilistic sampling
multi-objective optimization
GFlowNet
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Generative Flow Networks (GFlowNets) have been introduced as a method to sample a diverse set of candidates with probabilities proportional to a given reward. However, GFlowNets can only be used with a predefined scalar reward, which can be either computationally expensive or not directly accessible, in the case of multi-objective optimization (MOO) tasks for example. Moreover, to prioritize identifying high-reward candidates, the conventional practice is to raise the reward to a higher exponent, the optimal choice of which may vary across different environments. To address these issues, we propose Order-Preserving GFlowNets (OP-GFNs), which sample with probabilities in proportion to a learned reward function that is consistent with a provided (partial) order on the candidates, thus eliminating the need for an explicit formulation of the reward function. We theoretically prove that the training process of OP-GFNs gradually sparsifies the learned reward landscape in single-objective maximization tasks. The sparsification concentrates on candidates of a higher hierarchy in the ordering, ensuring exploration at the beginning and exploitation towards the end of the training. We demonstrate OP-GFN's state-of-the-art performance in single-objective maximization (totally ordered) and multi-objective Pareto front approximation (partially ordered) tasks, including synthetic datasets, molecule generation, and neural architecture search.

Published as a conference paper at ICLR 2024
ORDER-PRESERVING GFLOWNETS
Yihang Chen Section of Communication Systems EPFL, Switzerland
Lukas Mauch Sony Europe B.V.
Stuttgart Laboratory 1, Germany lukas www.mauch@soo.. sonchhh@oo.... com
yihang. chen@epfl. ch
ABSTRACT
Generative Flow Networks (GFlowNets) have been introduced as method to sample diverse set candidates with probabilities proportional to a given reward. However, GFlowNets can only be used with predefined scalar reward, which can be either computationally expensive or not directly accessible, in the case multi-objective optimization (MOO) tasks for example. Moreover, prioritize identifying high-reward candidates, the conventional practice to raise the reward higher exponent, the optimal choice of which may vary across different en- vironments. To address these issues, we propose Order-Preserving GFlowNets learned reward
(OP-GFNs), which sample with probabilities in proportion function that is consistent with a provided (partial) order on the candidates, thus eliminating the need for an explicit formulation of the reward function. We theoret- ically prove that the training process of OP-GFNs gradually sparsifies the learned reward landscape in single-objective maximization tasks. The sparsification con- centrates on candidates of higher hierarchy in the ordering, ensuring exploration at the beginning and exploitation towards the end the training. We demonstrate OP-GFN's state-of- tate--of-the-art performance in oingle-bbjective maximization (totally ordered) and multi-objective Pareto front approximation (partially ordered) tasks, including synthetic datasets, molecule generation, and neural architecture search.
INTRODUCTION
Generative Flow Networks (GFlowNets) are a novel class of generative machine learning models. As introduced by Bengio al. (2021b). these models generate composite objects x E X' with probabilities that are proportional to given reward function, i.e., p(x) x R(x). Notably, GFlowNets can represent distributions over composite objects such sets and graphs. Their applications include the design of biological structures (Bengio et al. 2021a Jain et al. (0022). Bayesian structure learning (Deleu et al. 2022; Tochikaww-Toomeer et al.] (2022), robust scheduling (Zhang et al. 2023a),
and graph combinatorial problems (Zhang et al. (023b). GFlowNets can also provide a route to unity many generative models (Zhang et al. 2022), including hierarchical VAEs (Ranganath et al. 2016). normalizing flows (Dinh et al. 2014), and diffusion models (Song et al. (222))
We investigate GetowNets for combinatorial stochastic optimization specifically. want maximize set of D objectives over X. uzx) E R., In the objective space IR,,, we define the the Pareto dominance on vectors u, u' E R such that u I u + Vk, uk S u/p- We remark that < induces total order on for D = 1, and a partial order for D We identify two problems:
black-box functions. More
1. GFlowNets require explicit formulation of a scalar reward R(x) that measures the global quality of an object æ. In the multi-objective optimization where D > GFlowNets cannot be directly applied and u(x) has to be scalarized prior (Jain al. 2023 Roy al. 2023). Besides, evaluating the exact value of the objective function might be costly. For example, in the neural architecture search (NAS) task (Dong al. 202 evaluating the architecture's test accuracy requires training the network to completion.
1 Our codes are availahle
A partial order is homogeneous binary relation that is reflexive, transitive, and antisymmetric. A total order is partial order in which any elements comparable.

---

Published as a conference paper at ICLR 2024
2. For scalar objecive function u(x) where D 1, to prioritize the identification of candidates with high u(x) value, GFlowNets typically operate on the exponentially scaled reward R(x) = (u(z))}, termed as GFN-3 (Bengio et al. 202 Note, that the optimal heavily depends on the geometric landscape of u(x)). A small / may hinder the exploitation of the maximal reward candidates since even perfectly fitted GFlowNet will still have high probability of sampling outside the areas that maximize u(r). On the contrary, a large 3 may hinder the exploration since the sampler will encounter a highly sparse reward landscape in the early training stages.
In this work, we propose a novel training criterion for GelowNets, termed Order- Pedder-reeseving (OP) GFlowNets. The main difference to standard GelooNetss is that OP-GFNs are not trained to sample proportional to an explicitly defined reward R: : X - I but proportional to a learned reward that should be compatible with a provided (partial) order over X' induced by u.
We summarize our contributions in the following:
1. We propose the OP-GFNs for both the single-objective maximization and multi-objeccive Pareto approximation, which require only the (partial- Jordering relations among candidates.
2. We empirically evaluate our method on synthesis environment HyperGrid (Bengio et al.. [ and two real-world applications: NATS-Bend (Dong et al. 2021), and molecular designs (Shen et al. 2023; Jain et al. (023) to demonstrate its advantages in the diversity and the top reward (or the closeness to the Pareto front) of the generated candidates.
3. In the single-objective maximization problem, we theoretically prove that the learned reward is piece-wise geometric progression with respect to the ranking by the objective function, and can efficiently assign high credit to the important substructures of the composite object.
4. We show that the learned prder-preserving reward will balance the exploration in the early stages and the exploitation in the later stages of the training, by gradually sparsifying the reward function during the training.
2 PRELIMINARIES
2.1
GEL
We give some essential definitions, following Section 3 of (Bengio et al. 2021b). For a directed acyelic graph g (S, A) with state space S and action space A, we call the vertices states, the edges actions. Let 8 E S be the initial state, i.e., the only state with no incoming edges; and X' the set of terminal states, i.e., the states with no outgoing edges. We note that Bengio et al. (202 la) allows terminal states with outgoing edges. Such difference can be dealt with by augmenting every such state s by a new terminal state s' with the terminal action $ + TT
A complete trajectory is a sequence of transitions T= (50-+11+7 state 8q to a terminal state 8n with ($c->8141) A for all S < 1. Let be the set of complete trajectories. A trajectory flow is a nonnegative function F T- For any state 8, define the state flow F(s) = ->ser F(T), and, for any edge 8-28', the edge flow F(s-t')= = Er=(..-o-o/_.. F()). The forward transition Pp and backward transition probability are defined as Pp(('(s) = F(s + (/).
s.,) going from the initial
= F(s - F()/F(s') for the consecutive state s, s'. We define the normalizing factor Z = E F(x). Suppose that nontrivial nonnegative reward function R: X + Rzo is given on the set of terminal states. GFlowNets (Bengio et al. 2021a) aim to approximate Markovian flow F(-) on the graph & such that F(x)= [x) E
In general, an objective optimizing for this equality cannot be minimized directly because F(x) is sum over all trajectories leading x. Previously, several objectives, such as flow matching, detailed balance, trajectory balance, and subtrajectory balance, have previously been proposed. We list their names, parametrizations and respective constraints in Table 2.1 I. By Proposition 10 in |engio et al. (2021b) for flow matching, Proposition 6 of [Bnggi et al.) (b02 for detailed balance, and Proposition of Malkin et al. (2022) for trajectory balance, the training policy has full support and respective constraints are reached, the GFlowNet sampler does sample from the target distribution.
2.2 MULTI-OBJECTIVE OPTIMIZATION
The Multi Molti-Objective Optimization (MOO) problem can be described as the desire to maximize a set of D 1 objectives over X. We summarize these objectives in the vector-valued function
2

---

of pref- However, the problem of Published as a conference paper at ICLR 2024
Method
Parametrization (by 9)
Constraint
Flow matching
P(s -> 8))
Euroee F(s" Esssica P(a-to').
(FM. Bengio et al.
2021aa)
Detailed balance
(r(s)s(s)s) Pa(s)1)
((s)Pr(s11)) F(s)))(()))))
(DB. Bengio al.
2021b
Trajectory balance
Po(o%s))
2 IT Po(a R(x)) [T= Pa(Se-1(8)
(TB. Malkin et al. 2022
SubTrajectory balance
F(a), (olle)) F(sm.) Nm Pr(s))s-1) F((m,) Th Pa(se-118s)
(subTB, |Madan et al. (022))
Table 2.1:
GFlowNet objectives. We define the complete trajectory T = (5o 1:00~+1100 7 + = =x),
and its subtrajectory Sm,
Smg S m. m2 S n.
u(x) E R.D. The Pareto front of the objective set defined by Parcto == fu E S: Aul * E S.s.t. 3 n'), and the Pareto front of sample set X is defined by Pareto(X)) == fi E X da' A x, s.t. u(x) 3 u(x'). The image of u(r) is u == {u(x) x E X.. Such MOO problems typically solved by scalarization methods, i.e. preference goal conditioning.
GFN-based approach to multi-objectivv optimization (Jain et al. 2023), called preference- conditioning GFNsG,,s), amounts to scalarizing the objective function by using erences w: Rw == w T u(x),, where 1 1Tw w = 1, w/ 2
this scalarization is that, only when the Pareto front is convex, Ew, x, E arg max Rw < x² E Pareto( In particular, the "<= relation does not hold on the non-convex Pareto front, which means that some Pareto front solutions never maximize any linear scalarization. Recently, Roy et al. (2023) proposed goal-conditioning (GC-GFNs), more controllable conditional model that can uniformly explore solutions along the entire Pareto front under challenging objective landscapes. A sample x is defined in the focus region if the cosine similarity between its objective vector u and the goal direction d, above the threshold
can obtain the equivalence
u.d, 2 Cg). The reward function Ry depends on Cg. i.e. 9 == {u IR* cos(u, dg) = ||
the current goal 9 that the conditioned reward will only be non-zero in the focus region, i.e., (x) == 1 u(r) 1 [u(xx) E g), where 1(: is the indicator function.
3 METHOD
3.1
OVERVIEW
We define some terminology here. Let Tn = {(ob + s/ = =x:)]= be a batch of trajectories of size B, XB {x,}_² be the set of terminal states that are reached by the trajectories in batch TB. and SB = {u{x:)}{{ be the (vector) objective set. We use u(x) to denote the single or multi- objective function for unified discussion, and u(x) for single-objective function only. We will drop the subscript B when the batch size is irrelevant. Let R(xr) be the predefined reward function used in previous GFlow methods, such as R(x) = (u(z)) when D = 1. We call / the reward exponent and assume B = 1 if not mentioned otherwise. We also let R(z) be an undetermined reward function that we will learn in the training.
In the GFlowNet framework, R(() is parametrized by the GFlowNet parameters The training of the order- wrder-preserving GelowNet can be divided into two parts: 1) The order-preserving criterion, and 2) The 1) Order preserving. We define local labeling X) for E X batch of terminal states (candidates) X. Let ( be the non-dominated candidates in X, the labeling is defined by ((:; x)= == [[x= E Pareto(X)), which induces the labeling distribution IP (x)X). The reward R(-) also induces a local distribution on the sample X, denoted by PPx R). We have
Process (MDP) constraints.
Py(x|X)= = 1 | E Pareto(X))
R((x)
P(IX,, = Evex R(x')
Vr E X.
(I)
[x))to(X)]
Since we want R(-) to keep the ordering of u, we minimize the KI. divergence. The order-preserving loss is, therefore,
Lor(X; R) KL (LXP,(X)) R?))
(2)
3

---

Published as a conference paper at ICLR 2024
In the TB parametrization, the state flow F(-) is not parametrized. By the trajectory balance equality constraints in Table 2.1 the order- preserirg reward of x on trajectory 7 ending at æ is therefore
RTE(I; 0) = Ze II Pe(salss-::00/Pa(st--l
(3)
t=1
For the non-TB parametrizations, where the state flow is parametrized, we let R(I; F(;;0)
2) MDP Constraints. For the TB parametrization, where each trajectory is balanced, the MDP constraint loss is 0. For non-TB parametrization, we introduce hyperparameter AOp to balance the order-preserving loss and MDP constraint loss. For the ease of discussion, we set AOP = 1 for TB parametrization. We defer the detailed descriptions to Appendix
3.2 SINGLE-OBIECTIVV MAXIMIZATION
In the single-objective maximization, i.e. D = 1, there exists a global ordering, or ranking induced by u(z),, among all the candidates. We consider the scenario where the GelowNet used to sample arg m u(x). We assume the local labeling is defined pairs, i.e., X (x,,²)) Therefore, from Equation T. the labeling and local distributions are,
Py (x(x ((u(x)= > u(x')) + ((u(x) 2 u(x'))
R(x) P(x|X, = R(x) + R(x')*
2
and consequently we can calculate Equation 2 of Lop = (x, x²); R2).
In the following, we provide some theoretical analysis under the single-objective maximization task. We defer our experimental verifications to Figure E.6 in Appendix E.2.3. where we visualize the
average of learned R?(.) on states of the same objective values.
1) Ordde-Preserving, We claim that the learned reward R(x;) is piece- piece-wise geometric progression with respect to the ranking i. We first consider the special case: when u(x;) is mutually different, log R(x;) is an arithmetic progression.
Proposition 1 (Mutually different). For {a E X, assume that u(x;) < u( < The order peese-prese reward R(z) E [1/7, is defined by the reward function that minimizes the order-pre nndee-press loss for neighboring pairs LOP-N. i.e.,
n
R(-) = arg min r=((=)((1/7.1]
for xor-x((z;) = arg
min r,(())([1>711 1=1
Cor Cor((ss-1, (4) Σ
We have R(x.)) = -//0-1 sis n, and Nor- R) = n log(1 + +17))
In practice, 7 is not fixed value. Instead, y is driven infinity when minimizing LOP-N with a variable 7. Combined with Proposition [ we claim that the order-preserving loss gradually sparsifies the learned reward R(x)) on mutually different objective u(x) during training, where R(z,) + 0,1 n.
We also consider the more general setting, where 3i * j. such that u(x;) = u(x;)) We first give an informal proposition in Proposition Proposition (Informal), For (a, Ji & E X, assume that u(x;) S u(x;), 0 S < S n. Using the notations in Proposition 7 when is sufficiently large, there exists any B., dependent on % such that R 1) R., if u(#;41) > u(x;), and R(2441) B,R if u(x;+1) = u(x;), for 0 < i 1. Also, minimize the LOP-N qith a variable 7 will drive ry + 00, 00, Br + 1.
[ and defer its formal version and proof to Proposition
If we do not fix positive lower bound 1/7, i.c., let R((x) E [0, 1]. minimizing CoP-N defined in Equation will drive a. + 00, - 1 as 7 + 00, which indicates ((z))/((x,) + u(x.) = uxxr), ). and R(x;)/R(xx,) + Do if u(x;;) < w(x;) as training goes on, i.c. R((-) enlarges the relative gap between different objective values, and assign similar reward to states of the same objective values. Since the GFlowNet sampler samples candidates with probabilities in proportion to R(.), it can sample high objective value candidates with a larger probability as the training progresses. We remark that the sparsification of the learned reward and the training of the GFlowNet happens simultaneously. Therefore, the exploration on the early training stages and the exploitation on the later sparsified reward R(.)) can both be achieved.
4

---

Published as a conference paper at ICLR 2024
(B) MDP Constraints, In this part, match the flow F(.) with R(-), where R(-) is fixed and learned in Proposition [ We consider the sequence prepend/append MDP (Shen et al. 2023).
Definition 1 (Sequence prepend/append MDP). In this MDP, states are strings, and actions are prepending appending symbol in an alphabet. This MDP generates strings of length I.
In the following proposition, we claim that matching the flow F(-) with sufficiently trained OP-GFN (i.e. sufficiently large 7) will assign high flow value on non-terminal states that on the trajectories ending in maximal reward candidates.
Proposition 3. In the MDP in Definition 7 consider the dataset {I,, F with u(zo) u(x1) ..... < u((n) = u(>²)) We define the important substring 8* as the longest substring shared by Tnz ty and sx(x)) as the sel of k-length substrings of x. Following Proposition 5 let the order- preserving reward R(-) E [1/7, 11, and the ratio Ocy. We fix PB to be uniform, and match the flow F() with R(-) on terminal states. Then, when ac, > 4, we have EF(s") > EF(s), Vs (1))( where the expectation is taken over the random positions of s* in Ing Zny
3.3 EVALUATION
3.3.1
GFlowNets for oingle-objective tasks are typically evaluated by measuring their ability to match the target distribution, e.g. by using the Spearman correlation between log p(z) and R(x) (Madan ct al. 2022 Nica et al.] 2 However, our method learns a GFlowNet to sample the terminal states proportional to the reward R(x) instead of R(x), which is unknown in prior. Therefore, we only focus on evaluating the GFlowNet's ability to discover the maximal objective.
3.3.2 OULTI-OBJECTIVE
The evaluation of the multi-objective sampler is significantly more difficult. We assume the reference set := {p,1,11 is the set of the true Pareto front points, and S = {s; ||| is the set of all the generated candidates, P' = (S) is non-dominated points in S. When the true Pareto front is unknown, we use discretization of the extreme faces of the objective space hypercube as P.
Audet et al. (021) summarized various performance indicators to measure the convergence and uniformity of the generated Pareto front. Specifically, we measure: 1) the convergence of all the generated candidates S to the true Pareto front P, by Averaged Hausdorff distance; 2) the convergence of the estimated Pareto front P' to the true Pareto front P. by IGD+, HyperVolume, Ra indicator. 3) the uniformity of the estimated front P' with respect to P, by PC-ent, R2 indicator. We discuss these metrics in Appendix [F.]] During the computation on the estimated Pareto front, we will not de-duplicate the identical objective vectors.
SINGLE-OBJECTIVE EXPERIMENTS
4.1 HYPERGRID
We study synthetic HyperGrid environment introduced by (Bengio et al. (02 In this environment, the states form a D-dimensional HyperGrid with side length H: ((s ....,," s (H - -1)- sd {0, 1,.., H - 1}, d = I...., D). and non-stop actions operations of incrementing one coordinate in a state by μ-1 without exiting the grid. The initial state is (0,...,0) O). For every state (x) ...... P)T is given by I|rd 0.5| E (0.25, 0.5]] + 2T22 p2 is an indicator function and Ro is constant controlling the difficulty of exploration. The ability of standard GFlowNets, i.e. R(x)) = u(x). is sensitive to Ro. Large Ro facilitates exploration but hinders exploitation, whereas low Ro facilitates exploitation but hinders exploration. However, OP-GFNs only deal with the pairwise order relation, so independent of Rg-
the terminal action is + S x. The objective at the state u(x)= = Ro + 0.5 0.5 Th II
0.5| (0.3, 0.4) where I
We set (D, H)) = (2, 64), (3,32) 32) and (4, 16), and compare TB and order-preserving TB (OP-TB). For (D, H) = (2. 64), we plot the observed distribution on 4000 most recently visited states in
Figure E.4 We consider the following three ratios measure exploration-exploitaion. 1) #(distinetly
visited stales)/#(all the states); 2) #(distinctly visited maximal states)/ #(all the maximal states): 3)
In the most recently 4000 visited states, #(distinctly maximal /4a0 in Figure E.5 A good
5

---

Published as a conference paper at ICLR 2024
sampling algorithm should have small ratio 1), and large ratio 2), 3). We confirm that TB's performance sensitive to the choice of Ry. and observe that OP-TB can recover all maximal areas more efficiently, and sample maximal candidates with higher probability after visiting fewer distinct candidates. The detailed discussions are in Appendix [..222 To conclude, OP-GFNs can balance exploration and exploitation without the selection of Ro.
4.2 MOLECULAR DESIGN
We study various molecular designs environments (Bengio et al. 2021a). including Bag, TFBind8, TFBind10, QM9, sEH, whose detailed descriptions are in Appendix|F.33 formulation (Shen al., 2023) for the molecule graph generation, i.e., sequence prepend/append MDP in Definition | We define the optimal candidates in Bag as the æ with objective value 30, in SIX6, PHO4, QM9 as the top 0.5% x ranked by the objective value, sEH as the top 0.1 a ranked by the objective value. The total number of such candidates for Bag, SIX6, PHO4, QM9, sEH is 3160, 328, 5764, 805, 34013 respectively.
We use the the sequence
We consider previous GFN methods and reward- coterddmaxiiiztiiino methods as baselines. Previous GFN methods include TB, DB, subTB, maximum entropy (MaxEnt, Malkin al. (202)). and substructure- guided trajectory balance (GTB, (Shen et al. (2023). For
methods, we consider a widely-used sampling-bas -based method in the molecule domain, Markov Molecular Sampling (MARS, Xie et al. (202T)), and RL-based methods, including actor-critic (A2C, Mnih et al. (2016)). Soft O-Learning (SQL, Hou et al. (2020)) and proximal policy optimization (PPO, Schulman et al. (2017)). We run the experiments over 3 seeds, and plot the mean and variance of the objective value of top- 100 candidates ranked by u(r), and also plot the number of optimal candidates being found among all the generated candidates in Figure 4.1 We find that the order- dedee--rese method outperforms all the baselines in both the ability to find the number of different optimal candidates and the average top-k performance.
ambdtr
3200
2000
2000
200
300
100
10:0
1393
2003
500
1993
1593
2003
500
3393
3580
2000
3200
1908
2030
5000
10303
15000 20000
Top Serouuucc
0.9
0.5
3.75
0.7
0.5
1099
1393
500
1097
1593
2002
500
2387 Roard
2007
1265
1958
2030
10303
15000
20000
Reund
Reurd
Acaad
40.0d
Ours2e 180
Meatel
148/11
619
bag
am/gt
sehctr
idd
Ubindd0
3000
3300
100
2000
2007
3700
1200
100
5.0
1050
1350
500
1003
1.29
5.5
0.3
1.03
4.0
100
350
1000
5d0
Reand
Rourd
Rlcard
fourd
Figure 4.1: Molecular design: In the environment Bag. QM9, sEH, TFBind8, TFBind10, we test our algorithm (OP-TB) against previous GFN methods (MaxEnt, TB, DB, subTB, GTB), and (RL- >sampling methods (MARS, A2C, SQL, PPO).

---

Published as a conference paper at ICLR 2024
4.3 NEURAL ARCHITECTURE SEARCH
4.3.1 NAS ENVIRONMENT
We study the neural architecture search environment WATS-Benc's (Dong et al. 2021 which includes three datasets: CIFARIO, IO, CIFAR-100 and ImageNet- 1 16-100. We choose the topology search space in NATS-Bench. The neural architecture search can be regarded as sequence generation problem, where each sequence of neural network operations uniquely determines an architecture. In the MDP, each forward action fills any empty position in the sequence, starting from the empty sequence. For sequence I E X. the objective function ur (z) is the test accuracy of x's corresponding architecture with the weights at the 7'-th epoch during its standard training pipeline. To measure the cost to compute ur(.), we introduce the simulated train and test (T&T) time, which is defined by the time to train the architecture to the epoch T and evaluate its test accuracy at epoch T. NATS-Benas provides APIs on (-) and its T&T time for T S 200. Following Dong et al. (021), when training the GFlowNet, we use the test accuracy at epoch 12 (-)) as the objective function in training; when evaluating the candidates, we use the test accuracy at epoch 200 (U200(-)) as the objective function in testing. We remark that (12(-) is proxy for U200 with lower T&T time, and the global ordering induced by u12(-) is also a proxy for the global ordering induced by u200() (-). OP-GFNs only access the ordering of (12()), ignoring the unnecessary information of #12(-)'s 's exact value.
4.3.2 EXPERIMENTAL DETAILS
We train the GFlowNet in multi-trial sampling procedure described in Appendix E.4.2 and option- ally use the backward KL regularization (-KL) and trajectories augmentation (-AUG) introduced in Appendix F. To monitor the training process, in each training round, we will record the architecture that has the highest objective function in training and the accumulated T&T time up to that round. We terminate the training when the accumulated T&T time reaches the threshold, of 50000, 100000 and 200000 seconds for CIFAR 10, CIFAR 100, and ImageNet-166 1oie--16-20000 respectively. We adopt the RANDOM as baselines, and compare our results against previous multi-trial sampling methods: 1) evolutionary strategy, e.g.. REA (Real et al. (2019): 2) reinforcement learning (RL)- methods, e.g.. REINFORCE (Williams) 1992). 3) HPO methods, e.g., BOHB (Falkner al. 2018), whose experimental settings are described in Appendix E.4.2] To evaluate the algorithms, we plot the averaged accuracy (at epochs 12 and 200) of the recorded sequence of architectures with respect to the recorded sequence of accumulated T&T time, over 200 random sceds. We report the results on trajectory balance (TB) and its order- preserrineeeiiiggggggg variants (OP-TB) in Figure 4.2" and defer the results of non-TB methods Appendix E.4.5] where include the ablation studies of (OP-)DB, (OP-) (OP-)subTB, and the hyperparameters Aop. We observe that OP-non-TB methods can achieve similar performance gain with OP-TB, which validates the effectiveness and generality of order-preserving methods. Moreover, we compare the OP-TB with the GFN-3 algorithm in Appendix [E.4.4] where 4, 16, 32, 64, 128, and plot the results in Figure E.8 the GFN-8 method by a significant margin.
We observe that the OP-GFN outperforms
We conclude that order- pieeerries GFlowNets consistently improve over the previous baselines in both the objective functions used in training and testing, especially in the early training stages. Besides, backward KL regularization and backward trajectory augmentation also contribute positively to the sampling efficiency. Finally, once we get trained GFlowNet sampler, we can also use the learned order-preserving reward as proxy to further boost the sampling efficiency, see Appendix F.4.3
5
MULTI-OBJECTIVE EXPERIMENTS
5.1 HYPERGRID
We study HyperGrid environment in Section 4.1 with HD = (2, 32). and four normalized objectives: brannin, currin, shubert, beale, Appendix F.2 for details. All the objectives are normalized to fall between 0 and 1. The true Pareto front of HyperGrid environment can be explicitly obtained by enumerating all the states.
"Sine different runs sample different sequences of architectures, and different architectures have different T&T times, each run's sequence of time coordinates may not he uniformly spaced. We first linearly interpolate each run's 's's sequences of points and then calculate the mean and variance on some fixed reference points.
'We remark that the first 64 candidates of GFN methods are generated by random policy. The point at which we observe sudden performance increase of the GFN methods indicates the start of the training.
7

---

Published as a conference paper at ICLR 2024
CHRARD
OFAR100
16-22
30
05
30
19300
33800
19283
20838
40002
60833
18000
33591
20032 ardes
n2é03 serce
12001
19900
Edivated
Edtrstte cheetive
KANDON
DATB-C
O6TB--,,00
3040
805
RESPOONCE
Figure 4.2: Multi-trial training of GelowNet sampler. Best test accuracy epoch and 200 of random baseline (Random), GFlowNet methods (TB, OP-TB, OP-TB-KI, OP-TB-KI-AUGY and other multi-trial algorithms (REA, BOHB, REINFORCE).
The training procedures are described in Appendix GFlowNet sampler in the evaluation as S, and report the metrics Table F.1 To visualize the sampler, plot all the objective vectors, and the true Pareto front, well the first 128 generated objective vectors, and their estimated Pareto front, in the objective space [0. 1]2 and [o, 1] in Figure F. We observe that sampler achieves better approximation (i.e. almost uniformity (i.e. higher PC-ent) to the Pareto front, especially in the non-convex Pareto front, such as currin- -anubeerr..... We also plot the learned reward distributions of OP-GFNs and compare them with the indicator functions of the true Pareto front solutions in Figure 5.1 where we observe that OP-GFNs can learn a highly sparse reward function that concentrates on the true Pareto solutions, outperforming PC-GFNs.
F.2
We generate 1280 candidates by the learned
IGD+ and smaller dæ). and
branin-currin
branin-shubert
branin-beale
currinsshubert
currn-beale
shubert-beald
0.8
8.05
OP-GFN
0.03
0.02
0.1
0.00
0.06
PC-GN
004
0.00
Figure 5.1: Reward Distribution: We plot the indicator function of the true Parcto front solutions and the learned reward distribution the OP-GFNs and PC-GFNs.
5.2 N-GRAM
The synthetic sequence design task o-gram proposed by Stanton et al. (2022 is to generate sequences of a fixed maximum length L 36. The vocabulary (action) to construct the sequence of size 21, with 20 characters and special token end the sequence. The objectives are defined by the number
8

---

Published as a conference paper at ICLR 2024
of occurrences of given set of n-grams in a sequence We consider unigrams and bigrams in our experiments and summarize the objectives in Table
a replay buffer help stabilize the training (Roy et al. 2023] Appendix C.I). Specifically, instead of the on-policy update, we push the online sampled batch into the replay buffer, and immediately sample a batch of the same size to stabilize the training. We defer detailed experimental settings to Appendix [:3] and the experiment results are summarized in
In the multi-objective optimization,
Table F.3
We observe that OP-GFNS achieve better performances on most of the tasks.
5.3
DNA SEQUENCE GENERATION
An instance illustrating a real-worid scenario where the GFlowNet graph takes the form of a tree is the creation of DNA aptamers, which are single-stran stranded sequences of nucleotides widely employed in the realm of biological polymer design (Zhou et al. 2017)) Yesselman et al.) 2019). We generate the DNA sequences by adding one nucleobase ("A", "C", "T", "G") at a time, with length of 30. We consider three objectives, (1) energy: the free energy of the secondary structure calculated by the software NUPACK (Zadeh et al. 201 (2) pins: DNA hairpin index; (3) pa rs: the number of base pairs. All the objectives
normalized to be bounded by 0 and 1. The experimental settings are detailed in Appendix F.4 We report the metrics in Table F.4] and plot the objective vectors in Figure F2 We conclude that OP-GFNs achieve similar or better performance than preference conditioning, especially in the diversity of the estimated Pareto front.
5.4 FRAGMENT-BASED MOLECULE GENERATION
The fragment-based molecule generation is four-objective molecular generation task, (1) qed: the well-known drug-likenest heuristic QED (Bickerton et al., ) (2) seh; the sEH binding energy prediction of pre-trained publicly available model (Bengio et 2021a); (3) sa: a standard heuristic of synthetic accessibility; (4) mw: a weight target region penalty, which favors molecules with a weight of under 300. All the objectives are normalized be bounded by 0 and
We compare our OP-GFNs to both the preference (PC) and goal conditioning (GC) GFN. To stabilize the training of OP-GENs and GC-GFNs, we use the same replay buffer as in Section [.2 The detailed
experimental settings are in Appendix F.5 In evaluation, we sample 64 candidates per round, 50 rounds using the trained sampler. We plot the estimated Pareto front Figure 5.2 and defer the
full results in Figure F.3 and Table F.5 We conclude that OP-GFNs achieve comparable or better
performance with condition- GFNs without scalarization in advance.
seh-qed
seh-sa
seh-mw
qed-sa
qed-mw
Sa-Mi
X PC-GFN
GC-GFN
CP-GFN
Figure 5.2: Fragment-Bann Based Molecule Generation: We plot the estimated Pareto front of the gener- ated samples in [0, 1Ja. The y-axis are the first, and second objective in the title of respectively,
6 CONCLUSION
In this paper, we propose the order-preserving GFlowNets that sample composite objects with probabilities in proportion to a learned reward function that consistent with a provided (partial) order. We theoretically prove that OP-GFNs learn to sample optimal candidates exponentially more often than non-optimal candidates. The main contribution that our method does not require an explicit scalar reward function in prior, and can be directly used in the MOO tasks without scalarization. Also, when evaluating the objective function's value costly, but the ordering relation, such the pairwise comparison is feasible, OP-GFNs can efficiently reduce the training cost.
We will continue exploring the order-based sampling methods, especially in the MOO tasks. For example, currently resample from the replay buffer to ensure that the training of OP-GFNs does not collapse to part of the Pareto front. In the future, we hope that we can introduce more controllable guidance to ensure the diversity of the OP-GENs sampling.

---

Published as a conference paper at ICLR 2024
REFERENCES
Charles Audet, Jean Bigeon, Dominique Cartier, Sébastien Le Digabel, and Ludovic Salomon. Performance indicators in multiobjective optimization. European journal operational research, 292(2):397-422, 2021.
Luis A Barrera, Anastasia Vedenko, Jesse V Kurland, Julia M Rogers, Stephen Gisselbrecht, Elizabeth Rossin, Jaie Woodard, Luca Mariani, Kian Hong Kock, Sachi Inukai, et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351 (6280):1450-a) 1454, 2016.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS), 2021a.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. arXiv preprint 2111.09266, 2021b.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90 98, 2012.
L. C. Blum and J.. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc., 131:8732, 2009.
Lars Buesing, Nicolas Heess, and Theophane Weber. Approximate inference in discrete distributions with monte carlo tree search and value functions. In International Conference on Artificial Intelligence and Statistics, pp. 624 634. PMLR, 2020.
Carlos A Coello Coello and Margarita Reyes Sierra. A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm. In MICAI 2004: Advances in Artificial Intelligence: Third Mexican International Conference Artificial Intelligence, Mexico City, Mexico, April 26-30, 2004. Proceedings 3. pp. 688-697.65 Springer, 2004.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Xuanyi Dong and Yi Yang. Nas-bench-2011 Extending the scope of reproducible neural architecture search. arXiv preprini :00i:::20010003220 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence, 44(7):3634 2021.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. In International Conference on Machine Learning. pp. 1437-1446. PMLR, 2018.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. International Conference on Machine Leurning, pp. 3061-3071. PMLR, 2020.
Carlos M Fonseca, Luís Paquete, and Manuel López-lbánez. An improved dimension-sweep algorithm for the hypervolume indicator. In 2006 IEEE international conference on evolutionary computation, pp. 1157-1163. IEEE, 2006.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep ensrgy-based policies. International Conference on Machine Learning (ICML), 2017.
Tuomas Haamojj, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference Machine Learning (ICMI.), 2018.
10

---

Published as a conference paper at ICLR 2024
Michael Pilegaard Hansen and Andrzej Jaszkiewicz. Evaluating the quality of approximations the non-dominated set. Citeseer, 1994.
Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforeement learning: Soft actor-critic with advantage weighted mixture policy (sac-awmp). arXiv preprint arXiv:2002.02829, 2020.
Hisao Ishibuchi, Hiroyuki Masuda, Yuki Tanigaki, and Yusuke Nojima. Modified distance calculation in generational distance and inverted generational distance. In Evolutionary Optimization: 8th International Conference, EMO 2015, Guimarães, Portugal, March 29-April 1, 2015. Proceedings, Part pp. 110-125. Springer, 2015.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcias Jarrid Rector-Brooks, Bonaventure F.P. Dossou, Chanakya Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequenee design with GFlowNets. International Conference Machine Learning (ICML), 2022.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Garcia. Jarrid Rector- Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International PMLR, 2023.
Conference Machine Learning, pp.
Minsu Kim, Joohwan Ko, Dinghuai Zhang, Ling Pan, Tacyoung Yun, Woochang Kim, Jinkyoo Park, and Yoshua Bengio. Learning to scale logits for temperature-condiitoonn gflownets. arXiv preprint arXiv:2310.02822 2023a.
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. arXiv preprini arXiv:2310.002110, 2023b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 69800, 2014.
Salem Lahlou, Joseph D Viviano, and Vietor Schmidt. torchgfn: A pytorch gflownet library. arXiv preprint arXiv:2305. 140.:44445...44, 2023.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: /001v 1806.09055 2018.
Kanika Madan, Jarrid Rector-Brooks. Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. arXiv preprint arXiv:2209. 12782, 2022.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201.13253 2022.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Grégoire Montavon, Matthias Rupp. Vivekanand Gobre, Alvaro Vazquez-Mayagoitta. Katja Hansen, Alexandre Tkatchenko, Klaus-Robert Müller, and O Anatole von Lilienfeld. Machine learning of molecular electronic properties in chemical compound space. New Journal of Physics, 15(9): 095003, 2013. URL http://ssscccaa.ooooooo...a/////
Ofr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Neural Information Processing Systems (NeurIPS), 2017.
Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Mizu Nishikawa- Toommy, Tristan Deleu, Jithendaraa Subramanian. Yoshua Bengio, and Laurent Charlin. Bayesian learning of causal structure and mechanisms with gflownets and variational bayes. arXiv preprint arXiv:22 1.02663, 2022.

---

Published as a conference paper at ICLR 2024
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. arXiv preprint arXiv:2210.03306 2022.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. arXiv preprint arXiv:2302.0168/ 2023.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International conference on machine learning, pp. 333. PMLR, 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. arXiv preprint arXiv:2306. 17693, 2023.
Julien Roy, Pierre-Luc Bacon, Christopher Pal, and Emmanuel Bengio. Goal-conditioned gflownets for controllable multi-objective molecular design. arXiv preprint urXiv:2306.04620, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.063: 1707.06347 2017.
Oliver Schutze, Xavier Esquivel, Adriana Lara, and Carlos A Coello Coello. Using the averaged hausdorff distance performance measure in evolutionary multiohjective optimization. IEEE Transactions on Evolutionary Computation, 16(4):504-522, 2012.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. arXiv preprint arXiv:2305.07170, 2023.
Julien Siems, Lucas Zimmer, Arber Zela. Jovita Lukasik, Margret Keuper, and Frank Hutter, Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.0977 2020.
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL /ttpnr/e///peeeeeee.... ne. forumorrmm
Id=PxTIG12RRHS
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pp. 20459-20478. PMLR, 2022.
David Allen Van Veldhuizen. Multiobjectivee evolutionary algorithms: classifications, analyses, and new innovations. Air Force Institute of Technology, 1999.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5-32, 1992.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: Markov molecular sampling for multi-objective drug discovery. International Conference on Learning Representations (ICLR), 2021.
Joseph D Yesselman, Daniel Eiler, Erik D Carlson, Michael R Gotrik, Anne E d' Aquino, Alexandra N Ooms, Wipapat Kladwang, Paul D Carlson, Xuesong Shi, David A Costantino, et al. Computational design of three-dimensional ma structure and function. Nature nanotechnology. 14(9):866-873, 2019.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101 101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114 PMLR, 2019.
12

---

Published as a conference paper at ICLR 2024
Joseph N Zadeh, Conrad D Steenberg, Justin S Bois, Brian R Wolfe, Marshall B Pierce, Asif R Khan, Robert M Dirks, and Niles A Pierce. Nupack: Analysis and design of nucleic acid systems. Journal of computational chemistry, 32(1):170- 173, 2011.
David W Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. arXiv preprint arXiv:2302005446, 2023a.
Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. arXiv preprint arXiv:2209.0060 02606, 2022.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint MrXi::2305.7001 2023b.
Dinghuai Zhang, Ling Pan. Ricky TQ Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows. arXiv preprint arXiv:2302.05793 2023c.
Wenhu Zhou, Runjhun Saran, and Juewen Liu, Metal sensing by dna. Chemical reviews, 117(12): 8272-8325, 2017.
13

---

Published as a conference paper at ICLR 2024
A OVERVIEW OF APPENDIX
We give a brief overview of the appendix here.
Appendix
we discuss related work on reinforcement learning, discrete Gets and
NAS Benchmark.
Appendix C
we incorporate order-preserving loss with GFlowNets criteria other than
trajectory balance.
Appendix D
we provide the formal descriptions and missing proofs of Proposition Proposition and Proposition in Section 3.2
Appendix
E
Complete single-objective experiments.
-Appendix E.1
Preliminaries and tricks in the single-objective experiments.
Appendix
E.2
HyperGrid Experiments: In Appendix E.2. we validate the effec-
tiveness of backward KL regularization proposed in Appendix E. In Appendix [3.222 provide the ablation study of Rp- In Appendix E.2.3 distribution.
plot the learned reward
Appendix E.3 Molecule Experiments: we provide the implementation details for each environment and present complete experimental results.
Appendix [E44, NAS Experiments: In Appendix [E.4.]] we give complete description of the NAS environment. In Appendix [E.42] we provide the implementation details. In Appendix E.4.3 we provide experimental results on boosting the sampler. In Appendix E.4.4 we provide the ablation study GFN-3 methods, KL regularization hyperparameter AKL. and the size of the randomly generated dataset at initialization. we provide the experimental results OP methods in FM, DB and
In Appendix E.4.5
subTB.
Appendix Complete multi-ohjeccive experiments.
In Appendix F.1
we discuss multiple multi-objective evaluation metrics. In Appendix N22 Appendix F.4 Appendix [55] we conduct experiments HyperGrid,
Appendix F.3
n B-gram, DNA sequence generation, and fragment-ben molecule generation separately.
B RELATED WORK
Reinforcement Learning GFlowNets are trained to sample proportionally the reward rather than maximize in standard RL. However, on tree-structured DAGs (autoregressive generation) are equiva- lent to RL with appropriate entropy regularization or soft Q-learning and control as inference (Buesing) ct al. 2020 Haarnoja et al. 2017 2018). The experiments and theoretical explanation of Bengio et al. (a) show how standard RL-based methods can fail in the general DAG case, while GFlowNets can handle it well. Signal propagation over sequences of several actions in trajectory balance is also related to losses used in RL computed subtrajectories (Nachum al., 2017). However, compared with our proposed OP-GFNs, previous RL baselines are limited in their application to the multi-objective optimization and are inferior in the diversity of the solutions.
Discrete GFlowNets GFlowNets were first formulated as a reinforcement learning algorithm (Bengio al. 2021a),, with discrete state and action spaces, that trains sequential sampler that samples the terminal states with probabilities in proportion to a given reward by the flow matching objective. Bengio et al. (202Tb) provides the theoretical foundations of GFlowNets, based on flow networks detined on MDPs, and proposes the detailed balance loss that bypasses the need to sum the flows over large sets of children and parents. Later, trajectory balance (Makin et al.) 2022), trajeeryy balance (Madan et al. (2022). augmented flow (Pan et al. 2022). forward-looking (Pan et al. 2023). quantile matching (Zhang et al. 2023c), local search (Kim et al. 2023b) were proposed to improve the credit assignments along trajectories.
NAS Benchmark The first tabular NAS benchmark to be released was NAS-Bench-lud (Ying et al. 2019). This benchmark consists of 423,622 architectures trained on CIFAR-10. NAS-Bench- 201 (Dong & Yang 2020) is another popular tabular NAS benchmark. The cell-based search space consists of DAG where each edge can take operations. The number of non non-isomorphic architectures is 6,466 and all are trained on CIFAR-10, CIFAR- 100, and ImageNet- 16-100. NATS- Bench (Dong et al. 202T) is extension of NAS-Bench-200 and provides algorithm-agnostic
14

---

Published as a conference paper at ICLR 2024
benchmark for most up-to-date NAS algorithms. The search spaces in NATS- MeTS-Bench includes both architecture topology and size. The DARTS (Liu et al. 2018) search space with CIFAR- 10, consisting of 1018 architectures, but is not queryable. 60 000 of the architectures were trained and used to create NAS-Bench-301 (Siems et al. 2020), the first surrogate NAS benchmark.
Exploration in GFlowNets The exploration and exploitation strategy in GFlowNets has been ana- lyzed recently, and we will discuss the difference here. |RRecto--rrodd -rotoks--rooss-- et al.) (2023) demonstrates how Thompson sampling with GFlowNets allows for improved exploration and optimization efficiency in GelowNets. We point out the key difference in the exploration strategy is that, Srector-Brooks et al. (2023) encourages the exploration by bootstrapping different policies, while in the early stages of OP-GFNs, the learned reward is almost uniform, which naturally facilitates the exploration. As the training goes on, the learned reward gets sparser, and the exploration declines and exploitation arises. We remark that the idea of T'hompson sampling can be used in the exploitation stages of OP-GFNs to further encourage the exploration the latter stages of training.
Cnmpperaaurr-CCoddttoonn GFlowNets Zhang et al. (2023a); Kim et al. (2023a) propose the Temperature-Condiins
GelowNets (TC-GFNs) to learn the generative distribution with any given temperature. Zhang al. (2023a) conditions the policy networks on the temperature, and Kim ct al. 2023a) directly scales probability logits regarding the temperature. We point out some critical differences to OP-GFN: 1) In TC-GFNs, suited B's prior must still be chosen. while OP-GFNs do not require such a choice. 2) TC-GFNs learn to match (x)x) for all 33 while OP-GFNs just learn to sample with the correct ordering statistics. However, using these few statistics, OP-GENs still achieve competitive results in both single and multi-objective optimization. 3) TC-GFNs require the scalar reward function, while OP-GFNs can be directly used in multi-objective optimization.
C ORDER-PRESERVING AS A PLUGIN SOLVER
In the next, we discuss how we can integrate the loss Lop into existing (non-TB) training criteria. In this case, will introduce hyperparameter ÅOp balance the MDP constraint loss. Although AOp is dependent on the MDP structure and the objective function, argue that OP-GFNs less sensitive different choices of Åor compared to the influence of different choices of (3 on GFN-3 methods in Appendix E.4.5.
loss and the
Flow Matching. In the parametrization of the flow matching, denote the order- pidee-rrese reward by (x 0) E( F(s", 0). The loss can be written as:
FMP-FM(TT: = ET Σ LFM(s):0) 0) AOP Lop (xn; (FM(::0))
where
2
(s
log Σ
"s" log Σ
F(s, (s->))EA
(s"-
is the flow matching loss for each state s.
Detailed Balance. In the parametrization of the detailed balance, denote the order-preserving reward at terminal states by RDB(E:; 0) = ((:; x E X. The loss can be written as:
ny-
COP-DB(TB::) EETTI
LDB(S st; + dop Lop (Xp; RDB(::
where
(nn = (log F(x; Pr()))) log F(''; P(s:0)))p()))))))03
is the detailed balance loss for each transition s'.
SubTrajectory Balance. Similar to the detailed balance, the order preservie reward also REuSTB (z; 0) =F(r; 0), E X. The loss can be written as
Lor subTB (To; E
Σ
XM-u Courreu + Aor Lor (X ( 0)),
05ucuSni
15

---

Published as a conference paper at ICLR 2024
where AubBB is the subtrajectory geometric reweighting hyperparameter, and F(NN; 0) [ii=u+ Pr (=(81101111)
C 0)
log
F(s.; 0) Ile=u+1 Pn(
the subtrajectory balance loss for each subtrajectory Tu,e = (su .... 8o). 0 S u n.
D MISSING PROOFS
For completeness, we also restate Proposition [ Proposition 3 Teere in Proposition 4 and Proposi- tion respectivvly.
Proposition 4 (Mutually different). For {xi}i=o E X, assume the objective function u(x) is known, and u(xa)) < u(ar),, < n. The order-preserving reward R(x) [1/7, 11, < < b, is defined by the reward function that minimizes the order-preserving loss for neighboring pairs COP-X:
R(-) == arg
min (
Cop COP-N(f2i)--0)) = arg
min
Σ 1=1
(.:( (5)
r
We have A(x,) = y s/n-1 0 sis n, and Cop- ((f R) = n log 117))
Proof of Proposition Let Lor(r) Lop Cop-N((Z;) r) for abbreviation. Since the objective
(rp(r) decreases as (xx) decreases or r(zn) increases, we have R(zo) = 1/7, R(xn) = We denote r(z;)) by T2. 0 < kism.
Let us consider R(x;),, Si
since
@L
1 ri-1 +Ts
1 1 =0 0 Vi-IT
Ōr:
F1+1 +r,
r;
Therefore, R(x,) =
1 ssis n L, and the second order derivative
82 (rr(r)
2
-= r3 (1+1/7)) > 0,
0r?
which proves it minimize (rp(r).
Proposition 5. For {x:)²=0 E X, assume the objective function u(x) is known, and u(x;) S u(z)), < i < S We define two subscript sets:
1 = {i: u(x;)= < u(xx), 0 < i n n-1), 1}, 1= (i: u((;)) = u( -(
We introduce two auxiliary states x-1 and an+ for boundary conditions, such that u(x-1)
-oo, ((xu()))) = + The order- pndee-preserving reward R(x) E [1/7, 1) is defined by minimizing the order-preserving loss with neighboring states:
arg min -(r-N( U (x-1, = arg min >r(()))l1/:11
Σ
Lor (or(2)-1
Let m [11, and define one auxiliary function:
4
n-w
[i(a): == m42
(1-
+3
Since f1(a), (1 m+T) are both monotonically increasing from to infinity for a, Y 2 1. There exists unique "/o, a2 > 1 such that
fi(0,)= =Y, f1(10 =o
For Yo: we have
((om)) a, h, R(( B, Is = a, -1
a, +3"
Also, minimizing for-N n will drive 7 +00, and hence ay - +00,
Note that for regular x X, we have u(z)) E (0,+
16

---

Published as a conference paper at ICLR 2024
Proof of Proposition 5 We can expand the order-preservin loss by separating two cases where -(;;-1) < u(x;)) or u(=;-1) = u(x;) for 0 < $ n + 1. We remark that u(x-1) < u(zo) and u(x,)) u(zx+1) by the definition of 2- and 2n++1
arg min
((p-N((z))E U (x-1, k
n+1
min == arg (
(or(1-11,
= arg
min (x())([L/sa)
log r(a++1)
+ r(ait1) log r(x,) (xx+1) r(xx) + r((541)))
r(ar))
Σ k,)
Σ
r(z;) + (((441)) 2 ich
(n+1, r) for abbreviation. We denote r(z;) by rss - i S n + 1, and define a;-1 such that r, = a4-19i-1.0 S i S n n decreases r-1 decreases rx+1 increases, we have R(x-1) 1/7, R(xn+1) 1.
Let (op(r) == Lop-N
+1+.111 1. Since the objective Lop(r)
We consider the terms involving rs. S i i m in the order- onder-preserving loss. (1) If u(x;=1) u(v;)) ((1441). the relevant term is
Cop(r) =
log ri-1 +r; Pi +ri
Ti-1 + log
T
log
T++1 + log
T/
+
Pi+1 +
Ti+1 + tri
Then.
@Lop(r)
Or:
T4 T141
Ti +r-1 Ti
Setting @Cor(r) 0, we have Dr.
0; 0
(2) uf u(x;)) < u(x441); the relevant term is
Cor(r)= -
1 log T:-1
+ log
r3 )
- log
Ti+1 + +....
ri-1 +
ri-1 + rs.
Tit1 +
Then,
@Lor(r)
dr:
Ti + rit1
2r; Tv +141
Setting @Cop(r) Dr.
0, we have
1 0;-1
a, =
1 ---14-1
(3) If [f < u(xi) < u(xi+1), the relevant term is
Lop(r) log
log
Ti+1
ri-1 + Pi
ri+1 + Ti
Then,
@Lop(r)
+
Or:
T:41
+141
T1+1
Setting @Lor(r)
0, we have
03 0;-1-
(4) [f u(x;) = u(x++1), the relevant term is
Lor(r) -loog
1
log
r;
log
Ti+1
+...
P;-1 + +r
2
ri+1 + rs
ri+1 + rj
17

---

Published as a conference paper at ICLR 2024
Then,
@Cop(r)
3 2ri
+
Or;
+ ri+1
1i41
ri Ti+1
Setting (Cor(r)
0, have
0i = 0/-1 -1
0;-1 +3
From (I), (2), (3), (4), assuming u(x;-1) < u(zi) = =... u(x;) < u(zz+1), then
1+ 01-1-1
1+ 1 = 2. 1 1 0i
1 = 2. 0;-143
Qj = 2. 1 a-aj-
= a:-1.
1- 0i
ac-1+3
Therefore, we can define a := a; if u(x;) < u(z;-1), and == a if u(x;) = u(x:41), and 0+3 2 1,0 s 3 s 1. Then, logr is piecewise linear with two different slope, log and
log By the definition of - and Øn+1, we have ro ar-1 ar .' Tn+1/a b/a 1/a.
According the previous definition, We have d 2 7 i... a is the solution the following equation
m-2
a
3
Therefore, a, by the definition of Ccy We finally need to ensure R()) preserves the order, i.e. if u(x,) u(x;), we have R(x,) R(z;). We can bound:
R(x;) a2
ar - n-
(6)
R(x;)
a2 +3 3
and the equality holds iff l2 C (i,i ..... jj- 13 and j-i- 1/2 1. We have by the definition
of Oryy and monotonic increasing of (a) and [1(7 m1 1- w.r.t. a and 7,
Ck: + 3
1 a
m+1
f1(7 2 70:
which satisfies the assumption, hence R(x,)) R(x.). The order-preserving loss can be explicitly written as:
Cop(r)
n log (1+ B,,)3
By
(m + 2)log 1 agy
Qy
n log
(a, 1)(ar, +3)
(m + 22 log 1 og
a₂
4(a, + 112
and taking the derivative w.r.t. ay, we have
dCor(r)
m + 2 da (a () + a, (a, + 2)
4n
0.
Therefore, minimizing the order pidee-preseriigg loss corresponds to making a, - +00, and therefore B, and 7 + 400.
uf)( < u(x1) or u(xxx=1) < u(z,), the auxiliary states I-1 or Fn+1 are not necessary be added. In this case, have u(xx) 1/1 u(zu) without auxiliary states, following a similar argument in Proposition / However, if there are multiple states of minimal or maximal objective value, need to introduce u(x-1) -o u(xat1) +00, so that ro or Ta appears in
Remark.
18

---

Published as a conference paper at ICLR 2024
two terms in (rr(r), unifying analysis and avoiding boundary condition difference. For example, if (fu(zo)= = u(z1) without x 2-1, we have
@Lop(r)
+
Oro
To +
2ro
+
(r)
1
Lop [Zr
dr vo +11 2r1
To
+ Ti
òri
and @Lov(r) and @Cop(r) Oru
cannot be zero at the same time.
dr:
Proposition 6. In the sequence prepend/append MDP in Definition 7 we consider fixed dataset MDP {o,, Fiji= with u(ro) u(x1) <.... < u(xx) u(z")). Denote as the important substring, defined as the longest substring shared by Tn, with length k, and (x(x;) as the set of k-length substrings of x. Following Proposition 5
let the order-preserving reward R [1/%, 11, and the ratio 0r. We fix PB to be uniform and match the flow F(-) with R(-) on terminal states. Then, when on > 4, we have (FF(s*) (F)(s), (8 where the expectation is taken over the random positions of s* in Nn: Z'y.
Proof of Proposition 6 By Proposition 5 the reward R(z,) = R(z))a, R(z²,) B.
i S n. We claim that uniform backward policy induces uniform trajectory distribution over trajectories connecting
80. On constructing we have - choices of prepending or appending.
Therefore, there are 2 trajectories ending at æ, and each trajectory has flow
R(w))
Let of length 1 is preceded by characters in x of length 1, in total there are
(1-)) 2 trajectories
passing through s that end at æ. The average number of trajectories over uniform distribution on 0 a 1 l: is uniformly random positions of in x is I-++1 for any 8,
2--1 I-IFI Thus, the expected flow passing through and ending at just x, over
R(z)
+ and for (n(sn)) RF(s) (t. (+x,)+> R(O 1-k+1
We have EF(s*) 2
As long
1-k+1
4 Cry +3 (1-0y
Q2y -1
1 + n-1 < ,, = (1-
Cryy
+3 Q2y
i.e On > is sufficient to make the inequality hold. we have EF(s*) > (s),), E (x,) U U (xx,))(s Therefore, then the order-preserving reward can help correctly assign credits to the high-reward intermediate state s'. Note our results does not contradict with those in Shen et al. 2023 since we are considering F(s), E sx(x)))s", instead of ((sx(z))(s))
E SINGLE-OBJECTIV EXPERIMENTS
In this section, our implementation is based on torchgín (Lahlou et al. (023), Shen et al. (202))ss implementation.
E.1 PRELIMINARIES
We introduce some important tricks we used to improve the performance in the experiments.
Backward KL Regularization Fixed uniform backward distribution Pa has been shown to avoid bias induced by joint training of Pg and Pp. but also suffers from the slow convergence (Malkin et al. 2022]. In this paper, we propose the regularize the backward distribution by its KL divergence w.r.t. uniform distribution. For a trajectory trajectory loss LKL as
(so
+ Sn z), define the KL regularized
1 n Σ (s)), where (KL(S:) (L
(KI(T)
where U-(+ is uniform distribution on valid backward actions. Such KL regularizer can be plugged into any training objective that parametrizes the backward probability PB. which includes DB and (sub)TB objective. In Appendix E.2.I we show that KL regularizer provides the advantages of both fixed and trainable Pa.
19

---

Published as a conference paper at ICLR 2024
Backward Trajectories Augmentation We adopt the prioritized replay training (PRT) (Shen et al. 2023). that focuses on high reward data. We form a replay batch from X, all terminal states seen so tar, so that an percentile of the batch is sampled from the top a2 percentile of the objective function, and the rest the batch is sampled from the bottom 1 02 percentile. We sample augmented trajectories from the replay batch using PB- In practice, select 01 50, 02 = 10.
R(z) as Proxy When evaluating the objective function u(z) costly, we propose to use R(x) as proxy. If we want to sample k terminal states with maximal rewards, we can first sample K > k terminal states, and pick states with top-k R(z). Then, we need only evaluate u(z) on k instead of K terminal states, We define the ratio boosting to be Tboost.
/Kk For GFlowNet objective parametrize F(s), we can directly let R(z) = F E X. For TB objective, we need to use Equation approximate R(x). Since the cost of evaluating R(x) is also non-negligible, we only adopt this strategy when obtaining u(z) directly is significantly more difficult. For example, in the neural architecture search environment (in Section 4.3 evaluating ur(x)) requires training network
to step T to get the test accuracy.
Training Procedure We adopt the hybrid of online and offline training of the GFlowNet. The full pseudo algorithm is summarized Algorithm
Algorithm 1 Order-Preserving GFlowNet
Inputs:
Nuit: number of forward sampled terminal states at initialization.
Nround: number of rounds for forward sampling.
Nnew: number of forward sampled terminal states in every round.
Not: number of backward augmented terminal states.
-oft-per: number of backward augmented trajectories per terminal state.
Initialize:
To: Random initialized trajectories of size Ninit-
0o: GFlowNet with parameters = 0g:
T(::T): Order preserving loss defined in Equation
for ->> Nround do
Update parameters 02 0:-1 trajectories set Ti-1-
Sample Nnew terminal trajectories T.' with the forward action sampler parameterized by 0z Update trajectory sets T, Ti-1 U T.'
Sample Not terminal states from Te Augment into trajectories T," by the uniform backward sampler, with Nott per trajectories per terminal state. Update parameters 0g t- 0z on trajectories set T.".
end for
E.2 HYPERGRID
Objective Function The objective function at the state x = (x',...,,>") D)T is given by
D
u(x)= Ro 0.5
II I [1
- 0.5| e (0.25, 0.5]] + II II [10a 0.5| E (0.3,0.4)]
(7)
d=1
d=1
where 1 is indicator function and Ro is constant controlling the difficulty of exploration. This objective function has peaks of height 2.5 + Ro near the four corners the HyperGrid, surrounded by plateaux of height 0.5 Rg. These plateau are situated on wide valley of height Ro-
Network Struture We use shared encoder to parameterize the state flow estimator F(s) and transition probability estimator (r(+s)) (=)-ss), and one tensor to parametrize normalizing constant Z. The encoder is an MLP with hidden layers and 256 hidden dimensions. We use ReLU as the activation function. We Adam optimizer with learning rate of 0.1 for Zo's ' parameters and learning rate of 0.001 for the neural network's 's parameters.
E.2.1 BACKWARD KL REGULARIZATION
In this subsection, we validate the effectiveness of backward KL regularization in Appendix E.I We set the reward R(x) u(x:) defined in Equation
20

---

Published as a conference paper at ICLR 2024
Following the definition of HyperGrid in Section 4.1 consider two grids with the number of terminal states: a 2-dimensional grid with H 64 and 4-dimensional grid with H = We set Ro 0.1, 0.01, 0.001, where Ro is defined in Equation (7). We remark that larger H expects longer trajectories and smaller Ro poses greater exploration challenges since models are less likely pass the low-reward valley. We analyze training behaviors of fixed and regularized Pg. During the training, we update the model on actively sampled 1000 terminal states in each round, for 1000 rounds in total.
We expect that KL regularized P3 converges faster than fixed P3 in hard instances, such as long trajectory lengths and small Rg, and avoid bias introduced by simply trainable PB without regulariza- tion. To validate this, we graph the progression of the l error between the target reward distribution and the observed distribution of the most recent 10 visited states, and the empirical KL divergence of learned Pn during training in Figure E. Figure E.2 We observe that fixed Pn makes it very slow to converge in & distance hard instances (i.e. Ro S 0.01 and H = 64), but a regularized converge faster, and keep near-uniform backward transition
PB with proper ÄKL. e.g. 0.1, probability during training. We also observe that in short trajectories, fixed or trainable PB does not have convergence speed difference.
To illustrate the bias of trainable Ps without regularization, we visualize the learned sampler after 10 states by plotting the probability of each state perform the terminal action in Figure E.3 The initial state is (0, 0), and the reward symmetric with respect to the diagonal from (0, 0) to (63, 63). Therefore, the learned probability of terminal action should be symmetric with respect the diagonal. We observe that standard TB training is biased towards the upper diagonal part, while fixed TB and regularized TB behave more symmetrically.
Ro 0.1
Ro 0.01
R. = 0.001
empinical 4 error
10-1
0.6
KL divergence
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
state visited
1e6
state visited
1e6
state visited
le6
Fixed Ps
Ave
Figure E.1: HyperGrid with D 2, H 64, different Rg
0.001. Pp is trainable with KL regularization weight λKI. 0, 0.01, 0,1, 10, or Pn is fixed.
F.2.2 ABLATION STUDY OF Ro
In this subsection, we provide the ablation study of Ro {0, 0.0001, 0.001, 0.01, 0.1, for Sec-
tion 4.1 wW set (D, H) = (2, 64), (3, 32) and (4, 16), and compare TB and order- pnder-preserving TB (OP-TB). For D, H) (2, 64), we plot the observed distribution
4000 most recently visited states in Figure E.4 We additionally plot the following three ratios: 1) #(distinctly visited states)/#(al the states); #(distinctly visited maximal states) #(all the maximal states); In the most recently 4000 visited states, # #ddistinctly maximal states) states)/4000 /4000 in Figure/E.5 We train the network for 500 steps, 200 trajectories per step, 20 steps per checkpoint.
21

---

Published as a conference paper at ICLR 2024
Rg 0.1
Rg = 0.01
Rg 0.001
empirical E, error
10-4
KL divergence
0.3
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
state visited
state visited
state visited
AxL 0
Fixed Ps
AuL 0.01
AxL 0.1
AKL
ÁKL 10
Figure E.2: HyperGrid with D 4, H 8, different Rs = 0.1,
PB is trainable with
KL regularization weight λKL. 0, 0,0.01,0.11 1, 10, Pa is fixed.
Trainable Ps
KL Regularized Ps
Fixed Ps
0.7
60
0.6
50
50
0.4
0.3
0.2
0.1
0
20
40
60
0
20
40
60
20
40
60
Figure E.3: HyperGrid with D = 2, 11 = 64, Ro = 0.01. We set ÅKL. for KL regularization. We plot the probability of each state to perform the terminal action.
22

---

Published as a conference paper at ICLR 2024
We remark that Ro plays similar role the reward exponent B to flatten sparsify the rewards. Large Rg facilitates exploration but hinders exploitation since a perfectly trained GFlowNet will also sample non-maximal objective candidates with high probability; whereas low Rg facilitates exploitation but hinders exploration since low reward valleys hinder the discovery of maximal objective candidates far from the initial state. A good sampling algorithm should have small ratio 1), and large ratios 2). 3), which means it can sample diverse maximal states (large ratio 2), exploration), and sample only maximal states (large ratio 3), exploitation), using the fewest distinct visited states (small ratio 1), efficiency). We observe from Figure E.5 that OP-TB outperforms TB in almost every Ru in terms of three ratios.
Vitted
Figure E.4: Full experiments on Rg 1, 0,1, 0.01, 0.001, 0.0001, 0. We plot observed distribution
on 4000 most recently visited states, when we have sampled 4000, 8000, 12000, 16000, 20000 states.
23

---

Published as a conference paper at ICLR 2024
1.0
0.3
0.5
0.6-
0.6-
0.4
04-
9.2
00020
40200
20200
200300
0.3
00,40-
0.6
5.3
0.4
0.2
02
531
0.025
0.5
0.020
0.4
0.000
02-
0.035
0.000
42000
02030
40200
200003
OPTS
10, R. 10,
TB. Ro. 0.51 ... 10 0.00
TA.4, 6.001
Ro
Figure E.5: Full experiments results on the HyperGrid with (D, 64). (3, 32), (4, 16) and Ro O, 0.0001, 0.001, 0.01, 0.1, We plot the ratio 1) #(distinctly visited states)##(a the states); 2) #(distinctly visited maximal states)/ #(all the maximal states); 3) In the most recently 4000 visited states, #(distinctly maximal states)/4000. by OP-TB (solid lines) and TB (dashed lines) method.
E.2.3 R(-) DISTRIBUTION
We consider the cosine objective function. The objective function at the state (x),,.. xD is given by
u(r) Ro
II d=1
()(0)) ()xx")),
where á is the standard normal p.d.f. Such choice offers more complex landscapes than those in Section 4.1, We set D = 2, H = 32, Rg = 1 in the HyperGrid. Since DB directly parametrizes F(s), we adopt the OP-DB method with Aop 0.1. Assume there distinct values of W1 < u2 < <... < Un, among all terminal states, we calculate the mean of the learned R(--) on all the states with objective value ui, i.e.,
(0) : Avg( = F(x;0), where u(x) = u.) 1sisn.
We sample 20 states per round and checkpoint the GFlowNet every 50 rounds. In Figure E.6 plot the log normalized R; and R, by linearly scaling their sum over to We observe that log R is approximately linear, confirming Propositions [ [ and the slope is increasing as the training goes.
We note that the active training process we used in practice, will put more mass on states of near- maximal objective values than Propositions and I predictions based on full batch training. In other words, we observe that the slope is larger in the near-maximal values, explained in the following. Since we are actively training the GFlowNets, later training stages. Therefore, the order- preserving method in active training pays more attention to high-value states as the training goes, resulting in a larger slope, compared with log linear's 's prediction in the near-maximal objective value states.
more likely to collect high-value states in the
in Propositions and T
24

---

Published as a conference paper at ICLR 2024
Log Normalized Reward
-4-
R(X)
R(x), Round
-8-
Rix). Round 50
R(x). Round 100
Rix), Round 200
Rix), Round 300
0
40
80
by Rix)
Figure E.6: Log normalized R(-) (red) and R(-). We checkpoint the GFlowNet sampler training round (randomly initialized), 50, 100, 200, 400.
E.3 MOLECULAR DESIGN
E.3.1 ENVIRONMENTS
In the following, we describe each environment and its training setup individually. We adopt the sequence prepend/append MDPs proposed by (Shen et al. (2023) simplify the graph sampling problems. For instance, in this section, we employ the sEH reward with 18 fragments and approxi- mately 10? candidates. is worth noting that the original sEH environment (Bengio al. 2021a) is fragment- caannent-graph--ee graph-based MDP with around 100 fragments and more than T0 candidates. We will utilize the sEH reward with this fragment-gran saaett-rraahha-sasss MDP for multi-objective optimization problems in Section 5.4
Bag ((X = 96, 889, 010, 407) A multiset of size 13, with 7 distinct elements. The base objective value is 0.01. A bag has a "substructure" if it contains 7 or more repeats of any letter: then it has objective value candidates as the X' with objective value 30, and the total number is 3160. The policy encoder is an MLP with 2 hidden layers and 16 hidden dimensions. We use an exploration epsilon & = 0.10. The number of active training rounds is 2000.
with 75% chance and 30 otherwise. We define the maximal objective value
SIX6 (TFBind8) ([X = 65, 536) A string of length 8 nucleotides. The objective function is wet-lab measured DNA binding activity human transcription factor, SIX6 REF R1, from Barrera et al. (2016); Fedus et al. (2020). We define the optimal candidates as the top 0.5% of X as ranked by the objective function, and the number is 328. The policy encoder is an MLP with 2 hidden layers 0.01, and a reward exponent 3 = 3.
and 128 hidden dimensions. We use an exploration epsilon
The number of active training rounds is 2000.
PHO4 (TFBind10) (|x') = 1, 048, 576) A string of length 10 of nucleotides. The objective function wet-lab measured DNA binding activity yeast transcription factor PHO4, from Barrera al. (2016): Fedus et al. (2020). We define the optimal candidates as the top 0.5% of X' as ranked by the objective function, and the total number is 5764. The policy encoder is an MLP with 3 hidden layers and 512 hidden dimensions. We use an exploration epsilon e 0.01, and reward exponent 3=33 3, and scale the reward a maximum of 10. The number of active training rounds 20000.
QM9 (Ix') 161, 051) A small molecule graph (Blum & Reymond 2009 Montavon et al. 2013)
The objective function is from a proxy deep neural network predicting the HOMO-LUMO gap from density functional theory. We build by 12 building blocks with 2 stems, and generate with 5 blocks per molecule. We define the optimal candidates as the top 0.5% of as ranked by the objective function, and the total number is 805. The policy encoder is MLP with 2 hidden layers and 1024 hidden dimensions. We use an exploration epsilon E = 0.10, and reward exponent & = and scale the reward maximum of 100. The number of active training rounds is 2000.
25

---

Published as a conference paper at ICLR 2024
SEH (1.2) 34, 012, 224) A small molecule graph. The objective function is from a proxy model trained to predict binding affinity to soluble epoxide hydrolase. Specifically, we use a gradient- boosted regressor on the graph neural network predictions from the proxy model provided by [Bnngio et al. (2021 la) and Kim et al. (2023b), to memorize the model. Their proxy model achieved an MSE of 0.375, and Pearson correlation of 0.905. We build by 18 building blocks with 2 stems and generate 6 blocks per molecule. We define the optimal candidates as the top 0.1% of X' as ranked by the objective function, and the total number 34013. The policy encoder is an MLP with 2 hidden layers and 1024 hidden dimensions. We use an exploration epsilon EF = 0.05, and reward exponent 6, and scale the reward to a maximum of 10. The number of active training rounds is 2000.
E.3.2 EXPERIMENTAL DETAILS
Our implementation is adapted from official implementation from [henn et al. (2023): Kim al. We follow their environment and training settings in the codes.
(0033
Network structure. When training the GFNs and OP- GFNs, instead of parametrizing forward and backward transition by policy parametrization, we first parametrize the edge flow F(s - s'; f), and define the forward transition probability as F(s - /)/ -s' F(s + 0); We clip the gradient norm to a maximum of 10.0, and the policy logit to maximum of absolute value of 50.0. We initialize log Zg to be 5.0, which is smaller than the ground-truth Z in every environment. We use Adam optimizer with learning rate of 0.01 for Za's 's parameters and a learning rate of 0.0001 for the neural network's parameters. We do not share the policy parameters between forward and backward transition functions. When training the RL-based methods, we set the actor and critic networks' initial learning rate to be 0.0001. We set the A2C's entropy regularization parameter to be 0.01, and SQL's temperature parameter to be 0.01.
Training and Evaluation In each round, we first update the model using the on-policy batch of size 32 and then perform additional update on one off-policy batch of size 32, which was generated by the backward trajectories augmentation PRT from the replay buffer. The number of active training rounds varies for different tasks, see Appendix E.3.1 for details. To monitor the training process, for every 10 active rounds, we sample 128 momtoring candidates from the current training policy without random noise injection. At each monitoring point, we record the average of the value of the top 100 candidates ranked by the objective function, and the number of optimal candidates being found, among all the generated candidates.
E.4 NEURAL ARCHITECTURE SEARCH
E.4.1 NAS ENVIRONMENT
In this subsection, we study the neural architecture search environment NATS-Bench (Dong et al.) 202T), which includes three datasets: CIFARIO, CIFAR-100, and ImageNet- 120. We choose the topology search space in NATS- i.e. the densely connected DAG of 4 nodes and the operation set of 5 representative candidates. The representative operations O = {ox J are 1) zero, 2) skip connection, 3) 1-by-1 convolution, 4) 3-by-3 convolution, and 5) 3-by-3 average pooling layer. Each architecture can be uniquely determined by a sequence x = {d', E O Ohsisjsı of length 6, where dyy indicates the operation from node to node j. Therefore, the neural architecture search can be regarded as an order-agnostic sequence generation problem, where the objective function of each sequence is determined by the accuracy of the corresponding architecture.
RugeRegees MDP Design We use the GFlowNet (S, X)) tackle the problem. Each state is sequence of operations of length 6, with possible empty positions, i.e. S = {s|s = foh, E OU 0 The initial state is the empty sequence, and terminal states are full sequences, i.e. X {z|x (oly O)sisjs4)). Each forward action fills the empty position in the non-terminal state with some Ok, and each backward action empties some non-empty position.
Objective Function Design For X, the objective function ur(x)) is the test accuracy of x's corresponding architecture with the weights at the T-th epoch during its standard training pipeline. To measure the cost to compute the objective function (z), we introduce the simulated train and test (T&T) time, which is defined by the time to train the architecture to the epoch T and then evaluate its test accuracy.
provides APIs on (x:) and its T&T time for T < 200. Following the
26

---

Published as a conference paper at ICLR 2024
experimental setups in Dong al. (2021). when training the GFlowNet, we use the test accuracy epoch 12 (#12) as the objective function; when evaluating the candidates, we use the test accuracy at epoch 200 U200 as the objective function. We remark that U12 is proxy for u200 with lower T&T time, and order-preserving methods only preserve the order of u12, ignoring the possible unnecessary information: U12' exact value.
E.4.2 EXPERIMENTAL DETAILS
Network Structure We the default Network structure in the package torchgfn (Lahlou et al. 2023). We use a shared encoder to parameterize the state flow estimator F(s) and transition probability estimator (=(+)s), (- and one tensor to parameterize normalizing constant Z. The encoder is an MLP with 2 hidden layers and 256 hidden dimensions. We use ReLU as the activation function. We use Adam optimizer with a learning rate of 0.1 for Zy" parameters and a learning rate of 0.001 for the neural network's 's parameters.
Training Pipeline We focus on training the GFlowNet multi-trial sampling procedure. We find is beneficial to use randomly generated initial datasets, and set the size to 64. In each active training round, we generate 10 new trajectories using the current training policy and update the Gel on all the collected trajectories. We optionally use backward trajectory augmentation to sample 20 terminal states from the replay buffer and generate 20 trajectories per terminal using the current backward policy to update the GFlowNet.
Multi- Trial Methods We use the official implementation from NATS-BencH where the hyperpa- rameters are specified below.
REA. We follow Algorithm 1 in Real al. (2019). We set the number of cycles C 200, the number of individuals to keep in the population P = 10, and the number of individuals that should participate in each tournament S
BOHB. We follow Algorithm 2 in Falkner et al. 2018 8). We set the fraction of random configurations = 0, the bandwidth factor b 3, and the number of candidates for the acquisition function N₂ 4.
REINFORCE (Williams) 1992). The code is directly adapted from the standard RL example [8) We use the Adam optimizer for the policy parametrization, and the learning rate to be 0.01. The expected reward is calculated by the exponential moving average with a momentum of 0.9.
E.4.3 BOOSTING THE SAMPLER
Once we get trained GelowNet sampler, we can also use the learned order-preserving reward as a proxy to further boost it, see Appendix EI We adopt the following experimental settings. The (unboosted) GelowNet samplers are obtained by training after the first round of the previous multi-triae training. We mcasure the sampler's performance by sequentially generating candidates and recording the highest validation accuracy obtained so far. We also plot each algorithm's sample efficiency gain Fgain- which indicates that the baseline (unboosted) takes rgain times of number of candidates reach a target accuracy compared that algorithm. We plot the average over 100 seeds in Figure E.7 observing that setting Tboost. RJ 8 reaches up to 300% gain.
fixed dataset, i.c. the checkpoint sampler
E.4.4 ABLATION STUDY
In this subsection, we provide the ablation study of hyperparameters on GFlowNet training. To keep the comparison fair, we use the TB as the GFlowNet objective, and disable the backward trajectories augmentation in all the following experiments. We plot the test accuracy at the 12th epoch and the 200th epoch w.r.t. the estimated T&T time, following experimental settings in Section 4.3
OP-GFN v.s. GFN-3. We use R(z)= := u(z)8 in TB-3 training. We set the reward exponent 3 = 4,8, 16, 32, 64, 128. We disable the KI. regularization in all the experiments. The results are plotted in Figure E.8 We observe that 3 = 8 ~ 32 are the best choices. Setting to be too large or small will both negatively impact the performance, by hindering exploration and exploitation
27

---

Published as a conference paper at ICLR 2024
CIFARIO
CIFAR100
ImageNet16-120
94.25
Teee 200
93.75
45
93.25
92.50
40
semples
500%
100%
250%
300%
1509
130%
2009
100%
100%
0%
09
93.0
93.2
93.4
93.6
93.8
0.
94.2
69
70
44 "arget Accuracy
Target Accurncy
Targer Accuracy
Pamm
Twoe
Figure E.7: Boosting GFlowNet sampler. To boost the sampler, we select the best candidates ranked by R(-)) from Tboost = 1, 2, 8, 32, 128 candidates states, where Thoost 1 denotes the unboosted sampler. We plot the highest test accuracy observed far in the 100 candidates, and the performance gain w.r.t. each target accuracy.
respectively. However, the best choices of B are still below the performance of the OP-TB method, especially in the early training stage.
ImageNet16-120
62
38
60
56
87-
34
28
28
50
10000
20000
30000
40000
50000
20000
40000
60209
80000
100000
50000
100000
150003
200000
20.0
65
or Sauch 200
72
90.0
70
69.0-
68
not
62.0
91.5
91.5
64
20203
30000
40200
52000
100000
30000
150002 Estimaled time
100020
Extmmttd wall kime
Extimated waliclese
18
18-6-22
1B,564
TB-9-123
Figure E.8: Ablation study of TB-3 with RANDOM, TB, and OP-TB.
64, 128.
We compare them against
28

---

Published as a conference paper at ICLR 2024
Ablation Study of ÄKL We set the KL regularization hyperparameter λKL 0.001. 0.01. 0.1, 1. We use the OP-TB as the OP-GFN criterion. The results are plotted in Figure E. We observe that positive Ägt. contribute positively towards the sampling efficiency, but the performance insensitive to the exact value of ÅKL. We empirically set λKI. = 0.1 Aop in default for KI. regularized OP-GFN methods (recall that set the Aor = 1 for OP-TB).
CIFAR1D
CIFAR100
ImageNett6-120
00
38
34
88
32
54-
86
52
26
10000
20000
30020
40300
53000
20000
40000
60203
100000
100000
30000
100020
150003
200020
15.0
84.5
Not 1000000000 Day ALE200 200
72-
53.5
70
42
62.5
68-
40
66
91.5
10000
20000
30000
40000
50000
20000
40000
60200
80000
200000
50000
100000
150000
200020
Pstmmtte wal--denc time
Extimated walle time
Est mated ual-<lock time
RANDOM
TB
OFTB
Av. 9.001
Au. 0.01
01
Ag
Figure E.9; Ablation study of KL regularization with Aςι. = 0.001, 0.01, 0.1, 1. We compare them against RANDOM, TB, and OP-TB without KL regularization.
Ablation Study of Nintt We set the size of the randomly generated dataset at initialization, Nint 0, 20, 40, 60, 80, 100. We use the OP-TB as the OP-GFN criterion and disable the KL regularization in all the experiments. The results plotted in Figure We observe that randomly generated initial dataset contributes positively to the final performance since it avoids bias from insufficient candidates in the early training stages.
E.4.5 OP-NON-TB METHODS
In this subsection, provide the experimental results the NAS environment by order-preserving loss on GFlowNet objectives other than TB, defined in Appendix C Since FM does not directly parametrize PB. choose not use the KL regularization for FM. We set ÅOς 0.1, and ÁKI. = 0.01 if the backward KL regularization is used. Other experimental settings are the same those in Section 4.3 We include RANDOM baseline and OP-TB as comparisons. The results are plotted in Figure E. 1..1 Figure E. Figure E.13 We observe that OP-non-TE methods can achieve similar performance gain with OP-TB, which validates the effectiveness and generality of order-preserving methods.
Ablation Study Aor. We set the hyperparameter to weight the onder-preserving loss Aor 0.01, 0.1, 1, 10. We adopt OP-DB the OP-GFN criterion and disable both the backward KL. regularization and trajectory augmentation. We include the RANDOM baseline as comparison. The results are plotted in Figure E.14 We observe that different XOP has an influence on the performance, and λOp = 0.1 is the best choice. Therefore, similar to 3, optimal AOp is dependent on the MDP and objective function function u(z). However, compared with previous ablation study on 3 in Figure E.8) OP-GFN methods are less sensitive to the different choices of AOp compared to the impact of different choices of (3 GFN-3 methods.
29

---

Published as a conference paper at ICLR 2024
ImageNel16-120
50
62
56
54-
50
30000
40000
52000
100000
30000
25.0
46
94.5
Tet Cocurazy SrEpoch 200
$4.0
70
22.0
68-
62.
56
91.0
36
Figure 1.10: Ablation study of initial datasets Nunt 10, 20, 60, 80, 100. We compare them against RANDOM.
CIFAR100
ImageNel16-120
90
02
56
30
54
28
50
20303
30000
40000
53000
20000
100000
50000
150000
94.5
1st DOCUIEE ot Epoch 220
83.3
70
44
53.0-
66
92.0
40
Figure E.11: Detailed balance. We include DB, OP-DB, OP-DB- KL, OP-DB- KU-G and compare them against RANDOM and OP-TB.
30

---

Published as a conference paper at ICLR 2024
ImageNet1
62
69
60-
34
30
67-
34
50
10000
20000
30000
40000
50000
20000
40000
60009
80000
100000
50000
100000
150000
200000
$4.5
74-
47
153 acCaro at Spodh
$0.5
70
$2.5
92.0
20303
30000
40000
53000
100000
50000
100000
150003
Extimeted mel--decc time
Extimated waliclloce
Ealimatedd time
Figure E.12: Flow matching. We include FM, OP-FM, and compare them against RANDOM and OP-TB.
CIFAR10
CIFAR100
ImageNelle-120
58
BB-
56
86-
52
85
10000
20503
30000
40500
50
<050n
60203
150500
Tu2t aouuacy 24 Eooch 220
96.0
72
45.
$3.5
70
68
52.5
66
92.0
30000
30000
40000
50000
40000
60000
100000
50000
150000
200000
Ertiatted
Estimated wall-lock
RANDOM
wihTB
SP-WihTB
3P-uSTB-4
OP-T8
Figure E.13: SubTrajectory balance We include subTB, OP-subTB, OP-subTB-KL, OP-subTB. KL-AUG, and compare them against RANDOM and OP-TB.
31

---

Published as a conference paper at ICLR 2024
ImageNet]
80-
02
60-
32
56
07-
34
52-
50
24
10000
20000
30000
40000
50000
20000
40000
80000
100000
50000
100000
150000
200000
95.0
Ceaccuuoc at Spees 300
72-
$4.0
63.5
70
99.0
68
$2.0
66
38
91.3
20000
30000
50000
150003 Extiattdd cookk time
wall
SANDON
App-
App
Figure E. 14: Ablation study of ÅO with Aop = 0.01, We use the OP-DB for different AOP.
F MULTTI-BBEECTIVE EXPERIMENTS
In this section, our implementation is based on Jain et al. (2023)s implementation and https 7/githubs cwt.://ussceeeuuoooeeoo httbbb.com/recurs..oopaaaaamm wtwpa//glooweeee.o Townet. .it
F.1 EVALUATION METRICS
We assume the reference set p: (D, و is the set of the true Pareto front points, and S {ii) ;= the set of all the generated candidates, p' Pareto(S)) S) is non-dominated points S. When the true Pareto front is unknown, we use a discretization of the extreme faces of the objective space hypercube as P. We first introduce the GD class metries:
Generational Distance (GD, Van Veldhuizen (1999)) GD takes the average of the distance the closest reference point for each generated sample: GD(S, P) == Avg milppe ||p s s|12, which measures how close all the generated candidates are to the true Pareto front. The forward calculation of distance ensures that the indicator does not miss any part of the generated candidates.
Inverted Generational Distance (IGD, Coello Coello & Reyes Sierra (0004) IGD
takes the average of the distance to the closest generated sample for each Pareto point: IGD(S, P) := A min.es ||p s||2, which measures how closely the whole Pareto front is approximated by some of generated candidates. The inverted calculation of distance ensures that the indicator does not miss any part on the Pareto front.
Generational Distance Plus (GD+, Ishibuchi et al. (2015)) GD takes the average of the modified distance to the closest reference point for each generated sample: GD+(S, P)
Avg.es minper max(p s.0) 1|2-
Inverted Generational Distance Plus (IGD+, Ishibuchi et al. (2055) IGD takes the average of the modified distance to the closest generated sample for each Pareto point: IGD+(S, P) Avgpe p min max(p 8, |]
Averaged Hausdorff distance (dz, Schutze et al. 2012 The averaged Hausdorff distance is defined by (S. P) = max IGD P)}. which combines the advantages of GD and IGD. Otherwise, if only one metric is used, candidates with small GD might
32

---

Published as a conference paper at ICLR 2024
concentrate on only part of the true Pareto front, and candidates with small IGD might have high density away from the Pareto front.
Furthermore, as suggested by Ishibuchi et al. (2013), GD and IGD indicators not Pareto compliant, which means it might contradict Pareto optimality in some cases. However, IGD+ is proved to be weakly Pareto compliant. In other words, if S 3 S" holds between two non-dominated sets in the objective space, IGD+(S) < IGD+(S") always holds, where we call set S non non-dominated iff no vector in S is dominated by any other vector in S. However, GD+ not weakly Pareto compliant, therefore, + is not either.
Hypervolume Indicator (HV, Fonsecaa al. (206)), which a standard metric reported in MOO. measuring the volume in the objective space with respect to a reference point spanned by set of non-dominated solutions in Pareto front approximation. However, HV does not ensure uniformity in the non-convex Pareto front, the sampler might have high HV but miss the concave part in the Pareto front.
Creeto-Cuusters Entropy (PC-ent, Roy et al. 2023)). PC-ent measures the sampler's uniformity on the estimated Pareto front. We use the reference points P and divide P' in the disjoint subset Pj :=
{p : Vj' Ip P, ll2 I p - Pyllla) [la). and the entropy of the histogram is PC-ent(P .
P) ==
IPj
-, I log TPT which attains maximum value when all the generated candidates are uniformly distributed around the true Pareto front.
Rg Indicator (Hansen & Jaszkiewicz/ 1994): R2 provides a monotonic metric comparing two Pareto front approximations using set of uniform reference vectors and utopian point =² representing the ideal solution of the MOO. Specifically, we define a set of uniform reference vectors À E A that cover the space of the MOO, and set of Pareto front approximations E S. we calculate: R2,S, A, =*)= == TAT ΣXEA minges maxicl,... (...,,(A:ll?? - 8: - In our experiments where all objective functions are normalized to [0, 1], we set 2" (1, (1, 1). R2 indicators capture both the convergence and uniformity at the same time.
F.2 HYPERGRID
We present the implementation of objectives bran n, currin, shubert, bea e from https // //www. s fu. ca/ sssuujajno opt imi zat on html in the following, where some constants, such as 308. 13, 13.77, are used for normalization to 10.
prannin(E1,
(19 +
l2 10) /308.13,
4 (15x1 5)² + 5
5.1
10
where t = 15x--
(15z1 - 5) - 6, t2 = (10
cos( [1x1 5):
8
2300x + 1900x? + 2092x; + 60
currin(21, (1 e mm -
13.77(100z3 + 500x? + 4r + +0))
s cos (i + 1)21 +i)
Σ cos
1).2 + i) /397 + 186.8/397; (i cos((i
22) = ((1.5 - x 22 (2.25- I1 + x x2)² + (2.625 x + x x113))3)338...
Network Structure For both the OP-GFNs and PC-GFNs, we fix the backward transition probability Ps to be uniform and parametrize the forward transition probability Pr by the MLP with layers, 64 hidden dimensions and Lea as the activation function.
Training Pipeline During training, models are trained by the Adam optimizer (Kingma & Ba 2014 with learning rate of 0.01 and a batch size of 128 for 1000 steps. When training the PC-GFNs, we sample u ~ Dirichlet and 8 T(16, 1). In evaluation, we sample ur ~ Dirichlet(1.55 and fix ( = 16. we generate 1280 candidates by the learned GelowNet sampler as S, and report the metrics in Table F. To visualize the sampler, we plot the all the objective function vectors, and the true Pareto front, well as the first 128 generated objective function vectors, and their estimated Pareto front, in the objective space [o, 1]2 and [0. 1] in Figure F.1 We observe that our sampler achieve better approximation (i.e. almost zero IGD+ and smaller d ) and uniformity (i.e. higher PC-ent) of the Pareto front. We also plot the learned reward distributions of OP-GFNs and compare them with the indicator functions of the true Pareto front solutions in Figure 5.1 where we observe that
33

---

Published as a conference paper at ICLR 2024
OP-GFN can learn highly sparse reward function that concentrates the Pareto solutions. We also plot the learned reward distribution of PC-GFNs where the preference w ~ (irichlett1.5) in Figure 5.I, where find that PC-GFNs might miss solutions and does not concentrate on the correct solutions very accurately.
branin-currin
Tranin-shubert
branin-beale
curri-sshubert
curin-beale
shubert-beale
branin currin shubert
branin-curtin
branin shubert beale
sorrin-shubrrt.o beale
True
DI
(00,0)
aISGFN
00.0.0)
10.0.
MOGFN PC
30.0.01
(0.0.0)
0.
Figure F.I: Reward Landscape: The first row above two figures contains all the states (blue) and the true Pareto front (orange). In the second and third rows, plot the first 128 generated candidates (bluc) and the estimated Pareto front (orange). The z-, y-axis, (and s-axis) are the first, second (and third) objectives in the title of respectively.
F.3 N-GRAMS
Network Structure For both the OP-GFNs and PC-GENs, we fix the backward transition probability Pa to be uniform, and parametrize the forward transition probability Pr(: (s, w)) by the Transformer
34

---

Published as a conference paper at ICLR 2024
OP-GFN(Ours) IGD+(4)
PC-GFN
Objective
du (4)
(C-
(4)
IGD+(1)
PC-ent(1)
branin-currin
0.018
2.66e-6
3.56
0.047
2.66e-6
3.49
branin-shubert
0.046
6.82e-6
2,56
0.080
0.055
2.08
branin-beale
0.0068
1.38e-8
3.75
0.064
1.38e-8
3.64
currin-shubert
0.039
1.28e-8
3.44
0.040
0.016
2.73
currin-beale
0.0028
1.56e-8
2.08
0.075
1.56e-8
2.04
shubert-beale
0.032
7.06e-9
3.06
0.034
0.016
2.21
branin-ourzin--hhubet
0.032
9.07e-7
4.57
0.05
0,016
4.32
branin-cuzzin-bealle
0.013
1.40e-4
5.28
0,015
0.0010
5,04
branin-shubertt-baale
0.023
1.44e-8
4.86
0.045
0.034
4.48
currin-shubert-beul
0.026
1.41e-8
4,4
0,048
0,029
3.58
All Objectives
0.020
1.77e-8
5.91
0.019
0.010
5.79
Table F.l: HyperGrid Task: (S, P), IGD+(P', P) and PC-ent(P',, of all generated candidates and generated Pareto front p' w.r.t. the true Pareto front P.
encoder. The Transformer encoder consists of hidden layers of dimension 64 and attention heads to embed the current state s. We encode the preference w ~ Dirichlet(1.0)) by thermometer encoding with bins and set reward exponent / 96 the PC-GFNs.
Training Pipeline We use the trajectory balance criterion in training the GelowNets. During training. models are trained by the Adam optimizer (Kingma & Ba 2014) with learning rate 10 for Pr parameters and 10-3 for Z's parameters, and the batch size 128 for 10000 steps. In the training, we inject the random noise with probability 0.01 in the Pr. In the OP-GFNs, We set the sampling temperature to be 0.1 during training and evaluation. We additionally use replay buffer of maximal size of 100000 and add 1000 warm-up candidates to the buffer at initialization. Instead of the on-policy update, push the online sampled batch into the replay buffer and immediately sample batch the same size.
Evaluation To evaluate the PC-GFNs, Jain et al. 2023) samples N candidates per test preference and then pick the top-k candidates with highest scalarized rewards. However, in our condition GelowNets, there is scalarization. In order align the evaluation setting, disable the selection process top-k candidates and take all the generated candidates into account. In other words, we sample N 128 candidates per w Dirichlet(1.. (1.fett(1.)) for 10 rounds, resulting in 1280 candidates in total.
# Objectives
Unigrams
Bigrams
"A", "C" "A", "C", "V"
" "AC", "Cv" "AC", "CV", "VA" "C", "V", "W" "AC", "CV" "VA", "AN"
Table F.2: Objectives considered for the n--gamm task
F.4 DNA SEQUENCE GENERATION
We use the same experimental settings as Appendix except for setting / = 80. We report the metries in Table F4, and plot the objective vectors in Figure F2
F33
We conclude that OP-GFNs achieve similar or better performance than preference conditioning. From Figure[F2 we notice that since 3 = 80 large, samples more closely to the Pareto front (in the second column), but lacks exploration (in the first and third column).
F.5 FRAGMENT BASED MOLECULE GENERATION
Network Structure We fix the backward transition probability P3 to be uniform, and parametrize the forward transition probability Pr by the GNN with layers, the node embedding size of 64. We encode the preference vector w or goal conditioning vector d, by thermometer encoding bins, and the temperature 3 by thermometer encoding of 32 bins.
35

---

Published as a conference paper at ICLR 2024
Algorithm
2 Bigrams
HV (1)
R2 (4)
PC-ent (t)
Diversity (t)
PC-GFN
0.50 + 0.03
1.45 + 0.05
2.12. 0.09
14.96 + 0.27
OP-GFN
0.56 + 0.05
1.51 + 0.26
2.38 + 0.12
7.56:10.27
2 Unigrams
PC-GFN
0.42 0.05
1.46 + 0.03
2.22 + 0.01
2.97 + 0.31
OP-GFN
0.47 0.00
1.45 + 0.01
2.24 + 0.00
7.30 + 0.04
3 Bigrams
PC-GEN
0.30 0.02
8.49 10.46
2.54
14.43.10.69
OP-GEN
0.30 + 0.02
9.95 + 0.64
1.46: 0.63
7.37 + 0.14
3 Unigrams
PC-GFN
0.03 0.01
10.68
0.58
2.77 + 0.29
3.86 1.117
OP-GFN
0.14 + 0.00
10.18 + 0.53
3.77 + 0.08
9.89 + 0.18
4 Bigrams
PC-GFN
0.04 0.006
46.58
11.47
3.74 + 0.23 4.18 + 0.10
15.35 + 0.27
OP-GEN
0.05 + 0.00
46.89 + 1.36
19.77 + 1.01
4 Unigrams
PC-GFN
0.005 = 0.00
49.37 00.4
3.52+0.15 4.87 + 0.12
4.56 0.41 11.57 + 0.37
OP-GEN 0.03 +.0000
50.78 + 2.24
Table F.3:
n-Grams Task: Hypervolume, R2 indicator, PC-entropy, and TopK diversity of OP-GFNs and PC-GENs on various tasks.
Objectives
Algorithm
HV (t)
Rg (L)
PC-ent (1)
Diversity (t)
PC-GFN
0.100
4.32
0.58
5.03
(1)-()
OP GFN
0.122
3,93
0.73
9.60
PC-GFN
0.352
2.65
0.0
5.50
(1)-()
OP GFN
0.352
2.65
0.0
13.70
PC-GFN
0.195
3,41
0.04
6.59
(2)-(3)
OP GFN
0.25
2.98
0.77
3.02
Table F.4: DNA sequence generation:
Hypervolume, R2 indicator, PC-entropy, and TopK diversity
of OP-GFN and PC-GEN on various tasks.
36

---

Published as a conference paper at ICLR 2024
energy_pins
energy_pairrs
pins_pairs
PC-GFN
OP GPN
Figure F.2: DNA sequence generation: We plot the generated 1280 candidates (blue) and the estimated Pareto front (orange). The I-, y-axis are the first, and second objective in the title of respectively.
Training Pipeline We use the trajectory balance criterion in training the GFlowNets. During training. models are trained by the Adam optimizer (Kingma & Ba 2014) with learning rate 10 for PF's parameters and 10 for Z's parameters, and the batch size 64 for 20000 steps. In the training, we inject the random noise with probability 0.01 in the Pp. For goal conditioning and order- preserving GFlowNets, we use replay buffer of size 100000, and 1000 warmup samples, and perform resampling. To prevent the sampling distribution from changing too abruptly, collect new trajectories from a sampling model PF (0sapiigg which uses a soft update with hyperparameter to track the learned GFN at the k-th update: sample + 7 gk-1 sample + (1 - r) gh, where @ the
current parameters from the learning, and we set 0.95.
Algorithm Specification In the preference conditioning GFlowNets, sample the preference vector w ~ (brichlet(1.O)) In the goal conditioning GFlowNets, a sample x is defined in the focus region g if the cosine similarity between its objective vector and the goal direction dg is above the threshold Cg. i.e. == {u R cus(u, dg) = || d 2 cg)- The reward function Rg depends on the current goal we set the reward to be
u.d,
lng mi Togeo
Rg(x) = (x) u(x) 1[u(x) g). where (z(x) (cos(u(())
dg))
We set the focus region cosine similarity threshold cy be 0.98, and the limit reward coefficient my to be 0.2. We sample the goal conditioning vector dy from the tabular goal-sampler Roy et al. (2023). In the predefined reward Rn and Rd, we the reward exponent
60.
Evaluation We sample 128 candidates per round, 50 rounds using the trained sampler. We report the metrics in the following:
37

---

Published as a conference paper at ICLR 2024
Objectives
PC-GFN
PC-GFN
OP-GFN (Ours)
(1)
IGD+(J)
PC-ent(1) (1(
IGD+(J) PC-ent(")
dy(())
IGD+(J) PC-ent(1)
seb-qed
0.56
0.20
1.88
0.55
0.20
1.84
0.55
0.20
1.87
seb-sa
0.33
0.15
1.71
0.34
0.14
1.58
0.33
0.15
2.18
seh-mw
0.14
0.15
1.33
0.14
0.14
1.56
0.14
0.13
1.69
qed-sa
0.35
0.06
1.73
0.34
0.05
1.47
0.33
0.07
1.52
ged mw
0.16
0.03
0.00
0.15
0.03
0.00
0.15
0.04
0,00
sa mw
0.13
0.00
0.00
0.14
0.00
0.00
0.13
0.00
0.00
Table F.5: Fragment Based Molecule Generation: (s(S, P), IGD+(P', P) and PC-ent(P', of all generated candidates S and generated Pareto front P w.r.t. the true Pareto front P.
seh-mw
Figure F.3; Fragment Based Molecule Generation (Full): We plot the generated 3200 candidates (blue) and the estimated Pareto front (orange). The x-, y-axis are the first, and second objectives in the title of respectively.
38


Reviews:
[{"review_id": "buhrqJgiNK", "reviewer": "Reviewer_BMH3", "summary": "This work proposes to extend GFlowNets, called Order-Preserving GFlowNets (OP-GFN), to sample candidates in proportion to a reward function such that the sampling is consistent with the provided partial order of the candidates. By this extension, they show how exploration and exploitation can be controlled, such that candidates higher in the hierarchy can be given more preference. The method shows benefits for both single objective and multi-objective cases by conducing experiments on a variety of tasks.", "questions": "1. Is it possible to include some more difficult configurations of the environments? Hypergrid is one easy example for such an experiment.\n2. It would be helpful if the variations across different seeds and runs could be provided for the experiments.\n3. A hyperparam search for the best learning rate can also be sometimes a big contributing factor across different learning methods.\n\nIf these could be provided along with the comments in the weaknesses, section it will help in addressing most of my concerns towards the experimental contribution of this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an important extension of the GFlowNets for multi-objective optimization when D > 1 objectives need to be optimized. \n2. The work also discusses how an efficient utilization of the GFlowNet policy can be achieved in difficult to explore settings.\n3. The theoretical results and analysis are useful to understand the proposed method and its advantages.\n4. The work also provides a good overview of the literature to benefit the reader.", "weaknesses": "1. The experiments section can be expanded to include more difficult environments. For example, for hypergrid,higher values of H and N can be tested as larger grids will help analyzing the exploration problems better.\n2. Detailed balance objective can perform reasonably well in many settings. It will be beneficial to include it in all methods and numbers reported.\n3. It will be useful to add standard deviation and error bars across experiments. It will also be useful to better understand the variance across different GFlowNet objectives and baseline methods.\n\nBy addressing these concerns with experiments, the authors will help address the empirical limitations to strengthen their theoretical discussions and the overall contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LKEVDVrJsv", "reviewer": "Reviewer_8SYj", "summary": "The paper introduces an innovative approach for training GFlowNets, eliminating the necessity for an explicit formulation of the reward function while ensuring compatibility with the provided order of candidates. This method involves the simultaneous training of a reward function, which maintains the order of samples in conjunction with GFlowNet. The authors conducted comprehensive experiments in both single and multi-objective settings, revealing that their proposed approach yields outstanding results in comparison to prior methods when dealing with only pairwise order relations and non-convex Pareto fronts.", "questions": "**Section 4.2**\n\nAuthors say that they compare the proposed method with GFN-$\\beta$, where $\\beta$ is the selected value from the previous work. As far as I know, the temperature is not tuned in the previous work. What if we assign high values to $\\beta$? It seems that we can achieve higher topk reward by sacrificing diversity when we assign high values to $\\beta$ and authors only consider maximal objective in single-experiment setting\n\n**Section 5.2, 5.4**\n\nAuthors say that they use a replay buffer and do off-policy training. I am curious that as the reward function is trained across training, off-policy training may lead to degrading performance. \n\n**Minor Questions**\n\nSection 4.1) $u_0\\$ seems a typo. Maybe $R_0$?\n\nSection 4.3) Authors propose two implementation tricks, KL regularization and trajectories augmentation. Especially trajectories augmentation has shown superior results in the previous paper. It is hard to capture when those two tricks are applied. While authors say trajectory augmentation is applied in molecule design (E.3.2), main paper say that trajectories augmentation is optionally used in NAS environment. It may be helpful when those two tricks are applied across differnt experiment settings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper conducts extensive experiments under different objective settings and domains. Especially, the paper conducts experiments on NAS benchmark, which is a first attempt to apply GFlowNets into NAS while it is natural as we can make neural architecture by adding operations in a sequence manner. It also achieves superior results compared to other baselines in NAS benchmark.\n\nThe paper also tackles multi-objective problems with a non-convex Pareto front, which is hard to solve with prior multi-objective GFN methods such as PC-GFN. Additionally, the paper eliminates the necessity for fine-tuning the temperature parameter, β, which plays a critical role in GFlowNets' performance.", "weaknesses": "1. There is a possibility of encountering non-stationarity issues when jointly training GFlowNets and the reward function. It might be worth exploring alternative training strategies to mitigate this potential challenge.\n\n2. Experimental results are not that persuasive, having little improvement over baselines. For example, this work just compares with simple GFN baseline in molecular tasks, more competitive baselines (e.g., subTB, FL-GFN, RL methods) are needed. \n\n---\n\n\n**Discussion needed regarding temperature conditioning methods**\n\nThere are some researches on temperature-conditioned GFlowNets, which learn a single model for multiple reward functions conditioned on different temperatures. It may be more persuasive to compare the proposed method with the following literature [1], [2], especially in single-objective settings. Note that reviewer understands that temperature-conditioned GFlowNets are too recently proposed, so the authors could not reflect this in this submission but suggest making some discussion in the final version for future researchers!\n\n[1] Zhang, David W., et al. \"Robust scheduling with GFlowNets.\" arXiv preprint arXiv:2302.05446 (2023).\n\n[2] Kim, Minsu, et al. \"Learning to Scale Logits for Temperature-Conditional GFlowNets.\" arXiv preprint arXiv:2310.02823 (2023).\n\n---\n\n**Decision**\n\nThis work is novel, well-written, easy to understand, and provides insight for GFN communities. Although experimental results do not strictly outperform every competitive baseline, this work provides a lot of experiments over various tasks. To this end, the score would be 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TnSsz3q0gd", "reviewer": "Reviewer_ipVV", "summary": "The paper proposes to use the GFlowNet framework to learn samplers of optimal points based on given orderings. This is useful because it applies to single and multi-objective problems alike.\n\nThe method works by learning a reward function that minimizes the divergence between the optimality distribution (i.e. the indicator function around the argmax of $\\mathcal{X}$) and the distribution induced by the learned reward function, as learned by a GFlowNet. This function is learned simultaneously with the corresponding GFlowNet sampler that tries to match this non-stationary reward, which at convergence neatly results in finding the optimal points.", "questions": "Notes:\n- For someone not familiar with GFNs, Eq (3) may not be obvious (R being parameterized \"through\" P_F P_B & Z)\n- I'm not sure that I fully appreciate the \"piecewise linear\" part of the contribution. Is there something I'm missing?\n    - First of all, the whole thing is discrete and operating on sets. It makes no sense to talk about piecewise linearity because we are not in $\\mathbb{R}$, the pieces \"in between\" don't even exist.\n    - Ignoring the above, the shape of the function doesn't really matter? What matters is $\\mathrm{order}(a,b) \\equiv \\mathrm{order}(\\hat R(a), \\hat R(b))$, which follows in a fairly straightforward way once we assume 0 loss.\n- Proposition 3/6 is similarly weird, why this specific choice of MDP? I'd recommend either introducing why this choice matters, or maybe at least state that the general case is much harder to provide theoretical statements for. I find the current form a bit awkward\n- 4.2; the authors seem to be using the molecular setup of Shen et al., not Bengio et al.. Shen et al.'s setup is quite different (and more simple, the state spaces are made to be much small so that computing $p(x; \\theta)$ becomes tractable). It would be appropriate to note this. It's not clear what \"choices\" the authors are using from Shen et al. since quite a few are introduced in that work (which appear to have a significant impact)\n- \"optionally use the backward KL regularization (-KL) and trajectories augmentation (-AUG) introduced in Appendix E.1\", these options don't seem to have that much impact, but it's a bit weird that they're just mentioned in passing in the main text.\n- \"we observe that OP-GFNs can learn a highly sparse reward function that concentrates on the true Pareto solutions, outperforming PC-GFNs.\" It's a bit weird to claim here that OP-GFNs are outperforming PC-GFNs, because they learn entirely different things. From Fig 5.1 it seems like PC-GFN is attributing some probability mass on the Pareto front points, which suggests to me that it has discovered them. \n    - This is maybe more of a comment on RL literature, but I'm also not a fan of comparing algorithms on a grid. Gridworlds are useful to sanity check algorithms, not to compare algorithms.\n- In Sections 4 & 5, it would be good to clarify what work introduced what task and what MDP setup which is being used.\n- Why build on GFlowNets? Don't get me wrong I'm a big fan of GFNs, but it seems to me like the use-case of that framework is to obtain \"smooth\" energy-based amortized samplers, whereas the proposed work is meant to converge to an extremely peaky distribution where essentially $p(x^*)=1$, and is much more akin to finding the optimal greedy policy in RL. There may be a very good reasons (I can think of some ;) but I think the paper could do a better job of explaining them.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea proposed in the paper is solid and the execution is well done; the method is tested on a whole variety of tasks and relevant setups.  \nThe idea certainly relates to other rank-based methods, such as those in RL & search, but stands on its own in the GFlowNet framework.", "weaknesses": "My biggest criticism of the paper is really its presentation.\n\nIt's really not clear what the algorithm actually is, readers have to go all the way into the appendix to find it, and even there some questions remain, are $\\mathcal{T}$ and $\\mathcal{D}$ distinct? What is $\\hat R$ trained on? Does it have distinct parameters? Shared? etc.   \nFrom scrolling through the appendix, it appears that there are, understandably, a number of tricks that can be used. Most of them have some form of ablation in one task or the other, but a cohesive summary is lacking. The main criticism here is that it's important to disentangle what the contribution is from the algorithm itself, and from the tricks.\n\nInvestigative figures like Fig. E.6 should really be in the main body of the paper, and it would be much nicer to have such a figure for more complex domains. Generally the paper doesn't do a great job of showing _why_ the method works empirically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yb38vgMPim", "reviewer": "Reviewer_Ndf3", "summary": "The paper proposes OP-GFN, order-preserving GFlowNets, that sample with probabilities in proportion to a learned reward function instead of explicitly defined reward, the learned reward is compatible with a provided partial order on the candidates. OP-GFN promotes exploration in the early stages and exploitation in the later stages of the training by gradually sparsifying the reward function during the training. The authors provide theoretical proof that the learned reward is piecewise log-linear with respect to the ranking by the objective function. OP-GFN are proposed for both the single-objective maximization and multi-objective Pareto approximation problems, where experimental results on synthesis environment HyperGrid and two real-world applications NATS-Bench and molecular designs demonstrate competitive performance.", "questions": "In section 4.2, the results shown are for the top-100 candidates, are the results similar for other choices of K? \n\nIn the experiment on neural architecture search, in Figure 4.2 the test accuracy comparison is against the clocktime, could you plot against the number of samples to understand the impact of learning the reward and sample efficiency of OR–GFN?\n\nHave you considered other reward schemes besides the exponentially scaled reward? For example, some UCB-variant of the reward to balance between exploration vs exploitation? \n\nRelated work on encouraging exploration of GFlowNets: https://arxiv.org/pdf/2306.17693.pdf\n\n\nMinor Comments:\nFigure 4.1 caption, typo “Topk”\nFigure 4.1, should the legend be ‘TB-OP’ instead of ‘TB-RP’?\nIn section 4.1, typo “TB‘s performance”", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical contribution of the paper is sound and novel, where1)  the log reward being piecewise linear with respect to the ranking enables exploration on the early training stages and the exploitation on the later sparsified reward R is a useful and desirable feature in many training procedures; 2) matching the flow F with a sufficiently trained OP-GFN will assign high flow value on non-terminal states that on the trajectories ending in maximal reward candidates, enabling sampling optimal candidates exponentially more often than non-optimal candidates.\t\t\t\n\nThere is a range of comprehensive experimental results with details in the main document or in the supplementary material\n\nThe paper is well written and organized, making it easy to follow.", "weaknesses": "There can be more comparison against other state-of-the-art methods in the experimental section. The experiments, such as the hypergrid, only compares against the GFN-TB method. If the exploration-exploitation is the main benefit of OR-GFN in single objective problems, having more baseline comparison that encourages exploration of TB (such as https://arxiv.org/pdf/2306.17693.pdf or other variations of the reward besides beta-exponentiating) can make the paper stronger. \n\nIn single-objective maximization, the form of the local label y of the Pareto set is explicitly given in section 3.2, however in the multi-objective case, it is not clear how the label y can be calculated a priori, when the Pareto fronts are unknown. “When the true Pareto front is unknown, we use a discretization of the extreme faces of the objective space hypercube as P.” It is not clear whether the proposed method works due to the good estimate of the P, i.e., whether GGN can also benefit from incorporating estimated Pareto fronts into reward definition somehow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L63cazreAh", "reviewer": "Reviewer_CCax", "summary": "A new method called Order-Preserving GFlowNets (OP-GFNs) has been proposed to address issues with Generative Flow Networks (GFlowNets) in sampling candidates with probabilities proportional to a given reward. GFlowNets can only be used with a predefined scalar reward, which can be computationally expensive or not directly accessible, and the conventional practice of raising the reward to a higher exponent may vary across environments. OP-GFNs eliminate the need for an explicit formulation of the reward function by sampling with probabilities in proportion to a learned reward function that is consistent with a provided (partial) order on the candidates. The training process of OP-GFNs gradually sparsifies the learned reward landscape in single-objective maximization tasks, concentrating on candidates of a higher hierarchy in the ordering to ensure exploration at the beginning and exploitation towards the end of the training. OP-GFNs demonstrate state-of-the-art performance in both single-objective maximization and multi-objective Pareto front approximation tasks.", "questions": "Please refer to the weaknesses part in the previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper studies an interesting problem of GFlowNets -- training an order-preserving GFlowNets. The paper is also well-written and easy to follow.", "weaknesses": "> Note, that the optimal $\\beta$ heavily depends on the geometric landscape of $u(x)$.\n\nThere have been a few related works that attempt to determine the optimal $\\beta$ either from the perspective of dynamically annealing temperature or training a temeperature-conditioned GFlowNet that can generalize across a set of different temperatures. \n\nIn addition, it would be better to also compare OP-GFN with these baselines besides GFN-$\\beta$.\n\nKim, Minsu, Joohwan Ko, Dinghuai Zhang, Ling Pan, Taeyoung Yun, Woochang Kim, Jinkyoo Park, and Yoshua Bengio. \"Learning to Scale Logits for Temperature-Conditional GFlowNets.\" *arXiv preprint arXiv:2310.02823* (2023).\n\nZhang, David W., Corrado Rainone, Markus Peschl, and Roberto Bondesan. \"Robust Scheduling with GFlowNets.\" In *The Eleventh International Conference on Learning Representations*. 2022.\n\n> Therefore, we only focus on evaluating the GFlowNet’s ability to discover the maximal objective.\n\nIf the goal is to discover the maximal objective, why do you use GFlowNets and not RL, wher the latter learns an optimal policy that maximizes the return.\n\n>  The ability of standard GFlowNets, i.e. $R(x) = u(x)$, is sensitive to $R_0$.\n\nThe performance of trajectory balance is indeed senstive to the value of $R_0$ (which is also mentioned in the paper), which leads to large variance. However, this is usually not the case for flow matching, detailed balance, and sub-trajectory balance. It would be better to make this claim in a more clear way. In addition, it would be better to include a more extensive study of OP-GFN with other more advanced learning objectives (e.g., FM, DB, SubTB with temporal difference based objective) with a more thorough discussion. \n\n> Large $R_0$ encourages exploration but hinders exploitation since a perfectly trained GFlowNet will also sample non-optimal candidates with high probability; whereas low $R_0$ encourages exploitation but hinders exploration since low reward valleys hinder the discovery of maximal reward areas far from the initial state. \n\n$R_0$ actually determines the sparsity of the reward function -- with a larger $R_0$, the reward function is less sparse, while a smaller $R_0$ corresponds to a high sparsity in the reward function. Therefore, it is inappropriate to say that \"Large $R_0$ encourages exploration\" and \"low $R_0$\" encourages exploitataion. Whether encouraging exploration (or exploitation) for the agent is determined by the learning algorithm itself, instead of $R_0$ (which is the underlying environment).\n\n> We observe that the OP-GFN outperforms the GFN-β method by a significant margin.\n\nIt would be better to also include more advanced baselines and methods in the field of NAS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)