Title: Let the Flows Tell:  Solving Graph Combinatorial Problems with GFlowNets
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph; combinatorial optimization; sampling; gflownets
Abstract: Combinatorial optimization (CO) problems are often NP-hard and thus out of reach for exact algorithms, making them a tempting domain to apply machine learning methods. The highly structured constraints in these problems can hinder either optimization or sampling directly in the solution space.
On the other hand, GFlowNets have recently emerged as a powerful machinery to efficiently sample from composite unnormalized densities sequentially and have the potential to amortize such solution-searching processes in CO, as well as generate diverse solution candidates.
In this paper, we design Markov decision processes (MDPs) for different combinatorial problems and propose to train conditional GFlowNets to sample from the solution space. 
Efficient training techniques are also developed to benefit long-range credit assignment.
Through extensive experiments on a variety of different CO tasks with synthetic and realistic data, we demonstrate that GFlowNet policies can efficiently find high-quality solutions.
Our implementation is open-sourced at https://github.com/zdhNarsil/GFlowNet-CombOpt.

Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNets

Abstract

Combinatorial optimization (CO) problems are often NP-hard and thus out of reach for exact algorithms, making them a tempting domain to apply machine learning methods. The highly structured constraints in these problems can hinder either optimization or sampling directly in the solution space. On the other hand, GFlowNets have recently emerged as a powerful machinery to efficiently sample from composite unnormalized densities sequentially and have the potential to amortize such solution-searching processes in CO, as well as generate diverse solution candidates. In this paper, we design Markov decision processes (MDPs) for different combinatorial problems and propose to train conditional GFlowNets to sample from the solution space. Efficient training techniques are also developed to benefit long-range credit assignment. Through extensive experiments on a variety of different CO tasks with synthetic and realistic data, we demonstrate that GFlowNet policies can efficiently find high-quality solutions. Our implementation is open-sourced at https://github.com/zdhNarsil/GFlowNet-CombOpt.

1 Introduction

Combinatorial optimization (CO) is a branch of optimization that studies problems of minimizing or maximizing some cost over a finite feasible set. CO problems usually involve discrete structures, such as graphs, networks, and permutations, and require optimizing an objective function subject to discrete constraints in the solution space, which is often NPhard. CO problems have broad applications, including in medicine, engineering, operations research, and management (Paschos, 2010), and have spurred the development of discrete mathematics and theoretical computer science for a century (Kuhn, 1955; Kruskal, 1956; Ford & Fulkerson, 1956).

During the past few decades, researchers have developed numerical solvers such as GUROBI (Gurobi Optimization, 2023) to give approximate solutions via integer programming. In recent years, interest in

…

…

…

…

…

…

…

learning-based methods for solving CO problems has grown significantly. These approaches leverage the power of deep networks to learn the inherent structure of CO problems and provide efficient

max ind set

and effective solutions. One family of machine learning methods utilizes solver-found solutions to provide a supervised learning (SL) signal for training neural networks (Selsam et al., 2018; Gasse et al., 2019; Nair et al., 2020). This line of algorithms requires expensive precomputation by the numerical solver to produce supervision labels. On the other hand, unsupervised learning (UL) methods search for the solution without the help of a solution oracle. One branch of UL methods, called probabilistic methods, decodes the heatmap generated from one pass of a neural network to get solutions (Toenshoff et al., 2019; Karalias & Loukas, 2020). These methods can achieve fast inference at the cost of a large optimality gap. Another branch of unsupervised methods is reinforcement learning (RL), which iteratively refines or constructs the problem solution with practitioner-specified MDPs (Bello et al., 2016; Deudon et al., 2018; Wu et al., 2019).

Despite many recent efforts to apply deep RL to CO problems, such approaches have fundamental limitations. For example, due to the symmetry in problem configurations, there could be multiple optimal solutions to the same CO problem (Li et al., 2018). Standard RL algorithms such as Fujimoto et al. (2018) are grounded in the nature of cumulative reward maximization and fail to promote diversity in the solutions. Although entropy-regularized RL (Haarnoja et al., 2017, 2018; Zhang et al., 2023b) converges to a stochastic policy instead of a deterministic one, these methods target trajectory-level entropy rather than solution-level entropy . Consequently, the agent may get trapped in solutions that can be reached by many trajectories and lack the ability to generate diverse candidate solutions. In addition, the performance of RL methods largely depends on the designed reward function and relies on a dense per-step reward for learning the value functions and policies. As a result, it is challenging to apply RL in our problems if only a terminal reward is provided.

Although there are attempts to fix these issues (Kwon et al., 2020; Ahn et al., 2020), they are mostly problem-specific and only achieve marginal improvement. In this work, we turn to a more principled framework, namely generative flow networks (Bengio et al., 2021, GFlowNets), to search for highquality diverse candidates in CO problems. GFlowNet is a novel decision-making framework for learning stochastic policies to sample composite objects with probability proportional to a given terminal reward, which is suitable for problems where the solution is only related to the terminal state of generative trajectories. We design MDPs for a variety of NP-hard CO problems, where the intermediate states form a flow network in the latent space, and GFlowNet learns an agent to sequentially make decisions in this environment (Figure 1). Another challenge of applying GFlowNets is learning from long trajectories. In large-scale graph CO problems, the GFlowNet agent will encounter a very long trajectory before termination, rendering the task of credit assignment challenging. To this end, we develop efficient learning algorithms to train GFlowNets from transitions rather than complete trajectories, which greatly helps the learning process, especially in large-scale setups. Through extensive experiments on different CO tasks, we demonstrate the advantage of our proposed GFlowNet approach. In summary, our contributions are as follows:

We design a problem-specific MDP for GFlowNet training on four different CO tasks.

We propose an efficient GFlowNet learning algorithm to enable fast credit assignment for the GFlowNet agents with long trajectories that emerge in our graph CO problems.

The empirical advantage of GFlowNets is validated through experiments on different CO problems.

2 Preliminaries

2.1 GFlowNets

Generative flow networks, or GFlowNets, are variational inference algorithms that treat sampling from a target probability distribution as a sequential decision-making process (Bengio et al., 2021, 2023). We briefly summarize the formulation and the main training algorithms for GFlowNets.

We assume that a fully observed, deterministic MDP with set of states S and set of actions A ⊆ S ×S is given. The MDP has a designated initial state , denoted s 0 . Certain states are designated as terminal and have no outgoing actions; the set of terminal states is denoted X . All states in S are assumed to be reachable from s 0 by a (not necessarily unique) sequence of actions (see Figure 2). A complete trajectory is a sequence of states τ = ( s 0 → s 1 → ··· → s n ) , where s n ∈ X and each pair of consecutive states is related by an action, i.e. , ∀ i ( s i , s i +1 ) ∈ A .

A policy on the MDP is a choice of distribution P F ( s ′ | s ) for each s ∈ S \ X over the states s ′ reachable from s in a single action. 2 A policy induces a distribution over complete trajectories via

The marginal distribution over the final states of complete trajectories is denoted P ⊤ F , a distribution on X that may in general be intractable to compute exactly, as P ⊤ F ( x ) = ∑ τ → x P F ( τ ) , with the sum taken over all complete trajectories that end in x .

A reward function is a mapping X → R > 0 , which is understood as an unnormalized probability mass on the set of terminal states (we will typically make the identification R ( x ) = exp( -E ( x ) /T ) , where E : X → R is an energy function and T > 0 is a temperature parameter). The learning problem approximately solved by a GFlowNet is to fit a policy P F ( s ′ | s ) such that the induced distribution P ⊤ F ( x ) is proportional to the reward function, i.e. ,

The policy P F ( s ′ | s ) is parametrized as a neural network with parameters θ taking s as input and producing the logits of transitioning to each possible subsequent states s ′ . This problem is made difficult both by the intractability of computing P ⊤ F given P F and by the unknown normalization constant (partition function) on the right side of (1). Learning algorithms overcome these difficulties by introducing auxiliary objects into the optimization. Next, we review two relevant objectives.

Detailed balance (DB) The DB objective (Bengio et al., 2023), requires learning two objects in addition to a parametric forward policy P F ( s ′ | s ; θ ) (we omit θ when it causes no ambiguity):

A backward policy , which is a distribution P B ( s | s ′ ; θ ) over the parents (predecessors) of any noninitial state in the MDP;

A state flow function F ( · ; θ ) : S → R > 0 .

The detailed balance loss for a single transition s → s ′ is defined as

The DB training theorem states that if ℓ DB ( s , s ′ ; θ ) = 0 for all transitions s → s ′ , then the policy P F satisfies (1), i.e. , samples proportionally to the reward. The manner of selecting transitions s → s ′ on which to minimize (2) is discussed below.

The loss that performs best for the problems in this paper (see (6) in §3.3) is equivalent to DB with a particular parametrization of log F ( s ) that bootstraps learning by expressing the log-state flow as an additive correction to a partially accumulated negative energy.

Trajectory balance (TB) The TB objective (Malkin et al., 2022) features a simpler parametrization: in addition to the action policy P F , one learns a backward policy P B and only a single scalar Z θ , an estimator of the partition function corresponding to the initial state flow F ( s 0 ) in the DB parametrization. The TB loss for a complete trajectory τ = ( s 0 → s 1 →··· → s n = x ) is

The TB training theorem states that if ℓ TB ( τ ; θ ) = 0 for all complete trajectories τ , then the policy P F satisfies (1). Furthermore, Z then equals the normalization constant of the reward, ˆ Z = ∑ x ∈X R ( x ) .

In practice, policies and flows are typically output in the log domain, i.e. , a neural network predicts logits of the distributions P F ( ·| s ) , P B ( ·| s ′ ) and the log-flows log F ( s ) and log Z .

Training policy and exploration The DB and TB losses depend on individual transitions or trajectories, but leave open the question of how to choose the transitions or trajectories on which they are minimized. A common choice is to train in an on-policy manner, i.e. , rollout trajectories τ ∼ P F ( τ ; θ ) and perform gradient descent steps on ℓ TB ( τ ; θ ) or on ℓ DB ( s i , s i +1 ; θ ) for transitions s i → s i +1 in τ . In this case, DB and TB have close connections to variational (ELBO maximization) objectives (Malkin et al., 2022).

However, an exploratory behaviour policy can also be used, for example, by sampling τ from a version of P F that is tempered or mixed with a uniform distribution (resembles ϵ -greedy exploration in RL). Note that, unlike policy gradient methods in RL, GFlowNet objectives require no differentiation through the sampling procedure that yields τ . The ability to stably learn from off-policy trajectories is a key advantage of GFlowNets over hierarchical variational models (Zimmermann et al., 2022; Malkin et al., 2023). See related study in Section C.

Conditional GFlowNets The MDP and the reward function in a GFlowNet can depend on some conditioning information. For example, in the tasks we study, a GFlowNet policy sequentially constructs the solution to a CO problem on a graph g , and the set of permitted actions depends on g . The conditional GFlowNets we train achieve amortization by sharing the policy model between different g , enabling generalization to g not seen in training.

2.2 Graph combinatorial optimization problems

We focus on the following four NP-hard CO problems on graphs: maximum independent set (MIS), maximum clique (MC), minimum dominating set (MDS), and maximum cut (MCut). A CO problem can be described with an undirected graph g = ( V, E ) , where V is the set of vertices and E is the set of edges. Such problems typically require one to optimize over variables in a finite composite space to maximize or minimize some particular graph properties, as follows. Without loss of generality, we assume that all the graph weights equal one for simplicity, as our method can be easily extended to weighted graphs.

Maximum independent set In graph theory, an independent set is a set of vertices S in a given graph structure g where any pair of vertices { i, j } ⊆ S are not neighbors i.e. , ∀ i, j ∈ S, ( i, j ) / ∈ E . The MIS problem is to find such an independent set that has the largest possible size | S | .

Maximum clique Aclique is a subset of the vertices S ⊆ V where all pairs of vertices are adjacent, i.e. , ∀ i, j ∈ S, ( i, j ) ∈ E . The MC problem is to find a clique that has the largest size. We also remark that MC can be considered as a complementary problem of MIS, in the sense that the MC of any graph is actually the MIS of its complementary graph.

Minimum dominating set A dominating set S ⊆ V for a graph g is a subset of vertices such that, for any vertex in this graph, it is either in S , or it has a neighbor in S . The MDS problem is to find the smallest dominating set for a given graph structure.

Maximum cut Given a subset of graph vertices S ⊆ V , the cut is defined as the number of edges between S and V \ S . The MCut problem is to find a set of vertices S that maximizes the cut.

3 Methodology

3.1 Optimization as probabilistic inference

A CO problem can be seen as a constrained energy minimization problem in a discrete composite space arg min x ∈X E ( x ) , where E is the target energy function and X is the solution space or feasible set. This optimization problem can be considered roughly as sampling from an energy-based model p ∗ T ( x ) ∝ exp {-E ( x ) /T } , where T is the temperature parameter that controls the smoothness of the density landscape. Sampling from such highly structured space is non-trivial, therefore we propose to use GFlowNets to amortize this inference process. According to the GFlowNet theory, a perfectly trained GFlowNet with the reward function R ( x ) = exp {-E ( x ) /T } will be able to accurately sample from p ∗ T ( x ) . In this way, GFlowNets trained with a reasonably small temperature T can be used to search for solutions to given CO problems, as stated below.

Proposition 1. Assume that the GFlowNet is perfectly trained for a given temperature T , i.e., the training loss over a policy with full support equals zero. Then, if T →∞ , the distribution sampled

max ind set

0

1

…

(a) Independent set

⊘

1

…

(b) Clique

⊘

0

1

…

min dom set

(c) Dominating set

⊘

0

1

…

(d) Cut

⊘

0

by GFlowNet will converge to a uniform distribution on X ; if T → 0 , the distribution sampled by GFlowNet will converge to the uniform distribution on optimal solutions to the optimization problem.

Notice that each graph g corresponds to one unique sampling problem; thus, here we learn a graph conditional GFlowNet to amortize this condition distribution p ( x | g ) , i.e. , every learnable GFlowNet component (like the forward / backward policy or the flow function) is conditioned on the graph g .

3.2 Designing Markov decision processes for GFlowNets

This section illustrates the design of appropriate Markov decision process (MDP) formulations for GFlowNet learning; see Figure 2 for a summary.

State For all the CO problems studied in this work, the solution x is a subset of vertices for a given graph, represented as a binary vector x = ( x 1 , . . . , x | V | ) ∈ X ⊆ { 0 , 1 } | V | , where x i = 1 indicates that the i -th vertex belongs to the set and x i = 0 indicates that it does not. Note that the feasible solution space X is a subset of the full space { 0 , 1 } | V | , as some binary vectors encode a vertex set outside the feasible set ( e.g. , a non-independent set in the MIS problem). The solution space is also the GFlowNet's terminal state space. The design of the GFlowNet state space is inspired by that in Zhang et al. (2022b). We begin by defining

where ⊘ represents an unspecified 'void' or 'yet unspecified' situation for a particular vertex. Notice that S is a superset of the terminal state space X . The initial state is the all-void vector s 0 = ( ⊘ , ⊘ , . . . , ⊘ ) . In our MDP design, we only allow transforming vertex values from void ( ⊘ ) to non-void ( 0 or 1 ) according to problem-specific rules. The trajectory terminates when all the entries are non-void, i.e. , every entry is either 0 or 1 . Notice that we do not need to have an explicit stop action in the agent's action space.

Notice that not all vectors in S that have no void entries lie in X . Therefore, we must restrict S so that the set of terminal states is exactly the set of vectors encoding feasible solutions X . To be precise, we define the state space S to be the set of states s ∈ S such that there exists at least one x ∈ X that can be obtained from s by valid transitions.

Care must be taken to modify the set of permitted actions from each state accordingly, so that actions always produce states in S . This turns out to be doable in the problems we study (as, for example, any subset of an independent set is independent). As an example, we next describe our action / transition / reward design for the MIS problem. Details for other tasks are deferred to the Appendix B.

Action The initial state is all-void, meaning that the partially constructed independent set is initialized with the empty set. The action of the MIS MDP is simply to choose one void vertex and add it to the current solution set by turning the entry value of the chosen vertex from ⊘ to 1 .

max clique

˜

s

0

s

1

s

2

…

s

n

s

n

-1

s

3

ℰ(

x

)

s

s

′

…

s

s

′

ℰ(

s

2

)

˜

ℰ(

s

3

)

˜

ℰ(

s

n

-1

)

Algorithm 1 GFlowNet training algorithm

Require: Graph dataset, GFlowNet model with parameters θ , reward function.

repeat

Sample graph g from dataset;

Sample τ with the GFlowNet forward policy P F ( ·|· ; g , θ ) ;

Randomly choose transitions from τ to create buffer B ;

△ θ ← ∑ ( s → s ′ ) ∈B ∇ θ ℓ FL ( s , s ′ ; g , θ ) (as per Equation 6);

Update θ with some optimizer; until some convergence condition

Transition To ensure that the state can always be completed to an independent set, it is essential to carefully handle the actions taken. When a void vertex is chosen and its entry value is modified to 1 , we also update the entry values of all its neighboring vertices to 0 , which excludes the possibility of getting two adjacent vertices in the following steps. This proactive approach ensures that the independent set constraint is not violated in subsequent steps.

We remark that the feasible set X in this problem consists not of all independent sets, but of ordermaximal independent sets, i.e. , those to which no vertex can be added while keeping them independent. Non-order-maximal independent sets cannot be constructed with such transitions.

Reward We set the log reward to be the resulting independent set size, i.e. , E ( x ) = -| x | 1 , where | x | 1 denotes the ℓ 1 norm, i.e. , the number of 1 's in the binary vector x .

3.3 Factors affecting training efficiency

Transition-based GFlowNet training Most successful GFlowNet implementations ( e.g. , those in https://github.com/GFNOrg/) require a complete trajectory to compute the training loss and its gradient. Existing implementations with DB also specify the training loss at the trajectory level:

and calculate the parameter gradient update with E τ ∼ π ( τ ) [ ∇ θ L ( τ ; θ )] with some potentially offpolicy distribution π ( τ ) . This is also the case for other GFlowNet algorithms, such as flow matching (Bengio et al., 2021) and subtrajectory balance (Madan et al., 2023).

These implementations work well for moderate-scale trajectories, as previous GFlowNet works have shown (Malkin et al., 2022; Madan et al., 2023). However, for very long trajectories, such a design hinders efficient training: for a single complete trajectory which contains n transitions, one needs n calls of the neural network forward passes and storing all the intermediate feature maps and parameters in the GPU memory. Each forward pass contains multiple message passing operations on given graph structures, which is computationally expensive (linearly increasing with n ) in terms of speed and memory storage. In our experiments, we encounter large-scale problems where the trajectory is as long as ∼ 400 in length; nonetheless, with such graphs, our adopted graph neural networks can only support

5

10

15

20

Epoch

16

18

MIS size

Method

FL (ours)

FL-traj

DB

DB-traj

TB

forward and backward passes with batch size approximately 250 (which is much smaller than 400 ) on a 40 GB GPU memory device. This prohibits the usage of these trajectory-based GFlowNet training objectives on large-scale graph applications. In addition, the correlation between consecutive samples may incur stability issues (Mnih et al., 2015).

To this end, we turn to use a transition-based GFlowNet training approach wihtout the knowledge of complete trajectories, which is first proposed in Deleu et al. (2022) and has shown effectiveness in Nishikawa-Toomey et al. (2022); Deleu et al. (2023). We randomly sample B transitions B = { ( s b , s ′ b ) } B b =1 from a complete trajectory and construct detailed balance-based loss: L ( θ ) = 1 B ∑ B b =1 ℓ ( s b , s ′ b ; θ ) . This enables GFlowNets to be efficiently trained with long trajectories and limited GPU memory, which also converges faster. A corresponding schematic illustration of the algorithm can be found in Figure 3. In Figure 4, we show a comparison of learning efficiency between transition-based and trajectory-based GFlowNet implementations, where '-traj' denotes the latter variant. Different methods here share similar speed for one epoch training, thus from the figure we can see that transition-based approaches learn more efficiently. Besides the improved computation efficiency, our transition-based approach is also more memory efficient - as the memory occupation is proportional to the batch size instead of trajectory lengths (as for trajectory-based implementation), demonstrating its applicability to long-horizon problems.

Improving credit assignment with intermediate learning signals For normal GFlowNet training methods, the only learning signal in a trajectory comes from the terminal states and their associated reward values. This results in a relatively slow credit assignment process, and is inefficient to propagate information from near-terminal states to the early states due to the difficulty of attributing credits of each action in a long trajectory. The ability to learn from incomplete trajectories is especially important in our proposed transition-based training, since we are not using all the transitions from the complete trajectories. Therefore, we incorporate intermediate learning signals via the forwardlooking (Pan et al., 2023a, FL) technique:

where ˜ E ( · ) : S → R is a continuation of the reward energy E ( · ) : X → R which is only defined in the terminal state space X . For simplicity, here we ignore the conditioning on graph structure g . This FL method enables dense supervision signals to GFlowNet training, resulting in faster credit assignment as can be seen in Figure 3. Notice here that we need to design a handcrafted reward ˜ E ( s ) for all possible latent states to reflect our estimation on intermediate signals. For MIS problems, we naturally define -˜ E ( s ) to be the number of vertices in the current set, i.e. , the number of 1 entries in state s . This semantically coincides with the definition of an MIS terminal reward. We defer the intermediate reward design for other tasks to the Appendix. The effectiveness of FL against the DB or TB algorithms can be seen in Figure 4. We summarize the resulting algorithm in Algorithm 1.

4 Related work

GFlowNets GFlowNets were intended as diversity-seeking samplers for biological sequence and molecule design, an application area that continues to motivate research (Bengio et al., 2021; Jain et al., 2022, 2023b; Shen et al., 2023; Jain et al., 2023a). However, much recent work has used GFlowNets as samplers for Bayesian posterior distributions, e.g. , causal discovery (Deleu et al., 2022; Nishikawa-Toomey et al., 2022; Atanackovic et al., 2023), amortized variational EM with discrete latents (Hu et al., 2023), neurosymbolic inference (van Krieken et al., 2022), and feature attribution in classifiers (Li et al., 2023). The theory and optimization techniques for GFlowNets have also evolved, with improved training objectives and exploration techniques (Malkin et al., 2022; Madan et al., 2023; Pan et al., 2022, 2023a; Shen et al., 2023), better understanding of their connections to variational methods (Zhang et al., 2022a; Malkin et al., 2023; Zimmermann et al., 2022), and extensions to stochastic (Zhang et al., 2023c; Pan et al., 2023b) and continuous (Lahlou et al., 2023) MDPs.

MLfor combinatorial optimization The surge of machine learning for CO problems alleviates the reliance on hand-crafted heuristics while enabling generalization to new instances (Bengio et al., 2018; Cappart et al., 2021). Some methods (Li et al., 2018; Gasse et al., 2019; Gupta et al., 2020; Sun & Yang, 2023) rely on supervised information from expert solvers, which can be hard to obtain. Alternative approaches that leverage reinforcement learning (Dai et al., 2017; Kool et al., 2019; Chen & Tian, 2019; Yolcu & Póczos, 2019; Ahn et al., 2020; Delarue et al., 2020; Drori et al., 2020) or other unsupervised learning objectives (Karalias & Loukas, 2020; Sun et al., 2022; Wang et al., 2022) broaden the applicability of learning for CO problems. However, the mode-collapse issue might hinder the diversity and thus the solution coverage. In this regard, GFlowNets are easy to train while also designed for discovering multiple modes. As the first example to show the advantage of

GFlowNets in CO problems, Zhang et al. (2023a) tackles the robust job scheduling problem that is central to compiler optimization. In our paper, we formalize CO under the GFlowNet framework with principled MDP design and demonstrate its effectiveness in a wide range of graph CO tasks.

5 Experiments

We conduct extensive experiments on various graph CO tasks to demonstrate the effectiveness of the proposed GFlowNet approach. For MIS problems, we follow the setup in the MIS benchmark from Böther et al. (2022), while for other tasks, we follow the experimental setup in Sun et al. (2022).

Datasets As Dai et al. (2021) have pointed out that problems in existing synthetic graph data are relatively easy for MIS and MC, we take the more complicated RB graphs (Xu & Li, 2000) following Karalias & Loukas (2020). For realistic data, we take the SATLIB dataset (Hoos et al., 2000), which is reduced from SAT instances in conjunctive normal form. For the other two tasks, namely MDS and MCut, we adopt BA graphs (Barabási & Albert, 1999) following Sun et al. (2022). For all types of synthetic graph data, we generate two scales of datasets (denoted SMALL and LARGE), which respectively contain around 200 to 300 vertices and 800 to 1200 vertices.

Baselines For MIS problems, we compare with the baselines in the MIS benchmark. For classical operation research (OR) methods, we include a general-purpose mixed-integer program solver (GUROBI) and a MIS-specific solver (Lamm et al., 2017, KAMIS). For learning-based methods, we compare with a reinforcement learning-based PPO method (Ahn et al., 2020), and supervised learning with tree search refinement, in two different implementations (Li et al. (2018, INTEL) and Böther et al. (2022, DGL)). For the non-MIS tasks, we compare with the GUROBI solver, two heuristic methods which are greedy and mean-field annealing (Bilbro et al., 1988, MFA), and two state-of-the-art probabilistic methods (Karalias & Loukas, 2020) and the annealed version (Sun et al., 2022). We use ERDOS and ANNEAL to denote these two learning-based methods. For max-cut problems, we also adopt a semi-definite programming baseline (coined SDP in the result tables) which aims at a relaxation of the MCut task. We set its maximal running time to 10 hours. For the methods already contained in the MIS benchmark, we train them from scratch and report performance with the same protocol; for other algorithms, we write our own implementations and test them in the same way.

Results & analysis We evaluate both the performance and the inference time and report the mean value of the objective ( e.g. , set size in MIS) and the approximation ratio relative to the best-performing non-ML solver, treated as an oracle. 3 The time denotes the total latency of evaluating on the test set. The best results among non-OR methods are marked in bold. MIS results are demonstrated in Table 1, where the problem-specific solver KAMIS is served as the gold standard for calculating the drop. For MIS methods, the larger the size of the independent set found, the better the algorithm. The 'UL" (unsupervised learning) refers to the algorithms that do not need labels ( i.e. , solutions found by solvers) in the training set. For SATLIB, none of the supervised learning baselines can

achieve meaniningful results (average MIS size < 400 ) within 10 hours, thus we use ' -' to mark performance. One can see that GFlowNets surpass all the learning-based baselines in the sense of achieving the largest independent set solution.

For the MC & MDS & MCut problems, we exhibit the experimental results on small graphs in Table 2 and results on large graphs in Table 3, where GUROBI serves as the gold standard to calculate the performance drop ratio. Notice that for MDS problems, the smaller the vertex size result, the better the performance, which is opposite to other tasks. We observe that GFlowNet outperforms other baselines across different tasks and problem scales, with the only exception being the large-scale max-cut problem. This reflects the fairly universal effectiveness of the proposed method. We also notice that GFlowNets require a longer inference time compared to other methods. This is because the GFlowNet only adds one vertex into the set at each step, thus it needs multiple steps to output a vertex set solution. In contrast, probabilistic methods such as 'Erd˝ os goes neural' (Karalias & Loukas, 2020) only require one neural network forward pass during inference time to give a coarse estimate of the solution. We can see how this compares with a sequential generation approach as in GFlowNets, where each decision is taken in the context of the previously already taken decisions, ensuring a better coordinated set of decisions.

Ablation study We now conduct in-depth ablation studies to evaluate the key design choices in our method. We first study the difference between a series of GFlowNet variants in Figure 4, based on the MIS task with small scale graph data. We compare transition-based FL, trajectory-based FL, transition-based DB, trajectory-based DB, and TB (which only has trajectory-based implementation). Our result indicates that GFlowNet's transition-based FL implementation learns the fastest for CO tasks, which supports our modeling choice in Section 3.3. Figure 5 in Section C summarizes additional ablation studies including the temperature annealing, off-policy exploration strategy, and network architectures. We vary the GFlowNet's temperature coefficient which scales the temperature hyperparameter; we ablate the off-policy exploration during the rollout stage of GFlowNet training; we also ablate the GFlowNet architecture. Our results indicate that the GFlowNet is robust to a

wide range of hyperparameters, making it appealing to be applied to different CO problems, which validates the effectiveness of our proposed methodology from another perspective.

6 Conclusion

Our work focuses on the challenges of solving CO problems using unsupervised learning approaches. This work contributes to this growing field by proposing to apply the principled GFlowNet decisionmaking framework to CO tasks. By combining the power of probabilistic inference and sequential decision-making, GFlowNets offer a promising direction for finding a diverse set of high-quality candidate solutions in CO problems. Technically, we have developed problem-specific MDPs and efficient learning algorithms to address the challenges associated with learning from long trajectories, making our approach practical and scalable in large-scale setups. Our extensive numerical results showcase the effectiveness and efficiency of GFlowNets in solving NP-hard level CO problems, highlighting their ability to generate a set of diverse high-quality solutions. We believe that our work opens up new possibilities for addressing the limitations of existing approaches and paves the way for future research at the intersection of machine learning and combinatorial optimization.

References

Ahn, Sungsoo, Seo, Younggyo, and Shin, Jinwoo. Learning what to defer for maximum independent sets. International Conference on Machine Learning (ICML) , 2020.

Atanackovic, Lazar, Tong, Alexander, Hartford, Jason, Lee, Leo J., Wang, Bo, and Bengio, Yoshua. DynGFN: Bayesian dynamic causal discovery using generative flow networks. arXiv preprint 2302.04178 , 2023.

Barabási, Albert-Laszlo and Albert, Reka. Emergence of scaling in random networks. Science , 286 5439:509-12, 1999.

Bello, Irwan, Pham, Hieu, Le, Quoc V., Norouzi, Mohammad, and Bengio, Samy. Neural combinatorial optimization with reinforcement learning. arXiv preprint 1611.09940 , 2016.

Bengio, Emmanuel, Jain, Moksh, Korablyov, Maksym, Precup, Doina, and Bengio, Yoshua. Flow network based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS) , 2021.

Bengio, Yoshua, Lodi, Andrea, and Prouvost, Antoine. Machine learning for combinatorial optimization: a methodological tour d'horizon. Eur. J. Oper. Res. , 290:405-421, 2018.

Bengio, Yoshua, Deleu, Tristan, Hu, Edward J., Lahlou, Salem, Tiwari, Mo, and Bengio, Emmanuel. Gflownet foundations. Journal of Machine Learning Research (JMLR) , 2023.

Bilbro, Griff L., Mann, Reinhold, Miller, Thomas K., Snyder, Wesley E., van den Bout, David E., and White, Mark W. Optimization by mean field annealing. Neural Information Processing Systems (NIPS) , 1988.

Böther, Maximilian, Kißig, Otto, Taraz, Martin, Cohen, Sarel, Seidel, Karen, and Friedrich, Tobias. What's wrong with deep learning in tree search for combinatorial optimization. International Conference on Learning Representations (ICLR) , 2022.

Cappart, Quentin, Chételat, Didier, Khalil, Elias, Lodi, Andrea, Morris, Christopher, and Veliˇ ckovi´ c, Petar. Combinatorial optimization and reasoning with graph neural networks. International Joint Conference on Artificial Intelligence (IJCAI) , 2021.

Chen, Xinyun and Tian, Yuandong. Learning to perform local rewriting for combinatorial optimization. Neural Information Processing Systems (NeurIPS) , 2019.

Dai, Hanjun, Khalil, Elias B., Zhang, Yuyu, Dilkina, Bistra, and Song, Le. Learning combinatorial optimization algorithms over graphs. Neural Information Processing Systems (NIPS) , 2017.

Dai, Hanjun, Chen, Xinshi, Li, Yu, Gao, Xin, and Song, Le. A framework for differentiable discovery of graph algorithms. 2021.

Delarue, Arthur, Anderson, Ross, and Tjandraatmadja, Christian. Reinforcement learning with combinatorial actions: An application to vehicle routing. Neural Information Processing Systems (NeurIPS) , 2020.

Deleu, Tristan, G'ois, Ant'onio, Emezue, Chris C., Rankawat, Mansi, Lacoste-Julien, Simon, Bauer, Stefan, and Bengio, Yoshua. Bayesian structure learning with generative flow networks. Uncertainty in Artificial Intelligence (UAI) , 2022.

Deleu, Tristan, Nishikawa-Toomey, Mizu, Subramanian, Jithendaraa, Malkin, Nikolay, Charlin, Laurent, and Bengio, Yoshua. Joint bayesian inference of graphical structure and parameters with a single generative flow network. ArXiv , abs/2305.19366, 2023. URL https://api. semanticscholar.org/CorpusID:258987383 .

Deudon, Michel, Cournut, Pierre, Lacoste, Alexandre, Adulyasak, Yossiri, and Rousseau, LouisMartin. Learning heuristics for the tsp by policy gradient. In Integration of AI and OR Techniques in Constraint Programming , 2018.

Drori, Iddo, Kharkar, Anant, Sickinger, William R., Kates, Brandon, Ma, Qiang, Ge, Suwen, Dolev, Eden, Dietrich, Brenda L, Williamson, David P., and Udell, Madeleine. Learning to solve combinatorial optimization problems on real-world graphs in linear time. 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) , pp. 19-24, 2020.

Ford, Lester Randolph and Fulkerson, Delbert Ray. Maximal flow through a network. Canadian Journal of Mathematics , 8:399 - 404, 1956.

Fujimoto, Scott, Hoof, Herke, and Meger, David. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning (ICML) , 2018.

Gasse, Maxime, Chételat, Didier, Ferroni, Nicola, Charlin, Laurent, and Lodi, Andrea. Exact combinatorial optimization with graph convolutional neural networks. Neural Information Processing Systems (NeurIPS) , 2019.

Gupta, Prateek, Gasse, Maxime, Khalil, Elias, Mudigonda, Pawan, Lodi, Andrea, and Bengio, Yoshua. Hybrid models for learning to branch. Neural Information Processing Systems (NeurIPS) , 2020.

Gurobi Optimization, LLC. Gurobi Optimizer. https://www.gurobi.com , 2023.

Haarnoja, Tuomas, Tang, Haoran, Abbeel, Pieter, and Levine, Sergey. Reinforcement learning with deep energy-based policies. International Conference on Machine Learning (ICML) , 2017.

Haarnoja, Tuomas, Zhou, Aurick, Abbeel, P., and Levine, Sergey. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML) , 2018.

Hoos, Holger H., Stützle, Thomas, Gent, Ian P., van Maaren, Hans, and Walsh, Toby. SATLIB: An online resource for research on SAT. 2000.

Hu, Edward J., Malkin, Nikolay, Jain, Moksh, Everett, Katie, Graikos, Alexandros, and Bengio, Yoshua. GFlowNet-EM for learning compositional latent variable models. International Conference on Machine Learning , 2023.

Jain, Moksh, Bengio, Emmanuel, García, Alex, Rector-Brooks, Jarrid, Dossou, Bonaventure F. P., Ekbote, Chanakya Ajit, Fu, Jie, Zhang, Tianyu, Kilgour, Micheal, Zhang, Dinghuai, Simine, Lena, Das, Payel, and Bengio, Yoshua. Biological sequence design with GFlowNets. International Conference on Machine Learning (ICML) , 2022.

Jain, Moksh, Deleu, Tristan, Hartford, Jason, Liu, Cheng-Hao, Hernández-García, Alex, and Bengio, Yoshua. GFlowNets for AI-driven scientific discovery. Digital Discovery , 2023a.

Jain, Moksh, Raparthy, Sharath Chandra, Hernández-García, Alex, Rector-Brooks, Jarrid, Bengio, Yoshua, Miret, Santiago, and Bengio, Emmanuel. Multi-objective gflownets. International Conference on Machine Learning (ICML) , 2023b.

Karalias, Nikolaos and Loukas, Andreas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. Neural Information Processing Systems (NeurIPS) , 2020.

Kool, Wouter, van Hoof, Herke, and Welling, Max. Attention, learn to solve routing problems! International Conference on Learning Representations (ICLR) , 2019.

Kruskal, Joseph B. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society , 7(1):48-50, 1956.

Kuhn, Harold W. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly , 2(1-2):83-97, 1955.

Kwon, Yeong-Dae, Choo, Jinho, Kim, Byoungjip, Yoon, Iljoo, Min, Seungjai, and Gwon, Youngjune. POMO: Policy optimization with multiple optima for reinforcement learning. Neural Information Processing Systems (NeurIPS) , 2020.

Lahlou, Salem, Deleu, Tristan, Lemos, Pablo, Zhang, Dinghuai, Volokhova, Alexandra, HernándezGarcía, Alex, Ezzine, Léna Néhale, Bengio, Yoshua, and Malkin, Nikolay. A theory of continuous generative flow networks. International Conference on Machine Learning (ICML) , 2023.

Lamm, Sebastian, Sanders, Peter, Schulz, Christian, Strash, Darren, and Werneck, Renato F. Finding near-optimal independent sets at scale. J. Heuristics , 23(4):207-229, 2017. doi: 10.1007/ s10732-017-9337-x. URL https://doi.org/10.1007/s10732-017-9337-x .

Li, Wenqian, Li, Yinchuan, Li, Zhigang, Hao, Jianye, and Pang, Yan. DAG Matters! GFlowNets enhanced explainer for graph neural networks. International Conference on Learning Representations (ICLR) , 2023.

Li, Zhuwen, Chen, Qifeng, and Koltun, Vladlen. Combinatorial optimization with graph convolutional networks and guided tree search. Neural Information Processing Systems (NeurIPS) , 2018.

Madan, Kanika, Rector-Brooks, Jarrid, Korablyov, Maksym, Bengio, Emmanuel, Jain, Moksh, Nica, Andrei Cristian, Bosc, Tom, Bengio, Yoshua, and Malkin, Nikolay. Learning GFlowNets from partial episodes for improved convergence and stability. International Conference on Machine Learning (ICML) , 2023.

Malkin, Nikolay, Jain, Moksh, Bengio, Emmanuel, Sun, Chen, and Bengio, Yoshua. Trajectory balance: Improved credit assignment in GFlowNets. Neural Information Processing Systems (NeurIPS) , 2022.

Malkin, Nikolay, Lahlou, Salem, Deleu, Tristan, Ji, Xu, Hu, Edward J., Everett, Katie Elizabeth, Zhang, Dinghuai, and Bengio, Yoshua. GFlowNets and variational inference. International Conference on Learning Representations (ICLR) , 2023.

Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. Humanlevel control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Nair, Vinod, Bartunov, Sergey, Gimeno, Felix, von Glehn, Ingrid, Lichocki, Pawel, Lobov, Ivan, O'Donoghue, Brendan, Sonnerat, Nicolas, Tjandraatmadja, Christian, Wang, Pengming, Addanki, Ravichandra, Hapuarachchi, Tharindi, Keck, Thomas, Keeling, James, Kohli, Pushmeet, Ktena, Ira, Li, Yujia, Vinyals, Oriol, and Zwols, Yori. Solving mixed integer programs using neural networks. arXiv preprint 2012.13349 , 2020.

Nishikawa-Toomey, Mizu, Deleu, Tristan, Subramanian, Jithendaraa, Bengio, Yoshua, and Charlin, Laurent. Bayesian learning of causal structure and mechanisms with gflownets and variational bayes. ArXiv , abs/2211.02763, 2022. URL https://api.semanticscholar.org/CorpusID: 253384724 .

Pan, Ling, Zhang, Dinghuai, Courville, Aaron C., Huang, Longbo, and Bengio, Yoshua. Generative augmented flow networks. International Conference on Learning Representations (ICLR) , 2022.

Pan, Ling, Malkin, Nikolay, Zhang, Dinghuai, and Bengio, Yoshua. Better training of GFlowNets with local credit and incomplete trajectories. International Conference on Machine Learning (ICML) , 2023a.

Pan, Ling, Zhang, Dinghuai, Jain, Moksh, Huang, Longbo, and Bengio, Yoshua. Stochastic generative flow networks. arXiv preprint 2302.09465 , 2023b.

Paschos, Vangelis Th. Applications of combinatorial optimization. 2010.

Qiu, Ruizhong, Sun, Zhiqing, and Yang, Yiming. Dimes: A differentiable meta solver for combinatorial optimization problems. ArXiv , abs/2210.04123, 2022.

Selsam, Daniel, Lamm, Matthew, Bünz, Benedikt, Liang, Percy, de Moura, Leonardo Mendonça, and Dill, David L. Learning a SAT solver from single-bit supervision. International Conference on Learning Representations (ICLR) , 2018.

Shen, Max W., Bengio, Emmanuel, Hajiramezanali, Ehsan, Loukas, Andreas, Cho, Kyunghyun, and Biancalani, Tommaso. Towards understanding and improving GFlowNet training. International Conference on Machine Learning (ICML) , 2023.

Sun, Haoran, Guha, Etash K., and Dai, Hanjun. Annealed training for combinatorial optimization on graphs. arXiv preprint 2207.11542 , 2022.

Sun, Zhiqing and Yang, Yiming. DIFUSCO: Graph-based diffusion solvers for combinatorial optimization. arXiv preprint 2302.08224 , 2023.

Toenshoff, Jan, Ritzert, Martin, Wolf, Hinrikus, and Grohe, Martin. Graph neural networks for maximum constraint satisfaction. Frontiers in Artificial Intelligence , 3, 2019.

van Krieken, Emile, Thanapalasingam, Thiviyan, Tomczak, Jakub M, van Harmelen, Frank, and ten Teije, Annette. A-NeSI: A scalable approximate method for probabilistic neurosymbolic inference. arXiv preprint 2212.12393 , 2022.

Wang, Haoyu Peter, Wu, Nan, Yang, Hang, Hao, Cong, and Li, Pan. Unsupervised learning for combinatorial optimization with principled objective relaxation. Neural Information Processing Systems (NeurIPS) , 2022.

Wu, Yaoxin, Song, Wen, Cao, Zhiguang, Zhang, Jie, and Lim, Andrew. Learning improvement heuristics for solving routing problems. IEEE Transactions on Neural Networks and Learning Systems , 33:5057-5069, 2019.

Xu, K. and Li, W. Exact phase transitions in random constraint satisfaction problems. Journal of Artificial Intelligence Research , 12:93-103, Mar 2000. ISSN 1076-9757. doi: 10.1613/jair.696. URL http://dx.doi.org/10.1613/jair.696 .

Xu, Keyulu, Hu, Weihua, Leskovec, Jure, and Jegelka, Stefanie. How powerful are graph neural networks? International Conference on Learning Representations (ICLR) , 2019.

Yolcu, Emre and Póczos, Barnabás. Learning local search heuristics for boolean satisfiability. Neural Information Processing Systems (NeurIPS) , 2019.

Zhang, David, Rainone, Corrado, Peschl, Markus, and Bondesan, Roberto. Robust scheduling with GFlowNets. International Conference on Learning Representations (ICLR) , 2023a.

Zhang, Dinghuai, Chen, Ricky T. Q., Malkin, Nikolay, and Bengio, Yoshua. Unifying generative models with GFlowNets and beyond. arXiv preprint 2209.02606 , 2022a.

Zhang, Dinghuai, Malkin, Nikolay, Liu, Zhen, Volokhova, Alexandra, Courville, Aaron C., and Bengio, Yoshua. Generative flow networks for discrete probabilistic modeling. International Conference on Machine Learning (ICML) , 2022b.

Zhang, Dinghuai, Courville, Aaron C., Bengio, Yoshua, Zheng, Qinqing, Zhang, Amy, and Chen, Ricky T. Q. Latent state marginalization as a low-cost approach for improving exploration. International Conference on Learning Representations (ICLR) , 2023b.

Zhang, Dinghuai, Pan, L., Chen, Ricky T. Q., Courville, Aaron C., and Bengio, Yoshua. Distributional GFlowNets with quantile flows. arXiv preprint 2302.05793 , 2023c.

Zimmermann, Heiko, Lindsten, Fredrik, van de Meent, J.-W., and Naesseth, Christian A. A variational perspective on generative flow networks. Transactions of Machine Learning Research (TMLR) , 2022.

A Notations

B MDPdesigns

In all MDPs, the agent maintain a set of vertices as the state. This is achieved by assign a value among { 0 , 1 , ⊘} for each of the vertex, which corresponds to 'not in the set" / 'in the set" / 'unspecified". At each step, the agent performs an action, which is essentially changing the entry value of one vertex from ⊘ to a specified binary value. After that, the designed transition may modify the value of other unspecified vertices to ensure the state will represent a feasible solution. The agent will receive a terminal reward when no action can be taken ( i.e. , termination). An illustration of the following descriptions can be found in Figure 2.

B.1 Maximum clique MDP

Here we think of the all-void initial state means that we have an empty set of vertex at the beginning.

Action The action is to choose an unspecified vertex and include it in the current set. This is changing the entry value of that vertex in the state vector representation from ⊘ to 1 .

Transition We need to ensure that the current set always represents a clique, thus it is crucial to enforce the connectivity between vertices in the set. If a vertex could potentially join the clique in the future, then it must connect to all the vertices in the current set. Hence, after performing an action, we identify the vertices that fail to meet this requirement and mark them as 0 . This ensures that the clique constraint is preserved throughout the generation process.

Reward We set the log reward to be the resulting clique size, i.e. , E ( x ) = -| x | 1 . For intermediate state s , we define its intermediate signal by setting -˜ E ( s ) to be the number of 1 in s .

B.2 Minimum dominating set MDP

In MDS, we start with an all-void initial state, where all vertices (which are in unspecified state) are considered to in the target dominating set. This is different from other MDP designs, where we think of unspecified void state as not in the vertex set. In this representation, the entire vertex set V forms a trivial dominating set. Starting from this, we perform actions to remove vertices from the current set but keep it to be a dominating set all the time.

5

10

15

20

Epoch

16

18

20

MIS size

Method

w/ anneal

w/o anneal

5

10

15

20

Epoch

18

MIS size

1

1×10 1

1×10 2

5×10 2

1×10 3

5×10 3

0%

25%

50%

75%

100%

Exploration noise level

10

20

MIS size

1

2

3

4

5

6

7

8

Number of layers

10

20

MIS size

Action Each action corresponds to selecting a vertex and removing it from the current dominating set. This is done by changing the entry value of the chosen vertex from ⊘ to 0 .

Transition For a vertex in the graph, if it is adjacent to the current set and all of its neighbors are also connected to the set, removing this vertex from the set will not violate the dominating set requirement. For other isolated vertices, we change their values to 1 to ensure they stay in the set, thus the state s consistently representing a dominating set. By making these adjustments, we maintain the integrity of the dominating set representation throughout the process.

Reward The log reward is set to be the negative dominating set size, i.e. , E ( x ) = | x | 1 . For intermediate state s , we define its intermediate signal by setting ˜ E ( s ) to be the number of 1 in s .

B.3 Maximum cut MDP

We start with an empty solution vertex set, i.e. , all the vertices are in ⊘ value.

Action Same as MIS and MC, the action is to add one vertex to the current solution set of vertices, by changing its entry value from ⊘ to 1 .

Transition After each action is carried out, we perform the following check: if involving one void vertex to the solution vertex set would decrease the cut value, then we exclude this vertex by labeling it to 0 from void ⊘ . This guarantees that when the trajectory terminates ( i.e. , no void unspecified vertex), the cut is locally maximal.

Reward We set the negative energy to be the size of the current cut set, i.e. , the number of edges between all 1 vertices and all 0 vertices. For intermediate state s , we define its intermediate signal by setting -˜ E ( s ) to be the number of edges between all 1 vertices and other vertices (note that we think of void vertices as not in the solution set).

C GFlowNet details

We start with the proof of Proposition 1 and then describe other algorithmic details.

Proof. According to the theory of GFlowNets (Bengio et al., 2023), if GFlowNet's training loss is zero on full support, then the GFlowNet is able to sample correctly from the target distribution p ∗ T ( x ) . When we gradually move temperature T from finite to ∞ , the distribution p ∗ T ( x ) will gradually become the uniform distribution over X , thus the distribution sampled by GFlowNet will converge to this uniform distribution; when the temperature gradually approaching zero, the distribution p ∗ T ( x ) will gradually become the uniform distribution over all argmax solution of arg max x E ( x ) , thus the distribution sampled by GFlowNet will converge to this uniform distribution.

For the GFlowNet architecture, we use graph isomorphism network (Xu et al., 2019, GIN) with 5 hidden layers and 256 dimensional hidden size, for both the forward policy and the state flow function. We double the number of hidden layers for SATLIB experiment. The input of the GIN is a integer

vector represented state in the form of { 0 , 1 , 2 } | V | followed by an embedding layer, where we use 2 to denote the unspecified void ⊘ . For the forward policy GIN, we set a unidimensional output node feature followed by a softmax operation over all vertices, which gives a categorical distribution on all the vertices. For the state flow GIN, we use a graph pooling layer to extract a unidimensional graph-level output to serve as the flow value. We fix the backward policy to be a uniform distribution over all chosen vertices for simplicity. Theoretically, we can only use one GIN for both the flow function and the forward policy, but we find it is much less stable. We hypothesize that the flow function training stability is very important in DB-based GFlowNet methods , and that most previous GFlowNet works underfit the flow function and underestimate the performance of DB .

We use the Adam optimizer with default 1 × 10 -3 learning rate without hyperparameter tuning. After breaking the complete trajectories into transition pieces, we combine them to many buffer with batchsize equals 64 to calculate forward-looking based detailed balance loss. The training keeps 20 epochs, although most experiments actually converge in the first 5 epochs. For the SATLIB experiment, our proposed method converges within the first epoch. During the inference phase, we simply the same evaluation protocol in Ahn et al. (2020). To be concrete, we sample 20 solutions for each graph configuration and take the best result.

We conduct a series of ablation study with small graph MIS experiments in Figure 5. Each algorithm is repeated with five different random seeds. For the temperature T used in GFlowNet's reward shaping, we try values of the inverse temperature from 1 to 5000 and find values higher or equal than 100 will obtain similar results. We thus simply use an inverse temperature of 500 for all the tasks. We also use temperature annealing, to start with temperature of 1 and terminate with the target temperature level (although the ablation study shows that GFlowNet could reach same performance without the annealing trick). During the training, GFlowNet will need to rollout ( i.e. , interact with the environment to collect complete trajectories) to collect training data. In this work, we do onpolicy exploration, which means the policy used in the rollout stage is the current forward policy. Nonetheless, GFlowNet has the ability of doing off-policy exploration. In the third ablation study, we mix the forward policy with different levels of uniform noise distribution, and find that GFlowNet is very robust under this off-policy setups. In this fourth ablation study, we change the number of hidden layer in the GIN architecture and find GFlowNet is also robust to this hyperparameter.

D More about experiments

For BA graphs, we use the variants with 4 attaching edges. For small scale datasets, we first sample the number of vertices uniformly from 200 to 300 and generate the graph instances. For large-scale problems, it is uniformly sampled from 800 to 1200 . Both the small and large synthetic graph trainsets contain 4000 data points for either RB graph or BA graph. For the SATLIB dataset, there are 40000 data points in total, where the largest graph has 1347 vertices. For all datasets, we use a validation set and a test set, both with 500 data points. We choose the hyperparameters of each algorithm based on the validation performance. Regarding hardware, we use NVIDIA Tesla V 100 Volta GPUs. We repeat the experiments for five different random seeds.

We also conduct MIS experiments on Erdos-Renyi (ER) graph data shown in Table 4. We use ER data as large as 700 ∼ 800 vertices and 9000 ∼ 11000 vertices for a comparison with Qiu et al. (2022, DIMES) and Sun & Yang (2023, DIFUSCO). The training and evaluation protocols are not exactly the same, but we still present results from these baselines for completeness. DIMES achieves 38 . 24 / 42 . 06 and 320 . 50 / 332 . 80 (' / " means two variants of their algorithm) respectively

for these two tasks; DIFUSCO achieves 38 . 83 / 41 . 12 (' / " means two variants of their algorithm) for the ER700 -800 problem, and is not scalable for extremely large problems like ER9000 -11000 . The difference in protocol includes: GFlowNet uses less training data samples; DIFUSCO is a supervised learning algorithm (requires a solver to provides solution), while DIMES and GFlowNet are unsupervised methods; the test data may not be the same (they generate their own fold). For GFlowNet in the largest task, we use a temperature of 1000 and a batch size of 32 . In summary, we can see that GFlowNet also achieves state-of-the-art performance in this setup, especially in large problems.

In MIS benchmark, for the MIS supervised learning baselines (INTEL and DGL), we disable the usage of the graph reduction and 2-opt local search tricks, but only adopt the queue pruning and weighted queue pop technique. This is because even random initialized neural network can be used to find near optimal solutions with graph reduction and local search as pointed out by Böther et al. (2022). Furthermore, these forbidden techniques are very time-consuming, making the inference time exceeding our time limit constraint. For KAMIS, we use the redumis algorithm. For GUROBI, we formulate MIS as quadratic programming problem and then use the pygurobi python package to solve it. We also run Erdos-based probabilistic methods (ERDOS and ANNEAL) for MIS, but their results are worse than any baselines methods in the MIS benchmark, thus we do not include them in Table 1. In the max cut task, we use the cvxpy python package to implement the SDP baseline.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a new graph combinatorial optimization method by leveraging a generative flow network (GFlowNet) and other novel technics to improve it. Maximum independent sets (MIS) and their variants are very important problems as they can be applied to several high-impact tasks such as network communication. As these problems are, NP-hard exact method cannot solve them within a reasonable time, so that approximated solver such as a deep learning policy can support the production of near-optimal solutions quickly. This work follows this research trends and makes state-of-the-art over the deep learning baselines. \n\nA major contribution is the usage of GFlowNet rather than PPO because exploration of GFlowNet can cover the symmetric nature lies in combinatorial solution space using DAG-based MDP construction. The novelty of this work is the improvement of GFlowNet for long trajectory generation of combinatorial optimization, so this work proposed (1) transition-based training and (2) intermediate learning signal training, which is a simple yet intuitive idea. \n\nWhile this work is concrete and interesting, I have several questions and concerns; please address these concerns in the rebuttal discussion phase. \n\nI just wrote the questions below.\n\nSTRENGTHS:\nThis work is novel, and its performance seems to be promising. First, they cleverly leverage existing Markov decision process formulation of learning what to defer (LwD), as the state is described as $s \\in \\{0,1,2\\}$ where 0 is included, one is excluded, and two is deferred; please make clear spotlight that this MDP idea is from prior work of LwD [1]. \n\nSecond, they leveraged GFlowNet rather than PPO so that it could consider the symmetric nature of combinatorial solution space. \n\nThird, their additional techniques are widely studied things in GFlowNet research (e.g., the intermediate reward is similar to Generative Augmented Flow Network [2], and the transition-based method is identical to sub-trajectory balance [3]) but made intuitive variations for combinatorial optimization. \n\n[1] Ahn, Sungsoo, Younggyo Seo, and Jinwoo Shin. \"Learning what to defer for maximum independent sets.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Pan, Ling, et al. \"Generative augmented flow networks.\" arXiv preprint arXiv:2210.03308 (2022).\n\n[3] Madan, Kanika, et al. \"Learning GFlowNets from partial episodes for improved convergence and stability.\" arXiv preprint arXiv:2209.12782 (2022).\n\nWEAKNESSES:\nOne can be said that technical novelty is limited as every technique are already actively explored; I think making the novel combination of existing technics is also significant. \n\nI think the description of many ideas follows existing trends of GFlowNet; I enjoyed reading this paper, but maybe some who don't know about GFlowNet work can miss many detailed parts of this technique. Please make an explicit description of each technique and give a reference idea of where you are inspired by and what is the major difference, e.g., sub-trajectory balance vs. transition-based learning (assume that reader also not familiar with the sub-trajectory balance). \n\nFinally, there are several neural combinatorial optimizations works [1,2] that study the symmetric nature of combinatorial optimization. I humbly suggest to include as a reference paper. \n\n[1] Kim, Minsu, Junyoung Park, and Jinkyoo Park. \"Sym-nco: Leveraging symmetricity for neural combinatorial optimization.\" Advances in Neural Information Processing Systems 35 (2022): 1936-1949.\n\n[2] Kim, Hyeonah, et al. \"Symmetric Exploration in Combinatorial Optimization is Free!.\" arXiv preprint arXiv:2306.01276 (2023).\n\nQUESTIONS:\n1. How does diversity (e.g., the edit distance between sampled solution) changes when your new techniques are applied? (to compare with another method, including the trajectory balance method)?\n\n2. What kinds of post-processing methods are used for solution generation? Did you samples multiple samples from the learned sampler? Then what is the actual sampling width? How about applying local search just as learning what to defer (LwD) [1] ?\n\n3. To compare with LwD, did you put 2opt for LwD following their original paper?\n\n4. Can this method extend to other combinatorial optimization, such as the traveling salesman problem (TSP)?\n\n[1] Ahn, Sungsoo, Younggyo Seo, and Jinwoo Shin. \"Learning what to defer for maximum independent sets.\" International Conference on Machine Learning. PMLR, 2020.\n\nLIMITATIONS:\nGFlowNet is an on-going framework having a lot of limitations. Please make explicit limitations on the main paper for future researchers.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach for learning to solve graph combinatorial optimization by GFlowNets. The GFlowNets (or broadly, the generative models) own the advantage of discovering multiple (near)-optimal solutions, which is better than standard RL or SL. The authors modify GFlowNets to work with larger-scale problems in CO and provide extensive experiments on 4 different problems.\n\n---------------------\n\nPost-rebuttal: I appreciate the authors for the feedback and I am now more positive about this paper.\n\nSTRENGTHS:\n* This paper is well-written. \n* The motivation for introducing GFlowNets to CO is sound and interesting.\n* The authors made modifications to GFlowNetw to fit larger-scale CO problems.\n\nWEAKNESSES:\n* My major concern of this paper is about the selection of baselines and the implementations in experiments.\n  * The authors mentioned in L55 that \"there are attempts to fix these issues (Kwon et al., 2020; Ahn et al., 2020)\" from the RL literature, but these methods are not implemented and compared in experiments.\n  * Why the selection of baselines are different for MIS and the other 3 problems, especially considering that MIS and Max clique are complement?\n  * What will the performance be like if we let Gurobi run the same amount of time compared to your GFlowNet solvers?\n  * During inference, do you implement a search algorithm after the GFlowNets? What about the other neural solver baselines?\n  * How do you implement, configure and tune the baselines? Do you implement the same GIN as your implementation of GFlowNets? For example, from my knowledge, PPO's performance is strongly dependent on hyperparameter tuning and the configuration of tricks. Besides, the probabilistic method (Karalias & Loukas, 2020) has two types of configurations (fast/accurate) and which one do you choose?\n  * I also notice you follow some recent neural CO solvers such as Qiu et al. (2022, DIMES) and Sun & Yang (2023, DIFUSCO) but do not consider them in main experiments. Can you offer any justifications? \n* The training process does not seem clear to me. Is the training objective to minimize $l_{DB}+l_{TB}$? How is the reward being used? Seeing that GFlowNet is quite a new framework, we should not expect the readers to be that familiar with the domain knowledge. \n* Some details seem to be missing: \n  * What is the meaning of \"MIS size\" in Fig 4? On which dataset did you get the numbers in Fig 4?\n* Other minor issues:\n  * The notation of $P_F^\\top$ is improper. As I understand, $\\top$ does not mean \"transpose\" here and causes confusion to readers.\n\nQUESTIONS:\nPlease see the questions in the \"weaknesses\" part.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages generative flow networks (GFlowNets) to obtain diverse solution candidates for combinatorial optimization problems on graphs without expert supervision.\n\nGFlowNets were recently introduced as a way to sample structured objects $\\mathbf{x}$ with a likelihood $P(\\mathbf{x}) \\propto R(\\mathbf{x})$ that is proportional to a terminal reward function (see _Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation_, Bengio et al. 2021).\nThis is done by casting the sampling as a Markov Decision Process, whose policy is defined by neural networks and trained with a suitable loss to enforce flow balance conditions.\n\nIn the study at hand, the reward is chosen as a Gibbs probability distribution $R(\\mathbf{x}) = \\exp(\\mathcal{E}(\\mathbf{x}) / T)$, whose temperature controls solution diversity by specifying the acceptable deviation from the global optimum.\nParametric policies are constructed conditionally on the graph, to allow for generalization.\nMDPs are designed in a problem-specific manner, by iteratively constructing sets of vertices that satisfy certain constraints.\nSince the MDP trajectories can be quite long, two tricks are suggested to improve training efficiency and credit assignment:\n- sample random subsets of transitions instead of a full trajectory\n- incorporate intermediate reward signals based on partial solutions\n\nThese strategies are implemented and benchmarked against exact solvers, as well as recent ML-based approaches.\n\nSTRENGTHS:\n### Originality\n\nThis paper mixes several existing ideas into a coherent whole:\n\n- GFlowNets and their loss functions\n- viewing optimization as probabilistic inference\n- MDPs for combinatorial problems\n- strategies for credit assignment\n\nThis unseen combination, along with extensive numerical experiments, is sufficient novelty in my view.\n\n### Quality\n\nThe authors demonstrate deep knowledge of the literature on GFlowNets, and the theoretical part is well supported.\nThorough benchmarks suggest their method performs better than its counterparts, although some comparisons may be biased (see below).\nI especially appreciated the care given to make the benchmarks transparent and fair (aligning with recent papers, avoiding easy instances, reimplementing and retraining algorithms from scratch to compare them on the same basis). \nAdditionally, a very welcome ablation study shows the limited impact of hyperparameter settings.\nIn a nutshell, this is good, serious science.\n\n### Clarity\n\nThe writing style is clear and nicely complemented by explanatory pictures.\nA lot of time is devoted to the prerequisites on GFlowNets and combinatorial problems, as well as a thorough literature review. \n\n### Significance\n\nI believe this contribution can inspire further developments, since the use of GFlowNets for combinatorial optimization seems well-justified and practically successful.\n\nWEAKNESSES:\n### Originality\n\nTo me, it appears that the main novel idea in the paper is the application of GFlowNets to learning combinatorial problems (which is already a worthy one).\nThe idea of training from subsampled transitions instead of a full trajectory is also new.\n\nHowever, I would be very surprised if the MDPs presented by the authors were absent from the previous literature, especially given the standard nature of problems such as Maximum Independent Set.\nPerhaps there are some necessary adjustments that are necessary for these MDPs to work with GFlowNets, in which case I will welcome clarification! \n\n### Quality\n\nI am not confident enough to comment on the choice of algorithms in the MIS benchmark, but I have an issue with the 3 non-MIS benchmarks, which seem a bit unfair to me.\nA major asset of GFlowNets is their sequential construction of the solution, which the authors mention as the reason for their success.\nOn the other hand, the benchmarked competitors ERDOS and ANNEAL rely on a hypothesis of independence between vertices of the graph, which explains their fast inference.\nIn light of this, I argue that it would make sense to benchmark GFlowNets against other methods (typically from the RL literature) that also adopt a sequential perspective, to see if the advantage still holds.\n\n### Clarity\n\nn.a.\n\n### Significance\n\nTo underline the impact of the paper, it would be helpful to explain why a diverse set of solution candidates is important in practice.\nAs a combinatorial optimizer myself, it is not obvious to me: when should I settle for a set of diverse, but possibly suboptimal solutions?\n\nQUESTIONS:\nL61: What is new in your MDP design compared to existing formulations?\n\nL105: Where do the final rewards appear?\n\nL242: I thought the whole point was to make epochs faster?\n\nL255: Are there cases where this reward continuation is not applicable? \n\nL640: Why not use a linear formulation? What formulation did you use for the three non-MIS problems?\n\nLIMITATIONS:\nThe authors do not really discuss the limitations of their approach, and more reflection on that would be welcome.\n\nSocietal impact is not relevant in this case.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach for using GFlowNets for solving some NP-Hard combinatorial optimization problems on graphs. The GFlowNet approach is designed to better perform credit assignment for fitting a constructive policy that adds nodes on the graph to a currently considered subset. The authors propose slight modifications of GFlowNet that help it perform well on graph optimization tasks which require long trajectories and improved credit assignment. The authors justify their design decisions empirically via ablation studies. Finally, the authors evaluate the proposed approach against supervised learning, unsupervised learning, heuristics, and traditional OR approaches where applicable. They evaluate on a variety of problems considering max independent set, max clique, minimum dominating set, and max cut. They demonstrate improved solution quality over the investigated baselines and mostly improved runtime over OR baselines. Overall, the paper is a promising initial attempt at applying GFlowNets to solve CO problems on graphs.\n\nSTRENGTHS:\nThe main strength of the paper is in addressing limitations of previous RL for CO approaches using GFlowNet. In previous RL-based optimization approaches, the issue of credit assignment is a major challenge as often individual choices have limited immediate impact on the overall solution quality while the long-term impact can be quite large. GFlowNet as an approach aims to perform this credit assignment well as it aims to learn a policy based on a long-term reward. \n\nA second strength is that the authors evaluate the given approach against a variety of baselines on a number of settings and present results on all settings. \n\nLastly, the writing is quite clear, and the paper gives a useful introduction of how to use GFlowNet for CO problems which can likely lead to future work of employing GFlowNet for broader types of CO problems.\n\nWEAKNESSES:\nThe main weakness is that the main contribution of the paper is that it is mainly directly applying GFlowNet to CO on graphs with minor modifications of the GFlowNet approach, and justifying their approach with intuition and empirical results. It might be good to either identify 1) specific ways that the GFlowNet methodology might be adapted to be specially suited to CO problems, or 2) ways in which the particularities of GFlowNet might be especially suited for answering interesting questions in CO problems. \n\nSome ideas for 1) it may be helpful to formulate the architecture used in GFlowNet to encode certain properties that help solvers such as being invariant to permutation of the solution generation procedure, special actions for GFlowNet such as deleting components of the current solution. Other ideas might be to modify GFlowNet to make use of alternative information such as supervision from known solutions on the training dataset, precomputed primal solutions with objective values, or integrating solver components like presolve etc.\n\nFor 2) it might be helpful to showcase the ability to generate diverse solutions by evaluating the diversity of solutions generated by the GFlowNet in terms of how many unique solutions it produces. Alternatively, one could generate a solution pool and using that to warm start solvers or generate solutions for multi-objective optimization problems. \n\nOverall, it would be interesting to see how the GFlowNet approach could be used outside of improving solve time.\n\n\nWhile the empirical results are promising, it is somewhat unfair to compare the given heuristic methods only against gurobi as an exact solver, since gurobi may be spending a lot of time proving optimality in addition to finding the optimal solution. It would be more reasonable to compare against gurobi which is tuned for quickly finding primal solutions and which is set at a very strict time limit, since gurobi may be finding the optimal solution quickly and taking most of its time proving optimality. Additionally, gurobi seems to be solving faster than the proposed approach in the MDS setting.\n\nQUESTIONS:\n[Paragraph starting at line 44] One listed downside of RL in CO is that RL may not produce diverse solutions. It would be helpful to demonstrate how this approach addresses that for instance by demonstrating that the GFlowNet approach does generate diverse solutions compared to the RL approach. For instance, Gurobi is able to return a pool of solutions. It would be interesting to compare the solution diversity between approaches.\n\nFor the forward looking technique, would it be possible to use the objective value of a “completed solution” which finds the optimal solution given the partial solution as fixed? Does this need to be a differentiable function of the solution?\n\nIt would be helpful to give the variance for the results to determine how close the distribution of results are since they are somewhat close to previous work. Additionally, it would be helpful to give some metrics such as win rate, i.e. the percent of instances for which the given approach was fastest / got the best quality solution. \n\n\nIt would also be helpful to explain the sizes of the train / test datasets and how they were split especially in the case of SATLIB.\n\nIt is unclear how this approach might handle CO problems where feasibility is “difficult”. For instance, for TSP on non-complete graphs it may be hard to find a feasible tour. It would be helpful to explain how this method might approach settings where feasibility is nontrivial via constructive methods. Here it may be difficult to sample feasible solutions in the first place and as such it may be helpful to understand how this method would perform. Similarly, it would be helpful to explain how this method would perform in settings where the decisions are more complex than selecting a subset of vertices. This is somewhat hinted at in the paragraph starting on line 195 but it would be helpful to explain this limitation and how it might be addressed.\n\n\nSmall questions:\n[line 29] Why is gurobi considered as giving approximate solutions? It not only should give optimal solutions given enough runtime but should also give a bound on the solution quality in order to prove optimality.\n\n[line 258] handcraft -> handcrafted\nTable 1, Time of Ours in Large has a leading 0 before the 4:22\n\nLIMITATIONS:\nRelevant limitations are addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}