Title: BQ-NCO: Bisimulation Quotienting for Efficient Neural Combinatorial Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Combinatorial Optimization

Markov Decision Processes

Bisimulation

Policy Learning

Out-of-Distribution Generalization

Routing Problems

TSP

CVRP

KP.
Abstract: Despite the success of neural-based combinatorial optimization methods for end-to-end heuristic learning, out-of-distribution generalization remains a challenge. In this paper, we present a novel formulation of Combinatorial Optimization Problems (COPs) as Markov Decision Processes (MDPs) that effectively leverages common symmetries of COPs to improve out-of-distribution robustness. Starting from a direct MDP formulation of a constructive method, we introduce a generic way to reduce the state space, based on Bisimulation Quotienting (BQ) in MDPs. Then, for COPs with a recursive nature, we specialize the bisimulation and show how the reduced state exploits the symmetries of these problems and facilitates MDP solving. Our approach is principled and we prove that an optimal policy for the proposed BQ-MDP actually solves the associated COPs. We illustrate our approach on five classical problems: the Euclidean and Asymmetric Traveling Salesman, Capacitated Vehicle Routing, Orienteering and Knapsack Problems. Furthermore, for each problem, we introduce a simple attention-based policy network for the BQ-MDPs, which we train by imitation of (near) optimal solutions of small instances from a single distribution. We obtain new state-of-the-art results for the five COPs on both synthetic and realistic benchmarks. Notably, in contrast to most existing neural approaches, our learned policies show excellent generalization performance to much larger instances than seen during training, without any additional search procedure. Our code is available at: [link](https://github.com/naver/bq-nco).

BQ-NCO: Bisimulation Quotienting for Efficient Neural Combinatorial Optimization

Abstract

Despite the success of neural-based combinatorial optimization methods for end-toend heuristic learning, out-of-distribution generalization remains a challenge. In this paper, we present a novel formulation of Combinatorial Optimization Problems (COPs) as Markov Decision Processes (MDPs) that effectively leverages common symmetries of COPs to improve out-of-distribution robustness. Starting from a direct MDP formulation of a constructive method, we introduce a generic way to reduce the state space, based on Bisimulation Quotienting (BQ) in MDPs. Then, for COPs with a recursive nature, we specialize the bisimulation and show how the reduced state exploits the symmetries of these problems and facilitates MDP solving. Our approach is principled and we prove that an optimal policy for the proposed BQ-MDP actually solves the associated COPs. We illustrate our approach on five classical problems: the Euclidean and Asymmetric Traveling Salesman, Capacitated Vehicle Routing, Orienteering and Knapsack Problems. Furthermore, for each problem, we introduce a simple attention-based policy network for the BQMDPs, which we train by imitation of (near) optimal solutions of small instances from a single distribution. We obtain new state-of-the-art results for the five COPs on both synthetic and realistic benchmarks. Notably, in contrast to most existing neural approaches, our learned policies show excellent generalization performance to much larger instances than seen during training, without any additional search procedure. Our code is available at: url.

1 Introduction

Combinatorial Optimization Problems (COPs) are crucial in many application domains such as transportation, energy, logistics, etc. Because they are generally NP-hard [12], their resolution at real-life scales is mainly done by problem-specific heuristics, which heavily rely on expert knowledge. Neural combinatorial optimization (NCO) is a relatively recent line of research that focuses on using deep neural networks to learn such heuristics from data, possibly exploiting regularities in problem instances of interest [5, 9]. Among NCO methods, the so-called constructive approaches view the process of building a solution incrementally as a sequential decision making problem, which can naturally be modeled with Markov Decision Processes (MDPs). Although many previous works have successfully used this strategy, the process of formulating an appropriate MDP (esp. the state and action spaces) is usually specific to each problem (e.g. [28] for routing problems, [49] for jobshop scheduling and [20] for bin packing). These design choices have a considerable impact when solving the MDP. In particular, exploiting the COP's symmetries can boost the efficiency and generalization of neural solvers (e.g. [29, 24] leverage the symmetries of routing problems in Euclidean graphs).

In this paper, we present a generic and principled framework to cast any COP as an MDP and then a way to account for common COPs' symmetries to design a more efficient MDP. More precisely,

given a user-defined solution space , we show how to automatically derive a direct MDP , where the states are partial solutions and the actions are the construction steps. While the application of this framework to some COPs encompasses some previously proposed MDPs (e.g [28], for routing problems), to the best of our knowledge, we are the first to provide a way to derive an MDP for any COP and prove the equivalence between the optimal MDP policies and solving the COP.

Next, noting the limitations of using the partial solutions as states, we introduce a generic bisimulation mapping that allows to reduce the state space. In particular, we show that for problems that satisfy a recursion property , the bisimulation simply maps a partial solution to a new (induced) instance, which corresponds to the remaining subproblem when the partial solution is fixed. As many partial solutions can induce the same instance, the resulting bisimulation quotiented (BQ-)MDP has a significantly smaller state space. Also, it enables more efficient learning by avoiding to independently learn the policy at states which are equivalent for the COP at hand. In contrast to previous works ([29, 24]), the symmetries that we exploit are not linked to the Euclidean nature of the graphs but to the recursive property of the problems, which is very common in CO as it includes the Optimality Principle of Dynamic Programming [3, 6].

We illustrate our framework on five well-known COPs: the Euclidean and Asymmetric Traveling Salesman Problems (TSP, ATSP), the Capacitated Vehicle Routing Problem (CVRP), the Orienteering Problem (OP) and the Knapsack Problem (KP). Furthermore, we propose a simple transformerbased architecture [43] for these problems, well-suited to the BQ-MDPs, and requiring only minor variations to work for the five problems. For each problem, we train our policy by imitation of expert trajectories derived from (near) optimal solutions of small instances sampled from a single distribution. We test on both synthetic and realistic benchmarks of varying size and node distributions. Our model provides new state-of-the-art results on the five problems and exhibits excellent outof-distribution generalization, especially to larger instances, a well-recognized challenge for NCO methods [22, 32]. Notably with a single greedy rollout, our learned policies outperform state-of-theart end-to-end learning-based approaches for instances with more than 200 nodes. We show that we can further improve the performance with a beam-search (using more computation) while we can significantly speed-up the execution (at the cost of a slight performance drop) by replacing the quadratic transformer model by a linear attention-based model, the PerceiverIO [19].

In summary, our contributions are as follows: 1) We present a generic and principled framework to derive a direct MDP given any COP with minimal requirements; 2) We propose a method to reduce the direct MDPs via symmetry-focused bisimulation quotienting and define an explicit bisimulation for the class of recursive COPs; 3) We design an adequate transformer-based architecture for the BQMDPs, with only minor adaptations to work for the TSP (both Euclidean and asymmetric versions), CVRP, OP and KP; 4) We achieve state-of-the-art generalization performance on these five problems, significantly out-performing other neural-based constructive methods.

2 Combinatorial Optimization as a Markov Decision Problem

In this section, we propose a generic framework to represent any Combinatorial Optimization Problem (COP) as a Markov Decision Process (MDP) amenable to standard Machine Learning techniques. We denote a COP instance by:

where X is the finite, non-empty set of feasible solutions , while the objective f is a real-valued function whose domain contains X . The complexity of CO is due to the cardinality of X , which, although finite, is generally exponential in the problem size. Constructive approaches to CO build a solution sequentially by growing a partial solution at each step. A required assumption of constructive heuristics (although often left implicit) is that the feasibility of the final solution can be ensured through tractable conditions on the partial solutions at each step of the construction process.

2.1 Solution Space

Let us denote by X the set of all possible partial solutions which can be obtained by a given construction procedure. We assume that X is equipped with an operation ◦ having a neutral element ϵ . Informally, ϵ is the 'empty' partial solution and if x, y are partial solutions, then x ◦ y denotes the result of applying the sequence of construction steps yielding x followed by that yielding y . We

denote by Z⊂X the subset of partial solutions obtained from ϵ by just one construction step. By identification, we call the elements of Z steps and assume that any partial (a fortiori feasible) solution can be obtained as the composition of a sequence of steps. We can now define the structure of the solution space as required by our framework.

Definition 1 (Solution Space) . A solution space is a tuple ( X , ◦ , ϵ, Z ) where ( X , ◦ , ϵ ) forms a monoid (see Appendix G.1 for background), and the step set Z⊂X\{ ϵ } is a generator of X , such that any element of X has a finite positive number of step decompositions:

Thus, solving a COP instance ( f, X ) given a solution space X containing X amounts to finding a sequence of steps z 1: n such that z 1 ◦ · · · ◦ z n ∈ arg min x ∈ X f ( x ) , a task which can naturally be modeled as an MDP, as shown below.

Examples of Solution Spaces. We provide examples of solution spaces for two classical COPs. In the Euclidean Traveling Salesman Problem (TSP), an instance is defined by a set of nodes (including a depot) associated with points in a Euclidean space and the goal is to find the shortest tour that starts and ends at the depot and visits each other node exactly once. A partial solution for the TSP can be represented by a sequence of nodes. Therefore we can define X TSP as the set of finite sequences of nodes. In the Knapsack Problem (KP), an instance is defined by a set of items associated with weight and value features, and the goal is to select a subset of these items such that the sum of their weights does not exceed a given capacity while their cumulated value is maximized. We can define X KP as the set of finite sets of items. For TSP (resp. KP), the operator ◦ is sequence concatenation (resp. set disjoint union), the neutral element ϵ is the empty sequence (resp. set) and a step is a sequence (resp. set) of length 1. Note that these problems admit other (e.g. graph-based rather than set or sequence-based) solution spaces. In fact, a solution space is not intrinsic to a problem, nor vice-versa.

2.2 The Direct MDP

Given a solution space ( X , ◦ , ϵ, Z ) , we consider the set F X of instances ( f, X ) where the set of feasible solutions X is a finite, non-empty subset of X , and the objective function f ∈ R X is defined on X rather than just X , i.e. it is well-defined for any partial solution . Now, given an instance ( f, X ) ∈F X , we can derive its direct MDP, denoted M ( f,X ) , as follows. Its state space is the set ¯ X = { x ∈X : ∃ y ∈X , x ◦ y ∈ X } of partial solutions which can potentially be expanded into a feasible one. Its action space is Z∪{ ϵ } , i.e. an action is either a step or the neutral action. Its transitions are deterministic, and can be represented as a labeled transition system with the following two rules, where the label of each transition consists of its action-reward pair, placed respectively above and below the transition arrow:

Here, x ∈ ¯ X is a state and z ∈Z is a step. The condition in each rule determines whether the action is allowed. Thanks to the structure of the solution space captured by Def. 1, the direct MDP of an instance ( f, X ) has three key properties, proved in Appendix F.1. (1) From any state, the number of allowed actions is finite (even if Z is infinite), therefore M ( f,X ) belongs to the simpler class of discrete-action MDPs. (2) From any state, there is always at least one allowed action (i.e. there are no dead-end states). This assumes that one can guarantee if a partial solution can be expanded into a feasible one, as required of the valid states in ¯ X . This is a common assumption of constructive heuristics (often left implicit) and it avoids the complications associated with dead ends in MDPs [27]. (3) In any infinite trajectory in M ( f,X ) , the number of transitions involving a step action is finite, while all the other transitions involve the neutral action. Since the neutral action yields a null reward, this means that the return of a trajectory is well defined, without having to discount the rewards. Also, if a 1: ∞ is the (infinite) sequence of actions of the trajectory, since all but finitely many of them are neutral actions, their composition a 1 ◦ a 2 ◦· · · is well defined and is called the outcome of the trajectory. We can now establish the main result of this section, proved in Appendix F.2:

Proposition 1 (Soundness of the Direct MDP) . Given a solution space X and an instance ( f, X ) ∈F X , let M ( f,X ) be its direct MDP. The set arg min x ∈ X f ( x ) is exactly the set of x such that there exists an optimal policy π for M ( f,X ) where x is the outcome of a trajectory starting at ϵ under policy π .

This result states the exact correspondence between the optimal solutions of a COP instance and the optimal policies of its direct MDP. Thus, the vast corpus of techniques developed to search for optimal policies in MDPs in general is applicable to solving COPs. The direct MDP encompasses many previously proposed MDPs, where the state is a partial solution and each action consists of adding an element to the partial solution, e.g. [23] for graph problems and [7] for the TSP.

3 Bisimulation Quotienting

3.1 State information and symmetries

Let ( X , ◦ , ϵ, Z ) be a solution space and M ( f,X ) the direct MDP of an instance ( f, X ) ∈F X . In a trajectory of M ( f,X ) , observe that each non neutral action z 'grows' the state from x to x ◦ z , as defined by (2). For many COPs, this is counter-intuitive, because it implies that the state information increases while the number of allowed actions generally decreases. For example, in the TSP, the state contains the sequence of visited nodes, which grows at each step, while the allowed actions are the set of unvisited nodes, which shrinks at each step. At the end, the state may carry the most information (a full-grown feasible solution), but it is not used in any decision, since there is a single allowed action anyway (the neutral action). To address this mismatch between the information carried by the state and the complexity of the decision it supports, we seek to define a new state which captures only the information needed for the continuation of the construction process. To do so, we observe that a partial solution y ∈X can be seen both as an operator on partial solutions x ↦→ x ◦ y which grows its operand, or, alternatively, as the following operator on instances , which reduces its operand:

In fact, ( f ∗ y, X ∗ y ) is the so-called the tail subproblem [6] of instance ( f, X ) after partial solution y has been constructed. We observe that for a given tail subproblem ( f ′ , X ′ ) ∈F X , there can be many pairs of an instance ( f, X ) ∈F X and partial solution x ∈ ¯ X such that ( f ∗ x, X ∗ x )=( f ′ , X ′ ) . For example in the TSP, the tail subproblem consists of finding the shortest path from the last node e of the partial solution back to the depot o of the instance, which visits the set I of not-yet-visited nodes. Therefore we can see that any TSP instance with the same depot o and any node sequence ending at e with not-yet-visited nodes I will lead to the same tail subproblem. This is a strong symmetry with respect to the MDP policy: intuitively it means that an optimal policy should produce the same action for these infinitely many (instance-partial solution) pairs. Treating them as distinct states, as in the direct MDP, forces a training procedure to learn the underlying symmetry in order to map them into a common representation.

3.2 Bisimulation Quotiented MDP

In order to leverage the above 'reduction operator' view and its potential symmetry, we define the following 'reduced' MDP M . Its state space is the set F X of instances. Its action space is Z∪{ ϵ } , as in the direct MDP. Its transitions are also deterministic, and can be expressed with two rules, dual of those in (2):

̸

This MDP is defined at the level of the whole solution space rather than an individual instance. Now, the mapping Φ ( f,X ) : ¯ X ↦→F X from the direct to the reduced states defined by Φ ( f,X ) ( x )=( f ∗ x, X ∗ x ) for all x ∈ ¯ X is a bisimulation between M ( f,X ) and M (background in Appendix G.2 and a proof in Appendix F.3). Formally, M is isomorphic to the quotient of the direct MDP by the bisimulation (precise statement and proof in Appendix F.3), hence is called the Bisimulation Quotiented (BQ-)MDP. By the bisimulation property, for any direct state x , the action-reward sequences spawned from x in M ( f,X ) and from Φ ( f,X ) ( x ) in M are identical. Therefore, there is a one-to-one correspondence between the trajectories of M ( f,X ) starting at ϵ and those of M starting at Φ ( f,X ) ( ϵ )=( f, X ) . Hence, the analogue of Prop. 1 holds for the BQ-MDP:

Proposition 2 (Soundness of the BQ-MDP) . Let M be the BQ-MDP of a solution space X , and ( f, X ) ∈F X an instance. The set arg min x ∈ X f ( x ) is exactly the set of x such that there exists an optimal policy π for M where x is the outcome of a trajectory starting at ( f, X ) under policy π .

Impact on model architecture. Although both the direct and BQ-MDPs are equivalent in terms of solving their associated COP, their practical interpretation leads to major differences. In the direct MDP view, it would not make sense to learn each instance-specific MDP separately. Instead a generic MDP conditioned on an input instance is learnt, similar to goal-conditioned Reinforcement Learning [40, 31] (the goal is here the input instance). A typical policy model architecture consists of an encoder in charge of computing an embedding of the input instance and a decoder that takes the instance embedding and the current partial solution to compute the next action (Fig. 1 left), e.g. the Attention Model [28] or PointerNetworks [45]. In the rollout of a trajectory, the encoder needs only be invoked once since the instance does not change throughout the rollout. For the BQ-MDP, only one, unconditional MDP is learnt for the whole solution space. The model can be simpler since the distinction between encoder and decoder vanishes (Fig. 1 right). On the other hand, the whole model must be applied to a new input instance at each step of a rollout.

encoder

decoder

z

t

ϵ

x

t

-

1

x

0

=

ϵ

x

t

=

x

t

-

1

◦

z

t

¬

ϵ

t

=0

agent

environment

(

f, X

)

⋆

model

z

t

ϵ

(

f

t

-

1

, X

t

-

1

)

f

t

=

f

t

-

1

∗

z

t

X

t

=

X

t

-

1

∗

z

t

¬

ϵ

agent

environment

(

f, X

)

t

=0

⋆

3.3 Instance parametrization and recursion

For a given COP, instances are generally described by a set of parameters, which are used as input to the policy network. In the BQ-MDP, the instance is updated at each step z , according to the equations X ′ = X ∗ z and f ′ = f ∗ z . In order to implement the BQ-MDP, a key requirement is that ( f ′ , X ′ ) can be represented in the same parametric space as ( f, X ) . In fact, this is the case for COPs that satisfy the tail-recursion property : after applying a number of construction steps to an instance, the remaining tail subproblem is itself an instance of the original COP. This is a very common property in CO and includes in particular the Optimality Principle of Dynamic Programming [3, 6]: all problems that are amenable to dynamic programming satisfy the tail-recursion property. For these tail-recursive COPs, the bisimulation simply maps a partial solution to the tail subproblem instance it induces.

Application to the KP and path-TSP. One can see that the KP naturally satisfies the tail-recursion property. Consider an instance of the KP with capacity c and items ( w i , v i ) i ∈ I described by their weight w i and value v i . A partial solution is a subset J ⊂ I of items. In this case, the bisimulation Φ maps J to a new KP instance with capacity c -∑ j ∈ J w j and item set ( w i , v i ) i ∈ I \ J (see Appendix C). For the TSP, we introduce a slightly more general problem, path-TSP, where the goal is to find a shortest path between an origin and a destination node visiting a set of customer nodes, instead of a tour. The tail subproblem for a partial path x 1: k consists of finding the shortest path from x k to the destination that visits all the remaining nodes, hence it is also a path-TSP instance. Thus path-TSP is tail-recursive, and TSP is simply a sub-problem of path-TSP where origin and destination are associated with the same Euclidean point (the depot). A similar reasoning holds for CVRP and OP, leading to path-CVRP and path-OP (see in Appendix A and B).

4 Policy Learning

We now describe our proposed transformer-based policy network for the BQ-MDPs. For simplicity, we focus here on the path-TSP; the models for path-CVRP, path-OP and KP differ only slightly and are presented in Appendix A, B and C.

Neural architecture. For the Euclidean path-TSP (Fig. 2 left), the Cartesian coordinates of each node (including origin and destination) are embedded via a linear layer. The remainder of the model is based on Vaswani et al. [43] with the following differences. First, we do not use positional encoding since the input nodes have no order. Instead, a learnable origin (resp. destination) encoding vector is added to the feature embedding of the origin (resp. destination) node. Second, we use ReZero [2]

(

x

o

, y

o

)

(

x

d

, y

d

)

(

x

1

, y

1

)

· · ·

(

x

n

, y

n

)

Node features: Cartesian coordinates

input embedding layer

Transformer layer

activation=ReLu

normalization=ReZero

L

×

linear layer

softmax

orig.

enc.

dest.

enc.

+

+

r

o

r

d

r

1

· · ·

r

n

Node feature: random identifier

input embedding layer

Graph convolution layer

weights=normalized travel costs between nodes

Transformer layer

activation=ReLu

normalization=ReZero

L

×

linear layer

softmax

orig.

enc.

dest.

enc.

+

+

normalization, which leads to more stable training and better performance in our experiments. Finally, a linear layer projects the output of the last attention block into a vector of size N , from which unfeasible actions, corresponding to the origin and destination nodes, are masked out, before applying a softmax operator so as to interpret the scalar node values for all allowed nodes as action probabilities. Note the absence of autoregressive decoder layers. In the asymmetric TSP, node coordinates are not given, but the travelling cost between nodes is provided through a (possibly asymmetric) matrix. The policy model for path-ATSP (Fig. 2 right) differs slightly from that for path-TSP. Initially, a randomly generated identifier is assigned to each node, which is embedded by a linear layer. The cost matrix is then incorporated by using a weighted graph convolution operation in each layer of the model before applying the attention mechanism. The edge weights of the graph are obtained by some standard normalization of the cost matrix. The use of random node identifiers as model input is justified in ATSP by the absence of node specific features. In problems where such features are available, a random identifier can still be added as extra node feature, as that often improves performance.

Trajectory generation. We train our model by imitation of expert trajectories, using a cross-entropy loss. Such trajectories are extracted from pre-computed (near) optimal solutions for instances of a relatively small and fixed size. Even for hard problems, we leverage the fact that it is generally possible to efficiently solve glyph[tie] small instances (e.g. using MILP solvers). Of course the quality of the learned policy will be assessed on larger and therefore more challenging instances. Note that optimal solutions are not directly in the form of trajectories, i.e. sequences of construction steps. While Prop. 2 guarantees that a trajectory exists for any solution, it is usually not unique. In the TSP, an optimal tour corresponds to two possible trajectories (one being the reverse of the other). In the CVRP, each subtour similarly corresponds to two possible trajectories, and in addition, the different orders of the subtours lead to different trajectories. We noticed that this final order has an impact on the performance (see Appendix E.3). On the other hand, any sub-sequence from a fixed trajectory solution can be interpreted as the solution of a sub-instance. Note that these sub-instances will vary both in size and node distribution, therefore by training on them, we implicitly encourage the model to work well across sizes and node distributions, and generalize better than if such variations were not seen during the training.

Complexity. Because of the quadratic complexity of self-attention, and the fact that we call our policy at each construction step, the total complexity of our model, for an instance of size N , is O ( N 3 ) , whereas closely related transformer-based models such as the TransformerTSP [7] and the Attention Model [28] have a total complexity of O ( N 2 ) . We experimentally show in Sec. 6 that our model still provides better-quality solutions faster than these baselines. This is because most neural models are called many times per instance in practice, typically for sampling from the learned policy or within a beam search. In contrast, we observe that doing a single rollout of our policy yields excellent results. Intuitively, given the complexity (NP-hardness) of the targeted COPs, it makes sense to spend as much computation as possible at each time step, provided the whole rollout is fast enough. On the other hand, the quadratic attention bottleneck can be addressed by using some linear attention models. In our experiments, we show that replacing the Transformer by the linear PerceiverIO [19] significantly accelerates inference while still providing a competitive performance.

5 Related Work

Generic frameworks for CO. Generic frameworks have been crucial in the development and adoption of CO since they allow to formulate a COP in a certain format and then give it as an input to a (black box) solver. Among the most widespread frameworks are Mixed-Integer Programming [11] and Constraint Programming [39]. On the other hand, Markov Decision Processes are a powerful and ubiquitous tool to model sequential decision making problems and make them amenable to Machine Learning. While using MDPs to solve CO is not new, there are very few attempts to make it a general framework. Khalil et al. [23] makes a first attempt to define a generic MDP for greedy heuristics on graph problems. Drori et al. [13] also focuses on graph problems and provides a flexible and efficient GNN-based architecture to train with Reinforcement Learning (RL). Unlike these methods, our approach does not assume a graph or any structure on the COPs to define the (direct and BQ) MDP and we are the first to prove the equivalence between our proposed MDPs and solving the COP.

Neural-based constructive heuristics. Many NCO approaches construct solutions sequentially, via auto-regressive models. Starting with the seminal work by Vinyals et al. [45], which proposed the Pointer network trained in a supervised way, Bello et al. [4] trained the same model by RL for the TSP and Nazari et al. [34] adapted it for the CVRP. Kool et al. [28] introduced an Attention-based encoderdecoder Model (AM) trained with RL to solve several routing problems. This architecture was reused, with a few extensions, in POMO [29] and further combined with efficient search procedures in [10]. TransformerTSP [7] uses a similar architecture with a different decoder for the TSP. While most of these auto-regressive constructive approaches used RL, another line of work focuses on learning, in a supervised way, a heatmap of solution segments: Nowak et al. [35] trained a Graph Neural Network to output an adjacency matrix, which is converted into a feasible solution using beam search; Joshi et al. [21] followed a similar framework and trained a Graph Convolutional Network instead, that was used by [14] as well. Recently Sun and Yang [42] proposed DIFUSCO, a graph-based diffusion model to output the heatmap. Although such non-autoregressive methods are promising for large instances since they avoid the sequential decoding, they generally rely on sophisticated search methods (such as Monte Carlo Tree Search) in order to get a high-quality solution from the heatmap. Our paper explores the less common combination of supervised (imitation) learning to train a policy (not a heatmap) and achieves excellent performance without any search. In the context of planning, a similar strategy was successfully used in [16] to learn generalized reactive policies on small planning problems instances while generalizing well to larger ones.

Step-wise approaches. Instead of encoding an instance once and decoding sequentially as in the AM[28], Peng et al. [36] proposed to update the encoding after each subtour completion for CVRP while Xin et al. [47] updated it after each node selection in TSP. Xin et al. [46] extended on this idea by introducing the Multi-Decoder Attention Model (MDAM) which contains a special layer to efficiently approximate the re-embedding process. Since MDAM constitutes the most advanced version, we employ it as a baseline in our experiments.

Generalizable NCO. Generalization to different instance distributions, and esp. larger instances, is regarded as one of the major challenges for current NCO approaches [22, 33]. Fu et al. [14] trained a Graph Convolution model in a supervised manner on small graphs and used it to solve large TSP instances, by applying the model on sampled subgraphs and using an expensive MCTS search to improve the final solution (Att-GCN+MCTS). While this method achieves excellent generalization on TSP instances, MCTS requires a lot of computing resources and is essentially a post-learning search strategy. Geisler et al. [15] investigate the robustness of NCO solvers through adversarial attacks and find that existing neural solvers are highly non-robust to out-of-distribution examples and conclude that one way to address this issue is through adversarial training. In particular, Xin et al. [48] trains a GANto generate instances that are difficult to solve for the current model. Manchanda et al. [32], Son et al. [41], Zhou et al. [50] take a different approach and leverage meta-learning to learn a model that is easily adaptable to new distributions. In DIMES, Qiu et al. [38] combines meta-learning with a continuous parametrization of the candidate solutions that allows to solve large-scale TSP instances. Accounting for symmetries in a given COP is a powerful idea to boost the generalization performance of neural solvers. For example, for problems on Euclidean graphs, Sym-NCO [24] make use of solution symmetries, to enforce robust representations while [29] exploits it as part of their loss function during training and at inference time to augment the set of solutions. Note that the above adversarial or meta-learning strategies and data augmentation are orthogonal to our approach of providing an efficient MDP which has beneficial effects irrespective of the training method. Finally,

some hybrid methods have been successful in solving large-scale routing problems [30, 51, 18]; these ate based on the divide-and-conquer principle: they learn how to split large instances into smaller ones, that are then efficiently solved by external specialized solvers.

6 Experimental Evaluation

To evaluate the effectiveness of our method, we test it on four routing problems - Euclidean TSP, CVRP, OP and Asymmetric TSP and one non-routing problem - KP (details and results for the latter are in Appendix C). For all routing problems, we train our model and all baselines on synthetic instances of size 100, following the same procedure of data generation as in [28]. For asymmetric TSP, we use Euclidean distance matrices and randomly break symmetries. We choose graphs of size 100 because it is the largest size for which (near) optimal solutions are still reasonably fast to obtain, and such training datasets are commonly used in the literature. Then, we test the trained models on synthetic instances of size 100, 200, 500 and 1K generated from the same distribution, as well as the standard TSPLib and CVRPLib datasets. Except for perceiver-based models and OP (dist), we limit the subgraphs to the 250 nearest neighbours of the origin node at test time. This reduces inference time while not significantly impacting the solution quality (see Appendix D).

Hyperparameters and training. We use the same hyperparameters for all problems. The model has 9 layers, each built with 12 attention heads with embedding size of 192 and dimension of feed-forward layer of 512. Our model is trained by imitation of expert trajectories, using a cross-entropy loss. Solutions of these problems are obtained by using the Concorde solver [1] for TSP, LKH heuristic [17] for ATSP and CVRP, and EA4OP heuristic [26] for OP. We use a dataset of 1 million solutions. To sample trajectories out of this dataset, we note that in the case of TSP, any sub-path of the optimal tour is also an optimal solution to the associated path-TSP sub-problem, hence amenable to our path-TSP model. We therefore form minibatches by first sampling a number n between 4 and N (path-TSP problems with less than 4 nodes are trivial), then sampling sub-paths of length n from the initial solution set. At each epoch, we sample a sub-path from each solution. By sampling subsequences among all possible infixes of the optimal solutions, we automatically get an augmented dataset, which some previous models had to explicitly design their model for [29]. We use a similar sampling strategy for CVRP, OP and KP (see Appendix .A, B and C). Batches of size 1024 are formed, and the model is trained for 500 epochs. We use Adam [25] as optimizer with an initial learning rate of 7 . 5e -4 and decay of 0.98 every 50 epochs.

Baselines and test datasets. We compare our model with the following methods: OR-Tools [37], LKH [17], and Hybrid Genetic Search (HGS) for the CVRP [44] as SOTA non-neural methods; DIFUSCO+2opt, Att-GCN+MCTS and SGBS as hybrid methods; and AM, TransformerTSP, MDAM, POMO, DIMES and Sym-NCO as deep learning-based constructive methods. Description of all baseline models are provided in Sec. 5. For all deep learning baselines we use the corresponding model trained on graphs of size 100 and the best decoding strategy 2 . Following the same procedure as in [14], we generate four test datasets with graphs of sizes 100, 200, 500 and 1000. For CVRP, we use capacities of 50, 80, 100 and 250, respectively. In addition, we report the results on TSPLib instances with up to 4461 nodes and all CVRPLib instances with node coordinates in the Euclidean space. For all models, we report the optimality gap and the inference time. The optimality gap for TSP is based on the optimal solutions obtained with Concorde. For CVRP, although HGS gives better results than LKH, we use the LKH solutions as a reference to compute the "optimality" gap, in order to be consistent (and easily comparable) with previous works. For the OP task, we train and test the model using a "distance" variant of the problem, as described in [28]. For fixed maximum lengths, we chose T 100 = 4 , T 200 = 5 , T 500 = 6 , and T 1000 = 7 . While the optimality gap is easy to compute and compare, running times are harder to compare since they strongly depend on the implementation platforms (Python, C++), hardware (GPU, CPU), parallelization, batch size, etc. In our experiments, we run all deep learning models on a single Nvidia Tesla V100-S GPU with 24GB memory, and other solvers on Intel(R) Xeon(R) CPU E5-2670 with 256GB memory, in one thread.

Results. Table 1 summarizes our results for Euclidean TSP, CVRP and OP as well as asymmetric TSP. For all problems, our model shows superior generalization on larger graphs, even with a

greedy decoding strategy, which generates a single solution while all other baselines generate several hundreds (and select the best among them). In terms of running time, with greedy decoding, our model is slightly slower that Sym-NCO baseline, competitive with POMO, and significantly faster than other models. Beam search further improves the optimality gaps, but as expected, it takes more time. On the other hand, our model can be accelerated by replacing the quadratic attention blocks by the PerceiverIO architecture [19] (more details in Appendix E.1). This results in a considerable reduction of inference time for larger graphs, at the cost of some performance drop. Even with this trade-off, our model achieves remarkable performance compared with other NCO baselines. Figure 3 shows optimality gap versus running time for our models and other baseline models. Our models clearly outperform other models in terms of generalization to larger instances. The only models that are competitive with ours are Att-GCN+MCTS and DIFUSCO+2opt, but both are hybrid methods with an expensive search on top of the output of the neural network, and they are 2 to 15 times slower while being designed for TSP only. Finally, in addition to the synthetic datasets, we test our models on TSPLib and VRPLib instances, which are of varying graph sizes, node distributions, demand distributions and vehicle capacities. Table 1 shows that our model significantly outperforms the end-to-end baseline policies even with the greedy decoding strategy.

10 -3

10 -2

10 -1

10 0

10 1

10 2

Inference -ime per ins-ance (s)

0

5

10

15

20

25

30

35

40

Optimality gap (%)

SymNCO

SGBS

POMO

Att-GCN+MCTS

DIMES

DIFUSCO

OURS transf. (G)

OURS transf. (BS)

OURS perc.

TSP 100

TSP 200

TSP 500

TSP 1000

10 -2

10 -1

10 0

10 1

Inference -ime per ins-ance (s)

0

20

40

60

80

100

120

140

Optimality gap (%)

AM

MDAM

POMO

OURS transf. (G)

OURS transf. (BS)

OURS perc.

CVRP 100

CVRP 200

CVRP 500

CVRP 1000

7 Conclusion

We proposed a principled framework to formulate COPs as efficient symmetry-invariant MDPs. First, we presented how to derive a direct MDP for any COP that is amenable to a constructive approach, and proved the exact correspondence between the optimal policies of this direct MDP and the optimal solutions of the COP it stems from. Second, for the widespread class of recursive COPs, we introduced bisimulation quotienting to leverage the symmetries of the partial solutions. We showed that the resulting BQ-MDP has a significantly reduced state space while its optimal policies still exactly solve the original COP. We then designed a simple Transformer-based architecture as an examplification of a possible policy network for the BQ-MDPs, and applied it to five classical COP problems: (A)TSP, CVRP, OP and KP. The resulting policy generalizes particularly well, and significantly outperforms not only similar end-to-end neural-based constructive solvers but also hybrid methods that combine learning with search. A limitation of our approach, shared with all constructive methods, is that it can only handle problems for which feasibility is easy to ensure throughout the construction process. In order to address problems with more challenging global constraints, approaches which combine learning and constraint programming (e.g. [8]) may be a promising direction. While training on relatively small instances allowed us to use imitation learning, our proposed MDPs and the associated model could also be trained with Reinforcement Learning. This would be particularly interesting to extend our framework from deterministic to stochastic COPs.

All (15-1K)

22.36%

15.58%

7.62%

4.83

%

References

David Applegate, Robert Bixby, Vasek Chvatal, and William Cook. Concorde TSP solver, 2015.

Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: Fast convergence at large depth. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , pages 1352-1361. PMLR, December 2021.

Richard Bellman. The Theory of Dynamic Programming. Technical Report P-550, The Rand Corporation, Santa Monica, CA, U.S.A., July 1954.

Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural Combinatorial Optimization with Reinforcement Learning. arXiv:1611.09940 [cs, stat] , January 2017.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Dimitri Bertsekas. Dynamic Programming and Optimal Control: Volume I . Athena Scientific, 2012. ISBN 978-1-886529-43-4.

Xavier Bresson and Thomas Laurent. The Transformer Network for the Traveling Salesman Problem. arXiv:2103.03012 [cs] , March 2021.

Quentin Cappart, Thierry Moisan, Louis-Martin Rousseau, Isabeau Prémont-Schwarz, and Andre Cire. Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization. arXiv:2006.01610 [cs] , June 2020.

Quentin Cappart, Didier Chételat, Elias B. Khalil, Andrea Lodi, Christopher Morris, and Petar Veliˇ ckovi´ c. Combinatorial Optimization and Reasoning with Graph Neural Networks. In TwentyNinth International Joint Conference on Artificial Intelligence , volume 5, pages 4348-4355, August 2021.

Jinho Choo, Yeong-Dae Kwon, Jihoon Kim, Jeongwoo Jae, Andre Hottung, Kevin Tierney, and Youngjune Gwon. Simulation-guided Beam Search for Neural Combinatorial Optimization. In NeurIPS 2022 , 2022.

Michele Conforti, Gérard Cornuéjols, and Giacomo Zambelli. Integer Programming , volume 271 of Graduate Texts in Mathematics . Springer International Publishing, Cham, 2014. ISBN 978-3-319-38432-0.

William J. Cook, William H. Cunningham, William R. Pulleyblank, and Alexander Schrijver. Combinatorial Optimization . Wiley-Interscience, New York, 1st edition edition, November 1997. ISBN 978-0-471-55894-1.

Iddo Drori, Anant Kharkar, William R. Sickinger, Brandon Kates, Qiang Ma, Suwen Ge, Eden Dolev, Brenda Dietrich, David P. Williamson, and Madeleine Udell. Learning to Solve Combinatorial Optimization Problems on Real-World Graphs in Linear Time, June 2020.

Zhang-Hua Fu, Kai-Bin Qiu, and Hongyuan Zha. Generalize a Small Pre-trained Model to Arbitrarily Large TSP Instances. Proceedings of the AAAI Conference on Artificial Intelligence , 35(8):7474-7482, May 2021. ISSN 2374-3468.

Simon Geisler, Johanna Sommer, Jan Schuchardt, Aleksandar Bojchevski, and Stephan Günnemann. Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness. In Advances in Neural Information Processing Systems , December 2022.

Edward Groshev, Maxwell Goldstein, Aviv Tamar, Siddharth Srivastava, and Pieter Abbeel. Learning Generalized Reactive Policies Using Deep Neural Networks. Proceedings of the International Conference on Automated Planning and Scheduling , 28:408-416, June 2018. ISSN 2334-0843.

Keld Helsgaun. An Extension of the Lin-Kernighan-Helsgaun TSP Solver for Constrained Traveling Salesman and Vehicle Routing Problems. Technical report, Roskilde University, Roskilde, Denmark, 2017.

Qingchun Hou, Jingwei Yang, Yiqiang Su, Xiaoqing Wang, and Yuming Deng. Generalize Learned Heuristics to Solve Large-scale Vehicle Routing Problems in Real-time. In The Eleventh International Conference on Learning Representations , September 2022.

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A General Architecture for Structured Inputs & Outputs. In International Conference on Learning Representations , January 2022.

Yuan Jiang, Zhiguang Cao, and Jie Zhang. Solving 3D Bin Packing Problem via Multimodal Deep Reinforcement Learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS '21, pages 1548-1550, Richland, SC, May 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-8307-3.

Chaitanya K. Joshi, Thomas Laurent, and Xavier Bresson. An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem. arXiv:1906.01227 [cs, stat] , June 2019.

Chaitanya K. Joshi, Quentin Cappart, Louis-Martin Rousseau, and Thomas Laurent. Learning the travelling salesperson problem requires rethinking generalization. Constraints , 27(1-2): 70-98, April 2022. ISSN 1383-7133.

Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning Combinatorial Optimization Algorithms over Graphs. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 6348-6358. Curran Associates, Inc., 2017.

Minsu Kim, Junyoung Park, and Jinkyoo Park. Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization. In Advances in Neural Information Processing Systems , New Orleans, December 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR) , San Diego, May 2015.

Gorka Kobeaga, María Merino, and Jose A. Lozano. An efficient evolutionary algorithm for the orienteering problem. Computers & Operations Research , 90:42-59, February 2018. ISSN 0305-0548.

Andrey Kolobov, Mausam, and Daniel S. Weld. A theory of goal-oriented MDPs with dead ends. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence , UAI'12, pages 438-447, Arlington, Virginia, USA, August 2012. AUAI Press. ISBN 978-09749039-8-9.

Wouter Kool, Herke van Hoof, and Max Welling. Attention, Learn to Solve Routing Problems! In International Conference on Learning Representations , 2019.

Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. POMO: Policy Optimization with Multiple Optima for Reinforcement Learning. In Advances in Neural Information Processing Systems , volume 33, pages 21188-21198. Curran Associates, Inc., 2020.

Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. In Advances in Neural Information Processing Systems 34 Pre-Proceedings , 2021.

Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-Conditioned Reinforcement Learning: Problems and Solutions. In Thirty-First International Joint Conference on Artificial Intelligence , volume 6, pages 5502-5511, July 2022.

Sahil Manchanda, Sofia Michel, Darko Drakulic, and Jean-Marc Andreoli. On the Generalization of Neural Combinatorial Optimization Heuristics. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , Grenoble, France, September 2022.

Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400, October 2021. ISSN 0305-0548.

MohammadReza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takac. Reinforcement Learning for Solving the Vehicle Routing Problem. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Alex Nowak, David Folqué, and Joan Bruna. Divide and Conquer Networks. In 6-Th International Conference on Learning Representations , Vancouver, Canada, April 2018.

Bo Peng, Jiahai Wang, and Zizhen Zhang. A Deep Reinforcement Learning Algorithm Using Dynamic Attention Model for Vehicle Routing Problems. In Kangshun Li, Wei Li, Hui Wang, and Yong Liu, editors, Artificial Intelligence Algorithms and Applications , Communications in Computer and Information Science, pages 636-650, Singapore, 2020. Springer. ISBN 9789811555770.

Laurent Perron and Vincent Furnon. OR-tools, 2022.

Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. DIMES: A Differentiable Meta Solver for Combinatorial Optimization Problems. In Advances in Neural Information Processing Systems , December 2022.

Francesca Rossi, Peter van Beek, and Toby Walsh. Handbook of Constraint Programming . Elsevier Science Inc., USA, July 2006. ISBN 978-0-08-046380-3.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal Value Function Approximators. In Proceedings of the 32nd International Conference on Machine Learning , pages 1312-1320. PMLR, June 2015.

Jiwoo Son, Minsu Kim, Hyeonah Kim, and Jinkyoo Park. Meta-SAGE: Scale Meta-Learning Scheduled Adaptation with Guided Exploration for Mitigating Scale Shift on Combinatorial Optimization, June 2023.

Zhiqing Sun and Yiming Yang. DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization, February 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Thibaut Vidal. Hybrid genetic search for the CVRP: Open-source implementation and SWAP* neighborhood. Computers & Operations Research , 140:105643, April 2022. ISSN 0305-0548.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer Networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28 , pages 2692-2700. Curran Associates, Inc., 2015.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems. Proceedings of the AAAI Conference on Artificial Intelligence , 35(13):12042-12049, May 2021. ISSN 2374-3468.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Step-Wise Deep Learning Models for Solving Routing Problems. IEEE Transactions on Industrial Informatics , 17(7):4861-4871, July 2021. ISSN 1941-0050.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Generative Adversarial Training for Neural Combinatorial Optimization Models, January 2022.

Cong Zhang, Wen Song, Zhiguang Cao, Jie Zhang, Puay Siew Tan, and Chi Xu. Learning to dispatch for job shop scheduling via deep reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, pages 16211632, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-954-6.

Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards Omni-generalizable Neural Methods for Vehicle Routing Problems, June 2023.

Zefang Zong, Hansen Wang, Jingwei Wang, Meng Zheng, and Yong Li. RBG: Hierarchically Solving Large-Scale Routing Problems in Logistic Systems via Reinforcement Learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4648-4658, Washington DC USA, August 2022. ACM. ISBN 978-1-4503-9385-0.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on Bisimulation Quotienting (BQ), this paper proposed a new MDP formulation named BQ-MDP, to better formalise the process of solution construction in neural constructive solvers for CO problems. The proposed BQ-MDP reduces the state space and is designed to better reflect the symmetry of sub-problems encountered during solution construction. A new deep model named BQ-Transformer (based on Transformer) is designed according to the BQ-MDP, which is leveraged to solve CO problems including TSP, CVRP, and KP. The deep model is trained via supervised learning and the results showed better generalization performance of the learned model across problem sizes and distributions compared to several baselines.\n\nSTRENGTHS:\n- The introduction of BQ into MDP formulations for neural constructive solvers is novel and interesting.\n- While several works like POMO (NeurIPS-20) [1], Sym-NCO (NeurIPS-22) [2] and Pointformer (AAAI-23) [3] have tried to explore the symmetries of CO problems, the idea of exploiting the symmetries during the recursive construction process is novel.\n- The proposed model showed impressive generalization performance across size and distribution, outperforming several baselines.\n- The exploration of BQ-PerceiverG to reduce computational complexity.\n\nWEAKNESSES:\n- The clarity of the paper, particularly the mathematical formulations in Section 2.1 and Section 3.1, is limited. Examples include but are not limited to: the concept of the \"step set Z\" being “a set of partial solutions obtained by one construction step” is unclear, as it is not explicitly stated whether this step is based on an empty solution or an existing partial solution; It is said that “if x, y are partial solutions, x◦y denotes the result of applying the sequence of construction steps yielding x followed by that yielding y”, but I did not follow this definition. If x = z_1◦ · · · ◦ z_n,  does it means that z_i are partial solutions?\n\n- The novelty (about the model architecture) may be limited. The proposed architecture seems to share similarities to an earlier exploration in [4] which also suggests re-encoding the sub-problems during solution construction. Furthermore, in the model designs, the claim about the elimination of the autoregressive decoder may not be compelling. Essentially, the proposed model, similar to AM and POMO, also requires multiple calls to select nodes for solution construction. And the proposed model still follows the encoder-decoder structure, as the last linear layer (used for generating the probability of selecting nodes via softmax) is the decoder.\n\n- The model is trained using imitation learning, which necessitates labeled data and pre-existing solvers to generate high-quality solutions. It is thus not clear if the credits for better performance should be given to the imitation learning/labeled data, or the proposed new architecture.\n\n- The experimental comparisons raise several concerns which make the advantages of the proposed method unclear:\n1. On TSP-100 and CVRP-100 (without generalization), the BQ-Transformer performs worse than POMO despite using a longer runtime, particularly on CVRP. The BQ-Transformer seems to struggle with handling VRP constraints effectively. \n2. Furthermore, when BQ-Transformer is equipped with beam search, there is no comparison made with POMO under similar settings, e.g., the POMO + beam search as proposed in the SGBS paper (NeurIPS2022) [5]. \n3. Are the test datasets the same as DIEMS and DIFUSCO? How many instances are in the test datasets? The reported performance gaps of DIMES and DIFUSCO may be inaccurate, as I could not locate the reported gaps in their papers. \n4. Since the models are trained on size 100, in Table 1, it would be clearer to label TSP200, TSP500, and TSP1000 as performance under generalization, rather than mixing them up, which could lead to confusion.\n5. While the proposed model exhibits better generalization on larger-scale instances, it is unclear whether the model has the ability to further reduce the objective values (gaps) given a longer time budget, and how good the performance would be when compared to baselines with longer run time.\n\nReferences: \n\n[1] Kwon, Yeong-Dae, et al. \"Pomo: Policy optimization with multiple optima for reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 21188-21198.\n\n[2] Kim, Minsu, Junyoung Park, and Jinkyoo Park. \"Sym-nco: Leveraging symmetricity for neural combinatorial optimization.\" Advances in Neural Information Processing Systems 35 (2022): 1936-1949.\n\n[3] Jin, Yan, et al. \"Pointerformer: Deep Reinforced Multi-Pointer Transformer for the Traveling Salesman Problem.\" arXiv preprint arXiv:2304.09407 (2023).\n\n[4] Xin, Liang, et al. \"Step-wise deep learning models for solving routing problems.\" IEEE Transactions on Industrial Informatics 17.7 (2020): 4861-4871.\n\n[5]  Choo, Jinho, et al. \"Simulation-guided beam search for neural combinatorial optimization.\" Advances in Neural Information Processing Systems 35 (2022): 8760-8772.\n\nQUESTIONS:\n- Are the test datasets the same as DIEMS and DIFUSCO? How many instances are in the test datasets? Why have the objective values, in addition to the gaps, not been reported in Table 1?\n\n- What are the principal differences between the proposed model architecture and the one in reference [4]? Can we apply [4] to reflect the features of BQ-MDP?\n\n- The claim about the generality of BQ-MDP seems to be overstated. Are there certain Combinatorial Optimization Problems (COPs) that the MDP might struggle to handle? For instance, how would the MDP manage problems where the feasibility of the solution can only be determined when the solution is complete, as in some Simulation Optimization tasks?\n\n- Can the authors clarify the mathematical notations mentioned above? It may be beneficial to include examples (for instance, toy TSP-5 instances) to illustrate the new concepts introduced.\n\n- I could not find the correct Appendix; the current zip file contains the submitted manuscript for ICLR23, not the NeurIPS appendix.\n\nLIMITATIONS:\nThe limitation mentioned in the weakness above should be discussed in the revised paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use the notion of bisimulation quotienting (BQ) to define state spaces with the minimal information that is required to solve a combinatorial optimization (sub)problem. This results in a simpler architecture for the model used in training. Empirically, the methods advance state-of-the-art performance on a couple of routing problems.\n\nSTRENGTHS:\nThe paper rigorously formulates BQ-MDP for CO problems. I find this method applicable to a wide range of CO problems whenever it could be formulated as dynamic programming.\n\nThe empirical results show that the method can generalize to larger instances than training. \n\nThe paper includes many baselines, including some recent ones e.g. DIFFUSCO. \n\nI also like the idea of testing the method with a lightweight version of transformers.\n\nWEAKNESSES:\nThis method is limited to problems where finding feasible solutions are easy since it is mainly a construction heuristic.  Other problems like 3-SAT  (decision version of the Satisfiability problem, as opposed to MAX-SAT) that have a much constrained solution space might not be suitable for this method, though the BQ-MDP formulation is still applicable.\n\nFor people who are familiar with dynamic programming (DP), the idea behind the approach is pretty straightforward — In DP, it is crucial to identify the minimal state representation to solve a subproblem to reduce complexity. This paper formulates this idea in a more mathematical fashion.  \n\nI find this method applicable to a wide range of CO problems whenever it could be formulated as dynamic programming. But the authors didn’t emphasize this enough — it simply states it as a generic framework for COP. It is not obvious to readers how widely this method is applicable to COP. \n\n\nIn experiments, it would be nice to also report variances. The benchmarks used are mostly in euclidean space, which gives the readers an impression that this might only work for this case. Are there non-euclidean routing problem benchmark that you could test your approach on?\n\nIt seems that the method is not restricted to a particular learning algorithm or architecture. Imitation learning is just a way to go, but it wasn’t clear in the paper. In other words, would other learning methods like RL also be applicable?\n\nQUESTIONS:\nSee the weaknesses section.\n\nThis paper seems relevant to this work: https://arxiv.org/pdf/2006.01610.pdf\nMight be good to discuss its relevance in the paper.\n\nIs the method more sample efficient that the other works? Intuitively, this should be one of the benefits because you are reducing states that are essentially the same but with different representations of partial solutions.\n\nLIMITATIONS:\nI don't see any limitations other than those pointed out already.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a generic framework for deriving direct MDPs for a given combinatorial optimization problem (COP) and propose a method for reducing the MDP using symmetry-focused bisimulation quotienting. Specifically, the paper presents a generic bisimulation that can be applied to a wide range of recursive problems. The paper presents an instantiation of the approach to three well known problems: TSP, CVRP, and Knapsack using a simple Transformer-based architecture with minor adaptation per application. The experiments show that the proposed approach obtains significantly better performance in generalizing to larger problems.\n\nSTRENGTHS:\n- Novel and interesting approach to enhance neural combinatorial optimization problem by exploiting the recursive nature of many COPs.\n- The proposed technical approach seems to be relatively generic and can potentially be applied to a range of combinatorial optimization problems.\n- Experiments show significant improvement in generalizing to larger problems compared to the baselines.\n\nWEAKNESSES:\n- Experiments in the paper are limited to three benchmarks domains. While these are important and somewhat diverse domains, previous work has also considered combinatorial optimization problems like maximum cut, maximum independent set, etc. Further, as TSP and CVRP are highly related and the results for KP show somewhat different performance patterns, it is not clear how well this approach works beyond the limited set of benchmarks. Specifically, the results on KP that are only presented in the appendix show more limited improvement over POMO compared to TSP and CVRP (no other baselines). Taking into account that POMO is a relatively poor baseline on TSP/CVRP in terms of generalizing to larger instances, this raises the concern that the proposed method may not perform as well on domains beyond TSP/CVRP.\n\n - Comparison with previous symmetry-related enhancements such as Sym-NCO is missing. While the authors note that such approaches are \"orthogonal to our approach\", it is still important to evaluate the benefit of the proposed approach in presence of these augmentations.\n\n- The approach still requires optimal solutions for training. Such optimal solutions are typically intractable to compute beyond certain problem size. While the proposed approach shows very good performance in generalizing to larger problems, there may be a limit to the scalability (the paper presents results for up to one order of magnitude larger test problems). It is interesting to see if this can be paired with RL fine-tuning to further improve performance on larger instances.\n\nQUESTIONS:\nI would appreciate the authors' response to my main concerns listed above.\n\nLIMITATIONS:\nThere is no explicit discussion of limitation, however I have no concerns about the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper first present a formulation of Combinatorial Optimization Problems (COPs) as Markov Decision Processes (MDPs) that can leverage symmetries of COPs to improve out-of-distribution robustness. Based on the MDP formulation, the authors then introduce the Bisimulation Quotienting to the MDP, namely BQ-MDP, which can reduce the state space. Finally, the authors introduce a simple attention-based policy network to learn policies in the BQ-MDP. Experiments demonstrate that the proposed method achieves the state-of-the-art performance on both  synthetic and realistic benchmarks.

STRENGTHS:
1. To the best of my knowledge, the idea of introducing the Bisimulation Quotienting (BQ) to exploit symmetries of COPs is novel.  
2. Experiments demonstrate that the proposed method achieves the state-of-the-art performance on both  synthetic and realistic benchmarks. Moreover, the method achieves impressive generalization performance to much larger instances than seen during training.

WEAKNESSES:
The presentation could be improved. 
1. The authors claim that their formulation is generic for any CO problems. However, the formulation seems specific for the binary integer linear programming problems. Thus, it would be more convincing if the authors could explain how the proposed formulation applies to production planning problems with both continuous and discrete variables. 
2. The details about the proposed method are unclear to me. For example, the authors may want to explain why and how introducing Bisimulation Quotiented can exploit the symmetries of COPs. Moreover, the details about the impact of the BQ-MDP on model architecture is unclear.

QUESTIONS:
Please refer to Weaknesses for my questions.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06