SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Darko Drakulic, Sofia Michel, Florian Mai* Arnaud Sors, Jean-Marc Andreoli
Naver Labs Europe
firstname.lastname@naverlabs.com
Despite the success of neural-based combinatorial optimization methods for end-to-end heuristic learning, out-of-distribution generalization remains a challenge. In this paper, we present a novel formulation of Combinatorial Optimization Problems (COPs) as Markov Decision Processes (MDPs) that effectively leverages common symmetries of COPs to improve out-of-distribution robustness. Starting from a direct MDP formulation of a constructive method, we introduce a generic way to reduce the state space, based on Bisimulation Quotienting (BQ) in MDPs. Then, for COPs with a recursive nature, we specialize the bisimulation and show how the reduced state exploits the symmetries of these problems and facilitates MDP solving. Our approach is principled and we prove that an optimal policy for the proposed BQ-MDP actually solves the associated COPs. We illustrate our approach on five classical problems: the Euclidean and Asymmetric Traveling Salesman, Capacitated Vehicle Routing, Orienteering and Knapsack Problems. Furthermore, for each problem, we introduce a simple attention-based policy network for the BQ-MDPs, which we train by imitation of (near) optimal solutions of small instances from a single distribution. We obtain new state-of-the-art results for the five COPs on both synthetic and realistic benchmarks. Notably, in contrast to most existing neural approaches, our learned policies show excellent generalization performance to much larger instances than seen during training, without any additional search procedure. Our code is available at: url.
Combinatorial Optimization Problems (COPs) are crucial in many application domains such as transportation, energy, logistics, etc. Because they are generally NP-hard [12], their resolution at real-life scales is mainly done by problem-specific heuristics, which heavily rely on expert knowledge. Neural combinatorial optimization (NCO) is a relatively recent line of research that focuses on using deep neural networks to learn such heuristics from data, possibly exploiting regularities in problem instances of interest [5, 9]. Among NCO methods, the so-called constructive approaches view the process of building a solution incrementally as a sequential decision making problem, which can naturally be modeled with Markov Decision Processes (MDPs). Although many previous works have successfully used this strategy, the process of formulating an appropriate MDP (esp. the state and action spaces) is usually specific to each problem (e.g. [28] for routing problems, [49] for jobshop scheduling and [20] for bin packing). These design choices have a considerable impact when solving the MDP. In particular, exploiting the COP's symmetries can boost the efficiency and generalization of neural solvers (e.g. [29, 24] leverage the symmetries of routing problems in Euclidean graphs).
In this paper, we present a generic and principled framework to cast any COP as an MDP and then a way to account for common COPs' symmetries to design a more efficient MDP. More precisely,
*IDIAP Research Institute, Work done during an internship at Naver Labs Europe
given a user-defined solution space, we show how to automatically derive a direct MDP, where the states are partial solutions and the actions are the construction steps. While the application of this framework to some COPs encompasses some previously proposed MDPs (e.g [28], for routing problems), to the best of our knowledge, we are the first to provide a way to derive an MDP for any COP and prove the equivalence between the optimal MDP policies and solving the COP.
Next, noting the limitations of using the partial solutions as states, we introduce a generic bisimulation mapping that allows to reduce the state space. In particular, we show that for problems that satisfy a recursion property, the bisimulation simply maps a partial solution to a new (induced) instance, which corresponds to the remaining subproblem when the partial solution is fixed. As many partial solutions can induce the same instance, the resulting bisimulation quotiented (BQ-)MDP has a significantly smaller state space. Also, it enables more efficient learning by avoiding to independently learn the policy at states which are equivalent for the COP at hand. In contrast to previous works ([29, 24]), the symmetries that we exploit are not linked to the Euclidean nature of the graphs but to the recursive property of the problems, which is very common in CO as it includes the Optimality Principle of Dynamic Programming [3, 6].
We illustrate our framework on five well-known COPs: the Euclidean and Asymmetric Traveling Salesman Problems (TSP, ATSP), the Capacitated Vehicle Routing Problem (CVRP), the Orienteering Problem (OP) and the Knapsack Problem (KP). Furthermore, we propose a simple transformer-based architecture [43] for these problems, well-suited to the BQ-MDPs, and requiring only minor variations to work for the five problems. For each problem, we train our policy by imitation of expert trajectories derived from (near) optimal solutions of small instances sampled from a single distribution. We test on both synthetic and realistic benchmarks of varying size and node distributions. Our model provides new state-of-the-art results on the five problems and exhibits excellent out-of-distribution generalization, especially to larger instances, a well-recognized challenge for NCO methods [22, 32]. Notably with a single greedy rollout, our learned policies outperform state-of-the-art end-to-end learning-based approaches for instances with more than 200 nodes. We show that we can further improve the performance with a beam-search (using more computation) while we can significantly speed-up the execution (at the cost of a slight performance drop) by replacing the quadratic transformer model by a linear attention-based model, the PerceiverIO [19].
In summary, our contributions are as follows: 1) We present a generic and principled framework to derive a direct MDP given any COP with minimal requirements; 2) We propose a method to reduce the direct MDPs via symmetry-focused bisimulation quotienting and define an explicit bisimulation for the class of recursive COPs; 3) We design an adequate transformer-based architecture for the BQ-MDPs, with only minor adaptations to work for the TSP (both Euclidean and asymmetric versions), CVRP, OP and KP; 4) We achieve state-of-the-art generalization performance on these five problems, significantly out-performing other neural-based constructive methods.
2 Combinatorial Optimization as a Markov Decision Problem
In this section, we propose a generic framework to represent any Combinatorial Optimization Problem (COP) as a Markov Decision Process (MDP) amenable to standard Machine Learning techniques. We denote a COP instance by:
$$
\min  _ {x \in X} f (x),
$$
where  $X$  is the finite, non-empty set of feasible solutions, while the objective  $f$  is a real-valued function whose domain contains  $X$ . The complexity of CO is due to the cardinality of  $X$ , which, although finite, is generally exponential in the problem size. Constructive approaches to CO build a solution sequentially by growing a partial solution at each step. A required assumption of constructive heuristics (although often left implicit) is that the feasibility of the final solution can be ensured through tractable conditions on the partial solutions at each step of the construction process.
2.1 Solution Space
Let us denote by  $\mathcal{X}$  the set of all possible partial solutions which can be obtained by a given construction procedure. We assume that  $\mathcal{X}$  is equipped with an operation  $\circ$  having a neutral element  $\epsilon$ . Informally,  $\epsilon$  is the "empty" partial solution and if  $x, y$  are partial solutions, then  $x \circ y$  denotes the result of applying the sequence of construction steps yielding  $x$  followed by that yielding  $y$ . We
denote by  $\mathcal{Z} \subset \mathcal{X}$  the subset of partial solutions obtained from  $\epsilon$  by just one construction step. By identification, we call the elements of  $\mathcal{Z}$  steps and assume that any partial (a fortiori feasible) solution can be obtained as the composition of a sequence of steps. We can now define the structure of the solution space as required by our framework.
Definition 1 (Solution Space). A solution space is a tuple  $(\mathcal{X},\circ ,\epsilon ,\mathcal{Z})$  where  $(\mathcal{X},\circ ,\epsilon)$  forms a monoid (see Appendix G.1 for background), and the step set  $\mathcal{Z}\subset \mathcal{X}\backslash \{\epsilon \}$  is a generator of  $\mathcal{X}$ , such that any element of  $\mathcal{X}$  has a finite positive number of step decompositions:
$$
\forall x \in \mathcal {X}, 0 <   \left| \left\{z _ {1: n} \in \mathcal {Z} ^ {n}: x = z _ {1} \circ \dots \circ z _ {n} \right\} \right| <   \infty . \tag {1}
$$
Thus, solving a COP instance  $(f,X)$  given a solution space  $\mathcal{X}$  containing  $X$  amounts to finding a sequence of steps  $z_{1:n}$  such that  $z_{1} \circ \dots \circ z_{n} \in \arg \min_{x \in X} f(x)$ , a task which can naturally be modeled as an MDP, as shown below.
Examples of Solution Spaces. We provide examples of solution spaces for two classical COPs. In the Euclidean Traveling Salesman Problem (TSP), an instance is defined by a set of nodes (including a depot) associated with points in a Euclidean space and the goal is to find the shortest tour that starts and ends at the depot and visits each other node exactly once. A partial solution for the TSP can be represented by a sequence of nodes. Therefore we can define  $\mathcal{X}_{\mathrm{TSP}}$  as the set of finite sequences of nodes. In the Knapsack Problem (KP), an instance is defined by a set of items associated with weight and value features, and the goal is to select a subset of these items such that the sum of their weights does not exceed a given capacity while their cumulated value is maximized. We can define  $\mathcal{X}_{\mathrm{KP}}$  as the set of finite sets of items. For TSP (resp. KP), the operator  $\circ$  is sequence concatenation (resp. set disjoint union), the neutral element  $\epsilon$  is the empty sequence (resp. set) and a step is a sequence (resp. set) of length 1. Note that these problems admit other (e.g. graph-based rather than set or sequence-based) solution spaces. In fact, a solution space is not intrinsic to a problem, nor vice-versa.
2.2 The Direct MDP
Given a solution space  $(\mathcal{X},\circ ,\epsilon ,\mathcal{Z})$  , we consider the set  $\mathcal{F}_{\mathcal{X}}$  of instances  $(f,X)$  where the set of feasible solutions  $X$  is a finite, non-empty subset of  $\mathcal{X}$  , and the objective function  $f\in \mathbb{R}^{\mathcal{X}}$  is defined on  $\mathcal{X}$  rather than just  $X$  , i.e. it is well-defined for any partial solution. Now, given an instance  $(f,X)\in \mathcal{F}_{\mathcal{X}}$  , we can derive its direct MDP, denoted  $\mathcal{M}_{(f,X)}$  , as follows. Its state space is the set  $\bar{X} = \{x\in \mathcal{X}\colon \exists y\in \mathcal{X}$ $x\circ y\in X\}$  of partial solutions which can potentially be expanded into a feasible one. Its action space is  $\mathcal{Z}\cup \{\epsilon \}$  , i.e. an action is either a step or the neutral action. Its transitions are deterministic, and can be represented as a labeled transition system with the following two rules, where the label of each transition consists of its action-reward pair, placed respectively above and below the transition arrow:
$$
x \xrightarrow [ f (x) - f (x \circ z) ]{z} x \circ z \text {i f} x \circ z \in \bar {X} \quad \text {a n d} \quad x \xrightarrow [ 0 ]{\epsilon} x \text {i f} x \in X. \tag {2}
$$
Here,  $x \in \bar{X}$  is a state and  $z \in \mathcal{Z}$  is a step. The condition in each rule determines whether the action is allowed. Thanks to the structure of the solution space captured by Def. 1, the direct MDP of an instance  $(f, X)$  has three key properties, proved in Appendix F.1. (1) From any state, the number of allowed actions is finite (even if  $\mathcal{Z}$  is infinite), therefore  $\mathcal{M}_{(f, X)}$  belongs to the simpler class of discrete-action MDPs. (2) From any state, there is always at least one allowed action (i.e. there are no dead-end states). This assumes that one can guarantee if a partial solution can be expanded into a feasible one, as required of the valid states in  $\bar{X}$ . This is a common assumption of constructive heuristics (often left implicit) and it avoids the complications associated with dead ends in MDPs [27]. (3) In any infinite trajectory in  $\mathcal{M}_{(f, X)}$ , the number of transitions involving a step action is finite, while all the other transitions involve the neutral action. Since the neutral action yields a null reward, this means that the return of a trajectory is well defined, without having to discount the rewards. Also, if  $a_{1: \infty}$  is the (infinite) sequence of actions of the trajectory, since all but finitely many of them are neutral actions, their composition  $a_1 \circ a_2 \circ \dots$  is well defined and is called the outcome of the trajectory. We can now establish the main result of this section, proved in Appendix F.2:
Proposition 1 (Soundness of the Direct MDP). Given a solution space  $\mathcal{X}$  and an instance  $(f,X)\in \mathcal{F}_{\mathcal{X}}$  let  $\mathcal{M}_{(f,X)}$  be its direct MDP. The set  $\arg \min_{x\in X}f(x)$  is exactly the set of  $x$  such that there exists an optimal policy  $\pi$  for  $\mathcal{M}_{(f,X)}$  where  $x$  is the outcome of a trajectory starting at  $\epsilon$  under policy  $\pi$ .
This result states the exact correspondence between the optimal solutions of a COP instance and the optimal policies of its direct MDP. Thus, the vast corpus of techniques developed to search for optimal policies in MDPs in general is applicable to solving COPs. The direct MDP encompasses many previously proposed MDPs, where the state is a partial solution and each action consists of adding an element to the partial solution, e.g. [23] for graph problems and [7] for the TSP.
3 Bisimulation Quotienting
3.1 State information and symmetries
Let  $(\mathcal{X},\circ ,\epsilon ,\mathcal{Z})$  be a solution space and  $\mathcal{M}_{(f,X)}$  the direct MDP of an instance  $(f,X)\in \mathcal{F}_{\mathcal{X}}$ . In a trajectory of  $\mathcal{M}_{(f,X)}$ , observe that each non-neutral action  $z$  "grows" the state from  $x$  to  $x\circ z$ , as defined by (2). For many COPs, this is counter-intuitive, because it implies that the state information increases while the number of allowed actions generally decreases. For example, in the TSP, the state contains the sequence of visited nodes, which grows at each step, while the allowed actions are the set of unvisited nodes, which shrinks at each step. At the end, the state may carry the most information (a full-grown feasible solution), but it is not used in any decision, since there is a single allowed action anyway (the neutral action). To address this mismatch between the information carried by the state and the complexity of the decision it supports, we seek to define a new state which captures only the information needed for the continuation of the construction process. To do so, we observe that a partial solution  $y\in \mathcal{X}$  can be seen both as an operator on partial solutions  $x\mapsto x\circ y$  which grows its operand, or, alternatively, as the following operator on instances, which reduces its operand:
$$
(f, X) \mapsto (f * y, X * y) \quad \text {w i t h :} \quad (f * y) (x) = f (y \circ x) \quad \text {a n d} \quad X * y = \{x | y \circ x \in X \}.
$$
In fact,  $(f*y,X*y)$  is the so-called the tail subproblem [6] of instance  $(f,X)$  after partial solution  $y$  has been constructed. We observe that for a given tail subproblem  $(f',X')\in \mathcal{F}_{\mathcal{X}}$ , there can be many pairs of an instance  $(f,X)\in \mathcal{F}_{\mathcal{X}}$  and partial solution  $x\in \bar{X}$  such that  $(f*x,X*x)=(f',X')$ . For example in the TSP, the tail subproblem consists of finding the shortest path from the last node  $e$  of the partial solution back to the depot  $o$  of the instance, which visits the set  $I$  of not-yet-visited nodes. Therefore we can see that any TSP instance with the same depot  $o$  and any node sequence ending at  $e$  with not-yet-visited nodes  $I$  will lead to the same tail subproblem. This is a strong symmetry with respect to the MDP policy: intuitively it means that an optimal policy should produce the same action for these infinitely many (instance-partial solution) pairs. Treating them as distinct states, as in the direct MDP, forces a training procedure to learn the underlying symmetry in order to map them into a common representation.
3.2 Bisimulation Quotiented MDP
In order to leverage the above "reduction operator" view and its potential symmetry, we define the following "reduced" MDP  $\mathcal{M}$ . Its state space is the set  $\mathcal{F}_{\mathcal{X}}$  of instances. Its action space is  $\mathcal{Z} \cup \{\epsilon\}$ , as in the direct MDP. Its transitions are also deterministic, and can be expressed with two rules, dual of those in (2):
$$
(f, X) \xrightarrow [ f (\epsilon) - f (z) ]{z} (f * z, X * z) \text {i f} X * z \neq \emptyset \quad \text {a n d} \quad (f, X) \xrightarrow [ 0 ]{\epsilon} (f, X) \text {i f} \epsilon \in X. \tag {3}
$$
This MDP is defined at the level of the whole solution space rather than an individual instance. Now, the mapping  $\Phi_{(f,X)}:\bar{X}\mapsto \mathcal{F}_{\mathcal{X}}$  from the direct to the reduced states defined by  $\Phi_{(f,X)}(x) = (f*x,X*x)$  for all  $x\in \bar{X}$  is a bisimulation between  $\mathcal{M}_{(f,X)}$  and  $\mathcal{M}$  (background in Appendix G.2 and a proof in Appendix F.3). Formally,  $\mathcal{M}$  is isomorphic to the quotient of the direct MDP by the bisimulation (precise statement and proof in Appendix F.3), hence is called the Bisimulation Quotiented (BQ-)MDP. By the bisimulation property, for any direct state  $x$ , the action-reward sequences spawned from  $x$  in  $\mathcal{M}_{(f,X)}$  and from  $\Phi_{(f,X)}(x)$  in  $\mathcal{M}$  are identical. Therefore, there is a one-to-one correspondence between the trajectories of  $\mathcal{M}_{(f,X)}$  starting at  $\epsilon$  and those of  $\mathcal{M}$  starting at  $\Phi_{(f,X)}(\epsilon) = (f,X)$ . Hence, the analogue of Prop. 1 holds for the BQ-MDP:
Proposition 2 (Soundness of the BQ-MDP). Let  $\mathcal{M}$  be the BQ-MDP of a solution space  $\mathcal{X}$ , and  $(f, X) \in \mathcal{F}_{\mathcal{X}}$  an instance. The set  $\arg \min_{x \in X} f(x)$  is exactly the set of  $x$  such that there exists an optimal policy  $\pi$  for  $\mathcal{M}$  where  $x$  is the outcome of a trajectory starting at  $(f, X)$  under policy  $\pi$ .
Impact on model architecture. Although both the direct and BQ-MDPs are equivalent in terms of solving their associated COP, their practical interpretation leads to major differences. In the direct MDP view, it would not make sense to learn each instance-specific MDP separately. Instead a generic MDP conditioned on an input instance is learnt, similar to goal-conditioned Reinforcement Learning [40, 31] (the goal is here the input instance). A typical policy model architecture consists of an encoder in charge of computing an embedding of the input instance and a decoder that takes the instance embedding and the current partial solution to compute the next action (Fig. 1 left), e.g. the Attention Model [28] or PointerNetworks [45]. In the rollout of a trajectory, the encoder needs only be invoked once since the instance does not change throughout the rollout. For the BQ-MDP, only one, unconditional MDP is learnt for the whole solution space. The model can be simpler since the distinction between encoder and decoder vanishes (Fig. 1 right). On the other hand, the whole model must be applied to a new input instance at each step of a rollout.
Figure 1: Policy model architectures for the direct MDP (left) and BQ-MDP (right).
3.3 Instance parametrization and recursion
For a given COP, instances are generally described by a set of parameters, which are used as input to the policy network. In the BQ-MDP, the instance is updated at each step  $z$ , according to the equations  $X' = X * z$  and  $f' = f * z$ . In order to implement the BQ-MDP, a key requirement is that  $(f', X')$  can be represented in the same parametric space as  $(f, X)$ . In fact, this is the case for COPs that satisfy the tail-recursion property: after applying a number of construction steps to an instance, the remaining tail subproblem is itself an instance of the original COP. This is a very common property in CO and includes in particular the Optimality Principle of Dynamic Programming [3, 6]: all problems that are amenable to dynamic programming satisfy the tail-recursion property. For these tail-recursive COPs, the bisimulation simply maps a partial solution to the tail subproblem instance it induces.
Application to the KP and path-TSP. One can see that the KP naturally satisfies the tail-recursion property. Consider an instance of the KP with capacity  $c$  and items  $(w_i, v_i)_{i \in I}$  described by their weight  $w_i$  and value  $v_i$ . A partial solution is a subset  $J \subset I$  of items. In this case, the bisimulation  $\Phi$  maps  $J$  to a new KP instance with capacity  $c - \sum_{j \in J} w_j$  and item set  $(w_i, v_i)_{i \in I \setminus J}$  (see Appendix C). For the TSP, we introduce a slightly more general problem, path-TSP, where the goal is to find a shortest path between an origin and a destination node visiting a set of customer nodes, instead of a tour. The tail subproblem for a partial path  $x_{1:k}$  consists of finding the shortest path from  $x_k$  to the destination that visits all the remaining nodes, hence it is also a path-TSP instance. Thus path-TSP is tail-recursive, and TSP is simply a sub-problem of path-TSP where origin and destination are associated with the same Euclidean point (the depot). A similar reasoning holds for CVRP and OP, leading to path-CVRP and path-OP (see in Appendix A and B).
4 Policy Learning
We now describe our proposed transformer-based policy network for the BQ-MDPs. For simplicity, we focus here on the path-TSP; the models for path-CVRP, path-OP and KP differ only slightly and are presented in Appendix A, B and C.
Neural architecture. For the Euclidean path-TSP (Fig. 2 left), the Cartesian coordinates of each node (including origin and destination) are embedded via a linear layer. The remainder of the model is based on Vaswani et al. [43] with the following differences. First, we do not use positional encoding since the input nodes have no order. Instead, a learnable origin (resp. destination) encoding vector is added to the feature embedding of the origin (resp. destination) node. Second, we use ReZero [2]
Figure 2: BQ-MDP policy model architecture for the TSP (left) and ATSP (right); origin in yellow, destination in red and the other nodes in blue.