SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Instantine Caramanis
UT Austin & Archimedes / Athena RC
constantine@utexas.edu
vkonton@gmail.com
UOA & Archimedes / Athena RC
Deep Neural Networks and Reinforcement Learning methods have empirically shown great promise in tackling challenging combinatorial problems. In those methods a deep neural network is used as a solution generator which is then trained by gradient-based methods (e.g., policy gradient) to successively obtain better solution distributions. In this work we introduce a novel theoretical framework for analyzing the effectiveness of such methods. We ask whether there exist generative models that (i) are expressive enough to generate approximately optimal solutions; (ii) have a tractable, i.e., polynomial in the size of the input, number of parameters; (iii) their optimization landscape is benign in the sense that it does not contain sub-optimal stationary points. Our main contribution is a positive answer to this question. Our result holds for a broad class of combinatorial problems including Max- and Min-Cut, Max- $k$ -CSP, Maximum-Weight-Bipartite-Matching, and the Traveling Salesman Problem. As a byproduct of our analysis we introduce a novel regularization process over vanilla gradient descent and provide theoretical and experimental evidence that it helps address vanishing-gradient issues and escape bad stationary points.
Gradient descent has proven remarkably effective for diverse optimization problems in neural networks. From the early days of neural networks, this has motivated their use for combinatorial optimization [HT85, Smi99, VFJ15, BPL $^{+}$ 16]. More recently, an approach by  $\left[\mathrm{BPL}^{+}16\right]$ , where a neural network is used to generate (sample) solutions for the combinatorial problem. The parameters of the neural network thus parameterize the space of distributions. This allows one to perform gradient steps in this distribution space. In several interesting settings, including the Traveling Salesman Problem, they have shown that this approach works remarkably well. Given the widespread application but also the notorious difficulty of combinatorial optimization [GLS12, PS98, S $^{+}$ 03, Sch05, CLS $^{+}$ 95], approaches that provide a more general solution framework are appealing.
This is the point of departure of this paper. We investigate whether gradient descent can succeed in a general setting that encompasses the problems studied in  $\left[\mathrm{BPL}^{+}16\right]$ . This requires a parameterization
of distributions over solutions with a "nice" optimization landscape (intuitively, that gradient descent does not get stuck in local minima or points of vanishing gradient) and that has a polynomial number of parameters. Satisfying both requirements simultaneously is non-trivial. As we show precisely below, a simple lifting to the exponential-size probability simplex on all solutions guarantees convexity; and, on the other hand, compressed parameterizations with "bad" optimization landscapes are also easy to come by (we give a natural example for Max-Cut in Remark 1). Hence, we seek to understand the parametric complexity of gradient-based methods, i.e., how many parameters suffice for a benign optimization landscape in the sense that it does not contain "bad" stationary points.
We thus theoretically investigate whether there exist solution generators with a tractable number of parameters that are also efficiently estimizable, i.e., gradient descent requires a small number of steps to reach a near-optimal solution. We provide a positive answer under general assumptions and specialize our results for several classes of hard and easy combinatorial optimization problems, including Max-Cut and Min-Cut, Max- $k$ -CSP, Maximum-Weighted-Bipartite-Matching and Traveling Salesman. We remark that a key difference between (computationally) easy and hard problems is not the ability to find a compressed and efficiently estimizable generative model but rather the ability to efficiently draw samples from the parameterized distributions.
1.1 Our Framework
We introduce a theoretical framework for analyzing the effectiveness of gradient-based methods on the optimization of solution generators in combinatorial optimization, inspired by  $\left[\mathrm{BPL}^{+}16\right]$ .
Let  $\mathcal{I}$  be a collection of instances of a combinatorial problem with common solution space  $S$  and  $L(\cdot; I): S \to \mathbb{R}$  be the cost function associated with an instance  $I \in \mathcal{I}$ , i.e.,  $L(s; I)$  is the cost of solution  $s$  given the instance  $I$ . For example, for the Max-Cut problem the collection of instances  $\mathcal{I}$  corresponds to all graphs with  $n$  nodes, the solution space  $S$  consists of all subsets of nodes, and the loss  $L(s; I)$  is equal to (minus) the weight of the cut  $(s, [n] \setminus s)$  corresponding to the subset of nodes  $s \in S$  (our goal is to minimize  $L$ ).
Definition 1 (Solution Cost Oracle). For a given instance  $I$  we assume that we have access to an oracle  $\mathcal{O}(\cdot; I)$  to the cost of any given solution  $s \in S$ , i.e.,  $\mathcal{O}(s; I) = L(s; I)$ .
The above oracle is standard in combinatorial optimization and query-efficient algorithms are provided for various problems [RSW17, GPRW19, LSZ21, AEG+22, PRW22]. We remark that the goal of this work is not to design algorithms that solve combinatorial problems using access to the solution cost oracle (as the aforementioned works do). This paper focuses on landscape design: the algorithm is fixed, namely (stochastic) gradient descent; the question is how to design a generative model that has a small number of parameters and the induced optimization landscape allows gradient-based methods to converge to the optimal solution without getting trapped at local minima or vanishing gradient points.
Let  $\mathcal{R}$  be some prior distribution over the instance space  $\mathcal{I}$  and  $\mathcal{W}$  be the space of parameters of the model. We now define the class of solution generators. The solution generator  $p(w)$  with parameter  $w\in \mathcal{W}$  takes as input an instance  $I$  and generates a random solution  $s$  in  $S$ . To distinguish between the output, the input, and the parameter of the solution generator, we use the notation  $p(\cdot ;I;w)$  to denote the distribution over solutions and  $p(s;I;w)$  to denote the probability of an individual solution  $s\in S$ . We denote by  $\mathcal{P} = \{p(w):w\in \mathcal{W}\}$  the above parametric class of solution generators. For some parameter  $w$ , the loss corresponding to the solutions sampled by  $p(\cdot ;I;w)$  is equal to
$$
\mathcal {L} (w) = \underset {I \sim \mathcal {R}} {\mathbf {E}} [ \mathcal {L} _ {I} (w) ], \mathcal {L} _ {I} (w) = \underset {s \sim p (\cdot ; I; w)} {\mathbf {E}} [ L (s; I) ]. \tag {1}
$$
Our goal is to optimize the parameter  $w \in \mathcal{W}$  in order to find a sampler  $p(\cdot; I; w)$  whose loss  $\mathcal{L}(w)$  is close to the expected optimal value opt:
$$
\operatorname {o p t} = \mathbf {E} _ {I \sim \mathcal {R}} \left[ \min  _ {s \in S} L (s; I) \right]. \tag {2}
$$
The policy gradient method [Kak01] expresses the gradient of  $\mathcal{L}$  as follows
$$
\nabla_ {w} \mathcal {L} (w) = \underset {I \sim \mathcal {R}} {\mathbf {E}} \underset {s \sim p (\cdot ; I; w)} {\mathbf {E}} \left[ L (s; I) \nabla_ {w} \log p (s; I; w) \right],
$$
and updates the parameter  $w$  using the gradient descent update. Observe that a (stochastic) policy gradient update can be implemented using only access to a solution cost oracle of Definition 1.
Solution Generators. In  $\left[\mathrm{BPL}^{+}16\right]$  the authors used neural networks as parametric solution generators for the TSP problem. They provided empirical evidence that optimizing the parameters of the neural network using the policy gradient method results to samplers that generate very good solutions for (Euclidean) TSP instances. Parameterizing the solution generators using neural networks essentially compresses the description of distributions over solutions (the full parameterization would require assigning a parameter to every solution-instance pair  $(s, I)$ ). Since for most combinatorial problems the size of the solution space is exponentially large (compared to the description of the instance), it is crucial that for such methods to succeed the parameterization must be compressed in the sense that the description of the parameter space  $\mathcal{W}$  is polynomial in the size of the description of the instance family  $\mathcal{I}$ . Apart from having a tractable number of parameters, it is important that the optimization objective corresponding to the parametric class  $\mathcal{P}$  can provably be optimized using some first-order method in polynomial (in the size of the input) iterations.
We collect these desiderata in the following definition. We denote by  $[\mathcal{I}]$  the description size of  $\mathcal{I}$ , i.e., the number of bits required to identify any element of  $\mathcal{I}$ . For instance, if  $\mathcal{I}$  is the space of unweighted graphs with at most  $n$  nodes,  $[I] = O(n^{2})$ .
Definition 2 (Complete, Compressed and Efficiently Optimizable Solution Generator). Fix a prior  $\mathcal{R}$  over  $\mathcal{I}$ , a family of solution generators  $\mathcal{P} = \{p(w):w\in \mathcal{W}\}$ , a loss function  $\mathcal{L}$  as in Equation (1) and some  $\epsilon >0$
1. We say that  $\mathcal{P}$  is complete if there exists some  $\overline{w} \in \mathcal{W}$  such that  $\mathcal{L}(\overline{w}) \leq \mathrm{opt} + \varepsilon$ , where  $\mathrm{opt}$  is defined in (2).
2. We say that  $\mathcal{P}$  is compressed if the description size of the parameter space  $\mathcal{W}$  is polynomial in  $[\mathcal{I}]$  and in  $\log(1/\varepsilon)$ .
3. We say that  $\mathcal{P}$  is efficiently estimizable if there exists a first-order method applied on the objective  $\mathcal{L}$  such that after  $T = \mathrm{poly}([W], 1 / \varepsilon)$  many updates on the parameter vectors, finds an (at most)  $\epsilon$ -sub-optimal vector  $\widehat{w}$ , i.e.,  $\mathcal{L}(\widehat{w}) \leq \mathcal{L}(\overline{w}) + \epsilon$ .
Remark 1. We remark that constructing parametric families that are complete and compressed, complete and efficiently estimable, or compressed and efficiently estimable (i.e., satisfying any pair of assumptions of Question 1 but not all 3) is usually a much easier task. Consider, for example, the Max-Cut problem on a fixed (unweighted) graph with  $n$  nodes. Note that  $\mathcal{I}$  has description size  $O(n^{2})$ . Solutions of the Max-Cut for a graph with  $n$  nodes are represented by vertices on the binary hypercube  $\{\pm 1\}^n$  (coordinate  $i$  dictates the side of the cut that we put node  $i$ ). One may consider the full parameterization of all distributions over the hypercube. It is not hard to see that this is a complete and efficiently estimable family (the optimization landscape corresponds to optimizing a linear objective). However, it is not compressed, since it requires  $2^n$  parameters. On the other extreme, considering a product distribution over coordinates, i.e., we set the value of node  $i$  to be  $+1$  with probability  $p_i$  and  $-1$  with  $1 - p_i$  gives a complete and compressed family. However, as we show in Appendix B, the landscape of this compressed parameterization suffers from highly sub-optimal local minima and therefore, it is not efficiently estimable.
Therefore, in this work we investigate whether it is possible to have all 3 desiderata of Definition 2 at the same time.
Question 1. Are there complete, compressed, and efficiently estimable solution generators (i.e., satisfying Definition 2) for challenging combinatorial tasks?
Our Contributions. Before we present our results formally, we summarize the contributions of this work.
- Our main contribution is a positive answer (Theorem 1) to Question 1 under general assumptions that capture many combinatorial tasks. We identify a set of conditions (see Assumption 1) that allow us to design a family of solution generators that are complete, compressed and efficiently estimizable.
- The conditions are motivated by obstacles that are important for any approach of this nature. This includes solutions that escape to infinity, and also parts of the landscape with vanishing gradient. See the discussion in Section 3 and Figure 1.
- We specialize our framework to several important combinatorial problems, some of which are NP-hard, and others tractable: Max-Cut, Min-Cut, Max- $k$ -CSP, Maximum-Weight-Bipartite-Matching, and the Traveling Salesman Problem.
- Finally, we investigate experimentally the effect of the entropy regularizer and the fast/slow mixture scheme that we introduced (see Section 3) and provide evidence that it leads to better solution generators.
We begin with the formal presentation of our assumptions on the feature mappings of the instances and solutions and on the structure of cost function of the combinatorial problem.
Assumption 1 (Structured Feature Mappings). Let  $S$  be the solution space and  $\mathcal{I}$  be the instance space. There exist feature mappings  $\psi_S: S \to X$ , for the solutions, and,  $\psi_{\mathcal{I}}: \mathcal{I} \to Z$ , for the instances, where  $X, Z$  are Euclidean vector spaces of dimension  $n_X$  and  $n_Z$ , such that
1. (Bounded Feature Spaces) The feature and instance mappings are bounded, i.e., there exist  $D_S, D_{\mathcal{I}} > 0$  such that  $\| \psi_S(s)\| _2\leq D_S$ , for all  $s\in S$  and  $\| \psi_{\mathcal{I}}(I)\| _2\leq D_{\mathcal{I}}$ , for all  $I\in \mathcal{I}$ .
2. (Bilinear Cost Oracle) The cost of a solution  $s$  under instance  $I$  can be expressed as a bilinear function of the corresponding feature vector  $\psi_S(s)$  and instance vector  $\psi_{\mathcal{I}}(I)$ , i.e., the solution oracle can be expressed as  $\mathcal{O}(s,I) = \psi_{\mathcal{I}}(I)^{\top}M\psi_{S}(s)$  for any  $s\in S,I\in \mathcal{I}$ , for some matrix  $M$  with  $\| M\|_{\mathrm{F}}\leq C$ .
3. (Variance Preserving Features) There exists  $\alpha > 0$  such that  $\mathbf{Var}_{s \sim U(S)}[v \cdot \psi_S(s)] \geq \alpha \| v \|_2^2$  for any  $v \in X$ , where  $U(S)$  is the uniform distribution over the solution space  $S$ .
4. (Bounded Dimensions/Diameters) The feature dimensions  $n_X, n_Z$ , and the diameter bounds  $D_S, D_{\mathcal{I}}, C$  are bounded above by a polynomial of the description size of the instance space  $\mathcal{I}$ . The variance lower bound  $\alpha$  is bounded below by 1/poly([I]).
Remark 2 (Boundedness and Bilinear Cost Assumptions). We remark that Items 1, 4 are simply boundedness assumptions for the corresponding feauture mappings and usually follow easily assuming that we consider reasonable feature mappings. At a high-level, the assumption that the solution is a bilinear function of the solution and instance features (Item 2) prescribes that "good" feature mappings should enable a simple (i.e., bilinear) expression for the cost function. In the sequel we see that this is satisfied by natural feature mappings for important classes of combinatorial problems.
Remark 3 (Variance Preservation Assumption). In Item 3 (variance preservation) we require that the solution feature mapping has variance along every direction, i.e., the feature vectors corresponding to the solutions must be "spread-out" when the underlying solution generator is the uniform distribution. As we show, this assumption is crucial so that the gradients of the resulting optimization objective are not-vanishing, allowing for its efficient optimization.
We mention that various important combinatorial problems satisfy Assumption 1. For instance, Assumption 1 is satisfied by Max-Cut, Min-Cut, Max- $k$ -CSP, Maximum-Weight-Bipartite-Matching and Traveling Salesman. We refer the reader to the upcoming Section 2 for an explicit description of the structured feature mappings for these problems. Having discussed Assumption 1, we are ready to state our main abstract result which resolves Question 1.
Theorem 1. Consider a combinatorial problem with instance space  $\mathcal{I}$  that satisfies Assumption 1. For any prior  $\mathcal{R}$  over  $\mathcal{I}$  and  $\epsilon >0$ , there exists a family of solution generators  $\mathcal{P} = \{p(w):w\in \mathcal{W}\}$  with parameter space  $\mathcal{W}$  that is complete, compressed and, efficiently estimizable.
A sketch behind the design of the family  $\mathcal{P}$  can be found in Section 3 and Section 4.
Remark 4 (Computational Barriers in Sampling). We note that the families of generative models (a.k.a., solution generators) that we provide have polynomial parameter complexity and are estimable in a small number of steps using gradient-based methods. Hence, in a small number of iterations, gradient-based methods converge to distributions whose mass is concentrated on nearly optimal solutions. This holds, as we show, even for challenging (NP-hard) combinatorial problems. Our results do not, however, prove  $\mathrm{P} = \mathrm{NP}$ , as it may be computationally hard to sample from our generative models. We remark that while such approaches are in theory hard, such models seem to perform remarkably well experimentally where sampling is based on Langevin dynamics techniques [SE20, SSDK+20]. Though as our theory predicts, and simulations support, landscape problems seem to be a direct impediment even to obtain good approximate solutions.
Remark 5 (Neural Networks as Solution Samplers). A natural question would be whether our results can be extended to the case where neural networks are (efficient) solution samplers, as in  $[BPL^{+}16]$ . Unfortunately, a benign landscape result for neural network solution generators most likely cannot exist. It is well-known that end-to-end theoretical guarantees for training neural networks are out of reach since the corresponding optimization tasks are provably computationally intractable, see, e.g., [CGKM22] and the references therein.
Finally, we would like to mention an interesting aspect of Assumption 1. Given a combinatorial problem, Assumption 1 essentially asks for the design of feature mappings for the solutions and the instances that satisfy desiderata such as boundedness and variance preservation. Max-Cut, Min-Cut, TSP and Max-  $k$ -CSP and other problems satisfy Assumption 1 because we managed to design appropriate (problem-specific) feature mappings that satisfy the requirements of Assumption 1. There are interesting combinatorial problems for which we do not know how to design such good feature mappings. For instance, the "natural" feature mapping for the Satisfiability problem (SAT) (similar to the one we used for Max-  $k$ -CSPs) would require feature dimension exponential in the size of the instance (we need all possible monomials of  $n$  variables and degree at most  $n$ ) and therefore, would violate Item 4 of Assumption 1.
Neural Combinatorial Optimization. Tackling combinatorial optimization problems constitute one of the most fundamental tasks of theoretical computer science [GLS12, PS98, S+03, Sch05, CLS+95] and various approaches have been studied for these problems such as local search methods, branch-and-bound algorithms and meta-heuristics such as genetic algorithms and simulated annealing. Starting from the seminal work of [HT85], researchers apply neural networks [Smi99, VFJ15, BPL+16] to solve combinatorial optimization tasks. In particular, researchers have explored the power of machine learning, reinforcement learning and deep learning methods for solving combinatorial optimization problems [BPL+16, YW20, LZ09, DCL+18, BLP21, MSIB21, NOST18, SHM+16, MKS+13, SSS+17, ER18, KVHW18, ZCH+20, CCK+21, MGH+19, GCF+19, KLMS19].
The use of neural networks in combinatorial problems is extensive [SLB+18, JLB19, GCF+19, YGS20, MSIB21, BPL+16, KDZ+17, YP19, CT19, YBV19, KCK+20, KCY+21, DAT20, NJS+20, TRWG21, AMW18, KL20, Jeg22, SBK22, ART23] and various papers aim to understand the theoretical ability of neural networks to solve such problems [HS23b, HS23a, Gam23]. Our paper builds on the framework of the influential experimental work of  $\left[\mathrm{BPL}^{+}16\right]$  to tackle combinatorial optimization problems such as TSP using neural networks and reinforcement learning.  $\left[\mathrm{KP}^{+}21\right]$  uses an entropy maximization scheme in order to generate diversified candidate solutions. This experimental heuristic is quite close to our theoretical idea for entropy regularization. In our work, entropy regularization allows us to design quasar-convex landscapes and the fast/slow mixing scheme to obtain diversification of solutions. Among other related applied works,  $\left[\mathrm{KCK}^{+}20\right.$ , KPP22] study the use of Transformer architectures combined with the Reinforce algorithm employing symmetries (i.e., the existence of multiple optimal solutions of a CO problem) improving the generalization capability of Deep RL NCO and  $\left[\mathrm{MLC}^{+}21\right]$  studies Transformer architectures and aims to learn improvement heuristics for routing problems using RL.
Gradient Descent Dynamics. Our work provides theoretical understanding on the gradient-descent landscape arising in NCO problems. Similar questions regarding the dynamics of gradient descent have been studied in prior work concerning neural networks; for instance, [AS20] and  $\mathrm{[AKM^{+}21]}$  fix the algorithm (SGD on neural networks) and aim to understand the power of this approach (which function classes can be learned). Various other works study gradient descent dynamics in neural networks. We refer to [AS18, AS20, ABAB  $^+$  21, MYSSS21, BEG  $^+$  22, DLS22, ABA22, AAM22, BBSS22, ABAM23,  $\mathrm{AKM^{+}21}$ ,  $\mathrm{EGK^{+}23}$ ] (and the references therein) for a small sample of this line of research.
2 Combinatorial Applications
We now consider concrete combinatorial problems and show that there exist appropriate and natural feature mappings for the solutions and instances that satisfy Assumption 1; so Theorem 1 is applicable for any such combinatorial task. For a more detailed treatment, we refer to Appendix G.
Min-Cut and Max-Cut. Min-Cut (resp. Max-Cut) are central graph combinatorial problems where the task is to split the nodes of the graph in two subsets so that the number of edges from one subset to the other (edges of the cut) is minimized (resp. maximized). Given a graph  $G$  with  $n$  nodes represented by its Laplacian matrix  $L_{G} = D - A$ , where  $D$  is the diagonal degree matrix and  $A$  is the adjacency matrix of the graph, the goal in the Min-Cut (resp. Max-Cut) problem is to find a solution vector  $s \in \{\pm 1\}^n$  so that  $s^\top L_G s / 4$  is minimized (resp. maximized).
We first show that there exist natural feature mappings so that the cost of every solution  $s$  under any instance/graph  $G$  is a bilinear function of the feature vectors, see Item 2 of Assumption 1. We consider the correlation-based feature mapping  $\psi_S(s) = (ss^\top)^{\flat} \in \mathbb{R}^{n^2}$ , where by  $(\cdot)^{\flat}$  we denote the vectorization/ flattening operation and the Laplacian for the instance (graph),  $\psi_{\mathcal{I}}(G) = (L_G)^\flat \in \mathbb{R}^{n^2}$ . Then simply setting the matrix  $M$  to be the identity  $I \in \mathbb{R}^{n^2 \times n^2}$  the cost of any solution  $s$  can be expressed as the bilinear function  $\psi_{\mathcal{I}}(G)^\top M\psi_S(s) = (L_G^{\flat})^\top (ss^\top)^{\flat} = s^\top L_Gs$ . We observe that for (unweighted) graphs with  $n$  nodes the description size of the family of all instances  $\mathcal{I}$  is roughly  $O(n^2)$ , and therefore the dimensions of the feature mappings  $\psi_S, \psi_{\mathcal{I}}$  are clearly polynomial in the description size of  $\mathcal{I}$ . Moreover, considering unweighted graphs, it holds that  $\| \psi_{\mathcal{I}}(G)\|_2, \| \psi_S(s)\|_2, \| M\|_{\mathrm{F}} \leq \mathrm{poly}(n)$ . Therefore, the constants  $D_S, D_{\mathcal{I}}, C$  are polynomial in the description size of the instance family.