SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lorenzo Brusca* Lars C.P.M. Quaedvlieg* Stratis Skoulakis* Grigorios G Chrysos† Volkan Cevher École Polytechnique Fédérale de Lausanne, Switzerland {{first name}. [last name]}@epfl.ch
This work presents a graph neural network (GNN) framework for solving the maximum independent set (MIS) problem, inspired by dynamic programming (DP). Specifically, given a graph, we propose a DP-like recursive algorithm based on GNNs that firstly constructs two smaller sub-graphs, predicts the one with the larger MIS, and then uses it in the next recursive call. To train our algorithm, we require annotated comparisons of different graphs concerning their MIS size. Annotating the comparisons with the output of our algorithm leads to a self-training process that results in more accurate self-annotation of the comparisons and vice versa. We provide numerical evidence showing the superiority of our method vs prior methods in multiple synthetic and real-world datasets.
Deep neural networks (DNNs) have achieved unprecedented success in extracting intricate patterns directly from data without the need for handcrafted rules, while still generalizing well to new and previously unseen instances [He et al., 2016, Vaswani et al., 2017]. Among other applications, this success has led to the development of frameworks that utilize DNNs to solve combinatorial optimization (CO) problems, such as the Traveling Salesman Problem [Xing and Tu, 2020, Hu et al., 2020, Prates et al., 2019], the Job-Shop Scheduling Problem [Zhang et al., 2020, Park et al., 2021], and the Quadratic Assignment Problem [Nowak et al., 2017].
A core challenge for deep learning approaches on CO is the lack of training data. Annotating such data requires the solution of a huge number of instances of the CO, hence such supervised learning approaches are computationally infeasible for NP-hard problems [Yehuda et al., 2020]. Circumventing this difficulty is key to unlocking the full potential of otherwise broadly applicable DNNs for CO.
Our work demonstrates how classical ideas in CO together with DNNs can lead to a scalable self-supervised learning approach, mitigating the lack of training data. Concretely, we focus on the Maximum Independent Set (MIS) problem: Given a graph  $G(V,E)$ , MIS asks for a set of nodes of maximum cardinality such that no two nodes in the selected set are connected with an edge. MIS is an NP-hard problem with several hand-crafted heuristics (e.g., greedy heuristic, local search). More recently, several deep learning approaches have been proposed [Karalias and Loukas, 2020, Toenhoff et al., 2019, Schuetz et al., 2022a].
Our approach involves the following steps to determine an MIS in a graph. We use graph neural networks (GNNs) [Wu et al., 2020] to enable a model to generate approximate maximum independent sets after training on data that was annotated by the model itself. For this purpose, we draw inspiration
†Work done while at EPFL. Currently at University of Wisconsin-Madison; [surname]@wisc.edu.
from dynamic programming (DP) and employ a DP-like recursive algorithm. Initially, we are given a graph. At each recursive step, we select a random vertex from that graph and create two sub-graphs: one by removing the selected vertex and another by removing all its neighboring vertices. We then make a comparison between these sub-graphs to determine which sub-graph is likely to have a larger independent set, and we use the sub-graph with the highest estimated independent set for the next recursive call. We repeat this process until we reach a graph consisting only of isolated vertices, which signifies the discovery of an independent set for the original graph.
Dynamic programming guarantees that if our predictions are accurate (i.e., we select the sub-graph with the largest MIS value), our recursive algorithm will always result in a maximum independent set. To make accurate predictions, we introduce "graph comparing functions," which take two graphs as input and output a winner. We implement such graph-comparing functions with GNNs.
We adopt a self-training approach to train our graph-comparing function and optimize the parameters of the GNN. In each epoch, we update the graph-comparing function parameters to ensure it accurately fits the data it has seen so far. The data comprises pairs of graphs  $(G,G^{\prime})$  along with a label  $\mathrm{Label}(G,G^{\prime})\in \{0,1\}$ . For annotating the labels, we utilize the output of the recursive algorithm that leverages the graph-comparing function. Supported by theoretical and experimental evidence, we demonstrate how the self-annotation process improves parameter selection.
We conduct a thorough validation of our self-training approach in three real-world graph distribution datasets. Our algorithm surpasses the performance of previous deep learning methods [Karalias and Loukas, 2020, Toenshoff et al., 2019, Ahn et al., 2020] in the context of the MIS problem. To further validate the efficacy of our method, we explore its robustness on out-of-distribution data. Notably, our results demonstrate that the induced algorithm achieves competitive performance, showcasing the generalization capability of the learned comparator across different graph structures and distributions. In addition, we extend the evaluation of our DP-based self-training approach to tackle the Minimum Vertex Cover (MVC) problem in Appendix E. Encouragingly, similar to the MIS case, our induced GNN-based algorithms for MVC admit competitive performance with respect to other deep-learning approaches.
The code for the experiments and models discussed in this paper is available at: https://github. com/LIONS-EPFL/dynamic-MIS.
Our work lies in the intersection of various domains, i.e., combinatorial optimization, Dynamic Programming, and (graph) neural networks. We review the most critical ideas in each domain here and defer a more detailed discussion in Appendix A.
Graph Neural Networks (GNNs) have gained widespread popularity due to their ability to learn representations of graph-structured data [Xiao et al., 2022, Zhang and Chen, 2018, Zhu et al., 2021, Errica et al., 2019] invariant to the size of the graph. More complex architectural blocks, such as the Graph Convolutional Network (GCN) [Kipf and Welling, 2017, Zhang et al., 2019], the Graph Attention Network (GAT) [Veličković et al., 2017], and the Graph Isomorphism Network [Xu et al., 2018] have become influential instances of GNNs. In our work, we utilize a simple GNN architecture to showcase the effectiveness of our proposed framework. While our choice of architecture is intentionally simple, we emphasize its modular nature, which enables us to incorporate more complex GNNs with ease.
**Combinatorial Optimization:** Supervised learning approaches have been used for tackling CO tasks, such as the Traveling Salesman Problem (TSP) [Vinyals et al., 2015], the Vehicle Routing Problem (VRP) [Shalaby et al., 2021], and Graph Coloring [Lemos et al., 2019]. Due to the graph structure of the problems, GNNs are often used for tackling those tasks [Prates et al., 2019, Nazari et al., 2018, Schuetz et al., 2022b]. However, owing to the computational overhead of obtaining supervised labels, such supervised approaches often do not scale well. Instead, unsupervised approaches have been deployed recently [Wang and Li, 2023]. A popular approach relies on a continuous relaxation of the loss function [Karalias and Loukas, 2020, Wang et al., 2022, Wang and Li, 2023]. In contrast to the previous unsupervised works, we adopt Dynamic Programming techniques to diminish the overall time complexity of the algorithm. Another approach uses reinforcement learning (RL) methods to address CO tasks, such as in Covering Salesman Problem [Li et al., 2021], the TSP [Zhang et al.,
[2022], the VRP [James et al., 2019], and the Minimum Vertex Cover (MVC) [Tian and Li, 2021]. However, applying RL to CO problems can be challenging because of the long learning time required and the non-differentiable nature of the loss function.
Dynamic Programming has been a powerful problem-solving technique since at least the 50s [Bellman, 1954]. In recent years, researchers have explored the use of deep neural networks (DNNs) to replace the function responsible for dividing a problem into subproblems and estimating the optimal decision at each step [Yang et al., 2018]. Despite the progress, implementing CO tasks with Dynamic Programming suffers from significant computational overheads, since the size of the search space grows exponentially with the problem size [Xu et al., 2020]. Our approach overcomes this issue by utilizing a model that approximates the standard lookup table from Dynamic Programming, meaning that we avoid the exponential search space typically associated with DP.
3 An Optimal Solution to Maximum Independent Set (MIS)
Firstly, let us introduce MIS and its relationship with Dynamic Programming.
Notation:  $G(V, E)$  denotes an undirected graph where  $V$  stands for vertices and  $E$  for the edges.  $\mathcal{N}(v)$  denotes the neighbors of vertex  $v \in V$ ,  $\mathcal{N}(v) = \{u \in V : (u, v) \in E\}$ . The degree of vertex  $v \in V$  is  $d(v) := |\mathcal{N}(v)|$ . Given a set of vertices  $S \subseteq V$ ,  $G / S$  denotes the remaining graph of  $G$  after removing all nodes  $v \in S$ .
Definition 1 (Maximum Independent Set). Given an undirected graph  $G(V, E)$ , find a maximum set of nodes  $S \subseteq V$  such that  $(u, v) \notin E$  for all vertices  $u, v \in S$ . We denote with  $|\mathrm{MIS}(G)|$  the size of the maximum independent set of graph  $G$ .
Dynamic Programming is a powerful technique for algorithmic design in which the optimal solution of the instance of interest is constructed by combining the optimal solution of smaller sub-instances. The combination step is governed by local optimality conditions which, in the context of MIS, take the form of Theorem 1. Theorem 1 establishes that the decision to remove a node  $v \in V$  or its neighbors  $\mathcal{N}(v)$  during the recursive process depends on whether  $|\mathrm{MIS}(G / \mathcal{N}(v))| \geq |\mathrm{MIS}(G / v)|$ . This decision continues until a graph with no edges is reached. According to Theorem 1, if at each step of the recursion, the choice is made based on whether  $|\mathrm{MIS}(G / \mathcal{N}(v))| \geq |\mathrm{MIS}(G / v)|$  or not, then the resulting empty graph is guaranteed to be an optimal solution. The proof is this theorem can be found in Appendix I.
Theorem 1. Let a graph  $G(V,E) \in \mathcal{G}$ . Then for any vertex  $v \in V$  with  $d(v) \geq 1$ ,
$$
\left| \operatorname {M I S} (G) \right| = \max  \left(\left| \operatorname {M I S} \left(G / \mathcal {N} (v)\right) \right|, \left| \operatorname {M I S} \left(G / \{v \}\right) \right|\right) ^ {1}.
$$
4 Graph Neural Network-Based Algorithm for MIS
In this section, we present our approach for developing algorithms for MIS parameterized by parameters  $\theta \in \Theta$ . Initially in Sec. 4.1 we present how any graph-comparing function taking as input two different graphs and outputting a  $\{0,1\}$  value can be used in the construction of an algorithm computing an independent set (not necessarily optimal). In Sec. 4.2 we present how Graph Neural Networks can be used in the construction of such graph-comparing functions. Finally, in Sec. 4.3, we present our inference algorithm that computes an independent set of any graph  $G \in \mathcal{G}$ .
4.1 MIS Algorithms induced by Graph Comparing Functions
Consider a function  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$  that compares two graphs  $G,G^{\prime}$  based on the size of their MIS. Namely, if  $|\mathrm{MIS}(G)|\geq |\mathrm{MIS}(G')|$  then  $\mathrm{CMP}(G,G^{\prime}) = 0$  and  $\mathrm{CMP}(G,G^{\prime}) = 1$  otherwise.
Theorem 1 ensures that if we have access in such a graph-comparing function then we can compute an independent set of maximum size for any graph  $G$ . From a starting node  $v$ , with an initial graph from  $G$ , and by recursively selecting either  $G / \{v\}$  or  $G / \mathcal{N}(v)$  based on  $|\mathrm{MIS}(G / \{v\})| \geq |\mathrm{MIS}(G / \mathcal{N}(v))|$ , we are ensured to end in an independent set of maximum size. The decision of
<sup>1</sup>Note: In the trivial case where  $G$  is an empty graph (i.e., it has no edges), the size of the maximum independent set is  $|V|$ .
Algorithm 1 Randomized Comparator-Induced Algorithm
1: function  $\mathcal{A}^{\mathrm{CMP}}(G(V,E))$  ▷ Algorithm  $\mathcal{A}^{\mathrm{CMP}}(G)$  takes a graph  $G$  as input  
2: if  $|E| = 0$  then return  $V$   
3: end if  
4: pick a vertex  $v \in V$  with  $d(v) > 0$  uniformly at random.  
5:  $G_0 \gets G \setminus \{v\}$  and  $G_1 \gets G \setminus \mathcal{N}(v)$   
6: if  $\mathrm{CMP}(G_0, G_1) = 0$  then  
7:  $G \gets G_0$  ▷ Remove vertex  $v$   
8: else  
9:  $G \gets G_1$  ▷ Remove the neighbors of  $v$   
10: end if  
11: return  $\mathcal{A}^{\mathrm{CMP}}(G)$   
12: end function
whether  $|\mathrm{MIS}(G / \{v\})| \geq |\mathrm{MIS}(G / \mathcal{N}(v))|$  at each recursive call can be made according to the output of  $\mathrm{CMP}(G / \{v\}, G / \mathcal{N}(v))$ .
The cornerstone idea of our approach is that any graph-comparing function CMP induces such a recursive algorithm for a MIS. Recursively selecting  $G / \{v\}$  or  $G / \mathcal{N}(v)$  based on the output of a graph generating function  $\mathrm{CMP}(G / \{v\}, G / \mathcal{N}(v)) \in \{0,1\}$  always guarantees to reach an independent set of the original graph. In case  $\mathrm{CMP}(G, G') \neq \mathbb{I}[[\mathrm{MIS}(G)] < |\mathrm{MIS}(G')|]$ , where  $\mathbb{I}$  is the indicator function, it is not guaranteed that the computed independent set is of the maximum size. However, there might exist reasonable graph comparing functions that  $i$  are efficiently computable  $ii$  lead to near-optimal solutions.
In Definition 2 and Algorithm 1 we formalize the idea above.
Definition 2. A comparator  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$  is a function taking as input two graphs  $G,G^{\prime}$  and outputting a  $\{0,1\}$  value.
Proposition 1. Any comparator  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$  induces a randomized algorithm  $\mathcal{A}^{\mathrm{CMP}}$  (Algorithm 1).
Remark 1. Given a graph-comparing function  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$ , the induced algorithm is randomized, since at Step 4 of Algorithm 1, a vertex  $v$  is randomly selected. Notice that Algorithm 1 recursively proceeds until a subgraph with 0 edges is reached (see Step 2).
Remark 2. Two different comparators CMP and  $\mathrm{CMP^{\prime}}$  induce two different algorithms  $\mathcal{A}^{\mathrm{CMP}}$  and  $\mathcal{A}^{\mathrm{CMP^{\prime}}}$  for estimating the maximum independent set.
4.2 Comparators through Graph Neural Networks
In this section, we discuss the architecture of a model  $M_{\theta}:\mathcal{G}\mapsto \mathbb{R}$ , parameterized by  $\theta \in \Theta$ , that is used for the construction of a comparator function
$$
\operatorname {C M P} _ {\theta} (G, G ^ {\prime}) = \mathbb {I} \left[ M _ {\theta} (G) <   M _ {\theta} \left(G ^ {\prime}\right) \right].
$$
In order to embed graph-level information, we introduce a new GNN module, which we refer to as the Graph Embedding Module (GEM). Unlike standard GNN modules, this module captures different semantic meanings of differing embeddings of a node, its neighbors, and anti-neighbors.
Graph Embedding Module (GEM):
The GEM operates using the following recursive formula:
$$
\mu_ {v} ^ {k + 1} = \ln \left(\operatorname {G E L U} \left(\left[ \theta_ {0} ^ {k} \mu_ {v} ^ {k} \left\| \theta_ {1} ^ {k} \sum_ {u \in \mathcal {N} (v)} \mu_ {u} ^ {k} \right\| \theta_ {2} ^ {k} \sum_ {u \notin \mathcal {N} (v)} \mu_ {u} ^ {k} \right]\right)\right). \tag {1}
$$
Initially, all nodes in this graph have zeros embeddings  $\mu_v^0 = \vec{0} \in \mathbb{R}^{3p}$ . Here,  $\mu_v^0$  denotes the initial embedding vector of node  $v$ . In Eq. (1), for all iterations  $k \in [0, \dots, K-1]$ , the embeddings of a
Figure 1: Architecture of model  $M_{\theta}(G)$ . From left to right: initially, an input graph  $G$  is passed into the model with zeros as node embeddings, which are displayed as white in the figure. The striped green edges connect the anti-neighbors, which are also used in the GEM. After  $K$  iterations of the GEM module, the final node embeddings are obtained. These are then averaged to obtain a graph embedding  $\mu_G$ . Finally, the graph embedding is put through multiple fully-connected layers to obtain a final logit value for the input graph.
node denoted by  $\mu_v^k\in \mathbb{R}^{3p}$ , its neighbors, and its anti-neighbors  $v$  are put through their own linear layers, denoted by  $\theta_0^k,\theta_1^k,\theta_2^k\in \mathbb{R}^{p\times 3p}$ , which are the parameters of the module. The bias term is omitted in the equation for readability purposes. We incorporate anti-neighbors in the GEM to capture complementary relationships between nodes. By using separate linear layers for different features, we emphasize the contrasting semantic meaning between neighbors and anti-neighbors, representing negative and positive relationships in the graph. Then, the individual feature embeddings are concatenated, which is denoted by  $[\dots \| \dots ]$ , followed by a GELU activation function [Hendrycks and Gimpel, 2016] and layer normalization [Ba et al., 2016a]. We note that the computational complexity of this module is  $\mathcal{O}(|V|^2)$ .
The complete model architecture is depicted in Fig. 1. At a high level, the architecture uses a Graph Embedding Module to extract a global graph embedding from the input graph, which is then passed through a set of fully connected layers to output a logit for that graph. During the training process of the comparator function, we utilize  $\mathrm{CMP}_{\theta}(G,G^{\prime}) = \mathrm{softmax}([M_{\theta}(G)\| M_{\theta}(G^{\prime})])$ , which forms a differentiable loss function for classification.
4.3 Inference Algorithm
In the previous section, we discussed how a parameterization  $\theta \in \Theta$  defines the graph-comparing function  $\mathrm{CMP}_{\theta}(G,G^{\prime}) = \mathbb{I}\left[M_{\theta}(G) < M_{\theta}(G^{\prime})\right]$ . As a result, the same parameterization  $\theta \in \Theta$  defines an algorithm  $\mathcal{A}^{\mathrm{CMP}_{\theta}}$ , where at Step 6 of Algorithm 1, the comparing function  $\mathrm{CMP}_{\theta}$  is used. Since the number of graph comparisons is upper-bounded by  $|V|$ , the inference algorithm admits to a computational complexity of  $\mathcal{O}(|V|^3)$ .
5 Self-Supervised Training through the Consistency Property
In this section, we present our methodology for selecting the parameters  $\theta \in \Theta$  so that the resulting inference algorithm  $\mathcal{A}^{\mathrm{CMP}_{\theta}}(\cdot)$  computes independent sets with (close to) the maximum value.
The most straightforward approach is to select the parameters  $\theta \in \Theta$  such that  $\mathrm{CMP}_{\theta}(G, G') \simeq \mathbb{I} [|\mathrm{MIS}(G)| < |\mathrm{MIS}(G')|]$  using labeled data. The problem with this approach is that a huge amount of annotated data of the form  $\{((G, G'), \mathbb{I} [|\mathrm{MIS}(G)| < |\mathrm{MIS}(G')|])\}$  are required. Since finding the MIS is an NP-Hard problem, annotating such data comes with an insurmountable computational burden.
The key idea to overcome the latter limitation is to annotate the data of the form  $\{(G, G')\}$  by using the algorithm  $\mathcal{A}^{\mathrm{CMP}_{\theta}}(\cdot)$  that runs in polynomial time with respect to the size of the graph. Intuitively, our proposed framework entails the optimization of the parameterized comparator function  $\mathrm{CMP}_{\theta}$  on data generated using algorithm  $\mathcal{A}^{\mathrm{CMP}_{\theta}}$ . A better comparator function leads to a better algorithm, which leads to better data, and vice versa. This mutually reinforcing relationship between the two components of our framework is theoretically indicated by Theorem 2 that we present in Section 5.1. The exact steps are detailed below.
5.1 Consistent Graph Comparing Functions
In this section, we introduce the notion of a consistent graph-comparing function (Definition 3) that plays a critical role in our self-supervised learning approach. Kindly take note that  $\mathcal{A}^{\mathrm{CMP}}$  utilizes the unparameterized variant of a comparator function, whereas  $\mathcal{A}^{\mathrm{CMP}_{\theta}}$  utilizes its parameterized counterpart.
Definition 3 (Consistency). A graph-comparing function  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$  is called consistent if and only if for any pair of graphs  $G,G^{\prime}\in \mathcal{G}$
$$
\operatorname {C M P} \left(G, G ^ {\prime}\right) = 0 \text {i f a n d o n l y i f} \mathbb {E} \left[ \left| \mathcal {A} ^ {\operatorname {C M P}} (G) \right| \right] \geq \mathbb {E} \left[ \left| \mathcal {A} ^ {\operatorname {C M P}} \left(G ^ {\prime}\right) \right| \right].
$$
Remark 3. In Definition 3 we use  $\mathbb{E}\left[\left|\mathcal{A}^{\mathrm{CMP}}(G)\right|\right],\mathbb{E}\left[\left|\mathcal{A}^{\mathrm{CMP}}(G')\right|\right]$  since, as we have already discussed, a comparator CMP induces a randomized algorithm  $\mathcal{A}^{\mathrm{CMP}}$ , where the expectation is over the nodes in the graphs.
In Theorem 2, we formally establish that any consistent graph-comparing function CMP induces an optimal algorithm for the MIS.
Theorem 2. Consider a consistent comparator  $\mathrm{CMP}:\mathcal{G}\times \mathcal{G}\mapsto \{0,1\}$ . Then, the algorithm  $\mathcal{A}^{\mathrm{CMP}}(\cdot)$  always computes a Maximum Independent Set,  $\mathbb{E}\left[\left|\mathcal{A}^{\mathrm{CMP}}(G)\right|\right] = |\mathrm{MIS}(G)|$  for all  $G\in \mathcal{G}$ .
Theorem 2 guarantees that any consistent graph comparing function CMP induces an optimal algorithm  $\mathcal{A}^{\mathrm{CMP}}$  for MIS. The proof for this theorem can be found in Appendix I. Hence, the selection of parameters  $\theta^{\star}\in \Theta$  should be selected such that  $\mathrm{CMP}_{\theta^{\star}}$  is consistent. More precisely:
Goal of Training: Find parameters  $\theta^{\star}\in \Theta$  such that for all  $G,G^{\prime}\in \mathcal{G}$
$$
\operatorname {C M P} _ {\theta^ {\star}} (G, G ^ {\prime}) = 0 \text {i f a n d o n l y i f} \mathbb {E} \left[ \left| \mathcal {A} ^ {\operatorname {C M P} _ {\theta^ {\star}}} (G) \right| \right] \geq \mathbb {E} \left[ \left| \mathcal {A} ^ {\operatorname {C M P} _ {\theta^ {\star}}} (G ^ {\prime}) \right| \right].
$$
5.2 Training a Consistent Comparator
The cornerstone idea of our self-supervised learning approach is to make the comparator more and more consistent over time. Namely, the idea is to update the parameters as follows:
$$
\theta^ {t + 1} := \operatorname {a r g m i n} _ {\theta \in \Theta} \mathbb {E} _ {G, G ^ {\prime}} \left[ \right. \ell \left( \right.\mathrm {C M P} _ {\theta} (G, G ^ {\prime}), \mathbb {I} \left[ \right. \mathbb {E} \left[ | \mathcal {A} ^ {\mathrm {C M P} _ {\theta_ {t}}} (G) | \right] <   \mathbb {E} \left[ | \mathcal {A} ^ {\mathrm {C M P} _ {\theta_ {t}}} (G ^ {\prime}) | ] \right]\left. \right)\left. \right], \tag {2}
$$
where  $\ell (\cdot ,\cdot)$  is a binary classification loss. In Eq. (2),  $\theta_{t}$  are the fixed parameters of the previous epoch. Thus, in the next few paragraphs, we only use the notation  $\mathcal{A}^{\mathrm{CMP}_{\theta_t}}$  to denote the fixed parameters. Gradient updates are only computed over  $\theta$
Remark 4. We remark that neither solving the non-convex minimization problem of Eq. (2) nor the existence of parameters  $\theta^{\star} \in \Theta$  such that  $\mathrm{CMP}_{\theta^{\star}}$  can be guaranteed. However, using a first-order method for Eq. (2) and a large enough parameterization can lead to an approximately consistent comparator with approximately optimal performance.