SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiqing Sun*
Language Technologies Institute
zhiqings@cs.cmu.edu
Yiming Yang
yiming@cs.cmu.edu
Neural network-based Combinatorial Optimization (CO) methods have shown promising results in solving various NP-complete (NPC) problems without relying on hand-crafted domain knowledge. This paper broadens the current scope of neural solvers for NPC problems by introducing a new graph-based diffusion framework, namely DIFUSCO. Our framework casts NPC problems as discrete  $\{0,1\}$ -vector optimization problems and leverages graph-based denoising diffusion models to generate high-quality solutions. We investigate two types of diffusion models with Gaussian and Bernoulli noise, respectively, and devise an effective inference schedule to enhance the solution quality. We evaluate our methods on two well-studied NPC combinatorial optimization problems: Traveling Salesman Problem (TSP) and Maximal Independent Set (MIS). Experimental results show that DIFUSCO strongly outperforms the previous state-of-the-art neural solvers, improving the performance gap between ground-truth and neural solvers from  $1.76\%$  to  $0.46\%$  on TSP-500, from  $2.46\%$  to  $1.17\%$  on TSP-1000, and from  $3.19\%$  to  $2.58\%$  on TSP-10000. For the MIS problem, DIFUSCO outperforms the previous state-of-the-art neural solver on the challenging SATLIB benchmark.
Combinatorial Optimization (CO) problems are mathematical problems that involve finding the optimal solution in a discrete space. They are fundamental challenges in computer science, especially the NP-Complete (NPC) class of problems, which are believed to be intractable in polynomial time. Traditionally, NPC solvers rely on integer programming (IP) or hand-crafted heuristics, which demand significant expert efforts to approximate near-optimal solutions [4, 31].
Recent development in deep learning has shown new promise in solving NPC problems. Existing neural CO solvers for NPC problems can be roughly classified into three categories based on how the solutions are generated, i.e., the autoregressive constructive solvers, the non-autoregressive constructive solvers, and the improvement heuristics solvers. Methods in the first category use autoregressive factorization to sequentially grow a valid partial solution [6, 64]. Those methods typically suffer from the costly computation in their sequential decoding parts and hence are difficult to scale up to large problems [27]. Methods in the second category rely on non-autoregressive modeling for scaling up, with a conditional independence assumption among variables as typical [53, 55, 92]. Such an assumption, however, unavoidably limits the capability of those methods to capture the multimodal nature of the problems [57, 33], for example, when multiple optimal solutions exist for the same graph. Methods in the third category (improvement heuristics solvers) use a Markov decision process (MDP) to iteratively refines an existing feasible solution with neural network-guided local operations such as 2-opt [71, 2] and node swap [17, 113]. These methods
*Our code is available at https://github.com/Edward-Sun/DIFUSCO.
have also suffered from the difficulty in scaling up and the latency in inference, partly due to the sparse rewards and sample efficiency issues when learning improvement heuristics in a reinforcement learning (RL) framework [113, 79].
Motivated by the recent remarkable success of diffusion models in probabilistic generation [102, 40, 94, 120, 96], we introduce a novel approach named DIFUSCO, which stands for the graph-based DIFfUsion Solvers for Combinatorial Optimization. To apply the iterative denoising process of diffusion models to graph-based settings, we formulate each NPC problem as to find a  $\{0,1\}$ -valued vector that indicates the optimal selection of nodes or edges in a candidate solution for the task. Then we use a message passing-based graph neural network [61, 36, 29, 107] to encode each instance graph and to denoise the corrupted variables. Such a graph-based diffusion model overcomes the limitations of previous neural NPC solvers from a new perspective. Firstly, DIFUSCO can perform inference on all variables in parallel with a few ( $\ll N$ ) denoising steps (Sec. 3.3), avoiding the sequential generation problem of autoregressive constructive solvers. Secondly, DIFUSCO can model a multimodal distribution via iterative refinements, which alleviates the expressiveness limitation of previous non-autoregressive constructive models. Last but not least, DIFUSCO is trained in an efficient and stable manner with supervised denoising (Sec. 3.2), which solves the training scalability issue of RL-based improvement heuristics methods.
We should point out that the idea of utilizing a diffusion-based generative model for NPC problems has been explored recently in the literature. In particular, Graikos et al. [32] proposed an image-based diffusion model to solve Euclidean Traveling Salesman problems by projecting each TSP instance onto a  $64 \times 64$  greyscale image space and then using a Convolutional Neural Network (CNN) to generate the predicted solution image. The main difference between such image-based diffusion solver and our graph-based diffusion solver is that the latter can explicitly model the node/edge selection process via the corresponding random variables, which is a natural design choice for formulating NPC problems (since most of them are defined over a graph), while the former does not support such a desirable formalism. Although graph-based modeling has been employed with both constructive [64] and improvement heuristics [20] solvers, how to use graph-based diffusion models for solving NPC problems has not been studied before, to the best of our knowledge.
We investigate two types of probabilistic diffusion modeling within the DIFUSCO framework: continuous diffusion with Gaussian noise [16] and discrete diffusion with Bernoulli noise [5, 44]. These two types of diffusion models have been applied to image processing but not to NPC problems so far. We systematically compare the two types of modeling and find that discrete diffusion performs better than continuous diffusion by a significant margin (Section 4). We also design an effective inference strategy to enhance the generation quality of the discrete diffusion solvers.
Finally, we demonstrate that a single graph neural network architecture, namely the Anisotropic Graph Neural Network [9, 54], can be used as the backbone network for two different NP-complete combinatorial optimization problems: Traveling Salesman Problem (TSP) and Maximal Independent Set (MIS). Our experimental results show that DIFUSCO outperforms previous probabilistic NPC solvers on benchmark datasets of TSP and MIS problems with various sizes.
2.1 Autoregressive Construction Heuristics Solvers
Autoregressive models have achieved state-of-the-art results as constructive heuristic solvers for combinatorial optimization (CO) problems, following the recent success of language modeling in the text generation domain [106, 11]. The first approach proposed by Bello et al. [6] uses a neural network with reinforcement learning to append one new variable to the partial solution at each decoding step until a complete solution is generated. However, autoregressive models [64] face high time and space complexity challenges for large-scale NPC problems due to their sequential generation scheme and quadratic complexity in the self-attention mechanism [106].
2.2 Non-autoregressive Construction Heuristics Solvers
Non-autoregressive (or heatmap) constructive heuristics solvers [53, 27, 28, 92] are recently proposed to address this scalability issue by assuming conditional independence among variables in NPC problems, but this assumption limits the ability to capture the multimodal nature [57, 33] of high-
quality solution distributions. Therefore, additional active search [6, 92] or Monte-Carlo Tree Search (MCTS) [27, 98] are needed to further improve the expressive power of the non-autoregressive scheme.
DIFUSCO can be regarded as a member in the non-autoregressive constructive heuristics category and thus can be benefited from heatmap search techniques such as MCTS. But DIFUSCO uses an iterative denoising scheme to generate the final heatmap, which significantly enhances its expressive power compared to previous non-autoregressive methods.
2.3 Diffusion Models for Discrete Data
Typical diffusion models [100, 102, 40, 103, 85, 56] operate in the continuous domain, progressively adding Gaussian noise to the clean data in the forward process, and learning to remove noises in the reverse process in a discrete-time framework.
Discrete diffusion models have been proposed for the generation of discrete image bits or texts using binomial noises [100] and multinomial/categorical noises [5, 44]. Recent research has also shown the potential of discrete diffusion models in sound generation [118], protein structure generation [77], molecule generation [108], and better text generation [52, 38].
Another line of work studies diffusion models for discrete data by applying continuous diffusion models with Gaussian noise on the embedding space of discrete data [30, 69, 24], the  $\{-1.0, 1.0\}$  real-number vector space [16], and the simplex space [37]. The most relevant work might be Niu et al. [86], which proposed a continuous score-based generative framework for graphs, but they only evaluated simple non-NP-hard CO tasks such as Shortest Path and Maximum Spanning Tree.
3 DIFUSCO: Proposed Approach
Following a conventional notation [88], we define  $\mathcal{X}_s = \{0,1\}^N$  as the space of candidate solutions  $\{\mathbf{x}\}$  for a CO problem instance  $s$ , and  $c_{s}:\mathcal{X}_{s}\to \mathbb{R}$  as the objective function for solution  $\mathbf{x}\in \mathcal{X}_s$ :
$$
c _ {s} (\mathbf {x}) = \operatorname {c o s t} (\mathbf {x}, s) + \operatorname {v a l i d} (\mathbf {x}, s). \tag {1}
$$
Here  $\mathrm{cost}(\cdot)$  is the task-specific cost of a candidate solution (e.g., the tour length in TSP), which is often a simple linear function of  $\mathbf{x}$  in most NP-complete problems, and  $\mathrm{valid}(\cdot)$  is the validation term that returns 0 for a feasible solution and  $+\infty$  for an invalid one. The optimization objective is to find the optimal solution  $\mathbf{x}^{s*}$  for a given instance  $s$  as:
$$
\mathbf {x} ^ {s *} = \underset {\mathbf {x} \in \mathcal {X} _ {s}} {\operatorname {a r g m i n}} c _ {s} (\mathbf {x}). \tag {2}
$$
This framework is generically applicable to different NPC problems. For example, for the Traveling Salesman Problem (TSP),  $\mathbf{x} \in \{0,1\}^N$  is the indicator vector for selecting a subset from  $N$  edges; the cost of this subset is calculated as:  $\mathrm{cost}_{\mathrm{TSP}}(\mathbf{x},s) = \sum_{i}x_{i}\cdot d_{i}^{(s)}$ , where  $d_{i}^{(s)}$  denotes the weight of the  $i$ -th edge in problem instance  $s$ , and the valid  $(\cdot)$  part of Formula (1) ensures that  $\mathbf{x}$  is a tour that visits each node exactly once and returns to the starting node at the end. For the Maximal Independent Set (MIS) problem,  $\mathbf{x} \in \{0,1\}^N$  is the indicator vector for selecting a subset from  $N$  nodes; the cost of the subset is calculated as:  $\mathrm{cost}_{\mathrm{MIS}}(\mathbf{x},s) = \sum_{i}(1 - x_{i})$ , and the corresponding valid  $(\cdot)$  validates  $\mathbf{x}$  is an independent set where each node in the set has no connection to any other node in the set.
Probabilistic neural NPC solvers [6] tackle instance problem  $s$  by defining a parameterized conditional distribution  $p_{\theta}(\mathbf{x}|s)$ , such that the expected cost  $\sum_{\mathbf{x}\in \mathcal{X}_s}c_s(\mathbf{x})\cdot p(\mathbf{x}|s)$  is minimized. Such probabilistic generative models are usually optimized by reinforcement learning algorithms [111, 63]. In this paper, assuming the optimal (or high-quality) solution  $\mathbf{x}_s^*$  is available for each training instance  $s$ , we optimize the model through supervised learning. Let  $\mathcal{S} = \{s_i\}_{1}^{N}$  be independently and identically distributed (IID) training samples for a type of NPC problem, we aim to maximize the likelihood of optimal (or high-quality) solutions, where the loss function  $L$  is defined as:
$$
L (\boldsymbol {\theta}) = \mathbb {E} _ {s \in \mathcal {S}} \left[ - \log p _ {\boldsymbol {\theta}} \left(\mathbf {x} ^ {s *} \mid s\right) \right] \tag {3}
$$
Next, we describe how to use diffusion models to parameterize the generative distribution  $p_{\theta}$ . For brevity, we omit the conditional notations of  $s$  and denote  $\mathbf{x}^{s*}$  as  $\mathbf{x}_0$  as a convention in diffusion models for all formulas in the rest of the paper.
3.2 Diffusion Models in DIFUSCO
From the variational inference perspective [60], diffusion models [100, 40, 102] are latent variable models of the form  $p_{\theta}(\mathbf{x}_0) \coloneqq \int p_{\theta}(\mathbf{x}_{0:T}) d\mathbf{x}_{1:T}$ , where  $\mathbf{x}_1, \ldots, \mathbf{x}_T$  are latents of the same dimensionality as the data  $\mathbf{x}_0 \sim q(\mathbf{x}_0)$ . The joint distribution  $p_{\theta}(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod_{t=1}^T p_{\theta}(\mathbf{x}_{t-1} | \mathbf{x}_t)$  is the learned reverse (denoising) process that gradually denoises the latent variables toward the data distribution, while the forward process  $q(\mathbf{x}_{1:T} | \mathbf{x}_0) = \prod_{t=1}^T q(\mathbf{x}_t | \mathbf{x}_{t-1})$  gradually corrupts the data into noised latent variables. Training is performed by optimizing the usual variational bound on negative log-likelihood:
$$
\begin{array}{l} \mathbb {E} \left[ - \log p _ {\boldsymbol {\theta}} (\mathbf {x} _ {0}) \right] \leq \mathbb {E} _ {q} \left[ - \log \frac {p _ {\boldsymbol {\theta}} (\mathbf {x} _ {0 : T})}{q _ {\boldsymbol {\theta}} (\mathbf {x} _ {1 : T} | \mathbf {x} _ {0})} \right] \\ = \mathbb {E} _ {q} \left[ \sum_ {t > 1} D _ {K L} \left[ q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) \| p _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) \right] - \log p _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {0} \mid \mathbf {x} _ {1}\right) \right] + C \tag {4} \\ \end{array}
$$
where  $C$  is a constant.
Discrete Diffusion In discrete diffusion models with multinomial noises [5, 44], the forward process is defined as:  $q(\mathbf{x}_t|\mathbf{x}_{t - 1}) = \mathrm{Cat}(\mathbf{x}_t;\mathbf{p} = \tilde{\mathbf{x}}_{t - 1}\mathbf{Q}_t)$ , where  $\mathbf{Q}_t = \begin{bmatrix} (1 - \beta_t) & \beta_t \\ \beta_t & (1 - \beta_t) \end{bmatrix}$  is the transition probability matrix;  $\tilde{\mathbf{x}}\in \{0,1\}^{N\times 2}$  is converted from the original vector  $\mathbf{x}\in \{0,1\}^N$  with a one-hot vector per row; and  $\tilde{\mathbf{x}}\mathbf{Q}$  computes a row-wise vector-matrix product.
Here,  $\beta_{t}$  denotes the corruption ratio. Also, we want  $\prod_{t=1}^{T}(1 - \beta_{t}) \approx 0$  such that  $\mathbf{x}_{T} \sim \mathrm{Uniform}(\cdot)$ . The  $t$ -step marginal can thus be written as:  $q(\mathbf{x}_{t}|\mathbf{x}_{0}) = \mathrm{Cat}\left(\mathbf{x}_{t}; \mathbf{p} = \tilde{\mathbf{x}}_{0}\overline{\mathbf{Q}}_{t}\right)$ , where  $\overline{\mathbf{Q}}_{t} = \mathbf{Q}_{1}\mathbf{Q}_{2}\ldots \mathbf{Q}_{t}$ . And the posterior at time  $t-1$  can be obtained by Bayes' theorem:
$$
q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) = \frac {q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1} , \mathbf {x} _ {0}\right) q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {0}\right)}{q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right)} = \operatorname {C a t} \left(\mathbf {x} _ {t - 1}; \mathbf {p} = \frac {\tilde {\mathbf {x}} _ {t} \mathbf {Q} _ {t} ^ {\top} \odot \tilde {\mathbf {x}} _ {0} \overline {{\mathbf {Q}}} _ {t - 1}}{\tilde {\mathbf {x}} _ {0} \overline {{\mathbf {Q}}} _ {t} \tilde {\mathbf {x}} _ {t} ^ {\top}}\right), \tag {5}
$$
where  $\odot$  denotes the element-wise multiplication.
According to Austin et al. [5], the denoising neural network is trained to predict the clean data  $p_{\theta}(\widetilde{\mathbf{x}}_0|\mathbf{x}_t)$ , and the reverse process is obtained by substituting the predicted  $\widetilde{\mathbf{x}}_0$  as  $\mathbf{x}_0$  in Eq. 5:
$$
p _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) = \sum_ {\widetilde {\mathbf {x}}} q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \widetilde {\mathbf {x}} _ {0}\right) p _ {\boldsymbol {\theta}} \left(\widetilde {\mathbf {x}} _ {0} \mid \mathbf {x} _ {t}\right) \tag {6}
$$
Continuous Diffusion for Discrete Data The continuous diffusion models [102, 40] can also be directly applied to discrete data by lifting the discrete input into a continuous space [16]. Since the continuous diffusion models usually start from a standard Gaussian distribution  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , Chen et al. [16] proposed to first rescale the  $\{0,1\}$ -valued variables  $\mathbf{x}_0$  to the  $\{-1,1\}$  domain as  $\hat{\mathbf{x}}_0$ , and then treat them as real values. The forward process in continuous diffusion is defined as:  $q(\hat{\mathbf{x}}_t|\hat{\mathbf{x}}_{t - 1})\coloneqq \mathcal{N}(\hat{\mathbf{x}}_t;\sqrt{1 - \beta_t}\hat{\mathbf{x}}_{t - 1},\beta_t\mathbf{I})$ .
Again,  $\beta_{t}$  denotes the corruption ratio, and we want  $\prod_{t=1}^{T}(1 - \beta_{t}) \approx 0$  such that  $\mathbf{x}_{T} \sim \mathcal{N}(\cdot)$ . The  $t$ -step marginal can thus be written as:  $q(\hat{\mathbf{x}}_t|\hat{\mathbf{x}}_0) \coloneqq \mathcal{N}(\hat{\mathbf{x}}_t; \sqrt{\bar{\alpha}_t}\hat{\mathbf{x}}_0, (1 - \bar{\alpha}_t)\mathbf{I})$  where  $\alpha_{t} = 1 - \beta_{t}$  and  $\bar{\alpha}_{t} = \prod_{\tau=1}^{t}\alpha_{\tau}$ . Similar to Eq. 5, the posterior at time  $t-1$  can be obtained by Bayes' theorem:
$$
q \left(\hat {\mathbf {x}} _ {t - 1} \mid \hat {\mathbf {x}} _ {t}, \mathbf {x} _ {0}\right) = \frac {q \left(\hat {\mathbf {x}} _ {t} \mid \hat {\mathbf {x}} _ {t - 1} , \hat {\mathbf {x}} _ {0}\right) q \left(\hat {\mathbf {x}} _ {t - 1} \mid \hat {\mathbf {x}} _ {0}\right)}{q \left(\hat {\mathbf {x}} _ {t} \mid \hat {\mathbf {x}} _ {0}\right)}, \tag {7}
$$
which is a closed-form Gaussian distribution [40]. In continuous diffusion, the denoising neural network is trained to predict the unscaled Gaussian noise  $\widetilde{\pmb{\epsilon}}_t = (\hat{\mathbf{x}}_t - \sqrt{\bar{\alpha}_t}\hat{\mathbf{x}}_0) / \sqrt{1 - \bar{\alpha}_t} = f_{\pmb{\theta}}(\hat{\mathbf{x}}_t,t)$ . The reverse process [40] can use a point estimation of  $\hat{\mathbf{x}}_0$  in the posterior:
$$
p _ {\boldsymbol {\theta}} \left(\hat {\mathbf {x}} _ {t - 1} \mid \hat {\mathbf {x}} _ {t}\right) = q \left(\hat {\mathbf {x}} _ {t - 1} \mid \hat {\mathbf {x}} _ {t}, \frac {\hat {\mathbf {x}} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} f _ {\boldsymbol {\theta}} \left(\hat {\mathbf {x}} _ {t} , t\right)}{\sqrt {\bar {\alpha} _ {t}}}\right) \tag {8}
$$
For generating discrete data, after the continuous data  $\hat{\mathbf{x}}_0$  is generated, a thresholding/quantization operation is applied to convert them back to  $\{0,1\}$ -valued variables  $\mathbf{x}_0$  as the model's prediction.
3.3 Denoising Schedule for Fast Inference
One way to speed up the inference of denoising diffusion models is to reduce the number of steps in the reverse diffusion process, which also reduces the number of neural network evaluations. The denoising diffusion implicit models (DDIMs) [101] are a class of models that apply this strategy in the continuous domain, and a similar approach can be used for discrete diffusion models [5].
Formally, when the forward process is defined not on all the latent variables  $\mathbf{x}_{1:T}$ , but on a subset  $\{\mathbf{x}_{\tau_1},\dots,\mathbf{x}_{\tau_M}\}$ , where  $\tau$  is an increasing sub-sequence of  $[1,\dots,T]$  with length  $M$ ,  $\mathbf{x}_{\tau_1} = 1$  and  $\mathbf{x}_{\tau_M} = T$ , the fast sampling algorithms directly models  $q(\mathbf{x}_{\tau_{i - 1}}|\mathbf{x}_{\tau_i},\mathbf{x}_0)$ . Due to the space limit, the detailed algorithms are described in the appendix.
We consider two types of denoising scheduled for  $\tau$  given the desired  $\operatorname{card}(\tau) < T$ : linear and cosine. The former uses timesteps such that  $\tau_{i} = \lfloor ci\rfloor$  for some  $c$ , and the latter uses timesteps such that  $\tau_{i} = \left\lfloor \cos \left(\frac{(1 - ci)\pi}{2}\right)\cdot T\right\rfloor$  for some  $c$ . The intuition for the cosine schedule is that diffusion models can achieve better generation quality when iterating more steps in the low-noise regime [85, 121, 13].
3.4 Graph-based Denoising Network
The denoising network takes as input a set of noisy variables  $\mathbf{x}_t$  and the problem instance  $s$  and predicts the clean data  $\widetilde{\mathbf{x}}_0$ . To balance both scalability and performance considerations, we adopt an anisotropic graph neural network with edge gating mechanisms [9, 54] as the backbone network for both discrete and continuous diffusion models, and the variables in the network output can be the states of either nodes, as in the case of Maximum Independent Set (MIS) problems, or edges, as in the case of Traveling Salesman Problems (TSP). Our choice of network mainly follows previous work [54, 92], as AGNN can produce the embeddings for both nodes and edges, unlike typical GNNs such as GCN [62] or GAT [107], which are designed for node embedding only. This design choice is particularly beneficial for tasks that require the prediction of edge variables.
Anisotropic Graph Neural Networks Let  $h_i^\ell$  and  $e_{ij}^\ell$  denote the node and edge features at layer  $\ell$  associated with node  $i$  and edge  $ij$ , respectively.  $\mathbf{t}$  is the sinusoidal features [106] of denoising timestep  $t$ . The features at the next layer is propagated with an anisotropic message passing scheme:
$$
\hat {\boldsymbol {e}} _ {i j} ^ {\ell + 1} = \boldsymbol {P} ^ {\ell} \boldsymbol {e} _ {i j} ^ {\ell} + \boldsymbol {Q} ^ {\ell} \boldsymbol {h} _ {i} ^ {\ell} + \boldsymbol {R} ^ {\ell} \boldsymbol {h} _ {j} ^ {\ell},
$$
$$
\boldsymbol {e} _ {i j} ^ {\ell + 1} = \boldsymbol {e} _ {i j} ^ {\ell} + \mathrm {M L P} _ {e} (\mathrm {B N} (\hat {\boldsymbol {e}} _ {i j} ^ {\ell + 1})) + \mathrm {M L P} _ {t} (\mathbf {t}),
$$
$$
\boldsymbol {h} _ {i} ^ {\ell + 1} = \boldsymbol {h} _ {i} ^ {\ell} + \alpha (\mathrm {B N} (\boldsymbol {U} ^ {\ell} \boldsymbol {h} _ {i} ^ {\ell} + \mathcal {A} _ {j \in \mathcal {N} _ {i}} (\sigma (\hat {\boldsymbol {e}} _ {i j} ^ {\ell + 1}) \odot \boldsymbol {V} ^ {\ell} \boldsymbol {h} _ {j} ^ {\ell}))),
$$
where  $U^{\ell}, V^{\ell}, P^{\ell}, Q^{\ell}, R^{\ell} \in \mathbb{R}^{d \times d}$  are the learnable parameters of layer  $\ell$ ,  $\alpha$  denotes the ReLU [66] activation, BN denotes the Batch Normalization operator [51],  $\mathcal{A}$  denotes the aggregation function SUM pooling [116],  $\sigma$  is the sigmoid function,  $\odot$  is the Hadamard product,  $\mathcal{N}_i$  denotes the neighborhoods of node  $i$ , and  $\mathrm{MLP}_{(\cdot)}$  denotes a 2-layer multi-layer perceptron.
For TSP,  $e_{ij}^{0}$  are initialized as the corresponding values in  $\mathbf{x}_t$ , and  $h_i^0$  are initialized as sinusoidal features of the nodes. For MIS,  $e_{ij}^{0}$  are initialized as zeros, and  $h_i^0$  are initialized as the corresponding values in  $\mathbf{x}_t$ . A 2-neuron and 1-neuron classification/regression head is applied to the final embeddings of  $\mathbf{x}_t$  ( $\{\mathbf{e}_{ij}\}$  for edges and  $\{\mathbf{h}_i\}$  for nodes) for discrete and continuous diffusion models, respectively.
Hyper-parameters For all TSP and MIS benchmarks, we use a 12-layer Anisotropic GNN with a width of 256 as described above.
3.5 Decoding Strategies for Diffusion-based Solvers