SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It remains to show that our solution feature mapping satisfies the variance preservation assumption, i.e., Item 3 in Assumption 1. A uniformly random solution vector  $s \in \{\pm 1\}^n$  is sampled by setting each  $s_i = 1$  with probability  $1/2$  independently. In that case, we have  $\mathbf{E}[v \cdot x] = 0$  and therefore  $\mathbf{Var}(v \cdot x) = \mathbf{E}[(v \cdot x)^2] = \sum_{i \neq j} v_i v_j \mathbf{E}[x_i x_j] = \sum_i v_i^2 = \|v\|_2^2$ , since, by the independence of  $x_i, x_j$ , the cross-terms of the sum vanish. We observe that the same hold true for the Max-Cut problem and therefore, structured feature mappings exist for Max-Cut as well (where  $L(s; G) = -s^\top L_G s$ ). We shortly mention that there also exist structured feature mappings for Max- $k$ -CSP. We refer to Theorem 4 for further details.
Remark 6 (Partial Instance Information/Instance Context). We remark that Assumption 1 allows for the "instance"  $I$  to only contain partial information about the actual cost function. For example, consider the setting where each sampled instance is an unweighted graph  $G$  but the cost oracle takes the form  $\mathcal{O}(G,s) = (L_G)^{\flat}M(ss^{\top})^{\flat}$  for a matrix  $M_{ij} = a_i$  when  $i = j$  and  $M_{ij} = 0$  otherwise. This cost function models having a unknown weight function, i.e., the weight of edge  $i$  of  $G$  is  $a_i$  if edge  $i$  exists in the observed instance  $G$ , on the edges of the observed unweighted graph  $G$ , that the algorithm has to learn in order to be able to find the minimum or maximum cut. For simplicity, in what follows, we will continue referring to  $I$  as the instance even though it may only contain partial information about the cost function of the underlying combinatorial problem.
Maximum-Weight-Bipartite-Matching and TSP. The Maximum-Weight-Bipartite-Matching (MWBP) problem is another graph problem that, given a bipartite graph  $G$  with  $n$  nodes and  $m$  edges, asks for the maximum-weight matching. The feature vector corresponding to a matching can be represented as a binary matrix  $R \in \{0,1\}^{n \times n}$  with  $\sum_{j} R_{ij} = 1$  for all  $i$  and  $\sum_{i} R_{ij} = 1$  for all  $j$ , i.e.,  $R$  is a permutation matrix. Therefore, for a candidate matching  $s$ , we set  $\psi_S(s)$  to be the matrix  $R$  defined above. Moreover, the feature vector of the graph is the (negative flattened) adjacency matrix  $E^{\flat}$ . The cost oracle is then  $L(R;E) = \sum_{ij} E_{ij} M_{ij} R_{ij}$  perhaps for an unknown weight matrix  $M_{ij}$  (see Remark 6). For the Traveling Salesman Problem (TSP) the feature vector is again a matrix  $R$  with the additional constraint that  $R$  has to represent a single cycle (a tour over all cities). The cost function for TSP is again  $L(R;E) = \sum_{ij} E_{ij} M_{ij} R_{ij}$ . One can check that those representations of the instance and solution satisfy the assumptions of Items 1 and 4. Showing that the variance of those representations has a polynomial lower bound is more subtle and we refer the reader to the Supplementary Material.
We shortly mention that the solution generators for Min-Cut and Maximum-Weight-Bipartite-Matching are also efficiently samplable.
3 Optimization Landscape
Exponential Families as Solution Generators. A natural candidate to construct our family of solution generators is to consider the distribution that assigns to each solution  $s \in S$  and instance  $I \in \mathcal{I}$  mass proportional to its score  $\exp(-\tau L(s; I)) = \exp(-\tau \psi_{\mathcal{I}}(I)^{\top} M \psi_S(s)) = \exp(-\tau z^{\top} M x)$ .
Figure 1: In the left plot, we show the landscape of the "vanilla" objective of Eq.(1) for the feature domain  $X = \{(1,0),(2,2),(0,2)\}$  and linear cost oracle  $c\cdot x$  for  $c = (-3, - 3)$ . We see that the "vanilla" objective is minimized at the direction of  $-c$ , i.e., along the direction  $\tau (1,1)$  for  $\tau \rightarrow +\infty$ . We observe the two issues described in Section 3, i.e., that the true minimizer is a point at infinity, and that gradients vanish so gradient descent may get trapped in sub-optimal solutions, (e.g., in the upper-right corner if initialized in the top corner). In the middle plot, we show the landscape of the entropy-regularized objective of Eq.(3) that makes the minimizer finite and brings it closer to the origin. Observe that even if a gradient iteration is initialized in the top corner it will eventually converge to the minimizer; however the rate of convergence may be very slow. The right plot corresponds to the loss objective where we combine a mixture of exponential families as solution generator, as in Eq.(5), and the entropy regularization approach. We observe that we are able to obtain a benign (quasar-convex) landscape via the entropy regularization while the mixture-generator guarantees non-vanishing gradients.
for some "temperature" parameter  $\tau$ , where  $\psi_{\mathcal{I}}$  and  $\psi_S$  are the feature mappings promised to exist due to Assumption 1,  $z = \psi_{\mathcal{I}}(I)$ , and,  $x = \psi_S(s)$ . Note that as long as  $\tau \to +\infty$ , this distribution tends to concentrate on solutions that achieve small loss.
Remark 7. To construct the above solution sampler one could artificially query specific solutions to the cost oracle of Definition 1 and try to learn the cost matrix  $M$ . However, we remark that our goal (see Definition 2) is to show that we can train a parametric family via gradient-based methods so that it generates (approximately) optimal solutions and not to simply learn the cost matrix  $M$  via some other method and then use it to generate good solutions.
Obstacle I: Minimizers at Infinity. One could naturally consider the parametric family  $\phi(x;z;W) \propto \exp(z^\top Wx)$  (note that with  $W = -\tau M$ , we recover the distribution of the previous paragraph) and try to perform gradient-based methods on the loss (recall that  $L(x;z) = z^\top Mx$ )<sup>1</sup>
$$
\mathcal {L} (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} \underset {x \sim \phi (\cdot ; z; W)} {\mathbf {E}} \left[ z ^ {\top} M x \right]. \tag {1}
$$
The question is whether gradient updates on the parameter  $W$  eventually converge to a matrix  $\overline{W}$  whose associated distribution  $\phi(\overline{W})$  generates near-optimal solutions (note that the matrix  $-\tau M$  with  $\tau \rightarrow +\infty$  is such a solution). After computing the gradient of  $\mathcal{L}$ , we observe that
$$
\nabla_ {W} \mathcal {L} (W) \cdot M = \operatorname {V a r} _ {z \sim \mathcal {R}, x \sim \phi (\cdot ; z; W)} [ z ^ {\top} M x ] \geq 0,
$$
where the inner product between two matrices  $A \cdot B$  is the trace  $\mathrm{Tr}(A^\top B) = \sum_{i,j} A_{ij} B_{ij}$ . This means that the gradient field of  $\mathcal{L}$  always has a contribution to the direction of  $M$ . Nevertheless the actual minimizer is at infinity, i.e., it corresponds to the point  $\overline{W} = -\tau M$  when  $\tau \to +\infty$ . While the correlation with the optimal point is positive (which is encouraging), having such contribution to this direction is not a sufficient condition for actually reaching  $\overline{W}$ . The objective has vanishing gradients at infinity and gradient descent may get trapped in sub-optimal stationary points, see the left plot in Figure 1.
Solution I: Quasar Convexity via Entropy Regularization. Our plan is to try and make the objective landscape more benign by adding an entropy-regularizer. Instead of trying to make the objective convex (which may be too much to ask in the first place) we are able obtain a much better landscape with a finite global minimizer and a gradient field that guides gradient descent to the minimizer. Those properties are described by the so-called class of "quasar-convex" functions. Quasar convexity (or weak quasi-convexity [HMR16]) is a well-studied notion in optimization [HMR16, HSS20, LV16, ZMB $^{+}$ 17, HLSS15] and can be considered as a high-dimensional generalization of unimodality.
<sup>1</sup>We note that we overload the notation and assume that our distributions generate directly the featurizations  $z$  (resp.  $x$ ) of  $I$  (resp.  $s$ ).
Definition 3 (Quasar Convexity [HMR16, HSS20]). Let  $\gamma \in (0,1]$  and let  $\overline{x}$  be a minimizer of the differentiable function  $f: \mathbb{R}^n \to \mathbb{R}$ . The function  $f$  is  $\gamma$ -quasar-convex with respect to  $\overline{x}$  on a domain  $D \subseteq \mathbb{R}^n$  if for all  $x \in D$ ,  $\nabla f(x) \cdot (x - \overline{x}) \geq \gamma (f(x) - f(\overline{x}))$ .
In the above definition, notice that the main property that we need to establish is that the gradient field of our objective correlates positively with the direction  $W - \overline{W}$ , where  $\overline{W}$  is its minimizer. We denote by  $H: \mathcal{W} \to \mathbb{R}$  the negative entropy of  $\phi(W)$ , i.e.,
$$
H (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} _ {x \sim \phi (\cdot ; z; W)} \left[ \log \phi (x; z; W) \right], \tag {2}
$$
and consider the regularized objective
$$
\mathcal {L} _ {\lambda} (W) = \mathcal {L} (W) + \lambda H (W), \tag {3}
$$
for some  $\lambda > 0$ . We show (follows from Lemma 4) that the gradient-field of the regularized objective indeed "points" towards a finite minimizer (the matrix  $\overline{W} = -M / \lambda$ ):
$$
\nabla_ {W} \mathcal {L} _ {\lambda} (W) \cdot (W + M / \lambda) =
$$
$$
\operatorname {V a r} \left[ z ^ {\top} (W + M / \lambda) x \right] \geq 0, \tag {4}
$$
where the randomness is over  $z \sim \mathcal{R}, x \sim \phi(\cdot; z; W)$ . Observe that now the minimizer of  $\mathcal{L}_{\lambda}$  is the point  $-M / \lambda$ , which for  $\lambda = \mathrm{poly}(\epsilon, \alpha, 1/C, 1/D_S, 1/D_{\mathcal{I}})$  (these are the parameters of Assumption 1) is promised to yield a solution sampler that generates  $\epsilon$ -sub-optimal solutions (see also Proposition 2 and Appendix C). Having the property of Equation (4) suffices for showing that a gradient descent iteration (with an appropriately small step-size) will eventually converge to the minimizer.
Obstacle II: Vanishing Gradients. While we have established that the gradient field of the regularized objective "points" towards the right direction, the regularized objective still suffers from vanishing gradients, see the middle plot in Figure 1. In other words,  $\gamma$  in the definition of quasar convexity (Definition 3) may be exponentially small, as it is proportional to the variance of the random variable  $z^{\top}(W + M / \lambda)x$ , see Equation (4). As we see in the middle plot of Figure 1, the main issue is the vanishing gradient when  $W$  gets closer to the minimizer  $-M / \lambda$  (towards the front-corner). For simplicity, consider the variance along the direction of  $W$ , i.e.,  $\mathbf{Var}[z^{\top}Wx]$  and recall that  $x$  is generated by the density  $\exp(z^{\top}Wx) / (\sum_{x \in X} \exp(z^{\top}Wx))$ . When  $\|W\|_2 \to +\infty$  we observe that the value  $z^{\top}Wx$  concentrates exponentially fast to  $\max_{x \in X} z^{\top}Wx$  (think of the convergence of the soft-max to the max function). Therefore, the variance  $\mathbf{Var}[z^{\top}Wx]$  may vanish exponentially fast making the convergence of gradient descent slow.
Solution II: Non-Vanishing Gradients via Fast/Slow Mixture Generators. We propose a fix to the vanishing gradients issue by using a mixture of exponential families as a solution generator. We define the family of solution generators  $\mathcal{P} = \{p(W):W\in \mathcal{W}\}$  to be
$$
\mathcal {P} = \left\{\left(1 - \beta^ {\star}\right) \phi (W) + \beta^ {\star} \phi \left(\rho^ {\star} W\right): W \in \mathcal {W} \right\}, \tag {5}
$$
for a (fixed) mixing parameter  $\beta^{\star}$  and a (fixed) temperature parameter  $\rho^{\star}$ . The main idea is to have the first component of the mixture to converge fast to the optimal solution (to  $-M / \lambda$ ) while the second "slow" component that has parameter  $\rho^{\star}W$  stays closer to the uniform distribution over solutions that guarantees non-trivial variance (and therefore non-vanishing gradients).
More precisely, taking  $\rho^{\star}$  to be sufficiently small, the distribution  $\phi (\rho^{\star}W)$  is almost uniform over the solution space  $\psi_S(S)$ . Therefore, in Equation (4), the almost uniform distribution component of the mixture will add to the variance and allow us to show a lower bound. This is where Item 3 of Assumption 1 comes into play and gives us the desired non-trivial variance lower bound under the uniform distribution. We view this fast/slow mixture technique as an interesting insight of our work: we use the "fast" component (the one with parameter  $W$ ) to actually reach the optimal solution  $-M / \lambda$  and we use the "slow" component (the one with parameter  $\rho^{\star}W$ ) that essentially generates random solutions) to preserve a non-trivial variance lower bound during optimization.
4 Complete, Compressed and Efficiently Optimizable Solution Generators
In this section, we discuss the main results that imply Theorem 1: the family  $\mathcal{P}$  of Equation (5) is complete, compressed and efficiently estimizable (for some choice of  $\beta^{\star},\rho^{\star}$  and  $\mathcal{W}$ ).
Completeness. First, we show that the family of solution generators of Equation (5) is complete. For the proof, we refer to Proposition 2 in Appendix C. At a high-level, we to pick  $\beta^{\star}$ ,  $\rho^{\star}$  to be of order poly  $(\epsilon, \alpha, 1/C, 1/D_S, 1/D_{\mathcal{I}})$ . This yields that the matrix  $\overline{W} = -M/\lambda$  is such that  $\mathcal{L}(\overline{W}) \leq \mathrm{opt} + \epsilon$ , where  $M$  is the matrix of Item 2 in Assumption 1 and  $\lambda$  is poly  $(\epsilon/[I])$ . To give some intuition about this choice of matrix, let us see how  $\mathcal{L}(\overline{W})$  behaves. By definition, we have that
$$
\mathcal {L} (\overline {{W}}) = \underset {z \sim \mathcal {R}} {\mathbf {E}} \underset {x \sim p (\cdot ; z; \overline {{W}})} {\mathbf {E}} \left[ z ^ {\top} M x \right],
$$
where the distribution  $p$  belongs to the family of Equation (5), i.e.,  $p(\overline{W}) = (1 - \beta^{\star})\phi (\overline{W}) + \beta^{\star}\phi (\rho^{\star}W)$ . Since the mixing weight  $\beta^{\star}$  is small, we have that  $p(\overline{W})$  is approximately equal to  $\phi (\overline{W})$ . This means that our solution generator draws samples from the distribution whose mass at  $x$  given instance  $z$  is proportional to  $\exp (-z^{\top}Mx / \lambda)$  and, since  $\lambda >0$  is very small, the distribution concentrates to solutions  $x$  that tend to minimize the objective  $z^{\top}Mx$ . This is the reason why  $\overline{W} = -M / \lambda$  is close to opt in the sense that  $\mathcal{L}(\overline{W})\leq \mathrm{opt} + \epsilon$ .
Compression. As a second step, we show (in Proposition 3, see Appendix D) that  $\mathcal{P}$  is a compressed family of solution generators. This result follows immediately from the structure of Equation (5) (observe that  $W$  has  $n_X$ $n_Z$  parameters) and the boundedness of  $\overline{W} = -M / \lambda$ .
Efficiently Optimizable. The proof of this result essentially corresponds to the discussion provided in Section 3. Our main structural result shows that the landscape of the regularized objective with the fast/slow mixture solution-generator is quasar convex. More precisely, we consider the following objective:
$$
\mathcal {L} _ {\lambda} (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} \underset {x \sim p (\cdot ; z; W)} {\mathbf {E}} [ z ^ {\top} M x ] + \lambda R (W), \tag {1}
$$
where  $p(W)$  belongs in the family  $\mathcal{P}$  of Equation (5) and  $R$  is a weighted sum of two negative entropy regularizers (to be in accordance with the mixture structure of  $\mathcal{P}$ ), i.e.,  $R(W) = (1 - \beta^{\star})H(W) + \beta^{\star} / \rho^{\star}H(\rho^{\star}W)$ . Our main structural results follow (for the proof, see Appendix E.1).
Proposition 1 (Quasar Convexity). Consider  $\epsilon >0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds. The function  $\mathcal{L}_{\lambda}$  of Equation (1) with domain  $\mathcal{W}$  is  $\mathrm{poly}(\epsilon ,\alpha ,1 / C,1 / D_S,1 / D_{\mathcal{I}})$ -quasar convex with respect to  $-M / \lambda$  on the domain  $\mathcal{W}$ .
Since  $\rho^{\star}$  is small (by Proposition 2),  $H(\rho^{\star}W)$  is essentially constant and close in value to the negative entropy of the uniform distribution. Hence, the effect of  $R(W)$  during optimization is essentially the same as that of  $H(W)$  (since  $\beta^{\star}$  is close to 0). We show that  $\mathcal{L}_{\lambda}$  is quasar convex with a non-trivial parameter  $\gamma$  (see Proposition 1). We can then apply (in a black-box manner) the convergence results from [HMR16] to optimize it using projected SGD. We show that SGD finds a weight matrix  $\widehat{W}$  such that the solution generator  $p(\widehat{W})$  generates solutions achieving actual loss  $\mathcal{L}$  close to that of the near optimal matrix  $\overline{W} = -M / \lambda$ , i.e.,  $\mathcal{L}(\widehat{W}) \leq \mathcal{L}(\overline{W}) + \epsilon$ . For further details, see Appendix E.3.
In this section, we investigate experimentally the effect of our main theoretical contributions, the entropy regularizer (see Equation (2)) and the fast/slow mixture scheme (see Equation (5)). We try to find the Max-Cut of a fixed graph  $G$ , i.e., the support of the prior  $\mathcal{R}$  is a single graph. Similarly to our theoretical results, our sampler is of the form  $e^{\mathrm{score}(s;w)}$ , where  $s \in \{-1,1\}^{n}$  (here  $n$  is the number of nodes in the graph) is a candidate solution of the Max-Cut problem. For the score function we used a simple linear layer (left plot of Figure 2) and a 3-layer ReLU network (right plot of Figure 2).
Small Graph Instances. Focusing on instances where the number of nodes  $n$  is small (say  $n = 15$ ), we can explicitly compute the density function and work with an exact sampler. We generate 100 random  $G(n, p)$  graphs with  $n = 15$  nodes and  $p = 0.5$  and train solution generators using both the "vanilla" loss  $\mathcal{L}$  and the entropy-regularized loss  $\mathcal{L}_{\lambda}$  with the fast/slow mixture scheme. We perform 600 iterations and, for the entropy regularization, we progressively decrease the regularization weight, starting from 10, and dividing it by 2 every 60 iterations. Out of the 100 trials we found that our proposed objective was always able to find the optimal cut while the model trained with the vanilla loss was able to find it for approximately  $65\%$  of the graphs (for 65 out of 100 using the linear network and for 66 using the ReLU network).
Hence, our experiments demonstrate that while the unregularized objective is often "stuck" at suboptimal solutions – and this happens even for very small instances ( $n = 15$  nodes) – of the Max-Cut problem, the objective motivated by our theoretical results is able to find the optimal solutions. For further details, see Appendix I.
Figure 2: Plot of the Max-Cut value trajectory of the "vanilla" objective and entropy-regularized objective with the slow/fast mixture scheme. We remark that we plot the value of the cut of each iteration (and not the value of the regularized-loss). On the horizontal axis we plot the number of iterations and on the vertical axis we plot the achieved value of the cut. Both graphs used were random  $G(n,p)$  graphs generated with  $n = 15$  nodes and edge probability  $p = 0.5$ . For the left plot we used a linear network (the same exponential family as the one used in our theoretical results). For the right plot we used a simple 3-Layer ReLU network to generate the scores. We observe that the "vanilla" loss gets stuck on sub-optimal solutions.
Large Graph Instances. A natural question is whether this improvement is also apparent in larger graphs. We focus on the case of random  $d$ -regular graphs with  $n$  nodes. It is well-known that for this family of graphs, with high probability as  $n \to \infty$ , the size of the maximum cut satisfies  $\mathrm{MaxCut}(G(n,d)) = n(d/4 + P_{\star}\sqrt{d/4} + o_d(\sqrt{d})) + o(n)$ , where  $P_{\star} \approx 0.7632$  is a universal constant [DMS17]. We aim to find a good approximation for the normalized cut-value, defined as  $(\mathrm{cut\_value}/n - d/4)/\sqrt{d/4}$ , which (roughly speaking) takes values in  $[0, P_{\star}]$ . We obtain approximate random samples from the density  $e^f$  using the Metropolis-Adjusted Langevin Algorithm (MALA). In particular, an approximate sample from this density is obtained by the Euler-Maruyama method for simulating the Langevin diffusion:  $x_{k+1} = x_k + \tau \nabla f(x_k) + \sqrt{2\tau} \xi_k$ , where  $\xi_k$  is an independent Gaussian vector  $\mathcal{N}(0, I)$ . MALA incorporates an additional step based on the Metropolis-Hastings algorithm (see [Bes94, SK21]). In our case, the score function  $f$  is a simple 3-layer ReLU network. In our experiments for 3 larger random regular graphs (600 nodes) using the fast/slow mixing technique along entropy regularization we see that our method leads to improvements over the vanilla objective. Plots of the trajectories of the vanilla and our method can be found in Figure 3. In the horizontal axis we plot the iterations and in the vertical axis we plot the normalized cut score of each method (higher is better) – we stop the plot of the vanilla trajectory after 200 iterations because we observed that its output has fully converged and is stuck.
Figure 3: Plot of the Max-Cut value trajectory of the "vanilla" objective and entropy-regularized objective with the slow/fast mixture scheme on large instances. On the horizontal axis we plot the number of iterations and on the vertical axis we plot the achieved value of the normalized cut-value. The 3 graphs used were random  $G(n,p)$  graphs generated with  $n = 600$  nodes and edge probability  $p = 0.5$ . We stop the plot of the vanilla trajectory after 200 iterations because we observed that its output has fully converged and is stuck.
This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.
Constantine Caramanis was partially supported by the NSF IFML Institute (NSF 2019844), and the NSF AI-EDGE Institute (NSF 2112471).
Dimitris Fotakis has been partially supported by the Hellenic Foundation for Research and Innovation (H.F.R.I.) under the "First Call for H.F.R.I. Research Projects to support Faculty members and Researchers and the procurement of high-cost research equipment grant", project BALSAM, HFRI-FM17-1424.
Christos Tzamos was partially supported by the NSF IFML Institute (NSF 2144298).