SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
g (w) = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ c \cdot x ] + \lambda H (w), \quad \phi (x; w) = \frac {\exp (w \cdot x)}{\sum_ {y \in \mathcal {X}} \exp (w \cdot y)},
$$
where  $H$  is the negative entropy regularizer, i.e.,
$$
H (w) = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ \log \phi (x; w) ] = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ w \cdot x ] - \log \left(\sum_ {y \in \mathcal {X}} e ^ {w \cdot y}\right).
$$
The gradient of  $g$  with respect to  $w \in \mathcal{W}$  is equal to
$$
\nabla_ {w} g (w) = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ (c \cdot x) \nabla_ {w} \log \phi (x; w) ] + \lambda \nabla_ {w} H (w).
$$
$$
\nabla_ {w} \log \phi (x; w) = \nabla_ {w} \left(w \cdot x - \log \sum_ {y \in \mathcal {X}} e ^ {w \cdot y}\right) = x - \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ],
$$
$$
\nabla H (w) = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ] + \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ (w \cdot x) \nabla_ {w} \log \phi (x; w) ] - \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ] = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ (w \cdot x) \nabla_ {w} \log \phi (x; w) ].
$$
So, we get that
$$
\nabla_ {w} g (w) = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} \left[ \left(\left(c + \lambda w\right) \cdot x\right) x \right] - \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} \left[ \left(c + \lambda w\right) \cdot x \right] \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ].
$$
Lemma 5 (Gradient with Regularization and Mixing). For any  $\epsilon >0$ , for the family of solution generators  $\mathcal{P} = \{p(\cdot ;w) = (1 - \beta^{\star})\phi (\cdot ;w) + \beta^{\star}\phi (\cdot ;\rho^{\star}w):w\in \mathcal{W}\}$  and the objective  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  of Equation (3), it holds that
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) = (1 - \beta^ {\star}) \operatorname {V a r} _ {x \sim \phi (\cdot ; w)} [ (c + \lambda w) \cdot x ] + \beta^ {\star} \rho^ {\star} \operatorname {V a r} _ {x \sim \phi (\cdot ; \rho^ {\star} w)} [ (c + \lambda w) \cdot x ],
$$
for any  $\lambda > 0$ .
Proof. Let us first consider the scaled parameter  $\rho w\in \mathcal{W}$  for some  $\rho >0$  . Then it holds that
$$
\nabla_ {w} \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ c \cdot x ] = \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} \left[ (c \cdot x) \left(\rho x - \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ \rho x ]\right) \right] = \rho \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} \left[ (c \cdot x) \left(x - \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ x ]\right) \right].
$$
Moreover, the negative entropy regularizer at  $\rho w$  is
$$
H (\rho w) = \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ (\rho w) \cdot x ] - \log \sum_ {y \in \mathcal {X}} e ^ {(\rho w) \cdot y}.
$$
$$
\nabla_ {w} H (\rho w) = \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ (\rho w \cdot x) \nabla_ {w} \log \phi (x; \rho w) ] = \rho^ {2} \left(\underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ (w \cdot x) x ] - \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ w \cdot x ] \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ x ]\right).
$$
We consider the objective function  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  to be defined as follows: first, we take
i.e.,  $p(\cdot; w)$  is the mixture of the probability measures  $\phi(\cdot; w)$  and  $\phi(\cdot; \rho^{\star} w)$  with weights  $1 - \beta^{\star}$  and  $\beta^{\star}$  respectively for some scale  $\rho^{\star} > 0$ . Moreover, we take  $R^{\mathrm{vec}}(w) = (1 - \beta^{\star}) H(w) + \frac{\beta^{\star}}{\rho^{\star}} H(\rho^{\star} w)$ .
Then we define our regularized loss  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  to be
$$
\mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) = \underset {x \sim p (\cdot ; w)} {\mathbf {E}} [ c \cdot x ] + \lambda R ^ {\mathrm {v e c}} (w).
$$
Using Lemma 4 and the above calculations, we have that
$$
\begin{array}{l} \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) = (1 - \beta^ {\star}) \nabla_ {w} \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ c \cdot x ] + \lambda (1 - \beta^ {\star}) \nabla_ {w} H (w) + \beta^ {\star} \nabla_ {x \sim \phi (\cdot ; \rho^ {\star} x)} \underset {x \sim \phi (\cdot ; \rho^ {\star} x)} {\mathbf {E}} [ c \cdot x ] + \lambda \frac {\beta^ {\star}}{\rho^ {\star}} \nabla_ {w} H (\rho^ {\star} w) \\ = (1 - \beta^ {\star}) \left(\mathbf {E} _ {x \sim \phi (\cdot ; w)} [ ((c + \lambda w) \cdot x) x ] - \mathbf {E} _ {x \sim \phi (\cdot ; w)} [ (c + \lambda w) \cdot x ] \mathbf {E} _ {x \sim \phi (\cdot ; w)} [ x ]\right) + \\ + \beta^ {\star} \rho^ {\star} \left(\underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] - \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x ]\right). \\ \end{array}
$$
The above calculations yield
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) = (1 - \beta^ {\star}) \mathbf {V a r} _ {x \sim \phi (\cdot ; w)} [ (c + \lambda w) \cdot x ] + \beta^ {\star} \rho^ {\star} \mathbf {V a r} _ {x \sim \phi (\cdot ; \rho^ {\star} w)} [ (c + \lambda w) \cdot x ],
$$
and this concludes the proof.
The above correlation being positive intuitively means that performing gradient descent to  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  gives that the parameter  $w$  converges to  $-c / \lambda$ , the point that achieves completeness for that objective.
However, to obtain fast convergence, we need to show that the above correlation is non-trivial. This means that our goal in order to prove Lemma 3 is to provide a lower bound for the above quantity, i.e., it suffices to give a non-trivial lower bound for the variance of the random variable  $(c + \lambda w) \cdot x$  with respect to the probability measure  $\phi(\cdot; \rho^{\star}w)$ . It is important to note that in the above statement we did not fix the value of  $\rho^{\star}$ . We can now make use of Proposition 8. Intuitively, by taking the scale parameter appearing in the mixture  $\rho^{\star}$  to be sufficiently small, we can manage to provide a lower bound for the variance of  $(c + \lambda w) \cdot x$  with respect to the almost uniform measure, i.e., the second summand of the above right-hand side expression has significant contribution. We remark that  $\rho$  corresponds to the inverse temperature parameter. Hence, our previous analysis essentially implies that policy gradient on combinatorial optimization potentially works if the variance  $\mathbf{Var}_{x \sim \phi(\cdot; \rho w)}[(c + \lambda w) \cdot x]$  is non-vanishing at high temperatures  $1 / \rho$ .
The proof of Lemma 3. Recall that
$$
\phi (x; w) = \frac {e ^ {w \cdot x}}{\sum_ {y \in \mathcal {X}} e ^ {w \cdot y}}.
$$
Let also  $D$  be the diameter of  $\mathcal{X}$  and  $B$  the diameter of  $\mathcal{Z}$ . Recall from Lemma 5 that we have that
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) = (1 - \beta^ {\star}) \mathbf {V a r} _ {x \sim p (\cdot ; w)} [ (c + \lambda w) \cdot x ] + \beta^ {\star} \rho^ {\star} \mathbf {V a r} _ {x \sim p (\cdot ; \rho^ {\star} w)} [ (c + \lambda w) \cdot x ],
$$
where the scale parameter  $\rho^{\star} > 0$  is to be decided. This means that
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) \geq \beta^ {\star} \rho^ {\star} \operatorname {V a r} _ {x \sim \phi (\cdot ; \rho^ {\star} w)} [ (c + \lambda w) \cdot x ].
$$
Our goal is now to apply Proposition 8 in order to lower bound the above variance. Applying Proposition 8 for  $\mu \gets \phi(\cdot; \rho^{\star}w)$ ,  $c \gets c + \lambda w \in \mathcal{Z}$  and, so for some absolute constant  $C_0$ , we can pick
$$
\rho^ {\star} = C _ {0} \frac {\alpha}{B D ^ {3}}.
$$
$$
\operatorname {V a r} _ {x \sim \phi (\cdot ; \rho^ {*} w)} [ (c + \lambda w) \cdot x ] \geq \Omega (\alpha \| c + \lambda w \| _ {2} ^ {2}).
$$
This implies the desired result since
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) \geq C _ {0} \beta^ {\star} \alpha^ {2} / \left(B D ^ {3}\right) \| c + \lambda w \| _ {2} ^ {2}.
$$
E.3 Convergence for Quasar Convex Functions
The fact that  $\mathcal{L}_{\lambda}$  is quasar convex with respect to  $-M / \lambda$  implies that projected SGD converges to that point in a small number of steps and hence the family  $\mathcal{P}$  is efficiently estimizable. The analysis is standard (see e.g., [HMR16]). For completeness a proof can be found in Appendix E.3.
Proposition 7 (Convergence). Consider  $\epsilon > 0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds with parameters  $C, D_S, D_{\mathcal{I}}, \alpha$ . Let  $W_1, \ldots, W_T$  be the updates of the SGD algorithm with projection set  $\mathcal{W}$  performed on  $\mathcal{L}_{\lambda}$  of Equation (2) with appropriate step size and parameter  $\lambda$ . Then, for the non-regularized objective  $\mathcal{L}$ , it holds that
$$
\underset {t \sim U ([ T ])} {\mathbf {E}} \left[ \mathcal {L} (W _ {t}) \right] \leq \mathcal {L} (- M / \lambda) + \epsilon ,
$$
when  $T\geq \mathrm{poly}(1 / \epsilon ,1 / \alpha ,C,D_S,D_{\mathcal{I}},\| W_0 + M / \lambda \|_{\mathrm{F}})$
Our next goal is to use Proposition 1 and show that standard projected SGD on the objective  $\mathcal{L}_{\lambda}$  converges in a polynomial number of steps. The intuition behind this result is that since the correlation between  $\nabla \mathcal{L}_{\lambda}(W)$  and the direction  $M + \lambda W$  is positive and non-trivial, the gradient field drives the optimization method towards the point  $-M / \lambda$ .
Proof. Consider the sequence of matrices  $W_{1}, \ldots, W_{t}, \ldots, W_{T}$  generated by applying PSGD on  $\mathcal{L}_{\lambda}$  with step size  $\eta$  (to be decided) and initial parameter vector  $W_{0} \in \mathcal{W}$ . We have that  $\mathcal{L}_{\lambda}$  is  $\gamma$ -quasar convex and is also  $O(\Gamma)$ -weakly smooth since we now show that it is  $\Gamma$ -smooth.
Lemma 6.  $\mathcal{L}_{\lambda}$  is poly  $(D_S,D_{\mathcal{I}},C)$  -smooth.
$$
\| \nabla \mathcal {L} _ {\lambda} (W) \| _ {\mathrm {F}} ^ {2} = \| \mathbf {E} _ {z \sim \mathcal {R}} [ \nabla \mathcal {L} _ {\lambda , z} (W) ] \| _ {\mathrm {F}} ^ {2} \leq \mathbf {E} _ {z \sim \mathcal {R}} \| z \| _ {2} ^ {2} \| \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (z ^ {\top} W) \| _ {\mathrm {F}} ^ {2}.
$$
It suffices to show that  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  is smooth. Recall that
$$
\begin{array}{l} \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) = (1 - \beta^ {\star}) \left(\mathbf {E} _ {x \sim \phi (\cdot ; w)} [ ((c + \lambda w) \cdot x) x ] - \mathbf {E} _ {x \sim \phi (\cdot ; w)} [ (c + \lambda w) \cdot x ] \mathbf {E} _ {x \sim \phi (\cdot ; w)} [ x ]\right) + \\ + \beta^ {\star} \rho^ {\star} \left(\underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] - \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x ]\right). \\ \end{array}
$$
As mentioned in [HMR16], a function  $f$  is  $\Gamma$ -weakly smooth if for any point  $\theta$ ,  $\| \nabla f(\theta) \|^{2} \leq \Gamma (f(\theta) - f(\theta^{\star}))$ . Moreover, a function  $f$  that is  $\Gamma$ -smooth (in the sense  $\| \nabla^2 f \| \leq \Gamma$ ), is also  $O(\Gamma)$ -weakly smooth.
$$
\| \nabla_ {w} ^ {2} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \| _ {\mathrm {F}} ^ {2} \leq (1 - \beta^ {\star}) (A _ {1} + A _ {2}) + \beta^ {\star} \rho^ {\star} (A _ {3} + A _ {4}),
$$
$$
A _ {1} = \left\| \nabla_ {w} \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] \right\| _ {\mathrm {F}} ^ {2}, A _ {2} = \left\| \nabla_ {w} \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ] \right\| _ {\mathrm {F}} ^ {2},
$$
$$
A _ {3} = \left\| \nabla_ {w} \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] \right\| _ {\mathrm {F}} ^ {2}, A _ {4} = \left\| \nabla_ {w} \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x ] \right\| _ {\mathrm {F}} ^ {2}.
$$
Standard computation of these values yields that, since  $D_S$  and  $D_{\mathcal{I}}$  are bounds to  $x$  and  $z$  respectively, we have that  $\mathcal{L}_{\lambda}$  is smooth with parameter poly  $(D_S, D_{\mathcal{I}}, C)$ .
Let  $V$  be the variance of the unbiased estimator used for  $\nabla_W\mathcal{L}_\lambda (W)$ . We can apply the next result of [HMR16].
Lemma 7 ([HMR16]). Suppose the objective function  $f$  is  $\gamma$ -weakly quasi convex and  $\Gamma$ -weakly smooth, and let  $r(\cdot)$  be an unbiased estimator for  $\nabla f(\theta)$  with variance  $V$ . Moreover, suppose the global minimum  $\overline{\theta}$  belongs to  $\mathcal{W}$ , and the initial point  $\theta_0$  satisfies  $\| \theta_0 - \overline{\theta} \|_2 \leq R$ . Then projected stochastic gradient descent with a proper learning rate returns  $\theta_T$  in  $T$  iterations with expected error
$$
\underset {t \sim U ([ T ])} {\mathbf {E}} f (\theta_ {t}) - f (\overline {{\theta}}) \leq \max  \left\{\frac {\Gamma R ^ {2}}{\gamma^ {2} T}, \frac {R \sqrt {V}}{\gamma \sqrt {T}} \right\}.
$$
We apply the above result to  $\mathcal{L}_{\lambda}$  in order to find matrices  $W_{1},\ldots ,W_{T}$  that achieve good loss on average compared to  $-M / \lambda$ . Moreover, using a batch SGD update, we can take  $V$  to be also polynomial in the crucial parameters of the problem. We note that one can adapt the above convergence proof and show that the actual loss  $\mathcal{L}$  (and not the loss  $\mathcal{L}_{\lambda}$ ) are close after sufficiently many iterations (as indicated by the above lemma). We know that the Frobenius norm of the gradient of  $\mathcal{L}(W)$  is at most of order  $O(D_{\mathcal{I}}^{2}CD_{S}^{2})$ . We can apply the mean value theorem in high dimensions (by taking  $\mathcal{W}$  to be an open ball of radius  $O(B)$ ) and this yields that the difference between the values of  $\mathcal{L}(W_T)$  and  $\mathcal{L}(-M / \lambda)$  is at most  $D_{\mathcal{I}}^{2}CD_{S}^{2}\| W_{t} + M / \lambda \|_{\mathrm{F}}^{2}$ . However, the right-hand side is upper bounded by the correlation between  $\nabla \mathcal{L}_{\lambda}(W_t)$  and  $W_{t} + M / \lambda$ . Hence, we can still use this correlation as a potential in order to minimize  $\mathcal{L}$ . This implies that the desired convergence guarantee holds as long as  $T\geq \mathrm{poly}(1 / \epsilon ,1 / \alpha ,C,D_S,D_\mathcal{I},\| W_0 + M / \lambda \|_{\mathrm{F}})$ .
F Deferred Proofs: Variance under Almost Uniform Distributions
This section is a technical section that states some properties of exponential families. We use some standard notation, such as  $w$  and  $x$ , for the statements and the proofs but we underline that these symbols do not correspond to the notation in the main body of the paper.
We consider the parameter space  $\Theta$  and for any parameter  $w\in \Theta$ , we define the probability distribution  $\phi (\cdot ;w)$  over a space  $\mathcal{X}$  with density
In this section, our goal is to relate the variance of  $c \cdot x$  under the measure  $\phi(\cdot; 0)$  (uniform case) and  $\phi(\cdot; \rho^{\star} w)$  for some  $w \in \mathcal{W}$  and some sufficiently small  $\rho^{\star}$  (almost uniform case). The main result of this section follows.
Proposition 8 (Variance Lower Bound Under Almost Uniform Distributions). Assume that the variance of  $c \cdot x$  under the uniform distribution over  $\mathcal{X}$ , whose diameter is  $D$ , is lower bounded by  $\alpha \| c \|_2^2$ . Moreover, assume that  $w \in \Theta$  with  $\| w \|_2 \leq B$ . Then, setting  $\rho^\star = O(\alpha/(BD^3))$ , it holds that  $\operatorname{Var}_{x \sim \phi(\cdot; \rho^\star w)}[c \cdot x] = \Omega(\alpha \| c \|_2^2)$ .
We first provide a general abstract lemma that relates the variance of the uniform distribution  $U$  over  $\mathcal{X}$  to the variance of an almost uniform probability measure  $\mu$ . For simplicity, we denote the uniform distribution over  $\mathcal{X}$  with  $U = U(\mathcal{X})$ .
Lemma 8. Let  $w \in \Theta$  and  $x \in \mathcal{X}$  with  $\| x \|_2 \leq D$ . Consider the uniform probability measure  $U$  over  $\mathcal{X}$  and let  $\mu$  over  $\mathcal{X}$  be such that there exist  $\epsilon_1, \epsilon_2 > 0$  with:
-  $\left\| \mathbf{E}_{x \sim U}[x] - \mathbf{E}_{x \sim \mu}[x]\right\|_2 \leq \epsilon_1$ , and,
-  $w^{\top}\mathbf{E}_{x\sim \mu}[xx^{\top}]w\geq w^{\top}\mathbf{E}_{x\sim U}[xx^{\top}]w - \epsilon_{2}\| w\|_{2}^{2}.$
Then it holds that  $\mathbf{Var}_{x\sim \mu}[w\cdot x]\geq \mathbf{Var}_{x\sim U}[w\cdot x] - 3\max \{\epsilon_1^2,\epsilon_1D,\epsilon_2\} \| w\| _2^2$
$$
\mathbf {V a r} _ {x \sim \mu} [ w \cdot x ] = \underset {x \sim \mu} {\mathbf {E}} \left[ (w \cdot x) ^ {2} \right] - \left(\underset {x \sim \mu} {\mathbf {E}} [ w \cdot x ]\right) ^ {2}.
$$
We first deal with upper-bounding the square of the first moment. Note that
$$
w \cdot \left(\mathbf {E} _ {x \sim \mu} [ x ] - \mathbf {E} _ {x \sim U} [ x ]\right) \leq \| w \| _ {2} \left\| \mathbf {E} _ {x \sim \mu} [ x ] - \mathbf {E} _ {x \sim U} [ x ] \right\| _ {2} \leq \epsilon_ {1} \| w \| _ {2}.
$$
Let us take  $\epsilon > 0$  (with  $\epsilon < \epsilon_1$ ) for simplicity to be such that  $(\mathbf{E}_{x \sim \mu}[w \cdot x])^2 = (\mathbf{E}_{x \sim U}[w \cdot x] + \epsilon \| w \|_2)^2$ . This means that
$$
\begin{array}{l} \left(\mathbf {E} _ {x \sim \mu} [ w \cdot x ]\right) ^ {2} \leq \left(\mathbf {E} _ {x \sim U} [ w \cdot x ]\right) ^ {2} + 2 \epsilon \| w \| _ {2} \left| \mathbf {E} _ {x \sim U} [ w \cdot x ] \right| + \epsilon^ {2} \| w \| _ {2} ^ {2} \\ \leq \left(\mathbf {E} _ {x \sim U} [ w \cdot x ]\right) ^ {2} + 2 \epsilon D \| w \| _ {2} ^ {2} + \epsilon^ {2} \| w \| _ {2} ^ {2}. \\ \end{array}
$$
Next we lower-bound the second moment. It holds that
$$
\underset {x \sim \mu} {\mathbf {E}} \left[ (w \cdot x) ^ {2} \right] = w ^ {\top} \underset {x \sim \mu} {\mathbf {E}} \left[ x x ^ {\top} \right] w \geq \underset {x \sim U} {\mathbf {E}} [ (w \cdot x) ^ {2} ] - \epsilon_ {2} \| w \| _ {2} ^ {2},
$$
for some  $\epsilon_2 > 0$ . This means that
$$
\mathbf {V a r} _ {x \sim \mu} (w \cdot x) \geq \underset {x \sim U} {\mathbf {E}} \left[ (w \cdot x) ^ {2} \right] - \epsilon_ {2} \| w \| _ {2} ^ {2} - \left(\underset {x \sim U} {\mathbf {E}} [ w \cdot x ]\right) ^ {2} - 2 \epsilon D \| w \| _ {2} ^ {2} - \epsilon^ {2} \| w \| _ {2} ^ {2}.
$$
$$
\mathbf {V a r} _ {x \sim \mu} [ w \cdot x ] \geq \mathbf {V a r} _ {x \sim U} [ w \cdot x ] - 3 \max  \left\{\epsilon_ {2}, \epsilon_ {1} ^ {2}, \epsilon_ {1} D \right\} \| w \| _ {2} ^ {2}.
$$
Our next goal is to relate  $\phi (\cdot ;\rho^{\star}w)$  with the uniform measure  $\phi (\cdot ;0)$ . According to the above general lemma, we have to relate the first and second moments of  $\phi (\cdot ;\rho^{\star}w)$  with the ones of the uniform distribution  $U = \phi (\cdot ;0)$ .
The Proof of Proposition 8. Our goal is to apply Lemma 8. First, let us set
$$
f _ {v} (\rho) = \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ v \cdot x ],
$$
for any unit vector  $v\in \Theta$  . Then it holds that
$$
\left\| \mathbf {E} _ {\left. x \sim \phi (\cdot ; 0) \right.} [ x ] - \mathbf {E} _ {\left. x \sim \phi (\cdot ; \rho^ {\star} w) \right.} [ x ] \right\| _ {2} = \sup  _ {v: \| v \| _ {2} = 1} | f _ {v} (0) - f _ {v} (\rho^ {\star}) |.
$$
Using the mean value theorem in  $[0, \rho^{\star}]$  for any unit vector  $v$ , we have that there exists a  $\xi = \xi_v \in (0, \rho^{\star})$  such that
$$
\left| f _ {v} (0) - f _ {v} \left(\rho^ {\star}\right) \right| = \rho^ {\star} \left| f _ {v} ^ {\prime} (\xi) \right|.
$$
It suffices to upper bound  $f_v'(\xi)$  for any unit vector  $v$  and  $\xi \in (0, \rho^\star)$ . Let us compute  $f_v'$ . We have that
$$
\frac {d f _ {v}}{d \rho} = \int_ {S} (v \cdot x) \frac {d}{d \rho} \frac {e ^ {\rho (w x)}}{\int_ {S} e ^ {\rho (w y)} d y} d x = \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ (v \cdot x) (w \cdot x) ] - \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ v \cdot x ] \underset {x \sim \phi (\cdot ; \rho w)} {\mathbf {E}} [ w \cdot x ].
$$
Since  $x\in \mathcal{X}$  , we have that
$$
\sup  _ {v: \| v \| _ {2} = 1} \sup  _ {\xi \in (0, \rho^ {\star})} | f _ {v} ^ {\prime} (\xi) | \leq 2 \| w \| _ {2} D ^ {2}.
$$
$$
\left\| \mathbf {E} _ {x \sim \phi (\cdot ; 0)} [ x ] - \mathbf {E} _ {x \sim \phi (\cdot ; \rho^ {\star} w)} [ x ] \right\| _ {2} \leq 2 \rho^ {\star} \| w \| _ {2} D ^ {2}.
$$
We then continue with controlling the second moment: it suffices to find  $\epsilon_{2}$  such that for any  $v\in \Theta$  it holds
$$
v ^ {\top} \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x x ^ {\top} ] v \geq v ^ {\top} \underset {x \sim \phi (\cdot ; 0)} {\mathbf {E}} [ x x ^ {\top} ] v - \epsilon_ {2} \| v \| _ {2} ^ {2}.
$$
Let us set  $g_{v}(\rho) = \mathbf{E}_{x\sim \phi (\cdot ;\rho w)}[(v\cdot x)^{2}]$  for any vector  $v\in \Theta$  . We have that
$$
\left| g _ {v} (0) - g _ {v} \left(\rho^ {\star}\right) \right| = \rho^ {\star} \left| g _ {v} ^ {\prime} (\xi) \right|,
$$
where  $\xi \in (0,\rho^{\star})$ . It holds that
$$
\left| \frac {d g _ {v}}{d \rho} \right| = \left| \mathbf {E} _ {x \sim \phi (\cdot ; \rho w)} [ (v \cdot x) ^ {2} (w \cdot x) ] - \mathbf {E} _ {x \sim \phi (\cdot ; \rho w)} [ (v \cdot x) ^ {2} ] \mathbf {E} _ {x \sim \phi (\cdot ; \rho w)} [ w \cdot x ] \right| \leq 2 \| v \| ^ {2} \| w \| _ {2} D ^ {3}.
$$
This gives that for any  $v \in \Theta$ , it holds
$$
v ^ {\top} \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x x ^ {\top} ] v \geq v ^ {\top} \underset {x \sim \phi (\cdot ; 0)} {\mathbf {E}} [ x x ^ {\top} ] v - 2 \rho^ {\star} \| w \| _ {2} D ^ {3} \| v \| _ {2} ^ {2}.
$$
Note that the above holds for  $v = c$  too. Lemma 8 gives us that
$$
\mathbf {V a r} _ {x \sim \phi (\cdot ; \rho^ {\star} w)} [ c \cdot x ] \geq \mathbf {V a r} _ {x \sim \phi (\cdot ; 0)} [ c \cdot x ] - 3 \max  \left\{\epsilon_ {2}, \epsilon_ {1} ^ {2}, \epsilon_ {1} D \right\} \| c \| _ {2} ^ {2},
$$
where  $\epsilon_{1} = 2\rho^{\star}BD^{2}$  and  $\epsilon_{2} = 2\rho^{\star}BD^{3}$ . This implies that by picking
$$
\rho^ {\star} = C _ {0} \frac {\alpha}{B D ^ {3}}
$$
for some universal constant  $C_0$ , we get that
$$
\operatorname {V a r} _ {x \sim \phi (\cdot ; \rho^ {*} w)} [ c \cdot x ] = \Omega (\alpha \| c \| _ {2} ^ {2}).
$$
In this section, we considered  $\rho^{\star}$  as indicated by the above Proposition 8.
G Applications to Combinatorial Problems
In this section we provide a series of combinatorial applications of our theoretical framework (Theorem 1). In particular, for each one of the following combinatorial problems (that provably satisfy Assumption 1), it suffices to specify the feature mappings  $\psi_S, \psi_{\mathcal{I}}$  and compute the parameters  $C, D_S, D_{\mathcal{I}}, \alpha$ .
G.1 Maximum Cut, Maximum Flow and Max- $k$ -CSPs
We first provide a general lemma for the variance of "linear tensors" under the uniform measure.
Lemma 9 (Variance Lower Bound Under Uniform). Let  $n, k \in \mathbb{N}$ . For any  $w \in \mathbb{R}^{\binom{n}{k}}$ , it holds that
$$
\mathbf{Var}_{x\sim U(\{-1,1\}^{n})}[w\cdot x^{\otimes k}] = \sum_{\emptyset \neq S\subseteq [n]:|S|\leq k}w_{S}^{2}.
$$
Proof. For any  $w \in \mathbb{R}_{\binom{n}{k}}^{\binom{n}{k}}$ , it holds that
$$
\mathbf {V a r} _ {x \sim U (\{- 1, 1 \} ^ {n})} [ w \cdot x ^ {\otimes k} ] = \underset {x \sim U (\{- 1, 1 \} ^ {n})} {\mathbf {E}} \left[ (w \cdot x ^ {\otimes k}) ^ {2} \right] - \underset {x \sim U (\{- 1, 1 \} ^ {n})} {\mathbf {E}} [ w \cdot x ^ {\otimes k} ] ^ {2}.
$$