SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[AAM22] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pages 4782-4887. PMLR, 2022. 5
[ABA22] Emmanuel Abbe and Enric Boix-Adsera. On the non-universality of deep learning: quantifying the cost of symmetry. arXiv preprint arXiv:2208.03113, 2022. 5
[ABAB+21] Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26989-27002, 2021. 5
[ABAM23] Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055, 2023. 5
$\left[\mathrm{AEG}^{+}22\right]$  Simon Apers, Yuval Efron, Pawel Gawrychowski, Troy Lee, Sagnik Mukhopadhyay, and Danupon Nanongkai. Cut query algorithms with star contraction. arXiv preprint arXiv:2201.05674, 2022. 2
$\left[\mathrm{AKM}^{+}21\right]$  Emmanuel Abbe, Pritish Kamath, Eran Malach, Colin Sandon, and Nathan Srebro. On the power of differentiable learning versus pac and sq learning. Advances in Neural Information Processing Systems, 34:24340-24351, 2021. 5
[AMW18] Saeed Amizadeh, Sergiy Matusevych, and Markus Weimer. Learning to solve circuitsat: An unsupervised differentiable approach. In International Conference on Learning Representations, 2018. 5
[ART23] Maria Chiara Angelini and Federico Ricci-Tersenghi. Modern graph neural networks do worse than classical greedy algorithms in solving combinatorial optimization problems like maximum independent set. Nature Machine Intelligence, 5(1):29-31, 2023. 5
[AS18] Emmanuel Abbe and Colin Sandon. Provable limitations of deep learning. arXiv preprint arXiv:1812.06369, 2018. 5
[AS20] Emmanuel Abbe and Colin Sandon. On the universality of deep learning. Advances in Neural Information Processing Systems, 33:20061-20072, 2020. 5
[BBSS22] Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advances in Neural Information Processing Systems, 35:9768-9783, 2022. 5
$\left[\mathrm{BEG}^{+}22\right]$  Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799, 2022. 5
[VFJ15] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural information processing systems, 28, 2015. 1, 5
[YBV19] Weichi Yao, Afonso S Bandeira, and Soledad Villar. Experimental performance of graph neural networks on random instances of max-cut. In Wavelets and Sparsity XVIII, volume 11138, pages 242-251. SPIE, 2019. 5
[YGS20] Gal Yehuda, Moshe Gabel, and Assaf Schuster. It's not what machines can learn, it's what we cannot teach. In International conference on machine learning, pages 10831-10841. PMLR, 2020. 5
[YP19] Emre Yolcu and Barnabás Póczos. Learning local search heuristics for boolean satisfiability. Advances in Neural Information Processing Systems, 32, 2019. 5
[YW20] Yunhao Yang and Andrew Whinston. A survey on reinforcement learning for combinatorial optimization. arXiv preprint arXiv:2008.12248, 2020. 5
$\left[\mathrm{ZCH}^{+}20\right]$  Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020. 5
$\left[\mathrm{ZMB}^{+}17\right]$  Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Stephen Boyd, and Peter W Glynn. Stochastic mirror descent in variationally coherent optimization problems. Advances in Neural Information Processing Systems, 30, 2017. 7
A Preliminaries and Notation
This lemma is a useful tool for quasar convex functions.
Lemma 1 ([HMR16]). Suppose that the functions  $f_{1}, \ldots, f_{n}$  are individually  $\gamma$ -quasar convex in  $X$  with respect to a common global minimum  $\overline{x}$ . Then for non-negative weights  $a_{1}, \ldots, a_{n}$ , the linear combination  $f = \sum_{i \in [n]} a_{i} f_{i}$  is also  $\gamma$ -quasar convex with respect to  $\overline{x}$  in  $X$ .
In the proofs, we use the following notation: for a matrix  $W$  and vectors  $x, z$ , we let
$$
\phi (x; z; W) = \frac {\exp \left(z ^ {\top} W z\right)}{\sum_ {y \in X} \exp \left(z ^ {\top} W y\right)}, \tag {1}
$$
be a probability mass function over  $X$  and we overload the notation as
$$
\phi (x; w) = \frac {\exp (w \cdot x)}{\sum_ {y \in X} \exp (w \cdot y)}. \tag {2}
$$
B The Proof of Remark 1
Proof. Let  $x_1, \ldots, x_n$  be the variables of the Max-Cut problem of interest and  $S = \{-1, 1\}^n$  be the solution space. Consider  $\mathcal{P}$  to be the collection of product distributions over  $S$ , i.e., for any  $p \in \mathcal{P}$ , it holds that, for any  $s \in S$ ,  $\mathbf{Pr}_{x \sim p}[x = s] = \prod_{i \in [n]} p_i^{\frac{1 + s_i}{2}} (1 - p_i)^{\frac{1 - s_i}{2}}$ . Let us consider the cube  $[\epsilon, 1 - \epsilon]^n$ . This family is complete since the  $O(\epsilon)$ -sub-optimal solution of  $I$  belongs to  $\mathcal{P}$  and is compressed since the description size is  $\mathrm{poly}(n, \log(1 / \epsilon))$ . We show that in this setting there exist bad stationary points. Let  $L_G$  be the Laplacian matrix of the input graph. For some product distribution  $p \in \mathcal{P}$ , it holds that
$$
\mathcal {L} (p) = - \underset {x \sim p (\cdot)} {\mathbf {E}} [ x ^ {\top} L _ {G} x ] = - (2 p - 1) ^ {\top} \overline {{L}} _ {G} (2 p - 1), \quad \nabla_ {p} \mathcal {L} (p) = - 4 \overline {{L}} _ {G} (2 p - 1),
$$
where  $\overline{L}_G$  is zero in the diagonal and equal to the Laplacian otherwise. Let us consider a vertex of the cube  $p\in [\epsilon ,1 - \epsilon ]^n$  which is highly and strictly sub-optimal, i.e., any single change of a node would strictly improve the number of edges in the cut and the score attained in  $p$  is very large compared to  $\min_{x\in S} - x^{\top}L_{G}x$ . For any  $i\in [n]$ , we show that
$$
(\nabla \mathcal {L} (p) \cdot e _ {i})) ((2 p - 1) \cdot e _ {i}) <   0.
$$
This means that if  $p_i$  is large (i.e.,  $1 - \epsilon$ ), then the  $i$ -th coordinate of the gradient of  $\mathcal{L}(p)$  should be negative since this would imply that the negative gradient would preserve  $p_i$  to the right boundary. Similarly for the case where  $p_i$  is small. This means that this point is a stationary point and is highly sub-optimal by assumption.
Let  $P$  (resp.  $N$ ) be the set of indices in  $[n]$  where  $p$  takes the value  $1 - \epsilon$  (resp.  $\epsilon$ ). For any  $i \in [n]$ , let  $\mathcal{N}(i)$  be its neighborhood in  $G$ . Let us consider  $i \in P$ . We have that  $(2p - 1) \cdot e_i > 0$  and so it suffices to show that
$$
\left(\bar {L} _ {G} (2 p - 1)\right) \cdot e _ {i} > 0,
$$
which corresponds to showing that
$$
\sum_ {j \in \mathcal {N} (i) \cap P} L _ {G} (i, j) (1 - 2 \epsilon) + \sum_ {j \in \mathcal {N} (i) \cap N} L _ {G} (i, j) (2 \epsilon - 1) > 0,
$$
and so we would like to have
$$
\sum_ {j \in P} L _ {G} (i, j) - \sum_ {j \in N} L _ {G} (i, j) > 0.
$$
Note that this is true for any  $i \in [n]$  since the current solution is a strict local optimum. The same holds if  $i \in N$ .
C Completeness
Proposition 2 (Completeness). Consider  $\epsilon > 0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds. There exist  $\beta^{\star}, \rho^{\star} \in (0,1)$  and  $\mathcal{W}$  such that the family of solution generators  $\mathcal{P}$  of Equation (5) is complete.
Proof. Assume that  $\mathcal{O}(s,I) = \psi_{\mathcal{I}}(I)^{\top}M\psi_{S}(s)$  and let  $z = \psi_{\mathcal{I}}(I)$  and  $x = \psi_S(s)$ . Moreover, let  $\alpha, C, D_S, D_{\mathcal{I}}$  be the parameters promised by Assumption 1. Let us consider the family  $\mathcal{P} = \{p(W): W \in \mathcal{W}\}$  with
$$
p (x; z; W) = (1 - \beta^ {\star}) \frac {e ^ {z ^ {\top} W x}}{\sum_ {y \in X} e ^ {z ^ {\top} W y}} + \beta^ {\star} \frac {e ^ {z ^ {\top} \rho^ {\star} W x}}{\sum_ {y \in X} e ^ {z ^ {\top} \rho^ {\star} W y}},
$$
where the mixing weight  $\beta^{\star} \in (0,1)$  and the inverse temperate  $\rho^{\star}$  are to be decided. Recall that
$$
\mathcal {L} (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} _ {x \sim p (\cdot ; z; W)} \mathbf {E} [ L (x; z) ] = \underset {z \sim \mathcal {R}} {\mathbf {E}} _ {x \sim p (\cdot ; z; W)} \mathbf {E} [ z ^ {\top} M x ].
$$
Let us pick the parameter matrix  $W = -M / \lambda$ . Let us now fix a  $z \in \psi_{\mathcal{I}}(\mathcal{I})$ . For the given matrix  $M$ , we can consider the finite set of values  $V$  obtained by the quadratic forms  $\{z^{\top}Mx\}_{x \in \psi_S(S)}$ . We further cluster these values so that they have distance at least  $\epsilon$  between each other. We consider the level sets  $C_i$  where  $C_1$  is the subset of  $S$  with minimum value  $v_1(= v_1(z)) \in V$ ,  $C_2$  is the subset with the second smallest  $v_2(= v_2(z)) \in V$ , etc. For fixed  $z \in \psi_{\mathcal{I}}(\mathcal{I})$ , we have that
$$
\underset {x \sim p (\cdot ; z; - M / \lambda)} {\mathbf {E}} [ z ^ {\top} M x ] = (1 - \beta^ {\star}) \underset {x \sim \phi (\cdot ; z; - M / \lambda)} {\mathbf {E}} [ z ^ {\top} M x ] + \beta^ {\star} \underset {x \sim \phi (\cdot ; z; - \rho^ {\star} M / \lambda)} {\mathbf {E}} [ z ^ {\top} M x ],
$$
where  $\phi$  comes from (1). We note that
$$
\mathop{\bf P r}\limits_{x\sim \phi (\cdot ;z; - M / \lambda)}[z^{\top}Mx\in C_{i}] = \frac{|C_{i}|e^{-v_{i} / \lambda}}{\sum_{j}|C_{j}|e^{-v_{j} / \lambda}}  .
$$
We claim that, by letting  $\lambda \to 0$ , the above measure concentrates uniformly on  $C_1$ . The worst case scenario is when  $|C_2| = |S| - |C_1|$  and  $v_{2} = v_{1} + \epsilon$ . Then we have that
$$
\mathop{\bf P r}\limits_{x\sim \phi (\cdot ;z; - M / \lambda)}[z^{\top}Mx\in C_{2}] = \frac{|C_{2}| / |C_{1}|e^{(-v_{2} + v_{1}) / \lambda}}{1 + |C_{2}| / |C_{1}|e^{(-v_{2} + v_{1}) / \lambda}}\leq \delta   ,
$$
when  $1 / \lambda >\log (|\psi_S(S)| / \delta) / \epsilon$ , since in the worst case  $|C_2| / |C_1| = \Omega (|\psi_S(S)|)$ . Using this choice of  $\lambda$  and taking expectation over  $z$ , we get that
$$
\mathbf{E}_{z\sim \mathcal{R}}\underset {x\sim p(\cdot ;z; - M / \lambda)}{\mathbf{E}}[z^{\top}Mx]\leq (1 - \beta^{\star})  \mathbf{E}_{z\sim \mathcal{R}}\left[(1 - \delta)\min_{x\in \psi_{S}(S)}L(x;z) + \delta v_{2}(z)\right] + \beta^{\star}\underset {x\sim \phi (\cdot ;z; - \rho^{\star}M / \lambda)}{\mathbf{E}}[z^{\top}Mx].
$$
First, we remark that by taking  $\rho^{\star} = \mathrm{poly}(\alpha, 1/C, 1/D_S, 1/D_{\mathcal{I}})$ , the last term in the right-hand side of the above expression can be replaced by the expected score of an almost-uniform solution (see Lemma 3 and Proposition 8), which is at most  $\mathrm{poly}(D_S, D_{\mathcal{I}}, C)2^{-|\psi_S(S)|}$  (and which is essentially negligible). Finally, one can pick  $\beta^{\star}, \delta = \mathrm{poly}(\epsilon, 1/C, 1/D_S, 1/D_{\mathcal{I}})$  so that
$$
\mathcal {L} (- M / \lambda) = \underset {z \sim \mathcal {R}} {\mathbf {E}} \underset {x \sim p (\cdot ; z; - M / \lambda)} {\mathbf {E}} [ z ^ {\top} M x ] \leq \underset {z \sim \mathcal {R}} {\mathbf {E}} \left[ \min  _ {x \in \psi_ {S} (S)} L (x; z) \right] + \epsilon .
$$
This implies that  $\mathcal{P}$  is complete by letting  $\overline{W} = -M / \lambda \in \mathcal{W}$ . This means that one can take  $\mathcal{W}$  to be a ball centered at 0 with radius (of  $\epsilon$ -sub-optimality) to be of order at least  $B = \| M\|_{\mathrm{F}} / \lambda$ .
D Compression
Proposition 3 (Compression). Consider  $\epsilon > 0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds. There exist  $\beta^{\star}, \rho^{\star} \in (0,1)$  and  $\mathcal{W}$  such that the family of solution generators  $\mathcal{P}$  of Equation (5) is compressed.
Proof. We have that the bit complexity to represent the mixing weight  $\beta^{\star}$  is  $\mathrm{polylog}(D_S, D_{\mathcal{I}}, C, 1 / \epsilon)$  and the description size of  $\mathcal{W}$  is polynomial in  $[\mathcal{I}]$  and in  $\log(1 / \epsilon)$ . This follows from Assumption 1 since the feature dimensions  $n_X$  and  $n_Z$  are poly([I]) and  $\mathcal{W}$  is a ball centered at 0 with radius  $O(B)$ , where  $B = \|M\|_{\mathrm{F}} / \lambda \leq C / \lambda$ , which are also poly([I] /  $\epsilon$ ).
E Efficiently Optimizable
Proposition 4 (Efficiently Optimizable). Consider  $\epsilon > 0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds. There exist  $\beta^{\star}, \rho^{\star} \in (0,1)$  and  $\mathcal{W}$  such that family of solution generators  $\mathcal{P}$  of Equation (5) is efficiently estimable using Projected SGD, where the projection set is  $\mathcal{W}$ .
The proof of this proposition is essentially decomposed into two parts: first, we show that the entropy-reularized loss of Equation (2) is quasar convex and then apply the projected SGD algorithm to  $\mathcal{L}_{\lambda}$ .
Recall that  $H(W) = \mathbf{E}_{z\sim \mathcal{R}}\mathbf{E}_{x\sim \phi (\cdot ;z;W)}[\log \phi (x;z;W)]$ . Let  $R$  be a weighted sum (to be in accordance with the mixture structure of  $\mathcal{P}$ ) of negative entropy regularizers
$$
R (W) = \left(1 - \beta^ {\star}\right) H (W) + \frac {\beta^ {\star}}{\rho^ {\star}} H \left(\rho^ {\star} W\right), \tag {1}
$$
where  $\beta^{\star},\rho^{\star}$  are the fixed parameters of  $\mathcal{P}$  (recall Equation (5)). We define the regularized loss
$$
\mathcal {L} _ {\lambda} (W) = \mathcal {L} (W) + \lambda R (W), \tag {2}
$$
$$
\mathcal {L} (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} \underset {x \sim p (\cdot ; z; W)} {\mathbf {E}} [ L (z; x) ], p (W) \in \mathcal {P}.
$$
E.1 Quasar Convexity of the Regularized Loss
In this section, we show that  $\mathcal{L}_{\lambda}$  of Equation (2) is quasar convex. We restate Proposition 1.
Proposition 5 (Quasar Convexity). Consider  $\epsilon >0$  and a prior  $\mathcal{R}$  over  $\mathcal{I}$ . Assume that Assumption 1 holds. The function  $\mathcal{L}_{\lambda}$  of Equation (2) with domain  $\mathcal{W}$  is poly  $(C,D_S,D_{\mathcal{I}},1 / \epsilon ,1 / \alpha)$ -quasar convex with respect to  $-M / \lambda$  on the domain  $\mathcal{W}$ .
Proof. We can write the loss  $\mathcal{L}_{\lambda}$  as
$$
\mathcal {L} _ {\lambda} (W) = \underset {z \sim \mathcal {R}} {\mathbf {E}} [ \mathcal {L} _ {\lambda , z} (W) ] = \underset {z \sim \mathcal {R}} {\mathbf {E}} [ \mathcal {L} _ {z} (W) + \lambda R _ {z} (W) ],
$$
where the mappings  $\mathcal{L}_z$  and  $R_{z}$  are instance-specific (i.e., we have fixed  $z$ ). We can make use of Lemma 1, which states that linear combinations of quasar convex (with the same minimizer) remain quasar convex. Hence, since the functions  $\mathcal{L}_{\lambda ,z}$  have the same minimizer  $-M / \lambda$ , it suffices to show quasar convexity for a particular fixed instance mapping, i.e., it suffices to show that the function
$$
\mathcal {L} _ {\lambda , z} (W) = \mathcal {L} _ {z} (W) + \lambda R _ {z} (W)
$$
is quasar convex. Recall that  $W$  is a matrix of dimension  $n_Z \times n_X$ . To deal with the function  $\mathcal{L}_{\lambda, z}$ , we consider the simpler function that maps vectors instead of matrices to real numbers. For some vector  $c$ , let  $\mathcal{L}_{\lambda}^{\mathrm{vec}}: \mathbb{R}^{n_X} \to \mathbb{R}$  be
$$
\mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) = \underset {x \sim p (\cdot ; w)} {\mathbf {E}} [ c \cdot x ] + \lambda R ^ {\mathrm {v e c}} (w), \tag {3}
$$
where for any vector  $w \in \mathbb{R}^{n_X}$ , we define the probability distribution  $\phi(\cdot; w)$  over the solution space  $X = \psi_S(S)$  with probability mass function
$$
\phi (x; w) = \frac {e ^ {w \cdot x}}{\sum_ {y \in X} e ^ {w \cdot y}}.
$$
We then define
$$
p (\cdot ; w) = (1 - \beta^ {\star}) \phi (\cdot ; w) + \beta^ {\star} \phi (\cdot ; \rho^ {\star} w),
$$
and  $R^{\mathrm{vec}}(w) = (1 - \beta^{\star})H(w) + \frac{\beta^{\star}}{\rho^{\star}} H(\rho^{\star}w)$  (this is a essentially a weighted sum of regularizers, needed to simplify the proof) with  $H(w) = \mathbf{E}_{x\sim \phi (\cdot ;w)}\log \phi (x,w)$ . These quantities are essentially the fixed-instance analogues of Equations (5) and (2). The crucial observation is that by taking  $c = z^{\top}M$  and applying the chain rule we have that
$$
\nabla_ {W} \mathcal {L} _ {\lambda , z} (W) = z \cdot \left[ \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} \left(z ^ {\top} W\right) \right] ^ {\top}. \tag {4}
$$
This means that the gradient of the fixed-instance objective  $\mathcal{L}_{\lambda, z}$  is a matrix of dimension  $n_Z \times n_X$  that is equal to the outer product of the instance featurization  $z$  and the gradient of the simpler function  $\mathcal{L}_w^{\mathrm{vec}}$  evaluated at  $z^\top W$ . Let us now return on showing that  $\mathcal{L}_{\lambda, z}$  is quasar convex. To this end, we observe that
$$
\nabla_ {W} \mathcal {L} _ {\lambda , z} (W) \cdot \left(W + \frac {M}{\lambda}\right) = \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} \left(z ^ {\top} W\right) \cdot \left(z ^ {\top} W + z ^ {\top} \frac {M}{\lambda}\right).
$$
This means that, since  $z$  is fixed, it suffices to show that the function  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  is quasar convex. We provide the next key proposition that deals with issue. This result is one of the main technical aspects of this work and its proof can be found in Appendix E.2.
In the following, intuitively  $\mathcal{X}$  is the post-featurization instance space and  $\mathcal{Z}$  is the parameter space.
Proposition 6. Consider  $\epsilon, \lambda > 0$ . Let  $\|c\|_2 \leq C_1$ . Let  $\mathcal{Z}$  be an open ball centered at  $0$  with diameter  $2C_1 / \lambda$ . Let  $\mathcal{X}$  be a space of diameter  $D$  and let  $\mathbf{Var}_{x \sim U(\mathcal{X})}[v \cdot x] \geq \alpha \|v\|_2^2$  for any  $v \in \mathcal{Z}$ . The function  $\mathcal{L}_{\lambda}^{\mathrm{vec}}(w) = \mathbf{E}_{x \sim p(:, w)}[c \cdot x] + \lambda R^{\mathrm{vec}}(w)$  is poly  $(1/C_1, 1/D, \epsilon, \alpha)$ -quasar convex with respect to  $-c / \lambda$  on  $\mathcal{Z}$ .
We can apply the above result with  $c = z^{\top}M$ ,  $w = z^{\top}W$ ,  $D = D_{S}$  and  $C_1 = D_{\mathcal{I}}C$ . These give that the quasar convexity parameter  $\gamma$  is of order  $\gamma = \mathrm{poly}(\epsilon, \alpha, 1/C, 1/D_S, 1/D_{\mathcal{I}})$ . Since we have that  $\mathcal{L}_{\lambda}^{\mathrm{vec}}(z^{\top}W) = \mathcal{L}_{\lambda,z}(W)$ , we get that
$$
\nabla_ {W} \mathcal {L} _ {\lambda , z} (W) \cdot (W + M / \lambda) \geq \gamma (\mathcal {L} _ {\lambda , z} (W) - \mathcal {L} _ {\lambda , z} (- M / \lambda)).
$$
This implies that  $\mathcal{L}_{\lambda, z}$  is  $\gamma$ -quasar convex with respect to the minimizer  $-M / \lambda$  and completes the proof using Lemma 1.
E.2 The Proof of Proposition 6
Let us consider  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  to be a real-valued differentiable function defined on  $\mathcal{Z}$ . Let  $w, -c / \lambda \in \mathcal{Z}$  and let  $L$  be the line segment between them with  $L \in \mathcal{Z}$ . The mean value theorem implies that there exists  $w' \in L$  such that
$$
\mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) - \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (- c / \lambda) = \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} \left(w ^ {\prime}\right) \cdot \left(w + c / \lambda\right) \leq \| \nabla \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} \left(w ^ {\prime}\right) \| _ {2} \| w + c / \lambda \| _ {2}.
$$
Now we have that  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  has bounded gradient (see Lemma 2) and so we get that
$$
\left\| \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\text {v e c}} \left(w ^ {\prime}\right) \right\| _ {2} \leq D ^ {2} \| c + \lambda w ^ {\prime} \| _ {2} = D ^ {2} \lambda \| w ^ {\prime} + c / \lambda \| _ {2} \leq D ^ {2} \lambda \| w + c / \lambda \| _ {2},
$$
since  $w^{\prime}\in L$  . This implies that
$$
\mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) - \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (- c / \lambda) \leq D ^ {2} \lambda \| w + c / \lambda \| _ {2} ^ {2} \leq \frac {1}{\gamma} \nabla \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (w + c / \lambda),
$$
where  $1 / \gamma = \frac{\mathrm{poly}(C_1,D)}{\epsilon^3\alpha^2}$ . The last inequality is an application of the correlation lower bound (see Lemma 3).
In the above proof, we used two key lemmas: a bound for the norm of the gradient and a lower bound for the correlation. In the upcoming subsections, we prove these two results.
E.2.1 Bounded Gradient Lemma and Proof
Lemma 2 (Bounded Gradient Norm of  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$ ). Consider  $\epsilon, \lambda > 0$ . Let  $\mathcal{Z}$  be the domain of  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  of (3). Let  $\mathcal{X}$  be a space of diameter  $D$ . For any  $w \in \mathcal{Z}$ , it holds that
$$
\left\| \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\text {v e c}} (w) \right\| _ {2} \leq O \left(D ^ {2}\right) \| c + \lambda w \| _ {2}.
$$
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) = (1 - \beta^ {\star}) G _ {w} + \beta^ {\star} \rho^ {\star} G _ {\rho^ {\star} w},
$$
$$
G _ {w} = \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] - \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; w)} {\mathbf {E}} [ x ],
$$
$$
G _ {\rho^ {\star} w} = \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ ((c + \lambda w) \cdot x) x ] - \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ (c + \lambda w) \cdot x ] \underset {x \sim \phi (\cdot ; \rho^ {\star} w)} {\mathbf {E}} [ x ].
$$
Note that since  $x\in \mathcal{X}$  , it holds that  $\| x\| _2\leq D$  . Hence
$$
\| G _ {w} \| _ {2} = \sup  _ {v: \| v \| _ {2} = 1} | v \cdot G _ {w} | \leq 2 D ^ {2} \| c + \lambda w \| _ {2}.
$$
$$
\| G _ {\rho^ {*} w} \| _ {2} \leq 2 D ^ {2} \| c + \lambda w \| _ {2}.
$$
$$
\| \nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \| _ {2} \leq 2 (1 - \beta^ {\star}) \| c + \lambda w \| _ {2} D ^ {2} + 2 \beta^ {\star} \rho^ {\star} \| c + \lambda w \| _ {2} D ^ {2} = O (D ^ {2}) \| c + \lambda w \| _ {2}.
$$
E.2.2 Correlation Lower Bound Lemma and Proof
The following lemma is the second ingredient in order to show Proposition 6.
Lemma 3 (Correlation Lower Bound for  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$ ). Let  $\lambda > 0$ . Let  $\|c\|_2 \leq C_1$ . Let  $\mathcal{Z}$  be an open ball centered at  $0$  with diameter  $B = 2C_1 / \lambda$ . Let  $\mathcal{X}$  be a space of diameter  $D$ . Assume that  $w \in \mathcal{Z}$  and  $\operatorname{Var}_{x \sim U(\mathcal{X})}[(c + \lambda w) \cdot x] \geq \alpha \|c + \lambda w\|_2^2$  for some  $\alpha > 0$ . Then, for any  $\beta^\star \in (0,1)$ , there exists  $\rho^\star > 0$  such that it holds that
$$
\nabla_ {w} \mathcal {L} _ {\lambda} ^ {\mathrm {v e c}} (w) \cdot (c + \lambda w) = \Omega (\beta^ {\star} \alpha^ {2} / (B D ^ {3}) \| c + \lambda w \| _ {2} ^ {2}),
$$
where  $\mathcal{L}_{\lambda}^{\mathrm{vec}}$  is the regularized loss of Proposition 6,  $\rho^{\star}$  is the scale in the second component of the mixture of (5) and  $\beta^{\star} \in (0,1)$  is the mixture weight.
First, in Lemma 4 and Lemma 5, we give a formula for the desired correlation  $\nabla_w\mathcal{L}_\lambda^{\mathrm{vec}}(w)\cdot (c + \lambda w)$  and, then we can provide a proof for Lemma 3 by lower bounding this formula.
Lemma 4 (Correlation with Regularization). Consider the function  $g(w) = \mathbf{E}_{x\sim \phi (\cdot ;w)}[c\cdot x] + \lambda H(w)$ , where  $H$  is the negative entropy regularizer. Then it holds that
$$
\nabla_ {w} g (w) \cdot (c + \lambda w) = \mathbf {V a r} _ {x \sim \phi (\cdot ; w)} [ (c + \lambda w) \cdot x ].
$$
Proof. Let us consider the following objective function: