SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. If  $\mathbb{P}_{\mathbf{x} \in S} [|\langle \mathbf{w}, \mathbf{x} \rangle| \in [0, \theta]] > C_1 \theta \lambda^{C_1}$ , then reject.
2. Let  $\mathrm{proj}_{\perp \mathbf{w}}: \mathbb{R}^d \to \mathbb{R}^{d-1}$  denote the operator that given any vector in  $\mathbb{R}^d$ , it outputs its projection into the  $(d-1)$ -dimensional subspace of  $\mathbb{R}^d$  that is orthogonal to  $\mathbf{w}$ .
3. Compute the  $(d - 1)\times (d - 1)$  matrix  $M_S$  as follows:
$$
M _ {S} = \underset {\mathbf {x} \in S} {\mathbb {E}} \left[ \sum_ {i = 2} ^ {\infty} \frac {(\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T}}{(i - 1) ^ {2}} \mathbb {1} \{| \langle \mathbf {w}, \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta) \} \right]
$$
4. Run the spectral tester of Proposition A.2 on  $M_S$  given  $\delta \gets \delta$ ,  $\lambda \gets C_1\lambda^{C_1}$  and  $\theta \gets \frac{C_1}{2}\theta \lambda^{C_1}$ , i.e., compute  $\| M_S\|_{\mathrm{op}}$  and if  $\| M_S\|_{\mathrm{op}} > C_1\theta \lambda^{C_1}$ , then reject. Otherwise, accept.
First, suppose the test accepts. For the following, consider the vector  $\mathbf{w}' \in \mathbb{R}^d$  to be an arbitrary unit vector and  $\mathbf{v} \in \mathbb{R}^d$  to be the unit vector that is perpendicular to  $\mathbf{w}$ , lies within the plane defined by  $\mathbf{w}$  and  $\mathbf{w}'$  and  $\langle \mathbf{v}, \mathbf{w}' \rangle \leq 0$ . Then we have:
$$
\begin{array}{l} \underset {\mathbf {x} \sim S} {\mathbb {P}} \left[ \mathrm {s i g n} (\langle \mathbf {w} ^ {\prime}, \mathbf {x} \rangle) \neq \mathrm {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) \right] \leq \\ \leq \sum_ {i = 1} ^ {\infty} _ {\mathbf {x} \sim S} \mathbb {P} \left[ \underbrace {| \langle \mathbf {v} , \mathbf {x} \rangle | > \frac {\theta}{\tan \theta} \cdot (i - 1)} _ {\text {I m p l i e s} | \langle \mathbf {v}, \mathbf {x} \rangle | > (i - 1) / 2} \& | \langle \mathbf {w}, \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta ] \right] \\ \leq \underbrace {\mathbb {P} _ {\mathbf {x} \in S} \left[ | \langle \mathbf {w} , \mathbf {x} \rangle | \in [ 0 , \theta ] \right]} _ {\leq C _ {1} \theta \lambda^ {C _ {1}}} + 4 \underbrace {\sum_ {i = 2} ^ {\infty} \frac {\mathbb {E} _ {\mathbf {x} \sim S} \left[ \langle \mathbf {v} , \mathbf {x} \rangle^ {2} \mathbb {1} _ {| \langle \mathbf {w} , \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta ]} \right]}{(i - 1) ^ {2}}} _ {\langle \operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {v}, M \operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {v} \rangle \leq \| M \| _ {\mathrm {o p}} \leq C _ {1} \theta \lambda^ {C _ {1}}} \\ \leq 5 C _ {1} \theta \lambda^ {C _ {1}} \\ \end{array}
$$
For part (b), we suppose that the distribution  $D_{\mathcal{X}}$  is indeed  $\lambda$ -nice. We will show that with probability at least  $1 - \delta$ , the tester will accept, i.e., that
$$
\underset {\mathbf {x} \in S} {\mathbb {P}} [ | \langle \mathbf {w}, \mathbf {x} \rangle | \in [ 0, \theta ] ] \leq C _ {1} \theta \lambda^ {C _ {1}} \text {a n d} \tag {B.1}
$$
$$
\left\| M _ {S} \right\| _ {\mathrm {o p}} \leq C _ {1} \theta \lambda^ {C _ {1}} \tag {B.2}
$$
We first observe that the corresponding quantities under distribution  $D_{\mathcal{X}}$  due to Proposition A.3. In particular, we have that for some universal constant  $C' > 0$
$$
\mathbb {P} _ {\mathbf {x} \in D _ {\mathcal {X}}} [ | \langle \mathbf {w}, \mathbf {x} \rangle | \in [ 0, \theta ] ] \leq C ^ {\prime} \theta \lambda^ {C ^ {\prime}} \text {a n d} \tag {B.3}
$$
$$
\underset {\mathbf {x} \in D _ {\mathcal {X}}} {\mathbb {E}} \left[ \langle \mathbf {v} ^ {\prime}, \mathbf {x} \rangle^ {2} \cdot \mathbb {1} \left\{\left| \langle \mathbf {w}, \mathbf {x} \rangle \right| \in [ c, c + \theta ] \right\} \right] \leq C ^ {\prime} \theta \lambda^ {C ^ {\prime}} \text {f o r a n y v ^ {\prime} \in S ^ {d - 1}} \tag {B.4}
$$
If we let  $M_{D\mathcal{X}} = \mathbb{E}_{D\mathcal{X}}[M_S]$ , we get that
$$
\begin{array}{l} \| M _ {D _ {\mathcal {X}}} \| _ {\mathrm {o p}} = \sup  _ {\mathbf {u} \in \mathbb {S} ^ {d - 2}} \mathbf {u} ^ {T} M _ {D _ {\mathcal {X}}} \mathbf {u} = \sup  _ {\mathbf {v} ^ {\prime} \in \mathbb {S} ^ {d - 1}: \langle \mathbf {v} ^ {\prime}, \mathbf {w} \rangle = 0} (\operatorname {p r o j} _ {\perp_ {\mathbf {w}}} \mathbf {v} ^ {\prime}) ^ {T} M _ {D _ {\mathcal {X}}} (\operatorname {p r o j} _ {\perp_ {\mathbf {w}}} \mathbf {v} ^ {\prime}) \\ \leq \sum_ {i = 2} ^ {\infty} \frac {1}{(i - 1) ^ {2}} \sup  _ {\mathbf {v} ^ {\prime} \in \mathbb {S} ^ {d - 1}} \mathbb {E} _ {\mathbf {x} \in D _ {\mathcal {X}}} [ \langle \mathbf {v} ^ {\prime}, \mathbf {x} \rangle^ {2} \cdot \mathbb {1} \{| \langle \mathbf {w}, \mathbf {x} \rangle | \in [ c, c + \theta ] \} ] \\ \leq \sum_ {i = 2} ^ {\infty} \frac {1}{(i - 1) ^ {2}} \sup  _ {\mathbf {v} ^ {\prime} \in \mathbb {S} ^ {d - 1}} C ^ {\prime} \theta \lambda^ {C ^ {\prime}} \leq \frac {C ^ {\prime} \pi^ {2}}{6} \theta \lambda^ {C ^ {\prime}} \\ \end{array}
$$
Note that only at most  $|S|$  many terms below are non-zero, hence  $M_S$  can be computed efficiently.
By Proposition A.2, in order to satisfy expression (B.2), it remains to show that  $\mathbb{E}_{\mathbf{z}\sim D}[(\mathbf{z}_\ell \mathbf{z}_j)^2 ]\leq C_1\lambda^{C_1}$  for any  $\ell ,j\in [d]$ , where  $\mathbf{z}$  is defined as follows
$$
\mathbf {z} = \sum_ {i = 2} ^ {\infty} \frac {\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}}{(i - 1)} \mathbb {1} _ {| \langle \mathbf {w}, \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta)}.
$$
Since  $\mathbb{E}_{\mathbf{z}\sim D}[(\mathbf{z}_\ell \mathbf{z}_j)^2 ]\leq \mathbb{E}_{\mathbf{z}\sim D}[\langle \mathbf{u},\mathbf{x}\rangle^2\langle \mathbf{u}',\mathbf{x}\rangle^2 ],$  for some unit vectors  $\mathbf{u},\mathbf{u}'\in \mathbb{S}^{d - 1}$  (orthogonal to  $\mathbf{w}$ ), the desired bound follows from Proposition A.3.
It remains to bound the absolute distance between the quantities of the left hand side of expressions (B.1) and (B.3). This can be achieved by an application of the Hoeffding bound, since the empirical version of the quantity is the average of independent Bernoulli random variables.
We restate Lemma 3.2 here for convenience.
Lemma B.2 (Universally Testable Weak Anti-Concentration). Let  $D$  be a distribution over  $\mathbb{R}^d$ . Then, there is a universal constant  $C > 0$  and a tester that given a unit vector  $\mathbf{w} \in \mathbb{R}^d$ ,  $\delta \in (0,1)$ ,  $\gamma > 0$ ,  $\lambda \geq 1$ ,  $\sigma \leq \frac{1}{2\lambda}$  and a set  $S$  of i.i.d. samples from  $D$  with size at least  $C \cdot \frac{d^4}{\sigma^2\delta} \log(d) \lambda^C$ , runs in time  $\mathrm{poly}(d, \lambda, \frac{1}{\sigma}, \frac{1}{\delta}, \log(\frac{1}{\gamma}))$  and satisfies the following specifications
Proof. The testing algorithm receives a set  $S \subset \mathbb{R}^d$ ,  $\mathbf{w} \in \mathbb{S}^{d-1}$ ,  $\delta \in (0,1)$ ,  $\gamma > 0$ ,  $\lambda \geq 1$  and  $\sigma \leq \frac{1}{2\lambda}$  and does the following for some sufficiently large  $C_1 > 0$ :
1. If  $\mathbb{P}_{\mathbf{x}\in S}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma ] > 2\sigma \cdot C_1\lambda^{C_1}$ , then reject.
2. Compute the  $(d - 1)\times (d - 1)$  matrix  $M_S$  as follows:
$$
M _ {S} = \underset {\mathbf {x} \in S} {\mathbb {E}} \left[ (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} \{| \langle \mathbf {w}, \mathbf {x} \rangle \leq \sigma | \} \right]
$$
3. Run the spectral tester of Proposition A.2 on  $M_S$  given  $\delta \gets \delta$ ,  $\lambda \gets C_1\lambda^{C_1}$  and  $\theta \gets \frac{2\sigma}{C_1\lambda^{C_1}}$ , i.e., reject if the minimum singular value of  $M_S$  is less than  $\frac{2\sigma}{C_1\lambda^{C_1}}$ .
4. Run the hypercontractivity tester (Prop. 3.5) on  $S' = \{\mathrm{proj}_{\perp \mathbf{w}} \mathbf{x} : \mathbf{x} \in S \text{ and } |\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma\}$ , i.e., solve an appropriate SDP and reject if the solution is larger than a specified threshold. Otherwise, accept.
For part (a), we apply the Paley-Zygmund inequality to the random variable  $Z = \langle \mathbf{v},\mathbf{x}\rangle^2$  conditioned on  $|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma$  and get
$$
\underset {\mathbf {x} \in S} {\mathbb {P}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {2} \geq \frac {1}{2} \underset {\mathbf {x} \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {2} \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right] \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right] \geq \frac {(\mathbb {E} _ {\mathbf {x} \in S} [ \langle \mathbf {v} , \mathbf {x} \rangle^ {2} | | \langle \mathbf {w} , \mathbf {x} \rangle | \leq \sigma ]) ^ {2}}{4 \mathbb {E} _ {\mathbf {x} \in S} [ \langle \mathbf {v} , \mathbf {x} \rangle^ {4} | | \langle \mathbf {w} , \mathbf {x} \rangle | \leq \sigma ]}
$$
Note that since  $\langle \mathbf{v},\mathbf{w}\rangle = 0$  we have  $\langle \mathbf{v},\mathbf{x}\rangle = \langle \mathrm{proj}_{\perp \mathbf{w}}\mathbf{v},\mathrm{proj}_{\perp \mathbf{w}}\mathbf{x}\rangle$  (where  $\| \mathbf{v}\| _2 = \| \mathrm{proj}_{\perp \mathbf{w}}\mathbf{v}\| _2$ ). Therefore, since  $S$  has passed the spectral tester as well as the tester for the probability of lying within the strip  $|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma$ , we have that
$$
\underset {\mathbf {x} \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {2} \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right] = \frac {\mathbb {E} _ {\mathbf {x} \in S} \left[ \langle \mathbf {v} , \mathbf {x} \rangle^ {2} \cdot \mathbb {1} \{\left| \langle \mathbf {w} , \mathbf {x} \rangle \right| \leq \sigma \} \right]}{\mathbb {P} _ {\mathbf {x} \in S} [ | \langle \mathbf {w} , \mathbf {x} \rangle | \leq \sigma ]} \geq \frac {1}{2 C _ {1} \lambda^ {2 C _ {1}}}
$$
Moreover,  $\{\mathbf{x}\in S:|\langle \mathbf{w},\mathbf{x}\rangle |\leq \sigma \}$  has passed the hypercontractivity tester, and therefore, according to Proposition 3.5 we have
$$
\underset {\mathbf {x} \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {4} \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right] \leq C _ {1} \cdot \gamma^ {4}
$$
Combining the above inequalities we conclude the proof of part (a).
For part (b), we assume that  $D$  is indeed  $\lambda$ -nice and  $\gamma$ -Poincaré. We first use Proposition A.3 as well as a Hoeffding bound, to get that  $\mathbb{P}_{\mathbf{x} \in S} [|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma] \in [\frac{2\sigma}{C' \lambda^{C'}}$ ,  $2\sigma \cdot C' \lambda^{C'}]$  with probability at least  $1 - \delta / 3$  over  $S$  (since  $|S|$  is large enough), for some universal constant  $C' > 0$ . Then, we use part (ii) of Proposition A.3 to lower bound the minimum eigenvalue of  $M_D = \mathbb{E}_D[M_S]$  by  $\frac{4\sigma}{C' \lambda^{C'}}$ . Using part (iii) of Proposition A.3 to bound the second moment of each of the elements of  $M_D$ , we may use Proposition A.2 to get that  $M_S \succeq \frac{2\sigma}{C' \lambda^{C'}} I_{d-1}$  (and our spectral test passes) with probability at least  $1 - \delta / 3$ . It remains to show that the hypercontractivity tester will accept with probability at least  $1 - \delta / 3$  (since, then, the result follows from a union bound).
We acquire samples from the hypercontractivity tester through rejection sampling (we keep only the samples within the strip). Since the probability of falling inside the strip is at least  $\frac{2\sigma}{C' \lambda^{C'}}$ , the number of samples we will keep is at least  $|S'| \geq \frac{|S| \sigma}{C'' \lambda^{C'}}$ , for some large enough constant  $C'' > 0$  (due to Chernoff bound) and with probability at least  $1 - \delta / 6$ . We now apply Lemma A.1 to get that the distribution of  $\mathrm{proj}_{\perp_{\mathbf{w}}} \mathbf{x}$  conditioned on the strip  $|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma$  is  $\gamma$ -Poincaré, since  $D$  is also  $\gamma$ -Poincaré. Hence, the hypercontractivity tester accepts with probability at least  $1 - \delta / 6$  due to Proposition 3.5.
C Proofs from Section 4
C.1 Proof of Proposition 4.2
We restate Proposition 4.2 here for completeness.
Proposition C.1 (Modification from [GKSV23, DKTZ20a, DKTZ20b]). For a distribution  $D_{\mathcal{X}\mathcal{Y}}$  over  $\mathbb{R}^d\times \{\pm 1\}$  let opt be the minimum error achieved by some origin-centered halfspace and  $\mathbf{w}^{*}\in \mathbb{S}^{d - 1}$  a corresponding vector. Consider  $\mathcal{L}_{\sigma}$  as in Equation (4.1) for  $\sigma >0$  and let  $\eta < 1 / 2$ . Let  $\mathbf{w}\in \mathbb{S}^{d - 1}$  with  $\angle (\mathbf{w},\mathbf{w}^{*}) = \theta < \frac{\pi}{2}$  and  $\mathbf{v}\in \operatorname {span}(\mathbf{w},\mathbf{w}^{*})$  such that  $\langle \mathbf{v},\mathbf{w}\rangle = 0$  and  $\langle \mathbf{v},\mathbf{w}^{*}\rangle < 0$ . Then, for some universal constant  $C > 0$  and any  $\alpha \geq \frac{\sigma}{2\tan\theta}$  we have  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};D_{\mathcal{X}\mathcal{Y}})\| _2\geq A_1 - A_2 - A_3$ , where
$$
A _ {1} = \frac {\alpha}{C \cdot \sigma} \cdot \mathbb {P} \left[ | \langle \mathbf {v}, \mathbf {x} \rangle | \geq \alpha a n d | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \right]
$$
$$
A _ {2} = \frac {C}{\tan \theta} \cdot \mathbb {P} \left[ | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{2} \right] a n d A _ {3} = \frac {C}{\sigma} \cdot \sqrt {\mathrm {o p t}} \cdot \sqrt {\mathbb {E} \left[ \langle \mathbf {v} , \mathbf {x} \rangle^ {2} \cdot \mathbb {1} _ {\{| \langle \mathbf {w} , \mathbf {x} \rangle | \leq \frac {\sigma}{2} \}} \right]}
$$
Moreover, if the noise is Massart with rate  $\eta$ , then  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};D_{\mathcal{XY}})\| _2\geq (1 - 2\eta)A_1 - A_2$
Proof. The proof is a slight modification of a part of the proof of Lemma 4.4 in [GKSV23], but we present it here for completeness.
For any vector  $\mathbf{x} \in \mathbb{R}^d$ , let:  $\mathbf{x}_{\mathbf{w}} = \langle \mathbf{w}, \mathbf{x} \rangle$  and  $\mathbf{x}_{\mathbf{v}} = \langle \mathbf{v}, \mathbf{x} \rangle$ . It follows that  $\mathrm{proj}_V(\mathbf{x}) = \mathbf{x}_{\mathbf{v}} \mathbf{e}_1 + \mathbf{x}_{\mathbf{w}} \mathbf{e}_2$ , where  $\mathrm{proj}_V$  is the operator that orthogonally projects vectors on  $V$ . Using the fact that  $\nabla_{\mathbf{w}} (\langle \mathbf{w}, \mathbf{x} \rangle / \| \mathbf{w} \|_2) = \mathbf{x} - \langle \mathbf{w}, \mathbf{x} \rangle \mathbf{w} = \mathbf{x} - \mathbf{x}_{\mathbf{w}} \mathbf{w}$  for any  $\mathbf{w} \in \mathbb{S}^{d-1}$ , the interchangeability of the gradient and expectation operators and the fact that  $\ell_{\sigma}'$  is an even function we get that
$$
\nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}) = \mathbb {E} \Big [ - \ell_ {\sigma} ^ {\prime} (| \langle \mathbf {w}, \mathbf {x} \rangle |) \cdot y \cdot (\mathbf {x} - \mathbf {x} _ {\mathbf {w}} \mathbf {w}) \Big ]
$$
Since the projection operator  $\mathrm{proj}_V$  is a contraction, we have  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w})\| _2\geq \| \mathrm{proj}_V\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w})\| _2$  and we can therefore restrict our attention to a simpler, two dimensional problem. In particular, since  $\mathrm{proj}_V(\mathbf{x}) = \mathbf{x_v}\mathbf{e}_1 + \mathbf{x_w}\mathbf{e}_2$ , we get
$$
\begin{array}{l} \left. \left\| \operatorname {p r o j} _ {V} \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}) \right\| _ {2} = \left| \mathbb {E} \left[ - \ell_ {\sigma} ^ {\prime} (| \mathbf {x} _ {\mathbf {w}} |) \cdot y \cdot \mathbf {x} _ {\mathbf {v}} \right] \right| \right. \\ = \Big | \mathbb {E} \bigg [ - \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathrm {s i g n} (\langle \mathbf {w} ^ {*}, \mathbf {x} \rangle) \cdot (1 - 2 \mathbb {1} \{y \neq \mathrm {s i g n} (\langle \mathbf {w} ^ {*}, \mathbf {x} \rangle) \}) \cdot \mathbf {x _ {v}} \bigg ] \Big | \\ \end{array}
$$
Let  $F(y,\mathbf{x})$  denote  $1 - 2\mathbb{1}\{y\neq \mathrm{sign}(\langle \mathbf{w}^{*},\mathbf{x}\rangle)\}$ . We may write  $\mathbf{x}_{\mathbf{v}}$  as  $|\mathbf{x}_{\mathbf{v}}|\cdot \mathrm{sign}(\mathbf{x}_{\mathbf{v}})$  and let  $\mathcal{G}\subseteq \mathbb{R}^2$  such that  $\mathrm{sign}(\mathbf{x}_{\mathbf{v}})\cdot \mathrm{sign}(\langle \mathbf{w}^{*},\mathbf{x}\rangle) = -1$  iff  $\mathbf{x}\in \mathcal{G}$ .
Figure 1: The Gaussian mass in each of the regions labelled  $A_{1}$  and  $A_{2}$  is proportional to the corresponding term appearing in the statement of Proposition 4.2. As  $\sigma$  tends to 0, the Gaussian mass of region  $A_{2}$  shrinks faster than the one of region  $A_{1}$ , since both the height  $(\sigma)$  and the width  $\left(\frac{\sigma}{\tan\theta}\right)$  of  $A_{2}$  are proportional to  $\sigma$ , while the width of  $A_{1}$  is not affected (the height is  $\sigma /3$ ). Lemma 4.3 demonstrates that a similar property is universally testable under any nice Poincaré distribution.
Then,  $\mathrm{sign}(\mathbf{x_v})\cdot \mathrm{sign}(\langle \mathbf{w}^*,\mathbf{x}\rangle) = \mathbb{1}\{\mathbf{x}\notin \mathcal{G}\} -\mathbb{1}\{\mathbf{x}\in \mathcal{G}\}$ . We get
$$
\begin{array}{l} \| \operatorname {p r o j} _ {V} \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}) \| _ {2} = \\ = \Big | \mathbb {E} \bigg [ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot (\mathbb {1} \{\mathbf {x} \in \mathcal {G} \} - \mathbb {1} \{\mathbf {x} \notin \mathcal {G} \}) \cdot F (y, \mathbf {x}) \cdot | \mathbf {x _ {v}} | \cdot \bigg ] \Big | \\ \geq \mathbb {E} \Big [ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathbb {1} \{\mathbf {x} \in \mathcal {G} \} \cdot F (y, \mathbf {x}) \cdot | \mathbf {x _ {v}} | \Big ] - \mathbb {E} \Big [ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathbb {1} \{\mathbf {x} \notin \mathcal {G} \} \cdot F (y, \mathbf {x}) \cdot | \mathbf {x _ {v}} | \Big ] \\ \end{array}
$$
Let  $A_1' = \mathbb{E}[\ell_{\sigma}'(|\mathbf{x_w}|) \cdot \mathbb{1}\{\mathbf{x} \in \mathcal{G}\} \cdot F(y, \mathbf{x}) \cdot |\mathbf{x_v}|]$  and  $A_2' = \mathbb{E}[\ell_{\sigma}'(|\mathbf{x_w}|) \cdot \mathbb{1}\{\mathbf{x} \notin \mathcal{G}\} \cdot F(y, \mathbf{x}) \cdot |\mathbf{x_v}|]$ .
In the Massart noise case  $\mathbb{E}_{y|\mathbf{x}}[F(y,\mathbf{x})] = 1 - 2\eta (\mathbf{x})\in [1 - 2\eta ,1]$ , where  $1 - 2\eta >0$ . Therefore, we have that  $A_1^\prime \geq (1 - 2\eta)\cdot \mathbb{E}[\ell_{\sigma}'(|\mathbf{x_w}|)\cdot \mathbb{1}\{\mathbf{x}\in \mathcal{G}\} \cdot |\mathbf{x_v}|]$ . When the noise is adversarial, we have  $A_1^\prime \geq \mathbb{E}[\ell_{\sigma}'(|\mathbf{x_w}|)\cdot \mathbb{1}\{\mathbf{x}\in \mathcal{G}\} \cdot |\mathbf{x_v}|] - 2\mathbb{E}[\ell_{\sigma}'(|\mathbf{x_w}|)\cdot \mathbb{1}\{\mathbf{x}\in \mathcal{G}\} \cdot \mathbb{1}\{y\neq \mathrm{sign}(\langle \mathbf{w}^*,\mathbf{x}\rangle)\} \cdot |\mathbf{x_v}|\}$ .
For any  $\alpha \geq \frac{\sigma}{2\tan\theta}$ , we have that
$$
\mathbb {E} \left[ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathbb {1} \{\mathbf {x} \in \mathcal {G} \} \cdot | \mathbf {x _ {v}} | \right] \geq \mathbb {E} \left[ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathbb {1} \{\mathbf {x} \in \mathcal {G} \} \cdot \mathbb {1} _ {\{| \mathbf {x _ {w}} | \leq \frac {\sigma}{6} \}} \cdot | \mathbf {x _ {v}} | \right]
$$
(since terms are positive)
$$
\geq \mathbb {E} \left[ \frac {1}{\sigma} \cdot \mathbb {1} \left\{\mathbf {x} \in \mathcal {G} \right\} \cdot \mathbb {1} \left\{| \mathbf {x} _ {\mathbf {w}} | \leq \frac {\sigma}{6} \right\} \cdot | \mathbf {x} _ {\mathbf {v}} | \right]
$$
(by Proposition A.4)
$$
\begin{array}{l} \geq \frac {\alpha}{\sigma} \cdot \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} \in \mathcal {G} \right\} \cdot \mathbb {1} _ {\{| \mathbf {x _ {w}} | \leq \frac {\sigma}{6} \}} \cdot \mathbb {1} _ {\{| \mathbf {x _ {v}} | \geq \alpha \}} \right] \\ \geq \frac {\alpha}{\sigma} \cdot \mathbb {E} \left[ \mathbb {1} _ {\{| \mathbf {x} _ {w} | \leq \frac {\sigma}{6} \}} \cdot \mathbb {1} _ {\{| \mathbf {x} _ {v} | \geq \alpha \}} \right] \quad (\text {s e e F i g u r e 1}) \\ = \frac {\alpha}{\sigma} \cdot \mathbb {P} \left[ | \mathbf {x} _ {\mathbf {w}} | \leq \frac {\sigma}{6} \text {a n d} | \mathbf {x} _ {\mathbf {v}} | \geq \alpha \right] \stackrel {\text {d e f}} {=} A _ {1} \\ \end{array}
$$
Moreover, for some universal constant  $C' > 0$ , we similarly have
$$
\mathbb {E} \left[ \ell_ {\sigma} ^ {\prime} \left(\left| \mathbf {x} _ {\mathbf {w}} \right|\right) \cdot \mathbb {1} \left\{\mathbf {x} \notin \mathcal {G} \right\} \cdot F (y, \mathbf {x}) \cdot \left| \mathbf {x} _ {\mathbf {v}} \right| \right] \leq \mathbb {E} \left[ \ell_ {\sigma} ^ {\prime} \left(\left| \mathbf {x} _ {\mathbf {w}} \right|\right) \cdot \mathbb {1} \left\{\mathbf {x} \notin \mathcal {G} \right\} \cdot \left| \mathbf {x} _ {\mathbf {v}} \right| \right] (s i n c e F (y, \mathbf {x}) \leq 1)
$$
$$
\leq \mathbb {E} \left[ \frac {C ^ {\prime}}{\sigma} \cdot \mathbb {1} _ {\{| \mathbf {x} _ {\mathbf {w}} | \leq \frac {\sigma}{2} \}} \cdot \mathbb {1} \left\{\mathbf {x} \notin \mathcal {G} \right\} \cdot | \mathbf {x} _ {\mathbf {v}} | \right]
$$
$$
\leq \frac {C ^ {\prime}}{\sigma} \cdot \mathbb {E} \left[ \mathbb {1} _ {\{| \mathbf {x} _ {\mathbf {w}} | \leq \frac {\sigma}{2} \}} \cdot \mathbb {1} _ {\{| \mathbf {x} _ {\mathbf {v}} | \leq \frac {\sigma}{2 \tan \theta} \}} \cdot | \mathbf {x} _ {\mathbf {v}} | \right]
$$
(see Figure 1)
$$
\leq \frac {C ^ {\prime}}{2 \cdot \tan \theta} \cdot \mathbb {E} \left[ \mathbb {1} _ {\{| \mathbf {x _ {w}} | \leq \frac {\sigma}{2} \}} \cdot \mathbb {1} _ {\{| \mathbf {x _ {v}} | \leq \frac {\sigma}{2 \tan \theta} \}} \right]
$$
$$
\leq \frac {C ^ {\prime}}{2 \cdot \tan \theta} \cdot \mathbb {P} \left[ | \mathbf {x _ {w}} | \leq \frac {\sigma}{2} \right] \stackrel {\mathrm {d e f}} {=} A _ {2}
$$
Hence, we have shown that, in the Massart noise case, we have  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w})\| _2\geq (1 - 2\eta)A_1 - A_2$  as desired. For the adversarial noise case, it remains to bound the following quantity
$$
\begin{array}{l} 2 \mathbb {E} \left[ \ell_ {\sigma} ^ {\prime} (| \mathbf {x _ {w}} |) \cdot \mathbb {1} _ {\{\mathbf {x} \in \mathcal {G} \}} \cdot \mathbb {1} _ {\{y \neq \mathrm {s i g n} (\langle \mathbf {w} ^ {*}, \mathbf {x} \rangle) \}} \cdot | \mathbf {x _ {v}} | \right] \leq \\ \leq \frac {2 C ^ {\prime}}{\sigma} \cdot \mathbb {E} \left[ \mathbb {1} _ {\{\mathbf {x} \in \mathcal {G} \}} \cdot \mathbb {1} _ {\{| \mathbf {x} _ {\mathbf {w}} | \leq \frac {\sigma}{2} \}} \cdot \mathbb {1} _ {\{y \neq \mathrm {s i g n} (\langle \mathbf {w} ^ {*}, \mathbf {x} \rangle) \}} \cdot | \mathbf {x} _ {\mathbf {v}} | \right] \\ \leq \frac {2 C ^ {\prime}}{\sigma} \cdot \mathbb {E} \left[ \mathbb {1} _ {\{| \mathbf {x _ {w}} | \leq \frac {\sigma}{2} \}} \cdot \mathbb {1} _ {\{y \neq \mathrm {s i g n} (\langle \mathbf {w} ^ {*}, \mathbf {x} \rangle) \}} \cdot | \mathbf {x _ {v}} | \right] \\ \leq \frac {2 C ^ {\prime}}{\sigma} \cdot \sqrt {\operatorname {o p t}} \cdot \sqrt {\mathbb {E} \left[ | \mathbf {x _ {v}} | ^ {2} \cdot \mathbb {1} _ {\{| \mathbf {x _ {w}} | \leq \frac {\sigma}{2} \}} \right]} \stackrel {\text {d e f}} {=} A _ {3} \\ \end{array}
$$
where the final inequality follows from Cauchy-Schwarz inequality.
We restate Lemma 4.3 here for convenience.