SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma C.2 (Universally Testable Structure of Surrogate Loss). Let  $D_{\mathcal{X}\mathcal{Y}}$  be any distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Consider  $\mathcal{L}_{\sigma}$  as in Equation (4.1). Then, there is a universal constant  $C > 0$  and a tester that given a unit vector  $\mathbf{w} \in \mathbb{R}^d$ ,  $\delta \in (0,1)$ ,  $\eta < 1/2$ ,  $\gamma > 0$ ,  $\lambda \geq 1$ ,  $\sigma \leq \frac{1}{C\lambda^C}$  and a set  $S$  of i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  with size at least  $C \cdot \frac{d^4}{\sigma^2\delta} \log(d)\lambda^C$ , runs in time  $\text{poly}(d,\lambda,\frac{1}{\sigma},\frac{1}{\delta},\log(\frac{1}{\gamma}))$  and satisfies the following specifications
- If the noise is Massart with associated rate  $\eta$  and  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S)\| _2\leq \frac{1 - 2\eta}{C\lambda^C\gamma^4}$  then either  $\angle (\mathbf{w},\mathbf{w}_S^*)\leq \frac{C\lambda^C(1 + \gamma^4)}{1 - 2\eta}\cdot \sigma$  or  $\angle (-\mathbf{w},\mathbf{w}_S^*)\leq \frac{C\lambda^C(1 + \gamma^4)}{1 - 2\eta}\cdot \sigma$
- If the noise is adversarial with  $\mathsf{opt}_S \leq \frac{\sigma}{C\lambda^C}$  and  $\|\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S)\|_2 < \frac{1}{C\lambda^C\gamma^4}$  then either  $\angle (\mathbf{w},\mathbf{w}_S^*) \leq C\lambda^C (1 + \gamma^4) \cdot \sigma$  or  $\angle (-\mathbf{w},\mathbf{w}_S^*) \leq C\lambda^C (1 + \gamma^4) \cdot \sigma$ .
Proof of Lemma 4.3. The testing algorithm receives  $\mathbf{w} \in \mathbb{S}^{d-1}$ ,  $\delta \in (0,1)$ ,  $\eta < 1/2$ ,  $\gamma > 0$ ,  $\lambda \geq 1$ ,  $\sigma \leq \frac{1}{2\lambda}$  and a set  $S \subset \mathbb{R}^d \times \{\pm 1\}$  and does the following for some sufficiently large  $C_1 > 0$
1. If  $\mathbb{P}_{(\mathbf{x},y)\in S}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \frac{\sigma}{6} ]\leq \frac{\sigma}{C_1\lambda^{C_1}}$  or  $\mathbb{P}_{(\mathbf{x},y)\in S}[\left|\langle \mathbf{w},\mathbf{x}\rangle \right|\leq \frac{\sigma}{2}] > \sigma \cdot C_1\lambda^{C_1}$ , then reject.
2. Compute the  $(d - 1)\times (d - 1)$  matrices  $M_S^+$  and  $M_S^-$  as follows:
$$
M _ {S} ^ {+} = \underset {(\mathbf {x}, y) \in S} {\mathbb {E}} \left[ (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} _ {\{| \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{2} \}} \right]
$$
$$
M _ {S} ^ {-} = \underset {(\mathbf {x}, y) \in S} {\mathbb {E}} \left[ (\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} _ {\{| \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \}} \right]
$$
3. Run the (maximum singular value) spectral tester of Proposition A.2 on  $M_S^+$  given  $\delta \gets \frac{\delta}{4}$ ,  $\lambda \gets C_1\lambda^{C_1}$  and  $\theta \gets \frac{C_1\sigma\lambda^{C_1}}{2}$ , i.e., reject if the maximum singular value of  $M_S^+$  is greater than  $\sigma \cdot C_1\lambda^{C_1}$ .
4. Run the (minimum singular value) spectral tester of Proposition A.2 on  $M_S^-$  given  $\delta \gets \frac{\delta}{4}$ ,  $\lambda \gets C_1\lambda^{C_1}$  and  $\theta \gets \frac{2\sigma}{C_1\lambda^{C_1}}$ , i.e., reject if the minimum singular value of  $M_S^-$  is less than  $\frac{\sigma}{C_1\lambda^{C_1}}$ .
5. Run the hypercontractivity tester on  $S' = \{\operatorname{proj}_{\perp \mathbf{w}} \mathbf{x} : (\mathbf{x}, y) \in S \text{ and } |\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma\}$ , i.e., solve an appropriate SDP (see Prop. 3.5 with  $\gamma \gets \gamma$ ,  $\delta \gets \delta / 4$ ) and reject if the solution is larger than a specified threshold. Otherwise, accept.
For part (a), we suppose that the testing algorithm has accepted  $S$ . Therefore,  $S$  has passed all the tests required for part (a) of Lemma 3.2 and there exists a universal constant  $C' > 0$  such that
$$
\mathbb {P} _ {(\mathbf {x}, y) \in S} \left[ | \langle \mathbf {v}, \mathbf {x} \rangle | \geq \frac {1}{C ^ {\prime} \lambda^ {C ^ {\prime}}} \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \sigma \right] \geq \frac {1}{C ^ {\prime} \lambda^ {C ^ {\prime}} \gamma^ {4}}
$$
Moreover, we have  $\frac{\sigma}{C' \lambda^{C'}} < \mathbb{P}_{(\mathbf{x}, y) \in S} [|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \frac{\sigma}{6}] \leq \mathbb{P}_{(\mathbf{x}, y) \in S} [|\langle \mathbf{w}, \mathbf{x} \rangle| \leq \frac{\sigma}{2}] < \sigma \cdot C' \lambda^{C'}$  and
$$
\underset {(\mathbf {x}, y) \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {2} \right| | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{2} ] \leq C ^ {\prime} \lambda^ {C ^ {\prime}}
$$
$$
\underset {(\mathbf {x}, y) \in S} {\mathbb {E}} \left[ \langle \mathbf {v}, \mathbf {x} \rangle^ {2} \mid | \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \right] \geq \frac {1}{C ^ {\prime} \lambda^ {C ^ {\prime}}}
$$
Since Proposition 4.2 holds for any distribution, it will also hold for the empirical distribution (uniform on  $S$ ). We apply Proposition 4.2 with  $\alpha = \frac{1}{C' \lambda^{C'}}$  to lower bound  $\| \nabla_{\mathbf{w}} \mathcal{L}_{\sigma}(\mathbf{w}; S) \|_2$  (or  $\| \nabla_{\mathbf{w}} \mathcal{L}_{\sigma}(-\mathbf{w}; S) \|_2$ ) as follows
$$
\| \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}; S) \| _ {2} \geq A _ {1} (\alpha) - A _ {2} - A _ {3}
$$
(adversarial noise case)
$$
\left\| \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}; S) \right\| _ {2} \geq (1 - 2 \eta) \cdot A _ {1} (\alpha) - A _ {2}
$$
(Massart noise case)
Combining the above inequalities with the bounds implied by the fact that  $S$  has passed the tests, concludes the proof of part (a), since (after observing that  $\tan \theta \geq \theta$ ) we get
$$
\left\| \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}; S) \right\| _ {2} \geq \frac {3}{C \lambda^ {C} \gamma^ {4}} - \frac {\sqrt {C} \sigma \lambda^ {C / 2}}{\theta} - \sqrt {\frac {\operatorname {o p t} \cdot C \cdot \lambda^ {C}}{\sigma}} \quad \text {(a d v e r s a r i a l n o i s e c a s e)}
$$
$$
\| \nabla_ {\mathbf {w}} \mathcal {L} _ {\sigma} (\mathbf {w}; S) \| _ {2} \geq \frac {3 (1 - \eta)}{C \lambda^ {C} \gamma^ {4}} - \frac {\sqrt {C} \sigma \lambda^ {C / 2}}{\theta} \quad \text {(M a s s a r t n o i s e c a s e)}
$$
For part (b), we follow a similar recipe as the one used to prove part (b) of Lemma 3.2, i.e., we use the following reasoning to show that the tests will pass with probability at least  $1 - \delta$
1. We assume that the marginal distribution  $D_{\mathcal{X}}$  is  $\lambda$ -nice and  $\gamma$ -Poincaré.
2. We use Proposition A.3 to bound the values of the tested quantities under the true distribution.
3. We use appropriate concentration results (Hoeffding/Chernoff Bounds and Proposition A.2) to show that, since  $|S|$  is large enough, each of the empirical quantities at hand does not deviate a lot from its mean.
This concludes the proof of Lemma 4.3.
C.3 Proof of Main Theorem
We restate the main Theorem here for convenience.
Theorem C.3 (Efficient Universal Tester-Learner for Halfspaces). Let  $D_{\mathcal{X}\mathcal{Y}}$  be any distribution over  $\mathbb{R}^d \times \{\pm 1\}$ . Let  $\mathcal{C}$  be the class of origin centered halfspaces in  $\mathbb{R}^d$ . Then, for any  $\lambda \geq 1$ ,  $\gamma > 0$ ,  $\epsilon > 0$  and  $\delta \in (0,1)$ , there exists an universal tester-learner for  $\mathcal{C}$  w.r.t. the class of  $\lambda$ -nice and  $\gamma$ -Poincaré marginals up to error  $\mathrm{poly}(\lambda) \cdot (1 + \gamma^4) \cdot \mathrm{opt} + \epsilon$ , where  $\mathrm{opt} = \min_{\mathbf{w} \in \mathbb{S}^{d - 1}} \mathbb{P}_{D_{\mathcal{X}\mathcal{Y}}}[y \neq \mathrm{sign}(\langle \mathbf{w}, \mathbf{x} \rangle)]$ , and error probability at most  $\delta$ , using a number of samples and running time  $\mathrm{poly}(d, \lambda, \gamma, \frac{1}{\epsilon}, \log \frac{1}{\delta})$ .
Proof of Theorem 4.1. Note that we will give the algorithm for  $\delta \gets \delta' = 1/3$  since we can reduce the probability of failure with repetition (repeat  $O(\log \frac{1}{\delta})$  times, accept if the rate of acceptance is  $\Omega(1)$  and output the halfspace achieving the minimum test error among the halfspaces returned).
For reader's convenience, we now restate the algorithm on page 9 (note that together with the algorithm we include additional detail relevant to the analysis). The algorithm receives  $\lambda \geq 1$ ,  $\gamma > 0$ ,  $\epsilon > 0$  and  $\eta \in (0, 1/2) \cup \{1\}$  (say  $\eta = 1$  when we are in the agnostic case) and does the following for some appropriately large universal constant  $C_1, C_2 > 0$ .
1. First, create a set of parameters  $\Sigma$  and parameters  $E = \frac{\epsilon}{C_1\lambda^{C_1}}$  and  $A > 0$  as follows. If  $\eta = 1$ , then  $\Sigma$  is an  $\frac{E}{C_1\lambda^{C_1}}$ -cover of the interval  $[0, \frac{1}{C_1\lambda^{C_1}}]$  and  $A = \frac{1}{C_1\lambda^{C_1}\gamma^4}$ . Otherwise, let  $\Sigma = \left\{\frac{E \cdot (1 - 2\eta)}{C_1\lambda^{C_1}(1 + \gamma^4)}\right\}$  and  $A = \frac{1 - 2\eta}{C_1\lambda^{C_1}\gamma^4}$ .
2. Then, draw a set  $S_{1}$  of  $C_2\left(\frac{\lambda d}{\gamma\epsilon}\right)^{C_2}$  i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  and run PSGD, as specified in Proposition A.5 with  $\epsilon \gets A$ ,  $\delta \gets \frac{\delta'}{C_1}$  on the loss  $\mathcal{L}_{\sigma}$  for each  $\sigma \in \Sigma$ .
4. Draw a fresh set  $S_{2}$  of  $C_2\left(\frac{\lambda d}{\gamma\epsilon}\right)^{C_2}$  i.i.d. samples from  $D_{\mathcal{X}\mathcal{Y}}$  and compute for each  $(\mathbf{w},\sigma)\in L$  the value  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S_2)\| _2$ . If, for some  $\sigma \in \Sigma$ ,  $\| \nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w};S_2)\| _2 > A$  for all  $(\mathbf{w},\sigma)\in L$ , then reject.
6. Run the following tests for each  $(\mathbf{w},\sigma)\in L$  to ensure that part (a) of Lemma 4.3 holds for each of the elements of  $L$ , i.e., that any stationary point of the surrogate loss that lies in  $L$  is angularly close to the empirical risk minimizer (or the same holds for the inverse vector).
- If  $\mathbb{P}_{(\mathbf{x},y)\in S_2}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \frac{\sigma}{6} ]\leq \frac{\sigma}{C_1\lambda^{C_1}}$  or  $\mathbb{P}_{(\mathbf{x},y)\in S_2}[\left|\left\langle \mathbf{w},\mathbf{x}\right\rangle \right|\leq \frac{\sigma}{2}] > \sigma \cdot C_1\lambda^{C_1}$ , then reject.
- Compute the  $(d - 1) \times (d - 1)$  matrices  $M_{S_2}^+$  and  $M_{S_2}^-$  as follows:
$$
M _ {S _ {2}} ^ {-} = \underset {(\mathbf {x}, y) \in S _ {2}} {\mathbb {E}} \left[ (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\operatorname {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T} \cdot \mathbb {1} _ {\{| \langle \mathbf {w}, \mathbf {x} \rangle | \leq \frac {\sigma}{6} \}} \right]
$$
- Run the (maximum singular value) spectral tester of Proposition A.2 on  $M_{S_2}^+$  given  $\delta \gets \frac{\delta'}{C_1}$ ,  $\lambda \gets C_1 \lambda^{C_1}$  and  $\theta \gets \frac{C_1 \sigma \lambda^{C_1}}{2}$ , i.e., reject if the maximum singular value of  $M_{S_2}^+$  is greater than  $\sigma \cdot C_1 \lambda^{C_1}$ .
- Run the (minimum singular value) spectral tester of Proposition A.2 on  $M_{S_2}^-$  given  $\delta \gets \frac{\delta'}{C_1}, \lambda \gets C_1\lambda^{C_1}$  and  $\theta \gets \frac{2\sigma}{C_1\lambda^{C_1}}$ , i.e., reject if the minimum singular value of  $M_{S_2}^-$  is less than  $\frac{\sigma}{C_1\lambda^{C_1}}$ .
- Run the hypercontractivity tester on  $S' = \{\mathrm{proj}_{\perp \mathbf{w}} \mathbf{x} : (\mathbf{x}, y) \in S_2 \text{ and } |\langle \mathbf{w}, \mathbf{x} \rangle| \leq \sigma\}$ , i.e., solve an appropriate SDP (see Prop. 3.5 with  $\gamma \gets \gamma$ ,  $\delta \gets \delta'/C_1$ ) and reject if the solution is larger than a specified threshold.
7. Run the following tests for each pair of the form  $(\mathbf{w},\sigma)$  and  $(- \mathbf{w},\sigma)$  where  $(\mathbf{w},\sigma) \in L$  to ensure that part (a) of Lemma 3.1 is activated, i.e., that the distance of a vector from the empirical risk minimizer is an accurate proxy for the error of the corresponding halfspace. Set  $\theta (\sigma) = \frac{(1 + \gamma^4)\sigma}{A\gamma^4}$ .
- If  $\mathbb{P}_{(\mathbf{x},\mathbf{y})\in S_2}[|\langle \mathbf{w},\mathbf{x}\rangle |\leq \theta ] > C_1\lambda^{C_1}\theta$  then reject.
- Compute the  $(d - 1) \times (d - 1)$  matrix  $M_{S_2}$  as follows:
$$
M _ {S _ {2}} = \underset {(\mathbf {x}, y) \in S _ {2}} {\mathbb {E}} \left[ \sum_ {i = 2} ^ {\infty} \frac {(\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) (\mathrm {p r o j} _ {\perp \mathbf {w}} \mathbf {x}) ^ {T}}{(i - 1) ^ {2}} \mathbb {1} \{| \langle \mathbf {w}, \mathbf {x} \rangle | \in [ (i - 1) \theta , i \theta) \} \right]
$$
- Run the spectral tester of Proposition A.2 on  $M_S$  given  $\delta \leftarrow \frac{\delta'}{C_1}$ ,  $\lambda \leftarrow C_1\lambda^{C_1}$  and  $\theta \leftarrow \frac{C_1}{2}\theta \lambda^{C_1}$ , i.e., compute  $\| M_S\|_{\mathrm{op}}$  and if  $\| M_S\|_{\mathrm{op}} > C_1\theta \lambda^{C_1}$ , then reject.
For the following, let  $\alpha = 1$  in the Massart noise case and  $\alpha = C_1\lambda^{C_1}\gamma^4$  in the adversarial noise case. Consider also  $\mathrm{opt}_{S_2}$  to be the error of the origin-centered halfspace with the minimum empirical error on  $S_{2}$  and  $\mathbf{w}_{S_2}^*$  the corresponding optimum vector.
Soundness. We first prove the soundness condition, i.e., that the following implication holds with probability at least  $1 - \delta'$  over the samples:
$$
\text {I f} \mathbb {P} _ {D x y} [ y \neq \operatorname {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) ] \leq \alpha \cdot \operatorname {o p t} + \epsilon
$$
The tester accepts only if for every  $\sigma \in \Sigma$ , we have some  $\mathbf{w} \in L$  with  $\|\nabla_{\mathbf{w}} \mathcal{L}_{\sigma}(\mathbf{w}; S_2)\|_2 \leq A$  (step 4) and for which part (a) of each of Lemmas 4.3 (step 6) and 3.1 (step 7) is activated. Therefore, in the Massart noise case, for any  $\sigma \in \Sigma$ , there is some  $\mathbf{w}$  such that either  $(\mathbf{w}, \sigma) \in L$  or  $(- \mathbf{w}, \sigma) \in L$  and also
$$
\angle \left(\mathbf {w}, \mathbf {w} _ {S _ {2}} ^ {*}\right) \leq \frac {1 + \gamma^ {4}}{\gamma^ {4}} \cdot \frac {\sigma}{A} \stackrel {\text {d e f}} {=} \theta \tag {C.1}
$$
$$
\underset {S _ {2}} {\mathbb {P}} [ y \neq \operatorname {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) ] \leq \operatorname {o p t} _ {S _ {2}} + C ^ {\prime} \lambda^ {C ^ {\prime}} \cdot \theta \tag {C.2}
$$
In the adversarial noise case, the above are true conditional on  $\sigma$  being such that  $\mathrm{opt}_{S_2} \leq \frac{\sigma}{C' \lambda^{C'}}$ .
Therefore, in the Massart noise case, the above are true for  $\sigma = \frac{E(1 - 2\eta)}{C_1\lambda^{C_1}(1 + \gamma^4)}$  which gives
$$
\underset {S _ {2}} {\mathbb {P}} [ y \neq \mathrm {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) ] \leq \mathsf {o p t} _ {S _ {2}} + C ^ {\prime} \lambda^ {C ^ {\prime}} E
$$
In the agnostic case, condition C.2 is true for some  $\sigma \in [0,\frac{1}{C_1\lambda^{C_1}} ]$  such that
$$
\frac {\sigma}{C ^ {\prime} \lambda^ {C ^ {\prime}}} - \frac {1}{C _ {1} \lambda^ {C _ {1}}} \leq \operatorname {o p t} _ {S _ {2}} \leq \frac {\sigma}{C ^ {\prime} \lambda^ {C ^ {\prime}}}
$$
unless  $\mathrm{opt} > \frac{1}{C_1C'\lambda^{C_1 + C'}}$ , in which case any halfspace has error at most  $1 = \mathrm{opt} \cdot (C_1C'\lambda^{C_1 + C'})$ . Hence we get
$$
\underset {S _ {2}} {\mathbb {P}} [ y \neq \mathrm {s i g n} (\langle \mathbf {w}, \mathbf {x} \rangle) ] \leq \mathrm {p o l y} (\lambda) \cdot (1 + \gamma^ {4}) \cdot \mathsf {o p t} _ {S _ {2}} + C ^ {\prime} \lambda^ {C ^ {\prime}} E
$$
Soundness follows from the fact that if  $|S_2|$  is sufficiently large (but still polynomial in every parameter, since the VC dimension of the class of halfspaces in  $\mathbb{R}^d$  is  $d + 1$ ), then  $|\mathrm{opt}_{S_2} - \mathrm{opt}| \leq \frac{\epsilon}{C_1\lambda^{C_1}(1 + \gamma)^4}$  with probability at least  $1 - \delta'$ .
Note that only at most  $|S_2|$  many terms below are non-zero, hence  $M_{S_2}$  can be computed efficiently.
Completeness. Suppose now that the marginal is indeed  $\lambda$ -nice and  $\gamma$ -Poincaré. Then, for sufficiently large  $S_{1}$ , after step 3,  $L$  will contain a stationary point of  $\mathcal{L}_{\sigma}(\cdot; D_{\mathcal{X}\mathcal{Y}})$  for each  $\sigma \in \Sigma$ , due to Proposition A.5. If  $S_{2}$  is large enough, then steps 4, 6 and 7 will each accept with probability at least  $1 - \delta' / C_{1}$ , due to part (b) of Lemmas 4.3 and 3.1, as well as the fact that each coordinate of  $\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w}; S_{2})$  has bounded second moment (Proposition A.3) and therefore  $\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w}; S_{2})$  is concentrated around  $\nabla_{\mathbf{w}}\mathcal{L}_{\sigma}(\mathbf{w}; D_{\mathcal{X}\mathcal{Y}})$  for any fixed  $\mathbf{w}$  such that  $(\mathbf{w}, \sigma) \in L$  (we also need a union bound over  $L$ ). Hence, in total, the tester will accept with probability at least  $1 - \delta'$ .