SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we have used that  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\cdot \mathbf{w}^{*} = \| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\|_{2}$  (by Equation (1)) and  $(\mathbf{u} - \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}])\cdot \mathbf{w}^{*}\geq -\sup_{\mathbf{w}:||\mathbf{w}||_{2} = 1}(\mathbf{u} - \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}])\cdot \mathbf{w}$ , which is greater than or equal to  $-\frac{\kappa^2}{8}\| \mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y\mathbf{x}]\|_{2}$ , by the concentration argument used above.
As  $\cos \theta (\mathbf{w}_0,\mathbf{w}^*)\leq 1 - \theta (\mathbf{w}_0,\mathbf{w}^*)^2 /4$  , we further have
$$
\theta (\mathbf {w} _ {0}, \mathbf {w} ^ {*}) ^ {2} / 4 \leq \frac {\kappa^ {2} / 4}{1 + \kappa^ {2} / 8} \leq \kappa^ {2} / 4,
$$
yielding the desired result that  $\theta (\mathbf{w}_0,\mathbf{w}^*)\leq \kappa$
We now leverage Lemma 2.1 to argue about correctness of Algorithm 4, which provides an implementable initialization procedure.
Proof. Let  $\epsilon' = \epsilon/(1 - 2\eta)$  and  $J = \lceil \log_2(1 / \epsilon') \rceil$ . As we have shown in the proof of Lemma 2.1, vector  $\mathbf{u}_j$  satisfies
$$
\mathbf {P r} \left[ \| \mathbf {u} _ {j} - \mathbf {E} [ y \mathbf {x} ] \| _ {2} \geq \frac {1}{4} \sqrt {\frac {2}{\pi}} (1 - 2 \eta) p _ {j} \right] \leq 2 \exp \left(- \frac {n _ {j} (1 - 2 \eta) ^ {2} p _ {j} ^ {2}}{3 2 \pi d}\right).
$$
Since the algorithm runs for at most  $J = \lceil \log_2(1 / \epsilon ') \rceil$  iterations, applying the union bound yields:
$$
\mathbf {P r} \left[ \| \mathbf {u} _ {j} - \mathbf {E} [ y \mathbf {x} ] \| _ {2} \geq \frac {1}{4} \sqrt {\frac {2}{\pi}} (1 - 2 \eta) p _ {j}, \forall j = 1, \dots , J \right] \leq 2 J \exp \left(- \frac {n _ {j} (1 - 2 \eta) ^ {2} p _ {j} ^ {2}}{3 2 \pi d}\right) \leq \delta ,
$$
where we plugged in  $n_j = \left\lceil \frac{32\pi d(\log(1 / \delta) + \log\log(1 / \epsilon'))}{(1 - 2\eta)^2p_j^2}\right\rceil$ . Furthermore, recall that we have proved in Lemma 2.1 that  $\| \mathbf{E}[y\mathbf{x}]\|_2 = \sqrt{2 / \pi}(1 - 2\eta)\exp(-t^2 / 2)$ . Hence, with probability at least  $1 - \delta$ , we have that for all  $j = 1,\dots,J$
$$
\sqrt {\frac {2}{\pi}} (1 - 2 \eta) (\exp (- t ^ {2} / 2) - p _ {j} / 4) \leq \| \mathbf {u} _ {j} \| _ {2} \leq \sqrt {\frac {2}{\pi}} (1 - 2 \eta) (\exp (- t ^ {2} / 2) + p _ {j} / 4). \tag {2}
$$
Since we have assumed  $0 \leq t \leq \sqrt{2\log(1/\epsilon')}$ , it must be  $\exp(-t^2/2) \geq \epsilon'$ , thus the claimed bound on  $\hat{p}$  holds if the algorithm loop ends because  $j = \lceil \log_2(1/\epsilon') \rceil$ . Consider now the case that the loop ends before reaching the upper bound on the number of iterations. Observe that when  $p_j$  is still far away from  $\exp(-t^2/2)$ , i.e., when  $p_j > 2\exp(-t^2/2)$ , Equation (2) shows that with probability at least  $1 - \delta$ ,  $\| \mathbf{u}_j \|_2 < (3/4)\sqrt{2/\pi}(1 - 2\eta)p_j$ ; thus, the algorithm will continue to decrease our guess  $p_j$ . On the other hand, if  $p_j$  is already small, i.e., if  $p_j \leq \exp(-t^2/2)$ , Equation (2) implies that  $\| \mathbf{u}_j \|_2 \geq (3/4)\sqrt{2/\pi}(1 - 2\eta)p_j$ , reaching the repeat-until loop termination condition. As any iteration of the algorithm reduces the value of  $p_j$  by a factor of 2, we conclude that the loop ends with  $p_j$  that satisfies  $\frac{1}{2}\exp(-t^2/2) \leq p_j \leq 2\exp(-t^2/2)$ , hence the bound on  $\hat{p}$  follows as  $\hat{p} = 2p_j$ .
The bound on the total number of samples drawn by the algorithm follows by observing that for each iteration  $j$  of the repeat-until loop,  $n_j = O(N_1)$ , while there are  $O(\log(1/\epsilon'))$  total loop iterations.
To complete the proof, it remains to note that the bound on  $\hat{p}$  implies
$$
\sqrt {2 \log (1 / \hat {p})} \leq t \leq \sqrt {2 (\log (4) + \log (1 / \hat {p}))}
$$
Hence  $\kappa$  selected in the algorithm satisfies  $\kappa \leq \frac{1}{5t}$ . Furthermore, since the algorithm runs for at least one iteration, it holds that  $\hat{p} \leq 2p_1 = 1$ . Thus, our choice of  $\kappa$  also guarantees that  $\kappa \leq 1 / (5\sqrt{2\log(4)}) \leq \frac{\pi}{2}$ . It remains to apply Lemma 2.1.
Algorithm 5 Optimization
1: Input:  $\mathbf{w}_0, \hat{\hat{\imath}}, \hat{\gamma}, \eta, N_2$  i.i.d. samples  $(\mathbf{x}^{(i)}, y^{(i)})$  from  $\mathcal{D}$
2:  $\mu_0\gets \frac{(1 - 4\rho)\sqrt{2\pi}}{16(1 - 2\eta)};\rho \gets 0.00098;P(\hat{t},\hat{\gamma})\gets \mathbf{Pr}_{z\sim \mathcal{N}}[-\hat{t}\leq z\leq -\hat{t} +\hat{\gamma} ]$
3: for  $k = 0$  to  $K$  do
4: Let  $\mathcal{E}(\mathbf{w}_k,\hat{t})\coloneqq \{\mathbf{x}: - \hat{t}\leq \mathbf{w}_k\cdot \mathbf{x}\leq -\hat{t} +\hat{\gamma}\}$
5: Let  $\mathbf{g}(\mathbf{w}_k;\mathbf{x}^{(i)},y^{(i)}) = \frac{1}{2} ((1 - 2\eta)\mathrm{sign}(\mathbf{w}_k\cdot \mathbf{x}^{(i)} + \hat{t}) - y^{(i)})\mathrm{proj}_{\mathbf{w}_k^\perp}(\mathbf{x}^{(i)})$
6:  $\widehat{\mathbf{g}} (\mathbf{w}_k)\gets \frac{1}{N_2}\sum_{i = 1}^{N_2}\mathbf{g}(\mathbf{w}_k;\mathbf{x}^{(i)},y^{(i)})\frac{\mathbb{1}\{\mathbf{x}^{(i)}\in\mathcal{E}(\mathbf{w}_k,\hat{t})\}}{P(\hat{t},\hat{\gamma})}$
7:  $\mu_{k}\gets \mu_{k - 1}(1 - \rho)$
8:  $\mathbf{w}_{k + 1}\gets \frac{\mathbf{w}_k - \mu_k\widehat{\mathbf{g}}(\mathbf{w}_k)}{\|\mathbf{w}_k - \mu_k\widehat{\mathbf{g}}(\mathbf{w}_k)\|_2}$
9: end for
10: return  $\mathbf{w}_{K + 1}$
B.2 Optimization
As discussed before, our Optimization procedure (Algorithm 5) can be seen as Riemannian subgradient descent on the unit sphere. Crucial to our analysis is the use of subgradient estimates from Line 5 and Line 6, where we condition on the event that the samples come from a thin band, defined in Line 4. Without this conditioning, the algorithm would correspond to projected subgradient descent of the LeakyReLU loss on the unit sphere. The conditioning effectively changes the landscape of the loss function being optimized, which cannot be argued anymore to even be convex, as the definition of the band depends on the weight vector  $\mathbf{w}$  at which the vector  $\widehat{\mathbf{g}}(\mathbf{w})$  is evaluated. Nevertheless, as we argue in this section, the optimization procedure can be carried out very efficiently, even exhibiting a linear convergence rate. To simplify the notation, in this section we denote the conditioned distribution  $\mathcal{D}|_{\mathcal{E}(\mathbf{w},\hat{t})}$  by  $\mathcal{D}(\mathbf{w},\hat{t})$ . We carry out the analysis assuming the estimate  $\hat{t}$  is within additive  $\epsilon^2$  of the true threshold value  $t$ ; as argued before, this has to be true for at least one estimate  $\hat{t}$  for which the Optimization procedure is invoked.
In the following lemma, we show that if the angle between a weight vector  $\mathbf{w}$  and the target vector  $\mathbf{w}^*$  is from a certain range, we can guarantee that  $\mathbf{g}(\mathbf{w})$  is sufficiently negatively correlated with  $\mathbf{w}^*$ . This condition is then used to argue about progress of our algorithm. The upper bound on  $\theta$  will hold initially, by our initialization procedure, and we will inductively argue that it holds for most iterations. The lower bound, when violated, will imply that the distance between  $\mathbf{w}$  and  $\mathbf{w}^*$  is small, in which case we would have converged to a sufficiently good solution  $\mathbf{w}$ .
Proof. To simplify the notation, in the following we write  $\mathbf{g}(\mathbf{w}) = \mathbf{g}(\mathbf{w};\mathbf{x},y)$ , as  $(\mathbf{x},y)$  is clear from the context. Using the definition of conditional expectations as well as the definition of  $\mathcal{D}(\mathbf{w},\hat{t})$ , we have
$$
\underset {(\mathbf {x}, y) \sim \mathcal {D} (\mathbf {w}, \hat {t})} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} ] = \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} | \mathcal {E} (\mathbf {w}, \hat {t}) ] = \frac {\mathbf {E} _ {(\mathbf {x} , y) \sim \mathcal {D}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} \mathbb {1} \{\mathcal {E} (\mathbf {w} , \hat {t}) \} ]}{\Pr [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}. \tag {3}
$$
We carry out the proof by bounding the numerator  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[\mathbf{g}(\mathbf{w})\cdot \mathbf{w}^*\mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t})\} ]$ . Recall that  $\mathbf{E}_{(\mathbf{x},y)\sim \mathcal{D}}[y|\mathbf{x}] = (1 - 2\eta)\mathrm{sign}(\mathbf{w}^*\cdot \mathbf{x} + t)$ . Furthermore, since the Gaussian distribution is rotationally invariant, we can assume without loss of generality that  $\mathbf{w} = \mathbf{e}_1$  and  $\mathbf{w}^{*} = \cos \theta \mathbf{e}_{1} + \sin \theta \mathbf{e}_{2}$ .
$$
\begin{array}{l} \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ \mathbf {g} (\mathbf {w}) \cdot \mathbf {w} ^ {*} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}) \} ] \\ = \frac {1 - 2 \eta}{2} \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbf {E}} [ (\operatorname {s i g n} (\mathbf {w} \cdot \mathbf {x} + \hat {t}) - \operatorname {s i g n} (\mathbf {w} ^ {*} \cdot \mathbf {x} + t)) \operatorname {p r o j} _ {\mathbf {w} ^ {\perp}} (\mathbf {x}) \cdot \mathbf {w} ^ {*} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}) \} ] \\ = \frac {1 - 2 \eta}{2} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ (\operatorname {s i g n} \left(\mathbf {x} _ {1} + \hat {t}\right) - \operatorname {s i g n} (\cos \theta \mathbf {x} _ {1} + \sin \theta \mathbf {x} _ {2} + t)) \left(\mathbf {x} _ {2} \mathbf {e} _ {2}\right) \right. \\ \cdot \left(\cos \theta \mathbf {e} _ {1} + \sin \theta \mathbf {e} _ {2}\right) \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}) \right\} \\ = (1 - 2 \eta) \sin \theta \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \mathbf {x} _ {2} \right], \tag {4} \\ \end{array}
$$
where in the final equality, we used the fact that under the event  $\mathcal{E}(\mathbf{w},\hat{t})$ $\mathbf{x}_1 = \mathbf{w}\cdot \mathbf{x}$  is greater than or equal to  $-\hat{t}$  and consequently, the expression  $\mathrm{sign}(\mathbf{x}_1 + \hat{t}) - \mathrm{sign}(\cos \theta \mathbf{x}_1 + \sin \theta \mathbf{x}_2 + t)$  equals 2 when  $\cos \theta \mathbf{x}_1 + \sin \theta \mathbf{x}_2 + t$  is less than or equal to zero, and it is zero in all other cases.
Recall that under the assumptions of the lemma,  $|t - \hat{t}| \leq \epsilon'^2 / 8$ . To bound the expectation in Equation (4), we consider two possible cases:  $t \geq 1$  and  $t \leq 1$ . The reason that we discuss these two cases is that when  $t$  is large, under the condition that  $-\hat{t} \leq \mathbf{x}_1 \leq -\hat{t} + \hat{\gamma}$  and  $\mathbf{x}_2 \sin \theta \leq -t - \mathbf{x}_1 \cos \theta$ , it is guaranteed that  $\mathbf{x}_2 \leq 0$ . On the other hand, when  $t$  is small, it is possible that  $\mathbf{x}_2 \geq 0$ . However, we can show that even though  $\mathbf{x}_2 \geq 0$  in some area of the band, the expectation  $\mathbf{E}[\mathbf{x}_2 \mathbb{1}\{\mathcal{E}(\mathbf{w}, \hat{t}), 0 \leq \mathbf{x}_2 \sin \theta \leq -t - \mathbf{x}_1 \cos \theta\}]$  is small since  $t$  is very small. This is handled in the following two claims, under the same assumptions as in the statement of the lemma.
Claim B.3. If  $t \geq 1$ , then
$$
\underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \leq - \frac {2 \operatorname* {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{3 \sqrt {2 \pi}}.
$$
Proof. When  $t \geq 1$ , under event  $\mathcal{E}(\mathbf{w},\hat{t})$ , we have
$$
\begin{array}{l} \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \leq - t + \hat {t} \cos \theta \\ \leq - t + t \cos (\theta) + \epsilon^ {\prime 2} \cos (\theta) / 8 \\ \leq - 2 \sin^ {2} (\theta / 2) t + \epsilon^ {\prime 2} / 8. \\ \end{array}
$$
Since we have assumed  $t \geq 1$  and  $\theta \geq \epsilon'\exp(t^2/2)$ , we further have
$$
\mathbf {x} _ {2} \sin \theta \leq - 2 \sin^ {2} (\theta / 2) t + \epsilon^ {\prime 2} / 8 \leq - \frac {\epsilon^ {\prime 2}}{8} (\exp (t ^ {2}) - 1) \leq 0,
$$
where we used that  $\sin (\theta /2)\geq \theta /4$ , which holds for any  $\theta \leq \pi$ . Therefore,  $\mathbf{x}_2\leq 0$  when  $\mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\sin \theta \leq -t - \mathbf{x}_1\cos \theta \} = 1$ .
Note that conditioning on  $\mathcal{E}(\mathbf{w},\hat{t})$  we have  $\mathbf{x}_1\leq -\hat{t} +\hat{\gamma}$ , hence it holds  $\mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\sin \theta \leq -t-$ $\mathbf{x}_1\cos \theta \} \geq \mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\sin \theta \leq -t - (-\hat{t} +\hat{\gamma})\cos \theta \}$ . In addition, since  $\hat{t}\geq t - \epsilon^{\prime 2} / 8$ , we have  $\mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\sin \theta \leq -t - \mathbf{x}_1\cos \theta \} \geq \mathbb{1}\{\mathcal{E}(\mathbf{w},\hat{t}),\mathbf{x}_2\sin \theta \leq -(1 - \cos \theta)t - (\epsilon^{\prime 2} / 8 + \hat{\gamma})\cos \theta \}$ . Therefore, we have the following upper bound:
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \\ \leq \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t \tan \left(\theta / 2\right) - \left(\epsilon^ {\prime 2} / 8 + \hat {\gamma}\right) \cot \theta \right\} \right], \tag {5} \\ \end{array}
$$
where we used the trigonometric identity  $(1 - \cos \theta) / \sin \theta = \tan (\theta /2)$ . Since  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are independent standard normal random variables, the expectation on the right-hand side of Equation (5) has the following closed form expression:
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t \tan (\theta / 2) - \left(\epsilon^ {\prime 2} / 8 + \hat {\gamma}\right) \cot \theta \right\} \right] \\ = \Pr [ \mathcal {E} (\mathbf {w}, \hat {t}) ] \int_ {- \infty} ^ {- t \tan (\theta / 2) - \left(\epsilon^ {\prime 2} / 8 + \hat {\gamma}\right) \cot \theta} \frac {x}{\sqrt {2 \pi}} \exp (- x ^ {2} / 2) d x \\ = - \frac {\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}} \exp \left(- \frac {1}{2} \left(t \tan \left(\frac {\theta}{2}\right) + \frac {\epsilon^ {\prime 2}}{8} \cot \theta + \hat {\gamma} \cot \theta\right) ^ {2}\right). \tag {6} \\ \end{array}
$$
Let us now bound  $t \tan \left(\frac{\theta}{2}\right) + \frac{\epsilon'^2}{8} \cot \theta + \hat{\gamma} \cot \theta$ . Using the trigonometric inequalities  $\tan (\theta / 2) \leq \theta / 2$  and  $\cot \theta \leq 1 / \theta$ , which hold for  $\theta \in [0, \pi / 2]$ , and recalling that  $\epsilon' \exp(t^2 / 2) \leq \theta \leq 1 / (5t)$ , we have
$$
\begin{array}{l} t \tan \left(\frac {\theta}{2}\right) + \frac {\epsilon^ {\prime 2}}{8} \cot \theta + \hat {\gamma} \cot \theta \leq \frac {t \theta}{2} + \frac {\epsilon^ {\prime 2}}{8 \theta} + \frac {\hat {\gamma}}{\theta} \\ \leq \frac {1}{1 0} + \frac {1}{8} + \frac {1}{2} e ^ {\frac {t ^ {2} - t ^ {2}}{2}} \\ \leq \frac {9}{4 0} + \frac {1}{2} e ^ {\frac {\epsilon^ {\prime 2}}{8} \sqrt {2 \log (1 / \epsilon^ {\prime})}} \leq \frac {7}{8}, \tag {7} \\ \end{array}
$$
where the second inequality is by the definition of  $\hat{\gamma}$  and the bounds on  $\theta$  and the third inequality is by  $\hat{t} - t \leq \frac{\epsilon'^2}{8}$  and  $\hat{t}, t \leq \sqrt{2\log(1 / \epsilon')}$ . Hence, combining Equation (5)-Equation (7), we get
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \leq - \frac {\Pr [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}} \exp (- (7 / 8) ^ {2} / 2) \\ \leq - \frac {2 \Pr [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{3 \sqrt {2 \pi}}, \tag {8} \\ \end{array}
$$
We now proceed to the case where  $t < 1$ .
Claim B.4. If  $t < 1$ , then
$$
\underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \sin \theta \leq - t - \mathbf {x} _ {1} \cos \theta \right\} \right] \leq - \frac {\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{2 \sqrt {2 \pi}}.
$$
Proof. In this case, it is possible that  $\mathbf{x}_2 \geq 0$  when  $\mathbb{1}\{\mathcal{E}(\mathbf{w}), \mathbf{x}_2 \sin \theta \leq -t - \mathbf{x}_1 \cos \theta\} = 1$ . However, since  $\hat{\gamma} = \epsilon' \exp(\hat{t}^2 / 2) / 2 \geq \epsilon'^2 / 8$ , it must be  $-\hat{t} + \hat{\gamma} \geq -t - \epsilon'^2 / 8 + \hat{\gamma} \geq -t$ , indicating that  $-t - (-\hat{t} + \hat{\gamma}) \cos \theta \leq -t + t \cos \theta \leq 0$ , hence  $\mathbf{x}_2 \leq 0$  when  $\mathbb{1}\{\mathcal{E}(\mathbf{w}, \hat{t}), \mathbf{x}_2 \leq -t / \sin \theta - (-\hat{t} + \hat{\gamma}) \cot \theta\} = 1$ . Therefore, we split the indicator  $\mathbb{1}\{\mathcal{E}(\mathbf{w}, \hat{t}), \mathbf{x}_2 \sin \theta \leq -t - \mathbf{x}_1 \cos \theta\}$  into the indicators of three sub-events:
$$
\mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} = \mathbb {1} \left\{\mathcal {E} _ {1} \right\} + \mathbb {1} \left\{\mathcal {E} _ {2} \right\} + \mathbb {1} \left\{\mathcal {E} _ {3} \right\},
$$
$$
\mathcal {E} _ {1} := \mathcal {E} (\mathbf {w}, \hat {t}) \cap \left\{\mathbf {x} _ {2} \leq - t / \sin \theta - (- \hat {t} + \hat {\gamma}) \cot \theta \right\}
$$
$$
\mathcal {E} _ {2} := \mathcal {E} (\mathbf {w}, \hat {t}) \cap \left\{0 \leq \mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\}
$$
$$
\mathcal {E} _ {3} := \mathcal {E} (\mathbf {w}, \hat {t}) \cap \left\{- t / \sin \theta - (- \hat {t} + \hat {\gamma}) \cot \theta \leq \mathbf {x} _ {2} \leq \min  \left\{- t / \sin \theta - \mathbf {x} _ {1} \cot \theta , 0 \right\} \right\}.
$$
For  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}[x_21\mathbb{1}\{\mathcal{E}_1\} ]$  , observe first that
$$
\begin{array}{l} \mathbb {1} \left\{\mathcal {E} _ {1} \right\} = \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t / \sin \theta + \hat {t} \cot \theta - \hat {\gamma} \cot \theta \right\} \\ \geq \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t / \sin \theta + (t - \epsilon^ {\prime 2} / 8) \cot \theta - \hat {\gamma} \cot \theta \right\} \\ \end{array}
$$
Since  $\mathbf{x}_2\leq 0$  under  $\mathcal{E}_1$  , it then holds
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {1} \} \right] \leq \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t / \sin \theta + (t - \epsilon^ {\prime 2} / 8) \cot \theta - \hat {\gamma} \cot \theta \} \right] \\ = \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), \mathbf {x} _ {2} \leq - t \tan (\theta / 2) - \epsilon^ {\prime 2} / 8 \cot \theta - \hat {\gamma} \cot \theta \} \right] \\ = - \frac {\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}} \exp \left(- \frac {1}{2} \left(t \tan \left(\frac {\theta}{2}\right) + \frac {\epsilon^ {\prime 2}}{8} \cot \theta + \hat {\gamma} \cot \theta\right) ^ {2}\right), \\ \end{array}
$$
following similar steps as in Claim B.3, Equation (5)-Equation (6). Again, note that we have assumed  $\epsilon' \exp(t^2/2) \leq \theta \leq 1/(5t)$  and have chosen  $\hat{\gamma} = \epsilon' \exp(t^2/2)/2$ , thus, using the fact that  $\tan(\theta/2) \leq \theta/2$ ,  $\cot \theta \leq 1/\theta$ , we further get:
$$
\underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} [ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {1} \} ] \leq - \frac {\operatorname* {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{\sqrt {2 \pi}} \exp \left(- \frac {1}{2} \left(\frac {t \theta}{2} + \frac {\epsilon^ {\prime 2}}{8 \theta} + \frac {\hat {\gamma}}{\theta}\right) ^ {2}\right) \leq - \frac {2 \operatorname* {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{3 \sqrt {2 \pi}},
$$
using the same arguments as in Equation (7)-Equation (8). For  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}\big[\mathbf{x}_2\mathbb{1}\{\mathcal{E}_3\} \big]$ , note that  $\mathbf{x}_2\leq 0$  under  $\mathcal{E}_3$ , hence  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}\big[\mathbf{x}_2\mathbb{1}\{\mathcal{E}_3\} \big]\leq 0$ .
We now study  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}[x_2\mathbb{1}\{\mathcal{E}_2\} ]$  .Observe that
$$
\begin{array}{l} \mathbb {1} \left\{\mathcal {E} _ {2} \right\} = \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), 0 \leq \mathbf {x} _ {2} \leq - t / \sin \theta - \mathbf {x} _ {1} \cot \theta \right\} \\ \leq \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), 0 \leq \mathbf {x} _ {2} \leq - t / \sin \theta + \hat {t} \cot \theta \right\} \\ \leq \mathbb {1} \left\{\mathcal {E} (\mathbf {w}, \hat {t}), 0 \leq \mathbf {x} _ {2} \leq | - t (1 - \cos \theta) / \sin \theta + \left(\epsilon^ {\prime 2} / 8\right) \cot \theta | \right\}, \\ \end{array}
$$
where the first inequality results from the condition that  $\mathbf{x}_1 \geq -\hat{t}$ . Hence, the expectation  $\mathbf{E}_{\mathbf{x} \sim \mathcal{D}_{\mathbf{x}}}[\mathbf{x}_2 \mathbb{1}\{\mathcal{E}_2\}]$  can be upper-bounded by
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} _ {2} \} \right] \leq \underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \{\mathcal {E} (\mathbf {w}, \hat {t}), 0 \leq \mathbf {x} _ {2} \leq | - t \tan (\theta / 2) + (\epsilon^ {\prime 2} / 8) \cot \theta | \} \right] \\ = \Pr [ \mathcal {E} (\mathbf {w}, \hat {t}) ] \int_ {0} ^ {\left| - t \tan \left(\theta / 2\right) + \left(\epsilon^ {\prime 2} / 8\right) \cot \theta \right|} \frac {x}{\sqrt {2 \pi}} \exp \left(- x ^ {2} / 2\right) d x \\ \leq \Pr [ \mathcal {E} (\mathbf {w}, \hat {t}) ] \int_ {0} ^ {\left| - t \tan (\theta / 2) + \left(\epsilon^ {\prime 2} / 8\right) \cot \theta \right|} \frac {x}{\sqrt {2 \pi}} d x \\ \leq \frac {\mathbf {P r} [ \mathcal {E} (\mathbf {w} , \hat {t}) ]}{2 \sqrt {2 \pi}} (t \tan (\theta / 2) - (\epsilon^ {\prime 2} / 8) \cot \theta) ^ {2}. \\ \end{array}
$$
We again use the fact that  $\tan (\theta /2)\leq \theta$  and  $\cot \theta \leq 1 / \theta$ , then recall that  $\epsilon^{\prime}\exp (t^{2} / 2)\leq \theta \leq 1 / (5t)$ $\epsilon^\prime /\theta \leq 1$ , thus, we get
$$
\underset {\mathbf {x} \sim \mathcal {D} _ {\mathbf {x}}} {\mathbf {E}} \left[ \mathbf {x} _ {2} \mathbb {1} \left\{\mathcal {E} _ {2} \right\} \right] \leq \frac {\Pr \left[ \mathcal {E} (\mathbf {w} , \hat {t}) \right]}{\sqrt {2 \pi}} \left(\left(t \theta\right) ^ {2} + \left(\epsilon^ {\prime 2} / (8 \theta)\right) ^ {2}\right) \leq \frac {\Pr \left[ \mathcal {E} (\mathbf {w} , \hat {t}) \right]}{\sqrt {2 \pi}} \left(1 / 2 5 + \epsilon^ {\prime 2} / 6 4\right) \leq \frac {\Pr \left[ \mathcal {E} (\mathbf {w} , t ^ {\prime}) \right]}{1 2 \sqrt {2 \pi}}.
$$