SlowGuess/NeurIPS_2023_OCR huggingface dataset:
University of Wisconsin, Madison ilias@cs.wisc.edu
University of California, San Diego dakane@ucsd.edu
University of Wisconsin, Madison jelena@cs.wisc.edu
Puqian Wang
University of Wisconsin, Madison  
pwang333@wisc.edu
We study the problem of learning general (i.e., not necessarily homogeneous) halfspaces with Random Classification Noise under the Gaussian distribution. We establish nearly-matching algorithmic and Statistical Query (SQ) lower bound results revealing a surprising information-computation gap for this basic problem. Specifically, the sample complexity of this learning problem is  $\widetilde{\Theta}(d/\epsilon)$ , where  $d$  is the dimension and  $\epsilon$  is the excess error. Our positive result is a computationally efficient learning algorithm with sample complexity  $\tilde{O}(d/\epsilon + d / (\max\{p, \epsilon\})^2)$ , where  $p$  quantifies the bias of the target halfspace. On the lower bound side, we show that any efficient SQ algorithm (or low-degree test) for the problem requires sample complexity at least  $\Omega(d^{1/2} / (\max\{p, \epsilon\})^2)$ . Our lower bound suggests that this quadratic dependence on  $1/\epsilon$  is inherent for efficient algorithms.
A halfspace or Linear Threshold Function (LTF) is any Boolean function  $h: \mathbb{R}^d \to \{\pm 1\}$  of the form  $h(\mathbf{x}) = \operatorname{sign}(\mathbf{w} \cdot \mathbf{x} + t)$ , where  $\mathbf{w} \in \mathbb{R}^d$  is the weight vector and  $t \in \mathbb{R}$  is the threshold. The function  $\operatorname{sign}: \mathbb{R} \to \{\pm 1\}$  is defined as  $\operatorname{sign}(u) = 1$  if  $u \geq 0$  and  $\operatorname{sign}(u) = -1$  otherwise. The problem of learning halfspaces is a classical problem in machine learning, going back to the Perceptron algorithm [Ros58] and has had a big impact in both the theory and the practice of the field [Vap98, FS97]. Here we study the problem of PAC learning halfspaces in the distribution-specific setting in the presence of Random Classification Noise (RCN) [AL88]. Specifically, we focus on the basic case in which the marginal distribution on examples is the standard Gaussian — one of the simplest and most extensively studied distributional assumptions.
In the realizable PAC model [Val84b] (i.e., when the labels are consistent with a concept in the class), the class of halfspaces on  $\mathbb{R}^d$  is efficiently learnable to 0-1 error  $\epsilon$  using  $\widetilde{O}(d/\epsilon)$  samples via linear programming (even in the distribution-free setting). This sample complexity upper bound is information-theoretically optimal, even if we know a priori that the distribution on examples is well-behaved (e.g., Gaussian or uniform). That is, in the realizable setting, there is an efficient algorithm for halfspaces achieving the optimal sample complexity (within logarithmic factors).
Learning Gaussian Halfspaces with RCN. The RCN model [AL88] is the most basic model of random noise. In this model, the label of each example is independently flipped with probability exactly  $\eta$ , for some noise parameter  $0 < \eta < 1/2$ . One of the classical results on PAC learning with RCN [Kea98] states that any Statistical Query (SQ) algorithm can be transformed into an RCN noise-tolerant PAC learner with at most a polynomial complexity blowup. Halfspaces are known to be efficiently PAC learnable in the presence of RCN, even in the distribution-free setting [BFKV97, Coh97, DKT21, DTK22]. Alas, all these efficient algorithms require sample complexity that is suboptimal within polynomial factors in  $d$  and  $1/\epsilon$ .
The sample complexity of PAC learning Gaussian halfspaces with RCN is  $\widetilde{\Theta} (d / ((1 - 2\eta)\epsilon))$ . This bound can be derived, e.g., from [MN06], and the lower bound essentially matches the realizable case, up to a necessary scaling of  $(1 - 2\eta)$ . Given the fundamental nature of this learning problem, it is natural to ask whether a computationally efficient algorithm with (near-) optimal sample complexity (i.e., within logarithmic factors of the optimal) exists. That is, we are interested in a fine-grained sample size versus computational complexity analysis of the problem. This leads us to the following question:
Is there a sample near-optimal and polynomial-time algorithm for learning Gaussian halfspaces with RCN?
In this paper, we explore the above question and provide two main contributions — essentially resolving the question within logarithmic factors. On the positive side, we give an efficient algorithm with sample complexity  $\tilde{O}_{\eta}(d / \epsilon + d / (\max \{p, \epsilon\})^2)$  for the problem. Here the parameter  $p \in [0, 1/2]$  (Definition 1.2) quantifies the bias of the target function; a "balanced" function has  $p = 1/2$  and a constant function has  $p = 0$ . The worst-case upper bound arises when  $p = \Theta(\epsilon)$ , in which case our algorithm has sample complexity of  $\tilde{O}_{\eta}(d / \epsilon^2)$ . Perhaps surprisingly, we provide formal evidence that the quadratic dependence on the quantity  $1 / \max \{p, \epsilon\}$  in the sample complexity cannot be improved for computationally efficient algorithms. Our lower bounds apply for two restricted yet powerful models of computation, namely Statistical Query algorithms and low-degree polynomial tests. Our lower bounds suggest an inherent statistical-computational tradeoff for this problem.
We study the complexity of learning halfspaces with RCN under the Gaussian distribution. Let  $\mathcal{C} = \{f:\mathbb{R}^d\to \{\pm 1\} \mid f(\mathbf{x}) = \mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + t)\}$  be the class of general (i.e., not necessarily homogeneous) halfspaces in  $\mathbb{R}^d$ . The following definition summarizes our learning problem.
Definition 1.1 (Learning Gaussian Halfspaces with RCN). Let  $\mathcal{D}$  be a distribution on  $(\mathbf{x},y)\in \mathbb{R}^d\times \{\pm 1\}$  whose  $\mathbf{x}$ -marginal  $\mathcal{D}_{\mathbf{x}}$  is the standard Gaussian. Moreover, there exists  $\eta \in (0,1 / 2)$  and a target  $f\in \mathcal{C}$  such that the label  $y$  of example  $\mathbf{x}$  satisfies  $y = f(\mathbf{x})$  with probability  $1 - \eta$  and  $y = -f(\mathbf{x})$  otherwise. Given  $\epsilon >0$  and sample access to  $\mathcal{D}$ , the goal is to output a hypothesis  $h$  that with high probability satisfies  $\mathrm{err}_{0 - 1}^{\mathcal{D}}(h)\coloneqq \mathbf{Pr}_{\mathcal{D}}[h(\mathbf{x})\neq y]\leq \eta +\epsilon$ .
Our main contribution is a sample near-optimal efficient algorithm for this problem coupled with a matching statistical-computational tradeoff for SQ algorithms and low-degree polynomial tests. It turns out that the sample complexity of our algorithm depends on the bias of the target halfspace, defined below.
Definition 1.2 ( $p$ -biased function). For  $p \in [0,1/2]$ , we say that a Boolean function  $f: \mathbb{R}^d \to \{\pm 1\}$  is  $p$ -biased with respect to the distribution  $\mathcal{D}_{\mathbf{x}}$ , if  $\min \left\{\mathbf{Pr}_{\mathbf{x} \sim \mathcal{D}_{\mathbf{x}}} [f(\mathbf{x}) = 1], \mathbf{Pr}_{\mathbf{x} \sim \mathcal{D}_{\mathbf{x}}} [f(\mathbf{x}) = -1]\right\} = p$ .
For example, a homogeneous halfspace  $f(\mathbf{x}) = \mathrm{sign}(\mathbf{w}\cdot \mathbf{x})$  under the standard Gaussian distribution  $\mathcal{D}_{\mathbf{x}} = \mathcal{N}(\mathbf{0},\mathbf{I})$  satisfies  $\mathbf{E}_{\mathbf{x}\sim \mathcal{D}_{\mathbf{x}}}[f(\mathbf{x})] = 0$ , and therefore has bias  $p = 1 / 2$ . For a general halfspace  $f(\mathbf{x}) = \mathrm{sign}(\mathbf{w}\cdot \mathbf{x} + t)$  with  $\| \mathbf{w}\| _2 = 1$ , it is not difficult to see that its bias under the standard Gaussian is approximately  $p\sim (1 / t)\exp (-t^2 /2)$  (see Fact B.1).
We can now state our algorithmic contribution.
Theorem 1.3. (Main Algorithmic Result) There exists an algorithm that, given  $\epsilon, \delta \in (0,1/2)$  and  $N$  samples from a distribution  $\mathcal{D}$  satisfying Definition 1.1, runs in time  $O(dN/\epsilon^2)$  and returns a
Throughout this introduction, it will be convenient to view  $\eta$  as a constant bounded away from  $1 / 2$ .
hypothesis  $h \in \mathcal{C}$  such that with probability at least  $1 - \delta$ , it holds  $\mathrm{err}_{0-1}^{\mathcal{D}}(h) \leq \eta + \epsilon$ . The sample complexity of the algorithm is  $N = \widetilde{O}\left(\frac{d}{(1 - 2\eta)\epsilon} + \frac{d}{\max(p(1 - 2\eta), \epsilon)^2}\right) \log(1 / \delta)$ .
Some comments are in order. We note that the first term in the sample complexity matches the information-theoretic lower bound (within a logarithmic factor), even for homogeneous halfspaces  $(p = 1/2)$ ; see, e.g., [MN06, HY15]. The second term — scaling quadratically with  $1/\max\{(1 - 2\eta)p, \epsilon\}$  — is not information-theoretically necessary and dominates the sample complexity when  $p = O_{\eta}(\sqrt{\epsilon})$ . In the worst-case, i.e., when  $p = O_{\eta}(\epsilon)$ , our algorithm has sample complexity  $\widetilde{\Theta}_{\eta}(d/\epsilon^2)$ . Perhaps surprisingly, we show in Theorem 1.5 that this quadratic dependence is required for any computationally efficient SQ algorithm; and, via  $[\mathrm{BBH}^{+}20]$ , for any low-degree polynomial test.
Basics on SQ Model. SQ algorithms are a broad class of algorithms that, instead of having direct access to samples, are allowed to query expectations of bounded functions of the distribution.
Definition 1.4 (SQ algorithms). Let  $D$  be a distribution on  $\mathbb{R}^d$ . A statistical query is a bounded function  $q: \mathbb{R}^d \to [-1,1]$ . For  $u > 0$ , the VSTAT  $(u)$  oracle responds to the query  $q$  with a value  $v$  such that  $|v - \mathbf{E}_{\mathbf{x} \sim D}[q(\mathbf{x})]| \leq \tau$ , where  $\tau = \max(1/u, \sqrt{\mathrm{Var}_{\mathbf{x} \sim D}[q(\mathbf{x})]/u})$ . We call  $\tau$  the tolerance of the statistical query. A Statistical Query algorithm is an algorithm whose objective is to learn some information about an unknown distribution  $D$  by making adaptive calls to the corresponding oracle.
The SQ model was introduced in [Kea98] as a natural restriction of the PAC model [Val84a]. Subsequently, the model has been extensively studied in a range of contexts, see, e.g., [Fel16]. The class of SQ algorithms is broad and captures a range of known supervised learning algorithms. More broadly, several known algorithmic techniques in machine learning are known to be implementable using SQs (see, e.g.,  $\mathrm{[FGR^{+}17}$ , FGV17]).
We can now state our SQ lower bound result.
Theorem 1.5 (SQ Lower Bound). Fix any constant  $c \in (0,1/2)$  and let  $d$  be sufficiently large. For any  $p \geq 2^{-O(d^c)}$ , any SQ algorithm that learns the class of  $p$ -biased halfspaces on  $\mathbb{R}^d$  with Gaussian marginals in the presence of RCN with  $\eta = 1/3$  to error less than  $\eta + p/3$  either requires queries of accuracy better than  $\widetilde{O}(pd^{c/2 - 1/4})$ , i.e., queries to VSTAT  $(\widetilde{O}(d^{1/2 - c}/p^2))$ , or needs to make at least  $2^{\Omega(d^c)}$  statistical queries.
Informally speaking, Theorem 1.5 shows that no SQ algorithm can learn  $p$ -biased halfspaces in the presence of RCN (with  $\eta = 1/3$ ) to accuracy  $\eta + O(\epsilon)$  (considering  $p > \epsilon/2$ ) with a sub-exponential in  $d^{\Omega(1)}$  many queries, unless using queries of small tolerance — that would require at least  $\Omega(\sqrt{d}/p^2)$  samples to simulate. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms. When  $p = 2\epsilon$ , the computational sample complexity lower bound we obtain is  $\Omega(\sqrt{d}/\epsilon^2)$ . That is, for sufficiently small  $\epsilon$ , the computational sample complexity of the problem (in the SQ model) is polynomially higher than its information-theoretic sample complexity.
Via  $\mathrm{[BBH^{+}20]}$ , we obtain a qualitatively similar lower bound in the low-degree polynomial testing model; see Appendix D.
Upper Bound. At a high level, our main algorithm consists of three main subroutines. We start with a simple Initialization (warm-start) subroutine which ensures that we can choose a weight vector  $\mathbf{w}_0$  with sufficiently small angle to the target vector  $\mathbf{w}^*$ . This subroutine essentially amounts to estimating the degree-one Chow parameters of the target function and incurs sample complexity  $\widetilde{O}(d / (\max(p(1 - 2\eta), \epsilon)^2)$ . We emphasize that our procedure does not require knowing the bias  $p$  of the target halfspace; instead, it estimates this parameter to a constant factor.
Our next (and main) subroutine is an optimization procedure that is run for  $\widetilde{O}(1/\epsilon^2)$  different guesses of the threshold  $t$ . At a high level, our optimization subroutine can be seen as a variant of Riemannian (sub)gradient descent on the unit sphere, applied to the empirical LeakyReLU loss — defined as  $\mathrm{LeakyReLU}_{\lambda}(u) = (1 - \lambda)u\mathbb{1}\{u \geq 0\} + \lambda u\mathbb{1}\{u < 0\}$  — with parameter  $\lambda$  set to  $\eta$ ,  $u = \mathbf{w} \cdot \mathbf{x}$ ,
and with samples restricted to a band, namely  $a < |\mathbf{w} \cdot \mathbf{x}| < b$  — with  $a$  and  $b$  chosen as functions of the guess for the threshold  $t$ . The band restriction is key in avoiding  $\Omega(d / \epsilon^2)$  dependence in the sample complexity; instead, we only require order- $(d / \epsilon)$  samples to be drawn for the empirical LeakyReLU loss subgradient estimate. Using the band, the objective is restricted to a region where the current hypothesis incorrectly classifies a constant fraction of the mass from which we can perform "denoising" with constantly many samples.
For a sufficiently accurate estimate  $\hat{t}$  of  $t$  (which is satisfied by at least one of the guesses for which our optimization procedure is run), we argue that there is a sufficiently negative correlation between the empirical subgradient and the target weight vector  $\mathbf{w}^*$ . This result, combined with our initialization, enables us to inductively argue that the distance between the weight vector constructed by the optimization procedure and the target vector  $\mathbf{w}^*$  contracts and becomes smaller than  $\epsilon$  within order- $\log(1/\epsilon)$  iterations. This result is quite surprising, since the LeakyReLU loss is nonsmooth (it is, in fact, piecewise linear) and we do not explicitly bound its growth outside the set of its minima (i.e., we do not prove a local error bound, which would typically be used to prove linear convergence). Thus, the result we establish is impossible to obtain using black-box results for nonsmooth optimization. Additionally, we never explicitly use the LeakyReLU loss function or prove that it is minimized by  $\mathbf{w}^*$ ; instead, we directly prove that the vectors  $\mathbf{w}$  constructed by our procedure converge to the target vector  $\mathbf{w}^*$ . At a technical level, our result is enabled by a novel inductive argument, which we believe may be of independent interest (see Lemma 2.8 for more details).
Since each run of our optimization subroutine returns a different hypothesis, at least one of which is accurate (the one using the "correct" guess of the threshold  $t$ ), we need an efficient way to select a hypothesis with the desired error guarantee. This is achieved via our third subroutine — a simple hypothesis testing procedure, which draws a fresh sample and selects a hypothesis with the lowest test error. By standard results [MN06], such a hypothesis satisfies our target error guarantee.
SQ Lower Bound. To prove our SQ lower bound, it suffices to establish the existence of a large set of distributions whose pairwise correlations are small  $\left[\mathrm{FGR}^{+}17\right]$ . Inspired by the methodology of [DKS17], we achieve this by selecting our distributions on labeled examples  $(\mathbf{x},y)$  to be random rotations of a single one-dimensional distribution that nearly matches low-order Gaussian moments, and embedding this in a hidden random direction. Our hard distributions are as follows: We define the halfspaces  $f_{\mathbf{v}}(\mathbf{x}) = \mathrm{sign}(\mathbf{v}\cdot \mathbf{x} - t)$ , where  $\mathbf{v}$  is a randomly chosen unit vector and the threshold  $t$  is chosen such that  $\mathbf{Pr}_{\mathbf{x}\sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x}) = 1] = p$ . We then let  $y = f_{\mathbf{v}}(\mathbf{x})$  with probability  $2 / 3$ , and  $-f_{\mathbf{v}}(\mathbf{x})$  otherwise. By picking a packing of nearly orthogonal vectors  $\mathbf{v}$  on the unit sphere (i.e., set of vectors with pairwise small inner product), we show that each pair of these  $f_{\mathbf{v}}$ ’s corresponding to distinct vectors in the packing have very small pairwise correlations (with respect to the distribution where  $\mathbf{x}$  is a standard Gaussian and  $y$  is independent of  $\mathbf{x}$ ). While the results of [DKS17] cannot be directly applied to give our desired correlation bounds, the Hermite analytic ideas behind them are useful in this context. In particular, the correlation between two such distributions can be computed in terms of their angle and the Hermite spectrum. A careful analysis (Lemma 3.3) gives an inner product that is  $\widetilde{O} (\cos (\theta)p)$ , where  $\theta$  is the angle between the corresponding vectors. Combined with our packing bound, this is sufficient to obtain our final SQ lower bound result.
1.3 Related and Prior Work
A long line of work in theoretical machine learning has focused on developing computationally efficient algorithms for learning halfspaces under natural distributional assumptions in the presence of RCN and related semi-random noise models; see, e.g., [ABHU15, ABHZ16, YZ17, ZLC17, DKTZ20a, DKTZ20b, DKK+20, DKK+21, DKK+22]. Interestingly, the majority of these works focused on the special case of homogeneous halfspaces. We next describe in detail the most relevant prior work.
Prior work [YZ17, ZSA20, ZL21] gave sample near-optimal and computationally efficient learners for homogeneous halfspaces with RCN (and, more generally, bounded noise). Specifically, these works developed algorithms using near-optimal sample complexity of  $\widetilde{O}_{\eta}(d / \epsilon)$ . However, their algorithms and analyses are customized to the homogeneous case, and it is not clear how to extend them for general halfspaces. In fact, since all of these algorithms are easily implementable in the SQ model, our SQ lower bound (Theorem 1.5) implies that these prior algorithms cannot be adapted to handle the general case without an increase in sample complexity. Finally, [DKTZ22] gave an algorithm with sample complexity  $\widetilde{O}(d / \epsilon^2)$  to learn general Gaussian halfspaces with adversarial
label noise to error  $O(\mathrm{OPT}) + \epsilon$ , where OPT is the optimal misclassification error. Unfortunately, this algorithm does not suffice for our RCN setting (where  $\mathrm{OPT} = \eta$ ), since its error guarantee is significantly weaker than ours.
Very recent work  $\mathrm{[DDK^{+}23]}$  gave an SQ lower bound for  $\gamma$ -margin halfspaces with RCN, which has some similarities to ours. Specifically,  $\mathrm{[DDK^{+}23]}$  showed that any efficient SQ algorithm for that problem requires sample complexity  $\Omega(1 / (\gamma^{1/2}\epsilon^2))$ . Intuitively, the margin assumption allows for a much more general family of distributions compared to our Gaussian assumption here. In particular, the SQ construction of that work does not have any implications in our setting. Even though the Gaussian distribution does not have a margin, it is easy to see that it satisfies an approximate margin property for  $\gamma \sim 1/\sqrt{d}$ . In fact, using an adaptation of our construction, we believe we can quantitatively strengthen the lower bound of  $\mathrm{[DDK^{+}23]}$  to  $\Omega(1 / (\gamma \epsilon^2))$ . For more details, see Appendix A.
1.4 Preliminaries
For  $n \in \mathbb{Z}_+$ , we define  $[n] := \{1, \ldots, n\}$ . We use lowercase bold characters for vectors and uppercase bold characters for matrices. For  $\mathbf{x} \in \mathbb{R}^d$  and  $i \in [d]$ ,  $\mathbf{x}_i$  denotes the  $i$ -th coordinate of  $\mathbf{x}$ , and  $\| \mathbf{x} \|_2 := (\sum_{i=1}^d \mathbf{x}_i^2)^{1/2}$  denotes the  $\ell_2$ -norm of  $\mathbf{x}$ . We use  $\mathbf{x} \cdot \mathbf{y}$  for the inner product of  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$  and  $\theta(\mathbf{x}, \mathbf{y})$  for the angle between  $\mathbf{x}$  and  $\mathbf{y}$ . We slightly abuse notation and denote by  $\mathbf{e}_i$  the  $i^{\text{th}}$  standard basis vector in  $\mathbb{R}^d$ . We further use  $\mathbb{1}_A$  to denote the characteristic function of the set  $A$ , i.e.,  $\mathbb{1}_A(\mathbf{x}) = 1$  if  $\mathbf{x} \in A$  and  $\mathbb{1}_A(\mathbf{x}) = 0$  if  $\mathbf{x} \notin A$ . We use the standard  $O(\cdot), \Theta(\cdot), \Omega(\cdot)$  asymptotic notation. We also use  $\widetilde{O}(\cdot)$  to omit poly-logarithmic factors in the argument. We use  $\mathbf{E}_{x \sim \mathcal{D}}[x]$  for the expectation of the random variable  $x$  according to the distribution  $\mathcal{D}$  and  $\mathbf{Pr}[\mathcal{E}]$  for the probability of event  $\mathcal{E}$ . For simplicity of notation, we omit the distribution when it is clear from the context. For  $(\mathbf{x}, y)$  distributed according to  $\mathcal{D}$ , we denote by  $\mathcal{D}_{\mathbf{x}}$  the distribution of  $\mathbf{x}$ . As is standard, we use  $\mathcal{N}$  to denote the standard normal distribution in  $d$  dimensions; i.e., with its mean being the zero vector and its covariance being the identity matrix.
2 Efficiently Learning Gaussian Halfspaces
In this section, we prove Theorem 1.3 by analyzing Algorithm 1. As discussed in the introduction and shown in Algorithm 1, there are three main procedures in our algorithm. The guarantees of our Initialization (warm start) procedure, which ensures sufficient correlation between the initial weight vector  $\mathbf{w}_0$  and the target vector  $\mathbf{w}^*$ , are stated in Section 2.1, while the proofs and pseudocode are in Appendix B.1. Our main results for this section, including the Optimization procedure and associated analysis, are in Section 2.2. The Testing procedure is standard and deferred to Appendix B.2, together with most of the technical details from this section.
Throughout this section, we assume that the parameter  $\eta$  (RCN parameter) is known. As will become clear from our analysis, a constant factor approximation to the value of  $1 - 2\eta$  is sufficient to obtain our results. For completeness, we show how to obtain such an approximation in Appendix B.3. For simplicity, we present the results for  $t \geq 0$  and  $t \leq \sqrt{2\log((1 - 2\eta) / \epsilon)}$ . This is without loss of
Algorithm 1 Main Algorithm
1: Input:  $\delta, \eta, \epsilon$ , sample access to distribution  $\mathcal{D}$
2:  $[\mathbf{w}_0,\hat{p} ] = \mathrm{Initialization}(\delta ,\eta ,\epsilon);\epsilon ' = \epsilon /(1 - 2\eta)$
3:  $t_0 = \sqrt{2\log(1 / \hat{p})}$ ,  $M = 8\left[\frac{\sqrt{2(\log(4 / \hat{p}))} - \sqrt{2\log(1 / \hat{p})}}{(\epsilon')^2}\right] + 1$
4: Draw  $N_{2} = O\left(\frac{d\log(1 / \delta)\log(1 / \epsilon^{\prime})}{(1 - 2\eta)^{2}\epsilon^{\prime}}\right)$  samples  $\{(x^{(i)},y^{(i)})\}_{i = 1}^{N_2}$  from  $\mathcal{D}$
5: for  $m = 1:M$  do
6:  $t_m = t_0 + (m - 1)\frac{(\epsilon')^2}{8}, \gamma_m = \frac{\epsilon'}{2}\exp(t_m^2 / 2)$
7:  $\widehat{\mathbf{w}}_m = \mathrm{Optimization}(\mathbf{w}_0, t_m, \gamma_m, \eta, \{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^{N_2})$
8: end for
9:  $[\widehat{\mathbf{w}}_{\mathrm{out}},t_{\mathrm{out}}] = \mathrm{Testing}((\widehat{\mathbf{w}}_1,t_1),(\widehat{\mathbf{w}}_2,t_2),\ldots ,(\widehat{\mathbf{w}}_M,t_M))$
10: return  $\widehat{\mathbf{w}}_{\mathrm{out}}$ $t_\mathrm{out}$
Algorithm 2 Optimization