SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aravind Gollakota  
Apple
Parikshit Gopalan  
Apple
Adam R. Klivans UT Austin
Konstantinos Stavropoulos UT Austin
We give the first result for agnostically learning Single-Index Models (SIMs) with arbitrary monotone and Lipschitz activations. All prior work either held only in the realizable setting or required the activation to be known. Moreover, we only require the marginal to have bounded second moments, whereas all prior work required stronger distributional assumptions (such as anticoncentration or boundedness). Our algorithm is based on recent work by Gopalan et al. [2023] on omniprediction using predictors satisfying calibrated multiaccuracy. Our analysis is simple and relies on the relationship between Bregman divergences (or matching losses) and  $\ell_p$  distances. We also provide new guarantees for standard algorithms like GLMtron and logistic regression in the agnostic setting.
Generalized Linear Models (GLMs) and Single-Index Models (SIMs) constitute fundamental frameworks in statistics and supervised learning [McCullagh, 1984, Agresti, 2015], capturing and generalizing basic models such as linear and logistic regression. In the GLM framework, labeled examples  $(\mathbf{x}, y)$  are assumed to satisfy  $u(\mathbb{E}[y|\mathbf{x}]) = \mathbf{w} \cdot \mathbf{x}$  (or  $\mathbb{E}[y|\mathbf{x}] = u^{-1}(\mathbf{w} \cdot \mathbf{x})$ ), where  $u$  is a known monotone function (called the link function) and  $\mathbf{w}$  is an unknown vector. Single-Index Models (SIMs) are a generalization in which the monotone link function  $u$  is also unknown.
In the realizable setting where the labels are indeed generated according to a GLM with a known Lipschitz link function, the GLMtron algorithm of Kakade et al. [2011] is a simple and efficient learning algorithm. When the ground truth is only assumed to be a SIM (i.e. the link function is unknown), it can be learned efficiently by the Isotron algorithm [Kalai and Sastry, 2009, Kakade et al., 2011].
In this work, we consider the significantly more challenging agnostic setting, where the labels are arbitrary and not necessarily realizable by any SIM. Importantly, we do not fix a link function in advance; our goal is to output a predictor that has squared error comparable to that of the optimal SIM with an arbitrary monotone and Lipschitz link function. We can equivalently view this as a natural squared-error regression problem in which the final optimality guarantee must hold with respect to all SIMs with bounded weights and monotone, Lipschitz link functions.<sup>1</sup>
Formally, consider a distribution  $D$  over  $\mathbb{R}^d\times [0,1]$  and denote the squared error of a function  $h:\mathbb{R}^d\to \mathbb{R}$  by  $\mathrm{err}_2(h) = \mathbb{E}_{(\mathbf{x},y)\sim D}[(y - h(\mathbf{x}))^2 ]$ . Let  $\mathrm{opt}(\mathrm{SIM})$  denote optimal value of  $\mathrm{err}_2(h)$  over all SIMs  $h$  with bounded weights and arbitrary 1-Lipschitz monotone activations (we call the inverse  $u^{-1}$  of a link function  $u$  the activation function). Given access to samples from  $D$ , the goal of an agnostic learning algorithm is to compute a predictor  $p:\mathbb{R}^d\rightarrow [0,1]$  with error  $\mathrm{err}_2(p)$  that, with high probability over the samples, is comparable to the error of the optimal SIM:
$$
\operatorname {e r r} _ {2} (p) \leq \operatorname {o p t} (\operatorname {S I M}) + \varepsilon .
$$
<sup>1</sup>In this context, in recent times it has also been common to refer to the class of GLMs (resp. SIMs) as the class of single neurons with known (resp. unknown) activation functions.
Our main result is the first efficient learning algorithm with a guarantee of this form.
Theorem 1.1 (Informal, see Theorem 3.1). Let  $\mathrm{SIM}_B$  denote the class of SIMs of the form  $\mathbf{x} \mapsto u^{-1}(\mathbf{w} \cdot \mathbf{x})$  for some 1-Lipschitz function  $u^{-1}$  and  $\|\mathbf{w}\|_2 \leq B$ . Let  $D$  be any distribution over  $\mathbb{R}^d \times [0,1]$  whose marginal on  $\mathbb{R}^d$  has bounded second moments. There is an efficient algorithm (Algorithm 1) that agnostically learns  $\mathrm{SIM}_B$  over  $D$  up to error
$$
\operatorname {e r r} _ {2} (p) \leq O \left(B \sqrt {\operatorname {o p t} \left(\operatorname {S I M} _ {B}\right)}\right) + \varepsilon .
$$
This result provides a guarantee comparable to that of the Isotron algorithm [Kalai and Sastry, 2009, Kakade et al., 2011] (which also tackles the SIM setting, where the link function is unknown) but for the challenging agnostic setting rather than the realizable setting (where  $\mathrm{opt}(\mathrm{SIM}_B,D) = 0$ ). Moreover, Isotron's guarantees require the distribution to be supported on the unit ball, whereas we only require a mild second moment condition.
In our view, this result helps establish Algorithm 1 (and indeed any algorithm to compute omnipredictors, as we introduce and discuss shortly) as an efficient and powerful off-the-shelf supervised learning algorithm, akin to random forests or gradient-boosted trees.
A natural question is whether our guarantees are near-optimal, e.g., whether we can obtain a guarantee of the form  $\mathrm{err}_2(p) \leq \mathrm{opt}(\mathrm{SIM}) + \varepsilon$ . However, there is strong evidence that such results cannot be obtained using efficient algorithms [Goel et al., 2019, Diakonikolas et al., 2020a, Goel et al., 2020, Diakonikolas et al., 2021]. We partially justify the form of our guarantee by showing in Section 5 (adapting a result due to Diakonikolas et al. [2022a]) that one cannot avoid a dependence on the norm bound  $B$ . Further closing the gap between upper and lower bounds is an important direction for future work.
Overview of techniques. We now describe the main ingredients and techniques that go into proving Theorem 1.1. Our starting point is the connection between GLMs and so-called matching losses [Auer et al., 1995]. This connection arises from the fact that fitting a GLM with a known link function is equivalent to minimizing (over the class of all linear functions) a certain convex loss known as the matching loss corresponding to that link function (see Definition 1.8). (We refer the reader to [Gopalan et al., 2023, Sec 5.1] for a more detailed discussion.)
Importantly for us, minimizing the matching loss corresponding to a link function  $u$  yields a meaningful guarantee even in the agnostic setting, where the Bayes optimal predictor (i.e.  $\mathbf{x} \mapsto \mathbb{E}[y|\mathbf{x}]$ ) is not necessarily a GLM at all. Specifically, it turns out to be equivalent (via Fenchel-Legendre duality) to finding the closest predictor to the Bayes optimal predictor in the metric of the Bregman divergence associated with the link function  $u$  (see e.g. [Gopalan et al., 2023, Lemma 5.4]).
This is a powerful observation, but it still assumes that we have a fixed and known link function  $u$  in mind. In the agnostic setting, it is arguably unrealistic to assume that we know the best link function for the distribution at hand. Remarkably, recent work by Gopalan et al. [2022, 2023] has shown that there exist efficient learning algorithms that simultaneously minimize all matching losses corresponding to arbitrary monotone and Lipschitz link functions. Their solution concept is called an omnipredictor, i.e., a single predictor that is able to compete with the best-fitting classifier in a class  $C$  as measured by a large class of losses (as opposed to just a single pre-specified loss, as is standard in machine learning). They obtain such predictors through calibrated multiaccuracy [Gopalan et al., 2023] or multicalibration [Gopalan et al., 2022].
From the point of view of ordinary supervised learning or regression, however, an optimality guarantee in terms of such matching losses or Bregman divergences is hard to interpret. A much more standard metric is simply that of squared error. The key final step for our results is to find a way of translating (a) optimality (ranging over all linear functions) in terms of all matching losses simultaneously into (b) optimality (ranging over all SIMs) in terms of squared error. We do so by proving simple analytic distortion inequalities relating matching losses to  $\ell_p$  losses, which we believe may be of independent interest.
On a technical level, to prove these distortion inequalities we first prove strong versions of such inequalities for matching losses arising from bi-Lipschitz link functions (see Lemma 2.2). We then obtain our results for general Lipschitz link functions by carefully approximating them using bi-Lipschitz link functions (see Lemma 3.3).
Further applications. As further applications of our approach, if we let  $\mathrm{opt}(\mathrm{GLM}_{u^{-1},B})$  denote the optimal value of  $\mathrm{err}_2(h)$  over all GLMs of the form  $\mathbf{x}\mapsto u^{-1}(\mathbf{w}\cdot \mathbf{x})$ , where  $\| \mathbf{w}\| _2\leq B$ , we obtain the following result about bi-Lipschitz link functions (including, for example, the Leaky ReLU).
Theorem 1.2 (Informal, see Theorem 2.1). Let  $u: \mathbb{R} \to \mathbb{R}$  be a bi-Lipschitz invertible link function. Then, any predictor  $p: \mathbb{R}^d \to \mathbb{R}$  that is an  $\varepsilon$ -approximate minimizer of the population matching loss that corresponds to  $u$ , with respect to a distribution  $D$  over  $\mathbb{R}^d \times [0,1]$  satisfies
$$
\operatorname {e r r} _ {2} (p) \leq O \left(\operatorname {o p t} \left(\mathrm {G L M} _ {u ^ {- 1}, B}\right)\right) + O (\varepsilon)
$$
This guarantee holds under milder distributional assumptions than are required by comparable prior work on agnostically learning GLMs or single neurons [Frei et al., 2020, Diakonikolas et al., 2022b]. Moreover, when we focus on distortion bounds between the logistic loss and the squared loss, we obtain a near-optimal guarantee of  $\widetilde{O} (\mathrm{opt}(\mathrm{GLM}_{u^{-1},B}))$  for logistic regression, when  $u$  is the logit link function (i.e., when  $\mathrm{GLM}_{u^{-1},B}$  is the class of sigmoid neurons).
Theorem 1.3 (Informal, see Theorem 4.1). Let  $u(t) = \ln \left(\frac{t}{1 - t}\right)$ . Then, any predictor  $p: \mathbb{R}^d \to \mathbb{R}$  that is an approximate  $\varepsilon$ -minimizer of the population logistic loss, with respect to a distribution  $D$  over  $\mathbb{R}^d \times [0,1]$  whose marginal on  $\mathbb{R}^d$  has subgaussian tails in every direction satisfies
$$
\operatorname {e r r} _ {2} (p) \leq \widetilde {O} \left(\operatorname {o p t} \left(\mathrm {G L M} _ {u ^ {- 1}, B}\right)\right) + O (\varepsilon)
$$
While our error guarantee for this problem is weaker than that of Diakonikolas et al. [2022b], we do not make the anti-concentration assumptions their results require.
1.1 Background and Relation to Prior Work
We note that matching losses have been studied in various previous works either implicitly [Kakade et al., 2011] or explicitly [Auer et al., 1995, Diakonikolas et al., 2020b, Gopalan et al., 2023] and capture various fundamental algorithms like logistic and linear regression [McCullagh, 1984, Agresti, 2015]. However, to the best of our knowledge, our generic and direct approach for transforming matching loss guarantees to squared error bounds, has not been explored previously. Furthermore, our results do not depend on the specific implementation of an algorithm, but only on the matching loss bounds achieved by its output. In this sense, we provide new agnostic error guarantees for various existing algorithms of the literature. For example, our results imply new guarantees for the GLMtron algorithm of Kakade et al. [2011] in the agnostic setting, since GLMtron can be viewed as performing gradient descent (with unit step size) on the matching loss corresponding to a specified link function.
Matching losses over linear functions are also linked to the Chow parameters [O'Donnell and Servedio, 2008] through their gradient with respect to  $\mathbf{w}$ , as observed by Diakonikolas et al. [2020b]. In fact, the norm of the matching loss gradient is also linked to multiaccuracy, a notion that originates to fairness literature [Hebert-Johnson et al., 2018, Kim et al., 2019]. A stationary point  $\mathbf{w}$  of a matching loss that corresponds to a GLM with link  $u$  is associated with a multiaccurate predictor  $p(\mathbf{x}) = u^{-1}(\mathbf{w} \cdot \mathbf{x})$ , i.e., a predictor such that for all  $i \in [d]$ ,  $\mathbb{E}[\mathbf{x}_i(y - p(\mathbf{x}))] = 0$ . The work of [Gopalan et al., 2022, 2023] on omnipredictors presents a single predictor that is better than any linear model  $\mathbf{w} \cdot \mathbf{x}$  for every matching loss. The results of Gopalan et al. [2022] show that a multicalibrated predictor (with respect to the features  $\mathbf{x}_i$ ) is an omnipredictor for all convex losses, whereas Gopalan et al. [2023] shows that the simpler condition of calibrated multiaccuracy suffices for matching losses that arise from GLMs. In view of the relationship between multiaccuracy and the gradient of the matching loss, our results show that, while multiaccuracy implies bounds on agnostically learning GLMs, the additional guarantee of calibration is sufficient for agnostically learning all SIMs.
The work of Shalev-Shwartz et al. [2011] showed strong agnostic learning guarantees in terms of the absolute error (rather than the squared error) of the form  $\mathrm{opt} + \varepsilon$  for a range of GLMs, but their work incurs an exponential dependence on the weight norm  $B$ . In comparison, for the absolute loss, we get a bound of the form  $B$  opt  $\log(1/\mathrm{opt})$  for logistic regression (see Theorem 4.3). In more recent years, the problem of agnostically learning GLMs has frequently also been phrased as the problem of agnostically learning single neurons (with a known activation). For the ReLU activation, work by Goel et al. [2017] showed an algorithm achieving error  $\mathrm{opt} + \varepsilon$  in time  $\mathrm{poly}(d)\exp(1/\varepsilon)$  over marginals on the unit sphere, and Diakonikolas et al. [2020b] showed an algorithm achieving error
$O(\mathrm{opt}) + \varepsilon$  in fully polynomial time over isotropic log-concave marginals. The work of Frei et al. [2020], Diakonikolas et al. [2022b] both show guarantees for learning general neurons (with known activations) using the natural approach of running SGD directly on the squared loss (or a regularized variant thereof). Frei et al. [2020] achieves error  $O(\mathrm{opt})$  for any given strictly increasing activation and  $O(\sqrt{\mathrm{opt}})$  for the ReLU activation, but they assume that the marginal distribution is bounded. In contrast, we only assume that the marginal distribution has bounded second moments and we do not consider that the activation is known. Diakonikolas et al. [2022b] proved an  $O(\mathrm{opt})$  guarantee for a wide range of activations (including the ReLU), in the setting where the activation is known and over a large class of structured marginals, which need, however, to satisfy strong concentration and anti-concentration properties.
In terms of lower bounds and hardness results for this problem, the work of [Goel et al., 2019, Diakonikolas et al., 2020a, Goel et al., 2020, Diakonikolas et al., 2021, 2022a] has established superpolynomial hardness even for the setting of agnostically learning single RLUs over Gaussian marginals.
Limitations and directions for future work. While we justify a certain dependence on the norm bound  $B$  in our main result on agnostically learning SIMs, we do not completely justify the exact form of Theorem 1.1. An important direction for future work is to tightly characterize the optimal bounds achievable for this problem, as well as to show matching algorithms.
1.2 Preliminaries
For the following,  $(\mathbf{x},y)$  is used to denote a labelled sample from a distribution  $D$  over  $\mathbb{R}^d\times \mathcal{V}$ , where  $\mathcal{V}$  denotes the interval [0, 1] unless it is specified to be the set  $\{0,1\}$ . We note that, although we provide results for the setting where the labels lie within [0, 1], we may get similar results for any bounded label space. We use  $\mathbb{P}_D$  (resp.  $\mathbb{E}_D$ ) to denote the probability (resp. expectation) over  $D$  and, similarly,  $\mathbb{P}_S$  (resp.  $\mathbb{E}_S$ ) to denote the corresponding empirical quantity over a set  $S$  of labelled examples. Throughout the paper, we will use the term differentiable function to mean a function that is differentiable except on finitely many points. Our main results will assume the following about the marginal distribution on  $\mathbb{R}^d$ .
Definition 1.4 (Bounded moments). For  $\lambda \geq 1$  and  $k\in \mathbb{N}$ , we say that a distribution  $D_{\mathbf{x}}$  over  $\mathbb{R}^d$  has  $\lambda$ -bounded  $2k$ -th moments if for any  $\mathbf{v}\in \mathbb{S}^{d - 1}$  we have  $\mathbb{E}_{\mathbf{x}\sim D_{\mathbf{x}}}[(\mathbf{v}\cdot \mathbf{x})^{2k}]\leq \lambda$ .
For a concept class  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}$ , we define  $\mathrm{opt}(\mathcal{C},D)$  to be the minimum squared error achievable by a concept  $c:\mathbb{R}^d\rightarrow \mathbb{R}$  in  $\mathcal{C}$  with respect to the distribution  $D$ . We now define our main learning task.
Definition 1.5 (Agnostic learning). Let  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}$  be a concept class, let  $\Psi :[0,1]\rightarrow [0,1]$  be an increasing function, let  $D$  be a distribution over  $\mathbb{R}^d\times [0,1]$  and  $\varepsilon >0$ . We say that an algorithm  $\mathcal{A}$  agnostically learns the class  $\mathcal{C}$  up to error  $\Psi (\mathrm{opt}(\mathcal{C},D)) + \varepsilon$  if algorithm  $\mathcal{A}$ , given a number of i.i.d. samples from  $D$ , outputs, with probability at least  $2 / 3$  over the samples and the randomness of  $\mathcal{A}$ , a hypothesis  $h:\mathbb{R}^d\to [0,1]$  with  $\mathbf{E}_D[(y - h(\mathbf{x}))^2 ]\leq \Psi (\mathrm{opt}(\mathcal{C},D)) + \varepsilon$ .
We will also provide results that are specific to the sigmoid activation and work under the assumption that the marginal distribution is sufficiently concentrated.
Definition 1.6 (Concentrated marginals). For  $\lambda >0$  and  $\gamma$ , we say that a distribution  $D_{\mathbf{x}}$  over  $\mathbb{R}^d$  is  $(\lambda ,\gamma)$ -concentrated if for any  $\mathbf{v}\in \mathbb{S}^{d - 1}$  and  $r\geq 0$  we have  $\mathbb{P}_{\mathbf{x}\sim D_{\mathbf{x}}}[\left|\mathbf{v}\cdot \mathbf{x}\right|\geq r]\leq \lambda \cdot \exp (-r^{\gamma})$ .
Definition 1.7 (Fenchel-Legendre pairs). We call a pair of functions  $(f, g)$  a Fenchel-Legendre pair if the following conditions hold.
1.  $g^{\prime}:\mathbb{R}\to \mathbb{R}$  is continuous, non-decreasing, differentiable and 1-Lipschitz with range  $\mathrm{ran}(g^{\prime})\supseteq (0,1)$  and  $g(t) = \int_0^t g'(\tau)d\tau$ , for any  $t\in \mathbb{R}$ .
2.  $f:\mathrm{ran}(g^{\prime})\to \mathbb{R}$  is the convex conjugate (Fenchel-Legendre transform) of  $g$  , i.e., we have  $f(r) = \sup_{t\in \mathbb{R}}r\cdot t - g(t)$  for any  $r\in \operatorname {ran}(g')$
For such pairs of functions (and their derivatives  $f^{\prime}, g^{\prime}$ ), the following are true for  $r \in \mathrm{ran}(g^{\prime})$  and  $t \in \mathrm{ran}(f^{\prime})$  (note that  $\mathrm{ran}(f^{\prime})$  is not necessarily  $\mathbb{R}$  when  $g^{\prime}$  is not invertible).
$$
g ^ {\prime} \left(f ^ {\prime} (r)\right) = r \text {a n d} f (r) = r f ^ {\prime} (r) - g \left(f ^ {\prime} (r)\right), \text {f o r} r \in \operatorname {r a n} \left(g ^ {\prime}\right) \tag {1.1}
$$
$$
f ^ {\prime} \left(g ^ {\prime} (t)\right) = t \text {a n d} g (t) = t g ^ {\prime} (t) - f \left(g ^ {\prime} (t)\right), \text {f o r} t \in \operatorname {r a n} \left(f ^ {\prime}\right) \tag {1.2}
$$
Note that  $g'$  will be used as an activation function for single neurons and  $f'$  corresponds to the unknown link function of a SIM (or the known link function of a GLM). We say that  $g'$  is bi-Lipschitz if for any  $t_1 < t_2 \in \mathbb{R}$  we have that  $(g'(t_2) - g'(t_1)) / (t_2 - t_1) \in [\alpha, \beta]$ . If  $g'$  is  $[\alpha, \beta]$  bi-Lipschitz, then  $f'$  is  $[\frac{1}{\beta}, \frac{1}{\alpha}]$  bi-Lipschitz. However, the converse implication is not necessarily true when  $g'$  is not strictly increasing.
Definition 1.8 (Matching Losses). For a non-decreasing and Lipschitz activation  $g^{\prime}:\mathbb{R}\to \mathbb{R}$ , the matching loss  $\ell_g:\mathcal{V}\times \mathbb{R}\rightarrow \mathbb{R}$  is defined pointwise as follows:
$$
\ell_ {g} (y, t) = \int_ {0} ^ {t} g ^ {\prime} (\tau) - y d \tau ,
$$
where  $g(t) = \int_0^t g'$ . The function  $\ell_g$  is convex and smooth with respect to its second argument. The corresponding population matching loss is
$$
\mathcal {L} _ {g} (c; D) = \underset {(\mathbf {x}, y) \sim D} {\mathbb {E}} \left[ \ell_ {g} (y, c (\mathbf {x})) \right] \tag {1.3}
$$
In Equation (1.3),  $c: \mathbb{R}^d \to \mathbb{R}$  is some concept and  $D$  is some distribution over  $\mathbb{R}^d \times [0,1]$ . In the specific case where  $c$  is a linear function, i.e.,  $c(\mathbf{x}) = \mathbf{w} \cdot \mathbf{x}$ , for some  $\mathbf{w} \in \mathbb{R}^d$ , then we may alternatively denote  $\mathcal{L}_g(c; D)$  with  $\mathcal{L}_g(\mathbf{w}; D)$ .
We also define the Bregman divergence associated with  $f$  to be  $\mathbb{D}_f(q,r) = f(q) - f(r) - (q - r)f'(r)$ , for any  $q, r \in \mathrm{ran}(g')$ . Note that  $\mathbb{D}_f(q,r) \geq 0$  with equality iff  $q = r$ .
Definition 1.9 (SIMs and GLMs as Concept Classes). For  $B > 0$ , we use  $\mathrm{SIM}_B$  to refer to the class of all SIMs of the form  $\mathbf{x} \mapsto g'(\mathbf{w} \cdot \mathbf{x})$  where  $\| \mathbf{w} \|_2 \leq B$  and  $g': \mathbb{R} \to \mathbb{R}$  is an arbitrary 1-Lipschitz monotone activation that is differentiable (except possibly at finitely many points). We define  $\mathrm{GLM}_{g',B}$  similarly except for the case where  $g'$  is fixed and known.
We also define the notion of calibrated multiaccuracy that we need to obtain omnipredictors in our context.
Definition 1.10 (Calibrated Multiaccuracy). A predictor  $p: \mathbb{R}^d \to [0,1]$  is called  $\varepsilon$ -multiaccurate if for all  $i \in [d]$ ,  $|\mathbb{E}[\mathbf{x}_i(y - p(\mathbf{x}))]| \leq \varepsilon$ . It is called  $\varepsilon$ -calibrated if  $|\mathbb{E}_{p(\mathbf{x})}\mathbb{E}_{y|p(\mathbf{x})}[y - p(\mathbf{x})]| \leq \varepsilon$ .
2 Distortion Bounds for the Matching Loss
In this section, we propose a simple approach for bounding the squared error of a predictor that minimizes a (convex) matching loss, in the agnostic setting. We convert matching loss bounds to squared loss bounds in a generic way, through appropriate pointwise distortion bounds between the two losses. In particular, for a given matching loss  $\mathcal{L}_g$ , we transform guarantees on  $\mathcal{L}_g$  that are competitive with the optimum linear minimizer of  $\mathcal{L}_g$  to guarantees on the squared error that are competitive with the optimum GLM whose activation  $(g')$  depends on the matching loss at hand.
We now provide the main result we establish in this section.
Theorem 2.1 (Squared Error Minimization through Matching Loss Minimization). Let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$ , let  $0 < \alpha \leq \beta$  and let  $(f,g)$  be a Fenchel-Legendre pair such that  $g':\mathbb{R}\to \mathbb{R}$  is  $[\alpha ,\beta ]$  bi-Lipschitz. Suppose that for a predictor  $p:\mathbb{R}^d\to \mathrm{ran}(g')$  we have
$$
\mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {\| \mathbf {w} \| _ {2} \leq B} \mathcal {L} _ {g} (\mathbf {w}; D) + \varepsilon \tag {2.1}
$$
Then we also have:  $\mathrm{err}_2(p) \leq \frac{\beta}{\alpha} \cdot \mathrm{opt}(\mathrm{GLM}_{g',B}) + 2\beta \varepsilon$ .
The proof of Theorem 2.1 is based on the following pointwise distortion bound between matching losses corresponding to bi-Lipschitz link functions and the squared distance.