SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 2.2 (Pointwise Distortion Bound for bi-Lipschitz link functions). Let  $0 < \alpha \leq \beta$  and let  $(f, g)$  be a Fenchel-Legendre pair such that  $f': \mathrm{ran}(g') \to \mathbb{R}$  is  $[\frac{1}{\beta}, \frac{1}{\alpha}]$  bi-Lipschitz. Then for any  $y, p \in \mathrm{ran}(g')$  we have
$$
\ell_ {g} (y, f ^ {\prime} (p)) - \ell_ {g} (y, f ^ {\prime} (y)) = \mathbb {D} _ {f} (y, p) \in \left[ \frac {1}{2 \beta} (y - p) ^ {2}, \frac {1}{2 \alpha} (y - p) ^ {2} \right]
$$
In the case that  $f'$  is differentiable on  $(0,1)$ , the proof of Lemma 2.2 follows from an application of Taylor's approximation theorem of degree 2 on the function  $f$ , since the Bregman divergence  $\mathbb{D}_f(y,p)$  is exactly equal to the error of the second degree Taylor's approximation of  $f(y)$  around  $p$  and  $f''(\xi) \in [\frac{1}{\beta}, \frac{1}{\alpha}]$  for any  $\xi \in \mathrm{ran}(g')$ . The relationship between  $\ell_g$  and  $\mathbb{D}_f$  follows from property (1.2). Note that when  $g'$  is  $[\alpha, \beta]$  bi-Lipschitz, then  $f'$  is  $[\frac{1}{\beta}, \frac{1}{\alpha}]$  bi-Lipschitz.
Theorem 2.1 follows by applying Lemma 2.2 appropriately to bound the error of a predictor  $p$  by its matching loss  $\mathcal{L}_g(f' \circ p)$  and bound the matching loss of the linear function corresponding to  $\mathbf{w}^*$  by the squared error of  $g'(\mathbf{w}^* \cdot \mathbf{x})$ , where  $g'(\mathbf{w}^* \cdot \mathbf{x})$  is the element of  $\mathrm{GLM}_{g',B}$  with minimum squared error.
Although Theorem 2.1 only applies to bi-Lipschitz activations  $g'$ , it has the advantage that the assumption it makes on  $p$  corresponds to a convex optimization problem and, when the marginal distribution has certain concentration properties (for generalization), can be solved efficiently through gradient descent on the empirical loss function. As a consequence, for bi-Lipschitz activations we can obtain  $O(\mathrm{opt})$  efficiently under mild distributional assumptions in the agnostic setting.
3 Agnostically Learning Single-Index Models
In this section, we provide our main result on agnostically learning SIMs. We combine the distortion bounds we established in Section 2 with results from Gopalan et al. [2023] on Omniprediction, which can be used to learn a predictor  $p$  that satisfies the guarantee of Theorem 2.1 simultaneously for all bi-Lipschitz activations. By doing so, we obtain a result for all Lipschitz and non-decreasing activations simultaneously.
Theorem 3.1 (Agnostically Learning SIMs). Let  $\lambda, B \geq 1$ ,  $\varepsilon > 0$  and let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$  with second moments bounded by  $\lambda$ . Then, Algorithm 1 agnostically learns the class  $\mathrm{SIM}_B$  over  $D$  up to squared error  $O(B\sqrt{\lambda}\sqrt{\mathrm{opt}(\mathrm{SIM}_B, D)}) + \varepsilon$  using time and sample complexity poly $(d, B, \lambda, \frac{1}{\varepsilon})$ . Moreover, the same is true for any algorithm with an omniprediction guarantee like the one of Theorem 3.2.
In order to apply Theorem 2.1, we use the following theorem which is a combination of results in Gopalan et al. [2023], where they show that the matching losses corresponding to a wide class of functions can all be minimized simultaneously by an efficiently computable predictor.
Theorem 3.2 (Omnipredictors for Matching Losses, combination of results in Gopalan et al. [2023]). Let  $\lambda, L, R, B \geq 1$ ,  $\varepsilon > 0$  and let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$  whose marginal on  $\mathbb{R}^d$  has  $\lambda$ -bounded second moments. Then, Algorithm 1, given sample access to  $D$ , with probability at least  $2/3$  returns a predictor  $p: \mathbb{R}^d \to (0,1)$  with the following guarantee. For any Fenchel-Legendre pair  $(f,g)$  such that  $g': \mathbb{R} \to \mathbb{R}$  is  $L$ -Lipschitz, and  $f'$  takes values within the interval  $[-R,R]$ ,  $p$  satisfies
$$
\mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {\| \mathbf {w} \| _ {2} \leq B} \mathcal {L} _ {g} (\mathbf {w}; D) + \varepsilon .
$$
The algorithm requires time and sample complexity  $\mathrm{poly}(\lambda, B, L, R, \frac{1}{\varepsilon})$ .
Algorithm 1 is a version of the algorithm of Gopalan et al. [2023] for calibrated multiaccuracy, specific to our setting. For a more quantitative version of Theorem 3.2, see Theorem C.3 in the appendix.
We aim to apply Theorem 3.2 to the class of all Lipschitz activations (which is wider than the class of bi-Lipschitz activations). This is enabled by the following lemma, whose proof is based on Theorem 2.1 and the fact that the error of a predictor is bounded by the sum of the error of another predictor and the squared expected distance between the two predictors.
Lemma 3.3. Let  $D$  be a distribution over  $\mathbb{R}^d\times [0,1]$ . Let  $g^{\prime}:\mathbb{R}\to \mathbb{R}$  be some fixed activation, and  $f^{\prime}$  its dual. Consider the class  $\mathrm{GLM}_{g',B}$ , and let  $\mathbf{w}^{*}$  be the weights achieving opt  $(\mathrm{GLM}_{g',B},D)$ . Let  $\phi^{\prime}:\mathbb{R}\rightarrow \mathbb{R}$  be an  $[\alpha ,\beta ]$  bi-Lipschitz function (differentiable except possibly at finitely many points) that we wish to approximate  $g^{\prime}$  by. Any predictor  $p:\mathbb{R}^d\to \mathbb{R}$  that satisfies
$$
\mathcal {L} _ {\phi} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {\| \mathbf {w} \| _ {2} \leq B} \mathcal {L} _ {\phi} (\mathbf {w}; D) + \varepsilon
$$
also satisfies the following  $\ell_2$  error guarantee:
$$
\operatorname {e r r} _ {2} (p) \leq \frac {2 \beta}{\alpha} \operatorname {o p t} \left(\mathrm {G L M} _ {g ^ {\prime}, B}\right) + \frac {2 \beta}{\alpha} \mathbb {E} \left[ \left(g ^ {\prime} (\mathbf {w} ^ {*} \cdot \mathbf {x}) - \phi^ {\prime} (\mathbf {w} ^ {*} \cdot \mathbf {x})\right) ^ {2} \right] + 2 \beta \varepsilon .
$$
Algorithm 1: Calibrated Multiaccuracy (modification of Algorithm 2 in Gopalan et al. [2023])
Input: Sufficiently large training set  $S$  over  $\mathbb{R}^d\times [0,1]$  parameters  $\varepsilon ,\lambda ,B,L,R$
Output: A predictor  $p:\mathbb{R}^d\to (0,1)$
Form a training set  $S'$  by substituting each element  $(\mathbf{x}, y)$  in  $S$  with  $(\mathbf{x}, y')$  where  $y' \in \{0, 1\}$  is formed as follows. Conditioned on  $y$ , let  $y'$  be an independent Bernoulli random variable with probability of success  $y$ .
Let  $A = B \cdot L$ ,  $C > 0$  a sufficiently large universal constant and let  $\delta = \frac{\varepsilon^2}{CA^2\lambda R^2}$ ,  $\sigma = \frac{\varepsilon - \delta R}{2CA^2\lambda R}$ . Set  $p \gets 0$  (the zero function).
Let  $S''$  be a subset of  $S'$  with  $|S''| = C\frac{d^2A^2\lambda R^2}{\varepsilon^2}$  and set  $S' \gets S' \setminus S''$ .
Form a set  $T$  by substituting each element  $(\mathbf{x},y^{\prime})$  of  $S''$  with  $(\mathbf{x},z)$ , where
$$
z = y ^ {\prime} - p (\mathbf {x}).
$$
$$
\operatorname {L e t} \mathbf {v} = \frac {1}{| T |} \sum_ {(\mathbf {x}, z) \in T} z \mathbf {x}.
$$
Reject if  $\| \mathbf{v}\| _2\leq 3(\varepsilon -A\sqrt{\lambda\delta}) / (4A)$
Otherwise, let  $\mathbf{w} = B\mathbf{v} / \| \mathbf{v}\| _2$  and update  $p$  to be the following predictor
$$
\mathbf {x} \mapsto (p (\mathbf {x}) + \sigma (\mathbf {w} \cdot \mathbf {x})) _ {[ 0, 1 ]}, \text {w h e r e} (t) _ {[ 0, 1 ]} = \min  \left\{\max  \{t, 0 \}, 1 \right\}
$$
until some iteration rejects (which happens when it has found a multiaccurate predictor);
Let  $p^\delta : \mathbb{R}^d \to [0,1]$  such that  $p^\delta(\mathbf{x}) = \sum_{j=0}^{1/\delta} j \delta \mathbb{1}\{p(\mathbf{x}) \in I_j\}$ , where
$$
I _ {j} = \left[ \left(j - \frac {1}{2}\right) \delta , \left(j + \frac {1}{2}\right) \delta \right].
$$
Let  $S''$  be a subset of  $S'$  with  $|S''| = C\frac{A^84R^8}{\varepsilon^8}\log \left(\frac{\lambda R}{\varepsilon}\right)$ , set  $S' \gets S' \setminus S''$  and let
$$
S _ {j} ^ {\prime \prime} = \left\{\left(\mathbf {x}, y ^ {\prime}\right) \in S ^ {\prime \prime}: p ^ {\delta} (\mathbf {x}) = j \delta \right\}.
$$
Estimate the calibration error  $|\mathbb{E}_{p^{\delta}(\mathbf{x})}\mathbb{E}_{y|p^{\delta}(\mathbf{x})}[y - p^{\delta}(\mathbf{x})]|$  of  $p^\delta$  using the samples in  $S''$  according to the following expression
$$
\sum_ {j = 0} ^ {1 / \delta} \frac {\left| S _ {j} ^ {\prime \prime} \right|}{\left| S ^ {\prime \prime} \right|} \left| j \delta - \bar {y} _ {j} \right|, \text {w h e r e} \bar {y} _ {j} = \frac {1}{\left| S _ {j} ^ {\prime \prime} \right|} \sum_ {(\mathbf {x}, y ^ {\prime}) \in S _ {j} ^ {\prime \prime}} y ^ {\prime}
$$
If the estimate has value at most  $\varepsilon /2$ , then output  $p^\delta$  and terminate.
Otherwise, update  $p$  to be the following (calibrated) predictor
$$
\mathbf {x} \mapsto \sum_ {j = 0} ^ {1 / \delta} \bar {y} _ {j} \mathbb {1} \left\{p (\mathbf {x}) \in I _ {j} \right\}
$$
until some iteration terminates the execution;
By combining Theorem 3.2 with Lemma 3.3 (whose proofs can be found in Appendix C), we are now ready to prove our main theorem.
Proof of Theorem 3.1. We will combine Theorem 3.2, which states that there is an efficient algorithm that simultaneously minimizes the matching losses corresponding to bounded, non-decreasing and Lipschitz activations, with Lemma 3.3, which implies that minimizing the matching loss corresponding to the nearest bi-Lipschitz activation is sufficient to obtain small error. Note that we may assume that  $\varepsilon < 1/2$ , since otherwise the problem is trivial (output the zero function and pick  $C = 2$ ).
As a first step, we show that link functions corresponding to bi-Lipschitz activations are bounded (according to Definition C.1, for  $\gamma = 0$ ). In particular, let  $\phi^{\prime}:\mathbb{R}\to \mathbb{R}$  be an  $[\alpha ,\beta ]$  bi-Lipschitz activation for some  $\beta \geq \alpha >0$  such that  $\phi^{\prime}(s)\in [-1,2]$  for some  $s\in \mathbb{R}$  and let  $\psi^\prime$  be the inverse of  $\phi^{\prime}$  ( $\phi^{\prime}$  is invertible since it is strictly increasing). We will show that  $\psi^{\prime}(r)\in [-R,R]$  for any  $r\in [0,1]$  and  $R = O(|s| + 1 / \alpha)$ .
We pick  $r_0 = 0$ ,  $r_1 = 1$  and get that  $|\psi'(\phi'(s)) - \psi'(r_0)| \leq \frac{1}{\alpha} |\phi'(s) - r_0| \leq \frac{2}{\alpha}$ . Hence  $\psi'(r_0) \geq \psi'(\phi'(s)) - \frac{2}{\alpha} = s - \frac{1}{\alpha}$ . Similarly, we get  $\psi'(r_1) \leq s + \frac{2}{\alpha}$ . Therefore,  $\psi'(r) \in [\psi'(0), \psi'(1)] \subseteq [-|s| - \frac{2}{\alpha}, |s| + \frac{2}{\alpha}]$ , for any  $r \in [0,1]$ , due to monotonicity of  $\psi'$ .
For a given non-decreasing and 1-Lipschitz  $g'$ , we will now show that there is a bounded bi-Lipschitz activation  $\phi'$  such that if the assumption of Lemma 3.3 is satisfied for  $\phi'$  by a predictor  $p$ , then the error of  $p$  is bounded by
$$
\operatorname {e r r} _ {2} (p) \leq O \left(B \sqrt {\lambda} \left(\operatorname {o p t} \left(\operatorname {G L M} _ {g ^ {\prime}, B}\right)\right) ^ {1 / 2}\right) + O \left(\lambda B ^ {2} \varepsilon\right)
$$
Suppose, first, that  $\mathrm{opt}(\mathrm{GLM}_{g',B})\leq \varepsilon^2$ . Then, we pick  $\phi '(t) = g'(t) + \varepsilon t$ . Note that  $\phi^\prime$  is  $[\varepsilon ,1 + \varepsilon ]$  bi-Lipschitz. Moreover, since  $\mathrm{opt}(\mathrm{GLM}_{g',B})\leq \varepsilon^2$ , we must have some  $s\in \mathbb{R}$  with  $|s|\leq 2\lambda B^2$  such that  $g^{\prime}(s)\in [-1,2]$ . Otherwise, we would have  $\mathrm{opt}(\mathrm{GLM}_{g',B}) = \mathbf{E}[(g'(\mathbf{w}^{*}\cdot \mathbf{x}) - y)^{2}]\geq$ $\mathbf{E}[(g'(\mathbf{w}^{*}\cdot \mathbf{x}) - y)^{2}]\mid |\mathbf{w}^{*}\cdot \mathbf{x}\leq 2\lambda B^{2}|]\cdot \mathbb{P}[\left|\mathbf{w}^{*}\cdot \mathbf{x}\right|\leq 2\lambda B^{2}]\geq \mathbb{P}[\left|\mathbf{w}^{*}\cdot \mathbf{x}\right|\leq 2\lambda B^{2}] = 1 - \mathbb{P}[\left|\mathbf{w}^{*}\cdot \mathbf{x}\right|>$ $2\lambda B^2 ]\geq \frac{1}{4} >\varepsilon^2$ , due to Chebyshev's inequality, the fact that  $\mathbf{w}^{*}\in \mathcal{W}$  and the bounded moments assumption. Therefore,  $\psi^\prime$  is  $(R = 2\lambda B^2 +\frac{2}{\varepsilon},\gamma = 0)$ -bounded and we have
$$
\mathbb {E} \left[ \left(g ^ {\prime} \left(\mathbf {w} ^ {*} \cdot \mathbf {x}\right) - \phi^ {\prime} \left(\mathbf {w} ^ {*} \cdot \mathbf {x}\right)\right) ^ {2} \right] \leq \varepsilon^ {2} \mathbb {E} [ (\mathbf {w} ^ {*} \cdot \mathbf {x}) ^ {2} ] \leq \varepsilon^ {2} \lambda B ^ {2}
$$
As a consequence, under the assumption of Lemma 3.3 for  $\phi'$ , the error of the corresponding predictor  $p$  is  $\mathrm{err}_2(p) \leq 2(1 + \varepsilon)\varepsilon + 2(1 + \varepsilon)\varepsilon\lambda B^2 + 2(1 + \varepsilon)\varepsilon = O(\lambda B^2\varepsilon)$ .
In the case that  $\mathrm{opt}(\mathrm{GLM}_{g',B}) > \varepsilon^2$ , we pick  $\phi'(t) = g'(t) + t \frac{\sqrt{\mathrm{opt}(\mathrm{GLM}_{g',B})}}{B \sqrt{\lambda}}$ . We may also assume that  $\mathrm{opt}(\mathrm{GLM}_{g',B}) \leq 1/2$ , since otherwise any predictor with range  $[0,1]$  will have error at most  $2\mathrm{opt}(\mathrm{GLM}_{g',B})$ . Then,  $\psi$  is  $(O(\lambda B^2 + \frac{B \sqrt{\lambda}}{\varepsilon}), 0)$ -bounded,  $\phi'$  is  $[\frac{1}{B} \sqrt{\mathrm{opt}(\mathrm{GLM}_{g',B})} / \sqrt{\lambda}, 1 + \frac{1}{B}]$  bi-Lipschitz which gives
$$
\mathbb {E} \left[ \left(g ^ {\prime} \left(\mathbf {w} ^ {*} \cdot \mathbf {x}\right) - \phi^ {\prime} \left(\mathbf {w} ^ {*} \cdot \mathbf {x}\right)\right) ^ {2} \right] \leq \frac {\operatorname {o p t} \left(\mathrm {G L M} _ {g ^ {\prime} , B}\right)}{B ^ {2} \lambda} \mathbb {E} [ (\mathbf {w} ^ {*} \cdot \mathbf {x}) ^ {2} ] \leq \operatorname {o p t} \left(\mathrm {G L M} _ {g ^ {\prime}, B}\right)
$$
As a consequence, under the assumption of Lemma 3.3 for  $\phi'$ , the error of the corresponding predictor  $p$  is  $\mathrm{err}_2(p) \leq 4(1 + \frac{1}{B}) B \sqrt{\lambda} \sqrt{\mathrm{opt}(\mathrm{GLM}_{g',B})} + 2(1 + \frac{1}{B}) \varepsilon$ . Using a similar approach as for the case  $\mathrm{opt}(\mathrm{GLM}_{g',B}) \leq \varepsilon$ , we can show that  $\psi'$  is polynomially bounded (as per Definition C.1), since  $\mathrm{opt}(\mathrm{GLM}_{g',B}) \leq \frac{1}{2}$ .
To conclude the proof of Theorem 3.1, we apply Theorem 3.2 with appropriate (polynomial) choice of parameters  $(R = O(\lambda B^2 +\frac{B\sqrt{\lambda}}{\varepsilon}),L = 2)$ , to show that there is an efficient algorithm that outputs a predictor  $p:\mathbb{R}^d\to (0,1)$  for which the assumption of Lemma 3.3 holds simultaneously for all bi-Lipschitz activations  $(\phi^{\prime})$  with sufficiently bounded inverses  $(\psi^{\prime})$ .
4 Stronger Guarantees for Logistic Regression
In this section, we follow the same recipe we used in Section 2 to get distortion bounds similar to Theorem 2.1 for the sigmoid activation (or, equivalently, the logistic model) under the assumption that the marginal distribution is sufficiently concentrated (see Definition C.1). In particular, Theorem 2.1 does not hold, since the sigmoid is not bi-Lipschitz and our main Theorem 3.1 only provides a guarantee of  $O(\sqrt{\mathrm{opt}})$  for squared error. We use appropriate pointwise distortion bounds for the matching loss corresponding to the sigmoid activation and provide guarantees of  $\widetilde{O} (\mathrm{opt})$  for logistic regression with respect to both squared and absolute error, under appropriate assumptions about the concentration of the marginal distribution. The proofs of this section are provided in Appendix D.
For the logistic model, the link function  $f'$  is defined as  $f'(r) = \ln \left( \frac{r}{1 - r} \right)$ , for  $r \in (0, 1)$  and the corresponding activation  $g'$  is the sigmoid  $g'(t) = \frac{1}{1 + e^{-t}}$  for  $t \in \mathbb{R}$ . The corresponding matching loss is the logistic loss.
Squared error. We first provide a result for squared loss minimization. In comparison to Theorem 2.1, the qualitative interpretation of the following theorem is that, while the sigmoid activation is not bi-Lipschitz, it is effectively bi-Lipschitz under sufficiently concentrated marginals.
Theorem 4.1 (Squared Loss Minimization through Logistic Loss Minimization). Let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$  whose marginal on  $\mathbb{R}^d$  is  $(1,2)$ -concentrated. Let  $g':\mathbb{R}\to \mathbb{R}$  be the sigmoid activation, i.e.,  $g'(t) = (1 + e^{-t})^{-1}$  for  $t\in \mathbb{R}$ . Assume that for some  $B > 0$ ,  $\varepsilon >0$  and a predictor  $p:\mathbb{R}^d\rightarrow (0,1)$  we have
$$
\mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {\mathbf {w}: \| \mathbf {w} \| _ {2} \leq B} \mathcal {L} _ {g} (\mathbf {w}; D) + \varepsilon \tag {4.1}
$$
If we let  $\mathrm{opt}_g = \min_{\| \mathbf{w}\| _2\leq B}\mathrm{err}_2(g_\mathbf{w}')$  , then for the predictor  $p$  and some universal constant  $C > 0$  we also have
$$
\operatorname {e r r} _ {2} (p) \leq C \operatorname {o p t} _ {g} \exp \left(B ^ {2} + \sqrt {B ^ {2} \log \frac {1}{\operatorname {o p t} _ {g}}}\right) + 2 \varepsilon .
$$
In particular, the squared error of  $p$  is upper bounded by  $\widetilde{O}(\mathrm{opt}_g)$ , since the function  $t \mapsto \exp(\log^{1/2} t)$  is asymptotically smaller than any polynomial function  $t \mapsto t^\gamma$  with  $\gamma > 0$ .
Once more, the proof of our result is based on an appropriate pointwise distortion bound which we provide below. It follows by the fact that the Bregman divergence corresponding to the sigmoid is the Kullback-Leibler divergence and by combining Pinsker's inequality (lower bound) with Lemma 4.1 of Götze et al. [2019] (upper bound).
Lemma 4.2 (Pointwise Distortion Bound for Sigmoid). Let  $g'$  be the sigmoid activation. Then, for any  $y, p \in (0,1)$  we have that  $\mathbb{D}_f(y,p) = \mathbb{D}_{\mathrm{KL}}(y\|p) = y\ln(y/p) + (1 - y)\ln(\frac{1 - y}{1 - p})$ . Moreover
$$
\ell_ {g} (y, f ^ {\prime} (p)) - \ell_ {g} (y, f ^ {\prime} (y)) = \mathbb {D} _ {\mathrm {K L}} (y \| p) \in \left[ \frac {1}{2} (y - p) ^ {2}, \frac {2}{\min  \{p , 1 - p \}} \cdot (y - p) ^ {2} \right]
$$
We translate Lemma 4.2 to a bound on the squared error of a matching loss minimizer (in this case, the logistic loss) using an approach similar to the one for Theorem 2.1. In order to use the upper bound on the surrogate loss provided by Lemma 4.2, we apply it to  $p \gets g'(\mathbf{w} \cdot \mathbf{x})$ , where  $g'$  is the sigmoid function, and observe that the quantity  $\frac{1}{p(1 - p)}$  is exponential in  $|\mathbf{w} \cdot \mathbf{x}|$ . Hence, when the marginal is  $(\lambda, 2)$ -concentrated (subgaussian concentration), then  $\frac{1}{p(1 - p)}$  is effectively bounded.
Absolute error. All of the results we have provided so far have focused on squared error minimization. We now show that our approach yields results even for the absolute error, which can also be viewed as learning in the p-concept model [Kearns and Schapire, 1994]. In particular, for a distribution  $D$  over  $\mathbb{R}^d\times [0,1]$ , we define the absolute error of a predictor  $p:\mathbb{R}^d\to [0,1]$  as follows.
$$
\operatorname {e r r} _ {1} (p) = \underset {(\mathbf {x}, y) \sim D} {\mathbb {E}} [ | y - p (\mathbf {x}) | ]
$$
In the specific case when the labels are binary, i.e.,  $y \in \{0,1\}$ , we have
$$
\operatorname {e r r} _ {1} (p) = \underset {(\mathbf {x}, y) \sim D} {\mathbb {E}} [ | y - p (\mathbf {x}) | ] = \underset {(\mathbf {x}, y, y _ {p}) \sim D _ {p}} {\mathbb {P}} [ y \neq y _ {p} ] \quad \text {(s e e P r o p o s i t i o n A . 2)}
$$
where the distribution  $D_{p}$  is over  $\mathbb{R}^d\times \{0,1\} \times \{0,1\}$  and is formed by drawing samples  $(\mathbf{x},y)$  from  $D$  and, given  $\mathbf{x}$ , forming  $y_{p}$  by drawing a conditionally independent Bernoulli random variable with parameter  $p(\mathbf{x})$ . We provide the following result.
Theorem 4.3 (Absolute Loss Minimization through Logistic Loss Minimization). Let  $D$  be a distribution over  $\mathbb{R}^d \times \{0,1\}$  whose marginal on  $\mathbb{R}^d$  is  $(1,1)$ -concentrated. Let  $g': \mathbb{R} \to \mathbb{R}$  be the sigmoid activation, i.e.,  $g'(t) = (1 + e^{-t})^{-1}$  for  $t \in \mathbb{R}$ . Assume that for some  $B > 0$ ,  $\varepsilon > 0$  and a predictor  $p: \mathbb{R}^d \to (0,1)$  we have
$$
\mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {\mathbf {w}: \| \mathbf {w} \| _ {2} \leq B} \mathcal {L} _ {g} (\mathbf {w}; D) + \varepsilon \tag {4.2}
$$
If we let  $\mathrm{opt}_g = \min_{\| \mathbf{w}\| _2\leq B}\mathrm{err}_1(g_\mathbf{w}')$ , then for the predictor  $p$  and some universal constant  $C > 0$  we also have
$$
\operatorname {e r r} _ {1} (p) \leq C B \operatorname {o p t} _ {g} \log \frac {1}{\operatorname {o p t} _ {g}} + \varepsilon
$$
The corresponding distortion bound in this case is between the absolute and logistic losses and works when the labels are binary.
Lemma 4.4 (Pointwise Distortion between Absolute and Logistic Loss). Let  $g'$  be the sigmoid activation. Then, there is a constant  $c \in \mathbb{R}$  such that for any  $y \in \{0,1\}$  and  $p \in (0,1)$ , we have
$$
\ell_ {g} (y, f ^ {\prime} (p)) - c \in \left[ | y - p |, 2 \cdot \ln \left(\frac {1}{p (1 - p)}\right) \cdot | y - p | \right]
$$
The bound of Theorem 4.3 implies an algorithm for learning an unknown sigmoid neuron in the p-concept model, by minimizing a convex loss. While there are algorithms achieving stronger guarantees [Diakonikolas et al., 2022b] for agnostically learning sigmoid neurons, such algorithms typically make strong distributional assumptions including concentration, anti-concentration and anti-anti-concentration or boundedness.
Moreover, it is useful to compare the bound we provide in Theorem 4.3 to a lower bound by Diakonikolas et al. [2020c, Theorem 4.1], which concerns the problem of agnostically learning halfspaces by minimizing convex surrogates. In particular, they show that even under log-concave marginals, no convex surrogate loss can achieve a guarantee better than  $O(\mathrm{opt}\log (1 / \mathrm{opt}))$ , where opt is measured with respect to the  $\ell_1$  error (which is equal to the probability of error). The result is not directly comparable to our upper bound, since we examine the sigmoid activation. Their setting can be viewed as a limit case of ours by letting the norm of the weight vector grow indefinitely (the sigmoid tends to the step function), but the main complication is that our upper bound is of the form  $O(B\mathrm{opt}\log (1 / \mathrm{opt}))$ , which scales with  $B$ . However, their lower bound concerns marginal distributions that are not only concentrated, but are also anti-concentrated and anti-anticoncentrated, while our results only make concentration assumptions.
5 Necessity of Norm Dependence
In this final section, we use a lower bound due to Diakonikolas et al. [2022a] on agnostic learning of GLMs using SQ algorithms and compare it with our main result (Theorem 3.1). For simplicity, we specialize to the case of the standard sigmoid or logistic function. A modification of their proof ensures that the bound holds under isotropic margins.
Theorem 5.1 (SQ Lower Bound for Agnostically Learning GLMs, variant of [Diakonikolas et al., 2022a, Thm C.3]). Let  $g^{\prime}:\mathbb{R}^{d}\to \mathbb{R}$  be the standard logistic function. Any SQ algorithm either requires  $d^{\omega (1)}$  queries or  $d^{-\omega (1)}$  tolerance to distinguish between the following two labeled distributions:
- (Labels have signal.)  $D_{\mathrm{signal}}$  on  $\mathbb{R}^d\times \mathbb{R}$  is such that  $\mathrm{opt}(\mathrm{GLM}_{g',B},D_{\mathrm{signal}})\leq \exp (-\Omega (\log^{1 / 4}d)) = o(1)$  for some  $B = \mathrm{poly}(d)$ .
- (Labels are random.)  $D_{\mathrm{random}}$  on  $\mathbb{R}^d \times \mathbb{R}$  is such that the labels  $y$  are drawn i.i.d. from  $\{a, b\}$  for certain universal constants  $a, b \in [0,1]$ . In particular,  $\mathrm{opt}(\mathrm{GLM}_{g',B}, D_{\mathrm{random}}) = \Omega(1)$  for any  $B$ .
Both  $D_{\mathrm{signal}}$  and  $D_{\mathrm{random}}$  have the same marginal on  $\mathbb{R}^d$ , with 1-bounded second moments.