SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let us consider applying our main theorem (Theorem 3.1) to this setting, with  $D$  being either  $D_{\mathrm{signal}}$  or  $D_{\mathrm{random}}$ , and with the same  $B = \mathrm{poly}(d)$  as is required to achieve small error in the "labels have signal" case. We would obtain a predictor with  $\ell_2$  error at most  $B\sqrt{\mathrm{opt}(\mathrm{GLM}_{g',B})}$  (or indeed with  $\mathrm{SIM}_B$  in place of  $\mathrm{GLM}_{g',B}$ ). Since this is  $\omega(1)$ , this guarantee is insufficient to distinguish the two cases above, which is as it should be since our main algorithm indeed fits into the SQ framework.
Theorem 5.1 does, however, justify a dependence on the norm  $B$  in our main result. In particular, it is clear that a guarantee of the form  $\mathrm{opt}(\mathrm{GLM}_{g',B})^c$  for any universal constant  $c > 0$  (independent of  $B$ ) would be too strong, as it would let us distinguish the two cases above. In fact, this lower bound rules out a large space of potential error guarantees stated as functions of  $B$  and  $\mathrm{opt}(\mathrm{GLM}_{g',B})$ . For instance, for sufficiently large  $d$ , it rules out any error guarantee of the form  $\exp(O(\log^{1/5}B)) \cdot \mathrm{opt}(\mathrm{GLM}_{g',B})^{c'}$  for any universal constant  $c' > 0$ .
${}^{2}$  Specifically,our features correspond to all multilinear monomials (or parities) of degree at most  $k$  over  ${\left\{  \pm  1\right\}  }^{n}$  ,whereas they use all monomials (not necessarily multilinear) of degree at most  $k$  . These yield equivalent representations since the hard distributions are obtained from the uniform distribution on  ${\left\{  \pm  1\right\}  }^{n}$  .
We wish to thank the anonymous reviewers of NeurIPS 2023 for their constructive feedback. Aravind Gollakota was at UT Austin while this work was done, supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Adam R. Klivans was supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Konstantinos Stavropoulos was supported by NSF award AF-1909204, the NSF AI Institute for Foundations of Machine Learning (IFML), and by scholarships from Bodossaki Foundation and Leventis Foundation.
Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of a single neuron with gradient descent. Advances in Neural Information Processing Systems, 33:5417-5428, 2020. 1, 1.1
Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning a single neuron with adversarial label noise via gradient descent. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4313-4361. PMLR, 02-05 Jul 2022b. URL https://proceedings.mlr.press/v178/diakonikolas22c.html. 1, 1, 1.1, 4
Ilias Diakonikolas, Surbhi Goel, Sushrut Karmalkar, Adam R Klivans, and Mahdi Soltanolkotabi. Approximation schemes for relu regression. In Conference on Learning Theory, pages 1452-1485. PMLR, 2020b. 1.1
Ryan O'Donnell and Rocco A Servedio. The chow parameters problem. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 517-526, 2008. 1.1
Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (Computationally-identifiable) masses. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1939–1948. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/hebert-johnson18a.html.1.1
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019. 1.1
Shai Shalev-Shwartz, Ohad Shamir, and Karthik Sridharan. Learning kernel-based halfspaces with the 0-1 loss. SIAM Journal on Computing, 40(6):1623-1646, 2011. 1.1
Surbhi Goel, Varun Kanade, Adam Klivans, and Justin Thaler. Reliably learning the relu in polynomial time. In Conference on Learning Theory, pages 1004-1042. PMLR, 2017. 1.1
Friedrich Götze, Holger Sambale, and Arthur Sinulis. Higher order concentration for functions of weakly dependent random variables. *Electronic Journal of Probability*, 24:1-19, 2019. 4
Michael J Kearns and Robert E Schapire. Efficient distribution-free learning of probabilistic concepts. Journal of Computer and System Sciences, 48(3):464-497, 1994. 4
Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Non-convex sgd learns halfspaces with adversarial label noise. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18540-18549. Curran Associates, Inc., 2020c. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/d785bf9067f8af9e078b93cf26de2b54-Paper.pdf. 4
A Technical Lemmas
In this section, we provide some technical Lemmas that we use in our proofs.
Proposition A.1 (Weak Learner for Linear Functions). Let  $D$  be a distribution over  $\mathbb{R}^d\times [-1,1]$  whose marginal on  $\mathbb{R}^d$  has  $\lambda$ -bounded second moments and  $B > 0$ . For any  $\varepsilon >0$  and  $\delta \in (0,1)$ , there is a universal constant  $C > 0$  and an algorithm that given a set  $S$  of i.i.d. samples from  $D$  of size at least  $C\cdot \frac{d^2\lambda B^2}{\varepsilon^2}\log \frac{1}{\delta}$ , runs in time  $O(d|S|)$  and satisfies the following specifications with probability at least  $1 - \delta$
1. If  $\mathbb{E}_{(\mathbf{x},z)\sim D}[z(\mathbf{w}\cdot \mathbf{x})]\geq \varepsilon$  for some  $\mathbf{w}\in \mathbb{R}^d$  with  $\| \mathbf{w}\| _2\leq B$ , then the algorithm accepts. Otherwise, it may or may not reject and return a special symbol.
2. If the algorithm accepts then it returns  $\mathbf{w} \in \mathbb{R}^d$  with  $\| \mathbf{w} \|_2 \leq B$  such that we have  $\mathbb{E}_{(\mathbf{x}, z) \sim D}[z(\mathbf{w} \cdot \mathbf{x})] \geq \varepsilon / 4$ .
Proof. We will prove the proposition for  $\delta = 1 / 6$ . We may boost the probability of success with repetition.
The algorithm computes the vector  $\mathbf{v} = \mathbb{E}_S[z\mathbf{x}]$ . If  $\| \mathbf{v}\| _2\leq \frac{3\varepsilon}{4B}$ , then the algorithm rejects and outputs a special symbol. Otherwise, it outputs the vector  $\frac{B}{\|\mathbf{v}\|_2}$ $\mathbf{v}$ .
Suppose, first, that  $\mathbb{E}_{(\mathbf{x},z)\sim D}[z(\mathbf{w}\cdot \mathbf{x})]\geq \varepsilon$  for some  $\mathbf{w}$  with  $\| \mathbf{w}\| _2\leq B$ . Then, due to Chebyshev's inequality we have for any  $i\in [d]$
$$
\mathbb {P} \left[ \left| \underset {S} {\mathbb {E}} [ z \mathbf {x} _ {i} ] - \underset {D} {\mathbb {E}} [ z \mathbf {x} _ {i} ] \right| > \frac {\varepsilon}{8 B \sqrt {d}} \right] \leq \frac {6 4 d B ^ {2}}{| S | \varepsilon^ {2}} \mathbb {E} [ \mathbf {x} _ {i} ^ {2} ] \leq \frac {6 4 d B ^ {2} \lambda}{| S | \varepsilon^ {2}} \leq \frac {1}{6 d} \quad (\text {f o r l a r g e e n o u g h} | S |)
$$
Hence, with probability at least  $5/6$ , we have  $\| \mathbb{E}_S[z\mathbf{x}] - \mathbb{E}_D[z\mathbf{x}]\|_2 \leq \frac{\varepsilon}{8B}$ , due to a union bound. Therefore,  $\| \mathbf{v}\|_2 \geq \| \mathbb{E}_D[z\mathbf{x}]\|_2 - \frac{\varepsilon}{8B} \geq \frac{\mathbb{E}_D[z(\mathbf{w} \cdot \mathbf{x})]}{B} - \frac{\varepsilon}{8B} \geq \frac{7B}{8}$  and the algorithm accepts.
Suppose, now, that the algorithm accepts. Then, we have  $\| \mathbf{v}\| _2 > \frac{3\varepsilon}{4B}$  and (with probability at least  $5 / 6$ ) we have
$$
\underset {D} {\mathbb {E}} \left[ \frac {B}{\| \mathbf {v} \| _ {2}} z (\mathbf {v} \cdot \mathbf {x}) \right] = \frac {B}{\| \mathbf {v} \| _ {2}} \mathbf {v} \cdot \mathbb {E} [ z \mathbf {x} ] \geq \varepsilon / 4
$$
since  $\| \mathbb{E}_S[z\mathbf{x}] - \mathbb{E}_D[z\mathbf{x}]\| _2\leq \frac{\varepsilon}{8B}$ . This concludes the proof.
Proposition A.2. Let  $D$  be a distribution over  $\mathbb{R}^d\times \{0,1\}$  and  $p:\mathbb{R}^d\to [0,1]$ . Consider the distribution  $D_{p}$  over  $\mathbb{R}^d\times \{0,1\} \times \{0,1\}$ , which is formed by drawing samples  $(\mathbf{x},y)$  from  $D$  and, given  $\mathbf{x}$ , forming  $y_{p}$  by drawing a conditionally independent Bernoulli random variable with parameter  $p(\mathbf{x})$ . Then we have
$$
\operatorname {e r r} _ {1} (p) = \underset {(\mathbf {x}, y) \sim D} {\mathbb {E}} [ | y - p (\mathbf {x}) | ] = \underset {(\mathbf {x}, y, y _ {p}) \sim D _ {p}} {\mathbb {P}} [ y \neq y _ {p} ]
$$
Proof. Since over  $D_p$ ,  $y$  and  $y_p$  are conditionally independent, we have
$$
\begin{array}{l} \operatorname {e r r} _ {1} (p) = \mathbb {E} [ | y - p (\mathbf {x}) | ] = \mathbb {E} [ (1 - p (\mathbf {x})) \mathbb {1} \{y = 1 \} + p (\mathbf {x}) \mathbb {1} \{y = 0 \} ] \\ = \underset {\mathbf {x}} {\mathbb {E}} \left[ \mathbb {P} [ y _ {p} = 0 | \mathbf {x} ] \mathbb {P} [ y = 1 | \mathbf {x} ] + \mathbb {P} [ y _ {p} = 1 | \mathbf {x} ] \mathbb {P} [ y = 0 | \mathbf {x} ] \right] \\ = \underset {x} {\mathbb {E}} [ \mathbb {P} [ y \neq y _ {p} | \mathbf {x} ] ] = \mathbb {P} [ y \neq y _ {p} ] \\ \end{array}
$$
B Proofs from Section 2
B.1 Proof of Theorem 2.1
To prove Theorem 2.1, we first prove the following more general theorem. Theorem 2.1 may then be easily recovered from this by setting  $\mathcal{H} = \mathrm{GLM}_{g',B}$  and observing that  $f'(g'(\mathbf{w} \cdot \mathbf{x})) = \mathbf{w} \cdot \mathbf{x}$ , since  $g'$  is invertible.
Theorem B.1 (Squared Error Minimization through Distorted Matching Loss Minimization). Let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$ , let  $0 < \alpha \leq \beta$  and let  $(f,g)$  be a pair of Fenchel-Legendre dual functions such that  $g':\mathbb{R}\to \mathbb{R}$  is continuous, non-decreasing and  $f':\mathrm{ran}(g')\to \mathbb{R}$  is  $[\frac{1}{\beta},\frac{1}{\alpha}]$  bi-Lipschitz. Let  $\varepsilon >0$  and  $\mathcal{H} \subseteq \{\mathbb{R}^d\to \mathrm{ran}(g')\}$ . Assume that for a predictor  $p:\mathbb{R}^d\to \mathrm{ran}(g')$  we have
$$
\mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) \leq \min  _ {h \in \mathcal {H}} \mathcal {L} _ {g} \left(f ^ {\prime} \circ h; D\right) + \varepsilon \tag {B.1}
$$
Then, for the predictor  $p$ , we also have:  $\mathrm{err}_2(p) \leq \frac{\beta}{\alpha} \cdot \min_{h \in \mathcal{H}} \mathrm{err}_2(h) + 2\beta \varepsilon$ .
Proof. We apply Lemma 2.2 with  $y \gets y$  and  $p \gets p(\mathbf{x})$  and take expectations over  $D$  on both sides. We have that
$$
\operatorname {e r r} _ {2} (p) \leq 2 \beta \cdot \mathbb {E} \ell_ {g} (y, f ^ {\prime} (p (\mathbf {x}))) - 2 \beta \cdot \mathbb {E} \ell_ {g} (y, f ^ {\prime} (y))
$$
Therefore, we can bound the squared error of  $p$  as follows.
$$
\begin{array}{l} \operatorname {e r r} _ {2} (p) \leq 2 \beta \cdot \mathcal {L} _ {g} \left(f ^ {\prime} \circ p; D\right) - 2 \beta \cdot Q ^ {*} \\ \leq 2 \beta \cdot \mathcal {L} _ {g} \left(f ^ {\prime} \circ h; D\right) - 2 \beta \cdot Q ^ {*} \quad (\text {b y}) \\ \end{array}
$$
where  $Q^{*} = \mathbb{E}\ell_{g}(y,f^{\prime}(y))$
We now apply Lemma 2.2 again with  $y \gets y$  and  $p \gets h(\mathbf{x})$  and we similarly have
$$
\mathbb {E} \ell_ {g} (y, f ^ {\prime} \circ h (\mathbf {x})) - Q ^ {*} \leq \frac {1}{2 \alpha} \operatorname {e r r} _ {2} (h)
$$
Therefore, for any  $h\in \mathcal{H}$  we have, in total:  $\mathrm{err}_2(p)\leq \frac{\beta}{\alpha}\mathrm{err}_2(h) + 2\beta \varepsilon$
We first prove Lemma 2.2, which we restate here for convenience.
Lemma B.2. Assume  $f'$  is  $[1/\beta, 1/\alpha]$  bi-Lipschitz and differentiable on all except from a finite number of points on any bounded interval. Then for any  $y, p \in \mathrm{ran}(g')$  we have
$$
\ell_ {g} (y, f ^ {\prime} (p)) - \ell_ {g} (y, f ^ {\prime} (y)) = D _ {f} (y, p) \in \left[ \frac {1}{2 \beta} (y - p) ^ {2}, \frac {1}{2 \alpha} (y - p) ^ {2} \right]
$$
Proof. We first show that  $\ell_g(y, f'(p)) - \ell_g(y, f'(y)) = D_f(y, p)$ . In particular, we have
$$
g \left(f ^ {\prime} (p)\right) = f ^ {\prime} (p) g ^ {\prime} \left(f ^ {\prime} (p)\right) - f \left(g ^ {\prime} \left(f ^ {\prime} (p)\right)\right) = p f ^ {\prime} (p) - f (p),
$$
since  $f'(p) \in \operatorname{ran}(f')$  and we know that  $g(t) = t g'(t) - f(g'(t))$  for any  $t \in \operatorname{ran}(f')$  as well as  $g'(f'(p)) = p$  for any  $p \in \operatorname{ran}(g')$ . Therefore, we have
$$
\begin{array}{l} \ell_ {g} (y, f ^ {\prime} (p)) - \ell_ {g} (y, f ^ {\prime} (y)) = g \left(f ^ {\prime} (p)\right) - y f ^ {\prime} (p) - g \left(f ^ {\prime} (y)\right) + y f ^ {\prime} (y) \\ = p f ^ {\prime} (p) - f (p) - y f ^ {\prime} (p) - y f ^ {\prime} (y) + f (y) + y f ^ {\prime} (y) = \\ = f (y) - f (p) - (y - p) f ^ {\prime} (p) = D _ {f} (y, p). \\ \end{array}
$$
Let  $\psi : \mathrm{ran}(g') \to \mathbb{R}$  be such that  $\psi'(p) = f'(p)$  and  $\psi'$  is differentiable on the open interval between  $y$  and  $p$ , with  $\psi''(\xi) \in [1/\beta, 1/\alpha]$  for any  $\xi$  between  $y$  and  $p$ . Let  $\gamma_y := f(y) - \psi(y)$ ,  $\gamma_p := f(p) - \psi(p)$  and  $\gamma_\psi := 2\max\{|\gamma_y|, |\gamma_p|\}$ . Then we have that
$$
D _ {f} (y, p) = \psi (y) - \psi (p) - (y - p) \psi^ {\prime} (p) + (\gamma_ {y} - \gamma_ {p}) = \frac {1}{2} \psi^ {\prime \prime} (\xi) (y - p) ^ {2} + (\gamma_ {y} - \gamma_ {p})
$$
$$
D _ {f} (y, p) \in \left[ \frac {1}{2 \beta} (y - p) ^ {2} - 2 \gamma_ {\psi}, \frac {1}{2 \alpha} (y - p) ^ {2} + 2 \gamma_ {\psi} \right],
$$
for any  $\psi$  as defined above (say  $\psi \in \Psi$ ). In particular, we have
$$
D _ {f} (y, p) \leq \frac {1}{2 \alpha} (y - p) ^ {2} + 2 \inf  _ {\psi \in \Psi} \gamma_ {\psi} \text {a n d}
$$
$$
D _ {f} (y, p) \geq \frac {1}{2 \beta} (y - p) ^ {2} - 2 \inf  _ {\psi \in \Psi} \gamma_ {\psi}
$$
Since, we have only a finite number of points where the derivative is not well defined, a simple smoothening technique may give us  $\Psi$  such that  $\inf_{\psi \in \Psi} \gamma_{\psi} = 0$ .
C Proofs from Section 3
C.1 Proof of Theorem 3.2
We first define a boundedness property which we use in order to apply the results from Gopalan et al. [2023]. The property states that the activation function (the partial inverse of the link function) must either have a range that covers all possible labels, or has a range whose closure covers all possible labels and the rate with which the labels are covered as we tend to the limits of the domain is at least polynomial. For example, the sigmoid activation tends to 1 (resp. 0) exponentially fast as its argument increases (resp. decreases).
Definition C.1 (Bounded Functions). Let  $u: (0,1) \to \mathbb{R}$  be a non-decreasing function defined on the interval  $(0,1)$ . For  $R, \gamma \geq 0$ , we say that  $u$  is  $(R,\gamma)$ -bounded on  $[0,1]$  if for any  $\varepsilon > 0$ , there are  $r_0 \leq r_1 \in [0,1]$  such that if we let  $u(r_i) = \lim_{r \to r_i} u(r)$ ,  $i = 0,1$  then
$$
\max  \{- u (r _ {0}), u (r _ {1}) \} \leq R \left(\frac {1}{\varepsilon}\right) ^ {\gamma}
$$
$$
(1 - r _ {1}) (u (r) - u (r _ {1})) \leq \varepsilon \text {f o r} r \geq r _ {1} \text {a n d}
$$
$$
r _ {0} (u (r _ {0}) - u (r)) \leq \varepsilon \text {f o r} r \leq r _ {0}
$$
Proposition C.2. Let  $u:(0,1)\to (-R,R)$  be non-decreasing and continuous, then  $u$  is  $(R,0)$ -bounded.
We restate a more quantitative version of Theorem 3.2 here for convenience.
Theorem C.3 (Omnipredictors for Matching Losses, combination of results in Gopalan et al. [2023]). Let  $D$  be a distribution over  $\mathbb{R}^d \times [0,1]$  whose marginal on  $\mathbb{R}^d$  has  $\lambda$ -bounded second moments. There is an algorithm that, given sample access to  $D$ , with high probability returns a predictor  $p: \mathbb{R} \to (0,1)$  with the following guarantee. For any pair of Fenchel-Legendre dual functions  $(f,g)$  such that  $g': \mathbb{R} \to \mathbb{R}$  is continuous, non-decreasing and  $L$ -Lipschitz, and  $f'$  is  $(R,\gamma)$ -bounded (see Definition C.1),  $p$  satisfies
The algorithm requires time
$$
O \left(d ^ {3} B ^ {4} L ^ {4} \lambda^ {2} R ^ {3} \left(\frac {3}{\varepsilon}\right) ^ {3 + 3 \gamma} + d B ^ {2} L ^ {2} \lambda R ^ {4} \left(\frac {3}{\varepsilon}\right) ^ {4 + 4 \gamma} + B ^ {1 0} L ^ {1 0} \lambda^ {5} R ^ {1 2} \left(\frac {3}{\varepsilon}\right) ^ {1 2 + 1 2 \gamma} \log \left(\frac {B L \lambda R}{\varepsilon^ {1 + \gamma}}\right)\right)
$$
and sample complexity
$$
O \left(d ^ {2} B ^ {4} L ^ {4} \lambda^ {2} R ^ {3} \left(\frac {3}{\varepsilon}\right) ^ {3 + 3 \gamma} + B ^ {8} L ^ {8} \lambda^ {4} R ^ {1 0} \left(\frac {3}{\varepsilon}\right) ^ {1 0 + 1 0 \gamma} \log \left(\frac {B L \lambda R}{\varepsilon^ {1 + \gamma}}\right)\right)
$$
Proof of Theorem 3.2. The idea of the proof (given by Gopalan et al. [2023]) is that in each repetition of both the inner and the outer loop of Algorithm 1, there is a potential function which reduces by some amount that is bounded away above zero. The potential function is in fact the function  $\mathbf{E}_D[(p^*(\mathbf{x}) - p(\mathbf{x}))^2]$ , where  $p^*(\mathbf{x}) = \mathbf{E}_D[y'| \mathbf{x}]$  (for us,  $y'$  is formed by drawing a Bernoulli random variable with probability of success  $y$ , given an example  $(\mathbf{x}, y) \in \mathbb{R}^d \times [0,1]$  drawn from  $D$ ). Since  $\mathbf{E}_D[(p^*(\mathbf{x}) - p(\mathbf{x}))^2] \in [0,1]$ , the number of iterations of each of the loops has to be bounded. Moreover, after the completion of each of the inner loops, the current value of  $p$  corresponds to an approximately multiaccurate predictor and note that the algorithm terminates if and only if a discretized version  $p^\delta$  of (the multiaccurate)  $p$  is approximately calibrated. The output is then  $p^\delta$  which is close to  $p$  (and hence also approximately multiaccurate), but also approximately calibrated. We will now present a small number of technical modifications we need to make in the proof of Gopalan et al. [2023] in order to specialize it to our setting. The main difference here is that we do not consider the distribution of  $\mathbf{x}$  to have bounded norm with probability 1, but we only assume it to have bounded second moments. For the following, we assume that  $g$  is 1-Lipschitz, since we can set  $B \gets B \cdot L$  and push the Lipschitz constant in the domain of  $\mathbf{w}$ .
Suppose first that the marginal of  $D$  on  $\mathbb{R}^d$  is supported on the unit ball  $\mathbb{B}_d$  and that the labels are binary. Then, the result would follow from Theorems 7.7 and A.4 of Gopalan et al. [2023]. In particular,
Theorem 7.7 states that given access to a weak learner with the specifications of Proposition A.1, there is an efficient algorithm that computes an  $\varepsilon_{1}$ -calibrated and  $(\mathcal{C},\varepsilon_1)$ -multiaccurate predictor  $p$  where the notions of calibration and multiaccuracy originate to the literature of fairness and are defined, e.g., in Definitions 3.1 and 3.2 of Gopalan et al. [2023] and  $\mathcal{C} = \{\mathbf{x}\to \mathbf{w}\cdot \mathbf{x}\mid \| \mathbf{w}\| _2\leq B\} \cup \{\mathbf{x}\to 1\}$  (the class  $\mathcal{C}$  is bounded as long as  $\| \mathbf{x}\| _2\leq 1$  almost surely). Theorem A.4 states that for  $\varepsilon_{2} > 0$  any  $\varepsilon_{1}$ -calibrated and  $(\mathcal{C},\varepsilon_1)$ -multiaccurate predictor  $p$  minimizes simultaneously the matching loss corresponding to any pair  $(f,g)\in \mathcal{F}$  (where  $f$  is  $(R,\gamma)$ -bounded) up to error
$$
R (1 / \varepsilon_ {2}) ^ {\gamma} \varepsilon_ {1} + \varepsilon_ {1} + \varepsilon_ {2}
$$
The expression above is formed by proving that any pair  $(f,g)\in \mathcal{F}$  has the property that  $f^{\prime}$  is  $(\varepsilon_{2},R(1 / \varepsilon_{2})^{\gamma})$  -approximately optimal (as per the Definition A.1 of Gopalan et al. [2023]), for any  $\varepsilon_2 > 0$ . In particular, we would like to show that for any  $\varepsilon_{2} > 0$  there exists  $\widehat{f}^\prime$  such that the following is true for any  $r\in [0,1]$
$$
\ell_ {g} (r, \widehat {f} ^ {\prime} (r)) \leq \ell_ {g} (r, f ^ {\prime} (r)) + \varepsilon_ {2}
$$
$$
| \widehat {f} ^ {\prime} (r) | \leq R \cdot (1 / \varepsilon_ {2}) ^ {\gamma}
$$
We may pick  $\widehat{f}'$  as follows (for  $r_0 \leq r_1$  as given by Definition C.1 for  $\varepsilon \gets \varepsilon_2$ , since  $f'$  is  $(R, \gamma)$ -bounded).
$$
\widehat {f} ^ {\prime} (r) = \left\{ \begin{array}{l} f ^ {\prime} (r), \text {i f} r \in [ r _ {0}, r _ {1} ] \\ f ^ {\prime} (r _ {0}), \text {i f} r <   r _ {0} \\ f ^ {\prime} (r _ {1}), \text {i f} r > r _ {1} \end{array} \right.
$$
The desired result follows from using the expression for  $\ell_g$ , the convexity of  $g$  (since  $g'$  is non-decreasing) and the guarantees of Definition C.1. In particular, we have that  $|\widehat{f}'| \leq \max \{|f'(r_0)|, |f'(r_1)|\}$  since  $f'$  is increasing. Moreover, for  $r \in [r_0, r_1]$  we have  $\ell_g(r, \widehat{f}'(r)) = \ell_g(r, f'(r))$  and for  $r < r_0$  we have
$$
\begin{array}{l} \ell_ {g} (r, \widehat {f} ^ {\prime} (r)) = \ell_ {g} (r, f ^ {\prime} (r _ {0})) = g \left(f ^ {\prime} (r _ {0})\right) - r f ^ {\prime} (r _ {0}) \\ \leq r _ {0} \left(f ^ {\prime} \left(r _ {0}\right) - f ^ {\prime} (r)\right) + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} \left(r _ {0}\right) \quad \text {(s i n c e} g \text {i s c o n v e x)} \\ \leq r _ {0} \left(f ^ {\prime} \left(r _ {0}\right) - f ^ {\prime} (r)\right) + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} (r) \quad (\text {s i n c e} f ^ {\prime} \text {i s i n c r e a s i n g a n d} r <   r _ {0}) \\ \leq \varepsilon_ {2} + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} (r) \quad \text {(s i n c e} f ^ {\prime} \text {i s b o u n d e d a c c o r d i n g t o D e f i n i t i o n C . 1)} \\ = \ell_ {g} (r, f ^ {\prime} (r)) + \varepsilon_ {2} \\ \end{array}
$$
Similarly, for  $r > r_1$ , we have
$$
\begin{array}{l} \ell_ {g} (r, \widehat {f} ^ {\prime} (r)) = \ell_ {g} (r, f ^ {\prime} (r _ {1})) = g (f ^ {\prime} (r _ {1})) - r f ^ {\prime} (r _ {1}) \\ \leq r _ {1} \left(f ^ {\prime} \left(r _ {1}\right) - f ^ {\prime} (r)\right) + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} \left(r _ {1}\right) \quad \text {(s i n c e} g \text {i s c o n v e x)} \\ \leq \left(1 - r _ {1}\right) \left(f ^ {\prime} \left(r _ {1}\right) - f ^ {\prime} (r)\right) + g \left(f ^ {\prime} (r)\right) + (1 - r) f ^ {\prime} (r _ {1}) - f ^ {\prime} (r) \\ \leq \left(1 - r _ {1}\right) \left(f ^ {\prime} \left(r _ {1}\right) - f ^ {\prime} (r)\right) + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} (r) \text {(s i n c e} f ^ {\prime} \text {i s i n c e r a s i n g ,} 1 > r > r _ {1}) \\ \leq \varepsilon_ {2} + g \left(f ^ {\prime} (r)\right) - r f ^ {\prime} (r) \quad \text {(s i n c e} f ^ {\prime} \text {i s b o u n d e d a c c o r d i n g t o D e f i n i t i o n C . 1)} \\ = \ell_ {g} (r, f ^ {\prime} (r)) + \varepsilon_ {2} \\ \end{array}
$$
In order to acquire  $R(1 / \varepsilon_2)^\gamma \varepsilon_1 + \varepsilon_1 + \varepsilon_2\leq \varepsilon$  , we set  $\varepsilon_{2} = \varepsilon /3$  and  $\varepsilon_{1} = (\varepsilon /3)^{1 + \gamma} / R$
However, we only assume that the marginal distribution has  $\lambda$ -bounded second moments and we, therefore, need to make certain modifications to the proof of their Theorem 7.7. In particular, the boundedness assumption is used in the proofs of Lemma 7.2, Lemma 7.6 and Theorem 7.7 in Gopalan et al. [2023].