SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present our results regarding online realizable regression. The next result resolves an open question of [DG22]. It provides an online learner with optimal (off by a factor of 2) cumulative loss in realizable regression.
Theorem 4 (Optimal Cumulative Loss). Let  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  and  $\varepsilon > 0$ . Then, there exists a deterministic algorithm (Algorithm 3) whose cumulative loss in the realizable setting is bounded by  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) + \varepsilon$ . Conversely, for any  $\varepsilon > 0$ , every deterministic algorithm in the realizable setting incurs loss at least  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) / 2 - \varepsilon$ .
Algorithm 3 Scaled SOA
Parameters:  $\{\varepsilon_t\}_{t\in \mathbb{N}}$
Initialize  $V^{(1)} = \mathcal{H}$
For  $t = 1,\ldots$
1. Receive  $x_{t} \in \mathcal{X}$ .
2. For every  $y \in [0,1]$ , let  $V_{(x_t,y)}^{(t)} = \{h \in V^{(t)} : h(x_t) = y\}$ .
3. Let  $\widehat{y}_t$  be an arbitrary label such that
$$
\mathbb {D} ^ {\mathrm {o n l}} \left(V _ {(x _ {t}, \widehat {y} _ {t})} ^ {(t)}\right) \geq \sup  _ {y ^ {\prime}} \mathbb {D} ^ {\mathrm {o n l}} \left(V _ {(x _ {t}, y ^ {\prime})} ^ {(t)}\right) - \varepsilon_ {t}.
$$
4. Predict  $\widehat{y_t}$
5. Receive the true label  $y_{t}^{\star}$  and incur loss  $\ell (\widehat{y}_t,y_t^\star)$
6. Update  $V^{(t + 1)} = \{h\in V^{(t)}:h(x_{t}) = y_{t}^{\star}\}$
Proof. Let us begin with the upper bound. Assume that  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) < \infty$ . Suppose we are predicting on the  $t$ -th point in the sequence and let  $V^{(t)}$  be the version space so far, i.e.,  $V^{(t)} =$
$\{h \in \mathcal{H} : \forall \tau \in [t - 1], h(x_{\tau}) = y_{\tau}\}$ . Let  $x_{t}$  be the next point to predict on. For each label  $y \in \mathbb{R}$ , let  $V_{(x_{t},y)}^{(t)} = \{h \in V^{(t)} : h(x_{t}) = y\}$ . From the definition of the dimension  $\mathbb{D}^{\mathrm{onl}}$ , we know that for all  $y, y' \in \mathbb{R}$  such that  $V_{(x_{t},y)}^{(t)}, V_{(x_{t},y')}^{(t)} \neq \emptyset$ ,
$$
\mathbb {D} ^ {\text {o n l}} (V ^ {t}) \geq \ell (y, y ^ {\prime}) + \min  \left\{\mathbb {D} ^ {\text {o n l}} \left(V _ {(x _ {t}, y)} ^ {(t)}\right), \mathbb {D} ^ {\text {o n l}} \left(V _ {(x _ {t}, y ^ {\prime})} ^ {(t)}\right) \right\}.
$$
Let  $\widehat{y}_t$  be an arbitrary label with  $\mathbb{D}^{\mathrm{onl}}\left(V_{(x_t,\widehat{y}_t)}^{(t)}\right)\geq \sup_{y'}\mathbb{D}^{\mathrm{onl}}\left(V_{(x_t,y')}^{(t)}\right) - \varepsilon_t$ , where  $\varepsilon_t$  is some sequence shrinking arbitrarily quickly in the number of rounds  $t$ . The learner predicts  $\widehat{y}_t$ . Assume that the adversary picks  $y_t^\star$  as the true label and, so, the learner incurs loss  $\ell (\widehat{y}_t,y_t^\star)$  at round  $t$ . Then, the updated version space  $V_{(x_t,y_t^\star)}^{(t)}$  has
$$
\mathbb {D} ^ {\mathrm {o n l}} \left(V _ {(x _ {t}, y _ {t} ^ {*})} ^ {(t)}\right) \leq \sup _ {y ^ {\prime}} \mathbb {D} ^ {\mathrm {o n l}} \left(V _ {(x _ {t}, y ^ {\prime})} ^ {(t)}\right) \leq \mathbb {D} ^ {\mathrm {o n l}} \left(V _ {(x _ {t}, \widehat {y} _ {t})} ^ {(t)}\right) + \varepsilon_ {t},
$$
$$
\min  \left\{\mathbb {D} ^ {\text {o n l}} \left(V _ {(x _ {t}, \widehat {y} _ {t})} ^ {(t)}\right), \mathbb {D} ^ {\text {o n l}} \left(V _ {(x _ {t}, y _ {t} ^ {*})} ^ {(t)}\right) \right\} \geq \mathbb {D} ^ {\text {o n l}} \left(V _ {(x, y _ {t} ^ {*})} ^ {(t)}\right) - \varepsilon_ {t}.
$$
This gives that
$$
\begin{array}{l} \mathbb {D} ^ {\text {o n l}} (V ^ {(t)}) \geq \ell (\widehat {y} _ {t}, y _ {t} ^ {*}) + \min  \left\{\mathbb {D} ^ {\text {o n l}} \left(V _ {\left(x _ {t}, \widehat {y} _ {t}\right)} ^ {(t)}\right), \mathbb {D} ^ {\text {o n l}} \left(V _ {\left(x _ {t}, y _ {t} ^ {*}\right)} ^ {(t)}\right) \right\} \\ \geq \ell (\widehat {y} _ {t}, y _ {t} ^ {*}) + \mathbb {D} ^ {\text {o n l}} \left(V _ {(x _ {t}, y _ {t} ^ {*})} ^ {(t)}\right) - \varepsilon_ {t}, \\ \end{array}
$$
and, by re-arranging,
$$
\mathbb {D} ^ {\text {o n l}} \left(V _ {\left(x _ {t}, y _ {t} ^ {*}\right)} ^ {(t)}\right) \leq \mathbb {D} ^ {\text {o n l}} \left(V ^ {(t)}\right) - \ell \left(\widehat {y} _ {t}, y _ {t} ^ {*}\right) + \varepsilon_ {t}. \tag {1}
$$
So every round reduces the dimension by at least the magnitude of the loss (minus  $\varepsilon_{t}$ ). Notice that  $\mathbb{D}^{\mathrm{onl}}(V^{(t + 1)}) = \mathbb{D}^{\mathrm{onl}}\left(V_{(x_t,y_t^\star)}^{(t)}\right)$ . Thus, by choosing the  $\{\varepsilon_t\}_{t\in \mathbb{N}}$  sequence such that
$$
\sum_ {t} \varepsilon_ {t} \leq \varepsilon^ {\prime},
$$
and summing up Equation (1) over all  $t \in \mathbb{N}$ , we get a cumulative loss bound
$$
\sum_ {t} \ell (\widehat {y} _ {t}, y _ {t} ^ {\star}) \leq \mathbb {D} ^ {\mathrm {o n l}} (\mathcal {H}) + \varepsilon^ {\prime}.
$$
Hence, we see that by taking the limit as  $\varepsilon^{\prime}$  goes to 0 shows that the cumulative loss is upper bounded by  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H})$ . This analysis shows that Algorithm 3 achieves the cumulative loss bound  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) + \varepsilon^{\prime}$ , for arbitrarily small  $\varepsilon^{\prime} > 0$ .
Let us continue with the lower bound. For any  $\varepsilon > 0$ , we are going to prove that any deterministic learner must incur cumulative loss at least  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) / 2 - \varepsilon$ . By the definition of  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H})$ , for any  $\varepsilon > 0$ , there exists a tree  $T_{\varepsilon}$  such that, for every path  $\pmb{y}$ ,
$$
\sum_ {i = 1} ^ {\infty} \gamma_ {\boldsymbol {y} \leq i} \geq \mathbb {D} ^ {\mathrm {o n l}} (\mathcal {H}) - 2 \varepsilon ,
$$
i.e., the sum of the gaps across the path is at least  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) - 2\varepsilon$ . The strategy of the adversary is the following: in the first round, she presents the learner with the instance  $x_{1} = x_{\emptyset}$ . Then, no matter what label  $\hat{y}_{1}$  the learner picks, the adversary can choose the label  $y_{1}^{\star}$  so that  $|\hat{y}_1 - y_1^\star |\geq \gamma_\emptyset /2$ . The adversary can keep picking the instances  $x_{t}$  based on the induced path of the choices of the true labels  $\{y_{\tau}^{\star}\}_{\tau < t}$  and the loss of the learner in every round  $t$  is at least  $\gamma_{\pmb {y}\leq t} / 2$ . Thus, summing up over all the rounds as  $T\to \infty$ , we see that the total loss of the learner is at least
$$
\frac {\mathbb {D} ^ {\mathrm {o n l}} (\mathcal {H})}{2} - \varepsilon .
$$
Remark 2 (Randomized Online Learners). We highlight that, unlike the setting of realizable online classification, in the case of realizable online regression randomization does not seem to help the learner (see also [FHMM23]). In particular, the lower bound of  $\frac{\mathbb{D}^{\mathrm{onl}}(\mathcal{H})}{2} - \varepsilon$  holds even for randomized learners. To see it, notice that for all distributions over  $\mathcal{D}$  over  $[0,1]$  it holds that
$$
\max  _ {c _ {1}, c _ {2}} \left\{\mathbf {E} _ {X \sim \mathcal {D}} [ \ell (X, c _ {1}) ], \mathbf {E} _ {X \sim \mathcal {D}} [ \ell (X, c _ {2}) ] \right\} \geq \ell (c _ {1}, c _ {2}) / 2.
$$
Example 2 (Sequential Complexity Measures). Sequential fat-shattering dimension and sequential covering numbers are two standard combinatorial measures for regression in online settings [RST15b, RST15a]. Note that Example 1 can be learned with 1 sample even in the online realizable setting. Hence, Example 1 shows that sequential fat-shattering dimension fails to characterize online realizable regression (since this dimension is at least as large as fat-shattering dimension which is infinite in this example). Moreover, we know that sequential covering numbers and sequential fat-shattering dimension are of the same order of magnitude and so they are also infinite in the case of Example 1.
F Dimension and Finite Character Property
$\left[\mathrm{BDHM}^{+}19\right]$  gave a formal definition of the notion of "dimension" or "complexity measure", that all previously proposed dimensions in statistical learning theory comply with. In addition to characterizing learnability, a dimension should satisfy the finite character property:
Definition 17 (Finite Character  $[\mathrm{BDHM}^+19]$ ). A dimension characterizing learnability can be abstracted as a function  $F$  that maps a class  $\mathcal{H}$  to  $\mathbb{N} \cup \{\infty\}$  and satisfies the finite character property: for every  $d \in \mathbb{N}$  and  $\mathcal{H}$ , the statement " $F(\mathcal{H}) \geq d$ " can be demonstrated by a finite set  $X \subseteq \mathcal{X}$  of domain points, and a finite set of hypotheses  $H \subseteq \mathcal{H}$ . That is, " $F(\mathcal{H}) \geq d$ " is equivalent to the existence of a bounded first order formula  $\phi(\mathcal{X}, \mathcal{H})$  in which all the quantifiers are of the form:  $\exists x \in \mathcal{X}, \forall x \in \mathcal{X}$  or  $\exists h \in \mathcal{H}, \forall h \in \mathcal{H}$ .
Claim 1. The scaled one-inclusion graph dimension  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  satisfies the finite character property.
Proof. To demonstrate that  $\mathbb{D}^{\mathrm{OIG}}(\mathcal{H})\geq d$ , it suffices to find a set  $S$  of  $n$  domain points and present a finite subgraph  $G = (V,E)$  of the one-inclusion hypergraph induced by  $S$  where every orientation  $\sigma :E\to V$  has out-degree at least  $n / 3$ . Note that  $V$  is, by definition, a finite collection of datasets realizable by  $\mathcal{H}$  and so this means that we can demonstrate that  $\mathbb{D}^{\mathrm{OIG}}(\mathcal{H})\geq d$  with a finite set of domain points and a finite set of hypotheses.
G Examples for Scaled Graph Dimension
These examples adaptations from [DSS14, DSBDSS15].
Example 3 (Large Gap Between ERM Learners). For every  $d \in \mathbb{N}$ , consider a domain  $\mathcal{X}_d$  such that  $|\mathcal{X}_d| = d$  and  $\mathcal{X}_d, \mathcal{X}_{d'}$  are disjoint for  $d \neq d'$ . For all  $d \in \mathbb{N}$ , let  $P(\mathcal{X}_d)$  denote the collection of all finite and co-finite subsets of  $\mathcal{X}_d$ . Let us fix  $\gamma \in (0,1)$ . Consider a mapping  $f: \cup_{d \in \mathbb{N}} P(\mathcal{X}_d) \to [0,1]$  such that  $f(A_d) \in (\gamma,1)$  for all  $d \in \mathbb{N}$ ,  $A_d \in P(\mathcal{X}_d)$ , and  $f(A_d) \neq f(A_{d'}')$  for all  $A_d \neq A_{d'}'$ ,  $A_d \in P(\mathcal{X}_d)$ ,  $A_{d'}' \in P(\mathcal{X}_{d'})$ . Such a mapping exists due to the density of the reals. For any  $d \in \mathbb{N}$ ,  $A_d \subseteq \mathcal{X}_d$ , let  $h_{A_d}(x) = f(A_d) \cdot \mathbb{1}\{x \in A_d\}$  and consider the scaled first Cantor class  $\mathcal{H}_{\mathcal{X}_d,\gamma} = \{h_{A_d}: A_d \in P(\mathcal{X}_d)\}$ . We claim that  $\mathbb{D}_{\gamma}^{\mathrm{Nat}}(\mathcal{H}_{\mathcal{X}_d,\gamma}) = 1$  and that  $\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H}_{\mathcal{X}_d,\gamma}) = |\mathcal{X}_d| = d$  since one can use  $f_{\emptyset}$  for the  $\gamma$ -graph shattering. Consider the following two ERM learners for the scaled first Cantor class  $\mathcal{H}_{\mathcal{X}_d,\gamma}$ :
1. Whenever a sample of the form  $S = \{(x_i,0)\}_{i\in [n]}$  is observed, the first algorithm outputs  $h_{\cup \{x_i\}_{i\in [n]}^c}$  which minimizes the empirical error. If the sample contains a non-zero element, the ERM learner identifies the correct hypothesis. The sample complexity of PAC learning is  $\Omega (d)$ .
2. The second algorithm either returns the all-zero function or identifies the correct hypothesis if the sample contains a non-zero label. This is a good ERM learner  $\mathcal{A}_{\text{good}}^{\text{ERM}}$  with sample complexity  $m(\varepsilon, \delta) = \frac{1}{\varepsilon} \log \left( \frac{1}{\delta} \right)$ .
$^6$  A set  $S \subseteq \mathcal{X}_d$  is co-finite if its complement  $S^c$  is finite.
The construction that illustrates the poor performance of the first learner is exactly the same as in the proof of the lower bound of Theorem 1. The second part of the example is formally shown in Claim 2, which follows.
Claim 2 (Good ERM Learner). Let  $\varepsilon, \delta \in (0,1)^2$ . Then, the good ERM learner of Example 3 has sample complexity  $\mathcal{M}(\varepsilon, \delta) = \frac{1}{\varepsilon} \log \left(\frac{1}{\delta}\right)$ .
Proof. Let  $d \in \mathbb{N}$ ,  $\mathcal{D}_{\mathcal{X}_d}$  be a distribution over  $\mathcal{X}_d$  and  $h_{A_d^*}$  be the labeling function. Consider a sample  $S$  of length  $m$ . If the learner observes a value that is different from 0 among the labels in  $S$ , then it will be able to infer  $h_{A_d^*}$  and incur 0 error. On the other hand, if the learner returns the all zero function its error can be bounded as
$$
\mathbf {E} _ {x \sim \mathcal {D} _ {\mathcal {X} _ {d}}} \left[ \ell (h _ {\emptyset} (x), h _ {A _ {d} ^ {\star}} (x)) \right] \leq \mathbf {P r} _ {x \sim \mathcal {D} _ {\mathcal {X} _ {d}}} \left[ x \in A _ {d} ^ {\star} \right].
$$
Since in all the  $m = \frac{1}{\varepsilon}\log \left(\frac{1}{\delta}\right)$  draws of the training set  $S$  there were no elements from  $A_{d}^{\star}$  we can see that, with probability at least  $1 - \delta$  over the draws of  $S$  it holds that
$$
\operatorname * {P r} _ {x \sim \mathcal {D} _ {\mathcal {X} _ {d}}} \left[ x \in A _ {d} ^ {\star} \right] \leq \varepsilon .
$$
Thus, the algorithm satisfies the desired guarantees.
The next example shows that no proper algorithm can be optimal in the realizable regression setting.
Example 4 (No Optimal PAC Learner Can be Proper). Let  $\mathcal{X}_d$  contain  $d$  elements and let  $\gamma \in (0,1)$ . Consider the subclass of the scaled first Cantor class (see Example 3) with  $\mathcal{H}_{d,\gamma}^{\prime} = \{h_A:A\in P(\mathcal{X}_d),|A| = \lfloor d / 2\rfloor \}$ . First, since this class is contained in the scaled first Cantor class, we can employ the good ERM and learn it. However, this learner is improper since  $h_\emptyset \notin \mathcal{H}_{d,\gamma}^{\prime}$ . Then, no proper algorithm is able to PAC learn  $\mathcal{H}_{d,\gamma}^{\prime}$  using  $o(d)$  examples.
Proof. Suppose that an adversary chooses  $h_A \in \mathcal{H}_{d,\gamma}'$  uniformly at random and consider the distribution on  $\mathcal{X}_d$  which is uniform on the complement of  $A$ , where  $|A| = O(d)$ . Note that the error of every hypothesis  $h_B \in \mathcal{H}_{d,\gamma}'$  is at least  $\gamma |B \setminus A| / d$ . Therefore, to return a hypothesis with small error, the algorithm must recover a set that is almost disjoint from  $A$  and so recover  $A$ . However the size of  $A$  implies that it cannot be done with  $o(d)$  examples.
Formally, fix  $x_0 \in \mathcal{X}_d$  and  $\varepsilon \in (0,1)$ . Let  $A \subseteq \mathcal{X}_d \setminus \{x_0\}$  of size  $d / 2$ . Let  $\mathcal{D}_A$  be a distribution with mass  $\mathcal{D}_A((x_0, h_A(x_0))) = 1 - 16\varepsilon$  and is uniform on the points  $\{(x, h_A(x)): x \in A^c\}$ , where  $A^c$  is the complement of  $A$  (without  $x_0$ ).
Consider a proper learning algorithm  $\mathcal{A}$ . We will show that there is some algorithm-dependent set  $A$ , so that when  $\mathcal{A}$  is run on  $\mathcal{D}_A$  with  $m = O(d / \varepsilon)$ , it outputs a hypothesis with error at least  $\gamma$  with constant probability.
Pick  $A$  uniformly at random from all sets of size  $d/2$  of  $\mathcal{X}_d \setminus \{x_0\}$ . Let  $Z$  be the random variable that counts the number of samples in the  $m$  draws from  $\mathcal{D}_A$  that are not  $(x_0, h_A(x_0))$ . Standard concentration bounds imply that with probability at least  $1/2$ , the number of points from  $(\mathcal{X}_d \setminus \{x_0\}) \setminus A$  is at most  $d/4$ . Conditioning on this event,  $A$  is a uniformly chosen random set of size  $d/2$  that is chosen uniformly from all subsets of a set  $\mathcal{X}' \subset \mathcal{X}_d$  with  $|\mathcal{X}'| \geq 3d/4$  (these points are not present in the sample). Now assume that the learner returns a hypothesis  $h_B$ , where  $B$  is a subset of size  $d/2$ . Note that  $\mathbf{E}[|B \setminus A|] \geq d/6$ . Hence there exists a set  $A$  such that with probability  $1/2$ , it holds that  $|B \setminus A| \geq d/6$ . This means that  $A$  incurs a loss of at least  $\gamma$  on all points in  $B \setminus A$  and the mass of each such point is  $\Omega(\varepsilon/d)$ . Hence, in total, the learner will incur a loss of order  $\gamma \cdot \varepsilon$ .
H Extension to More General Loss Functions
Our results can be extended to loss functions that satisfy approximate pseudo-metric axioms (see e.g., [HKLM22, CKW08]). The main difference from metric losses is that we allow an approximate triangle inequality instead of a strict inequality. Many natural loss functions are captured by this definition, such as the well-studied  $\ell_p$  losses for the regression setting. Abstractly, in this context, the
label space is an abstract non-empty set  $\mathcal{V}$ , equipped with a general loss function  $\ell : \mathcal{V}^2 \to \mathbb{R}_{\geq 0}$  satisfying the following property.
Definition 18 (Approximate Pseudo-Metric). For  $c \geq 1$ , a loss function  $\ell: \mathcal{Y}^2 \to \mathbb{R}_{\geq 0}$  is  $c$ -approximate pseudo-metric if (i)  $\ell(x, x) = 0$  for any  $x \in \mathcal{V}$ , (ii)  $\ell(x, y) = \ell(y, x)$  for any  $x, y \in \mathcal{V}$ , and, (iii)  $\ell$  satisfies a  $c$ -approximate triangle inequality; for any  $y_1, y_2, y_3 \in \mathcal{V}$ , it holds that  $\ell(y_1, y_2) \leq c(\ell(y_1, y_3) + \ell(y_2, y_3))$ .
Furthermore, note that all dimensions for  $\mathcal{H}$ ,  $\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H})$ ,  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$ ,  $\mathbb{D}_{\gamma}^{\mathrm{DS}}(\mathcal{H})$ , and  $\mathbb{D}_{\gamma}^{\mathrm{onl}}(\mathcal{H})$  are defined for loss functions satisfying Definition 18.
Next, we provide extensions of our main results for approximate pseudo-metric losses and provide proof sketches for the extensions.
ERM Learnability for Approximate Pseudo-Metrics. For ERM learnability and losses satisfying Definition 18, we can obtain the next result.
Theorem 5. Let  $\ell$  be a loss function satisfying Definition 18. Then for every class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ ,  $\mathcal{H}$  is learnable by any ERM in the realizable PAC regression setting under  $\ell$  if and only if  $\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H}) < \infty$  for all  $\gamma \in (0,1)$ .
The proof of the upper bound and the lower bound follow in the exact same way as with the absolute loss.
PAC Learnability for Approximate Pseudo-Metrics. As for PAC learning with approximate pseudo-metric losses, we can derive the next statement.
Theorem 6. Let  $\ell$  be a loss function satisfying Definition 18. Then every class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  is PAC learnable in the realizable PAC regression setting under  $\ell$  if and only if  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H}) < \infty$  for any  $\gamma \in (0,1)$ .
Proof Sketch. We can generalize the upper bound in Theorem 2 for the scaled OIG dimension as follows. One of the ingredients of the proof for the absolute loss is to construct a sample compression scheme through the median boosting algorithm (cf. Algorithm 2). While the multiplicative update rule is defined for any loss function, the median aggregation is no longer the right aggregation for arbitrary (approximate) pseudo-metrics. However, for each such loss function, there exists an aggregation such that the output value of the ensemble is within some cutoff value from the true label for each example in the training set, which means that we have a sample compression scheme for some cutoff loss. In particular, we show that by using weak learners with cutoff parameter  $\gamma/(2c)$ , where  $c$  is the approximation level of the triangle inequality, the aggregation of the base learners can be expressed as a sample compression scheme for cutoff loss with parameter  $\gamma$ .
Indeed, running the boosting algorithm with  $(\gamma /(2c),1 / 6)$ -weak learners yields a set  $h_1,\ldots ,h_N$  of weak predictors, with the property that for each training example  $(x,y)$ , at least  $2 / 3$  of the functions  $h_i$  (as weighted by coefficients  $\alpha_{i}$ ),  $1\leq i\leq N$ , satisfy  $\ell (h_i(x),y)\leq \gamma /(2c)$ . For any  $x$ , let  $\hat{h} (x)$  be a value in  $\mathcal{V}$  such that at least  $2 / 3$  of  $h_i$  (as weighted by  $\alpha_{i}$ ),  $1\leq i\leq N$ , satisfy  $\ell (h_i(x),\hat{h} (x))\leq \gamma /(2c)$ , if such a value exists, and otherwise  $\hat{h} (x)$  is an arbitrary value in  $\mathcal{V}$ . In particular, note that on the training examples  $(x,y)$ , the label  $y$  satisfies this property, and hence  $\hat{h} (x)$  is defined by the first case. Thus, for any training example, there exists  $h_i$  (indeed, at least  $2 / 3$  of them) such that both  $\ell (h_i(x),y)\leq \gamma /(2c)$  and  $\ell (h_i(x),\hat{h} (x))\leq \gamma /(2c)$  are satisfied, and therefore we have
$$
\ell (\hat {h} (x), y) \leq c (\ell (\hat {h} (x), h _ {i} (x)) + \ell (h _ {i} (x), y)) \leq \gamma .
$$
This function  $\hat{h}$  can be expressed as a sample compression scheme of size  $O\left(\mathbb{D}_{\gamma/(2c)}^{\mathrm{OIG}}(\mathcal{H}) \log(m)\right)$  for cutoff loss with parameter  $\gamma$ : namely, it is purely defined by the  $h_i$  functions, where each  $h_i$  is
<sup>7</sup>We would like to mention that, in general, we do not require that the label space is bounded. In contrast, we have to assume that the loss function takes values in a bounded space. This is actually necessary since having an unbounded loss in the regression task would potentially make the learning task impossible. For instance, having some fixed accuracy goal, one could construct a learning instance (distribution over labeled examples) that would make estimation with that level of accuracy trivially impossible.
specified by  $O\left(\mathbb{D}_{\gamma/(2c)}^{\mathrm{OIG}}(\mathcal{H})\right)$  training examples, and we have  $N = O(\log(m))$  such functions, and  $\hat{h}$  satisfies  $\ell(\hat{h}(x), y) \leq \gamma$  for all  $m$  training examples  $(x, y)$ . Thus, by standard generalization bounds for sample compression, we get an upper bound that scales with  $\tilde{O}\left(\mathbb{D}_{\gamma/(2c)}^{\mathrm{OIG}}(\mathcal{H})\frac{1}{m}\right)$  for the cutoff loss with parameter  $\gamma$ , and hence by Markov's inequality, an upper bound
$$
\mathbf {E} [ \ell (\hat {h} (x), y) ] = \tilde {O} \bigg (\mathbb {D} _ {\gamma / (2 c)} ^ {\mathrm {O I G}} (\mathcal {H}) \frac {1}{m \gamma} \bigg).
$$
We next deal with the lower bound. For the absolute loss, we scale the dimension by  $2\gamma$  instead of  $\gamma$  since for any two possible labels  $y_{1}, y_{2}$  the learner can predict some intermediate point, and we want to make sure that the prediction will be either  $\gamma$  far from  $y_{1}$  or  $y_{2}$ . For an approximate pseudo-metric, we should take instead  $2c\gamma$  in order to ensure that the prediction is  $\gamma$  far, which means that the lower bounds in Theorem 2 hold with a scale of  $2c\gamma$ .
Online Learnability for Approximate Pseudo-Metrics. Finally, we present the more general statement for online learning.
Theorem 7. Let  $\ell$  be a loss function satisfying Definition 18 with parameter  $c\geq 1$ . Let  $\mathcal{H}\subseteq \mathcal{V}^{\mathcal{X}}$  and  $\varepsilon >0$ . Then, there exists a deterministic algorithm whose cumulative loss in the realizable setting is bounded by  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) + \varepsilon$ . Conversely, for any  $\varepsilon >0$ , every deterministic algorithm in the realizable setting incurs loss at least  $\mathbb{D}^{\mathrm{onl}}(\mathcal{H}) / (2c) - \varepsilon$ .