SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 From orientation  $\sigma$  to learner  $\mathbb{A}_{\sigma}$
Input: An  $\mathcal{H}$ -realizable sample  $\{(x_i, y_i)\}_{i=1}^{n-1}$  and a test point  $x \in \mathcal{X}, \gamma \in (0,1)$ .
Output: A prediction  $\mathbb{A}_{\sigma}(x)$
1. Create the one-inclusion graph  $G_{\mathcal{H}|_{(x_1,\ldots ,x_{n - 1},x)}}^{\mathrm{OIG}}$ .
2. Consider the edge in direction  $n$  defined by the realizable sample  $\{(x_i, y_i)\}_{i=1}^{n-1}$ ; let
$$
e = \left\{h \in \mathcal {H} \mid_ {\left(x _ {1}, \dots , x _ {n - 1}, x\right)}: \forall i \in [ n - 1 ] h (i) = y _ {i} \right\}.
$$
3. Return  $\mathbb{A}_{\sigma}(x) = \sigma (e)(n)$
Proof. By the classical leave-one-out argument, we have that
$$
\mathop{\bf E}_{S\sim \mathcal{D}^{n - 1}_{\mathcal{X}}}\left[ \begin{array}{c}\mathbf{Pr}_{x\sim \mathcal{D}_{\mathcal{X}}}[\ell (\mathbb{A}_{\sigma}(x),h^{\star}(x)) > \gamma ] \end{array} \right] = \\ \mathop{\bf E}_{(S,(x,y))\sim \mathcal{D}^{n}}[\mathbb{I}\{\ell (h_{S}(x),y) > \gamma \} ] = \mathop{\bf E}_{S^{\prime}\sim \mathcal{D}^{n},I\sim \mathbb{U}([n])}[\mathbb{I}\{\ell (h_{S_{-I}^{\prime}}(x_{I}^{\prime}),y_{I}^{\prime}) > \gamma \} ],
$$
where  $h_S$  is the predictor  $\mathbb{A}_{\sigma}$  using the examples  $S$ , and  $\mathbb{U}([n])$  is the uniform distribution on  $\{1, \ldots, n\}$ . Now for every fixed  $S'$  we have that
$$
\underset {I \sim \mathbb {U} ([ n ])} {\mathbf {E}} \left[ \mathbb {I} \left\{\ell \left(h _ {S ^ {\prime} - I} \left(x _ {I} ^ {\prime}\right), y _ {I} ^ {\prime}\right) > \gamma \right\} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {I} \left\{\ell \left(\sigma \left(e _ {i}\right) (i), y _ {i} ^ {\prime}\right) > \gamma \right\} = \frac {\operatorname {o u t d e g} \left(y ^ {\prime} ; \sigma , \gamma\right)}{n},
$$
where  $y^\prime$  is the node of the scaled OIG that corresponds to the true labeling of  $S^{\prime}$ . By taking expectation over  $S^{\prime}\sim \mathcal{D}^{n}$  we get that
$$
\mathbf {E} _ {S ^ {\prime} \sim \mathcal {D} ^ {n}, I \sim \mathbb {U} ([ n ])} [ \mathbb {I} \{\ell (h _ {S _ {- I} ^ {\prime}} (x _ {I} ^ {\prime}), y _ {I} ^ {\prime}) > \gamma \} ] \leq \mathbf {E} _ {S ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ \frac {\operatorname {o u t d e g} (y ^ {\prime} ; \sigma , \gamma)}{n} \right] \leq \frac {\max  _ {v \in V _ {n}} \operatorname {o u t d e g} (v ; \sigma , \gamma)}{n}.
$$
Equipped with the previous result, we are now ready to show that when the learner gets at least  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  samples as its training set, then its expected  $\gamma$ -cutoff loss is bounded away from  $1/2$ .
Lemma 8 (Scaled OIG Guarantee (Weak Learner)). Let  $\mathcal{D}_{\mathcal{X}}$  be a distribution over  $\mathcal{X}$  and  $h^{\star} \in \mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ , and  $\gamma \in (0,1)$ . Then, for all  $n > \mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  there exists an orientation  $\sigma^{\star}$  such that for the prediction error of the one-inclusion graph algorithm  $\mathbb{A}_{\sigma^{\star}}^{\mathrm{OIG}}: (\mathcal{X} \times [0,1])^{n-1} \times \mathcal{X} \to [0,1]$ , it holds that
$$
\mathop{\sum \nolimits E}_{S\sim \mathcal{D}^{n - 1}_{\mathcal{X}}}\left[ \begin{array}{c}\mathbf{Pr}_{x\sim \mathcal{D}_{\mathcal{X}}}\left[\ell (\mathbb{A}^{\mathrm{OIG}}_{\sigma^{\star}}(x),h^{\star}(x)) > \gamma \right] \end{array} \right]\leq 1 / 3.
$$
Proof. Fix  $\gamma \in (0,1)$ . Assume that  $n > \mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  and let  $G_{\mathcal{H}|(S,x)}^{\mathrm{OIG}} = (V_n,E_n)$  be the possibly infinite scaled one-inclusion graph. By the definition of the  $\gamma$ -OIG dimension (see Definition 9), for every finite subgraph  $G = (V,E)$  of  $G_{\mathcal{H}|(S,x)}^{\mathrm{OIG}}$  there exists an orientation  $\sigma : E \to V$  such that for every vertex in  $G$  the out-degree is at most  $n/3$ , i.e.,
$$
\forall S \in \mathcal {X} ^ {n}, \forall \text {f i n i t e} G = (V, E) \text {o f} G _ {\mathcal {H} _ {| s}} ^ {\mathrm {O I G}}, \exists \text {o r i e n t a t i o n} \sigma_ {E} \text {s . t .} \forall v \in V, \text {i t h o l d s o u t d e g} (v; \sigma_ {E}, \gamma) \leq n / 3.
$$
First, we need to create an orientation of the whole (potentially infinite) one-inclusion graph.
We will create this orientation using the compactness theorem of first-order logic which states that a set of formulas  $\Phi$  is satisfiable if and only if it is finitely satisfiable, i.e., every finite subset  $\Phi' \subseteq \Phi$  is satisfiable. Let  $G_{\mathcal{H}|_S}^{\mathrm{OIG}} = (V_n, E_n)$  be the (potentially infinite) one-inclusion graph of  $\mathcal{H}|_S$ . Let  $\mathcal{Z}$  be the set of pairs  $z = (v, e) \in V_n \times E_n$  so that  $v \in e$ . Our goal is to assign binary values to each  $z \in \mathcal{Z}$ . We define the following sets of formulas:
- For each  $e \in E_n$  we let  $\Phi_e \coloneqq \exists$  exactly one  $v \in e : z(v, e) = 1$ .
- For each  $v \in V_n$  we let  $\Phi_v := \exists$  at most  $n/3$  different  $e_{i,f} \in E_n : v \in e_{i,f} \land (\exists v' \in e_{i,f} : (z(v',e) = 1 \land \ell(v_i',v_i) > \gamma))$
It is not hard to see that each  $\Phi_e, \Phi_v$  can be expressed in first-order logic. Then, we define
$$
\Phi := \left(\bigcap_ {e \in E _ {n}} \Phi_ {e}\right) \cap \left(\bigcap_ {v \in V _ {n}} \Phi_ {v}\right).
$$
Notice that an orientation of the edges of  $G_{\mathcal{H}|s}^{\mathrm{OIG}}$  is equivalent to picking an assignment of the elements of  $\mathcal{Z}$  that satisfies all the  $\Phi_e$ . Moreover, notice that for such an assignment, if all the  $\Phi_v$  are satisfied then then maximum  $\gamma$ -scaled out-degree of  $G_{\mathcal{H}|s}^{\mathrm{OIG}}$  is at most  $n/3$ .
We will now show that  $\Phi$  is finitely satisfiable. Let  $\Phi'$  be a finite subset of  $\Phi$  and let  $E' \subseteq E_n$ ,  $V' \subseteq V_n$ , be the set of edges, vertices that appear in  $\Phi'$ , respectively. If  $V' = \emptyset$ , then we can orient the edges in  $E'$  arbitrarily and satisfy  $\Phi'$ . Similarly, if  $E' = \emptyset$  we can let all the  $z(e,v) = 0$  and satisfy all the  $\Phi_v, v \in V'$ . Thus, assume that both sets are non-empty. Consider the finite subgraph of  $G_{\mathcal{H}|_S}^{\mathrm{OIG}}$  that is induced by  $V'$  and let  $E''$  be the set of edges of this subgraph. For every edge  $e \in E' \setminus E''^4$ , pick an arbitrary orientation, i.e., for exactly one  $v \in e$  set  $z(e,v) = 1$  and for the remaining  $v' \in e$  set  $z(e,v') = 0$ . By the definition of  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  there is an orientation  $\sigma_{E''}$  of the edges in  $E''$  such that  $\forall v \in V'$  outdeg  $(v; \sigma_{E''}, \gamma) \leq n/3$ . For every  $e \in E''$  pick the assignment of all the  $z(v,e), v \in e$ , according to the orientation  $\sigma_{E''}$ . Thus, because of the maximum out-degree property of  $\sigma_{E''}$  we described before, we can also see that all the  $\Phi_v, v \in V'$ , are satisfied. Hence, we have shown that  $\Phi$  is finitely satisfiable, so it is satisfiable. This assignment on  $z(v,e)$  induces an orientation  $\sigma^\star$  under which all the vertices of the one-inclusion graph have out-degree at most  $n/3$ .
We will next use the orientation  $\sigma^{\star}$  of  $G_{\mathcal{H}|s}^{\mathrm{OIG}} = (V_n,E_n)$  to design a learner  $\mathbb{A}_{\sigma^{\star}}^{\mathrm{OIG}}:(\mathcal{X}\times [0,1])^{n - 1}\times$ $\mathcal{X}\to [0,1]$ , invoking Lemma 7. In particular, we get that, from Lemma 7 with the chosen orientation,
$$
\mathbf {E} _ {S \sim \mathcal {D} _ {\mathcal {X}} ^ {n - 1}} \left[ \mathbf {P r} _ {x \sim \mathcal {D} _ {\mathcal {X}}} \left[ \ell (\mathbb {A} _ {\sigma^ {\star}} ^ {\mathrm {O I G}} (x), h ^ {\star} (x)) > \gamma \right] \right] \leq \frac {\max  _ {v \in V _ {n}} \operatorname {o u t d e g} (v ; \sigma^ {\star} , \gamma)}{n} \leq 1 / 3,
$$
C.2.2 Boosting Real-Valued Functions
Definition 15 (Weak Real-Valued Learner). Let  $\ell$  be a loss function. Let  $\gamma \in [0,1]$ ,  $\beta \in (0,\frac{1}{2})$ , and  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ . For a distribution  $\mathcal{D}_{\mathcal{X}}$  over  $\mathcal{X}$  and true target function  $h^{\star} \in \mathcal{H}$ , we say that  $f: \mathcal{X} \to [0,1]$  is  $(\gamma, \beta)$ -weak learner with respect to  $\mathcal{D}_{\mathcal{X}}$  and  $h^{\star}$ , if
$$
\mathbf {P r} _ {x \sim \mathcal {D} _ {\mathcal {X}}} \left[ \ell (f (x), h ^ {\star} (x)) > \gamma \right] <   \frac {1}{2} - \beta .
$$
<sup>4</sup>Since the edges in  $E''$  are of finite length, we first need to map them to the appropriate edges in  $E'$ .
Following [HKS19], we define the weighted median as
$$
\operatorname {M e d i a n} (y _ {1}, \ldots , y _ {T}; \alpha_ {1}, \ldots , \alpha_ {T}) = \min  \left\{y _ {j}: \frac {\sum_ {t = 1} ^ {T} \alpha_ {t} \mathbb {I} [ y _ {j} <   y _ {t} ]}{\sum_ {t = 1} ^ {T} \alpha_ {t}} <   \frac {1}{2} \right\},
$$
and the weighted quantiles, for  $\theta \in [0,1 / 2]$ , as
$$
Q _ {\theta} ^ {+} \left(y _ {1}, \dots , y _ {T}; \alpha_ {1}, \dots , \alpha_ {T}\right) = \min  \left\{y _ {j}: \frac {\sum_ {t = 1} ^ {T} \alpha_ {t} \mathbb {I} \left[ y _ {j} <   y _ {t} \right]}{\sum_ {t = 1} ^ {T} \alpha_ {t}} <   \frac {1}{2} - \theta \right\}
$$
$$
Q _ {\theta} ^ {-} (y _ {1}, \ldots , y _ {T}; \alpha_ {1}, \ldots , \alpha_ {T}) = \max  \left\{y _ {j}: \frac {\sum_ {t = 1} ^ {T} \alpha_ {t} \mathbb {I} [ y _ {j} > y _ {t} ]}{\sum_ {t = 1} ^ {T} \alpha_ {t}} <   \frac {1}{2} - \theta \right\},
$$
and we let  $Q_{\theta}^{+}(x) = Q_{\theta}^{+}(h_1(x), \ldots, h_T(x); \alpha_1, \ldots, \alpha_T), Q_{\theta}^{-}(x) = Q_{\theta}^{-}(h_1(x), \ldots, h_T(x); \alpha_1, \ldots, \alpha_T)$ , where  $h_1, \ldots, h_T, \alpha_1, \ldots, \alpha_T$  are the values returned by Algorithm 2. The following guarantee holds for this procedure.
Lemma 9 (MedBoost guarantee [Keg03]). Let  $\ell$  be the absolute loss and  $S = \{(x_i, y_i)\}_{i=1}^m$ ,  $T = O\left(\frac{1}{\theta^2} \log(m)\right)$ . Let  $h_1, \ldots, h_T$  and  $\alpha_1, \ldots, \alpha_T$  be the functions and coefficients returned from MedBoost. For any  $i \in \{1, \ldots, m\}$  it holds that
$$
\left. \right. \max  \left\{\ell \left(Q _ {\theta / 2} ^ {+} (x _ {i}), y _ {i}\right), \ell \left(Q _ {\theta / 2} ^ {-} (x _ {i}), y _ {i}\right)\right\} \leq \gamma .
$$
Algorithm 2 MedBoost [Keg03]
Input:  $S = \{(x_{i},y_{i})\}_{i = 1}^{m}$
Parameters:  $\gamma, \beta, T$ .
Initialize  $\mathcal{P}_1 = \mathrm{Uniform}(S)$
1. Find a  $(\gamma, \beta)$ -weak learner  $h_t$  with respect to  $(x_i, y_i) \sim \mathcal{P}_t$ , using a subset  $S_t \subseteq S$ .
2. For  $i = 1, \dots, m$ :
(a) Set  $w_{i}^{(t)} = 1 - 2\mathbb{I}\big[\ell (h_{t}(x_{i}),y_{i}) > \gamma \big]$
(b) Set  $\alpha_{t} = \frac{1}{2}\log \left(\frac{(1 - \gamma)\sum_{i = 1}^{n}\mathcal{P}_{t}(x_{i},y_{i})\mathbb{I}\left[w_{i}^{(t)} = 1\right]}{(1 + \gamma)\sum_{i = 1}^{n}\mathcal{P}_{t}(x_{i},y_{i})\mathbb{I}\left[w_{i}^{(t)} = -1\right]}\right)$ .
(c) If  $\alpha_{t} = \infty$ : return  $T$  copies of  $h_t$ ,  $(\alpha_1 = 1, \dots, \alpha_T = 1)$ , and  $S_t$ .
- Else:  $P_{t + 1}(x_i,y_i) = P_t(x_i,y_i)\exp (-\alpha_tw_i^t) / Z_t,\quad \mathrm{where}\quad Z_t$
Output: Functions  $h_1, \ldots, h_T$ , coefficients  $\alpha_1, \ldots, \alpha_T$  and sets  $S_1, \ldots, S_T$ .
C.2.3 Generalization via Sample Compression Schemes
Sample compression scheme is a classic technique for proving generalization bounds, introduced by [LW86, FW95]. These bounds proved to be useful in numerous learning settings, such as binary classification [GHST05, MY16, BHMZ20], multiclass classification [DSBDSS15, DSS14, DMY16,  $\mathrm{BCD^{+}22}$ ], regression [HKS18, HKS19], active learning [WHEY15], density estimation  $[\mathrm{ABDH}^+ 20]$ , adversarially robust learning [MHS19, MHS20, MHS21, MHS22, AHM22, AH22], learning with partial concepts [AHHM22], and showing Bayes-consistency for nearest-neighbor methods [GKN14, KSW17]. As a matter of fact, compressibility and learnability are known to be equivalent for general learning problems [DMY16]. Another remarkable result by [MY16] showed that VC classes enjoy a sample compression that is independent of the sample size.
We start with a formal definition of a sample compression scheme.
Definition 16 (Sample compression scheme). A pair of functions  $(\kappa, \rho)$  is a sample compression scheme of size  $\ell$  for class  $\mathcal{H}$  if for any  $n \in \mathbb{N}$ ,  $h \in \mathcal{H}$  and sample  $S = \{(x_i, h(x_i))\}_{i=1}^n$ , it holds for the compression function that  $\kappa(S) \subseteq S$  and  $|\kappa(S)| \leq \ell$ , and the reconstruction function  $\rho(\kappa(S)) = \hat{h}$  satisfies  $\hat{h}(x_i) = h(x_i)$  for any  $i \in [n]$ .
We show a generalization bound that scales with the sample compression size. The proof follows from [LW86].
Lemma 10 (Sample compression scheme generalization bound). Fix a margin  $\gamma \in [0,1]$ . For any  $k \in \mathbb{N}$  and fixed function  $\phi: (\mathcal{X} \times [0,1])^k \to [0,1]^\mathcal{X}$ , for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times [0,1]$  and any  $m \in \mathbb{N}$ , for  $S = \{(x_i, y_i)\}_{i \in [m]}$  i.i.d.  $\mathcal{D}$ -distributed random variables, if there exist indices  $i_1, \ldots, i_k \in [m]$  such that  $\sum_{(x,y) \in S} \mathbb{I}\{\ell(\phi((x_{i_1}, y_{i_1}), \ldots, (x_{i_k}, y_{i_k}))(x), y) > \gamma\} = 0$ , then
$$
\mathbf {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {I} \left\{\ell \left(\phi \left(\left(x _ {i _ {1}}, y _ {i _ {1}}\right), \dots , \left(x _ {i _ {k}}, y _ {i _ {k}}\right)\right) (x), y\right) > \gamma \right\} \right] \leq \frac {1}{m - k} \left(k \log m + \log \left(1 / \delta\right)\right).
$$
with probability at least  $1 - \delta$  over  $S$ .
Proof. Let us define  $\widehat{\ell}_{\gamma}(h;S) = \frac{1}{|S|}\sum_{(x,y)\in S}\mathbb{I}\{\ell(h(x),y) > \gamma\}$  and  $\ell_{\gamma}(h;\mathcal{D}) = \mathbf{E}_{(x,y)\sim \mathcal{D}}[\mathbb{I}\{\ell(h(x),y) > \gamma\}]$ . For any indices  $i_1,\ldots,i_k\in [m]$ , the probability of the bad event
$$
\mathbf {P r} _ {S \sim \mathcal {D} ^ {m}} [ \widehat {\ell} _ {\gamma} (\phi ((x _ {i _ {1}}, y _ {i _ {1}}),..., (x _ {i _ {k}}, y _ {i _ {k}})); S) = 0 \land \ell_ {\gamma} (\phi ((x _ {i _ {1}}, y _ {i _ {1}}),..., (x _ {i _ {k}}, y _ {i _ {k}})); \mathcal {D}) > \varepsilon ]
$$
is at most
$$
\begin{array}{l} \mathbf {E} \left[ \mathbb {I} \{\ell_ {\gamma} (\phi (\{(x _ {i _ {j}}, y _ {i _ {j}}) \} _ {j \in [ k ]}); \mathcal {D}) > \varepsilon \} \mathbf {P r} [ \widehat {\ell} _ {\gamma} (\phi (\{(x _ {i _ {j}}, y _ {i _ {j}}) \} _ {j \in [ k ]}); S \setminus \{(x _ {i _ {j}}, y _ {i _ {j}}) \} _ {j \in [ k ]}) = 0 | \{(x _ {i _ {j}}, y _ {i _ {j}}) \} _ {j \in [ k ]} ] \right] \\ <   (1 - \varepsilon) ^ {m - k} \\ \end{array}
$$
where the expectation is over  $(x_{i_1},y_{i_1}),\ldots ,(x_{i_k},y_{i_k})$  and the inner probability is over  $S\setminus (x_{i_1},y_{i_1}),\ldots ,(x_{i_k},y_{i_k})$ . Taking a union bound over all  $m^k$  possible choices for the  $k$  indices, we get that the bad event occurs with probability at most
$$
m ^ {k} \exp (- \varepsilon (m - k)) \leq \delta \Rightarrow \varepsilon = \frac {1}{m - k} (k \log m + \log (1 / \delta)).
$$
C.3 Putting it Together
We now have all the necessary ingredients in place to prove the upper bound of Theorem 2. First, we use Lemma 8 on a sample of size  $n_0 = \mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  to obtain a learner which makes  $\gamma$ -errors with probability at most  $1/3^5$ . Then, we use the boosting algorithm we described (see Algorithm 2) to obtain a learner that does not make any  $\gamma$ -mistakes on the training set. Notice that the boosting algorithm on its own does not provide any guarantees about the generalization error of the procedure. This is obtained through the sample compression result we described in Appendix C.2.3. Since we run the boosting algorithm for a few rounds on a sample whose size is small, we can provide a sample compression scheme following the approach of [DMY16, HKS19].
Lemma 11 (Upper Bound of PAC Regression). Let  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  and  $\varepsilon, \delta, \gamma \in (0,1)^3$ . Then,
$$
\mathcal {M} (\mathcal {H}; \varepsilon , \delta , \gamma) \leq O \left(\frac {\mathbb {D} _ {\gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon} \log^ {2} \frac {\mathbb {D} _ {\gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon} + \frac {1}{\varepsilon} \log \frac {1}{\delta}\right).
$$
Proof. Let  $n$  be the number of samples  $S = ((x_{1},y_{1}),\ldots ,(x_{n},y_{n}))$  that are available to the learner,  $n_0 = \mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H})$  and let  $A$  be the algorithm obtained from Lemma 8. We have that
$$
\underset {S \sim \mathcal {D} _ {\mathcal {X}} ^ {n _ {0} - 1}} {\mathbf {E}} \left[ \underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\mathbf {P r}} \left[ \ell (A (S; x), h ^ {\star} (x)) > \gamma \right] \right] \leq 1 / 3.
$$
This means that, for any distribution  $\mathcal{D}_{\mathcal{X}}$  and any labeling function  $h^\star \in \mathcal{H}$  we can draw a sample  $S^{\star} = ((x_{1},y_{1}),\ldots ,(x_{n_{0} - 1},y_{n_{0 - 1}}))$  with non-zero probability such that
$$
\underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\operatorname * {P r}} [ \ell (A (S ^ {\star}; x), h ^ {\star} (x)) > \gamma ] \leq \frac {1}{3}.
$$
In expectation over the training set.
Notice that such a classifier is a  $(\gamma, 1/6)$ -weak learner (see Definition 15). Thus, by executing the MedBoost algorithm (see Algorithm 2) for  $T = O(\log n)$  rounds we obtain a classifier  $\hat{h}: \mathcal{X} \to \mathbb{R}$  such that,  $\ell(\hat{h}(x_i), y_i) \leq \gamma, \forall i \in [n]$ . We underline that the subset  $S_t$  that is used in line 1 of Algorithm 2 has size at most  $n_0$ , for all rounds  $t \in [T]$ . Thus, the total number of samples that is used by MedBoost is at most  $O(n_0 \log n)$ . Hence, following the approach of [MY16] we can encode the classifiers produced by MedBoost as a compression set that consists of  $k = O(n_0 \log n)$  samples that were used to train the classifiers along with  $k \log k$  extra bits that indicate their order. Thus, using generalization based on sample compression scheme as in Lemma 10, we have that with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}^n$ ,
$$
\underset {(x, y) \sim \mathcal {D}} {\mathbf {E}} \left[ \mathbb {I} \left\{\ell (\hat {h} (x), y) > \gamma \right\} \right] \leq \frac {C}{n - n _ {0} \log (n)} \left(n _ {0} \log^ {2} n + \log (1 / \delta)\right),
$$
which means that for large enough  $n$
$$
\mathbf {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {I} \left\{\ell (\hat {h} (x), y) > \gamma \right\} \right] \leq O \left(\frac {n _ {0} \log^ {2} n}{n} + \frac {\log (1 / \delta)}{n}\right).
$$
$$
\mathop{\bf P r}\limits_{(x,y)\sim \mathcal{D}}\left[\ell (\hat{h} (x),y) > \gamma \right]\leq O\left(\frac{n_{0}\log^{2}n}{n} +\frac{\log(1 / \delta)}{n}\right).
$$
Hence, we can see that
D  $\gamma$ -DS Dimension and Learnability
In this section, we will show that finiteness of  $\gamma$ -DS dimension is necessary for PAC learning in the realizable case.
Theorem 3. Let  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}, \varepsilon, \delta, \gamma \in (0,1)^3$ . Then,
$$
\mathcal {M} (\mathcal {H}; \varepsilon , \delta , \gamma) \geq \Omega \left(\frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {D S}} (\mathcal {H}) + \log (1 / \delta)}{\varepsilon}\right).
$$
Proof. Let  $d = \mathbb{D}_{2\gamma}^{\mathrm{OIG}}(\mathcal{H})$ . Then, there exists some  $S = (x_{1},\ldots ,x_{d})\in \mathcal{X}^{d}$  such that  $\mathcal{H}|_S$  contains a  $2\gamma$ -pseudo-cube, which we call  $\mathcal{H}'$ . By the definition of the scaled pseudo-cube,  $\forall h\in \mathcal{H}',i\in [d]$ , there is exactly one  $h^\prime \in \mathcal{H}'$  such that  $h(x_{j}) = h^{\prime}(x_{j}),j\neq i$ , and  $\ell (h(x_i),h'(x_i)) > 2\gamma$ . We pick the target function  $h^\star$  uniformly at random among the hypotheses of  $\mathcal{H}'$  and we set the marginal distribution  $\mathcal{D}_{\mathcal{X}}$  of  $\mathcal{D}$  as follows
$$
\mathbf {P r} [ x _ {1} ] = 1 - 2 \varepsilon , \quad \mathbf {P r} [ x _ {i} ] = 2 \varepsilon / (d - 1), \forall i \in \{2, \dots , d \}.
$$
Consider  $m$  samples  $\{(z_i, h^\star(z_i)\}_{i \in [m]}$  drawn i.i.d. from  $\mathcal{D}$ . Let us take  $m \leq \frac{d-1}{6\varepsilon}$ . Then, the sample will include at most  $(d-1)/2$  examples which are not  $x_1$  with probability  $1/100$ , using Chernoff's bound. Let us call this event  $E$ . Conditioned on  $E$ , the posterior distribution of the unobserved points is uniform among the vertices of the  $d/2$ -dimensional  $2\gamma$ -pseudo-cube. Thus, if the test point  $x$  falls among the unobserved points, the learner will make a  $\gamma$ -mistake with probability at least  $1/2$ . To see that, let  $\widehat{y}$  be the prediction of the learner on  $x$ . Since every hyperedge has size at least 2 and all the vertices that are on the hyperedge differ by at least  $2\gamma$  in the direction of  $x$ , no matter what  $\widehat{y}$  is the correct label  $y^\star$  is at least  $\gamma$ -far from it. Since  $\mathbf{Pr}[E] \geq 1/100$ , we can see that  $\mathcal{M}_{\mathcal{A}}(\mathcal{H}; \varepsilon, \delta, \gamma) = \Omega\left(\frac{d}{\varepsilon}\right)$ . Moreover, by the law of total probability there must exist a deterministic choice of the target function  $h^\star$ , that could depend on  $\mathcal{A}$ , which satisfies the lower bound. For the other part of the lower bound, notices the probability that the sample will only contain  $x_1$  is
$(1 - 2\varepsilon)^{m} \geq e^{-4\varepsilon m}$  which is greater than  $\delta$  whenever  $m \leq \log(1 / \delta) / (4\varepsilon)$ . This implies that the  $\gamma$ -cut-off sample complexity is lower bounded by
$$
\max  \left\{C _ {1} \cdot \frac {d}{\varepsilon}, C _ {2} \cdot \frac {\log (1 / \delta)}{\varepsilon} \right\} = C _ {0} \cdot \frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {D S}} (\mathcal {H}) + \log (1 / \delta)}{\varepsilon}.
$$
Thus  $\mathcal{M}_{\mathcal{A}}(\mathcal{H};\varepsilon ,\delta ,\gamma)$ , satisfies the desired bound when the dimension is finite. Finally, it remains to claim about the case where  $\mathbb{D}_{2\gamma}^{\mathrm{DS}}(\mathcal{H}) = \infty$  for the given  $\gamma$ . We consider a sequence of  $2\gamma$ -DS shattered sets  $S_{n}$  with  $|S_{n}| = n$  and repeat the claim for the finite case. This will yield that for any  $n$  the  $\gamma$ -cut-off sample complexity is lower bounded by  $\Omega ((n + \log (1 / \delta)) / \varepsilon)$  and this yields that  $\mathcal{M}(\mathcal{H};\varepsilon ,\delta ,\gamma) = \infty$ .
We further conjecture that this dimension is also sufficient for PAC learning.
Conjecture 2. A class  $\mathcal{H} \subseteq (0,1)^{\mathcal{X}}$  is PAC learnable in the realizable regression setting with respect to the absolute loss function if and only if  $\mathbb{D}_{\gamma}^{\mathrm{DS}}(\mathcal{H}) < \infty$  for any  $\gamma \in (0,1)$ .
We believe that there must exist a modification of the approach of  $\left[\mathrm{BCD}^{+}22\right]$  that will be helpful in settling the above conjecture.
**Conjecture 3.** There exists  $\mathcal{H} \subseteq (0,1)^{\mathcal{X}}$  for which  $\mathbb{D}_{\gamma}^{\mathrm{Nat}}(\mathcal{H}) = 1$  for all  $\gamma \in (0,1)$  but  $\mathbb{D}_{\gamma}^{\mathrm{DS}}(\mathcal{H}) < \infty$  for some  $\gamma \in (0,1)$ .
In particular, we believe that one can extend the construction of  $\left[\mathrm{BCD}^{+}22\right]$  (which uses various tools from algebraic topology as a black-box) and obtain a hypothesis class  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  that has  $\gamma$ -Natarajan dimension 1 but is not PAC learnable (it will have infinite  $\gamma$ -DS dimension). This construction though is not immediate and requires new ideas related to the works of [JS03, Osa13]
E Online Realizable Regression