SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For  $r \in Q_{\varepsilon, \gamma}$ , fix  $h_r \in \mathcal{H}$  that satisfies  $\widehat{\mathrm{er}}_{r,\gamma}(h_r) = 0$ ,  $\mathrm{er}_{\mathcal{D}, \gamma}(h) > \varepsilon$ . It suffices to show that for  $h_r$
$$
\mathcal {D} ^ {n} \left(s: \widehat {\operatorname {e r}} _ {s, \gamma} \left(h _ {r}\right) \geq \varepsilon / 2\right) \geq 1 / 2.
$$
Then, the proof of the lemma follows immediately. Since  $\mathrm{er}_{\mathcal{D},\gamma}(h_r) > \varepsilon$ , we know that  $n \cdot \widehat{\mathrm{er}}_{s,\gamma}(h)$  follows a binomial distribution with probability of success on every try at least  $\varepsilon$  and  $n$  number of tries. Thus, the multiplicative version of Chernoff's bound gives us
$$
\mathcal {D} ^ {n} (s: \widehat {\mathrm {e r}} _ {s, \gamma} (h _ {r}) <   \varepsilon / 2) \leq e ^ {- \frac {n \cdot \varepsilon}{8}}.
$$
Thus, if  $n = c / \varepsilon$ , for some appropriate absolute constant  $c$  we see that
$$
\mathcal {D} ^ {n} \left(s: \widehat {\operatorname {e r}} _ {s, \gamma} \left(h _ {r}\right) <   \varepsilon / 2\right) <   1 / 2,
$$
Next, we can use a random swap argument to upper bound  $\mathcal{D}^{2n}(R_{\varepsilon,\gamma})$  with a quantity that involves a set of permutations over the sample of length  $2n$ . The main idea behind the proof is to try to leverage the fact that each of the labeled examples is as likely to occur among the first  $n$  examples or the last  $n$  examples.
Following [AB99], we denote by  $\Gamma_{n}$  the set of all permutations on  $\{1, \ldots, 2n\}$  that swap  $i$  and  $n + i$ , for all  $i$  that belongs to  $\{1, \ldots, n\}$ . In other words, for all  $\sigma \in \Gamma_{n}$  and  $i \in \{1, \ldots, n\}$  either  $\sigma(i) = i$ ,  $\sigma(n + i) = n + i$  or  $\sigma(i) = n + i$ ,  $\sigma(n + i) = i$ . Thus, we can think of  $\sigma$  as acting on coordinates where it (potentially) swaps one element from the first half of the sample with the corresponding element on the second half of the sample. For some  $z \in Z^{2n}$  we overload the notation and denote  $\sigma(z)$  the effect of applying  $\sigma$  to the sample  $z$ .
We are now ready to state the bound. Importantly, it shows that by (uniformly) randomly choosing a permutation  $\sigma \in \Gamma_{n}$  we can bound the probability that a sample falls into the bad set  $R_{\varepsilon,\gamma}$  by a quantity that does not depend on the distribution  $\mathcal{D}$ .
Lemma 3 (Random Swaps; Adaptation of Lemma 4.5 in [AB99]). Fix  $Z = \mathcal{X} \times [0,1]$ . Let  $R_{\varepsilon,\gamma}$  be any subset of  $Z^{2n}$  and  $\mathcal{D}$  any probability distribution on  $Z$ . Then
$$
\mathcal {D} ^ {2 n} (R _ {\varepsilon , \gamma}) = \underset {z \sim \mathcal {D} ^ {2 n}} {\mathbf {E}} \underset {\sigma \sim \mathbb {U} (\Gamma_ {n})} {\mathbf {P r}} [ \sigma (z) \in R _ {\varepsilon , \gamma} ] \leq \max _ {z \in Z ^ {2 n}} \underset {\sigma \sim \mathbb {U} (\Gamma_ {n})} {\mathbf {P r}} [ \sigma (z) \in R _ {\varepsilon , \gamma} ],
$$
where  $\mathbb{U}(\Gamma_n)$  is the uniform distribution over the set of swapping permutations  $\Gamma_{n}$ .
Proof. First, notice that the bound
$$
\mathop{\bf E}\limits_{z\sim \mathcal{D}^{2n}}\mathop{\bf Pr}\limits_{\sigma \sim \mathbb{U}(\Gamma_{n})}\left[ \sigma (z)\in R_{\varepsilon ,\gamma}\right]\leq \max_{z\in Z^{2n}}\mathop{\bf Pr}\limits_{\sigma \sim \mathbb{U}(\Gamma_{n})}\left[ \sigma (z)\in R_{\varepsilon ,\gamma}\right],
$$
follows trivially and the maximum exists since  $\mathbf{Pr}_{\sigma \sim \mathbb{U}(\Gamma_n)}[\sigma (z)\in R_{\varepsilon ,\gamma}]$  takes finitely many values for any finite  $n$  and all  $z\in Z^{2n}$ . Thus, the bulk of the proof is to show that
$$
\mathcal {D} ^ {2 n} (R _ {\varepsilon , \gamma}) = \underset {z \sim \mathcal {D} ^ {2 n}} {\mathbf {E}} \underset {\sigma \sim \mathbb {U} (\Gamma_ {n})} {\mathbf {P r}} [ \sigma (z) \in R _ {\varepsilon , \gamma} ].
$$
First, notice that since example is drawn i.i.d., for any swapping permutation  $\sigma \in \Gamma_{n}$  we have that
$$
\mathcal {D} ^ {2 n} \left(R _ {\varepsilon , \gamma}\right) = \mathcal {D} ^ {2 n} \left(\left\{z \in Z ^ {2 n}: \sigma (z) \in R _ {\varepsilon , \gamma} \right\}\right). \tag {5}
$$
Thus, the following holds
$$
\begin{array}{l} \mathcal {D} ^ {2 n} \left(R _ {\varepsilon , \gamma}\right) = \int_ {Z ^ {2 n}} \mathbb {I} \left\{z \in R _ {\varepsilon , \gamma} \right\} d \mathcal {D} ^ {2 n} (z) \\ = \frac {1}{\left| \Gamma_ {n} \right|} \sum_ {\sigma \in \Gamma_ {n}} \int_ {Z ^ {2 n}} \mathbb {I} \left\{\sigma (z) \in R _ {\varepsilon , \gamma} \right\} d \mathcal {D} ^ {2 n} (z) \\ = \int_ {Z ^ {2 n}} \left(\frac {1}{| \Gamma_ {n} |} \sum_ {\sigma \in \Gamma_ {n}} \mathbb {I} \{\sigma (z) \in R _ {\varepsilon , \gamma} \}\right) d \mathcal {D} ^ {2 n} (z) \\ = \underset {z \sim \mathcal {D} ^ {2 n}} {\mathbf {E}} \underset {\sigma \sim \mathbb {U} (\Gamma_ {n})} {\mathbf {P r}} [ \sigma (z) \in R _ {\varepsilon , \gamma} ], \\ \end{array}
$$
where the first equation follows by definition, the second by Equation (5), the third because the number of terms in the summation is finite, and the last one by definition.
As a last step we can bound the above RHS by using all possible patterns when  $\mathcal{H}$  is (roughly speaking) projected in the sample  $rs\in Z^{2n}$ .
Lemma 4 (Bounding the Bad Event). Fix  $Z = \mathcal{X} \times [0,1]$ . Let  $R_{\varepsilon,\gamma} \subseteq Z^{2n}$  be the set
$$
R _ {\varepsilon , \gamma} = \left\{(r, s) \in Z ^ {n} \times Z ^ {n}: \exists h \in \mathcal {H}: \widehat {\mathrm {e r}} _ {r, \gamma} (h) = 0, \widehat {\mathrm {e r}} _ {s, \gamma} (h) \geq \varepsilon / 2 \right\}.
$$
$$
\mathop{\bf P r}\limits_{\sigma \sim \mathbb{U}(\Gamma_{n})}\bigl[\sigma (z)\in R_{\varepsilon ,\gamma}\bigr] \leq (2n)^{O(\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H})\log (2n))}2^{-n\varepsilon /2}.
$$
Proof. Throughout the proof we fix  $z = (z_{1},\dots,z_{2n}) \in Z^{2n}$ , where  $z_{i} = (x_{i},y_{i}) = (x_{i},h^{\star}(x_{i}))$  and let  $S = \{x_1,\ldots ,x_{2n}\}$ . Consider the projection set  $\mathcal{H}|_{S}$ . We define a partial binary concept class  $\mathcal{H}' \subseteq \{0,1,\star\}^{2n}$  as follows:
$$
\mathcal {H} ^ {\prime} := \left\{h ^ {\prime} \in \{0, 1, \star \} ^ {2 n}: \exists h \in \mathcal {H} | _ {S}: \forall i \in [ 2 n ] \left\{ \begin{array}{l l} h (x _ {i}) = y _ {i}, & h ^ {\prime} (i) = 0 \\ \ell (h (x _ {i}), y _ {i}) > \gamma , & h ^ {\prime} (i) = 1 \\ 0 <   \ell (h (x _ {i}), y _ {i}) \leq \gamma , & h ^ {\prime} (i) = \star \end{array} \right\} \right\}.
$$
Importantly, we note that, by definition,  $\mathrm{VC}(\mathcal{H}') \leq \mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H})$ .
Currently, we have a partial binary concept class  $\mathcal{H}'$ . As a next step, we would like to replace the  $\star$  symbols and essentially reduce the problem to a total concept class. This procedure is called disambiguation (cf. Definition 14). The next key lemma shows that there exists a compact (in terms of cardinality) disambiguation of a VC partial concept class for finite instance domains.
Lemma 5 (Compact Disambiguations, see [AHHM22]). Let  $\mathcal{H}$  be a partial concept class on a finite instance domain  $\mathcal{X}$  with  $\mathrm{VC}(\mathcal{H}) = d$ . Then there exists a disambiguation  $\overline{\mathcal{H}}$  of  $\mathcal{H}$  with size  $|\overline{\mathcal{H}}| = |\mathcal{X}|^{O(d\log |\mathcal{X}|)}$ .
This means that there exists a disambiguation  $\overline{\mathcal{H}'}$  of  $\mathcal{H}'$  of size at most
$$
(2 n) ^ {O \left(\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) \log (2 n)\right)}.
$$
Since this (total) binary concept class is finite, we can apply the following union bound argument. We have that  $\sigma(z) \in R$  if and only if some  $h \in \mathcal{H}$  satisfies
$$
\frac {\sum_ {i = 1} ^ {n} \mathbb {I} \{\ell (h (x _ {\sigma (i)}) , y _ {\sigma (i)}) > \gamma \}}{n} = 0, \quad \frac {\sum_ {i = 1} ^ {n} \mathbb {I} \{\ell (h (x _ {\sigma (n + i)}) , y _ {\sigma (n + i)}) > \gamma \}}{n} \geq \varepsilon / 2.
$$
We can relate this event with an event about the disambiguated partial concept class  $\overline{\mathcal{H}'}$  since the number of 1's can only increase. In particular, for any swapping permutation  $\sigma$  of the  $2n$  points, if there exists a function in  $\mathcal{H}$  that is correct on the first  $n$  points and is off by at least  $\gamma$  on at least  $\varepsilon n/2$  of the remaining  $n$  points, then there is a function in the disambiguation  $\overline{\mathcal{H}'}$  that is 0 on the first  $n$  points and is 1 on those same  $\epsilon n/2$  of the remaining points.
If we fix some  $\sigma \in \Gamma_{n}$ , and some  $\overline{h^{\prime}} \in \overline{\mathcal{H}^{\prime}}$  then  $\overline{h^{\prime}}$  is a witness that  $\sigma(z) \in R_{\varepsilon, \gamma}$  only if  $\forall i \in [n]$  we do not have that  $\overline{h^{\prime}}(i) = 1$ ,  $\overline{h^{\prime}}(i + n) = 1$ . Thus at least one of  $\overline{h^{\prime}}(i)$ ,  $\overline{h^{\prime}}(n + i)$  must be zero. Moreover, at least  $n\varepsilon / 2$  entries must be non-zero. Thus, when we draw random swapping permutation the probability that all the non-zero entries land on the second half of the sample sample is at most  $2^{-n\varepsilon / 2}$ .
Crucially since the number of possible functions is at most  $|\overline{\mathcal{H}}^{\prime}|\leq (2n)^{O(\mathbb{D}_{\gamma}^{G}(\mathcal{H})\log (2n))}$  a union bound gives us that
$$
\mathop{\bf P r}\limits_{\sigma \sim \mathbb{U}(\Gamma_{n})}\left[\sigma (z)\in R_{\varepsilon ,\gamma}\right]\leq (2n)^{O(\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H})\log (2n))}\cdot 2^{-n\varepsilon /2}.
$$
Thus, since  $z \in Z^{2n}$  was arbitrary we have that
$$
\max  _ {z \in Z ^ {2 n}} \mathbf {P r} _ {\sigma \left(\Gamma_ {n}\right)} [ \sigma (z) \in R _ {\varepsilon , \gamma} ] \leq (2 n) ^ {O (\mathbb {D} _ {\gamma} ^ {G} (\mathcal {H}) \log (2 n))} \cdot 2 ^ {- n \varepsilon / 2}.
$$
Lower Bound for the ERM learner. Our next goal is to show that
$$
\mathcal {M} _ {\mathrm {E R M}} (\mathcal {H}; \varepsilon , \delta , \gamma) \geq C _ {0} \cdot \frac {\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) + \log (1 / \delta)}{\varepsilon}.
$$
To this end, we will show that there exists an ERM learner satisfying this lower bound. This will establish that the finiteness of  $\gamma$ -graph dimension for any  $\gamma \in (0,1)$ , is necessary for PAC learnability using a worst-case ERM. It suffices to show that there exists a bad ERM algorithm that requires at least  $C_0 \frac{\mathbb{D}_\gamma^{\mathrm{G}}(\mathcal{H}) + \log(1 / \delta)}{\varepsilon}$  samples to cut-off PAC learn  $\mathcal{H}$ . First let us consider the case where  $d = \mathbb{D}_\gamma^{\mathrm{G}}(\mathcal{H}) < \infty$  and let  $S = \{x_1,\dots,x_d\}$  be a  $\gamma$ -graph-shattered set by  $\mathcal{H}$  with witness  $f_0$ . Consider the ERM learner  $\mathcal{A}$  that works as follows: Upon seeing a sample  $T \subseteq S$  consistent with  $f_0$ ,  $\mathcal{A}$  returns a function  $\mathcal{A}(T)$  that is equal to  $f_0$  on elements of  $T$  and  $\gamma$ -far from  $f_0$  on  $S \setminus T$ . Such a function exists since  $S$  is  $\gamma$ -graph-shattered with witness  $f_0$ . Let us take  $\delta < 1/100$  and  $\varepsilon < 1/12$ . Define a distribution over  $S \subseteq \mathcal{X}$  such that
$$
\mathbf {P r} \left[ x _ {1} \right] = 1 - 2 \varepsilon , \quad \mathbf {P r} \left[ x _ {i} \right] = 2 \varepsilon / (d - 1), \forall i \in \{2, \dots , d \}.
$$
Let us set  $h^{\star} = f_{0}$  and consider  $m$  samples  $\{(z_{i},f_{0}(z_{i})\}_{i\in [m]}$ . Since we work in the scaled PAC model,  $\mathcal{A}$  will make a  $\gamma$ -error on all examples from  $S$  which are not in the sample (since in that case the output will be  $\gamma$ -far from the true label). Let us take  $m \leq \frac{d - 1}{6\varepsilon}$ . Then, the sample will include at most  $(d - 1) / 2$  examples which are not  $x_{1}$  with probability  $1 / 100$ , using Chernoff's bound. Conditioned on that event, this implies that the ERM learner will make a  $\gamma$ -error with probability at least  $\frac{2\varepsilon}{d - 1}\cdot (d - 1 - \frac{d - 1}{2}) = \varepsilon$ , over the random draw of the test point. Thus,  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta ,\gamma) = \Omega (\frac{d - 1}{\varepsilon})$ . Moreover, the probability that the sample will only contain  $x_{1}$  is  $(1 - 2\varepsilon)^{m} \geq e^{-4\varepsilon m}$  which is greater than  $\delta$  whenever  $m \leq \log (1 / \delta) / (4\varepsilon)$ . This implies that the  $\gamma$ -cut-off ERM sample complexity is lower bounded by
$$
\max \left\{\frac {d - 1}{6 \varepsilon}, \frac {\log (1 / \delta)}{2 \varepsilon} \right\} = C _ {0} \cdot \frac {\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) + \log (1 / \delta)}{\varepsilon}.
$$
Thus  $\mathcal{M}_{\mathrm{ERM}}(\mathcal{H};\varepsilon ,\delta ,\gamma)$ , satisfies the desired bound when the dimension is finite. Finally, it remains to claim about the case where  $\mathbb{D}_{\gamma}^{\mathrm{G}}(\mathcal{H}) = \infty$  for the given  $\gamma$ . We consider a sequence of  $\gamma$ -graphshattered sets  $S_{n}$  with  $|S_{n}| = n$  and repeat the claim for the finite case. This will yield that for any  $n$  the cut-off ERM sample complexity is lower bounded by  $\Omega ((n + \log (1 / \delta)) / \varepsilon)$  and this yields that  $\mathcal{M}_{\mathrm{ERM}}(\mathcal{H};\varepsilon ,\delta ,\gamma) = \infty$ .
However, as Example 4 shows, the optimal learner cannot be proper and as a result, this dimension does not characterize PAC learnability for real-valued regression (there exist classes whose  $\gamma$ -graph dimension is infinite but are PAC learnable in the realizable regression setting).
C  $\gamma$ -OIG Dimension and Learnability
In this section we identify a dimension characterizing qualitatively and quantitatively what classes of predictors  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  are PAC learnable and we provide PAC learners that achieve (almost) optimal sample complexity. In particular, we show the following result.
Theorem 2. Let  $\ell$  be the absolute loss function. For every class  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  and for any  $\varepsilon, \delta, \gamma \in (0,1)^3$ , the sample complexity bound for realizable PAC regression satisfies
$$
\Omega \left(\frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon}\right) \leq \mathcal {M} (\mathcal {H}; \varepsilon , \delta , \gamma) \leq O \left(\frac {\mathbb {D} _ {\gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon} \log^ {2} \frac {\mathbb {D} _ {\gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon} + \frac {1}{\varepsilon} \log \frac {1}{\delta}\right).
$$
In particular, there exists an algorithm  $A$  such that
$$
\mathbf{E}_{x\sim \mathcal{D}_{\mathcal{X}}}\left[ \ell (A(S;x),h^{\star}(x)\right]\leq \inf_{\gamma \in [0,1]}\gamma +\widetilde{\Theta}\left(\frac{\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{H}) + \log(1 / \delta)}{n}\right),
$$
with probability at least  $1 - \delta$  over  $S\sim \mathcal{D}^n$
A finite fat-shattering dimension implies a finite OIG dimension. Let  $\mathcal{F} \subseteq [0,1]^{\mathcal{X}}$  be a function class with finite  $\gamma$ -fat shattering dimension for any  $\gamma > 0$ . We show that  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{F})$  is upper bounded (up to constants and log factors) by  $\mathbb{D}_{c\gamma}^{\mathrm{fat}}(\mathcal{F})$ , for some  $c > 0$ , where the OIG dimension is defined with respect to the  $\ell_1$  loss. Note that the opposite direction does not hold. Example 1 exhibits a function class with an infinite fat-shattering dimension that can be learned with a single example, and as a result, the OIG dimension has to be finite. On the one hand, we have an upper bound on the sample complexity of  $O\left(\frac{1}{\epsilon}\left(\mathbb{D}_{c\varepsilon}^{\mathrm{fat}}(\mathcal{F})\log^{2}\frac{1}{\epsilon} + \log \frac{1}{\delta}\right)\right)$ , for any  $\varepsilon, \delta \in (0,1)$ . See sections 19.6 and 20.4 about the restricted model in [AB99]. On the other hand, we prove in Lemma 6 a lower bound on the sample complexity of  $\Omega\left(\frac{\mathbb{D}_{2\varepsilon}^{\mathrm{OIG}}(\mathcal{F})}{\epsilon}\right)$ , for any  $\varepsilon \in (0,1)$ , and so  $\mathbb{D}_{\gamma}^{\mathrm{OIG}}(\mathcal{F})$  is upper bounded by  $\mathbb{D}_{c\gamma}^{\mathrm{fat}}(\mathcal{F})$  up to constants and log factors.
C.1 Proof of the Lower Bound
Lemma 6. [Lower Bound of PAC Regression] Let  $A$  be any learning algorithm and  $\varepsilon, \delta, \gamma \in (0,1)^3$  such that  $\delta < \varepsilon$ . Then,
$$
\mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta , \gamma) \geq \Omega \left(\frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon}\right).
$$
Proof. Let  $n_0 = \mathbb{D}_{2\gamma}^{\mathrm{OIG}}(\mathcal{H})$ . Let  $n \in \mathbb{N}, 1 < n \leq n_0$ . We know that for each such  $n$  there exists some  $S \in \mathcal{X}^n$  such that the one-inclusion graph of  $\mathcal{H}|_S$  has the property that: there exists a finite subgraph  $G = (V, E)$  of  $G_{\mathcal{H}|_S}^{\mathrm{OIG}}$  such that for any orientation  $\sigma : E \to V$  of the subgraph, there exists a vertex  $v \in V$  with  $\text{outdeg}(v; \sigma, 2\gamma) > \mathbb{D}_{2\gamma}^{\mathrm{OIG}}(\mathcal{H}) / 3$ .
Given the learning algorithm  $A: (\mathcal{X} \times [0,1])^\star \times \mathcal{X} \to [0,1]$ , we can describe an orientation  $\sigma_A$  of the edges in  $E$ . For any vertex  $v = (v_1, \ldots, v_n) \in V$  let  $P_v$  be the distribution over  $(x_1, v_1), \ldots, (x_n, v_n)$  defined as
$$
P _ {v} \left(\left(x _ {1}, v _ {1}\right)\right) = 1 - \varepsilon , P _ {v} \left(\left(x _ {t}, v _ {t}\right)\right) = \frac {\varepsilon}{n - 1}, t \in \{2, \dots , n \}.
$$
Let  $m = n / (2\varepsilon)$ . For each vertex  $v \in V$  and direction  $t \in [n]$ , consider the hyperedge  $e_{t,v}$ . For each  $u \in e_{t,v}$  we define
$$
p _ {t} (u) = \underset {S \sim P _ {u} ^ {m}} {\mathbf {P r}} [ \ell (A (S; x _ {t}), u _ {t}) > \gamma | (x _ {t}, u _ {t}) \notin S ],
$$
and let  $C_{e_{t,v}} = \{u \in e_{t,v} : p_t(u) < 1/2\}$ . If  $C_{e_{t,v}} = \emptyset$ , we orient the edge  $e_{t,v}$  arbitrarily. Since for all  $u, v \in e_{t,v}$  the distributions  $P_u^m, P_v^m$  conditioned on the event that  $(x_t, u_t), (x_t, v_t)$  respectively
are not in  $S$  are the same, we can see that  $\forall u, v \in C_{e_{t,v}}$  it holds that  $\ell(u_t, v_t) \leq 2\gamma$ . We orient the edge  $e_{t,v}$  using an arbitrary element of  $C_{e_{t,v}}$ .
Because of the previous discussion, we can bound from above the out-degree of all vertices  $v \in V$  with respect to the orientation  $\sigma_A$  as follows:
$$
\operatorname {o u t d e g} (v; \sigma_ {A}, 2 \gamma) \leq \sum_ {t} \mathbb {I} \{p _ {t} (v) \geq 1 / 2 \} \leq 1 + 2 \sum_ {t = 2} ^ {n} \underset {S \sim P _ {v} ^ {m}} {\operatorname {P r}} [ \ell (A (S, x _ {t}), y _ {t}) > \gamma | (x _ {t}, y _ {t}) \notin S ].
$$
$$
\mathop{\bf P r}\limits_{S\sim P_{v}^{m}}[\ell (A(S,x_{t}),y_{t}) > \gamma |(x_{t},y_{t})\notin S] = \frac{\mathop{\bf P r}_{S\sim P_{v}^{m}}[\{\ell(A(S,x_{t}),y_{t}) > \gamma \} \wedge \{(x_{t},y_{t})\notin S\}]}{\mathop{\bf P r}_{S\sim P_{v}^{m}}[(x_{t},y_{t})\notin S]},
$$
and by the definition of  $P_{v}$ , we have that
$$
\mathbf {P r} _ {S \sim P _ {v} ^ {m}} [ (x _ {t}, y _ {t}) \notin S ] = \left(1 - \frac {\varepsilon}{n - 1}\right) ^ {m} \geq 1 - \frac {n}{2 (n - 1)},
$$
since  $m = n / (2\varepsilon)$ . Combining the above, we get that
$$
\mathrm {o u t d e g} (v; \sigma_ {A}, 2 \gamma) \leq 1 + 2 \left(1 - \frac {n}{2 (n - 1)}\right) \sum_ {t = 2} ^ {n} _ {S \sim P _ {v} ^ {m}} \mathbf {E} \{\ell (A (S, x _ {t}), y _ {t}) > \gamma \} \cdot \mathbb {I} \{(x _ {t}, y _ {t}) \notin S \},
$$
$$
\begin{array}{l} \operatorname {o u t d e g} (v; \sigma_ {A}, 2 \gamma) \leq 1 + 2 \left(1 - \frac {n}{2 (n - 1)}\right) \underset {S \sim P _ {v} ^ {m}} {\mathbf {E}} \left[ \sum_ {t = 2} ^ {n} \mathbb {I} \{\ell (A (S, x _ {t}), y _ {t}) > \gamma \} \right] \\ = 1 + 2 \left(1 - \frac {n}{2 (n - 1)}\right) \frac {n - 1}{\varepsilon} \underset {S \sim P _ {v} ^ {m}} {\mathbf {E}} \left[ \frac {\varepsilon}{n - 1} \sum_ {t = 2} ^ {n} \mathbb {I} \left\{\ell \left(A \left(S, x _ {t}\right), y _ {t}\right) > \gamma \right\} \right] \\ \leq 1 + 2 \left(1 - \frac {n}{2 (n - 1)}\right) \frac {n - 1}{\varepsilon} \underset {S \sim P _ {v} ^ {m}} {\mathbf {E}} \left[ \underset {(x, y) \sim P _ {v}} {\mathbf {E}} [ \mathbb {I} \{\ell (A (S; x), y) > \gamma \} ] \right] \\ = 1 + 2 \left(1 - \frac {n}{2 (n - 1)}\right) \frac {n - 1}{\varepsilon} \underset {S \sim P _ {v} ^ {m}} {\mathbf {E}} \left[ \underset {(x, y) \sim P _ {v}} {\mathbf {P r}} [ \ell (A (S; x), y) > \gamma ] \right] \\ \leq 1 + \frac {n - 2}{\varepsilon} \underset {S \sim P _ {v} ^ {m}} {\mathbf {E}} \left[ \underset {(x, y) \sim P _ {v}} {\mathbf {P r}} [ \ell (A (S; x), y) > \gamma ] \right] \\ \end{array}
$$
By picking "hard" distribution  $\mathcal{D} = P_{v^{\star}}$ , where  $v^{\star} \in \arg \max_{v' \in V} \mathrm{outdeg}(v'; \sigma_A, 2\gamma)$  we get that that
$$
\begin{array}{l} \underset {S \sim P _ {v ^ {\star}} ^ {m}} {\mathbf {E}} \left[ \underset {(x, y) \sim P _ {v ^ {\star}}} {\mathbf {P r}} [ \ell (A (S; x), y) > \gamma ] \right] \geq (\operatorname {o u t d e g} (v ^ {\star}; \sigma_ {A}, 2 \gamma) - 1) \cdot \frac {\varepsilon}{n - 2} \\ \geq \frac {\varepsilon}{6}, \\ \end{array}
$$
since  $\mathrm{outdeg}(v^{\star};\sigma_A,2\gamma) > n / 3$ . By picking  $n = n_0$  we see that when the learner uses  $m = n_0 / \varepsilon$  samples then its expected error is at least  $\varepsilon /6$ . Notice that when the learner uses  $m^{\prime} = \mathcal{M}_{A}(\mathcal{H};\varepsilon ,\delta ,\gamma)$  samples we have that
$$
\mathbf{E}_{S\sim P_{v^{\star}}^{m^{\prime}}}\left[ \begin{array}{c}\mathbf{Pr}\\ (x,y)\sim P_{v^{\star}} \end{array} \left[\ell (A(S;x),y) > \gamma \right] \right]\leq \delta +(1 - \delta)\varepsilon \leq \delta +\varepsilon \leq 2\varepsilon .
$$
Thus, we see that for any algorithm  $A$
$$
\mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta , \gamma) \geq \Omega \left(\frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon}\right),
$$
$$
\mathcal {M} (\mathcal {H}; \varepsilon , \delta , \gamma) \geq \Omega \left(\frac {\mathbb {D} _ {2 \gamma} ^ {\mathrm {O I G}} (\mathcal {H})}{\varepsilon}\right).
$$
C.2 Proof of the Upper Bound
Let us present the upper bound. For this proof, we need three tools: we will provide a weak learner based on the scaled one-inclusion graph, a boosting algorithm for real-valued functions, and consistent sample compression schemes for real-valued functions.
To this end, we introduce the one-inclusion graph (OIG) algorithm  $\mathbb{A}_{\gamma}^{\mathrm{OIG}}$  for realizable regression at scale  $\gamma$ .
C.2.1 Scaled One-Inclusion Graph Algorithm and Weak Learning
First, we show that every scaled orientation  $\sigma$  of the one-inclusion graph gives rise to a learner  $\mathbb{A}_{\sigma}$  whose expected absolute loss is upper bounded by the maximum out-degree induced by  $\sigma$ .
Lemma 7 (From Orientations to Learners). Let  $\mathcal{D}_{\mathcal{X}}$  be a distribution over  $\mathcal{X}$  and  $h^{\star} \in \mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ , let  $n \in \mathbb{N}$  and  $\gamma \in (0,1)$ . Then, for any orientation  $\sigma : E_n \to V_n$  of the scaled-one-inclusion graph  $G_{\mathcal{H}}^{\mathrm{OIG}} = (V_n, E_n)$ , there exists a learner  $\mathbb{A}_{\sigma} : (\mathcal{X} \times [0,1])^{n-1} \to [0,1]^{\mathcal{X}}$ , such that
$$
\mathop{\bf E}\limits_{S\sim \mathcal{D}_{\mathcal{X}}^{n - 1}}\left[\mathop{\bf Pr}\limits_{x\sim \mathcal{D}_{\mathcal{X}}}\left[ \ell (\mathbb{A}_{\sigma}(x),h^{\star}(x)) > \gamma \right]\right]\\ \leq \frac{\max_{v\in V_{n}}\operatorname{outdeg}(v;\sigma,\gamma)}{n},
$$
where  $\mathbb{A}_{\sigma}$  is trained using a sample  $S$  of size  $n - 1$  realized by  $h^{\star}$ .