SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $\| f - \widehat{f}\|_{L_1(\mu_n)}\leq \gamma (1 + 2\widehat{\Lambda}_{\widehat{f}}^\beta)$ . In particular  $L_{S}(f)\leq L_{S}(\widehat{f}) + \gamma (1 + 2\widehat{\Lambda}_{\widehat{f}}^{\beta})$
$\overline{\Lambda}_f^\beta (\mu)\leq 5\widehat{\Lambda}_\widehat{f}^\beta .$
Such a set does not exist only in the case of atoms  $x_0 \in \Omega$  with large probability mass  $\mu(x_0)$ . If that is the case, consider a "copy" metric space  $\widetilde{\Omega}$  with  $x_0$  split into two points  $x_1, x_2 \in \widetilde{\Omega}$  at distance  $\varepsilon$  apart and each of mass  $\mu(x_0)/2$ . Any function  $f: \Omega \to \mathbb{R}$  is extended to  $\widetilde{f}: \widetilde{\Omega} \to \mathbb{R}$  via  $\widetilde{f}(x_1) = \widetilde{f}(x_2) = f(x_0)$ . Repeating the split if necessary and taking  $\varepsilon \downarrow 0$ , we obtain a space  $\widetilde{\Omega}$  with all of the relevant properties of  $\Omega$  but no atoms of large mass.
Proof. Throughout the proof, we denote for any point  $x \in \Omega$ , subset  $B \subset \Omega$  and function  $g: B \to [0,1]$ :
Give the sample  $S = (X_{i},Y_{i})_{i = 1}^{n}$ , we denote  $S_{x} = (X_{i})_{i = 1}^{n}$ . Let  $\gamma >0$ . The algorithm constructs  $f:\Omega \to [0,1]$  as follows:
1. Let  $S_x(\gamma) \subset S_x$  consist of the  $\lfloor \gamma n \rfloor$  points whose  $\Lambda_{\widehat{f}}(\cdot, S_x)$  values are the largest (with ties broken arbitrarily), and  $S_x'(\gamma) \coloneqq S_x \setminus S_x(\gamma)$  be the rest.
2. Let  $A \subset S_x'(\gamma)$  be a  $\gamma^{1/\beta}$ -net of  $S_x'(\gamma)$ .
3. Define  $f: \Omega \to [0,1]$  to be the  $\beta$ -PMSE extension of  $\widehat{f}$  from  $A$  to  $\Omega$  as defined in Definition A.2 (and analyzed throughout Appendix A).
We will prove that  $f$  satisfies both requirements. For the first requirement, since  $f|_{A} = \widehat{f}|_{A}$  and  $S_{x} = S_{x}^{\prime}(\gamma) \oplus S_{x}(\gamma)$  we have
$$
\| f - \widehat {f} \| _ {L _ {1} (\mu_ {n})} := \frac {1}{n} \sum_ {i = 1} ^ {n} | f (x _ {i}) - g (x _ {i}) | = \frac {1}{n} \sum_ {x \in S _ {x} (\gamma) \backslash A} | f (x) - \widehat {f} (x) | + \frac {1}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} | f (x) - \widehat {f} (x) |.
$$
The first summand above is bounded by  $\gamma$  since  $0 \leq f, \widehat{f} \leq 1 \implies |f(x) - \widehat{f}(x)| \leq 1$  and  $|S_x(\gamma)| \leq \gamma n$ . In order to bound the second term, we denote by  $N_A: S_x'(\gamma) \to A$  to be the mapping of each element to its nearest neighbor in the net, and note that  $\rho(x, N_A(x)) \leq \gamma^{1/\beta}$ . Then
$$
\begin{array}{l} \frac {1}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} | f (x) - \widehat {f} (x) | \leq \frac {1}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} \frac {\gamma}{\rho (x , N _ {A} (x)) ^ {\beta}} | f (x) - \widehat {f} (x) | \\ \leq \frac {\gamma}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} \frac {| f (x) - \widehat {f} (N _ {A} (x)) | + | \widehat {f} (N _ {A} (x)) - \widehat {f} (x) |}{\rho (x , N _ {A} (x)) ^ {\beta}} \\ = \frac {\gamma}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} \frac {| f (x) - f (N _ {A} (x)) |}{\rho (x , N _ {A} (x)) ^ {\beta}} + \frac {| \widehat {f} (N _ {A} (x)) - \widehat {f} (x) |}{\rho (x , N _ {A} (x)) ^ {\beta}} \\ \leq \frac {\gamma}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} \Lambda_ {f} ^ {\beta} (x, A) + \Lambda_ {\widehat {f}} ^ {\beta} (x, A) \\ \end{array}
$$
$$
\begin{array}{l} \left[ \text {T h e o r e m} A. 3 \right] \leq \frac {2 \gamma}{n} \sum_ {x \in S _ {x} ^ {\prime} (\gamma) \backslash A} \Lambda_ {\widehat {f}} ^ {\beta} (x, A) \\ \leq 2 \gamma L. \\ \end{array}
$$
So overall we get  $\| f - \widehat{f}\|_{L_1(\mu_n)}\leq \gamma +2\gamma L = \gamma (1 + 2L)$  as claimed in the first bullet.
We move on to prove the second bullet. Let  $U \subset \Omega$  be a  $\frac{\gamma^{1/\beta}}{4}$ -net of  $\Omega$ ,  $\Pi$  be its induced Voronoi partition and let  $m := |\Pi| \leq \mathcal{N}_{\Omega}(\gamma^{1/\beta}/4)$ . Let Consider the following partition of  $\Pi$  into "light" and "heavy" cells:
$$
\Pi_ {l} := \left\{C \in \Pi : \mu_ {n} (C) <   n \gamma / m \right\}, \quad \Pi_ {h} := \Pi \setminus \Pi_ {l}.
$$
We will now state three lemmas required for the proof, two of which are due to [Ashlagi et al., 2021].
Lemma B.4. Suppose  $A \subset \Omega$  and that  $f: \Omega \to [0,1]$  is the  $\beta$ -PMSE extension of some function from  $A$  to  $\Omega$ . Let  $E \subset \Omega$  such that  $\mathrm{diam}(E)^{\beta} \leq \frac{1}{2} \min_{x \neq x' \in A} \rho(x,x')^{\beta}$ . Then  $\sup_{x,x' \in E} \frac{\Lambda_f^\beta(x)}{\Lambda_f^\beta(x')} \leq 2$ .
Proof. Let  $u_x^*, v_x^* \in A$  be the pair of points which satisfy  $\Lambda_f^\beta(x) = \frac{f(v_x^*) - f(u_x^*)}{\rho(v_x^*, x)^\beta + \rho(u_x^*, x)^\beta}$ . By assumption on  $E$ , we know that  $2\mathrm{diam}(E)^\beta \leq \rho(v_x^*, u_x^*)^\beta \leq \rho(v_x^*, x)^\beta + \rho(u_x^*, x)^\beta$ , hence
$$
\rho (v _ {x} ^ {*}, x) ^ {\beta} + \rho (u _ {x} ^ {*}, x) ^ {\beta} + 2 \mathrm {d i a m} (E) ^ {\beta} \leq 2 (\rho (v _ {x} ^ {*}, x) ^ {\beta} + \rho (u _ {x} ^ {*}, x) ^ {\beta}). \mathrm {W e g e t}
$$
$$
\begin{array}{l} \Lambda_ {f} ^ {\beta} (x ^ {\prime}) \geq \frac {f (v _ {x} ^ {*}) - f (u _ {x} ^ {*})}{\rho (v _ {x} ^ {*} , x ^ {\prime}) ^ {\beta} + \rho (u _ {x} ^ {*} , x ^ {\prime}) ^ {\beta}} \\ \geq \frac {f (v _ {x} ^ {*}) - f (u _ {x} ^ {*})}{\rho (v _ {x} ^ {*} , x) ^ {\beta} + \operatorname {d i a m} (E) ^ {\beta} + \rho (u _ {x} ^ {*} , x) ^ {\beta} + \operatorname {d i a m} (E) ^ {\beta}} \\ \geq \frac {f (v _ {x} ^ {*}) - f (u _ {x} ^ {*})}{2 (\rho (v _ {x} ^ {*} , x) ^ {\beta} + \rho (u _ {x} ^ {*} , x) ^ {\beta})} = \frac {1}{2} \Lambda_ {f} ^ {\beta} (x). \\ \end{array}
$$
Lemma B.5 (Ashlagi et al., 2021, Lemma 16). If  $n\gamma^2 \geq m$ , then
$$
\begin{array}{l} \Pr_ {S \sim \mathcal {D} ^ {n}} \left[ \min  _ {C \in \Pi_ {h}} \frac {\mu_ {n} (C)}{\mu (C)} > \frac {1}{2} \right] \geq 1 - m \exp (- n \gamma / 4 m), \\ \operatorname * {P r} _ {S \sim \mathcal {D} ^ {n}} \left[ \max  _ {C \in \Pi_ {h}} \frac {\mu_ {n} (C)}{\mu (C)} <   2 \right] \geq 1 - m \exp (- n \gamma / 3 m), \\ \operatorname * {P r} _ {S \sim \mathcal {D} ^ {n}} \left[ \sum_ {C \in \Pi_ {l}} \mu (C) <   2 \gamma \right] \geq 1 - \exp \left(- n \left(\gamma - \sqrt {m / n}\right) ^ {2} / 2\right). \\ \end{array}
$$
Lemma B.6 (Ashlagi et al., 2021, Lemma 17).  $\| f\|_{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}^{\beta}}\leq \frac{2L}{\gamma}$
Equipped with the three lemmas, we calculate
$$
\bar {\Lambda} _ {f} ^ {\beta} (\mu) = \int_ {\Omega} \Lambda_ {f} ^ {\beta} (x) d \mu = \sum_ {C \in \Pi_ {l}} \int_ {C} \Lambda_ {f} ^ {\beta} (x) d \mu + \sum_ {C \in \Pi_ {h}} \int_ {C} \Lambda_ {f} ^ {\beta} (x) d \mu . \tag {14}
$$
The first summand above is bounded with high probability using Lemma B.5 and Lemma B.6, since under the event described in Lemma B.5 we have:
$$
\begin{array}{l} \sum_ {C \in \Pi_ {l}} \int_ {C} \Lambda_ {f} ^ {\beta} (x) d \mu \leq \sum_ {C \in \Pi_ {l}} \int_ {C} \frac {2 L}{\gamma} d \mu = \frac {2 L}{\gamma} \sum_ {C \in \Pi_ {l}} \mu (C) \\ \leq \frac {2 L}{\gamma} \cdot 2 q = \frac {L}{4}. \\ \end{array}
$$
In order to bound the second term in Eq. (14), let  $C \in \Pi$ ,  $x' \in C$  and note that by applying Lemma B.4 to  $E \coloneqq S_x \cap C$  we get that  $\Lambda_f^\beta(x') \leq 2 \min_{x \in S_x \cap C} \Lambda_f^\beta(x)$ . Thus, under the high probability event described in Lemma B.5 we have
$$
\begin{array}{l} \sum_ {C \in \Pi_ {h}} \int_ {C} \Lambda_ {f} ^ {\beta} (x) d \mu \leq \sum_ {C \in \Pi_ {h}} \int_ {C} 2 \min  _ {x \in S _ {x} \cap C} \Lambda_ {f} ^ {\beta} (x) d \mu = 2 \sum_ {C \in \Pi_ {h}} \min  _ {x \in S _ {x} \cap C} \Lambda_ {f} ^ {\beta} (x) \mu (C) \\ \leq 4 \sum_ {C \in \Pi_ {h}} \min  _ {x \in S _ {x} \cap C} \Lambda_ {f} ^ {\beta} (x) \mu_ {n} (C) = \frac {4}{n} \sum_ {C \in \Pi_ {h}} \sum_ {x ^ {\prime} \in S _ {x} \cap C} \min  _ {x \in S _ {x} \cap C} \Lambda_ {f} ^ {\beta} (x) \\ \leq \frac {4}{n} \sum_ {C \in \Pi_ {h}} \sum_ {x ^ {\prime} \in S _ {x} \cap C} \Lambda_ {f} ^ {\beta} \left(x ^ {\prime}\right) \leq \frac {4}{n} \sum_ {x ^ {\prime} \in S _ {x}} \Lambda_ {f} ^ {\beta} \left(x ^ {\prime}\right) \leq 4 L, \\ \end{array}
$$
where the last inequality is due to the extension property of Theorem A.3. Overall, plugging these bounds into Eq. (14) and using the union bound to ensure all required events to hold simultaneously, we see that the desired second bullet holds with probability at least  $1 - m \exp(-n\gamma / 4m) - \exp\left(-n(\gamma - \sqrt{m / n})^2 / 2\right)$ . A straightforward computation shows that by our assumption on  $n$  being large enough, this probability exceeds  $1 - \delta / 2$  as required.
We are now ready to finish the proof of Theorem 4.1. Let  $\gamma >0$ . By Corollary B.2, we can construct  $\widehat{f}:S\to [0,1]$  such that with probability at least  $1 - \delta /2:L_S(\widehat{f}) = 0$  and  $\widehat{\Lambda}_{\widehat{f}}^{\beta}\leq$
$5\log^{2}(4n / \delta)L$ . Assuming  $n$  is appropriately large, we further apply Proposition B.3 in order to obtain  $f:\Omega \to [0,1]$  such that with probability at least  $1 - \delta /2:f\in \overline{\mathrm{H}\bar{\mathrm{o}}\mathrm{l}}_{25\log^{2}(4n / \delta)L}^{\beta}(\Omega)$  and also  $L_{S}(f)\leq L_{S}(\widehat{f}) + \gamma (1 + 2L) = \gamma (1 + 2L)$ . By the union bound, we get that with probability at least  $1 - \delta$  :
$$
\begin{array}{l} L _ {\mathcal {D}} (f) = 1. 0 1 L _ {S} (f) + \left(L _ {\mathcal {D}} (f) - 1. 0 1 L _ {S} (f)\right) \\ \leq \gamma (1 + 2L) + \sup_{f\in \overline{\mathrm{H}\bar{\mathrm{o}}\mathrm{l}}_{25\log^{2}(4n / \delta)L}^{\beta}}(L_{\mathcal{D}}(f) - 1.01L_{S}(f)). \\ \stackrel {(*)} {\leq} \frac {\varepsilon}{2} + \frac {\varepsilon}{2} = \varepsilon , \\ \end{array}
$$
where  $(\ast)$  is justified by setting  $\gamma = \Theta (\varepsilon /L)$  and applying Theorem 3.4 for appropriately large  $n$
B.4 Proof of Theorem 5.1
Given a sample  $S = (X_{i},Y_{i})_{i = 1}^{n}\sim \mathcal{D}^{n}$ , denote the empirically smooth class
$$
\widehat {\mathrm {H o l}} := \left\{f: \left\{X _ {1}, \dots , X _ {\lfloor n / 2 \rfloor} \right\}\rightarrow [ 0, 1 ]: \widehat {\Lambda} _ {f} ^ {\beta} \leq 5 \log^ {2} (4 n / \delta) L \right\}.
$$
Consider the following procedure:
1. (Empirical cover) Construct  $h_1, \ldots, h_N \in \widehat{\mathrm{H\ddot{o}l}}$  for maximal  $N$  such that  $\forall i \neq j \in [N]$ :  $\| h_i - h_j \|_{L_1(\mu_n)} \geq \frac{\epsilon}{4}$ .
2. (Run realizable algorithm on cover) For any  $j \in [N]$ , execute the realizable algorithm  $\mathcal{A}_{\mathrm{realizable}}$  of Theorem 4.1 on the "relabeled" dataset  $(X_i, h_j(X_i))_{i=1}^{\lfloor n/2 \rfloor}$ , and obtain  $f_j: \Omega \to [0,1]$ .
3. (ERM) Return  $\arg \min_{f_1,\dots ,f_N}\sum_{i = \lfloor n / 2\rfloor +1}^n |f_j(X_i) - Y_i|$ .
We will now prove that the algorithm above satisfies the theorem. Let  $f^{*} \in \arg \min_{f \in \overline{\mathrm{H}\tilde{\mathrm{o}}l}_{L}^{\beta}(\Omega, \mu)} L_{\mathcal{D}}(f)$ , and note that by Proposition B.1 (as explained in Corollary B.1) we have  $f^{*} \in \widehat{\mathrm{H}\tilde{\mathrm{o}}l}$  with probability at least  $1 - \delta / 2$ . By construction,  $h_1, \ldots, h_N$  is a maximal  $\frac{\epsilon}{4}$ -packing of  $\widehat{\mathrm{H}\tilde{\mathrm{o}}l}$ , which is known to imply that it is also a  $\frac{\epsilon}{4}$ -net [Vershynin, 2018, Lemma 4.2.8] with respect to the metric  $L_1(\mu_n)$ . In particular, this implies that there exists  $j^{*} \in [N]$  such that
$$
\left\| f ^ {*} - h _ {j ^ {*}} \right\| _ {L _ {1} \left(\mu_ {n}\right)} \leq \frac {\epsilon}{4} \Rightarrow L _ {S} \left(h _ {j ^ {*}}\right) \leq L _ {S} \left(f ^ {*}\right) + \frac {\epsilon}{4}.
$$
Further note for any  $j \in [N]: h_j \in \widehat{\mathrm{H\ddot{o}l}}$ , so our realizable algorithm (as manifested in Proposition B.3) for  $\gamma = \Theta(\epsilon / L)$  when fed the "smoothed" labels  $(X_i, h_j(X_i))_{i=1}^{\lfloor n/2 \rfloor}$  will produce  $f_j$  such that  $L_S(f_j) \leq L_S(h_j) \leq \frac{\epsilon}{4}$  and  $\overline{\Lambda}_{f_j}^\beta(\mu) \leq 5\widehat{\Lambda}_{h_j}^\beta \leq 25\log^2(4n/\delta)L$ . In particular
$$
L _ {S} \left(f _ {j ^ {*}}\right) \leq L _ {S} \left(h _ {j ^ {*}}\right) + \frac {\epsilon}{4} \leq L _ {S} \left(f ^ {*}\right) + \frac {\epsilon}{2}.
$$
Finally, by Eq. (1) and Theorem 3.1 (which holds for any measure, in particular for the empirical measure  $\mu_{n}$ )
$$
\begin{array}{l} \log N \leq \log \mathcal {N} _ {\text {H o l}} (\epsilon / 2) \\ \leq \log \mathcal {N} _ {[ l ]} (\widehat {\mathrm {H o l}}, L _ {1} (\mu_ {n}), \epsilon) \\ \leq \log \mathcal {N} _ {\Omega} \left(\left(\frac {\varepsilon}{6 4 0 \log^ {2} (4 n / \delta) L \log (1 / \varepsilon)}\right) ^ {1 / \beta}\right) \cdot \log \left(\frac {1 6 \log_ {2} (1 / \varepsilon)}{\varepsilon}\right). \\ \end{array}
$$
Hence, by a standard Chernoff-Hoeffding bound over the finite class  $\{f_1,\ldots ,f_N\}$ , step (3) of the algorithm yields  $\frac{\epsilon}{2}$  excess risk as long as  $\frac{n}{2} = \Omega \left(\frac{\log(N) + \log(1 / \delta)}{\epsilon^2}\right)$ .
<sup>6</sup>We assume without loss of generality that the infimum is obtained. Otherwise we can take a function whose loss is arbitrarily close enough to the optimal value and continue with the proof verbatim.
B.5 Proof of Theorem 6.1
We start by providing a simple structural result which we will use for our lower bound construction, showing that in any metric space there exists a sufficiently isolated point from a large enough subset.
Lemma B.7. There exists a point  $x_0 \in \Omega$  and a subset  $K \subset \Omega$  such that
$\forall x\in K:\rho (x_0,x)\geq \frac{\mathrm{diam}(\Omega)}{4}$
-  $\forall x \neq y \in K: \rho(x, y) \geq (\varepsilon / L)^{1 / \beta}$ .
$|K| = \left\lfloor \frac{\mathcal{N}_{\Omega}((\varepsilon / L)^{1 / \beta})}{2}\right\rfloor$
Proof. Denote  $D \coloneqq \mathrm{diam}(\Omega)$ , let  $x_0, x_1$  be two points such that  $\rho(x_0, x_1) > D/2$ , and let  $\Pi = \{C_0, C_1\}$  be a Voronoi partition of  $\Omega$  induced by  $\{x_0, x_1\}$ . For  $\gamma > 0$ , let  $N_{\gamma}$  be a maximal  $\gamma$ -packing of  $\Omega$ . By the pigeonhole principle there must exist a cell  $C_i \in \Pi$  such that  $|C_i \cap N_{\gamma}| \geq |N_{\gamma}|/2$ , which we assume without loss of generality to be  $C_1$ . Now note that any  $x \in C_1$  satisfies  $\rho(x, x_0) \geq \frac{1}{2}\rho(x, x_0) + \frac{1}{2}\rho(x, x_1) \geq \frac{1}{2}\rho(x_0, x_1) > D/4$ . Finally, set  $\gamma := \varepsilon^{1/\beta}$  and let  $K \subset C_1 \cap N_{\gamma}$  be any subset of size  $\left\lfloor \frac{\mathcal{N}_{\Omega}((\varepsilon/L)^{1/\beta})}{2} \right\rfloor$ .
Given  $x_0, K$  from the lemma above, we denote  $\overline{K} = \{x_0\} \cup K$  and define the distribution  $\mu$  over  $\Omega$  supported on  $\overline{K}$  such that  $\mu(x_0) = 1 - \frac{\varepsilon}{2}$  and  $\mu(x) = \frac{\varepsilon}{2|K|}$  for all  $x \in K$ . From now on, the proof is similar to a classic lower bound strategy for VC classes in the realizable case (e.g. Kearns and Vazirani, 1994, Proof of Theorem 3.5). To that end, it is enough to provide a distribution over functions in  $\overline{\mathrm{Hoi}}_L^\beta(\Omega, \mu)$  such that with constant probability any algorithm must suffer significant loss for some function supported by the distribution.
We define such a distribution over functions  $\overline{f}:\overline{K}\to \{0,1\}$  as follows:  $\operatorname *{Pr}[\overline{f} (x_0) = 0] = 1$  , while for any  $x\in K:\operatorname *{Pr}[\overline{f} (x) = 0] = \operatorname *{Pr}[\overline{f} (x) = 1] = \frac{1}{2}$  independently of other points. We will now show that any such  $\overline{f}:\overline{K}\rightarrow \{0,1\}$  is average Hölder smooth with respect to  $\mu$  . Indeed, for every  $x\in K$  ..
$$
\Lambda_ {\overline {{f}}} ^ {\beta} (x) = \sup  _ {x ^ {\prime} \in \overline {{K}} \backslash \{x \}} \frac {| \overline {{f}} (x) - \overline {{f}} \left(x ^ {\prime}\right) |}{\rho (x , x ^ {\prime}) ^ {\beta}} \leq \frac {1}{\varepsilon / L} = \frac {L}{\varepsilon},
$$
$$
\Lambda_ {\overline {{f}}} ^ {\beta} \left(x _ {0}\right) = \sup  _ {x ^ {\prime} \in \overline {{K}} \backslash \{x _ {0} \}} \frac {| \overline {{f}} \left(x _ {0}\right) - \overline {{f}} \left(x ^ {\prime}\right) |}{\rho \left(x _ {0} , x ^ {\prime}\right) ^ {\beta}} \leq \frac {1}{\operatorname {d i a m} (\Omega) / 4} = \frac {4}{\operatorname {d i a m} (\Omega)},
$$
$$
\overline {{\Lambda}} _ {f} ^ {\beta} (x) = \mu (x _ {0}) \Lambda_ {\overline {{f}}} ^ {\beta} (x _ {0}) + \sum_ {x \in K} \mu (x) \Lambda_ {\overline {{f}}} ^ {\beta} (x) \leq \frac {4}{D} + \frac {L}{2} \leq L.
$$
Finally, we define the (random) function  $f^{*}:\Omega \to [0,1]$  to be the  $\beta$ -PMSE extension of  $\overline{f}$  from  $\overline{K}$  to  $\Omega$  as defined in Definition A.2, and note that  $f^{*}$  satisfies the required smoothness assumption. Setting  $\mathcal{D}$  over  $\Omega \times [0,1]$  to have marginal  $\mu$  and  $Y = f^{*}(X)$ , we ensure that  $\mathcal{D}$  is indeed realizable by  $\overline{\mathrm{H}\mathrm{o}\mathrm{l}}_L^\beta (\Omega)$ .
Now assume  $A$  is a learning algorithm which is given a sample  $S$  of size  $|S| \leq \frac{|K|}{4\varepsilon}$  and produces  $A(S): \Omega \to [0,1]$ . We call a point  $x \in K$  "misclassified" by the algorithm if  $|A(S)(x) - f^{*}(x)| \geq \frac{1}{2}$ , and denote the set of misclassified points by  $M \subset K$ . Recalling that  $\forall x \in K: \operatorname{Pr}[f(x) = 0] = \operatorname{Pr}[f(x) = 1] = \frac{1}{2}$  independently, and that  $\mu(x) = \frac{\varepsilon}{2|K|}$ , we observe that with probability at least  $\frac{1}{2}$  the algorithm will misclassify more than  $|K| / 2$  points. Thus, we get that with probability at least  $\frac{1}{2}$ :
$$
L _ {\mathcal {D}} (A (S)) = \underset {X \sim \mu} {\mathbb {E}} \left[ | A (S) (X) - f ^ {*} (X) | \right] \geq \sum_ {x \in M} \mu (x) \cdot | A (S) (x) - f ^ {*} (x) | \geq \frac {| K |}{2} \cdot \frac {\varepsilon}{2 | K |} \cdot \frac {1}{2} = \frac {\varepsilon}{8}.
$$
Indeed, denoting  $C = K \setminus M$  we see that  $\operatorname*{Pr}[|C| \geq |K|/8] \leq \frac{8}{|K|} \cdot \mathbb{E}[|C|] = \frac{8}{|K|} \cdot \frac{|S|}{2} \cdot \mu(K) \leq \frac{8}{|K|} \cdot \frac{|K|}{8\varepsilon} \cdot \frac{\varepsilon}{2} = \frac{1}{2}$ .
By rescaling  $\varepsilon$ , we see that in order to obtain  $L_{\mathcal{D}}(A(S)) \leq \varepsilon$  the sample size must be of size
$$
\Omega \left(\frac {| K |}{\varepsilon}\right) = \Omega \left(\frac {\mathcal {N} _ {\Omega} ((\varepsilon / L) ^ {1 / \beta})}{\varepsilon}\right).
$$
B.6 Proofs from Section 7
Proof of Claim 7.1. Let  $\beta \in (0,1)$ . Consider the unit segment  $\Omega = [0,1]$  with the standard metric, equipped with the probability measure  $\mu$  with density  $\frac{d\mu}{dx} = \frac{1}{Z} |x - \frac{1}{2}|^{\frac{\beta - 1}{2}}$  (where  $Z = \int_0^1 |x - \frac{1}{2}|^{\frac{\beta - 1}{2}} < \infty$  is a normalizing constant). We examine the function  $f(x) = \mathbf{1}[x > \frac{1}{2}]$  which is clearly not Hölder continuous since it is discontinuous. Furthermore,
$$
\begin{array}{l} \mu \left(\left\{x: \Lambda_ {f} ^ {1} (x) \geq t \right\}\right) = \mu \left(\left\{\left| x - \frac {1}{2} \right| \leq \frac {1}{t} \right\}\right) = \frac {2}{Z} \int_ {0} ^ {1 / t} x ^ {\frac {\beta - 1}{2}} d x \asymp t ^ {- \frac {\beta + 1}{2}} \\ \Rightarrow \widetilde {\Lambda} _ {f} ^ {1} = \sup  _ {t > 0} t \cdot \mu (\{x: \Lambda_ {f} ^ {1} (x) \geq t \}) \asymp \sup  _ {t > 0} t ^ {\frac {1 - \beta}{2}} = \infty , \\ \end{array}
$$
hence  $f \notin \widetilde{\mathrm{Lip}}_M(\Omega, \mu)$  for all  $M > 0$ . On the other hand,  $\Lambda_f^\beta(x) = \frac{1}{|x - \frac{1}{2}|^\beta}$  so
$$
\overline {{\Lambda}} _ {f} ^ {\beta} = \int_ {0} ^ {1} \Lambda_ {f} ^ {\beta} (x) d \mu = \frac {1}{Z} \int_ {0} ^ {1} \frac {\left| x - \frac {1}{2} \right| ^ {\frac {\beta - 1}{2}}}{\left| x - \frac {1}{2} \right| ^ {\beta}} d x = \frac {1}{Z} \int_ {0} ^ {1} \frac {1}{\left| x - \frac {1}{2} \right| ^ {\frac {\beta + 1}{2}}} d x <   \infty ,
$$
thus  $f \in \overline{\mathrm{H}\vec{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega)$  for some  $L < \infty$ . Note that by normalizing the function, the claim holds even for  $L = 1$ .
Proof of Claim 7.2. Let  $\beta \in (0,1)$ . Consider the unit segment  $\Omega = [0,1]$  with the standard metric, equipped with the probability measure  $\mu$  with density  $\frac{d\mu}{dx} = \frac{1}{Z} |x - \frac{1}{2}|^{\beta - 1}$  (where  $Z = \int_0^1 |x - \frac{1}{2}|^{\beta - 1} dx < \infty$  is a normalizing constant). We examine the function  $f(x) = \mathbf{1}[x > \frac{1}{2}]$ . Note that for any  $x \neq \frac{1}{2}: \Lambda_f^1(x) = \frac{1}{|x - \frac{1}{2}|}$ , hence
$$
\mu \left(\left\{x: \Lambda_ {f} ^ {1} (x) \geq t \right\}\right) = \mu \left(\left\{x: \left| x - \frac {1}{2} \right| \leq \frac {1}{t} \right\}\right) = \frac {2}{Z} \int_ {0} ^ {1 / t} x ^ {\beta - 1} d x \asymp t ^ {- \beta}.
$$
$$
\widetilde {\Lambda} _ {f} ^ {1} = \sup  _ {t > 0} t \cdot \mu (\{x: \Lambda_ {f} ^ {1} (x) \geq t \}) \asymp \sup  _ {t > 0} t ^ {1 - \beta} = \infty ,
$$
hence  $f \notin \widetilde{\mathrm{Lip}}_M(\Omega, \mu)$  for all  $M > 0$ . Furthermore, for  $x \neq \frac{1}{2} : \Lambda_f^\beta(x) = \frac{1}{|x - \frac{1}{2}|^\beta}$  so
$$
\overline {{\Lambda}} _ {f} ^ {\beta} = \int_ {0} ^ {1} \frac {1}{| x - \frac {1}{2} | ^ {\beta}} d \mu = \frac {1}{Z} \int_ {0} ^ {1} \frac {1}{| x - \frac {1}{2} |} d x = \infty ,
$$
hence  $f \notin \widetilde{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}}_M^\beta (\Omega ,\mu)$  for all  $M > 0$ . On the other hand
$$
\begin{array}{l} \mu \left(\left\{x: \Lambda_ {f} ^ {\beta} (x) \geq t \right\}\right) = \mu \left(\left\{\left| x - \frac {1}{2} \right| \leq t ^ {- 1 / \beta} \right\}\right) = \frac {2}{Z} \int_ {0} ^ {t ^ {- 1 / \beta}} x ^ {\beta - 1} d x \asymp t ^ {- 1} \\ \Rightarrow \widetilde {\Lambda} _ {f} ^ {\beta} = \sup  _ {t > 0} t \cdot \mu (\{x: \Lambda_ {f} ^ {\beta} (x) \geq t \}) <   \infty , \\ \end{array}
$$
thus  $f \in \widetilde{\mathrm{H}\tilde{o}\mathrm{l}}_L^\beta (\Omega)$  for some  $L < \infty$ . Note that by normalizing the function, the claim holds even for  $L = 1$ .