SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conclude that if  $\mathrm{diam}(A) < \infty$  then  $\lim_{\varepsilon \to 0^{+}}\Phi_{x}(\varepsilon)$  indeed exists.
It remains to prove the optimality of the  $\beta$ -slope. Throughout the proof we will denote for any  $u \neq v \in \Omega$ :
$$
S (u, v) := \frac {\left| F _ {A} (u) - F _ {A} (v) \right|}{\rho (u , v) ^ {\beta}},
$$
and for any point  $x\in \Omega$  , subset  $B\subset \Omega$  and function  $g:\Omega \to [0,1]$  we let
$$
\Lambda_{g}^{\beta}(x,B):= \sup_{y\in B\setminus \{x\}}\frac{|g(x) - g(y)|}{\rho(x,y)^{\beta}}  .
$$
The proof is split into three claims.
Claim I.  $\forall x\in \Omega \setminus A:\Lambda_{F_A}^\beta (x,A)\leq \Lambda_f^\beta (x,A).$
Let  $x \in \Omega \setminus A$ , and let  $(u^{*}, v^{*}) \in A \times A$  be its associated maximizer of  $R_{x}$ . Recall Eq. (4) from which we can deduce that  $F_{A}(u^{*}) \leq F_{A}(x) \leq F_{A}(v^{*})$ . Also note that a simple rearrangement based on Eq. (3) (and the fact that  $f$  and  $F_{A}$  agree on  $A$ ) shows that  $S(u^{*}, x) = R_{x}(u^{*}, v^{*}) = S(x, v^{*})$ . Furthermore, we claim that  $\Lambda_{F_{A}}^{\beta}(x, A) := \sup_{y \in A \setminus \{x\}} S(x, y) = S(x, u^{*})$ . If this were not true then we would have  $S(x, y) > S(x, u^{*}) = S(x, v^{*})$  for some  $y \in A \setminus \{x, u^{*}, v^{*}\}$ . Using the median inequality, if  $f(y) \geq f(x)$  this implies
$$
R _ {x} (u ^ {*}, y) = \frac {f (y) - f (u ^ {*})}{\rho (x , y) ^ {\beta} + \rho (x , u ^ {*}) ^ {\beta}} = \frac {F _ {A} (y) - F _ {A} (x) + F _ {A} (x) - F _ {A} (u ^ {*})}{\rho (x , y) ^ {\beta} + \rho (x , u ^ {*}) ^ {\beta}} > S (x, u ^ {*}) = R _ {x} (u ^ {*}, v ^ {*}),
$$
while if  $f(y) < f(x)$  then
$$
R _ {x} (y, v ^ {*}) = \frac {f (v ^ {*}) - f (y)}{\rho (x , v ^ {*}) ^ {\beta} + \rho (x , y) ^ {\beta}} = \frac {F _ {A} (v ^ {*}) - F _ {A} (x) + F _ {A} (x) - F _ {A} (y)}{\rho (x , v ^ {*}) ^ {\beta} + \rho (x , y) ^ {\beta}} > S (x, v ^ {*}) = R _ {x} (u ^ {*}, v ^ {*}),
$$
both contradicting the maximizing property of  $(u^{*},v^{*})$  - so indeed  $\Lambda_{F_A}^\beta (x,A) = S(x,u^*) = S(x,v^*)$ . In particular, we see that if  $F_{A}(x)\geq f(x)$  then
$$
\Lambda_ {f} ^ {\beta} (x, A) = \sup  _ {y \in A \backslash \{x \}} \frac {| f (y) - f (x) |}{\rho (y , x) ^ {\beta}} \geq \frac {f (v ^ {*}) - f (x)}{\rho (v ^ {*} , x) ^ {\beta}} \geq \frac {F _ {A} (v ^ {*}) - F _ {A} (x)}{\rho (v ^ {*} , x) ^ {\beta}} = S (x, v ^ {*}) = \Lambda_ {F _ {A}} ^ {\beta} (x, A),
$$
while if  $F_A(x) < f(x)$  then
$$
\Lambda_ {f} ^ {\beta} (x, A) = \sup _ {y \in A \backslash \{x \}} \frac {| f (x) - f (u) |}{\rho (x , y) ^ {\beta}} \geq \frac {f (x) - f (u ^ {*})}{\rho (x , u ^ {*}) ^ {\beta}} > \frac {F _ {A} (x) - F _ {A} (u ^ {*})}{\rho (x , u ^ {*}) ^ {\beta}} = S (x, u ^ {*}) = \Lambda_ {F _ {A}} ^ {\beta} (x, A),
$$
proving Claim I in either case.
Claim II.  $\forall x\in \Omega \setminus A:\Lambda_{F_A}^\beta (x,\Omega \setminus A)\leq \Lambda_{F_A}^\beta (x,A),$  in particular  $\Lambda_{F_A}^{\beta}(x,\Omega) = \Lambda_{F_A}^{\beta}(x,A).$
It suffices to show that for any  $x, y \in \Omega \setminus A$ :
$$
S (x, y) \leq \min  \left\{\Lambda_ {F _ {A}} ^ {\beta} (x, A), \Lambda_ {F _ {A}} ^ {\beta} (y, A) \right\},
$$
since taking the supremum of the left hand side over  $y \in \Omega \setminus A$  shows the claim. Let  $(u_x^*, v_x^*), (u_y^*, v_y^*)$  the associated maximizers of  $R_x, R_y$  respectively, and note that by definition we have
$$
\Lambda_ {F _ {A}} ^ {\beta} (x, A) = \sup  _ {z \in A \backslash \{x \}} S (x, z) \geq \max  \left\{S \left(x, u _ {y} ^ {*}\right), S \left(x, v _ {y} ^ {*}\right) \right\}. \tag {5}
$$
We assume without loss of generality that  $\Lambda_{F_A}^\beta (x,A)\leq \Lambda_{F_A}^\beta (y,A)$ , and recall that by Eq. (4) we can deduce that  $F_{A}(u_{x}^{*})\leq F_{A}(x)\leq F_{A}(v_{x}^{*})$  and  $F_{A}(u_{y}^{*})\leq F_{A}(y)\leq F_{A}(v_{y}^{*})$ . Now suppose by contradiction that  $S(x,y) > \Lambda_{F_A}^\beta (x,A)$ . If  $F_{A}(x)\leq F_{A}(y)$  then
$$
\begin{array}{l} F _ {A} (v _ {y} ^ {*}) = F _ {A} (x) + \rho (x, y) ^ {\beta} S (x, y) + \rho (y, v _ {y} ^ {*}) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (y, A) \\ > F _ {A} (x) + \rho (x, y) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A) + \rho (y, v _ {y} ^ {*}) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A) \\ \geq F _ {A} (x) + \rho (x, v _ {y} ^ {*}) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A), \\ \end{array}
$$
thus  $S(x, v_y^*) = \frac{|F_A(x) - F_A(v_y^*)|}{\rho(x, v_y^*)^\beta} > \Lambda_{F_A}^\beta(x, A)$  which contradicts Eq. (5). On the other hand, if  $F_A(x) > F_A(y)$  then
$$
\begin{array}{l} F _ {A} (x) = F _ {A} \left(u _ {y} ^ {*}\right) + \rho \left(u _ {y} ^ {*}, y\right) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (y, A) + \rho (y, x) ^ {\beta} S (x, y) \\ > F _ {A} \left(u _ {y} ^ {*}\right) + \rho \left(u _ {y} ^ {*}, y\right) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A) + \rho (y, x) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A) \\ \geq F _ {A} \left(u _ {y} ^ {*}\right) + \rho \left(u _ {y} ^ {*}, x\right) ^ {\beta} \Lambda_ {F _ {A}} ^ {\beta} (x, A), \\ \end{array}
$$
thus  $S(x, u_y^*) = \frac{|F_A(x) - F_A(u_y^*)|}{\rho(x, u_y^*)^\beta} > \Lambda_{F_A}^\beta(x, A)$  which contradicts Eq. (5), and proves claim II.
Claim III.  $\forall x\in A:\Lambda_{F_A}^\beta (x,\Omega) = \Lambda_{F_A}^\beta (x,A)\leq \Lambda_f^\beta (x,\Omega).$
Let  $x \in A$ . Assume towards contradiction that there exists  $y \notin A$  such that
$$
\Lambda_ {F _ {A}} (x, \Omega) \geq S (x, y) > \Lambda_ {F _ {A}} ^ {\beta} (x, A).
$$
We denote by  $(u_y^*, v_y^*) \in A \times A$  the maximizer of  $R_y(\cdot, \cdot)$ . Recall that since  $x \in A$ , in the proof of Claim I we showed that  $S(x, y) \leq S(y, u_y^*) = S(y, v_y^*)$ . If  $F_A(x) \leq F_A(y) \leq F_A(v_y^*)$  then
$$
\begin{array}{l} S (x, v _ {y} ^ {*}) = \frac {F _ {A} (v _ {y} ^ {*}) - F _ {A} (x)}{\rho (v _ {y} ^ {*} , x) ^ {\beta}} \geq \frac {F _ {A} (v _ {y} ^ {*}) - F _ {A} (y) + F _ {A} (y) - F _ {A} (x)}{\rho (v _ {y} ^ {*} , y) ^ {\beta} + \rho (x , y) ^ {\beta}} \\ \geq \min  \{S (y, v _ {y} ^ {*}), S (x, y) \} = S (x, y) > \Lambda_ {F _ {A}} ^ {\beta} (x, A), \\ \end{array}
$$
while on the other hand if  $F_A(x) > F_A(y) \geq F_A(u_y^*)$  then
$$
\begin{array}{l} S (x, u _ {y} ^ {*}) = \frac {F _ {A} (x) - F _ {A} (u _ {y} ^ {*})}{\rho (x , u _ {y} ^ {*}) ^ {\beta}} \geq \frac {F _ {A} (x) - F _ {A} (y) + F _ {A} (y) - F _ {A} (u _ {y} ^ {*})}{\rho (x , y) ^ {\beta} + \rho (u _ {y} ^ {*} , y) ^ {\beta}} \\ \geq \min  \{S (x, y), S (y, u _ {y} ^ {*}) \} = S (x, y) > \Lambda_ {F _ {A}} ^ {\beta} (x, A), \\ \end{array}
$$
where in both calculations we used the median inequality. Both inequalities above contradict the definition of  $\Lambda_{F_A}^\beta (x,A)$ , thus proving Claim III.
Combining the ingredients. We are now ready to finish the proof. For  $x \in \Omega$ , if  $x \in A$  then Claim III provides the desired inequality. Otherwise, if  $x \in \Omega \setminus A$  then
$$
\Lambda_ {F _ {A}} ^ {\beta} (x, \Omega) \stackrel {{\text {C l a i m I I}}} {=} \Lambda_ {F _ {A}} ^ {\beta} (x, A) \stackrel {{\text {C l a i m I}}} {\leq} \Lambda_ {f} ^ {\beta} (x, A) \leq \Lambda_ {f} ^ {\beta} (x, \Omega).
$$
We start by stating a strengthened version of the triangle inequality (also known as the "snowflake" triangle inequality) which we will use later on. For any  $\beta \in (0,1]$ ,  $x \neq y$ ,  $z \in \Omega$ :
$$
\rho (x, y) ^ {\beta} \leq \rho (x, z) ^ {\beta} + \rho (z, y) ^ {\beta}. \tag {6}
$$
Indeed, this follows from
$$
\begin{array}{l} \frac {\rho (x , z) ^ {\beta} + \rho (z , y) ^ {\beta}}{\rho (x , y) ^ {\beta}} \geq \frac {\rho (x , z) ^ {\beta} + \rho (z , y) ^ {\beta}}{(\rho (x , z) + \rho (z , y)) ^ {\beta}} = \left(\frac {\rho (x , z)}{\rho (x , z) + \rho (z , y)}\right) ^ {\beta} + \left(\frac {\rho (z , y)}{\rho (x , z) + \rho (z , y)}\right) ^ {\beta} \\ \geq \left(\frac {\rho (x , z)}{\rho (x , z) + \rho (z , y)}\right) + \left(\frac {\rho (z , y)}{\rho (x , z) + \rho (z , y)}\right) = 1. \\ \end{array}
$$
Let  $0 < \varepsilon < \frac{1}{4}$ , denote  $K := \lceil \log_2(1 / \varepsilon) \rceil$ ,  $\varepsilon' := \frac{1}{(K + 1)2^K}$  and note that
$$
\varepsilon^ {\prime} \geq \frac {1}{\left(\log_ {2} (1 / \varepsilon) + 2\right) 2 ^ {\log_ {2} (1 / \varepsilon) + 1}} = \frac {\varepsilon}{2 \left(\log_ {2} (1 / \varepsilon) + 2\right)} \geq \frac {\varepsilon}{4 \log_ {2} (1 / \varepsilon)}. \tag {7}
$$
Let  $N = \{x_{1},\ldots ,x_{|N|}\}$  be a  $\left(\frac{\varepsilon'}{32L}\right)^{1 / \beta}$ -net of  $\Omega$  of size  $|N| = \mathcal{N}_{\Omega}\left(\left(\frac{\varepsilon'}{32L}\right)^{1 / \beta}\right)$ , and let  $\Pi = \{C_1,\dots ,C_{|N|}\}$  be its induced Voronoi partition. We define  $\mathcal{B} = \{[l_j,u_j]\}_{j\in J}\subset [0,1]^{\Omega}\times [0,1]^{\Omega}$  to be the pairs of functions constructed as follows:
-  $l, u$  are both constant over every cell  $C_i \in \Pi$ , and map each cell to a value in  $\{0, \frac{\varepsilon'}{2}, \varepsilon', \frac{3\varepsilon'}{2}, \dots, 1\}$ .
- Choose some cells  $S_1 \subset \Pi$  such that  $\mu(\bigcup_{C_i \in S_1} C_i) \leq \varepsilon'$  and set for any  $C_i \in S_1 : l|_{C_i} = 0$ ,  $u|_{C_i} = 1$ .
- For  $m = 2, \ldots, K$  choose some "unchosen" cells  $S_{m} \subset \Pi \setminus \bigcup_{j < m} S_{j}$  such that  $\mu(\bigcup_{C_{i} \in S_{m}} C_{i}) \leq 2^{m - 1} \varepsilon'$  and set for any  $C_{i} \in S_{m} : l|_{C_{i}} \in \{0, \frac{1}{2^{m}}, \frac{2}{2^{m}}, \ldots, \frac{2^{m} - 2}{2^{m}}\}$ ,  $u|_{C_{i}} = l + \frac{1}{2^{m - 1}}$ .
- In the "remaining" cells  $S_{K+1} \coloneqq \Pi \setminus \bigcup_{j \leq K} S_j$  set for any  $C_i \in S_{K+1}$ :
$$
l | _ {C _ {i}} \in \left\{0, \frac {1}{2 ^ {K + 1}}, \frac {2}{2 ^ {K + 1}}, \dots , \frac {2 ^ {K + 1} - 2}{2 ^ {K + 1}} \right\}, u | _ {C _ {i}} = l + \frac {1}{2 ^ {K}}.
$$
Notice that for any  $[l,u]\in \mathcal{B}$  we have
$$
\begin{array}{l} \| l - u \| _ {L ^ {1} (\mu)} = \sum_ {C _ {i} \in \Pi} \int_ {C _ {i}} | l (x) - u (x) | d \mu (x) = \sum_ {m = 1} ^ {K + 1} \sum_ {C _ {i} \in S _ {m}} \int_ {C _ {i}} | l (x) - u (x) | d \mu (x) \\ = \sum_ {m = 1} ^ {K + 1} \sum_ {C _ {i} \in S _ {m}} \int_ {C _ {i}} \frac {1}{2 ^ {m - 1}} d \mu (x) = \sum_ {m = 1} ^ {K + 1} \frac {1}{2 ^ {m - 1}} \sum_ {C _ {i} \in S _ {m}} \mu (C _ {i}) \\ = \sum_ {m = 1} ^ {K + 1} \frac {2 ^ {m - 1} \varepsilon^ {\prime}}{2 ^ {m - 1}} = \varepsilon^ {\prime} (K + 1) = \frac {1}{2 ^ {K}} \leq \varepsilon . \\ \end{array}
$$
Furthermore, we can bound  $|\mathcal{B}|$  by noticing that any such  $l$  is defined by its values over  $|N|$  cells who all belong to  $\{0, \frac{\varepsilon'}{2}, \varepsilon', \dots, 1\}$ , and that once  $l$  is fixed then any associated  $u$  has at most  $K + 1$  possible values over each cell since it equals  $l + \frac{1}{2^{m - 1}}$  for some  $m \in [K + 1]$ . Thus
$$
| \mathcal {B} | \leq (K + 1) \left(\frac {8}{\varepsilon^ {\prime}}\right) ^ {| N |} \leq \log_ {2} \left(\frac {1}{\varepsilon}\right) \cdot \left(\frac {1 6 \log_ {2} (1 / \varepsilon)}{\varepsilon}\right) ^ {\mathcal {N} \left(\left(\frac {\varepsilon}{1 2 8 L \log (1 / \varepsilon)}\right) ^ {1 / \beta}\right)},
$$
where the last inequality uses Eq. (7) and definition of  $K$ . In order to finish the proof, in remains to show that  $\mathcal{B}$  indeed cover  $\widetilde{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega ,\mu)$  as brackets. Namely, that for any  $f\in \widetilde{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega ,\mu)$  there exist  $[l,u]\in \mathcal{B}$  such that  $l\leq f\leq u$ . To that end, let  $f\in \widetilde{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega ,\mu)$ . Denote
$$
S _ {1} ^ {f} := \left\{C _ {i} \in \Pi : \forall x \in C _ {i}: \Lambda_ {f} ^ {\beta} (x) \geq \frac {L}{\varepsilon^ {\prime}} \right\}
$$
and notice that  $\bigcup \{C_i\in S_1^f\} \subseteq \{x:\Lambda_f^\beta (x)\geq L / \varepsilon '\} \Rightarrow \mu (\bigcup \{C_i\in S_1^f\})\leq \varepsilon '.$  Hence we can pick  $[l,u]\in \mathcal{B}$  such that  $(l|_{C_i},u|_{C_i})\equiv (0,1)$  for any  $C_i\in S_1^f$  (serving as  $S_{1}$  in their construction). Clearly any such  $l,u$  bound  $f$  over these cells. Furthermore, for  $m = 2,\ldots ,K$  we denote
$$
S _ {m} ^ {f} := \left\{C _ {i} \in \Pi \setminus \bigcup_ {j <   m} S _ {j} ^ {f}: \forall x \in C _ {i}: \Lambda_ {f} ^ {\beta} (x) \geq \frac {L}{2 ^ {m - 1} \varepsilon^ {\prime}} \right\},
$$
and notice that  $\bigcup \{C_i\in S_m^f\} \subseteq \{x:\Lambda_f^\beta (x)\geq L / (2^{m - 1}\varepsilon ')\} \Rightarrow \mu (\bigcup \{C_i\in S_m^f\})\leq 2^{m - 1}\varepsilon '.$  Consequently we can let  $S_{m}^{f}$  serve as  $S_{m}$  in the construction of  $[l,u]\in \mathcal{B}$ , assuming we will show such a choice can serve as a bracket of  $f$  over such cells. Indeed, for any  $x\in C_i$  we have
$$
| f (x) - f (z _ {i}) | \leq \Lambda_ {f} ^ {\beta} (z _ {i}) \cdot \rho (x, z _ {i}) ^ {\beta} \stackrel {{E q.}} {{\leq}} ^ {(6)} \frac {L}{2 ^ {m - 2} \varepsilon^ {\prime}} \cdot \frac {2 \varepsilon^ {\prime}}{3 2 L} = \frac {1}{2 ^ {m + 2}},
$$
which by the triangle inequality shows in particular that for any  $x, y \in C_i$ :
$$
| f (x) - f (y) | \leq | f (x) - f (z _ {i}) | + | f (z _ {i}) - f (y) | \leq \frac {1}{2 ^ {m + 1}} = \frac {1}{4 \cdot 2 ^ {m - 1}}.
$$
So clearly there exists  $\alpha_{i}\in \{0,\frac{1}{2^{m}},\frac{2}{2^{m}},\ldots ,\frac{2^{m} - 2}{2^{m}}\}$  such that  $\alpha_{i}\leq f|_{C_{i}}\leq \alpha_{i} + \frac{1}{2^{m - 1}}$  and by setting  $l|_{C_i},u|_{C_i} = (\alpha_i,\alpha_i + \frac{1}{2^{m - 1}})$  for any  $C_i\in S_m^f$  we ensure the bracketing property. Finally, for any of the remaining cells  $S_{K + 1}^{f}\coloneqq \Pi \setminus \bigcup_{j\leq K}S_{j}^{f}$  we get by construction that  $\exists z_{i}\in C_{i}:\Lambda_{f}^{\beta}(z_{i}) <   \frac{L}{2^{K}\varepsilon^{\prime}}$  (otherwise they would satisfy the condition for some previously constructed  $S_{m}^{f}$ ). Hence for any  $x\in C_i$  we have
$$
| f (x) - f (z _ {i}) | \leq \Lambda_ {f} ^ {\beta} (z _ {i}) \cdot \rho (x, z _ {i}) ^ {\beta} \stackrel {{E q.}} {{\leq}} ^ {(6)} \frac {L}{2 ^ {K} \varepsilon^ {\prime}} \cdot \frac {2 \varepsilon^ {\prime}}{3 2 L} = \frac {1}{2 ^ {K + 4}},
$$
which by the triangle inequality shows that for any  $x, y \in C_i$ :
$$
| f (x) - f (y) | \leq \frac {1}{2 ^ {K + 3}} = \frac {1}{8 \cdot 2 ^ {K}}.
$$
So as before, there clearly exists  $\alpha_{i}\in \{0,\frac{1}{2^{K + 1}},\frac{2}{2^{K + 1}},\ldots ,\frac{2^{K + 1} - 2}{2^{K + 1}}\}$  such that  $\alpha_{i}\leq f|_{C_{i}}\leq \alpha_{i} + \frac{1}{2^{K}}$  and by setting  $l|_{C_i},u|_{C_i} = (\alpha_i,\alpha_i + \frac{1}{2^K})$  for any  $C_i\in S_m^f$  we ensure the bracketing property over all of  $\Omega$ , which finishes the proof.
B.2 Proof of Proposition 3.2
Recalling that the realizability assumption ensures a "perfect" predictor  $f^{*} \in \mathcal{F}$ , we start by introducing the loss class  $\mathcal{L}_{\mathcal{F}} \subset [0,1]^{\Omega}$ :
$$
\mathcal {L} _ {\mathcal {F}} = \left\{\ell_ {f} (x) := | f (x) - f ^ {*} (x) |: f \in \mathcal {F} \right\}.
$$
Fix  $\alpha > 0$ . We observe that  $\mathcal{L}_{\mathcal{F}}$  is no larger than  $\mathcal{F}$  in terms of bracketing entropy, namely
$$
\mathcal {N} _ {[ ]} \left(\mathcal {L} _ {\mathcal {F}}, L _ {1} (\mu), \alpha\right) \leq \mathcal {N} _ {[ ]} \left(\mathcal {F}, L _ {1} (\mu), \alpha\right). \tag {8}
$$
Indeed, suppose we are given an  $\alpha$ -bracketing of  $\mathcal{F}$  denoted by  $\mathcal{B}_{\alpha}$ , and denote for any  $f \in \mathcal{F}$  by  $[l_f, u_f] \in \mathcal{B}_{\alpha}$  its associated bracket. Then any  $\ell_f \in \mathcal{L}_{\mathcal{F}}$  is inside the bracket  $[l_{\ell_f}, u_{\ell_f}]$  where
$$
l _ {\ell_ {f}} := \max  \left\{0, \min  \left\{l _ {f} - f ^ {*}, f ^ {*} - u _ {f} \right\} \right\},
$$
$$
u _ {\ell_ {f}} := \min  \{1, \max  \{u _ {f} - f ^ {*}, f ^ {*} - l _ {f} \} \}.
$$
It is straightforward to verify that  $\| u_{\ell_f} - l_{\ell_f}\|_{L_1(\mu)}\leq \| u_f - l_f\|_{L_1(\mu)}\leq \alpha$  , and clearly the set of all such brackets is of size at most  $|\mathcal{B}_{\alpha}|$  , yielding Eq. (8).
Now notice that for any  $f\in \mathcal{F}$  ..
$$
L _ {\mathcal {D}} (f) - 1. 0 1 L _ {S} (f) = \| \ell_ {f} \| _ {L _ {1} (\mu)} - 1. 0 1 \| \ell_ {f} \| _ {L _ {1} (\mu_ {n})} \leq \alpha + \| l _ {\ell_ {f}} \| _ {L _ {1} (\mu)} - 1. 0 1 \| l _ {\ell_ {f}} \| _ {L _ {1} (\mu_ {n})},
$$
$$
\sup  _ {f \in \mathcal {F}} \left(L _ {\mathcal {D}} (f) - 1. 0 1 L _ {S} (f)\right) \leq \alpha + \max  _ {l _ {\ell_ {f}}} \left(\left\| l _ {\ell_ {f}} \right\| _ {L _ {1} (\mu)} - 1. 0 1 \left\| l _ {\ell_ {f}} \right\| _ {L _ {1} \left(\mu_ {n}\right)}\right). \tag {9}
$$
In order to bound the right hand side, fix some  $l_{\ell_f}$ , and note that  $\mathrm{Var}(l_{\ell_f}) \leq \| l_{\ell_f}^2 \|_{L_1(\mu)} \leq \| l_{\ell_f} \|_{L_1(\mu)}$  since  $l_{\ell_f}(x) \in [0, 1]$ . Thus by Bernstein's inequality and the AM-GM inequality we get that with probability at least  $1 - \gamma$ :
$$
\begin{array}{l} \left\| l _ {\ell_ {f}} \right\| _ {L _ {1} (\mu)} - \left\| l _ {\ell_ {f}} \right\| _ {L _ {1} (\mu_ {n})} \leq \frac {\log (1 / \gamma)}{n} + \sqrt {\frac {2 \left\| l _ {\ell_ {f}} \right\| _ {L _ {1} (\mu)} \log (1 / \gamma)}{n}} \\ \leq \frac {2 0 2 \log (1 / \gamma)}{n} + \frac {1}{1 0 1} \| l _ {\ell_ {f}} \| _ {L _ {1} (\mu)} \\ \Longrightarrow \| l _ {\ell_ {f}} \| _ {L _ {1} (\mu)} - 1. 0 1 \| l _ {\ell_ {f}} \| _ {L _ {1} (\mu_ {n})} \leq \frac {2 0 5 \log (1 / \gamma)}{n}. \\ \end{array}
$$
Setting  $\gamma = \delta /\mathcal{N}_{[]}(\mathcal{F},L_1(\mu),\alpha)$  and taking a union bound over  $l_{\ell_f}$  whose number is bounded due to Eq. (8), we see that with probability  $1 - \delta$  :
$$
\max  _ {l _ {\ell_ {f}}} \left(\| l _ {\ell_ {f}} \| _ {L _ {1} (\mu)} - 1. 0 1 \| l _ {\ell_ {f}} \| _ {L _ {1} \left(\mu_ {n}\right)}\right) \leq \frac {2 0 5 \log \mathcal {N} _ {[ ]} \left(\mathcal {F} , L _ {1} (\mu) , \alpha\right) + 2 0 5 \log (1 / \delta)}{n}.
$$
Plugging this back into Eq. (9), and minimizing over  $\alpha > 0$  finishes the proof.
B.3 Proof of Theorem 4.1
Proposition B.1. Let  $f: \Omega \to [0, 1]$ . Then with probability at least  $1 - \delta / 2$  over drawing a sample it holds that
$$
\widehat {\Lambda} _ {f} ^ {\beta} \leq 4 \log^ {2} (4 n / \delta) \overline {{\Lambda}} _ {f} ^ {\beta} (\mu) + \frac {4 \log^ {2} (4 n / \delta)}{n}.
$$
Corollary B.2. If  $\mathcal{D}$  is realizable by  $\overline{\mathrm{Hol}}_L^\beta (\Omega ,\mu)$ , then for  $f^{*}:\Omega \to [0,1]$  such that  $L_{\mathcal{D}}(f^{*}) = 0$  it holds with probability at least  $1 - \delta /2:\widehat{\Lambda}_{f^*}^\beta \leq 5\log^2 (4n / \delta)L$ . Hence,  $\widehat{f} (X_i)\coloneqq f^* (X_i) = Y_i$  satisfies  $L_{S}(\widehat{f}) = 0$  and  $\widehat{\Lambda}_{\widehat{f}}^{\beta}\leq 5\log^{2}(4n / \delta)L$ .
Proof. (of Proposition B.1) Fix  $f: \Omega \to [0,1]$ . Given a sample  $(X_{i})_{i = 1}^{n}\sim \mu^{n}$  which induces an empirical measure  $\mu_{n}$ , we get
$$
\widehat {\Lambda} _ {f} ^ {\beta} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \sup  _ {z \neq X _ {i}} \frac {| f (X _ {i}) - f (z) |}{\rho (X _ {i} , z) ^ {\beta}} = \underset {X \sim \mu_ {n}} {\mathbb {E}} [ \Lambda_ {f} ^ {\beta} (X) ] \leq 2 \log (n) \mathbb {W} _ {X \sim \mu_ {n}} [ \Lambda_ {f} ^ {\beta} (X) ], \tag {10}
$$
where the last inequality follows from the reversed strong-weak mean inequality for uniform measures. We will now show that with high probability  $\mathbb{W}_{X\sim \mu_n}[\Lambda_f^\beta (X)]\lesssim \mathbb{W}_{X\sim \mu}[\Lambda_f^\beta (X)] = \widetilde{\Lambda}_f^\beta$ . To that end, we denote for any  $t > 0:M_f(t)\coloneqq \{x:\Lambda_f^\beta (x)\geq t\} \subset \Omega$ , let  $K\coloneqq \widetilde{\Lambda}_f^\beta (\mu)$ ,  $N\coloneqq [2\log (4n / \delta)\log \log (4n / \delta)]$  and note that
$$
\begin{array}{l} \mathbb {W} _ {X \sim \mu_ {n}} \left[ \Lambda_ {f} ^ {\beta} (X) \right] = \sup  _ {t > 0} t \mu_ {n} \left(M _ {f} (t)\right) \tag {11} \\ \leq \sup  _ {0 <   t \leq K} t \mu_ {n} (M _ {f} (t)) + 2 \max  _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} K \mu_ {n} (M _ {f} (2 ^ {j} K)) + \sup  _ {t \geq 2 ^ {N} K} t \mu_ {n} (M _ {f} (t)). \\ \end{array}
$$
We will bound all three summands above. We easily bound the first term by
$$
\sup  _ {0 <   t \leq K} t \mu_ {n} \left(M _ {f} (t)\right) \leq K \cdot 1 = \widetilde {\Lambda} _ {f} ^ {\beta} (\mu). \tag {12}
$$
For the second term, denote for any  $t > 0$  by  $M_f^+(t) \supset M_f(t)$  a containing set for which  $\frac{1}{n} \leq \mu(M_f^+(t)) \leq \mu(M_f(t)) + \frac{1}{n}$ . We can always assume without loss of generality that such a set exists. By the multiplicative Chernoff bound we have for any  $t, \alpha > 0$ :
$$
\operatorname * {P r} _ {S} \left[ \mu_ {n} (M _ {f} ^ {+} (t)) \geq (1 + \alpha) \mu (M _ {f} ^ {+} (t)) \right] \leq \frac {e ^ {\alpha}}{(1 + \alpha) ^ {1 + \alpha}},
$$
hence by the union bound we get with probability at least  $1 - \frac{Ne^{\alpha}}{(1 + \alpha)^{1 + \alpha}}$ :
$$
\begin{array}{l} \max  _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} K \mu_ {n} (M _ {f} (2 ^ {j} K)) \leq \max  _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} K \mu_ {n} (M _ {f} ^ {+} (2 ^ {j} K)) \\ \leq (1 + \alpha) \max _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} L \mu (M _ {f} ^ {+} (2 ^ {j} K)) \\ \leq (1 + \alpha) \max  _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} K \left(\mu \left(M _ {f} \left(2 ^ {j} K\right)\right) + \frac {1}{n}\right) \\ \leq (1 + \alpha) \widetilde {\Lambda} _ {f} ^ {\beta} (\mu) + \frac {1 + \alpha}{n}. \\ \end{array}
$$
Letting  $\alpha = \log (4n / \delta) - 1$ , by our choice of  $N = \lceil 2\log (4n / \delta)\log \log (4n / \delta)\rceil$  we get that with probability at least  $1 - \delta /4$ :
$$
2 \max  _ {j \in \{0, 1, \dots , N - 1 \}} 2 ^ {j} K \mu_ {n} \left(M _ {f} \left(2 ^ {j} K\right)\right) \leq 2 \log (4 n / \delta) \widetilde {\Lambda} _ {f} ^ {\beta} (\mu) + \frac {2 \log (4 n / \delta)}{n}. \tag {13}
$$
In order to bound the last term in Eq. (11), we observe that the empirical measure satisfies for any  $A \subset \Omega : \mu_n(A) < \frac{1}{n} \iff \mu_n(A) = 0$ , and that  $M_f(s) \subset M_f(t)$  for  $s > t$ . Furthermore, by definition of  $K = \widetilde{\Lambda}_f^\beta(\mu)$  we have  $\mu(M_f(t)) \leq \frac{K}{t}$ , hence by Markov's inequality
$$
\operatorname * {P r} _ {S} \left[ \sup _ {s \geq t} \mu_ {n} (M _ {f} (s)) \neq 0 \right] \leq \operatorname * {P r} _ {S} \left[ \mu_ {n} (M _ {f} (t)) \neq 0 \right] = \operatorname * {P r} _ {S} \left[ \mu_ {n} (M _ {f} (t)) \geq \frac {1}{n} \right] \leq \frac {n K}{t}.
$$
For  $t := 2^N K$  yields  $\operatorname{Pr}_S\left[\sup_{s \geq 2^n K} \mu_n(M_f(s)) \neq 0\right] \leq \frac{n}{2^N} \leq \frac{\delta}{4}$ . Combining this with Eq. (12), Eq. (13) and plugging back into Eq. (11), we get that with probability at least  $1 - \delta / 2$ :
$$
\mathbb {W} _ {X \sim \mu_ {n}} [ \Lambda_ {f} ^ {\beta} (X) ] \leq (1 + 2 \log (4 n / \delta)) \widetilde {\Lambda} _ {f} ^ {\beta} (\mu) + \frac {2 \log (4 n / \delta)}{n} \leq (1 + 2 \log (4 n / \delta)) \overline {{\Lambda}} _ {f} ^ {\beta} (\mu) + \frac {2 \log (4 n / \delta)}{n}.
$$
Recalling Eq. (10), we get overall that
$$
\widehat {\Lambda} _ {f} ^ {\beta} \leq 2 \log (n) \left[ (1 + 2 \log (4 n / \delta)) \overline {{\Lambda}} _ {f} ^ {\beta} (\mu) + \frac {2 \log (4 n / \delta)}{n} \right].
$$
Simplifying the expression above finishes the proof.
Proposition B.3. Under the same setting, for any  $\gamma > 0$  there exists an algorithm that given a sample  $S \sim \mathcal{D}^n$  and any function  $\widehat{f}: S \to [0,1]$ , provided that  $n \geq N$  for  $N = \widetilde{O}\left(\frac{\mathcal{N}_{\Omega}(\gamma) + \log(1 / \delta)}{\gamma}\right)$ , constructs a function  $f: \Omega \to [0,1]$  such that with probability at least  $1 - \delta / 2$ :