SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 3 (Standard single function setting.). Everything in the proof of Theorem 16 up to Equation (31) holds for a non-compositional model and where  $t = 1$  which corresponds to the standard setting. In this context, there is no need to apply the Gaussian chain rule, Theorem 8. Let  $\mathcal{Q} : \mathcal{X} \to \mathbb{R}$  be a hypothesis class and note that we have  $n$  i.i.d. samples.
If we set  $\alpha = \min \left(\sqrt{br}, 10\sqrt{3}\tilde{\mathfrak{G}}_n(\mathcal{Q})\sqrt{Hr}\right)$  then we have
$$
\tilde {\mathfrak {N}} _ {n} (\mathcal {L} _ {\ell} (\mathcal {Q}), r) \leq 4 0 \sqrt {3} \tilde {\mathfrak {G}} _ {n} (\mathcal {Q}) \sqrt {H r} \log \left(\frac {e \sqrt {b}}{\min \left(\sqrt {b} , 1 0 \sqrt {3} \tilde {\mathfrak {G}} _ {n} (\mathcal {Q}) \sqrt {H}\right)}\right),
$$
and if we set  $\alpha = \sqrt{\frac{br}{n}}$
$$
\tilde {\mathfrak {R}} _ {n} (\mathcal {L} _ {\ell} (\mathcal {Q}), r) \leq 2 0 \sqrt {3} \mathfrak {G} _ {n} (\mathcal {Q}) \sqrt {H r} \log (n) + \frac {3 \sqrt {b r}}{\sqrt {n}}.
$$
Using Lemma 5, we also get the following bounds for local Rademacher complexity (as opposed to width),
$$
\mathfrak {R} _ {n} (\mathcal {L} _ {\ell} (\mathcal {Q}), r) \leq 4 0 \sqrt {3} \tilde {\mathfrak {G}} _ {n} (\mathcal {Q}) \sqrt {H r} \log \left(\frac {e \sqrt {b}}{\min \left(\sqrt {b} , 1 0 \sqrt {3} \tilde {\mathfrak {G}} _ {n} (\mathcal {Q}) \sqrt {H}\right)}\right) + \sqrt {\frac {b r}{n}},
$$
$$
\mathfrak {R} _ {n} (\mathcal {L} _ {\ell} (\mathcal {Q}), r) \leq 2 0 \sqrt {3} \mathfrak {G} _ {n} (\mathcal {Q}) \sqrt {H r} \log (n) + \frac {5 \sqrt {b r}}{\sqrt {n}}.
$$
D.5 Theorems for transition from empirical to population
Lemma 19. Let  $\mathcal{F}$  be a class of functions that map  $\mathcal{X}$  into  $[0,b]$  with  $b > 0$ . Fix  $\alpha > 0$ . For every  $\delta > 0$  and  $r$  that satisfy
$$
r \geq 4 (\alpha + 1) \mathbb {E} \Re_ {\mathbf {Z}} \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P \boldsymbol {f} \leq r \right\} + \frac {b \delta}{n t} \left(\frac {8}{3} + \frac {2}{\alpha}\right)
$$
we have with probability at least  $1 - e^{-\delta}$  that
$$
\left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P \boldsymbol {f} \leq r \right\} \subseteq \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid \hat {P} ^ {n} \boldsymbol {f} \leq 2 r \right\}.
$$
The proof is similar to the proof of Corollary 2.2 within Bartlett et al. [2005].
Proof of Lemma 19. For  $\pmb{f} \in \{\pmb{f} \in \mathcal{F}^{\otimes t} \mid P\pmb{f} \leq r\}$  we have  $\mathrm{Var}\big[\pmb{f}\big] \leq P\pmb{f}^2 \leq bP\pmb{f} \leq br$ . By Theorem 10, with probability  $1 - e^{-\delta}$ , every  $\pmb{f} \in \{\pmb{f} \in \mathcal{F}^{\otimes t} \mid P\pmb{f} \leq r\}$  satisfies, with  $\alpha = \sqrt{2}$ ,
$$
\begin{array}{l} \hat {P} ^ {n} \boldsymbol {f} \leq P \boldsymbol {f} + 2 (1 + \alpha) \mathbb {E} \Re_ {\mathbf {Z}} \mathcal {F} _ {r} ^ {\otimes t} + \sqrt {\frac {2 b r \delta}{n t}} + b \left(\frac {1}{3} + \frac {1}{\alpha}\right) \frac {\delta}{n t} \\ \leq r + 2 (1 + \alpha) \mathbb {E} \Re_ {\mathbf {Z}} \mathcal {F} _ {r} ^ {\otimes t} + \sqrt {\frac {2 b r \delta}{n t}} + b \left(\frac {1}{3} + \frac {1}{\alpha}\right) \frac {\delta}{n t} \\ \leq r + 2 (1 + \alpha) \mathbb {E} \Re_ {\mathbf {Z}} \mathcal {F} _ {r} ^ {\otimes t} + \frac {b \delta}{n t} + \frac {r}{2} + b \left(\frac {1}{3} + \frac {1}{\alpha}\right) \frac {\delta}{n t} \\ \leq 2 r. \\ \end{array}
$$
D.6 Smooth Learning Bounds
Lemma 20. Under the setting of Theorem 16 for any  $\delta > 0$ , the fixed point  $r^*$  is bounded by,
$$
c _ {1} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H b \log (n t) ^ {4} + \frac {2 3 3 2 8 0 0 D ^ {2} H b \log (n t) ^ {2}}{n t} + \frac {1 1 2 b ^ {2} \delta}{3 n t} + \frac {1 9 4 4 b ^ {2}}{n t}
$$
where  $c_{1} < 1.5 \times 10^{8}$  and  $G(\mathcal{F}^{\otimes t}(\mathcal{H})) = L\tilde{\mathfrak{G}}_{nt}(\mathcal{H}) + \tilde{\mathfrak{G}}_n(\mathcal{F})$
Further, if
$$
1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \leq D \quad a n d \quad \sqrt {b} \geq 6 4 0 \sqrt {3} \Pi (\mathcal {F} ^ {\otimes t} (H)),
$$
where  $\Pi (\mathcal{F}^{\otimes t}(H)) = \sqrt{H} G\left(\mathcal{F}^{\otimes t}(\mathcal{H})\right)\log \left(\frac{eD}{16G(\mathcal{F}^{\otimes t}(\mathcal{H}))}\right)$ , then for any  $\delta >0$ , the fixed the fixed point  $r^*$  is bounded by,
$$
\begin{array}{l} c _ {2} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H b \log \left(\frac {D e}{1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right) ^ {2} \log \left(\frac {2 \sqrt {3} \sqrt {b}}{1 9 2 0 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log \left(\frac {e D}{1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right)}\right) ^ {2} \\ + \frac {1 1 2 b ^ {2} \delta}{3 n t} \\ \end{array}
$$
with  $c_{2} < 8 \times 10^{8}$ .
Proof of Lemma 20. Let
$$
\gamma (r) = 4 (\alpha + 1) \mathbb {E} \Re_ {\mathbf {Z}} (b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b}) + \frac {b ^ {2} \delta}{n t} \left(\frac {8}{3} + \frac {2}{\alpha}\right).
$$
$$
\begin{array}{l} b \mathbb {E} \Re_ {\mathbf {Z}} \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \leq 4 (1 + \alpha) \mathbb {E} \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} + \frac {b ^ {2} \delta}{n t} \left(\frac {8}{3} + \frac {2}{\alpha}\right) \\ = \gamma (r). \tag {32} \\ \end{array}
$$
Note that from Lemma 19, we have that, for any  $\delta >0$ , with probability at least  $1 - e^{-\delta}$
$$
\left. \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P \boldsymbol {f} \leq \frac {r}{b} \right\} \leq \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid \hat {P} ^ {n} \boldsymbol {f} \leq \frac {2 r}{b} \right\}. \right.
$$
Further, we can apply the concentration of Rademacher complexity, Lemma 11, to bound the expected value of the left-hand side as follows. With probability at least  $1 - e^{-\delta}$ ,
$$
\mathbb {E} \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \leq 2 \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} + \frac {b ^ {2} \delta}{n t}.
$$
Hence, with probability  $1 - 2e^{-\delta}$ , we get,
$$
\mathbb {E} \left[ \Re_ {\mathbf {Z}} \left\{b f \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \right] \leq 2 \Re_ {\mathbf {Z}} \left\{b f \in \mathcal {F} ^ {\otimes t} \mid \hat {P} ^ {n} f \leq \frac {2 r}{b} \right\} + \frac {b ^ {2} \delta}{n t}.
$$
We now apply Lemma 16 and upper bound the right-hand side by taking supremum over  $\mathbf{Z}$ . This gives us,
$$
\mathbb {E} \left[ \Re_ {\mathbf {Z}} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \right] \leq 2 \Re_ {n t} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid \hat {P} ^ {n} f \leq \frac {2 r}{b} \right\} + \frac {b ^ {2} \delta}{n t}.
$$
Plugging the above in Eqn. (32), we get,
$$
\begin{array}{l} b \mathbb {E} \Re_ {\mathbf {Z}} \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \leq 8 (1 + \alpha) \mathbb {E} \Re_ {n t} \left\{b \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid \hat {P} ^ {n} f \leq \frac {2 r}{b} \right\} \\ + \frac {b ^ {2} \delta}{n t} \left(\frac {8}{3} + 4 (1 + \alpha) + \frac {2}{\alpha}\right). \\ \end{array}
$$
Suppose that the local Rademacher complexity is bounded by a multiple of  $\sqrt{r}$ , say  $\sqrt{r} E$ . Now bounding the local Rademacher complexity with Theorem 16, we have that,
$$
\begin{array}{l} b \mathbb {E} \Re_ {\mathbf {Z}} \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\} \leq 8 (1 + \alpha) \sqrt {b} \sqrt {2 r} E + \frac {b ^ {2} \delta}{n t} \left(\frac {8}{3} + 4 (1 + \alpha) + \frac {2}{\alpha}\right) \\ = A \sqrt {r} + D \\ \end{array}
$$
$$
A = 8 (1 + \alpha) \sqrt {b} \sqrt {2} E D = \frac {b ^ {2} \delta}{n t} \left(\frac {8}{3} + 4 (1 + \alpha) + \frac {2}{\alpha}\right).
$$
Note that  $A\sqrt{r} + D$  is sub-root and larger than  $\gamma(r)$ . Thus if we solve for a fixed point  $r^*$  of this expression we have  $A\sqrt{r^*} + D = r^*$ . By Lemma 12 we have  $r^* \leq A^2 + 2D$ .
Therefore, with  $\alpha = 1 / 8$
$$
b \Re_ {n} \left\{\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \mid P f \leq \frac {r}{b} \right\}
$$
is bounded by a sub-root function of  $r$  with the following fixed point:
$$
b 7 2 (E) ^ {2} b + \frac {1 3 9}{3} \frac {b ^ {2} \delta}{n t}. \tag {33}
$$
We have seen from Theorem 16 that if  $16G(\mathcal{F}^{\otimes t}(\mathcal{H}))\leq D$  and  $\sqrt{b}\geq 640\sqrt{3}\Pi (\mathcal{F}^{\otimes t}(H))$  then we can set
$$
E = 2 5 6 0 \sqrt {3} \Pi (\mathcal {F} ^ {\otimes t} (H)) \log \left(\frac {e \sqrt {3} \sqrt {b}}{1 9 2 0 \Pi (\mathcal {F} ^ {\otimes t} (H))}\right).
$$
In this setting, using Equation (33), the fixed point is bounded by:
$$
7 9 6 2 6 2 4 0 0 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H b \log \left(\frac {D e}{1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right) ^ {2} \log \left(\frac {2 \sqrt {3} \sqrt {b}}{1 9 2 0 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log \left(\frac {e D}{1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right)}\right) ^ {2} + \frac {1 1 2 b ^ {2} \delta}{3 n t}
$$
Alternatively, we can always set  $E$  as follows:
$$
E = 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log (n t) ^ {2} + \frac {8 0 \sqrt {3} D \sqrt {H} \log (n t) + 5 \sqrt {b}}{\sqrt {n t}}.
$$
therefore we also have that the fixed point is bounded as:
$$
1 4 9 2 9 9 2 0 0 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H b \log (n t) ^ {4} + \frac {2 3 3 2 8 0 0 D ^ {2} H b \log (n t) ^ {2}}{n t} + \frac {1 1 2 b ^ {2} \delta}{3 n t} + \frac {1 9 4 4 b ^ {2}}{n t}.
$$
D.7 Smooth MTL and MTL via MTRL
When we add the additional assumption that the loss function is smooth we have the following extensions to Theorems 12 and 15
Theorem 17. Let  $(\hat{f},\hat{h})$  be an empirical risk minimizer as given in Eqn. (2). Let  $\psi (r)\geq b\Re_{n}(\mathcal{F}^{\otimes t}(\mathcal{H}),r)$  with  $r^*$  the fixed point of  $\psi (r)$ . Then, under Assumption 1 along with  $\ell$  being  $H$ -smooth, with probability  $1 - 2e^{-\delta}$ ,
$$
R _ {\text {s o u r c e}} (\hat {\boldsymbol {f}}, \hat {h}) \leq R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) + \sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*} , h ^ {*}\right)} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r ^ {*}}{b}. \tag {34}
$$
$$
\frac {r ^ {*}}{b} \leq c \left(G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H \log (n t) ^ {4} + \frac {D ^ {2} H b \log (n t) ^ {2}}{n t} + \frac {b \delta}{n t} + \frac {b ^ {2}}{n t}\right)
$$
with  $c < 1.5 \times 10^{8}$  and  $G(\mathcal{F}^{\otimes t}(\mathcal{H})) = L\tilde{\mathfrak{G}}_{nt}(\mathcal{H}) + \tilde{\mathfrak{G}}_n(\mathcal{F})$ .
Proof of Theorem 17. Use the fixed point bound from Lemma 20 that always holds and substitute it into Theorem 12.  $\square$
Theorem 18. Let  $\hat{h}$  and  $\hat{f}_0$  be the learned representation and target predictor, as described in Eqns. (2) and (3). Let  $\psi_1(r) \geq b\Re_n(\ell \circ \mathcal{F}^{\otimes t} \circ \mathcal{H}, r)$  and  $\psi_2(r) \geq b\Re_n(\ell \circ \mathcal{F}_0, r)$  with  $r_1^*$  and  $r_2^*$  the fixed points of  $\psi_1(r)$  and  $\psi_2(r)$ , respectively. Then, under Assumption 1 along with  $\ell$  being  $H$ -smooth and that  $\pmb{f}^{*}$  is  $(\nu, \epsilon)$ -diverse over  $\mathcal{F}_0$  w.r.t.  $h^*$ , with probability at least  $1 - 2e^{-\delta}$ , the transfer learning risk is upper-bounded by,
$$
\begin{array}{l} R _ {\text {t a r g e t}} \left(\hat {f} _ {0}, \hat {h}\right) \leq R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r _ {1} ^ {*}}{b}}\right) + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r _ {1} ^ {*}}{2 b} (35) \\ + \frac {1}{\nu} \left(\sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {* , h ^ {*}}\right)} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r _ {2} ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r _ {2} ^ {*}}{b}\right) + \varepsilon , (36) \\ \end{array}
$$
$$
\frac {r _ {2} ^ {*}}{b} \leq c \left(G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) ^ {2} H \log (n t) ^ {4} + \frac {D ^ {2} H b \log (n t) ^ {2}}{n t} + \frac {b \delta}{n t} + \frac {b ^ {2}}{n t}\right)
$$
where  $c < 1.5 \times 10^{8}$  and
$$
\frac {r _ {1} ^ {*}}{b} \leq 9 7 2 0 0 \tilde {\mathfrak {G}} _ {m} ^ {2} (\mathcal {F} _ {0} \circ \hat {h}) H \log (m) ^ {2} + \frac {1 1 2 b \delta}{3 m} + \frac {1 2 9 6 b}{m}.
$$
Proof of Theorem 18. Use the fixed point bound from Lemma 20 that always holds and substitute it into Theorem 15.  $\square$
D.8 Local Rademacher complexity chain rule
Theorem 19. Suppose the loss function  $\ell$  is  $L_{\ell}$ -Lipschitz. Define the restricted representation and predictor classes as follows,
$$
\tilde {\mathfrak {G}} _ {n} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H}), r\right)\right) \leq c \Big (L L _ {\ell} \psi_ {\mathcal {F}} (r) + \psi_ {\mathcal {H}} (r) \big) \log (n t) + \frac {D}{(n t) ^ {2}} \Big).
$$
Proof of Theorem 19. Define
$$
\ell \circ \mathcal {F} _ {\mathbf {X}} ^ {\otimes t} (\mathcal {H}) (r) = \left\{\left(\ell \circ \boldsymbol {f}, h\right): V \left(\ell \circ \boldsymbol {f} \circ h\right) \leq r \right\}.
$$
Observe that for any  $(\ell \circ \pmb {f},h)\in \mathcal{F}_{\mathbf{X}}^{\otimes t}(\mathcal{H})$  , we have  $\pmb {f}\in \mathcal{F}_{\mathbf{X}}(r)$  and  $h\in \mathcal{H}_{\mathbf{X}}(r)$  . Hence,
$$
\ell \circ \mathcal {F} _ {\mathbf {X}} ^ {\otimes t} (\mathcal {H}) (r) \subseteq \mathcal {F} _ {\mathbf {X}} (r) \circ \mathcal {H} _ {\mathbf {X}} (r)
$$
Further, by assumption, the composed function  $\ell \circ f$  is  $L_{\ell}L$ -Lipschitz. We can now apply the (standard) chain rule of Tripuraneni et al. [2021] which gives us,
$$
\begin{array}{l} \mathfrak {S} _ {n} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} _ {\mathbf {X}} ^ {\otimes t} (\mathcal {H}) (r)\right)\right) \leq \mathfrak {S} _ {n t} \left(\ell \circ \mathcal {F} _ {\mathbf {X}} (r) \circ \mathcal {H} _ {\mathbf {X}} (r)\right) \\ \leq c \left(\left(L L _ {\ell} \psi_ {\mathcal {F}} (r) + \psi_ {\mathcal {H}} (r)\right) \log (n t) + \frac {D}{(n t) ^ {2}}\right) \\ \end{array}
$$
E Detailed comparison with prior works
In this section, we provide a more detailed comparison with prior works.
E.1 Comparison with Srebro et al. [2010]
Firstly, we identify some erroneous or missing though fixable steps in the proof of Srebro et al. [2010].
1. Dudley's integral formula. The work of Srebro et al. [2010] seems to interchange the concepts of Rademacher width and complexity with the same notation  $\Re_n(\mathcal{F})$ . The original Dudley's formula and its truncated version proved by Srebro et al. [2010] is for Rademacher (or Gaussian) "width", yet in their work, it is used to bound Rademacher complexity. It is possible to translate from width to complexity, but this will lead to an additional term of  $\sqrt{\frac{br}{n}}$  as detailed below. This additional term only changes their result by constant factors.
Lemma 21. Consider a class of functions  $\mathcal{F}$  with range in  $[0, b]$ . Then given input  $\mathbf{Z}$  of  $n$  points,  $\Re_{\mathbf{Z}}\left\{f \in \mathcal{F} : \hat{P}^n f \leq r\right\} \leq 2\tilde{\Re}_{\mathbf{Z}}\left\{f \in \mathcal{F} : \hat{P}^n f \leq r\right\} + \sqrt{\frac{br}{n}}$ .
Proof. The proof follows by an application of Lemma 5. In particular, given input  $\mathbf{Z}$  of  $n$  points, for any  $f\in \left\{f\in \mathcal{F}:\hat{P}^n f\leq r\right\}$ , we have that
$$
\frac {1}{\sqrt {n}} \| f \| _ {L ^ {2} (\mathbf {Z})} = \sqrt {\frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} f (z _ {i}) ^ {2}} \leq \sqrt {\frac {b}{n} \frac {1}{n} \sum_ {i = 1} ^ {n} f (z _ {i})} \leq \sqrt {\frac {b r}{n}}
$$
Plugging this in Lemma 5 gives the claimed bound.
2. Centering. Within Srebro et al. [2010] they consider loss with the "bounded difference" property:  $\forall \hat{y}, \hat{y}, y'$  we have that  $|\ell(\hat{y}, y) - \ell(\hat{y}', y)| \leq b$ . To bound the local Rademacher complexity of their loss class which is empirically constrained with  $\hat{P}^n(\ell \circ f) \leq r$ , they use Dudley's integral. Note the upper limit of integration in Dudley's integral is  $\sqrt{\hat{P}^n(\ell \circ f)^2}$  and it is claimed that  $\sqrt{\hat{P}^n(\ell \circ f)^2} \leq \sqrt{br}$  because  $\hat{P}^n(\ell \circ f)^2 \leq bP(\ell \circ f) \leq br$ . Yet, this reasoning only follows for  $b$ -bounded losses not under this weaker condition of bounded difference. Yet, it is possible to center the process and perform a comparable analysis. To resolve this issue let  $\tilde{f} = \arg \inf_{f \in \mathcal{F}} \hat{P}^n(\ell \circ f)$  and consider  $(\ell \circ f) - (\ell \circ \tilde{f})$ . This process is now  $b$ -bounded. Centering the process in this way does not affect the downstream results because Gaussian/Rademacher width is shift agnostic due to symmetry.
3. Missing condition on  $n$ . This omission is related to the subsequent discussion about comparison with Srebro et al. [2010]. In the proof of Lemma 2.2, after Eqn. 23, the limits of integration are required to satisfy  $\sqrt{12Hr}\Re_n(\mathcal{F})\leq \sqrt{br}$ . Using the fact that in the setup, in general,  $\Re_n(\mathcal{F}) = \Theta \left(\frac{B}{\sqrt{n}}\right)$ , the above is equivalent to  $n = \Omega \left(\frac{HB^2}{b}\right)$ , This condition on  $n$  is missing from their main statement. Further, as we detail in Appendix E.1.1, this term, in general, can be unbounded.
4. Fat-Shattering inequality. Using the notation from Srebro et al. [2010]. Note that  $\mathrm{fat}_x$  is a decreasing function in  $x$ . Also  $x\log \left(\frac{n}{x}\right)$  is a decreasing function in  $x$  for  $x\geq 1$ . Therefore  $\mathrm{fat}_x\log \left(\frac{n}{\mathrm{fat}_x}\right)$  is an increasing function in  $x$ . So therefore, for  $\varepsilon \in [\gamma ,\theta ]$  we have  $\mathrm{fat}_{\gamma}\log \left(\frac{n}{\mathrm{fat}_{\gamma}}\right)\leq \mathrm{fat}_{\varepsilon}\log \left(\frac{n}{\mathrm{fat}_{\varepsilon}}\right)\leq \mathrm{fat}_{\theta}\log \left(\frac{n}{\mathrm{fat}_{\theta}}\right)$ . This in inequality is in the wrong direction on page 27 of Srebro et al. [2010].
Disregarding the above issues, the bound obtained on Srebro et al. [2010] on the local Rademacher complexity is,
$$
\tilde {\Re} _ {n} \left(\mathcal {L} _ {\ell} (\mathcal {F}, r)\right) = \mathcal {O} \left(\tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H r} (\log (n)) ^ {3 / 2}\right). \tag {37}
$$
In contrast ours is,
$$
\tilde {\mathfrak {R}} _ {n} \left(\mathcal {L} _ {\ell} (\mathcal {F}, r)\right) \leq c \tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H r} \log \left(\frac {e \sqrt {b}}{\tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H}}\right). \tag {38}
$$
Worst-case improvement for large enough samples. We first perform a general comparison. Accordingly, we bound the above as,
$$
\begin{array}{l} \tilde {\mathfrak {R}} _ {n} (\mathcal {L} _ {\ell} (\mathcal {F}, r)) \leq c \tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H r} \log \left(\frac {e \sqrt {b}}{\tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H}}\right) \\ \leq c \tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H r} \sqrt {\log (n)} \log \left(\frac {e \sqrt {b}}{\tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H} \sqrt {\log (n)}}\right) \\ \leq c ^ {\prime} \tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H r} \sqrt {\log (n)} \log \left(\frac {e b}{H \log (n)} \frac {n}{B ^ {2}}\right). \\ \end{array}
$$
The above is of the same form as the bound of Srebro et al. [2010] in Eqn. (37), and it is easy to see that ours is better whenever  $n = e^{\Omega \left(\frac{b}{HB^2}\right)}$ .
Worst-case improvement for constant samples. Unfortunately, the term  $\frac{b}{HB^2}$  can be unbounded, from above, in general, as detailed in Appendix E.1.1. However, a reasonable and standard assumption removes this issue. If we assume that loss at zero is bounded as follows,  $\ell(0) \leq HB^2$ , then we show in Lemma 22, that  $\frac{b}{HB^2} = \Omega(1)$ , thereby, making the requirement to be a constant number of samples. A uniform bound on  $\ell(0)$  features in the characterization of sample complexity of learning with non-negative smooth losses as well as the regime described by the above bound on  $\ell(0)$  has been considered in prior works [Arora et al., 2022, Shamir, 2015].
Improvement for certain hypothesis class. In the above chain of inequalities, we use the bound that  $\tilde{\mathfrak{G}}_n(\mathcal{F})\leq \Re_n(\mathcal{F})\sqrt{\log n}$  which is tight in the worst case. However, there are natural situations where the two are of the same order. A prominent example is linear predictors with data bounded in (any) norm  $\| \cdot \|$  and hypothesis class bounded in the via a (regularization) function  $R$  which is strongly convex with respect to the dual norm  $\| \cdot \|_{*}$  [Kakade et al., 2008].
E.1.1 Understanding the  $\frac{HB^2}{b}$  term
No non-trivial lower bound. Firstly, we see that the term is not non-trivially lower bounded, in general. Consider the function  $\ell(f(x)) = \frac{1}{2} (q - f(x))^2$ . This function is 1-smooth and considers  $B = 1$  (which bounds the size of  $f(x)$ ), however,  $b$  here is controlled by the size of  $q$  and thus could be unbounded. This means that the is a non-trivial lower bound on  $\frac{HB^2}{b}$  in general. The above example also works if relax the definition of  $b$  to be "bounded difference",  $\sup_{f(x), f'(x') \in \mathcal{F}} (\ell(f(x)) - \ell(f'(x')))$  as opposed to a uniform absolute bound.
A lower bound under  $\ell(0)$  bound. We give a lower bound under the assumption of bound on  $\ell(0)$ .
Lemma 22. Let  $\mathcal{F}$  be a class of real-valued functions and let the loss function  $\ell : \mathbb{R} \to \mathbb{R}$  be a non-negative  $H$ -smooth function with  $\ell(0) \leq L_0$ , and  $\sup_{z \in \text{Range}(\mathcal{F})} \ell(z) \leq b$ . Then,
$$
\frac {H B ^ {2}}{b} \geq \frac {1}{1 + \frac {2 L _ {0}}{H B ^ {2}}}
$$
$$
\begin{array}{l} b = \sup  _ {z} \ell (z) = \ell (z) - \ell (0) + \ell (0) \\ \leq \langle \ell^ {\prime} (0), z - z ^ {\prime} \rangle + \frac {H}{2} | z | ^ {2} + L _ {0} \\ \leq \frac {\left| \ell^ {\prime} (0) \right| ^ {2}}{2 H} + \frac {H \left| z \right| ^ {2}}{2} + \frac {H B ^ {2}}{2} + L _ {0} \\ \leq \frac {\ell (0)}{2} + H B ^ {2} + L _ {0} \\ = 2 L _ {0} + H B ^ {2} \\ \end{array}
$$
where the first inequality uses smoothness, the second AM-GM inequality, and the third uses the self-bounding property of non-negative smooth losses (Lemma 2.1 in Srebro et al. [2010]).
No non-trivial upper bound. Consider the function  $\ell(z) = H(1 + \sin(z))$ ; this is non-negative,  $H$ -smooth, and  $b = 2H$ . However, we can define the size of the domain  $B$  as arbitrary, without affecting the smoothness and range boundedness. Thus,  $\frac{HB^2}{b}$  is unbounded from above, in general. Note that even assuming an upper bound on  $\ell(0)$  doesn't help here.
E.2 Comparison with Denevi et al. [2019], Khodak et al. [2019]