SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L ^ {*} + \frac {1 7 8 9 K b \delta}{8 8 2 n t} + \frac {1 4 4 K r ^ {*}}{b} + \sqrt {2} \sqrt {\frac {L ^ {*} b \delta}{n t}} + \frac {2 2 3 6 9 b \delta}{8 8 2 n t} + \frac {1 4 4 r ^ {*}}{b} + \frac {2 L ^ {*}}{K} + \frac {5 b \delta}{3 K n t}.
$$
These are the terms without  $K$ :
$$
L ^ {*} + \sqrt {2} \sqrt {\frac {L ^ {*} b \delta}{n t}} + \frac {2 2 3 6 9 b \delta}{8 8 2 n t} + \frac {1 4 4 r ^ {*}}{b}. \tag {28}
$$
Considering only those terms a function of  $K$ :
$$
\frac {1 7 8 9 K b \delta}{8 8 2 n t} + \frac {1 4 4 K r ^ {*}}{b} + \frac {2 L ^ {*}}{K} + \frac {5 b \delta}{3 K n t}.
$$
We set  $K = \frac{\max\left(\sqrt{L^{*}},\sqrt{\frac{b\delta}{nt}}\right)}{\max\left(\sqrt{\frac{r^{*}}{b}},\sqrt{\frac{b\delta}{nt}}\right)}$ . Terms with  $K$  in the numerator are upper-bounded by
$$
\frac {1 7 8 9 b \delta \left(\sqrt {\boldsymbol {L} ^ {*}} + \sqrt {\frac {b \delta}{n t}}\right)}{8 8 2 n t \max \left(\sqrt {\frac {r ^ {*}}{b}} , \sqrt {\frac {b \delta}{n t}}\right)} + \frac {1 4 4 r ^ {*} \left(\sqrt {\boldsymbol {L} ^ {*}} + \sqrt {\frac {b \delta}{n t}}\right)}{b \max \left(\sqrt {\frac {r ^ {*}}{b}} , \sqrt {\frac {b \delta}{n t}}\right)}.
$$
Terms with  $K$  in the denominator are upper-bounded by
$$
\frac {2 \pmb {L} ^ {*} \left(\sqrt {\frac {b \delta}{n t}} + \sqrt {\frac {r ^ {*}}{b}}\right)}{\max \left(\sqrt {\pmb {L} ^ {*}} , \sqrt {\frac {b \delta}{n t}}\right)} + \frac {5 b \delta \left(\sqrt {\frac {b \delta}{n t}} + \sqrt {\frac {r ^ {*}}{b}}\right)}{3 n t \max \left(\sqrt {\pmb {L} ^ {*}} , \sqrt {\frac {b \delta}{n t}}\right)}.
$$
Expanding both expressions gives eight terms where we will choose an appropriate value to maximize the quantity:
$$
\frac {2 \boldsymbol {L} ^ {*} \sqrt {r ^ {*}}}{\sqrt {b} \max  \left(\sqrt {\boldsymbol {L} ^ {*}}, \sqrt {\frac {b \delta}{n t}}\right)} \leq 2 \sqrt {\frac {\boldsymbol {L} ^ {*} r ^ {*}}{b}}
$$
$$
\frac {1 4 4 \sqrt {\boldsymbol {L} ^ {*} r ^ {*}}}{b \max  \left(\sqrt {\frac {r ^ {*}}{b}} , \sqrt {\frac {b \delta}{n t}}\right)} \leq 1 4 4 \sqrt {\frac {\boldsymbol {L} ^ {*} r ^ {*}}{b}}
$$
$$
\frac {5 b ^ {\frac {3}{2}} \delta^ {\frac {3}{2}}}{3 n ^ {\frac {3}{2}} t ^ {\frac {3}{2}} \max  \left(\sqrt {\boldsymbol {L} ^ {*}}, \sqrt {\frac {b \delta}{n t}}\right)} \leq \frac {5 b \delta}{3 n t}
$$
$$
\frac {1 7 8 9 b ^ {\frac {3}{2}} \delta^ {\frac {3}{2}}}{8 8 2 n ^ {\frac {3}{2}} t ^ {\frac {3}{2}} \max  \left(\sqrt {\frac {r ^ {*}}{b}}, \sqrt {\frac {b \delta}{n t}}\right)} \leq \frac {1 7 8 9 b \delta}{8 8 2 n t}
$$
$$
\frac {2 \boldsymbol {L} ^ {*} \sqrt {b} \sqrt {\delta}}{\sqrt {n} \sqrt {t} \max  \left(\sqrt {\boldsymbol {L} ^ {*}}, \sqrt {\frac {b \delta}{n t}}\right)} \leq 2 \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}}
$$
$$
\frac {1 4 4 r ^ {*} \sqrt {\delta}}{\sqrt {b} \sqrt {n} \sqrt {t} \max \left(\sqrt {\frac {r ^ {*}}{b}}, \sqrt {\frac {b \delta}{n t}}\right)} \leq 1 4 4 \sqrt {\frac {r ^ {*} \delta}{n t}}
$$
$$
\frac {5 \sqrt {b} \sqrt {r ^ {*}} \delta}{3 n t \max  \left(\sqrt {\boldsymbol {L} ^ {*}} , \sqrt {\frac {b \delta}{n t}}\right)} \leq \frac {5}{3} \sqrt {\frac {r ^ {*} \delta}{n t}}
$$
$$
\frac {1 7 8 9 \sqrt {\boldsymbol {L} ^ {*} b \delta}}{8 8 2 n t \operatorname* {m a x} \left(\sqrt {\frac {r ^ {*}}{b}} , \sqrt {\frac {b \delta}{n t}}\right)} \leq \frac {1 7 8 9}{8 8 2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}}.
$$
Adding all of the right-hand side expressions together gives:
$$
\frac {3 5 5 3}{8 8 2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + 1 4 6 \sqrt {\frac {\boldsymbol {L} ^ {*} r ^ {*}}{b}} + \frac {3 2 5 9 b \delta}{8 8 2 n t} + \frac {4 3 7}{3} \sqrt {\frac {r ^ {*} \delta}{n t}}.
$$
Adding the terms without  $K$ , Equation (28) gives:
$$
\sqrt {2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + \frac {3 5 5 3}{8 8 2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + 1 4 6 \sqrt {\frac {\boldsymbol {L} ^ {*} r ^ {*}}{b}} + \boldsymbol {L} ^ {*} + \frac {1 2 8 1 4 b \delta}{4 4 1 n t} + \frac {4 3 7}{3} \sqrt {\frac {r ^ {*} \delta}{n t}} + \frac {1 4 4 r ^ {*}}{b}.
$$
Applying AM-GM again
$$
\sqrt {\boldsymbol {L} ^ {*}} \left(\sqrt {2} \sqrt {\frac {b \delta}{n t}} + \frac {3 5 5 3}{8 8 2} \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \boldsymbol {L} ^ {*} + \frac {8 9 8 6 7 b \delta}{8 8 2 n t} + \frac {1 3 0 1 r ^ {*}}{6 b}.
$$
Rounding up to the nearest integers:
$$
\sqrt {\boldsymbol {L} ^ {*}} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \boldsymbol {L} ^ {*} + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r ^ {*}}{b}.
$$
D.3 Multi-Task Learning via Representation Learning
Theorem 13. Let  $\hat{h}$  be an empirical risk minimizer as in (2). Let  $\psi(r) \geq b\mathbb{E}\tilde{\mathfrak{R}}_{\mathbf{Z}}(\ell \circ \mathcal{F}^{\otimes t} \circ \mathcal{H}, r)$  with  $r^*$  the fixed point of  $\psi(r)$ . Then, if Assumption 1.A holds. Then, with probability at least  $1 - e^{-\delta}$ ,
$$
\bar {d} _ {\mathcal {F}, f ^ {*}} (\hat {h}, h ^ {*}) \leq \sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {* , h ^ {*}}\right)} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r ^ {*}}{b},
$$
where recall that  $\bar{d}_{\mathcal{F},f^*}$  is defined in Equation (12).
Proof of Theorem 13.
$$
\begin{array}{l} \bar {d} _ {\mathcal {F}, f ^ {*}} (\hat {h}, h ^ {*}) = \inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} \otimes t} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime}, \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\} \\ \leq R _ {\text {s o u r c e}} (\hat {\boldsymbol {f}}, \hat {h}) - R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {*}, h ^ {*}) \\ \leq \sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {* , h ^ {*}}\right)} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r ^ {*}}{b}. \\ \end{array}
$$
where the last inequality is due to Theorem 12.
By using our hypothesis which rates the task-averaged representation distance to the worst case representation difference we can bound the excess transfer risk.
Theorem 14. Let  $\hat{f}_0$  be an empirical risk minimizer of  $\hat{R}_{\text {target }}(\cdot, \hat{h})$  in Eqn. (3) for any representation  $\hat{h}$ . Let  $\psi(r) \geq b \mathbb{E} \tilde{\mathfrak{A}}_{\mathbf{Z}}(\ell \circ \mathcal{F}_0, r)$  with  $r^*$  the fixed point of  $\psi(r)$ . Then if Assumption 1.A holds, with probability at least  $1 - \delta$ :
$$
\begin{array}{l} R _ {\text {t a r g e t}} \left(\hat {f} _ {0}, \hat {h}\right) \leq R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r}{b}}\right) \\ + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r}{2 b} + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}). \\ \end{array}
$$
Proof of Theorem 14. Let  $\tilde{f}_0 = \arg \min_{f\in \mathcal{F}}R_{\mathrm{target}}(f,\hat{h})$ . By adding and subtracting  $d_{\mathcal{F},\mathcal{F}_0}(\hat{h};h^*)$  then applying Theorem 12 to  $R_{\mathrm{target}}(\hat{f}_0,\hat{h})$ , we have convenient cancellation of  $R_{\mathrm{target}}(\tilde{f}_0,\hat{h})$ . For notational succinctness let
$$
A _ {t} = 6 \sqrt {\frac {b \delta}{m}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}} \quad \text {a n d} \quad B _ {t} = \frac {1 0 2 b \delta}{m} + \frac {2 1 7 r ^ {*}}{b}.
$$
$$
\begin{array}{l} R _ {\text {t a r g e t}} (\hat {f} _ {0}, \hat {h}) = R _ {\text {t a r g e t}} (\hat {f} _ {0}, \hat {h}) - d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ = R _ {\text {t a r g e t}} \left(\hat {f} _ {0}, \hat {h}\right) - \left(R _ {\text {t a r g e t}} \left(\tilde {f} _ {0}, \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right)\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq R _ {\text {t a r g e t}} \left(\tilde {f} _ {0}, \hat {h}\right) + \sqrt {R _ {\text {t a r g e t}} \left(\tilde {f} _ {0} , \hat {h}\right)} A _ {t} + B _ {t} \\ - \left(R _ {\text {t a r g e t}} \left(\tilde {f} _ {0}, \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right)\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ = \sqrt {R _ {\text {t a r g e t}} (\tilde {f} _ {0} , \hat {h})} A _ {t} + B _ {t} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}). \\ \end{array}
$$
Now note that  $R_{\mathrm{target}}(\tilde{f}_0,\hat{h}) = R_{\mathrm{target}}(f_0^*,h^*) + \bar{d}_{\mathcal{F}_0,f_0^*}(\hat{h},h^*)$ . After applying subadditivity of the square root  $\sqrt{\cdot}$  and AM-GM we can bound  $\bar{d}_{\mathcal{F}_0,f_0^*}(\hat{h},h^*)$  with Theorem 13. This is concretely
demonstrated with the following chain of inequalities:
$$
\begin{array}{l} R _ {\mathrm {t a r g e t}} (\hat {f} _ {0}, \hat {h}) \leq \sqrt {R _ {\mathrm {t a r g e t}} (f _ {0} ^ {*} , h ^ {*}) + \bar {d} _ {\mathcal {F} _ {0} , f _ {0} ^ {*}} (\hat {h} , h ^ {*})} A _ {t} + B _ {t} + R _ {\mathrm {t a r g e t}} (f _ {0} ^ {*}, h ^ {*}) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} A _ {t} + \sqrt {\bar {d} _ {\mathcal {F} _ {0} , f _ {0} ^ {*}} (\hat {h} , h ^ {*})} A _ {t} + B _ {t} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} A _ {t} + \frac {\bar {d} _ {\mathcal {F} _ {0} , f _ {0} ^ {*}} (\hat {h} , h ^ {*})}{2} + \frac {A _ {t} ^ {2}}{2} + B _ {t} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} A _ {t} + \frac {\sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} A _ {t} + B _ {t}}{2} \\ + \frac {A _ {t} ^ {2}}{2} + B _ {t} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} \frac {3}{2} A _ {t} + \frac {A _ {t} ^ {2}}{2} + \frac {3}{2} B _ {t} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}). \\ \end{array}
$$
We can apply AM-GM to  $A_{t}^{2} = \left(6\sqrt{\frac{b\delta}{m}} + 146\sqrt{\frac{r^{*}}{b}}\right)$  to get  $A^{2} \leq \frac{72b\delta}{m} + \frac{42632r}{b}$ .
Finally, by collecting like terms, we have that  $R_{\mathrm{target}}(\hat{f}_0,\hat{h})$  is bounded by
$$
\sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right)} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r}{b}}\right) + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r}{2 b} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}).
$$
Theorem 15. Let  $\hat{h}$  and  $\hat{f}_0$  be the learned representation and target predictor, as described in Eqns. (2) and (3). Let  $\psi_1(r) \geq b\mathbb{E}\tilde{\mathfrak{R}}_{\mathbf{Z}}(\ell \circ \mathcal{F}^{\otimes t} \circ \mathcal{H}, r)$  and  $\psi_2(r) \geq b\mathbb{E}\tilde{\mathfrak{R}}_{\mathbf{Z}}(\ell \circ \mathcal{F}_0, r)$  with  $r_1^*$  and  $r_2^*$  the fixed points of  $\psi_1(r)$  and  $\psi_2(r)$ , respectively. Then, under Assumption 1.A and that  $\pmb{f}^{*}$  is  $(\nu, \epsilon)$ -diverse over  $\mathcal{F}_0$  w.r.t.  $h^*$ , with probability at least  $1 - 4e^{-\delta}$ , the transfer learning risk is upper-bounded by,
$$
\begin{array}{l} R _ {t a r g e t} (\hat {f} _ {0}, \hat {h}) \leq R _ {t a r g e t} (f _ {0} ^ {*}, h ^ {*}) + \sqrt {R _ {t a r g e t} (f _ {0} ^ {*} , h ^ {*})} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r _ {1} ^ {*}}{b}}\right) + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r _ {1} ^ {*}}{2 b} \\ + \frac {1}{\nu} \left(\sqrt {R _ {\mathrm {s o u r c e}} (\boldsymbol {f} ^ {*}, h ^ {*})} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r _ {2} ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r _ {2} ^ {*}}{b}\right) + \varepsilon . \\ \end{array}
$$
Proof of Theorem 15. The proof is by first applying Theorem 14, then  $(\nu, \varepsilon)$ -diversity assumption, and finally Theorem 13. This is demonstrated with the following series of inequalities.
$$
\begin{array}{l} R _ {\text {t a r g e t}} (\hat {f} _ {0}, \hat {h}) \leq \sqrt {R _ {\text {t a r g e t}} (f _ {0} ^ {*} , h ^ {*})} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r _ {1} ^ {*}}{b}}\right) + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r _ {1} ^ {*}}{2 b} + R _ {\text {t a r g e t}} (f _ {0} ^ {*}, h ^ {*}) \\ + d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) \\ \leq \sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right)} \left(9 \sqrt {\frac {b \delta}{m}} + 2 1 9 \sqrt {\frac {r _ {1} ^ {*}}{b}}\right) + \frac {1 7 1 b \delta}{m} + \frac {2 1 9 6 7 r _ {1} ^ {*}}{2 b} + R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) \\ + \frac {1}{\nu} \left(\sqrt {R _ {\mathrm {s o u r c e}} (\boldsymbol {f} ^ {* , h ^ {*}})} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r _ {2} ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r _ {2} ^ {*}}{b}\right) + \varepsilon . \\ \end{array}
$$
D.4 Empirical Local Rademacher Complexity Bound
The following lemma allows us to discard the composition of a loss so long as we appropriately scale the cover.
Lemma 18. For a non-negative  $H$ -smooth loss and any function class  $\mathcal{F}$ , we have:
$$
\mathcal {N} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t}, r\right), \epsilon , n t\right) \leq \mathcal {N} \left(\mathcal {F} ^ {\otimes t}, \frac {\epsilon}{\sqrt {1 2 H r n t}}, n t\right).
$$
We follow the same first few steps as in Srebro et al. [2010] but then bound the maximum by summation to recover an empirical  $L_{2}$  cover.
Proof of Lemma 18. By Lemma 13 we see that for a non-negative  $H$ -smooth function  $f$ , we have that  $(f(t) - f(r))^2 \leq 6H(f(t) + f(r))(t - r)^2$ . Using this inequality, for any sample  $\mathbf{X} = \{(x_j^i, y_j^i)\}_{j \in [t], i \in [n]}$  of  $nt$  points, for an  $f_\epsilon$ , we have,
$$
\begin{array}{l} \sup  _ {\mathbf {x}} \sqrt {\frac {1}{n t} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {t} \left(\ell \left(f \left(x _ {j} ^ {i}\right) , y _ {j} ^ {i}\right) - \ell \left(f _ {\varepsilon} \left(x _ {j} ^ {i}\right) , y _ {j} ^ {i}\right)\right) ^ {2}} \\ \leq \sup  _ {\mathbf {X}} \sqrt {\frac {6 H}{n t} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {t} \left(\ell \left(f \left(x _ {j} ^ {i}\right) , y _ {j} ^ {i}\right) + \ell \left(f _ {\varepsilon} \left(x _ {j} ^ {i}\right) , y _ {j} ^ {i}\right)\right) \left(f \left(x _ {j} ^ {i}\right) - f _ {\varepsilon} \left(x _ {j} ^ {i}\right)\right) ^ {2}} \\ \leq \sup  _ {\mathbf {X}} \sqrt {\frac {6 H}{n t} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {t} \left(\ell \left(f \left(x _ {j} ^ {i}\right) , y _ {j} ^ {i}\right) + \ell \left(f _ {\varepsilon} \left(x _ {i}\right) , y _ {i}\right)\right)} \sqrt {\max  _ {i \in [ n ]} \left(f \left(x _ {i}\right) - f _ {\varepsilon} \left(x _ {i}\right)\right) ^ {2}} \\ \leq \sup _ {\mathbf {X}} \sqrt {1 2 H r} \sqrt {\max _ {i \in [ n ] , j \in [ t ]} \left(f \left(x _ {j} ^ {i}\right) - f _ {\varepsilon} \left(x _ {j} ^ {i}\right)\right) ^ {2}} \\ \leq \sup  _ {\mathbf {X}} \sqrt {1 2 H r} \sqrt {\sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {t} \left(f \left(x _ {j} ^ {i}\right) - f _ {\varepsilon} \left(x _ {j} ^ {i}\right)\right) ^ {2}} \\ = \sup  _ {\mathbf {X}} \sqrt {1 2 H r n t} \sqrt {\frac {1}{n t} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {t} \left(f \left(x _ {j} ^ {i}\right) - f _ {\varepsilon} \left(x _ {j} ^ {i}\right)\right) ^ {2}}. \\ \end{array}
$$
That is, a cover of  $\{\pmb{f} \in \mathcal{F}^{\otimes t}: P\pmb{f} \leq r\}$  at radius  $\epsilon / \sqrt{12Hrt}$  is also a cover of  $\mathcal{L}_{\ell}(\mathcal{F}^{\otimes}, r)$  at radius  $\epsilon$ , and we can conclude that,
$$
\mathcal {N} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t}, r\right), \epsilon , n t\right) \leq \mathcal {N} \left(\mathcal {F} ^ {\otimes t}, \frac {\epsilon}{\sqrt {1 2 H t r n}}, n t\right),
$$
Theorem 16 (Smooth non-negative local Rademacher complexity bound). Under the setting of Theorem 15 along with  $\ell$  being  $H$ -smooth and Assumptions 1.B and 1.C, we have:
$$
\tilde {\Re} _ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \log (n t) ^ {2} + \frac {8 0 \sqrt {3} D \sqrt {H r} \log (n t) + 4 \sqrt {b r}}{\sqrt {n t}}, \tag {29}
$$
and if  $16G(\mathcal{F}^{\otimes t}(\mathcal{H})) \leq D$  and  $\sqrt{b} \geq 640\sqrt{3}\Pi(\mathcal{F}^{\otimes t}(H))$ , which holds for large enough samples,
$$
\tilde {\mathfrak {R}} _ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq 2 5 6 0 \sqrt {3} \sqrt {r} \Pi \left(\mathcal {F} ^ {\otimes t} (H)\right) \log \left(\frac {e \sqrt {3} \sqrt {b}}{1 9 2 0 \Pi \left(\mathcal {F} ^ {\otimes t} (H)\right)}\right) \tag {30}
$$
$$
\begin{array}{l l l l l l} w h e r e \quad G (\mathcal {F} ^ {\otimes t} (\mathcal {H})) & = & L \tilde {\mathfrak {G}} _ {n t} (\mathcal {H}) & + & \tilde {\mathfrak {G}} _ {n} (\mathcal {F}) & a n d & \Pi (\mathcal {F} ^ {\otimes t} (H)) \\ \sqrt {H} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \log \left(\frac {e D}{1 6 G (\mathcal {F} ^ {\otimes t} (\mathcal {H}))}\right). \end{array}
$$
Proof of Theorem 16. Let  $\mathbf{Z}$  denote the dataset of  $nt$  points. We first compute  $\sup_{\mathbf{g} \in \mathcal{F}^{\otimes t}(\mathcal{H})} \sup_{\mathbf{Z}} \|f\|_{L^2(\mathbf{Z})}$ . Note that,
$$
\begin{array}{l} \sup  _ {\boldsymbol {g} \in \ell (\mathcal {F} ^ {\otimes t} (\mathcal {H}))} \sup  _ {\boldsymbol {\mathbf {Z}}} \| \boldsymbol {g} \| _ {L ^ {2} (\boldsymbol {\mathbf {Z}})} \leq \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}, h \in \mathcal {H}} \sup  _ {\boldsymbol {\mathbf {Z}}} \| \ell \circ \boldsymbol {f} \circ h \| _ {L ^ {2} (\boldsymbol {\mathbf {Z}})} \\ = \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}, h \in \mathcal {H}} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \left(\ell \left(f _ {j} \left(h \left(x _ {j} ^ {i}\right)\right)\right)\right) ^ {2} \\ \leq \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}, h \in \mathcal {H}} b \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = n} ^ {n} \ell \left(f _ {j} \left(h \left(x _ {j} ^ {i}\right)\right)\right) \leq b r. \\ \end{array}
$$
Therefore, by refined Dudley's entropy integral formula, we have
$$
\begin{array}{l} \tilde {\Re} _ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq \inf  _ {\sqrt {b r} \geq \alpha \geq 0} \left\{4 \alpha + 1 0 \int_ {\alpha} ^ {\sqrt {b r}} \sqrt {\frac {\log \mathcal {N} \left(\mathcal {L} _ {\ell} (r) , \varepsilon , n t\right)}{n t}} d \varepsilon \right\} (byLemma7) \\ \leq \inf  _ {\sqrt {b r} \geq \alpha \geq 0} \left\{4 \alpha + 1 0 \int_ {\alpha} ^ {\sqrt {b r}} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} ^ {\otimes t} (\mathcal {H}) , \frac {\varepsilon}{\sqrt {1 2 H r n t}} , n t)}{n t}} d \varepsilon \right\} \quad (b y \\ \leq \inf  _ {\sqrt {b r} \geq \alpha \geq 0} \left\{4 \alpha + 2 0 \int_ {\alpha} ^ {\sqrt {b r}} \frac {\sqrt {1 2 H r n t} \tilde {\mathcal {G}} _ {n t} (\mathcal {F} ^ {\otimes t} (\mathcal {H}))}{\varepsilon} \sqrt {\frac {1}{n t}} d \varepsilon \right\} \quad \text {(b y L e m m a 2)} \\ = \inf  _ {\sqrt {b r} \geq \alpha \geq 0} \left\{4 \alpha + 4 0 \sqrt {3 H r} \tilde {\mathcal {G}} _ {n t} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \log \left(\frac {\sqrt {b r}}{\alpha}\right) \right\}. (31) \\ \end{array}
$$
Now we apply the Gaussian chain rule Theorem 8 to  $\tilde{\mathcal{G}}_{nt}(\mathcal{F}^{\otimes t}(\mathcal{H}))$ , giving us:
$$
\tilde{\mathfrak{R}}_{nt}(\mathcal{L}_{\ell}(\mathcal{F}^{\otimes t}(\mathcal{H})),r)\leq \inf_{\sqrt{br}\geq \alpha \geq 0}\left\{4\alpha +40\sqrt{3Hr}\inf_{D\geq \delta \geq 0}\left\{4\delta +64G\left(\mathcal{F}^{\otimes t}(\mathcal{H})\right)\log \left(\frac{D}{\delta}\right)\right\} \log \left(\frac{\sqrt{br}}{\alpha}\right)\right\} .
$$
Setting  $\delta = \min(16G(\mathcal{F}^{\otimes t}(\mathcal{H}), D)$  and simplifying by bounding the minimum with  $16G(\mathcal{F}^{\otimes t}(\mathcal{H}))$  we have:
$$
\tilde {\mathfrak {N}} _ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq 2 5 6 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \log \left(\frac {e D}{\min \left(1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) , D\right)}\right) \log \left(\frac {\sqrt {b r}}{\alpha}\right) + 4 \alpha
$$
Doing the same for  $\delta$  with  $\min \left(\sqrt{br},640\sqrt{3} G\left(\mathcal{F}^{\otimes t}(\mathcal{H})\right)\sqrt{Hr}\log \left(\frac{eD}{\min(16G(\mathcal{F}^{\otimes t}(\mathcal{H}),D)}\right)\right)$  we have
$$
\begin{array}{l} \tilde {\Re} _ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq 2 5 6 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \\ \times \log \left(\frac {e D}{\operatorname* {m i n} \left(1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) , D\right)}\right) \\ \times \log \left(\frac {e \sqrt {b}}{\operatorname* {m i n} \left(\sqrt {b} , 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log \left(\frac {e D}{\operatorname* {m i n} (1 6 G (\mathcal {F} ^ {\otimes t} (\mathcal {H})) , D)}\right)\right)}\right). \\ \end{array}
$$
The above expression is optimal with respect to the parameters  $\alpha$  and  $\delta$  and clearly gives four different possibilities based on the two minima. Since, under a bounded setting,  $G(\mathcal{F}^{\otimes t}(\mathcal{H})$  would typically go down with  $nt$ , for large enough  $nt$  we have that
$$
\min  \left(1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), D\right) = 1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)
$$
$$
\begin{array}{l} \min  \left(\sqrt {b r}, 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \log \left(\frac {e D}{\min  \left(1 6 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) , D\right)}\right)\right) \\ = 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \log \left(\frac {e D}{1 2 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right). \\ \end{array}
$$
Under these conditions, we have
$$
\tilde {\mathfrak {N}} _ {n t} (\tilde {\mathcal {L}} _ {\ell} (\mathcal {F} ^ {\otimes t} (\mathcal {H})), r) \leq 2 5 6 0 \sqrt {3} \sqrt {r} \Pi (\mathcal {F} ^ {\otimes t} (H)) \log \left(\frac {e \sqrt {3} \sqrt {b}}{1 9 2 0 \Pi (\mathcal {F} ^ {\otimes t} (H))}\right),
$$
where  $\Pi (\mathcal{F}^{\otimes t}(H)) = G\left(\mathcal{F}^{\otimes t}(\mathcal{H})\right)\sqrt{H}\log \left(\frac{eD}{16G(\mathcal{F}^{\otimes t}(\mathcal{H}))}\right).$
Under the other three possibilities for the minima some or all the logarithmic factors are identically one and lead to a simplified bound.
Alternatively, for a more simple bound which always holds set  $\alpha = \sqrt{\frac{br}{nt}}$  and  $\delta = \frac{D}{\sqrt{nt}}$  to get
$$
\tilde {\Re} _ {n t} (\tilde {\mathcal {L}} _ {\ell} (\mathcal {F} ^ {\otimes t} (\mathcal {H})), r) \leq 6 4 0 \sqrt {3} G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H r} \log (n t) ^ {2} + \frac {8 0 \sqrt {3} D \sqrt {H r} \log (n t) + 4 \sqrt {b r}}{\sqrt {n t}}.
$$