SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\mathfrak {G}} _ {\mathbf {Z}} \left(\mathcal {Q} ^ {\otimes p}\right) = \mathbb {E} _ {g _ {i, j, k}} \left[ \sup  _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \frac {1}{n p} \sum_ {i, j, k = 1} ^ {n, p, u} g _ {i, j, k} \left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k} \right],
$$
where  $g_{i,j,k}$  are i.i.d.  $\mathcal{N}(0,1)$  random variables. We define the worst-case Rademacher width as  $\tilde{\Re}_n(\mathcal{Q}^{\otimes p}) = \sup_{\mathbf{Z}\in \mathcal{Z}^{p^n}}\tilde{\Re}_{\mathbf{Z}}(\mathcal{Q}^{\otimes p})$  and, analogously, the worst-case Gaussian width as  $\tilde{\mathfrak{G}}_n(\mathcal{Q}^{\otimes p}) = \sup_{\mathbf{Z}\in \mathcal{Z}^{p^n}}\tilde{\mathfrak{G}}_{\mathbf{Z}}(\mathcal{Q}^{\otimes p})$ . The above definitions generalize the standard Rademacher and Gaussian width for real-valued functions and can be derived as a special case of the more general set-based definitions [Wainwright, 2019] (see Appendix A).
We now define local Rademacher width as  $\tilde{\mathfrak{R}}_{\mathbf{Z}}(\mathcal{Q}^{\otimes p},r) = \tilde{\mathfrak{R}}_{\mathbf{Z}}(\{q\in \mathcal{Q}^{\otimes p}:V(\boldsymbol {q})\leq r\})$  , where  $V:\mathcal{Q}^p\to \mathbb{R}$  . That is, the local Rademacher width is simply the Rademacher width restricted by a functional. In our applications, we consider any  $V$  satisfying  $V(\pmb {q})\leq b\frac{1}{np}\sum_{i,j,k = 1}(q_j(z_j^i))_k,$  where  $b$  is the uniform bound on the range of  $q$  in  $\ell_2$  -norm. Note this recovers the classical local Rademacher width for real-valued functions and non-product spaces  $(p = u = 1)$  . We are mostly interested in the local Rademacher width of the loss applied to the pairwise composition between our vector-valued hypothesis class and our representation class. Specifically, if we define such a class as  $\mathcal{L}_{\ell}(\mathcal{F}^{\otimes t}(\mathcal{H})) = \{(\ell \circ \pmb {f}_1\circ h,\dots ,\ell \circ \pmb {f}_t\circ h)\mid h\in \mathcal{H},\pmb {f}\in \mathcal{F}^{\otimes t}\}$  , then we seek to bound  $\tilde{\Re}_n(\mathcal{L}_\ell (\mathcal{F}^{\otimes t}(\mathcal{H})),r)$  . Crucially, our bound will be a sub-root function in  $r$  , where we define this type of function below.
Definition 3 (Sub-root Function). A function  $\psi : [0, \infty) \to [0, \infty)$  is sub-root if it is nonnegative, nondecreasing, and if  $r \mapsto \psi(r) / \sqrt{r}$  is nonincreasing for  $r > 0$ .
Sub-root functions have the desirable properties of always being continuous and having a unique fixed point, i.e.,  $r^* = \psi(r^*)$  is only satisfied by some  $r^*$ ; see Lemma 8 in Appendix A.
In this section, we detail our local Rademacher complexity results for transfer learning (TL) and MTL via MTRL, a bound on the local Rademacher complexity of smooth bounded non-negative losses, and the application of this bound to both TL and MTRL.
Our first result is a local Rademacher complexity result for TL via MTRL.
Theorem 1. Let  $\hat{h}$  and  $\hat{f}_0$  be the learned representation and target predictor, as described in Eqns. (2) and (3). Under Assumption 1.A, if  $f^{*}$  is  $(\nu ,\epsilon)$ -diverse over  $\mathcal{F}_0$  w.r.t.  $h^*$  and let  $\psi_{1}$  and  $\psi_{2}$  be sub-root functions such that  $\psi_1(r)\geq \Re_n(\ell \circ \mathcal{F}^{\otimes t}\circ \mathcal{H},r)$  and  $\psi_{2}(r)\geq \Re_{m}(\ell \circ \mathcal{F}_{0},r)$ , then with probability at least  $1 - 4e^{-\delta}$ , the transfer learning risk is upper-bounded by
$$
\begin{array}{l} R _ {\text {t a r g e t}} \left(\hat {f} _ {0}, \hat {h}\right) \leq R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) + c _ {1} \left(\sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} \left(\sqrt {\frac {b \delta}{m}} + \sqrt {\frac {r _ {1} ^ {*}}{b}}\right) + \frac {b \delta}{m} + \frac {r _ {1} ^ {*}}{b}\right) \\ + \frac {1}{\nu} \left(c _ {2} \left(\sqrt {R _ {\mathrm {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right)} \left(\sqrt {\frac {b \delta}{n t}} + \sqrt {\frac {r _ {2} ^ {*}}{b}}\right) + \frac {b \delta}{n t} + \frac {r _ {2} ^ {*}}{b}\right)\right) + \varepsilon , \tag {4} \\ \end{array}
$$
where  $r_1^*$  and  $r_2^*$  are the fixed points of  $\psi_1(r)$  and  $\psi_2(r)$ , respectively, and  $c_1$  and  $c_2$  are absolute constants.
2 Bartlett et al. [2005] uses "complexity", but we use "width", as common in stochastic processes.
For exact constants for each term, please refer to Appendix D.
Inequality (4) bounds the transfer risk in terms of task diversity parameters  $\nu$  and  $\epsilon$ ; local Rademacher complexity parameters, fixed points  $r_1^*$  and  $r_1^*$ ; the minimum risk for the target task; and the minimum average risk for the source tasks.
As discussed in Xu and Tewari [2021], there exist settings where the task diversity parameters  $\nu$  and  $\epsilon$  are favorable, i.e.,  $\epsilon = 0$  and  $\nu = \Theta(1)$ , so we will disregard them in the subsequent discussion. The bound is an optimistic rate because it interpolates between  $\sqrt{r_1^*} + \sqrt{r_2^*}$  and  $r_1^* + r_2^*$ , depending on both  $R_{\mathrm{target}}(f_0^*, h^*)$  and  $R_{\mathrm{source}}(f^*, h^*)$ .
The fixed point  $r_1^*$  is a function of  $\mathcal{F}^{\otimes t}(\mathcal{H})$  and encodes its complexity measured with respect to the number of samples for the source tasks (among other class-specific parameters). The same reasoning holds for  $r_2^*$  w.r.t.  $\mathcal{F}_0$ .
We consider  $\mathcal{H}$  to be complex and both  $\mathcal{F}$  and  $\mathcal{F}_0$  to be simple. So, if we instead learned the target task directly, then we would pay the complexity of  $\mathcal{F}_0(\mathcal{H})$  against  $m$  samples. In contrast, in our bound, we only pay the complexity of  $\mathcal{F}_0$  for  $m$  samples and the complexity of  $\mathcal{F}^{\otimes t}(\mathcal{H})$  for  $nt$  samples. Since in many settings  $nt$  is much larger than  $m$ , we are successfully leveraging our representation to learn the target task.
The dependence on minimum risk is useful in various settings. An instructive regime is when  $nt \gg m$  with  $R_{\mathrm{target}}(f_0^*, h^*) = 0$  and  $R_{\mathrm{source}}(\pmb{f}^*, h^*) > 0$ ; in this case, we achieve a bound of  $\sqrt{r_1^*} + r_2^*$ . This demonstrates that data-abundant environments can be leveraged to learn representations that work well in small sample regimes.
The next result uses a decoupling of function classes, the Gaussian chain rule of Tripuraneni et al. [2020], and smoothness to bound  $r_1^*$  and  $r_2^*$ . We see that under this additional assumption that  $\ell$  is  $H$ -smooth,  $r_1^*$  and  $r_2^*$  decay quickly.
Theorem 2. Under the setting of Theorem 1 along with  $\ell$  being  $H$ -smooth and Assumptions 1.B and 1.C, with probability at least  $1 - 6e^{-\delta}$ , the fixed points in Inequality (4),
$$
r _ {1} ^ {*} \leq c _ {3} b \left(H \tilde {\mathfrak {G}} _ {m} ^ {2} \left(\mathcal {F} _ {0} \circ \hat {h}\right) \log^ {2} (m) + (1 + \delta) \frac {b}{m}\right), a n d
$$
$$
r _ {2} ^ {*} \leq c _ {4} b \left(\left(L ^ {2} \tilde {\mathfrak {G}} _ {n t} ^ {2} (\mathcal {H}) + \tilde {\mathfrak {G}} _ {n} ^ {2} (\mathcal {F})\right) H \log (n t) ^ {4} + \frac {D ^ {2} H \log^ {2} (n t) + b (1 + \delta)}{n t}\right),
$$
where  $c_3$  and  $c_4$  are absolute constants.
Remark 1. Note that in Theorem 2 it is not necessary to apply the chain rule. If the chain rule is not used, Assumptions 1.B and 1.C is not needed. The fixed point  $r_2^*$  would be bounded as a function of  $\mathcal{F}^{\otimes t}(\mathcal{H})$ . In the following, we use the above decomposed form for interpretability.
Note that for constant  $L, D, H, b$ , and  $\delta$ , the terms on the right are always fast, so the rate of decay depends crucially on the behavior of the Gaussian widths for each class. For many function classes used in machine learning, it is common that  $\mathfrak{G}_{nt}(\mathcal{H}) \approx \sqrt{\frac{C(\mathcal{H})}{nt}}$  and  $\mathfrak{G}_n(\mathcal{F}) \approx \sqrt{\frac{C(\mathcal{F})}{n}}$ , where  $C(\cdot)$  is some notion of complexity of the function class, which could be, for instance, the VC dimension, pseudo-dimension, fat-shattering dimension, etc. Therefore, it is common that
$$
L ^ {2} \tilde {\mathfrak {G}} _ {n t} ^ {2} (\mathcal {H}) + \tilde {\mathfrak {G}} _ {n} ^ {2} (\mathcal {F}) \approx L ^ {2} \frac {C (\mathcal {H})}{n t} + \frac {C (\mathcal {F})}{n} \mathrm {a n d} \tilde {\mathfrak {G}} _ {n} ^ {2} (\mathcal {F} _ {0} \circ \hat {h}) \approx \frac {C (\mathcal {F} _ {0})}{m}.
$$
Recall, from our discussion following Theorem 1, that these fixed points control the order of the rate. Therefore, this theorem interpolates between a rate of  $\frac{1}{\sqrt{m}} + \frac{1}{\sqrt{nt}}$  and  $\frac{1}{m} + \frac{1}{nt}$ , where the fast rate is achieved in a realizable setting.
Linear classes. As an example let us consider a linear projection onto a lower dimensional space along with linear regressors and squared loss. Let  $\mathcal{X} = \mathbb{R}^d$ ,  $\mathcal{Y} = \mathbb{R}$ , and
$$
\mathcal {F} = \left\{f \mid f (\mathbf {z}) = \boldsymbol {\alpha} ^ {\top} \mathbf {z}, \boldsymbol {\alpha} \in \mathbb {R} ^ {k}, \| \boldsymbol {\alpha} \| \leq c \right\},
$$
$$
\mathcal {H} = \left\{\mathbf {h} \mid \mathbf {h} (\mathbf {x}) = \mathbf {B} ^ {\top} \mathbf {x}, \mathbf {B} \in \mathbb {R} ^ {d \times k}, \mathbf {B} \text {i s a m a t r i x w i t h o r t h o n o r m a l c o l u m n s} \right\}.
$$
If we assume that  $P_{x}$  is sub-Gaussian, then standard arguments (see Tripuraneni et al. [2020]) show that  $\tilde{\mathfrak{G}}_m^2 (\mathcal{F}_0\circ \hat{h})\leq \mathcal{O}(\frac{k}{m}),\tilde{\mathfrak{G}}_n^2 (\mathcal{F})\leq \mathcal{O}(\frac{k}{n})$  and  $\tilde{\mathfrak{G}}_{nt}^{2}(\mathcal{H})\leq \mathcal{O}(\frac{k^{2}d}{nt})$  . To simplify the bounds let  $L = 1$ $H = 1$ $D = 1$ $b = 1$  , and  $\delta = 0.05$  . Thus, under the assumptions of Theorem 2, the fixed points are bounded by
$$
r _ {1} ^ {*} \lesssim \left(\frac {k}{m} + \frac {1}{m}\right), \text {a n d} r _ {2} ^ {*} \lesssim \left(\frac {k ^ {2} d}{n t} + \frac {k}{n} + \frac {1}{n t}\right),
$$
$$
\begin{array}{l} R _ {\text {t a r g e t}} \left(\hat {f} _ {0}, \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) \lesssim \left(\sqrt {R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)} \left(\sqrt {\frac {k}{m}}\right) + \frac {k}{m}\right) \\ + \frac {1}{\nu} \left(\left(\sqrt {R _ {\mathrm {s o u r c e}} (\boldsymbol {f} ^ {*}, h ^ {*})} \left(\sqrt {\frac {k ^ {2} d}{n t} + \frac {k}{n}}\right) + \frac {k ^ {2} d}{n t} + \frac {k}{n}\right)\right) + \varepsilon . \\ \end{array}
$$
The following result is such a bound on the local Rademacher complexity for a non-negative  $H$ -smooth loss and any function class  $\mathcal{H}$ .
Proposition 1 (Smooth non-negative local Rademacher complexity bound). Under the setting of Theorem 1 along with  $\ell$  being  $H$ -smooth and Assumptions 1.B and 1.C, there exists an absolute constants  $c_{3}$  and  $c_{4}$  such that
$$
\Re_ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq c _ {3} \sqrt {r} \left(G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log (n) ^ {2} + \frac {D \sqrt {H} \log (n)}{\sqrt {n}} + \sqrt {\frac {b}{n t}}\right), \tag {5}
$$
$$
\Re_ {n t} \left(\mathcal {L} _ {\ell} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right), r\right) \leq c _ {4} \sqrt {r} \left(\Pi \left(\mathcal {F} ^ {\otimes t} (H)\right) \log \left(\frac {e \sqrt {b}}{\Pi \left(\mathcal {F} ^ {\otimes t} (H)\right)}\right) + \sqrt {\frac {b}{n t}}\right), \tag {6}
$$
where  $G(\mathcal{F}^{\otimes t}(\mathcal{H})) = L\tilde{\mathfrak{G}}_{nt}(\mathcal{H}) + \tilde{\mathfrak{G}}_n(\mathcal{F})$  and  $\Pi (\mathcal{F}^{\otimes t}(H)) = \sqrt{H} G\left(\mathcal{F}^{\otimes t}(\mathcal{H})\right)\log \left(\frac{eD}{G(\mathcal{F}^{\otimes t}(\mathcal{H}))}\right)$
When we specialize the above bound to the standard single-task non-compositional setting and compare with Srebro et al. [2010], then
$$
\begin{array}{l} \tilde {\mathfrak {R}} _ {n} (\mathcal {L} _ {\ell} (\mathcal {F}, r)) \leq c \tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H r} \log \left(\frac {e \sqrt {b}}{\tilde {\mathfrak {G}} _ {n} (\mathcal {F}) \sqrt {H}}\right) \\ \leq c \tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H r} \sqrt {\log (n)} \log \left(\frac {e \sqrt {b}}{\tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H \log (n)}}\right) \\ \leq c ^ {\prime} \tilde {\Re} _ {n} (\mathcal {F}) \sqrt {H r} \sqrt {\log (n)} \log \left(\frac {e b}{H \log (n)} \frac {n}{B ^ {2}}\right). \\ \end{array}
$$
The second inequality uses the relation  $\tilde{\mathfrak{G}}_n(\mathcal{F})\leq 2\sqrt{\log(n)}\tilde{\mathfrak{R}}_n(\mathcal{F})$  [Wainwright, 2019, p. 155] and that  $x\mapsto x\log eC / x$  is increasing in  $x$  until  $x = C$ . The third inequality uses Khintchine's inequality,  $\tilde{\mathfrak{R}}_n(\mathcal{F})\geq \frac{\sqrt{2}B}{\sqrt{n}}$ , where  $B$  is the  $\ell_2$ -bound on the range of  $\mathcal{F}$ . Thus, the upper bound above is always better than the bound in Srebro et al. [2010] $^4$  for large enough  $n$ . Furthermore, under the additional though well-studied assumption that  $\ell (0)\leq HB^2$  [Arora et al., 2022, Shamir, 2015], ours is better for  $n = \Omega (1)$ . Finally, the Gaussian and Rademacher widths are of the same order for the class of linear predictors, bounded in norm, by a strongly convex regularizer, in general non-Euclidean settings. Hence, in this setting, our bound is smaller by a factor of  $\sqrt{\log n}$  in Kakade et al. [2008] - see Appendix E for a detailed comparison.
3.1 Multi-Task Learning
Next, we detail some theorems foundational to the results above. For those results, we were able to link MTL with MTRL by using the task diversity assumption. Therefore, we naturally have the following local Rademacher result for MTL.
Theorem 3. Let  $(\hat{f},\hat{h})$  be an empirical risk minimizer of  $\hat{R}_{source}(\cdot ,\cdot)$  as given in Eqn. (2). Under Assumption 1.A, let  $\psi$  be a sub-root function such that  $\psi (r)\geq \Re_n(\mathcal{F}^{\otimes t}(\mathcal{H}),r)$  with  $r^*$  the fixed point of  $\psi (r)$ , then with probability  $1 - 2e^{-\delta}$
$$
R _ {\text {s o u r c e}} (\hat {\boldsymbol {f}}, \hat {h}) \leq R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) + c \left(\sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*} , h ^ {*}\right)} \left(\sqrt {\frac {b \delta}{n t}} + \sqrt {\frac {r ^ {*}}{b}}\right) + \frac {b \delta}{n t} + \frac {r ^ {*}}{b}\right), \tag {7}
$$
<sup>4</sup>We note that Srebro et al. [2010] incur an additional  $\sqrt{rb / n}$  term when converting from complexity to width
As above, we can use our bound on the local Rademacher complexity of a  $H$ -smooth loss class to get the following result. This result is similar to Srebro et al. [2010], Theorem 1, yet ours is in an MTL via MTRL setting.
Theorem 4. Under the setting of Theorem 3 along with  $\ell$  being  $H$ -smooth and Assumptions 1.B and 1.C, with probability at least  $1 - 3e^{-\delta}$ ,  $r^*$  of  $\psi (r)$  is bounded by
$$
c b \Big (\Big (L ^ {2} \tilde {\mathfrak {G}} _ {n t} ^ {2} (\mathcal {H}) + \tilde {\mathfrak {G}} _ {n} ^ {2} (\mathcal {F}) \Big) H \log (n t) ^ {4} + \frac {D ^ {2} H \log (n t) ^ {2}}{n t} + \frac {b (1 + \delta)}{n t} \Big),
$$
3.2 Local Rademacher complexity chain rule
Using the chain rule of Tripuraneni et al. [2021], as we did above, to decouple the complexities of the representation and prediction classes is desirable. Yet, our general local Rademacher complexity Theorems 1 and 3, as stated, do not seem to have this property. Consequently, we develop a local chain rule, which aims to separate the local complexities of learning the representation and predictor. The main result is the following.
Theorem 5. Suppose the loss function  $\ell$  is  $L_{\ell}$ -Lipschitz. Define the restricted representation and predictor classes as follows,
$$
\ell \circ \mathcal {F} _ {\mathbf {X}} (r) := \left\{\ell \circ \boldsymbol {f} \in \ell \circ \mathcal {F} ^ {\otimes t}: \exists h \in \mathcal {H}: V (\ell \circ \boldsymbol {f} \circ h) \leq r \right\}
$$
$$
\mathcal {H} _ {\mathbf {X}} (r) := \left\{h \in \mathcal {H}: \exists \boldsymbol {f} \in \mathcal {F} ^ {\otimes t}: V (\ell \circ \boldsymbol {f} \circ h) \leq r \right\},
$$
where  $V$  is the functional in the local Rademacher complexity description. Under Assumptions 1.B and 1.C and that the worst-case Gaussian width of the above is bounded by the sub-root functions  $\psi_{\mathcal{F}}$  and  $\psi_{\mathcal{H}}$ , respectively, there exists an absolute constant  $c$  such that
$$
\tilde {\mathfrak {G}} _ {n} (\mathcal {L} _ {\ell} (\mathcal {F} ^ {\otimes t} (\mathcal {H}), r)) \leq c \Big ((L L _ {\ell} \psi_ {\mathcal {F}} (r) + \psi_ {\mathcal {H}} (r)) \log (n t) + \frac {D}{(n t) ^ {2}} \Big).
$$
4 Proof Techniques
In this section, we give details on some of the tools we use by giving proof sketches for the results in Section 3. In the next section, we cover some theorems at the root of all our results.
Proof sketch for Theorem 1. Let  $\tilde{f}_0 = \arg \min_{f\in \mathcal{F}}R_{\mathrm{target}}(f,\hat{h})$ . By a risk decomposition, we can bound the excess transfer risk by  $R_{\mathrm{target}}(\hat{f}_0,\hat{h}) - R_{\mathrm{target}}(\hat{f}_0,\hat{h}) + \sup_{f_0\in \mathcal{F}_0}\inf_{f'\in \mathcal{F}}\{R_{\mathrm{target}}(f',\hat{h}) - R_{\mathrm{target}}(f_0,h^*)\}$ . We can now use Theorem 3, with  $t = 1$ , to bound  $R_{\mathrm{target}}(\hat{f}_0,\hat{h})$ . Note that  $R_{\mathrm{target}}(f_0^*,\hat{h}) - R_{\mathrm{target}}(\tilde{f}_0,\hat{h}) \leq 0$ . Now  $\inf_{f'\in \mathcal{F}}R_{\mathrm{target}}(f',\hat{h}) - R_{\mathrm{target}}(f_0^*,h^*)$  is less than  $\frac{1}{\nu} (\inf_{f'\in \mathcal{F}^{\otimes t}}R_{\mathrm{source}}(\pmb {f}',\hat{h}) - R_{\mathrm{source}}(\pmb {f}^*,h^*) + \varepsilon$ , by the task-diversity assumption. As shown in Tripuraneni et al. [2020], we can bound  $\inf_{f'\in \mathcal{F}^{\otimes t}}\{R_{\mathrm{source}}(\pmb {f}',\hat{h}) - R_{\mathrm{source}}(\pmb {f}^*,h^*)\}$  with  $R_{\mathrm{source}}(\hat{\pmb{f}},\hat{\pmb{h}})$ . We can then apply Theorem 3 to the remaining  $R_{\mathrm{source}}(\hat{\pmb{f}},\hat{\pmb{h}})$  term. By leveraging task diversity again we can bound the remaining  $R_{\mathrm{target}}(f_0^*,\hat{h})$  as a function of  $R_{\mathrm{target}}(f_0^*,h^*)$ .
Proof sketch for Proposition 1. We achieve the following bound on the local Rademacher complexity of the loss class by using a truncated version of Dudley's integral [Srebro et al., 2010, see Lemma A.3].
$$
\mathfrak {R} _ {n t} (\mathcal {L} _ {\ell} (\mathcal {F} ^ {\otimes t} (\mathcal {H})), r) \leq \inf  _ {0 \leq \alpha \leq \sqrt {b r}} \left\{4 \alpha + 1 0 (n t) ^ {- 1 / 2} \int_ {\alpha} ^ {\sqrt {b r}} \sqrt {\log \mathcal {N} _ {2} (\mathcal {L} _ {\ell} (\mathcal {F} ^ {\otimes t} (\mathcal {H})) , r) , \varepsilon , n t)} \right\}
$$
We observe that we can use smoothness to move the dependence on the loss to an appropriate scaling, which gives that  $\mathcal{N}_2(\mathcal{L}_{\ell}(\mathcal{F}^{\otimes t}(\mathcal{H})),r),\varepsilon ,nt)\leq \mathcal{N}_2(\mathcal{F}^{\otimes t}(\mathcal{H}),\varepsilon /\sqrt{12Hrt},nt)$  (Lemma 18 in the appendix). Next, we apply Sudakov minoration to bound the log covering number by the Gaussian width of our function class:  $\sqrt{\log\mathcal{N}_2(\mathcal{F}^{\otimes t}(\mathcal{H}),\varepsilon / \sqrt{12Hrt},nt)}\leq \frac{\sqrt{12Hrt}}{\epsilon}\tilde{\mathcal{G}}_n(\mathcal{F}^{\otimes t}(\mathcal{H}))$  . With these simplifications, the parameter  $\alpha$  from Dudley's integral can either be solved for exactly (as in Inequality 6), so long as  $\alpha \leq \sqrt{br}$  , or set in such a way that it always holds (like in Inequality 5). Finally, we can optionally apply the Gaussian chain rule from Tripuraneni et al. [2020] to bound  $\tilde{\mathcal{G}}_n(\mathcal{F}^{\otimes t}(\mathcal{H}))$  with  $L\tilde{\mathfrak{G}}_{nt}(\mathcal{H}) + \tilde{\mathfrak{G}}_n(\mathcal{F})$
Proof sketch for Theorem 2. The right hand side of Inequality 5 is a sub-root function in  $r$ , because it is an affine transformation of  $\sqrt{r}$ . Therefore, we just solve for the fixed point of this sub-root function, i.e., solve for  $r^*$  w.r.t. the equation below.
$$
c \sqrt {r ^ {*}} \left(G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \sqrt {H} \log (n) ^ {2} + \frac {D \sqrt {H} \log (n)}{\sqrt {n}} + \sqrt {\frac {b}{n t}}\right) = r ^ {*}. \tag {8}
$$
All that remains of the proof is basic algebraic simplifications. The proof for Inequality 6 is similar.
The proofs of the above results hinge on the following local Rademacher complexity result.
Theorem 6. Let  $\mathcal{F}$  be a class of non-negative  $b$  bounded functions. Assume that  $X_{j}^{1},\ldots ,X_{j}^{n}$  are  $n$  draws from  $P_{j}$  for  $j\in [t]$  and all nt samples are independent. Suppose that  $\operatorname {Var}\bigl [\pmb {f}\bigr ]\leq V(\pmb {f})\leq BP\pmb{f}$ , where  $V$  is a functional from  $\mathcal{F}^{\otimes t}$  to  $\mathbb{R}$  and  $B > 0$ .
Let  $\psi$  be a sub-root function with fixed point  $r^*$  such that  $B\Re (\mathcal{F}^{\otimes t},r)\leq \psi (r)$  for all  $r\geq r^{*}$ , then for  $K > 1$  and  $\delta >1$ , with probability at least  $1 - e^{-\delta}$
$$
\forall \pmb {f} \in \mathcal {F} ^ {\otimes t} P \pmb {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \pmb {f} + 2 0 0 (1 + \alpha) ^ {2} \frac {K r ^ {*}}{B} + \frac {5}{4} \frac {B K \delta}{n t} + 2 (\frac {1}{\alpha} + \frac {1}{3}) \frac {b \delta}{n t}.
$$
Yousefi et al. [2018], Theorem 9, showed a result of this form in a more general setting of Bernstein classes and a function class that is not necessarily a product space. In contrast, for simplicity, we do not consider Bernstein classes, though we believe such a generalization is possible, and our function classes are product spaces. By doing so, we achieve better constants than Yousefi et al. [2018] by using the next theorem.
Theorem 7 (Concentration with  $t$  functions). Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$  and assume that all functions  $f$  in  $\mathcal{F}$  are  $P_{j}$ -measurable for all  $j\in [t]$ , square-integrable, and satisfy  $\mathbb{E}f = 0$ . Let  $\sup_{f\in \mathcal{F}}$  essup  $f\leq 1$  and  $Z = \sup_{\pmb {f}\in \mathcal{F}^{\otimes t}}\sum_{j = 1}^{t}\sum_{i = 1}^{n}f_{j}(X_{j}^{i})$ . Let  $\sigma >0$  such that  $\sigma^2\geq \sup_{\pmb {f}\in \mathcal{F}^{\otimes t}}\frac{1}{t}\sum_{j = 1}^{t}\mathrm{Var}\big[f_j(X_j^1)\big]$  almost surely. Then, for all  $\delta \geq 0$ , we have that
$$
\mathbb {P} \{Z \leq \mathbb {E} Z + \sqrt {2 \delta (t n \sigma^ {2} + 2 \mathbb {E} Z)} + \frac {\delta}{3}) \} \leq e ^ {- \delta}.
$$
This result also holds for  $\sup_{\pmb{f}\in \mathcal{F}^{\otimes t}}\left|\sum_{j = 1}^{t}\sum_{i = 1}^{n}f_{j}(X_{j}^{i})\right|$  under the condition that  $\sup_{f\in \mathcal{F}}\| f\|_{\infty}\leq 1$
Yousefi et al. [2018] also proves something similar to the above from scratch using "Logarithmic Sobolev" inequalities. We make the observation that within our setting a proof similar to the original proof given by Bousquet [2002] still holds. To clarify this further, Yousefi et al. [2018] state:
Note that the difference between the constants in (2) [Bousquet, 2002, Theorem 2.3] and (3) [Yousefi et al., 2018, Theorem 1] is due to the fact that we were unable to directly apply Bousquet's version of Talagrand's inequality (like it was done in Bartlett et al. [2005] for scalar-valued functions) to the class of vector-valued functions.
We show that within the product space structure, which we consider to be the most natural for MTL, it is possible to achieve the same constants. Concretely, when  $t = 1$ , we realize the single function version of Bousquet [2002] exactly.