SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note, here we are again leveraging product space structure to simplify prior work. Recall in Section 5 we observed that this structure allows us to recover existing single function theorems exactly, unlike Yousefi et al. [2018]. Specifically in the proof of Theorem 9 we "commute" suprema and summations.
Here we are "commuting" infima and summations to simplify the framework specified in Tripuraneni et al. [2020] within our setting. Note they work with product spaces but do not make the simplifying observation above. Nonetheless, the definitions they provide are suitable for the more general function classes considered in Yousefi et al. [2018], which are not necessarily product spaces.
Now consider the setting detailed in the preliminaries applied to our approximation  $\hat{h}, h^* \in \mathcal{H}$  and  $\pmb{f}^* \in \mathcal{F}^{\otimes t}$  with the following ratio
$$
\begin{array}{l} \frac {d _ {\mathcal {F} , \mathcal {F} _ {0}} (\hat {h} ; h ^ {*})}{\bar {d} _ {\mathcal {F} , \boldsymbol {f} ^ {*}} (\hat {h} ; h ^ {*})} = \frac {\sup  _ {f _ {0} \in \mathcal {F} _ {0}} \inf  _ {f ^ {\prime} \in \mathcal {F}} \left\{R _ {\text {t a r g e t}} \left(f ^ {\prime} , \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} , h ^ {*}\right) \right\}}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime} , \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\}} \\ = \frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} \left(f ^ {\prime} , \hat {h}\right) - \inf  _ {f _ {0} \in \mathcal {F} _ {0}} R _ {\text {t a r g e t}} \left(f _ {0} , h ^ {*}\right)}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime} , \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\}} \tag {9} \\ = \frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} \left(f ^ {\prime} , \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*} , h ^ {*}\right)}{\inf  _ {f ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime} , \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\}}. \\ \end{array}
$$
Note every sup and inf will realize a function that minimizes their respective risk.
This ratio is a measure of how similar the performance we can expect when we use our representation learned from the source tasks to and apply it the target task. That is if we seek to control this quantity we do not care per se if the risks are small nor do we care that we are getting good performance in comparison to the ground truth, but how comparable is our performance between source and target.
We seek to control this ratio with some  $\nu$
$$
\frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} (f ^ {\prime} , \hat {h}) - R _ {\text {t a r g e t}} (f _ {0} ^ {*} , h ^ {*})}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {\prime} , \hat {h}) - R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {*} , h ^ {*}) \right\}} \leq \frac {1}{\nu}.
$$
If the risk on our source tasks and target task is similar w.r.t. to the ground truth for all tasks then  $\nu \approx 1$ . That is low (high) average risk on the source tasks leads to low (high) risk on the target task.
This is even more apparent in the realizable setting.
$$
\frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} (f ^ {\prime} , \hat {h})}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {\prime} , \hat {h})} \leq \frac {1}{\nu}.
$$
First, imagine a setting in which there are several tasks and on all but a few source tasks, we perform well.
$$
\frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} (f ^ {\prime} , \hat {h})}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {\prime} , \hat {h})} \approx \frac {\text {r e l a t i v e l y s m a l l}}{\text {r e l a t i v e l y b i g}}.
$$
Then  $\frac{1}{\nu}$  will be small, which is okay if we do not care about performance across all  $t$  tasks, as long as the target task performance is good.
Alternatively, imagine that the task does relatively poorly on the target task but is still small. This could lead to a very large  $\frac{1}{\nu}$  which is not desirable.
$$
\frac {\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} (f ^ {\prime} , \hat {h})}{\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {\prime} , \hat {h})} \approx \frac {\text {r e l a t i v e l y b i g}}{\text {r e l a t i v e l y s m a l l}}.
$$
Therefore we allow for an additive  $\varepsilon$  to capture this quantity when we get risk on the source task is very small, but we are okay with relatively poor performance on the target task.
That is we care about the following inequality
$$
\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} \left(f ^ {\prime}, \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) \leq \frac {1}{\nu} \inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime}, \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\} + \varepsilon , \tag {10}
$$
where  $\varepsilon$  encodes how much tolerance we have between the performance on the source tasks and the target task. Hence, it is Inequality (10) which we care to control over all functions in  $\mathcal{H}$ . Therefore, we will assume uniformly for all  $h' \in \mathcal{H}$  that:
$$
\inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} (f ^ {\prime}, \hat {h}) - R _ {\text {t a r g e t}} (f _ {0} ^ {*}, h ^ {*}) \leq \frac {1}{\nu} \inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {\prime}, \hat {h}) - R _ {\text {s o u r c e}} (\boldsymbol {f} ^ {*}, h ^ {*}) \right\} + \varepsilon .
$$
Although we observe this simplification, to be succinct we will still use that
$$
d _ {\mathcal {F}, \mathcal {F} _ {0}} (\hat {h}; h ^ {*}) = \inf  _ {f ^ {\prime} \in \mathcal {F}} R _ {\text {t a r g e t}} \left(f ^ {\prime}, \hat {h}\right) - R _ {\text {t a r g e t}} \left(f _ {0} ^ {*}, h ^ {*}\right) \tag {11}
$$
$$
\bar {d} _ {\mathcal {F}, \boldsymbol {f} ^ {*}} (\hat {h}; h ^ {*}) = \inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime}, \hat {h}\right) - R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) \right\} \tag {12}
$$
and use this notation in the proofs of Appendix D.3.
C Technical Miscellanea
In this section, we present technical miscellanea that we shall later refer to in obtaining our proofs of the main results in Appendix D.
C.1 Martingale Concentration Inequality
For the sake of notational succinctness, we forgo the compositional model by overloading the notation to just consider a generic function class  $\mathcal{F}:\mathcal{X}\to \mathbb{R}$
Lemma 9 (Theorem 2.1 in Bousquet [2002]). Let  $W_{1}, \ldots, W_{n}$  be independent random variables in a Polish space  $\mathcal{W}$ .
$$
\mathcal {A} = \sigma (W _ {1}, \dots , W _ {n})
$$
$$
\forall k \in [ n ] \quad \mathcal {A} _ {k} = \sigma \left(W _ {1},, \dots , W _ {k - 1}, W _ {k + 1}, \dots , W _ {n}\right)
$$
be  $\sigma$ -algebras generated by these random variables.
We denote by  $\mathbb{E}_k[\cdot ]$  the expectation taken conditionally on  $\mathcal{A}_k$ . Let  $\varphi (x) = (1 + x)\log (1 + x) - x$  and  $\psi (x) = e^{-x} - 1 + x$ .
Let  $(Z,Z_1^{\prime},\ldots ,Z_n^{\prime})$  be a sequence of  $\mathcal{A}$ -measurable random variables and let  $(Z_{k})_{k\in [n]}$  be a sequence of random variables that are  $\mathcal{A}_k$  measurable, respectively. Assume that there exists  $u > 0$  such that for all  $k\in [n]$  the following inequalities are satisfied
$$
Z _ {k} ^ {\prime} \leq Z - Z _ {k} \leq 1 \quad a. s., \quad \mathbb {E} _ {k} \left[ Z _ {k} ^ {\prime} \right] \geq 0 \quad a n d \quad Z _ {k} ^ {\prime} \leq u \quad a. s.
$$
Let  $\sigma \in \mathbb{R}$  be a real number satisfying  $\sigma^2 \geq \frac{1}{n} \sum_{k=1}^{n} \mathbb{E}_k \left[(Z_k')^2\right]$  almost surely and let  $v = (1 + u)\mathbb{E}[Z] + n\sigma^2$ . If the following condition holds
$$
\sum_ {k = 1} ^ {n} Z - Z _ {k} \leq Z \quad a. s.,
$$
we obtain, for all  $\lambda \geq 0$
$$
\log \mathbb {E} \left[ e ^ {\lambda (Z - \mathbb {E} [ Z ])} \right] \leq \psi (- \lambda) v,
$$
which gives the following bounds for all  $\delta > 0$ ,
$$
\mathbb {P} \{Z \geq \mathbb {E} [ Z ] + \delta \} \leq \exp \left(- v \varphi \left(\frac {\delta}{v}\right)\right) \quad a n d \quad \mathbb {P} \{Z \geq \mathbb {E} [ Z ] + \sqrt {2 v \delta} + \frac {\delta}{3} \} \leq e ^ {- \delta}.
$$
The following corollary is immediate as independence is sufficient Bousquet [2003] and we can express the summation over  $nt$  random variables as "double summation" where we first sum over  $q \in [n]$  and then over  $j \in [t]$ .
Corollary 1 (Lemma 9 in block structure). Assume that  $W_{j}^{1}, \ldots, W_{j}^{n}$  are  $n$  independent draws from  $P_{j}$  for  $j \in [t]$  with random variables in some polish space, where all  $nt$  samples are independent.
Let  $\mathcal{A}$  be the sigma algebra generated by  $(W_j^i)_{j\in [t],i\in [n]}$  and  $\mathcal{A}_k^q$  be the sigma algebras generated by  $(W_j^i)_{\substack{j\in [t],i\in [n]\\ (j,i)\neq (k,q)}}$ . We denote by  $\mathbb{E}_k^q [\cdot ]$  the expectation taken conditionally on  $\mathcal{A}_k^q$ .
Let  $Z$  be a  $\mathcal{A}$ -measurable r.v.,  $(\tilde{Z}_k^q)_{k \in [t], q \in [n]}$  be a sequence of  $\mathcal{A}$ -measurable random variables, and  $(Z_k^q)_{k \in [t], q \in [n]}$  be a sequence of random variables that are  $\mathcal{A}_k^q$  measurable, respectively. Assume that there exists  $u > 0$  such that for all  $k \in [t]$  and  $q \in [n]$  the following inequalities are satisfied
$$
\tilde {Z} _ {k} ^ {q} \leq Z - Z _ {k} ^ {q} \leq 1 \quad a. s., \tag {13}
$$
$$
\mathbb {E} _ {k} ^ {q} \left[ \tilde {Z} _ {k} ^ {q} \right] \geq 0, \tag {14}
$$
$$
a n d \quad \tilde {Z} _ {k} ^ {q} \leq u \quad a. s. \tag {15}
$$
Let  $\sigma \in \mathbb{R}$  be a real number satisfying
$$
\sigma^ {2} \geq \frac {1}{n t} \sum_ {k = 1} ^ {t} \sum_ {q = 1} ^ {n} \mathbb {E} _ {k} ^ {q} \left[ \left(\tilde {Z} _ {k} ^ {q}\right) ^ {2} \right] \tag {16}
$$
almost surely and let  $v = (1 + u)\mathbb{E}[Z] + n\sigma^2$ . If the following condition holds
$$
\sum_ {k = 1} ^ {t} \sum_ {q = 1} ^ {n} Z - Z _ {k} ^ {q} \leq Z \quad a. s., \tag {17}
$$
C.2 Empirical Process Lemmas
The following result is standard symmetrization.
Lemma 10. Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$  and let  $\mathbf{X}$  denote the set of inputs  $\left\{X_j^i\right\}_{i,j=1}^{n,t}$ . Assume that all functions  $f$  in  $\mathcal{F}$  are  $P_j$ -measurable for all  $j \in [t]$ . Then,
$$
\mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) - \mathbb {E} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) \right] \leq 2 \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {F} ^ {\otimes t}\right)
$$
where  $\{\sigma_j^i\}$  is a sequence of nt independent Rademacher variables.
The result follows by standard symmetrization proof extended to the multi-function setting.
Proof. Let  $\tilde{X}_j^1, \ldots, \tilde{X}_j^n$  be independent copies of  $X_j^1, \ldots, X_j^n$  on probability space  $P_j$  for all  $j \in [t]$ . Let  $\mathbb{E}_{\tilde{\mathbf{X}}}^1$  be the expectation with respect to  $\tilde{X}_j^1, \ldots, \tilde{X}_j^n$  for all  $j \in [t]$  and  $\mathbb{E}_{\mathbf{X}}^2$  be the expectation with respect to  $X_j^1, \ldots, X_j^n$  for all  $j \in [t]$ . Let  $\sigma_j^i$  be i.i.d. Rademacher random variables for all  $j \in [t]$
and  $i \in [n]$ . Fixing  $X_{j}^{1}, \ldots, X_{j}^{n}$  for all  $j \in [t]$ , and using the copied variables we have that
$$
\begin{array}{l} \mathbb {E} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) - \mathbb {E} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right)\right) \\ = \mathbb {E} _ {\mathbf {X}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) - \mathbb {E} _ {\tilde {X}} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(\tilde {X} _ {j} ^ {i}\right)\right). \\ \end{array}
$$
By Jensen's inequality, we have
$$
\begin{array}{l} \mathbb {E} \sup _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} (X _ {j} ^ {i}) - \mathbb {E} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} (X _ {j} ^ {i})\right) \\ \leq \mathbb {E} _ {\mathbf {X}, \tilde {\mathbf {X}}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) - \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(\tilde {X} _ {j} ^ {i}\right)\right) \\ = \frac {1}{n t} \mathbb {E} _ {\mathbf {X}, \tilde {\mathbf {X}}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \left(f _ {j} \left(\tilde {X} _ {j} ^ {i}\right) - f _ {j} \left(X _ {j} ^ {i}\right)\right)\right) \\ \end{array}
$$
If  $\tilde{X}_j^i$  and  $X_j^i$  have the same distribution, we could exchange  $f_{j}(\tilde{X}_{j}^{i}) - f_{j}(X_{j}^{i})$  for  $f_{j}(X_{j}^{i}) - f_{j}(\tilde{X}_{j}^{i})$  and the equation would be equivalent. We can do this with Rademacher random variables, by randomly flipping the sign with probability 1/2. This yields the following
$$
\begin{array}{l} \mathbb {E} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) - \mathbb {E} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right)\right) \\ = \frac {1}{n t} \mathbb {E} _ {\sigma , \mathbf {X}, \tilde {\mathbf {X}}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \left(\sigma_ {j} ^ {i} f _ {j} \left(\tilde {X} _ {j} ^ {i}\right) - \sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right)\right)\right) \\ \leq \frac {1}{n t} \mathbb {E} _ {\sigma , \tilde {\mathbf {X}}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \left(\sigma_ {j} ^ {i} f _ {j} \left(\tilde {X} _ {j} ^ {i}\right)\right)\right) \\ + \frac {1}{n t} \mathbb {E} _ {\sigma , \mathbf {X}} \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left(\sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \left(\sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right)\right)\right) \\ = 2 \mathbb {E} \tilde {\mathfrak {R}} _ {\mathbf {X}} (\mathcal {F} ^ {\otimes t}). \\ \end{array}
$$
where the last inequality follows from triangle inequality.
The next result, a Gaussian chain rule given in Tripuraneni et al. [2020], is crucial to decoupling the complexity of the representation function class and the hypothesis class.
Theorem 8 (Theorem 7 from Tripuraneni et al. [2020]). Let the function class  $\mathcal{F}$  consist of functions that satisfy Assumption 1. Then the (empirical) Gaussian width of the function class  $\mathcal{F}^{\otimes t}(\mathcal{H})$  satisfies,
$$
\tilde {\mathfrak {G}} _ {Z} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \leq \inf  _ {D \geq \alpha > 0} \left\{4 \alpha + 6 4 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \log \left(\frac {D}{\alpha}\right) \right\}.
$$
where  $G(\mathcal{F}^{\otimes t}(\mathcal{H})) = L\tilde{\mathfrak{G}}_{nt}(\mathcal{H}) + \tilde{\mathfrak{G}}_n(\mathcal{F})$ . Further, if  $G(\mathcal{F}^{\otimes t}(\mathcal{H})) \leq D$  then by computing the exact infima of the expression,
$$
\tilde {\mathfrak {G}} _ {\mathbf {X}} \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \leq 6 4 G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right) \log \left(\frac {e D}{G \left(\mathcal {F} ^ {\otimes t} (\mathcal {H})\right)}\right). \tag {18}
$$
Lemma 11 (Lemma A.4 in Bartlett et al. [2005]). Let  $\mathcal{F}$  be a class of real-valued functions with range in  $[a,b]$ . Given a set  $\mathbf{Z}$  of nt i.i.d. points, with probability at least  $1 - e^{-\delta}$ ,
$$
\mathbb {E} \tilde {\Re} _ {\mathbf {Z}} (\mathcal {F}) \leq \inf  _ {\alpha \in (0, 1)} \left(\frac {\tilde {\Re} _ {\mathbf {Z}} (\mathcal {F})}{1 - \alpha} + \frac {(b - a) \delta}{4 n \alpha (1 - \alpha)}\right).
$$
C.3 Analytic Inequalities
Lemma 12. If  $x - \sqrt{x} A - B = 0$  for  $A, B > 0$  then  $x \leq A^2 + 2B$ .
Proof. As the discriminant of the quadratic in  $\sqrt{x}$  is positive we are guaranteed two real roots  $x_{1}, x_{2}$ . Taking the larger of the two solutions we have  $\sqrt{x_{2}} = \frac{A}{2} + \frac{\sqrt{A^{2} + 4B}}{2}$ . So  $x_{2} = \frac{A^{2}}{2} + B + A\frac{\sqrt{A^{2} + 4B}}{2}$ . By AM-GM,  $x_{2} \leq \frac{A^{2}}{2} + B + \frac{A^{2}}{4} + \frac{A^{2} + 4B}{4} = A^{2} + 2B$
Lemma 13 (Lemma B.1. from Srebro et al. [2010]). For any  $H$ -smooth non-negative function  $\ell : \mathbb{R} \mapsto \mathbb{R}$  and any  $t, r \in \mathbb{R}$  we have that
$$
(\ell (t) - \ell (r)) ^ {2} \leq 6 H (\ell (t) + \ell (r)) (t - r) ^ {2}.
$$
Lemma 14. For  $k \in [t], q \in [n]$ ,
$$
\left(\left|\sum_{j,i = 1}^{t,n}a_{ji}\right| - \left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}a_{ji}\right|\right)^{2}\leq a_{kq}^{2}.
$$
Proof. First note that for any  $k \in [n]$ ,
$$
\left(\left|\sum_{i = 1}^{n}a_{i}\right| - \left|\sum_{\substack{i = 1\\ i\neq k}}^{n}a_{i}\right|\right)^{2}\leq a_{k}^{2}.
$$
This follows from applying reverse triangle inequality
$$
\left|\left|\sum_{i = 1}^{n}a_{i}\right| - \left|\sum_{\substack{i = 1\\ i\neq k}}^{n}a_{i}\right|\right|\leq \left|\sum_{i = 1}^{n}a_{i} - \sum_{\substack{i = 1\\ i\neq k}}^{n}a_{i}\right| = |a_{k}|.
$$
Finally, we square both sides and rearrange the single summation into a double summation, which completes the proof.
Lemma 15. Consider the function  $f(x) = x + a\ln (b / x)$  for  $a,b\geq 0$ .
1. The minimizer of the function over  $0 < x \leq b$  is  $x^{*} = \min(a, b)$ .
2. The corresponding minimum value is  $a \ln \left( \frac{eb}{\min(a, b)} \right)$ .
3. For any  $c$  such that  $a \leq c \leq b$ ,  $a \ln (eb / a) \leq c \ln (eb / c)$ .
Proof. The first and second claims follow from the fact that the function  $f$  decreases from  $x = 0$  to  $a$ , and then increases. The third claim follows since the function  $x \mapsto x \ln (eb / x)$  increases till  $x = b$ .
Lemma 16. Let  $Z$  be a random variable taking values in  $\mathbb{R}$  and let  $A \in \mathbb{R}$  be such that  $\mathbb{P}[Z \geq A] > 0$ , then  $\sup |Z| \geq \operatorname{ess} \sup |Z| = \| Z \|_{\infty} \geq A$ .
Proof. The first inequality is standard. The second simply follows from the definition of ess sup:
$$
\operatorname {e s s} \sup  | Z | = \inf  \left\{a \in \mathbb {R}: \mathbb {P} [ Z \geq a ] = 0 \right\}.
$$
D Proofs of main results
Figure 1: A graph demonstrating the dependency structure of our results. The top blue node is the concentration inequality given in Bousquet [2002]. We abbreviate "local Rademacher complexity" with "LRC".
In this section, we restate the main theorems in our main paper (with precise constants) and give a detailed proof of our results. A graph in Appendix D shows the dependency between our theorems.
Note the relationship between the theorems in the main paper and the theorems in the appendix are as follows:
Theorem 1 becomes Theorem 15
Theorem 2 becomes Theorem 18
Proposition 1 becomes Theorem 16
Theorem 3 becomes Theorem 12
Theorem 4 becomes Theorem 17
Theorem 5 becomes Theorem 19
Theorem 6 becomes Theorem 11
Theorem 7 becomes Theorem 9
In Appendix D.1, we present Theorems 9 to 11 which are generalizations of prior work and also recover the single function setting. In particular, we give a concentration inequality and two risk bounds. In Section D.2, we present Theorem 12 our result for multitask learning with bounded non-negative loss classes. Next, Appendix D.3 contains Theorems 13 to 15 the MTL via MTRL learning setting.
The remaining theorems add the additional assumption of smoothness. We start, in section Appendix D.4, by giving Theorem 16, our bound on the empirical local Rademacher complexity of smooth bounded non-negative loss classes. However, in order to use the above result with our prior theorems we need to bound the local Rademacher complexity constrained on the risk to one
constrained by the empirical risk. So, in Appendix D.5, we introduce Lemmas 19 and 20 to make this conversion. In Appendix D.6 we conclude by invoking our bound on the local Rademacher complexity of smooth bounded non-negative loss classes within our MTL and MTRL theorems with Theorems 17 and 18.
D.1 Concentration Inequalities and Risk Bounds
In this section we forgo the compositional model and will consider a generic function class  $\mathcal{F}$ .
Theorem 9 (Concentration with  $t$  functions). Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$  and assume that all functions  $f$  in  $\mathcal{F}$  are  $P_{j}$ -measurable for all  $j\in [t]$ , square-integrable, and satisfy  $\mathbb{E}f = 0$ . Let  $\sup_{f\in \mathcal{F}}\| f\|_{\infty}\leq 1$  and
$$
Z = \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \left| \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right) \right| \tag {19}
$$
or let  $\sup_{f\in \mathcal{F}}\operatorname {ess}\sup f\leq 1$  and
$$
Z = \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} \left(X _ {j} ^ {i}\right). \tag {20}
$$
Let  $\sigma > 0$  such that  $\sigma^2 \geq \sup_{\boldsymbol{f} \in \mathcal{F}^{\otimes t}} \frac{1}{t} \sum_{j=1}^{t} \operatorname{Var}\left[f_j(X_j^1)\right]$  almost surely, then for all  $\delta \geq 0$ , we have
Proof of Theorem 9. We prove the result for  $Z$  as defined in Equation (19); the proof for the definition of  $X$  in Equation (20) is similar. Let
$$
Z = \sup_{\boldsymbol {f}\in \mathcal{F}^{\otimes t}}\left|\sum_{j,i = 1}^{t,n}f_{j}(X_{j}^{i})\right|.
$$
and let  $f_1^{(0)}, \ldots, f_t^{(0)}$  be the functions that achieve the supremum w.r.t.  $Z$ .
$$
Z = \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right|.
$$
Next, for each  $k \in [t]$  and  $q \in [n]$  let
$$
Z_{k}^{q} = \sup_{\boldsymbol {f}\in \mathcal{F}^{\otimes t}}\left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}(X_{j}^{i})\right| = \sup_{\boldsymbol {f}\in \mathcal{F}^{\otimes t}}\left| - f_{k}(X_{k}^{q}) + \sum_{j,i = 1}^{t,n}f_{j}(X_{j}^{i})\right|.
$$
The random variable  $Z_{k}^{q}$  is essentially removing a  $f_{k}(X_{k}^{q})$  term from  $Z$ . We define  $f_{1}^{(k,q)},\ldots ,f_{t}^{(k,q)}$  as the functions which achieve the supremum w.r.t  $Z_{k}^{q}$ . That is,
$$
Z_{k}^{q} = \left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(k,q)}(X_{j}^{i})\right|.
$$
Finally define,
$$
\tilde {Z} _ {k} ^ {q} = \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(k, q)} \left(X _ {j} ^ {i}\right) \right| - Z _ {k} ^ {q}.
$$