SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \tilde {Z} _ {k} ^ {q} = \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(k, q)} \left(X _ {j} ^ {i}\right) \right| - Z _ {k} ^ {q} \\ \leq \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| - \left| \sum_ {\substack {j, i = 1 \\ (j, i) \neq (k, q)}} ^ {t, n} f _ {j} ^ {(k, q)} \left(X _ {j} ^ {i}\right) \right| (21) \\ \leq \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| - \left| \sum_ {\substack {j, i = 1 \\ (j, i) \neq (k, q)}} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| (22) \\ \leq \left| \sum_ {\substack {j, i = 1 \\ (j, i) \neq (k, q)}} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| + \left| f _ {k} ^ {(0)} \left(X _ {k} ^ {q}\right) \right| - \left| \sum_ {\substack {j, i = 1 \\ (j, i) \neq (k, q)}} ^ {t, n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| \quad (\text{by triangle inequality}) \\ = \left| f _ {k} ^ {(0)} \left(X _ {k} ^ {q}\right) \right| \\ \leq 1 \quad \mathrm {a . s} \quad (\text {b y}) \\ \end{array}
$$
where Inequality 21 follows from  $f_1^{(0)}, \ldots, f_t^{(0)}$  being the functions which achieve the supremum w.r.t  $Z$  and Inequality 22 follows from  $f_1^{(k,q)}, \ldots, f_t^{(k,q)}$  achieving the supremum w.r.t  $Z_k^q$ . In addition, we show that  $\mathbb{E}_k^q\left[\tilde{Z}_k^q\right]$  is non-negative. Recall that  $\mathbb{E}_k^q$  is conditioned on  $\mathcal{A}_k^q$ .
$$
\begin{array}{l} \mathbb {E} _ {k} ^ {q} \left[ \tilde {Z} _ {k} ^ {q} \right] = \mathbb {E} _ {k} ^ {q} \left[ \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(k, q)} (X _ {j} ^ {i}) \right| - Z _ {k} ^ {q} \right] \\ = \mathbb {E} _ {k} ^ {q} \left[ \left| \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(k, q)} \left(X _ {j} ^ {i}\right) \right| \right] - Z _ {k} ^ {q} \quad (\text {s i n c e} Z _ {k} ^ {q} \text {i s a c o n s t a n t w . r . t} \mathbb {E} _ {k} ^ {q}) \\ \geq \left| \mathbb {E} _ {k} ^ {q} \left[ \sum_ {j, i = 1} ^ {t, n} f _ {j} ^ {(k, q)} \left(X _ {j} ^ {i}\right) \right] \right| - Z _ {k} ^ {q} \quad \text {(u s i n g J e n s e n ' s i n e q u a l i t y)}. \\ \end{array}
$$
Finally, each  $f_{j}^{(k,q)}$  is a constant w.r.t  $\mathbb{E}_k^q$  besides when  $j = k$  and  $i = q$  so the above reduces to:
$$
\begin{array}{l} \mathbb{E}_{k}^{q}\left[\tilde{Z}_{k}^{q}\right]\geq \left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(k,q)}(X_{j}^{i})\right| - Z_{k}^{q} \\ = 0. \\ \end{array}
$$
We now prove that the conditions in Corollary 1 are satisfied.
$$
\begin{array}{l} \left(n t - 1\right) Z = \left| \left(n t - 1\right) \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f _ {j} ^ {(0)} \left(X _ {j} ^ {i}\right) \right| \\ = \left|\sum_{k,q = 1}^{t,n}\left[\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(0)}(X_{j}^{i})\right]\right| \\ \leq \sum_{k,q = 1}^{t,n}\left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(0)}(X_{j}^{i})\right| \\ \leq \sum_{k,q = 1}^{t,n}\left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(k,q)}(X_{j}^{i})\right| \\ = \sum_ {k, q = 1} ^ {t, n} Z _ {k} ^ {q}. \\ \end{array}
$$
Therefore,  $\sum_{j=1}^{t} \sum_{i=1}^{n} (Z - Z_k^q) \leq Z$ .
$$
\begin{array}{l} \sum_{k,q = 1}^{t,n}\mathbb{E}_{k}^{q}\left[\left(\tilde{Z}_{k}^{q}\right)^{2}\right] = \sum_{k,q = 1}^{t,n}\mathbb{E}_{k}^{q}\left[\left(\left|\sum_{j,i = 1}^{t,n}f_{j}^{(k,q)}(X_{j}^{i})\right| - \left|\sum_{\substack{j,i = 1\\ (j,i)\neq (k,q)}}^{t,n}f_{j}^{(k,q)}(X_{j}^{i})\right|\right)^{2}\right] \\ \leq \sum_ {k, q = 1} ^ {t, n} \mathbb {E} _ {k} ^ {q} \left[ \left(f _ {k} ^ {(k, q)} \left(X _ {k} ^ {q}\right)\right) ^ {2} \right] \tag {byCorollary14} \\ \end{array}
$$
As the samples for the same task are being drawn from the same distribution,
$$
\begin{array}{l} \sum_ {k, q = 1} ^ {t, n} \mathbb {E} _ {k} ^ {q} \left[ \left(\tilde {Z} _ {k} ^ {q}\right) ^ {2} \right] = \sum_ {k, q = 1} ^ {t, n} \mathbb {E} _ {k} ^ {1} \left[ \left(f _ {k} ^ {(k, 1)} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right] \\ = n \sum_ {k = 1} ^ {t} \mathbb {E} _ {k} ^ {1} \left[ \left(f _ {k} ^ {(k, 1)} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right]. \\ \end{array}
$$
Note that  $f_{k}^{(k,1)}$  is a fixed function w.r.t.  $X_{k}^{1}$ , therefore  $f_{k}^{(k,1)}$  is a fixed function as we integrate over  $X_{k}^{1}$ . So,  $\mathbb{E}_k^1\left[\left(f_k^{(k,1)}(X_k^1)\right)^2\right] \leq \sup_{f_j \in \mathcal{F}} \mathbb{E}_k^1\left[\left(f_j(X_k^1)\right)^2\right]$ . Applying this reasoning to the  $t$  functions gives the desired bound.
$$
\begin{array}{l} n \sum_ {k = 1} ^ {t} \mathbb {E} _ {k} ^ {1} \left[ \left(f _ {k} ^ {(k, 1)} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right] \leq n \sum_ {k = 1} ^ {t} \sup  _ {f _ {k} \in \mathcal {F}} \mathbb {E} _ {k} ^ {1} \left[ \left(f _ {k} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right] \\ = n \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \sum_ {k = 1} ^ {t} \mathbb {E} _ {k} ^ {1} \left[ \left(f _ {k} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right]. \\ \end{array}
$$
Note, crucially, that the product space structure of our function class  $\mathcal{F}^{\otimes t}$  allows us to move the suprema outside the summation. Dividing by  $nt$  gives the desired bound.
$$
\frac {1}{n t} \sum_ {k, q = 1} ^ {t, n} \mathbb {E} _ {k} ^ {q} \left[ \left(\tilde {Z} _ {k} ^ {q}\right) ^ {2} \right] \leq \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \frac {1}{t} \sum_ {k = 1} ^ {t} \mathbb {E} _ {k} ^ {q} \left[ \left(f _ {k} \left(X _ {k} ^ {1}\right)\right) ^ {2} \right] = \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \frac {1}{t} \sum_ {k = 1} ^ {t} \operatorname {V a r} \left[ f _ {k} \left(X _ {k} ^ {1}\right) \right].
$$
The right-hand side is bounded by  $\sigma^2$  by hypothesis.
Finally, now we apply Corollary 1 as all the conditions are satisfied.
Theorem 10 (Analog of Theorem 2.1 in Bartlett et al. [2005]). Let  $\mathcal{F}$  be a class of functions that map  $\mathcal{X}$  into  $[a,b]$ . Given an input  $\mathbf{X} = \left\{X_j^i\right\}_{i,j=1}^{t,n}$ , assume that there is some  $r > 0$  such that for all  $f_1, \ldots, f_t$  that  $\frac{1}{t}\sum_{j=1}^{t}\operatorname{Var}\left[f_j(X_j^1)\right] \leq r$ . Then, for every  $\delta > 0$ , with probability  $1 - e^{-\delta}$ ,
$$
\sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} (P \boldsymbol {f} - \hat {P} ^ {n} \boldsymbol {f}) \leq \inf  _ {\alpha} \left\{2 (1 + \alpha) \mathbb {E} \Re_ {\mathbf {X}} (\mathcal {F} ^ {\otimes t}) + \sqrt {\frac {2 \delta r}{t n}} + (b - a) \left(\frac {1}{\alpha} + \frac {1}{3}\right) \frac {\delta}{n t} \right\}.
$$
Proof of Theorem 10. Let  $V^{+} = \sup_{\pmb{f}\in \mathcal{F}^{\otimes t}}(P\pmb{f} - \hat{P}^{n}\pmb{f})$  where  $\hat{P}^n\pmb {f} = \frac{1}{nt}\sum_{j = 1}^t\sum_{i = 1}^n f_j(X_j^i)$  and  $P\pmb {f} = \mathbb{E}\hat{P}^{n}\pmb {f}$ . Let  $\tilde{\mathcal{F}} = \left\{\frac{f - \mathbb{E}f}{(b - a)}\mid f\in \mathcal{F}\right\}$  be a set of scaled functions. For each  $\tilde{f}\in \tilde{\mathcal{F}}$  we have that  $\mathbb{E}\tilde{f} = 0$  and  $|\tilde{f} |\leq 1$ . In addition, for all  $\tilde{\pmb{f}}\in \tilde{\mathcal{F}}^{\otimes t}$  we have that  $\frac{1}{t}\sum_{j = 1}^{t}\mathrm{Var}\bigl [\tilde{f}_j(X_j^1)\bigr ] = \frac{1}{(b - a)^2}\frac{1}{t}\sum_{j = 1}^{t}\mathrm{Var}\bigl [f_j(X_j^1)\bigr ]\leq \frac{r}{(b - a)^2}$ . Therefore, the conditions for Theorem 9 are satisfied.
Letting  $Z = \sup_{\tilde{f} \in \tilde{\mathcal{F}}^{\otimes t}} \sum_{j=1}^{t} \sum_{i=1}^{n} \tilde{f}_j(X_j^i) = \frac{nt}{(b-a)} V^+$  as in Equation (20), we thus have with probability  $1 - e^{-\delta}$ :
$$
\begin{array}{l} V ^ {+} \leq \mathbb {E} V ^ {+} + \sqrt {\frac {2 \delta r}{t n} + \frac {4 \delta \mathbb {E} V ^ {+} (b - a)}{n t}} + \frac {(b - a) \delta}{n t 3} \\ \leq \mathbb {E} V ^ {+} + \sqrt {\frac {2 \delta r}{t n}} + \sqrt {\frac {4 \delta \mathbb {E} V ^ {+} (b - a)}{n t}} + \frac {(b - a) \delta}{n t 3} (23) \\ \leq \mathbb {E} V ^ {+} + \sqrt {\frac {2 \delta r}{t n}} + \alpha \mathbb {E} V ^ {+} + \frac {\delta (b - a)}{\alpha n t} + \frac {(b - a) \delta}{n t 3} (24) \\ \leq 2 (1 + \alpha) \mathbb {E} \tilde {\Re} _ {\mathbf {X}} (\mathcal {F} ^ {\otimes t}) + \sqrt {\frac {2 \delta r}{t n}} + (b - a) \left(\frac {1}{\alpha} + \frac {1}{3}\right) \frac {\delta}{n t} (25) \\ \end{array}
$$
where Inequality 23 is by subadditivity of the square root, Inequality 24 is by application of the AM-GM inequality, and Inequality 25 is by Lemma 10.
Theorem 11. Let  $\mathcal{F}$  be a class non-negative  $b$  bounded functions. Assume that  $X_{j}^{1},\ldots ,X_{j}^{n}$  are  $n$  draws from  $P_{j}$  for  $j\in [t]$  and all nt samples are independent. Suppose that
$$
\operatorname {V a r} [ \boldsymbol {f} ] \leq V (\boldsymbol {f}) \leq B P \boldsymbol {f}
$$
where  $V$  is a functional from  $\mathcal{F}^{\otimes t}$  to  $\mathbb{R}$  and  $B > 0$ . Given an input  $\mathbf{X} = \{X_{j}^{i}\}_{i,j = 1}^{t,n}$ , let  $\psi$  be a sub-root function with fixed point  $r^*$  such that  $B\mathbb{E}\tilde{\mathfrak{R}}_{\mathbf{X}}(\mathcal{F}^{\otimes t},r)\leq \psi (r)$  for all  $r\geq r^*$ . Then, the following holds: For  $K > 1$ ,  $\alpha >0$ , and  $x > 1$ , with probability at least  $1 - e^{-x}$ .
$$
\forall \boldsymbol {f} \in \mathcal {F} ^ {\otimes t} \quad P \boldsymbol {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \boldsymbol {f} + 2 0 0 (1 + \alpha) ^ {2} \frac {K r ^ {*}}{B} + \frac {5}{4} \frac {B K x}{n t} + 2 \left(\frac {1}{\alpha} + \frac {1}{3}\right) \frac {b x}{n t}.
$$
Remark 2 (Comparison to Yousefi et al. [2018]). A Theorem of this form was shown by Yousefi et al. [2018], which generalized results from Bartlett et al. [2005] to vector valued functions. The proof of Theorem 11 follows the same steps as the proof of Theorem Yousefi et al. [2018]. Yet, we will use our Theorem 10 which has better constants than Theorem 1 within Yousefi et al. [2018] for when the class that the suprema is indexing over is has product space structure.
The following lemma, and its proof, is a modified version of work in [Yousefi et al., 2018, Lemma B.2], which is a generalized version of work within [Bartlett et al., 2005, Lemma 3.8.]. We remove the Bernstein class structure, make  $B > 0$ , and only consider non-negative bounded functions<sup>5</sup>.
5 The reason we restrict to non-negative functions instead of functions that map to  $[a,b]$  as done in prior work is because we failed to reproduce a step within Bartlett et al. [2005] and Yousefi et al. [2018]. Consider the last sentence within the first paragraph of the proof of Lemma 3.8. in Bartlett et al. [2005]. It is shown that  $Pf\leq \hat{P}^n f + r / (\lambda BK)$ . Now  $\hat{P}^n f$  may be negative for  $f$  which maps to  $[a,b]$  with  $a < 0$ . Therefore, we cannot multiply this quantity by a value greater than one. In this case  $K / (K - 1)$  for  $K > 1$ .
The theorem can be generalized to Bernstien classes like Theorem B.3 from Yousefi et al. [2018] with worse dependence on  $B$ .
Lemma 17. Let  $K > 1, r > 0$  and  $B > 0$ . Suppose that for all  $\pmb{f} \in \mathcal{F}^{\otimes t}$  we have  $V(\pmb{f}) \leq BP\pmb{f}$ . Define the re-scaled version of  $\mathcal{F}^{\otimes t}$  as
$$
\mathcal {F} _ {r} ^ {\otimes t} = \left\{\boldsymbol {f} ^ {\prime} = \left(\boldsymbol {f} _ {1} ^ {\prime}, \dots , \boldsymbol {f} _ {t} ^ {\prime}\right) \mid f _ {j} ^ {\prime} = \frac {r f _ {j}}{\max  (r , V (\boldsymbol {f}))}, \boldsymbol {f} = \left(\boldsymbol {f} _ {1}, \dots , \boldsymbol {f} _ {t}\right) \in \mathcal {F} ^ {\otimes t} \right\}. \tag {26}
$$
If  $V_{r}^{+} = \sup_{\pmb{f}^{\prime}\in \mathcal{F}_{r}^{\otimes t}}\left[P\pmb{f}^{\prime} - \hat{P}^{n}\pmb{f}^{\prime}\right]\leq \frac{r}{BK}$ , then
$$
\forall \boldsymbol {f} \in \mathcal {F} ^ {\otimes t}, \quad P \boldsymbol {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \boldsymbol {f} + \frac {r}{B K}.
$$
Proof of Lemma 17. Let  $\pmb{f}$  be any element in  $\mathcal{F}^{\otimes t}$ .
Case 1. If  $V(\pmb{f}) \leq r$ , then  $\pmb{f}' = \pmb{f}$  and the inequality  $V_r^+ \leq \frac{r}{BK}$  leads to
$$
P (\boldsymbol {f}) \leq \hat {P} ^ {n} (\boldsymbol {f}) + \frac {r}{B K}.
$$
If all component functions of  $\pmb{f}$  are positive then  $\hat{P}^n (\pmb {f})$  is positive and as  $K / (K - 1) > 1$  for all  $K > 1$  we have
$$
P (\boldsymbol {f}) \leq \frac {K}{K - 1} \hat {P} ^ {n} (\boldsymbol {f}) + \frac {r}{B K}.
$$
Case 2. If  $V(\pmb{f}) \geq r$ , then  $\pmb{f}' = r\pmb{f} / V(\pmb{f})$  and the inequality  $V_r^+ \leq \frac{r}{BK}$  yields
$$
P (\boldsymbol {f}) \leq \hat {P} ^ {n} (\boldsymbol {f}) + \frac {V (\boldsymbol {f})}{B K} \leq \hat {P} ^ {n} (\boldsymbol {f}) + \frac {P (\boldsymbol {f})}{K}.
$$
Solving for  $P(\pmb{f})$  gives
$$
P (\boldsymbol {f}) \leq \frac {K}{K - 1} \hat {P} ^ {n} (\boldsymbol {f}) + \leq \frac {K}{K - 1} \hat {P} ^ {n} (\boldsymbol {f}) + \frac {r}{B K}.
$$
Proof of Theorem 11. We modify the proof within Yousefi et al. [2018], which generalises the proof within Bartlett et al. [2005] by extending it to multi-function and Bernstein classes. At a high level we apply the same steps as Yousefi et al. [2018], yet we use our concentration inequality and do not need to consider the more complicated inequality [Yousefi et al., 2018, Lemma B.1] as we do not have the additional Bernstein structure. The proof closely follows from Yousefi et al. [2018] with minor modifications as described and presented for completeness.
Let  $r \geq r^*$  be a fixed real number.
Verifying Conditions of Theorem 10. We seek to use Theorem 10 on  $\mathcal{F}_r^{\otimes t}$ , see Equation (26), therefore we will validate the conditions of the theorem.
Fix  $\pmb{f}^{\prime} \in \mathcal{F}_{r}^{\otimes t}$ . Thus,  $\pmb{f}^{\prime} = r\pmb{f} / \max(r, V(\pmb{f}))$  for some  $\pmb{f} \in \mathcal{F}$ . If  $\max(r, V(\pmb{f})) = r$ , then  $\pmb{f}^{\prime} = r\pmb{f} / \max(r, V(\pmb{f})) = \pmb{f}$ . Thus,  $\operatorname{Var}\left[\pmb{f}^{\prime}\right] = \operatorname{Var}\left[\pmb{f}\right] \leq V(\pmb{f}) \leq r$ . If  $\max(r, V(\pmb{f})) = V(\pmb{f})$ , then  $\operatorname{Var}\left[\pmb{f}^{\prime}\right] = \frac{r^{2}}{V^{2}(\pmb{f})}\operatorname{Var}\left[\pmb{f}\right] \leq \frac{r^{2}}{V(\pmb{f})} \leq r$ . Therefore,  $\frac{1}{t}\sup_{\pmb{f}' \in \mathcal{F}_{r}^{\otimes t}}\sum_{j=1}^{t}\mathbb{E}\left[f_{j}'(X_{j})\right]^{2} \leq r$ . Finally, recall, functions in  $\mathcal{F}$  are positive and  $b$ -bounded.
Thus, by Theorem 10, with probability at least  $1 - e^{-\delta}$  and  $\forall \delta > 0$
$$
\sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} (P \boldsymbol {f} - \hat {P} ^ {n} \boldsymbol {f}) \leq \inf  _ {\alpha} \left\{\leq 2 (1 + \alpha) \mathbb {E} \tilde {\Re} _ {\mathbf {X}} (\mathcal {F} _ {r} ^ {\otimes t}) + \sqrt {\frac {2 \delta r}{t n}} + b \left(\frac {1}{\alpha} + \frac {1}{3}\right) \frac {\delta}{n t} \right\}.
$$
Bounding  $\Re_{\mathbf{X}}(\mathcal{F}_r^{\otimes t})$ . Now we must control the Rademacher complexity within the above inequality. This part of the proof is the same for Bartlett et al. [2005], Yousefi et al. [2018].
Let  $\mathcal{F}^{\otimes t}(u,v)\coloneqq \{\pmb {f}\in \mathcal{F}^{\otimes t}:u\leq V(\pmb {f})\leq v\}$  for all  $0\leq u\leq v$  , and let
$$
\mathcal {R} _ {\mathbf {X}} \left(\boldsymbol {f} ^ {\prime}\right) = \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \sigma_ {j} ^ {i} f _ {j} ^ {\prime} \left(X _ {j} ^ {i}\right),
$$
$$
\mathcal {R} _ {\mathbf {X}} \left(\mathcal {F} _ {r} ^ {\otimes t}\right) = \sup  _ {f ^ {\prime} \in \mathcal {F} _ {r} ^ {\otimes t}} \left[ \mathcal {R} _ {\mathbf {X}} \left(\boldsymbol {f} ^ {\prime}\right) \right],
$$
Observe that  $\tilde{\mathfrak{R}}_{\mathbf{X}}(\mathcal{F}_r^{\otimes t}) = \mathbb{E}\mathcal{R}_{\mathbf{X}}(\mathcal{F}_r^{\otimes t})$ . By assumption we have that  $V(\pmb {f})\leq B(P\pmb {f})\leq Bb,\forall \pmb {f}\in$ $\mathcal{F}^{\otimes t}$ . Fix some  $\lambda >1$  and define  $k$  as the smallest integer such that  $r\lambda^{k + 1}\geq Bb$ . By leveraging the principle that combining function classes on a uniform set of inputs is equivalent to the Minkowski sum of the created image, and utilizing the trait of Rademacher width breakdown under Minkowski sum as detailed by Vershynin [2018], we have,
$$
\tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {G} _ {1} \cup \mathcal {G} _ {2}\right) \leq \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {G} _ {1}\right) + \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {G} _ {2}\right).
$$
We thus the following inequalities
$$
\begin{array}{l} \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {F} _ {r} ^ {\otimes t}\right) = \mathbb {E} \left[ \sup  _ {f ^ {\prime} \in \mathcal {F} _ {r} ^ {\otimes t}} \mathcal {R} _ {\mathbf {X}} \left(\boldsymbol {f} ^ {\prime}\right) \right] = \mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \frac {r}{\max  (r , V (\boldsymbol {f}))} \sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right) \right] \\ \leq \mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} (0, r)} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right) \right] + \mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} (r, B b)} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \frac {r}{V (\boldsymbol {f})} \sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right) \right] \\ \leq \mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} (0, r)} \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \sigma_ {j} ^ {i} f _ {j} \left(X _ {j} ^ {i}\right) \right] + \sum_ {j = 0} ^ {k} \lambda^ {- j} \mathbb {E} \left[ \sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t} (r \lambda^ {j}, r \lambda^ {j + 1})} \mathcal {R} _ {\mathbf {X}} (\boldsymbol {f}) \right] \\ \leq \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {F} ^ {\otimes t}, r\right) + \sum_ {j = 0} ^ {k} \lambda^ {- j} \tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {F} ^ {\otimes t}, r \lambda^ {j + 1}\right) \\ \end{array}
$$
As  $r \geq r^*$ , we can bound the local Rademacher width with the sub-root function.
$$
\leq \frac {\psi (r)}{B} + \frac {1}{B} \sum_ {j = 0} ^ {k} \lambda^ {- j} \psi \left(r \lambda^ {j + 1}\right).
$$
Now the sub-root property of  $\psi$  gives us that that  $\psi (\xi r)\leq \xi^{\frac{1}{2}}\psi (r)$  for any  $\xi \geq 1$  and, hence,
$$
\tilde {\mathfrak {R}} _ {\mathbf {X}} \left(\mathcal {F} _ {r} ^ {\otimes t}\right) \leq \frac {\psi (r)}{B} \left(1 + \sqrt {\lambda} \sum_ {j = 0} ^ {k} \lambda^ {- \frac {j}{2}}\right) \leq \frac {\psi (r)}{B} \left(1 + \frac {\lambda}{\sqrt {\lambda} - 1}\right).
$$
Pick  $\lambda = 4$  in the above inequality, which gives  $\tilde{\mathfrak{R}}_{\mathbf{X}}(\mathcal{F}_r^{\otimes t})\leq 5\psi (r) / B$ . Also,  $r^*$  is the fixed point of  $\psi$ , we have for all  $r\geq r^{*}$  that  $\psi (r)\leq \sqrt{r / r^{*}}\psi (r^{*}) = \sqrt{rr^{*}}$ . Taken together, we have
$$
\tilde {\mathfrak {N}} _ {\mathbf {X}} \left(\mathcal {F} _ {r} ^ {\otimes t}\right) \leq \frac {5}{B} \sqrt {r r ^ {*}}, \quad \forall r \geq r ^ {*}.
$$
Applying Lemma 17. The remainder of the proof is simple manipulations and applying Lemma 17. Here our proof deviates from [Yousefi et al., 2018]. Using our above bound on the Rademacher complexity gives for any  $r \geq r^*$  and  $\delta > 0$ , we have with probability at least  $1 - e^{-\delta}$ ,
$$
\sup  _ {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}} (P \boldsymbol {f} - \hat {P} ^ {n} \boldsymbol {f}) \inf  _ {\alpha} \left\{(1 + \alpha) \frac {1 0}{B} \sqrt {r r ^ {*}} + \sqrt {\frac {2 \delta r}{t n}} + b \left(\frac {1}{\alpha} + \frac {1}{3}\right) \frac {\delta}{n t} \right\}
$$
Now, letting  $A = 10(1 + \alpha)\sqrt{r^*} / B + \sqrt{2\delta / nt}$  and  $C = b\left(\frac{1}{\alpha} + \frac{1}{3}\right)\frac{\delta}{nt}$ ,
$$
\sup  _ {f ^ {\prime} \in \mathcal {F} _ {r} ^ {\otimes t}} \left[ P \boldsymbol {f} ^ {\prime} - \hat {P} ^ {n} \boldsymbol {f} ^ {\prime} \right] \leq A \sqrt {r} + C.
$$
Setting  $A\sqrt{r} + C = \frac{r}{BK}$  gives a quadratic, which has both roots bounded by  $(ABK)^2 + 2BKC$  by Lemma 12 now applying Lemma 17 we have that
$$
P \boldsymbol {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \boldsymbol {f} + B K A ^ {2} + 2 C.
$$
Plugging in our values for  $A$  and  $C$  gives
$$
P \boldsymbol {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \boldsymbol {f} + 1 0 0 (1 + \alpha) ^ {2} \frac {K r ^ {*}}{B} + 2 \frac {B K \delta}{n t} + 2 0 \sqrt {2} (1 + \alpha) K \sqrt {r ^ {*}} \sqrt {\frac {\delta}{n t}} + 2 \left(\frac {1}{3} + \frac {1}{\alpha}\right) \frac {b \delta}{n t}.
$$
Applying AM-GM inequality,
$$
P \boldsymbol {f} \leq \frac {K}{K - 1} \hat {P} ^ {n} \boldsymbol {f} + 1 0 1 (1 + \alpha) ^ {2} \frac {K r ^ {*}}{B} + 8 0 2 \frac {B K \delta}{n t} + 2 \left(\frac {1}{3} + \frac {1}{\alpha}\right) \frac {b \delta}{n t},
$$
D.2 Multi-Task Learning
The following theorem is a vector-valued generalization of Theorem 5.2 within Bartlett et al. [2005] with several differences: we do not require  $\frac{\delta}{n} \leq r^*$ ,  $b = 1$ , and we solve for constants.
Theorem 12. Let  $(\hat{\pmb{f}},\hat{\pmb{h}})$  be an empirical risk minimizer as given in Eqn. (2). Let  $\psi (r)\geq$ $b\mathbb{E}\tilde{\mathfrak{R}}_{\mathbf{X}}(\mathcal{F}^{\otimes t}(\mathcal{H}),r)$  with  $r^*$  the fixed point of  $\psi (r)$ . Then, under Assumption 1.A with probability  $1 - 2e^{-\delta}$ ,
$$
R _ {\text {s o u r c e}} (\hat {\boldsymbol {f}}, \hat {h}) \leq R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*}, h ^ {*}\right) + \sqrt {R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {*} , h ^ {*}\right)} \left(6 \sqrt {\frac {b \delta}{n t}} + 1 4 6 \sqrt {\frac {r ^ {*}}{b}}\right) + \frac {1 0 2 b \delta}{n t} + \frac {2 1 7 r ^ {*}}{b}. \tag {27}
$$
Proof of Theorem 12. To simplify the notation within the proof let  $\pmb{L}^{*} = R_{\mathrm{source}}(\pmb{f}^{*}, h^{*})$ .
We follow steps similar to the proof of Theorem 5.2 within Bartlett et al. [2005]. Assume  $f_{j}^{*} = \arg \min_{f\in \mathcal{F}}P_{j}\ell_{f}$  exists. (As mentioned in Bartlett et al. [2005] if it does not exist we can consider a sequence converging to the infimum.) Note that  $\{f_j^* (X_j^i)\}_{i\in [n],j\in [t]}$  are  $nt$  independent r.v.. Note that
$$
\frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {j = 1} ^ {n} \operatorname {V a r} \left[ f _ {j} ^ {*} (X _ {j} ^ {i}) \right] \leq \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {j = 1} ^ {n} \mathbb {E} \left(f _ {j} ^ {*} (X _ {j} ^ {i})\right) ^ {2} \leq \frac {b}{n t} \sum_ {j = 1} ^ {t} \sum_ {j = 1} ^ {n} \mathbb {E} f _ {j} ^ {*} (X _ {j} ^ {i}) = b L ^ {*}.
$$
By Bernstein inequality, with probability  $1 - e^{-\delta}$
$$
\hat {P} ^ {n} \hat {\boldsymbol {f}} \leq \hat {P} ^ {n} \boldsymbol {f} ^ {*} \leq \boldsymbol {L} ^ {*} + \sqrt {\frac {2 \boldsymbol {L} ^ {*} b \delta}{t n}} + \frac {2}{3} \frac {b \delta}{t n}.
$$
By Theorem 11, setting  $\alpha = \frac{1}{11}$  and  $B = b$ , and union bound, with probability  $1 - 2e^{-\delta}$ ,
$$
P \ell_ {\hat {f}} \leq \frac {K}{K - 1} \left(\sqrt {2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + \boldsymbol {L} ^ {*} + \frac {2 b \delta}{3 n t}\right) + \frac {1 7 8 9 b K \delta}{8 8 2 n t} + \frac {6 8 b \delta}{3 n t} + \frac {1 4 4 K r ^ {*}}{b}.
$$
By change of variables we consider  $K > 0$  by adding 1 to  $K$ :
$$
\begin{array}{l} \frac {1 7 8 9 b \delta (K + 1)}{8 8 2 n t} + \frac {6 8 b \delta}{3 n t} + \frac {1 4 4 r ^ {*} (K + 1)}{b} + \frac {(K + 1) \left(\sqrt {2} \sqrt {\frac {L ^ {*} b \delta}{n t}} + L ^ {*} + \frac {2 b \delta}{3 n t}\right)}{K} \\ = \frac {1 7 8 9 K b \delta}{8 8 2 n t} + \frac {1 4 4 K r ^ {*}}{b} + \sqrt {2} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + \boldsymbol {L} ^ {*} + \frac {2 2 3 6 9 b \delta}{8 8 2 n t} + \frac {1 4 4 r ^ {*}}{b} + \frac {\sqrt {2}}{K} \sqrt {\frac {\boldsymbol {L} ^ {*} b \delta}{n t}} + \frac {\boldsymbol {L} ^ {*}}{K} + \frac {2 b \delta}{3 K n t}. \\ \end{array}
$$
Applying AM-GM to  $\frac{\sqrt{2}}{K}\sqrt{\frac{L^{*}b\delta}{nt}}$