SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof of Theorem 6 is similar to the one given in Bartlett et al. [2005], Sec. 3, and generalized to MTL by Yousefi et al. [2018], Appendix B. Indeed, we follow the same steps within these proofs but use our inequality above.
We provide the first optimistic rates for transfer learning and multi-task learning via multi-task representation learning. This is achieved by establishing a local Rademacher complexity result for multi-task representation learning. We provide distribution-free optimistic rates for smooth nonnegative losses by bounding the local Rademacher complexity for multi-task loss classes. Besides our contributions to multi-task representation learning and multi-task learning, our work provides several foundational results and improved rates in the standard single-task setting. Directions for future work include: exploring adversarial robustness, active learning, and fine-tuning of the representation within multi-task representation learning.
Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Differentially private generalized linear models revisited. In Advances in Neural Information Processing Systems, volume 35, pages 22505-22517, 2022.
Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Annals of Statistics, 33:1497-1537, 2005.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Witold Bednorz and Rafal Latała. On the boundedness of bernoulli processes. Annals of Mathematics, 180(3):1167-1203, 2014.
Shai Ben-David and Reba Schuller Borbely. A notion of task relatedness yielding provable multiple-task learning guarantees. Machine learning, 73:273-287, 2008.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Gilles Blanchard, Olivier Bousquet, and Laurent Zwald. Statistical properties of kernel principal component analysis. Machine Learning, 66:259-294, 2007.
Olivier Bousquet. A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Mathematique, 334(6):495-500, 2002.
Olivier Bousquet. Concentration inequalities for sub-additive functions using the entropy method. In Stochastic inequalities and applications, pages 213-247. Springer, 2003.
Olivier Bousquet, Vladimir Koltchinskii, and Dmitriy Panchenko. Some local measures of complexity of convex hulls and generalization bounds. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002, pages 59-73. Springer, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research, 11:2901-2934, 2010.
Corinna Cortes, Marius Kloft, and Mehryar Mohri. Learning kernels using local rademacher complexity. Advances in neural information processing systems, 26, 2013.
Corinna Cortes, Mehryar Mohri, and Ananda Theertha Suresh. Relative deviation margin bounds. In International Conference on Machine Learning, pages 2122-2131. PMLR, 2021.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 13089-13099, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171-4186, 2019.
Lijia Zhou, Frederic Koehler, Danica J. Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. CoRR, abs/2112.04470, 2021.
1.1 Our techniques 3
1.2 Related work 4
2 Problem setup and preliminaries 4
3 Main Results 6
3.1 Multi-Task Learning 8
3.2 Local Rademacher complexity chain rule 9
4 Proof Techniques 9
5 Discussion 10
A Additional preliminaries on Gaussian processes and complexities 16
A.1 Preliminaries 16
A.2 Sub-root functions and local Rademacher Complexity 19
B Task diversity digression 19
C Technical Miscellanea 21
C.1 Martingale Concentration Inequality 21
C.2 Empirical Process Lemmas 22
C.3 Analytic Inequalities 24
D Proofs of main results 25
D.1 Concentration Inequalities and Risk Bounds 26
D.2 Multi-Task Learning 32
D.3 Multi-Task Learning via Representation Learning 34
D.4 Empirical Local Rademacher Complexity Bound 36
D.5 Theorems for transition from empirical to population 39
D.6 Smooth Learning Bounds 40
D.7 Smooth MTL and MTL via MTRL 42
D.8 Local Rademacher complexity chain rule 42
E Detailed comparison with prior works 43
E.1 Comparison with Srebro et al. [2010] 43
E.2 Comparison with Denevi et al. [2019], Khodak et al. [2019] 45
A Additional preliminaries on Gaussian processes and complexities
We recall basic preliminaries on (sub) Gaussian processes [Vershynin, 2018, Wainwright, 2019]. Given a set  $\Theta \subseteq \mathbb{R}^k$ , a random process is a collection of random variables  $\{Z_{\theta}\}_{\theta \in \Theta}$  defined on the same probability space, indexed by elements of set  $\Theta$ . The random process induces the following canonical psuedometric on the (abstract) set  $\Theta$ , defined as  $d(\theta, \theta') = \| Z_{\theta} - Z_{\theta'} \|_{L_2} = \sqrt{\mathbb{E}(Z_{\theta} - Z_{\theta'})^2}$ , (pseudo)metrizing the set  $T$ . This allows us to define the covering and packing number of the set  $\Theta$  with respect to the metric  $d$ , denoted as  $\mathcal{N}(\Theta, d, \epsilon)$  and  $\mathcal{M}(\Theta, d, \epsilon)$  respectively.
Definition 4 (Covering number). An  $\epsilon$ -cover of a set  $\Theta$  with respect to a metric  $d$ , is a set  $C \subseteq \Theta$  such that for any  $\theta \in \Theta$ , there exists a  $c \in C$ , such that  $d(\theta, c) \leq \epsilon$ . The  $\epsilon$ -covering number,  $\mathcal{N}(\Theta, d, \epsilon)$  is the cardinality of the smallest  $\epsilon$ -cover.
Definition 5 (Packing number). An  $\epsilon$ -packing of a set  $\Theta$  with respect to a metric  $d$ , is a set  $P \subseteq \Theta$  such that for all  $p, q \in P$ ,  $p \neq q$ ,  $d(p, q) > \epsilon$ . The  $\epsilon$ -packing number,  $\mathcal{M}(\Theta, d, \epsilon)$  is the cardinality of the largest  $\epsilon$ -packing.
These are related as follows.
Lemma 1 (Wainwright [2019]). For any totally-bounded set  $\Theta$  and  $\epsilon > 0$ ,
$$
\mathcal {M} (\Theta , d, 2 \epsilon) \leq \mathcal {N} (\Theta , d, \epsilon) \leq \mathcal {M} (\Theta , d, \epsilon)
$$
A well-studied problem is to control the process uniformly,  $\mathbb{E}[\sup_{\theta \in \Theta}Z_{\theta}]$  in terms of the geometric properties of the set  $\Theta$
A random process is a Gaussian process, if for all finite sets  $\Theta_0\subseteq \Theta$  , the distribution of the random vector  $\{Z_{\theta}\}_{\theta \in \Theta_{0}}$  is Gaussian. A well-known result regarding controlling  $\mathbb{E}[\sup_{\theta \in \Theta}Z_{\theta}]$  is the Sudakov minoration inequality.
Lemma 2 (Sudakov minoration [Wainwright, 2019]). Let  $\{Z_{\theta}, \theta \in \Theta\}$  be a zero-mean Gaussian process. Then
$$
\mathbb {E} \left[ \sup  _ {\theta \in \Theta} Z _ {\theta} \right] \geq \sup  _ {\alpha > 0} \frac {\alpha}{2} \sqrt {\log \mathcal {M} (\Theta , d , \alpha)},
$$
An important example is the so-called canonical Gaussian process,  $Z_{\theta} = \langle \theta ,g\rangle$ , where  $g\sim \mathcal{N}(0,\mathbb{I}_k)$  is the standard Normal vector. In this case, the canonical metric simply becomes the  $\ell_2$ -norm,  $d(\theta ,\theta^{\prime}) = \| \theta -\theta^{\prime}\|_{2}$ . The uniform control of the random process becomes a standard object, known as Gaussian width of the set  $\Theta$ .
$$
\tilde {\mathfrak {G}} (\Theta) = \mathbb {E} \left[ \sup  _ {\theta \in \Theta} Z _ {\theta} \right] = \mathbb {E} _ {g} \left[ \sup  _ {z \in \Theta} \langle g, \theta \rangle \right].
$$
The celebrated Fernique-Talagrand theory of majorizing measures establishes a characterization of the Gaussian width of a set in terms of its metric geometry - see Vershynin [2018] for more details.
A related concept is that of Gaussian complexity, where the goal is to control the process absolutely and uniformly, defined as,
$$
\mathfrak {G} (\Theta) = \mathbb {E} \left[ \sup  _ {z \in \Theta} | Z _ {\theta} | \right] = \mathbb {E} _ {g} \left[ \sup  _ {z \in T} | \langle g, \theta \rangle | \right].
$$
These two are related as follows.
Lemma 3 (Vershynin [2018]). The following are true about the relationship between Gaussian width and complexity.
1.  $\tilde{\mathfrak{G}} (\Theta)\leq \mathfrak{G}(\Theta).$
2. For any  $\theta \in \Theta$ , we have  $\frac{1}{3}\left(\tilde{\mathfrak{G}}(\Theta) + \|\theta\|\right) \leq \mathfrak{G}(\Theta) \leq 2\left(\tilde{\mathfrak{G}}(\Theta) + \|\theta\|\right)$ .
An important category, generalizing Gaussian process, is that of sub-Gaussian processes, in which for all  $\theta, \theta' \in \Theta$ , the random variable  $Z_{\theta} - Z_{\theta'}$  is  $d(\theta, \theta')$ -sub-Gaussian,
$$
\mathbb {E} \exp \left(\lambda (Z _ {\theta} - Z _ {\theta^ {\prime}}) ^ {2}\right) \leq \exp \left(\frac {\lambda^ {2} d (\theta , \theta^ {\prime}) ^ {2}}{2}\right) \quad \text {f o r a l l} \lambda .
$$
The supremum of sub-Gaussian process is upper bound by Dudley's entropy integral formula, as follows.
Lemma 4 (Refined Dudley's entropy integral formula Wainwright [2019]). Let  $\{Z_{\theta}\}_{\theta \in \Theta}$  be a zero-mean sub-Gaussian process indexed on set  $\Theta$  with diameter  $\bar{B}$ , then
$$
\mathbb {E} [ \sup  _ {\theta \in \Theta} Z _ {\theta} ] \leq \mathbb {E} [ \sup  _ {\theta , \theta^ {\prime} \in \Theta} Z _ {\theta} - Z _ {\theta^ {\prime}} ] \leq \inf  _ {0 \leq \alpha \leq B} \left\{4 \alpha + 1 0 \int_ {\alpha} ^ {B} \sqrt {\log \mathcal {N} (\Theta , d , \alpha)} d \epsilon \right\}.
$$
An important example of sub-Gaussian process is the so-called Bernoulli/Rademacher process, where  $Z_{\theta} = \langle \sigma ,\theta \rangle$ , where  $\sigma \in \{-1,1\} ^k$ , whose co-ordinates are i.i.d. Rademacher random variables. This leads to the notions of Rademacher width  $\tilde{\mathfrak{N}} (\Theta)$  and Rademacher complexity,  $\Re (\Theta)$ , defined as follows.
$$
\tilde {\Re} (\Theta) = \mathbb {E} _ {\sigma} \left[ \sup _ {\theta \in \Theta} \langle \sigma , \theta \rangle \right] \qquad \tilde {\Re} (\Theta) = \mathbb {E} _ {\sigma} \left[ \sup _ {\theta \in \Theta} | \langle \sigma , \theta \rangle | \right].
$$
The Rademacher width and complexity are related in the same way as Gaussian width and complexity (akin to Lemma 3).
Lemma 5 (Vershynin [2018]). The following is true about the relationship between Rademacher width and complexity.
1.  $\tilde{\Re} (\Theta)\leq \Re (\Theta)$
2. For any  $\theta \in \Theta$ , we have  $\frac{1}{3}\left(\tilde{\Re}(\Theta) + \|\theta\|\right) \leq \Re(\Theta) \leq 2\left(\tilde{\Re}(\Theta) + \|\theta\|\right)$ .
Further, the Gaussian and Rademacher widths are related as follows.
Lemma 6 (Wainwright [2019]). For a set  $\Theta \subset \mathbb{R}^k$
$$
\sqrt {\frac {2}{\pi}} \tilde {\Re} (\Theta) \leq \tilde {\mathfrak {G}} (\Theta) \leq 2 \tilde {\Re} (\Theta) \sqrt {\log k}.
$$
Akin to the Fernique-Talagrand majorizing measures theory for Gaussian processes, a complete characterization of the expected suprema for Bernoulli/Rademacher processes in terms of metric and co-ordinate geometry is obtained via (the now proved) Talagrand's Bernoulli conjecture Bednorz and Latała [2014], Ledoux and Talagrand [1991].
Complexity of real-valued function classes. Relatedly, another well-studied concept is Gaussian/Rademacher width/complexity of a set of real-valued functions on a fixed set of inputs. We first define it and then argue how this is simply a special case of the above "set"-view of these. Consider a class of functions  $\mathcal{Q}$  which map an abstract set  $\mathcal{Z}$  to  $\mathbb{R}$ . Given input  $\mathbf{Z}$  consisting of  $n$  points in the domain of  $q\in \mathcal{Q}$ , the Gaussian/Rademacher complexity and width of  $\mathcal{Q}$ , on  $\mathbf{Z}$  are defined as follows.
$$
\tilde {\mathfrak {R}} _ {\mathbf {Z}} (\mathcal {Q}) = \mathbb {E} \left[ \sup  _ {q \in \mathcal {Q}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} q (z _ {i}) \right], \quad \mathfrak {R} _ {\mathbf {Z}} (\mathcal {Q}) = \mathbb {E} \left[ \sup  _ {q \in \mathcal {Q}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} q (z _ {i}) \right| \right],
$$
$$
\tilde {\mathfrak {G}} _ {\mathbf {Z}} (\mathcal {Q}) = \mathbb {E} \left[ \sup _ {q \in \mathcal {Q}} \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} q (z _ {i}) \right], \quad \mathfrak {G} _ {\mathbf {Z}} (\mathcal {Q}) = \mathbb {E} \left[ \sup _ {q \in \mathcal {Q}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} q (z _ {i}) \right| \right],
$$
where  $\sigma_{i}$  and  $g_{i}$  are i.i.d. Rademacher and standard Normal random variables, respectively. The metric induced on the set  $\mathcal{Q}$  is the the  $L_{2}$  distance with respect to the empirical measure on input  $\mathbf{Z}$ ,
$$
d _ {\mathbf {Z}} (q, q ^ {\prime}) = \| q - q ^ {\prime} \| _ {L ^ {2} (\mathbf {Z})} = \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(q \left(z _ {i}\right) - q \left(z _ {i}\right)\right) ^ {2}}.
$$
The corresponding covering and packing numbers of  $\mathcal{Q}$  are denoted as  $\mathcal{N}(\mathcal{Q},\epsilon ,\mathbf{Z})$  and  $\mathcal{M}(\mathcal{Q},\epsilon ,\mathbf{Z})$ . To reduce the above from the set-definition, consider the set
$$
\mathcal {Q} (\mathbf {Z}) = \left\{\frac {1}{n} (q (z _ {1}), q (z _ {2}), \dots , q (z _ {n})): q \in \mathcal {Q} \right\}.
$$
In that case,  $\Re_{\mathbf{Z}}(\mathcal{Q}) = \Re (\mathcal{Q}(\mathbf{Z}))$ . Further,  $d_{\mathbf{Z}}(q,q^{\prime}) = \sqrt{n} d(\theta_{q},\theta_{q^{\prime}})$  where  $\theta_q = \frac{1}{n}\left(q(z_1),q(z_2),\ldots ,q(z_n)\right)$  and  $\theta_{q^{\prime}} = \frac{1}{n}\left(q^{\prime}(z_{1}),q^{\prime}(z_{2}),\ldots ,q^{\prime}(z_{n})\right)$  and thus lie in  $\mathcal{Q}(\mathbf{Z})$ .
Complexity of function classes in the multi-task learning setting. We now generalize the above to functions encountered in multi-task representation learning. For input space  $\mathcal{Z}$ ,  $p, q, n \in \mathbb{N}$ , consider a class of vector-valued functions  $\mathcal{Q}: \mathcal{Z} \to \mathbb{R}^q$ , and a dataset  $\mathbf{Z} = (z_j^i)_{j \in [p], i \in [n]}$ , where  $z_j^i \in \mathcal{Z}$ . For  $\mathcal{Q}^{\otimes p}$ , the  $p$ -fold Cartesian product of  $\mathcal{Q}$ , we define its data-dependent Rademacher width,  $\tilde{\mathfrak{R}}_{\mathbf{Z}}(\cdot)$ , data-dependent Gaussian width,  $\tilde{\mathfrak{G}}_{\mathbf{Z}}(\cdot)$ , data-dependent Rademacher complexity,  $\mathfrak{R}_{\mathbf{Z}}(\cdot)$ , data-dependent Gaussian complexity,  $\mathfrak{G}_{\mathbf{Z}}$ , with respect to input  $\mathbf{Z}$  as,
$$
\tilde {\mathfrak {R}} _ {\mathbf {Z}} \left(\mathcal {Q} ^ {\otimes p}\right) = \mathbb {E} _ {\sigma_ {i, j, k}} \left[ \sup  _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \frac {1}{n p} \sum_ {i, j k = 1} ^ {n, p, q} \sigma_ {i j k} \left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k} \right], \quad (\text {R a d e m a c h e r w i d t h})
$$
$$
\tilde {\mathfrak {G}} _ {\mathbf {Z}} \left(\mathcal {Q} ^ {\otimes p}\right) = \mathbb {E} _ {g _ {i, j, k}} \left[ \sup  _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \frac {1}{n p} \sum_ {i, j, k = 1} ^ {n, p, q} g _ {i, j, k} \left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k} \right], \quad (\text {G a u s s i a n w i d t h})
$$
$$
\mathfrak {R} _ {\mathbf {Z}} \left(\mathcal {Q} ^ {\otimes p}\right) = \mathbb {E} _ {\sigma_ {i, j, k}} \left[ \sup  _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \left| \frac {1}{n p} \sum_ {i, j k = 1} ^ {n, p, q} \sigma_ {i j k} \left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k} \right| \right], \quad (\text {R a d e m a c h e r c o m p l e x i t y})
$$
$$
\mathfrak {G} _ {\mathbf {Z}} \left(\mathcal {Q} ^ {\otimes p}\right) = \mathbb {E} _ {g _ {i, j, k}} \left[ \sup  _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \left| \frac {1}{n p} \sum_ {i, j, k = 1} ^ {n, p, q} g _ {i, j, k} \left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k} \right| \right], \quad (\text {G a u s s i a n c o m p l e x i t y})
$$
where  $\sigma_{i,j,k}$  and  $g_{i,j,k}$  are i.i.d. Rademacher and standard Normal random variables, respectively.
To reduce the above from the set-based definition, consider the set
$$
\mathcal {Q} ^ {\otimes p} (\mathbf {Z}): \left\{\frac {1}{n p} \left(\left(\left(q _ {j} \left(z _ {j} ^ {i}\right)\right) _ {k}\right) _ {j}\right) _ {i}: \mathbf {q} \in \mathcal {Q} ^ {\otimes p} \right\}.
$$
In the above, the notation  $\frac{1}{np} (((q_j(z_j^i))_k)_j)_i$  denotes a vector in  $\mathbb{R}^{npq}$  obtained via enumerating  $i,j$  and  $k$ .
The corresponding metric analogously is,
$$
d _ {\mathbf {Z}} (\boldsymbol {q}, \boldsymbol {q} ^ {\prime}) = \| \boldsymbol {q} - \boldsymbol {q} ^ {\prime} \| _ {L ^ {2} (\mathbf {Z})} = \sqrt {\frac {1}{n p} \sum_ {i , j , k} \left((q _ {j} (z _ {j} ^ {i})) _ {k} - (q _ {j} (z _ {j} ^ {i})) _ {k}\right) ^ {2}} = \sqrt {n p} d _ {\mathbf {Z}} (\theta_ {\boldsymbol {q}}, \theta_ {\boldsymbol {q} ^ {\prime}}),
$$
where  $\theta_{\mathbf{q}} = \frac{1}{np} ((q_j(z_j^i))_k)_j)_i$  and similarly  $\theta_{\mathbf{q}'}$ , which lie in  $\mathcal{Q}^{\otimes p}(\mathbf{Z})$ .
The above can be used to define covering and packing numbers of  $\mathcal{Q}^{\otimes p}$ , denoted as  $\mathcal{N}(\mathcal{Q}^{\otimes p},\epsilon ,\mathbf{Z})$  and  $\mathcal{M}(\mathcal{Q}^{\otimes p},\epsilon ,\mathbf{Z})$  respectively.
The concepts of Dudley's entropy integral formula and Sudakov minoration thus generalize accordingly. A result of the following form, for real-valued functions, appears in Srebro et al. [2010].
Lemma 7 (Refined Dudley's entropy integral). Consider a class of vector-valued functions  $\mathcal{Q}$  and input set  $\mathbf{Z} = (z_j^i)_{j\in [p],i\in [n]}$ . Define  $B := \sup_{\boldsymbol {q}\in \mathcal{Q}^{\otimes p}}\| \boldsymbol {q}\|_{L^2 (\mathbf{Z})} = \sqrt{np}\sup_{\boldsymbol {q}\in \mathcal{Q}^{\otimes p}}\left\| \frac{1}{np} ((q_j(z_j^i))_k)_j)_i\right\|$ . Then,
$$
\begin{array}{l} \mathfrak {R} _ {\mathbf {Z}} (\mathcal {F}) \leq \inf  _ {0 \leq \alpha \leq B / \sqrt {n p}} \left\{4 \alpha + 1 0 \int_ {\alpha} ^ {B / \sqrt {n p}} \sqrt {\log \mathcal {N} \left(\mathcal {Q} ^ {\otimes p} , \epsilon , \mathbf {Z}\right)} d \epsilon \right\} \\ = \inf  _ {0 \leq \alpha \leq B} \left\{4 \alpha + 1 0 \int_ {\alpha} ^ {B} \sqrt {\frac {\log \mathcal {N} (\mathcal {Q} ^ {\otimes p} , \epsilon , \mathbf {Z})}{n p}} d \epsilon \right\}. \\ \end{array}
$$
The equality above follows from the change of variables.
Worst-case complexities of functions. Often, we take the sup over datasets of a specified size, in the above definitions of Rademacher/ Gaussian width/complexity, yielding their worst-case counterparts, denoted as,
$$
\tilde {\mathfrak {G}} _ {n p} (\mathcal {Q} ^ {\otimes p}) = \sup  _ {\mathbf {Z} \in \mathcal {Z} ^ {n p}} \tilde {\mathfrak {G}} _ {\mathbf {Z}} (\mathcal {Q} ^ {\otimes p}), \qquad \mathfrak {G} _ {n p} (\mathcal {Q} ^ {\otimes p}) = \sup  _ {\mathbf {Z} \in \mathcal {Z} ^ {n p}} \mathfrak {G} _ {\mathbf {Z}} (\mathcal {Q} ^ {\otimes p}),
$$
$$
\tilde {\mathfrak {R}} _ {n p} (\mathcal {Q} ^ {\otimes p}) = \sup  _ {\mathbf {Z} \in \mathcal {Z} ^ {n p}} \tilde {\mathfrak {R}} _ {\mathbf {Z}} (\mathcal {Q} ^ {\otimes p}), \qquad \mathfrak {R} _ {n p} (\mathcal {Q} ^ {\otimes p}) = \sup  _ {\mathbf {Z} \in \mathcal {Z} ^ {n p}} \mathfrak {R} _ {\mathbf {Z}} (\mathcal {Q} ^ {\otimes p}).
$$
The corresponding metric similarly is
$$
d _ {n p} (\boldsymbol {q}, \boldsymbol {q} ^ {\prime}) = \sup  _ {\mathbf {Z} \in \mathcal {Z} ^ {n p}} d _ {\mathbf {Z}} (\boldsymbol {q}, \boldsymbol {q} ^ {\prime}),
$$
which can then be used to define covering and packing numbers of  $\mathcal{Q}^{\otimes p}$ , denoted as  $\mathcal{N}(\mathcal{Q}^{\otimes},\epsilon ,np)$  and  $\mathcal{M}(\mathcal{Q}^{\otimes},\epsilon ,np)$ , respectively.
As before, Dudley's entropy integral formula and Sudakov minoration generalize analogously.
A.2 Sub-root functions and local Rademacher Complexity
We present a key property about sub-root functions from [Bartlett et al., 2005] below.
Lemma 8 (Lemma 3.2. [Bartlett et al., 2005]). If  $\psi :[0,\infty)\to [0,\infty)$  is a nontrivial sub-root function, then it is continuous on  $[0,\infty)$  and the equation  $\psi (r) = r$  has a unique positive solution. Moreover, if we denote the solution by  $r^*$ , then for all  $r > 0,r\geq \psi (r)$  if and only if  $r^{*}\leq r$
B Task diversity digression
In this section we make a series of observations regarding the task-diversity definition introduced by Tripuraneni et al. [2020]. Within this work, they introduce two definitions that are used to relate average source tasks performance to target task performance: task-averaged representation difference and worst-case representation difference.
Definition 6 (The task-averaged representation difference w.r.t  $h\in \mathcal{H}$
$$
\bar {d} _ {\mathcal {F}, \boldsymbol {f}} (h ^ {\prime}; h) = \frac {1}{t} \sum_ {j = 1} ^ {t} \inf  _ {f ^ {\prime} \in \mathcal {F}} \mathbb {E} _ {x _ {j}, y _ {j} \sim P _ {j}} \left\{\ell \left(f ^ {\prime} \circ h ^ {\prime} (x _ {j}), y _ {j}\right) - \ell \left(f _ {j} \circ h (x _ {j}), y _ {j}\right) \right\}
$$
Importantly, like the work which introduces these concepts, it is possible to statistically control  $\bar{d}_{\mathcal{F},f}\left(\hat{h};h^{*}\right)$ , where  $\hat{h}$  results from the first step of the two-tage ERM procedure, see the variational definition Eqn. (2).
The other quantity they introduce is the worst-case representation difference.
Definition 7 (The worst-case representation difference w.r.t  $h, h' \in \mathcal{H}$ ).
$$
d _ {\mathcal {F}, \mathcal {F} _ {0}} (h ^ {\prime}; h) = \sup _ {f _ {0} \in \mathcal {F}} \inf _ {f ^ {\prime} \in \mathcal {F}} \mathbb {E} _ {x, y \sim P _ {0}} \Bigg \{\ell (f ^ {\prime} \circ h ^ {\prime} (x), y) - \ell (f _ {0} \circ h (x), y) \Bigg \},
$$
Finally, they introduce the following assumption on the two quantities defined above. It has a multiplicative parameter  $\nu$  and additive parameter  $\varepsilon$ .
Definition 8  $((\nu, \varepsilon)$ -task diversity w.r.t  $h \in \mathcal{H}$  [Tripuraneni et al., 2020]). For a function class  $\mathcal{F}$ , we say  $\pmb{f} \in \mathcal{F}^{\otimes t}$  is  $(\nu, \epsilon)$ -diverse over  $\mathcal{F}_0$  for a representation  $h$ , if uniformly for all  $h' \in \mathcal{H}$ ,
$$
d _ {\mathcal {F}, \mathcal {F} _ {0}} (h ^ {\prime}; h) \leq \bar {d} _ {\mathcal {F}, \mathbf {f}} (h ^ {\prime}; h) / \nu + \epsilon
$$
We will make a series of observations about these quantities which provide substantial simplification in our setting.
First, observe in the setting of product space,
$$
\bar {d} _ {\mathcal {F}, \boldsymbol {f}} \left(h ^ {\prime}; h\right) = \inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} \left\{R _ {\text {s o u r c e}} \left(\boldsymbol {f} ^ {\prime}, h ^ {\prime}\right) - R _ {\text {s o u r c e}} (\boldsymbol {f}, h) \right\}
$$
$$
d _ {\mathcal {F}, \mathcal {F} _ {0}} \left(h ^ {\prime}; h\right) = \sup  _ {f _ {0} \in \mathcal {F} _ {0}} \inf  _ {f ^ {\prime} \in \mathcal {F}} \left\{R _ {\text {t a r g e t}} \left(f ^ {\prime}, h ^ {\prime}\right) - R _ {\text {t a r g e t}} \left(f _ {0}, h\right) \right\}
$$