SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Austin Watkins
awatki29@jhu.edu
Enayat Ullah
enayat@jhu.edu
We study the problem of transfer learning via Multi-Task Representation Learning (MTRL), wherein multiple source tasks are used to learn a good common representation, and a predictor is trained on top of it for the target task. Under standard regularity assumptions on the loss function and task diversity, we provide new statistical rates on the excess risk of the target task, which demonstrate the benefit of representation learning. Importantly, our rates are optimistic, i.e., they interpolate between the standard  $\mathcal{O}(m^{-1/2})$  rate and the fast  $\mathcal{O}(m^{-1})$  rate, depending on the difficulty of the learning task, where  $m$  is the number of samples for the target task. Besides the main result, we make several new contributions, including giving optimistic rates for excess risk of source tasks (Multi-Task Learning (MTL)), a local Rademacher complexity theorem for MTRL and MTL, as well as a chain rule for local Rademacher complexity for composite predictor classes.
Transfer learning has emerged as a powerful tool in modern machine learning. The goal is to find a good predictor on a given target task with little data, by extracting knowledge from source task(s). The source tasks can either be explicitly given (supervised learning), or constructed from domain knowledge (unsupervised/self-supervised learning). An example of the supervised learning approach is a widely successful practice in deep learning, wherein a complex model is first trained on a source task with plentiful data, then applied to a target task by extracting representations of data by freezing the last layer of the complex model, and training a simple (linear) classifier on top of it [Bengio et al., 2013, Donahue et al., 2014]. Similarly, the unsupervised approaches of training a (large) neural network on a general objective has seen tremendous success, especially in natural language processing [Brown et al., 2020, Devlin et al., 2019] and computer vision. As before, a simple linear classifier trained on the output of the penultimate layer works well on a target task.
Representation learning, where the goal is to learn representations of data that are useful across multiple domains and tasks, is a common theme in the examples above. We study a popular formalization of this type of machine learning called Multi-Task Representation Learning (MTRL). As the name suggests, we are given  $t$  source tasks each with a small number of data points, say  $n$ . We seek to effectively pool the  $nt$  data points to extract a common good representation useful for the target task. Therefore, if our tasks are sufficiently "diverse", we hope to effectively circumvent our data scarcity by sharing knowledge between tasks. As a by-product, since the shared representation
can also be used for the source tasks themselves, this procedure is also effective for the related problem of multi-task learning (MTL).
To study the transfer of knowledge via representations within the MTRL framework, we consider a composite-learning model. This model consists of  $\mathcal{F}$  and  $\mathcal{H}$ , a class of hypotheses and representations respectively, where the final end-to-end predictors are given via the composition  $f\circ h$ , for all  $f\in \mathcal{F},h\in \mathcal{H}$ . Since we must decide between compositions, we use a two-stage empirical risk minimization procedure. First, the multi-task representation learning stage, the procedure selects one common representation and  $t$  predictors, one for each task, by minimizing the average empirical loss on the source tasks. Second, the transfer learning stage, the procedure selects a predictor for the target task that minimizes the empirical loss, w.r.t.  $m$  samples, on top of the fixed representation from stage one.
We desire to learn a good predictor – i.e., one that has small excess risk – from  $\mathcal{F} \circ \mathcal{H}$  for the target task. In this work, we consider the class  $\mathcal{H}$  to be complex (e.g., deep neural networks), whereas the class  $\mathcal{F}$  consists of simple functions (e.g., linear predictors). Therefore, the class  $\mathcal{F} \circ \mathcal{H}$  is complex, so directly learning it would require a large number of samples. Yet, if we can effectively pool all our data from each task, the total number of examples could be sufficiently plentiful to successfully learn a complex representation. Further, a good representation can significantly reduce the statistical burden of learning a good predictor for the target task. This phenomenon is formalized by the prior work of Tripuraneni et al. [2020], which is the work most related to ours. Specifically, they show that if the loss function is Lipschitz and the tasks are sufficiently diverse (see Definition 2), the excess (transfer) risk for the target task is bounded as,
$$
\text {E x c e s s t r a n s f e r} \quad \text {r i s k} = \tilde {O} \left(\sqrt {\frac {C (\mathcal {H}) + t C (\mathcal {F})}{n t}} + \sqrt {\frac {C (\mathcal {F})}{m}}\right),
$$
where  $C(\mathcal{H})$  and  $C(\mathcal{F})$  denote the complexity measures associated with learning  $\mathcal{H}$  and  $\mathcal{F}$  respectively (precise definition to follow in a later section).
While the above rate has the desirable properties described above, it is insufficient on many fronts, which our work seeks to address. First, it is akin to the standard rate of  $n^{-1/2}$ , albeit optimal in agnostic settings, for single-task PAC learning. However, it is well-known that a fast rate is achievable under realizability in the standard supervised learning. But, the above guarantee does not capture this possibility. Importantly, the work of Tripuraneni et al. [2020] operates under the assumption that the loss function is Lipschitz – in such settings, it is generally not possible to get fast rates, despite realizability, even for the special case of learning half-spaces [Srebro et al., 2010]. In the single task setting, the literature contains a rich theory of guarantees that interpolate between the slow and fast rate depending on the level of realizability, such guarantees are called "optimistic rates". We derive such optimistic rates for MTRL. Consequently, as an important application of our general theory, we show that the (transfer) risk of the MTRL approach on the target task is bounded as,
$$
\tilde {O} \left(L _ {\text {t a r g e t}} ^ {*} + \sqrt {\frac {L _ {\text {s o u r c e}} ^ {*} (C (\mathcal {H}) + t C (\mathcal {F}))}{n t}} + \sqrt {\frac {L _ {\text {t a r g e t}} ^ {*} C (\mathcal {F})}{m}} + \frac {C (\mathcal {H}) + t C (\mathcal {F})}{n t} + \frac {C (\mathcal {F})}{m}\right) \tag {1}
$$
when using non-negative smooth losses (such as squared loss in linear regression or smoothed ramp loss for classification [Cortes et al., 2021, Srebro et al., 2010]). The values  $L_{\mathrm{source}}^*$  and  $L_{\mathrm{target}}^*$  represent the minimum average risk over the source tasks and target task respectively, hence (1) yields a fast rate under mild realizability, i.e. when they are small.
In the course of proving the above, we extend a number of foundational results in the single-task setting to the multi-task setting. We list them in our contributions below.
1. We give a general local Rademacher complexity result for transfer learning via multitask representation learning – see Theorem 1.
2. We also provide a local Rademacher complexity result for multitask learning – see Theorem 3. Our result has two benefits compared to prior work [Yousefi et al., 2018]. First, we perform MTL via MTRL that leads to improved bounds which show the benefit of pooling data as opposed to learning each task separately. Second, we operate under the special, yet fairly natural setting, of hypothesis classes with product-space structure – this simplifies the proof significantly as well as recovers the result for single-task setting with the exact same constants (unlike rates in Yousefi et al. [2018]).
3. For non-negative smooth losses, we derive optimistic rates (such as Eqn. (1)) for transfer learning and MTL via MTRL. Notably, when we restrict our rates to the single task setting, our proof is significantly simpler and our obtained bound on local Rademacher complexity is smaller compared to the prior work of Srebro et al. [2010].
4. Finally, we present a chain rule for local Rademacher complexity for composite settings, which allows us to decouple the (local) complexities of representation and predictor classes.
1.1 Our techniques
In this section, we expand on our contributions and provide an overview of techniques and challenges overcome in the context of prior art.
Local Rademacher complexity for transfer learning via MTRL. We build on the work of Tripuraneni et al. [2020], which introduced the notion of task diversity. They showed that task diversity leads to improved rates on excess transfer risk via MTRL, compared to training only on the target task. Their work however yields the standard  $O(m^{-1/2})$  rate on excess risk, akin to what is obtained in agnostic settings. We provide a local Rademacher complexity result for transfer learning via MTRL, yielding optimistic rates depending on distributional properties. Our methods are based on tools and techniques from the seminal local Rademacher complexity paper of Bartlett et al. [2005]. A crucial component of our results is relative deviation Bennett concentration inequalities established via log-Sobolev inequalities together with the entropy method [Bousquet, 2002]. We make the crucial observation that the above concentration inequalities, though established for the single-task setting, are general enough to be used to extend the local Rademacher complexity framework to the multi-task and transfer learning setting.
Local Rademacher complexity for multi-task learning via MTRL. Intermediate to our above contribution, we provide a local Rademacher complexity result for MTL via MTRL, yielding optimistic rates. We note that the prior work of Yousefi et al. [2018] already provided a local Rademacher complexity result for MTL, though not via MTRL; importantly, our work establishes the benefits of pooling data compared to learning each task separately. However, in contrast to Yousefi et al.  $[2018]^1$ , our setting is limited to hypothesis classes with a product space structure rather than a general vector-valued hypothesis. This simply means that the tasks are permutation-invariant in our setting, which we argue is a fairly reasonable assumption in the multi-task context. This assumption makes the analysis much simpler as we can reuse concentration inequalities for single task setting whereas Yousefi et al. [2018] had to extend log-Sobolev concentration inequalities to the multi-task setting. Additionally, this yields bounds with better constants than Yousefi et al. [2018] – in the special case of a single task, this recovers the known result with the exact same constants.
Optimistic rates for non-negative smooth losses. An important application of our theorems is providing optimistic rates when using non-negative smooth losses, which is known to enable distribution-free optimistic rates in a single task setting [Cortes et al., 2021, Srebro et al., 2010]. This result shows the provable benefits of representation learning. Further, it can be combined with the chain rule for Gaussian complexity of Tripuraneni et al. [2020]. This allows us to separate the composite class  $\mathcal{F} \circ \mathcal{H}$  into the complexities of representation class  $\mathcal{H}$  and predictor class  $\mathcal{F}$  and also reuse existing complexity bounds in the literature. While we borrow some of the tools for optimistic rates theory in the single task setting from Srebro et al. [2010], the prior proof technique does not directly extend to the multi-task setting. Consequently, we modify the proof, simplifying it significantly as well as getting an improved bound, even in the single task setting. In particular, Srebro et al. [2010] bounds the local Rademacher complexity of the loss class by an  $\ell_2$ -covering number via Dudley's entropy integral formula, which is then bound by  $\ell_{\infty}$ -covering number of hypothesis class using smoothness, which in turn is bound by the fat-shattering dimension, which eventually is bound by the (global) Rademacher complexity of the hypothesis class. Some of the steps seemingly do not generalize in the multi-task setting because there are no (well-studied) multi-task analogues, e.g. the fat-shattering dimension. Our proof starts with the above bound on the local Rademacher complexity of the loss class by an  $\ell_2$ -covering number via Dudley's entropy integral formula but then applies the well-known Sudakov minoration inequality that yields a bound in terms of Gaussian width; this is in turn bound by Rademacher complexity by standard known relations between the two quantities.
Chain rule for local Rademacher complexities. In the application to non-negative smooth losses, the local Rademacher complexity of the loss class is bounded by a non-trivial function of the
<sup>1</sup> Yousefi et al. [2018] additionally considers Bernstein classes, which we do not. However, we think that our results can be extended to that setting.
(global) Gaussian complexity of the hypothesis class which is further bounded using the chain rule of Tripuraneni et al. [2020]. For general loss functions, there may not be any such non-trivial function. To this end, for Lipschitz losses, we develop a chain rule of local Rademacher complexity, Theorem 5, that similarly allows separating the local complexities of the representation and predictor class.
As mentioned before, our work is most related to, and builds on, Tripuraneni et al. [2020]. Other related work includes an early work of Baxter [2000] that gives guarantees on excess risk of transfer learning and multi-task learning via MTRL under a certain generative model for tasks. This was subsequently improved by Maurer et al. [2016], Pontil and Maurer [2013]. These early works give rates of the form  $\mathcal{O}\left(t^{-1/2} + m^{-1/2}\right)$  and do not capture the advantage of having a large number of source samples per task. Prior work has extensively studied the special case of linear representations and/or linear predictors, partly in the context of meta learning, for instance see Du et al. [2021], Tripuraneni et al. [2021], Xu and Tewari [2021]. Further, multitask learning has been explored under various notions of task relatedness [Ben-David and Borbely, 2008, Cavallanti et al., 2010].
Regarding optimistic rates, the seminal work of Vapnik and Cervonenkis [1971] provided the first optimistic rates, via a normalized uniform convergence analysis, but was limited to the PAC learning setting. Much later, a line of work of Bousquet et al. [2002], Koltchinskii and Panchenko [2000] derived optimistic rates in more general settings using the technique of localization. In this area, an important result is the local Rademacher complexity theorem of Bartlett et al. [2005], which many works have leveraged to yield improved rates for various problems [Blanchard et al., 2007, Cortes et al., 2013, Ullah et al., 2018]. Importantly, this theorem applies to learning with non-negative smooth losses [Srebro et al., 2010] where the local Rademacher complexity-based technique can be used to give distribution-free optimistic rates. Further, optimistic rates have seen renewed interest owing to the interpolation/benign overfitting phenomenon in deep learning. In certain problems, predictors may interpolate the data (zero training error), yet have small risk. A line of work [Zhou et al., 2020, 2021] shows that in certain linear regression problems, optimistic rates can be used to derive risk bounds that explain the phenomenon, where the usual tool of uniform convergence provably fails.
There has been some work on optimistic rates in the multi-task setting. The work of Yousefi et al. [2018] established a local Rademacher complexity theorem for multi-task learning (see Section "Our techniques" for detailed comparison). Besides that, the work of Reeve and Kaban [2020] proved optimistic rates for vector-valued hypothesis classes for self-bounding loss functions (which includes non-negative smooth losses).
Notation. Denote  $\| \cdot \|_2$  be the Euclidean norm and  $\| \cdot \|_\infty$  be the infinity norm. We use the convention that  $[m] = [1, \ldots, m]$  is the list of contiguous integers starting at 1 and ending at  $m$ . For  $f = (f_1, \ldots, f_t)$  let  $f^2 = (f_1^2, \ldots, f_t^2)$  and denote
$$
P \pmb {f} := \frac {1}{t} \sum_ {j = 1} ^ {t} P _ {j} f _ {j} = \frac {1}{t} \sum_ {j = 1} ^ {t} \mathbb {E} \left(f _ {j} (X _ {j})\right), \quad \hat {P} ^ {n} \pmb {f} := \frac {1}{t} \sum_ {j = 1} ^ {t} \hat {P} _ {j} ^ {n} f _ {j} = \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} f (X _ {j} ^ {i}).
$$
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{V} \subseteq \mathbb{R}$  denote the input feature space and the output label space respectively. The source tasks and the target task are represented using probability distributions  $\{P_j\}_{j=1}^t$  and  $P_0$ , respectively. Let  $\mathcal{F}$  and  $\mathcal{F}_0$  be a class of predictor functions from  $\mathbb{R}^k$  to  $\mathcal{V}$  for the source tasks and target task respectively, and  $\mathcal{H}$  a class of representation functions from  $\mathbb{R}^d$  to  $\mathbb{R}^k$ .
Following Tripuraneni et al. [2020], we assume that marginal distribution of  $P_{j}$  over  $\mathcal{X}$  is the same for all the  $j$  from 0 to  $t$ , and that there exists a common representation  $h^{*} \in \mathcal{H}$  and task-specific predictors  $f_{j}^{*} \in \mathcal{F}$  for  $j \in [t]$  and  $f_{0} \in \mathcal{F}_{0}$  such that  $P_{j}$  can be decomposed as  $P_{j}(x,y) = P_{x}(x)P_{y|x}(y|f_{j}^{*} \circ h^{*}(x))$ . Note that this decomposition does not assume that  $f_{j}^{*} \circ h^{*}$  is the optimal in-class predictor; however, we will assume it is to make our optimistic rates more meaningful (although this is not strictly necessary). Also the decomposition implicitly assumes that  $y$  depends on  $x$  only via  $f_{j}^{*} \circ h^{*}(x)$ , and thus any additional noise in  $y$  is independent of  $x$ . Note that like Tripuraneni et al. [2020], we allow the predictor class for the target task  $\mathcal{F}_{0}$  to be different than on the source tasks  $\mathcal{F}$ . For example, when performing logistic regression with a linear representation
$B \in \mathbb{R}^{k \times d}$  and linear predictor  $w \in \mathbb{R}^k$ , we have  $P_{y|x}(y = 1|f_j^* \circ h^*(x)) = \sigma(w^TB^Tx)$ , where  $\sigma$  is the sigmoid function.
Given a loss function  $\ell : \mathbb{R} \times \mathcal{Y} \to \mathbb{R}$ , the goal is to find a good predictor  $\hat{f} \circ \hat{h}$  from the composite-class  $\mathcal{F}_0 \circ \mathcal{H}$  with small transfer risk,
$$
R _ {\mathrm {t a r g e t}} (\hat {f} _ {0}, \hat {h}) = \mathbb {E} _ {(x, y) \sim P _ {0}} [ \ell (\hat {f} _ {0} \circ \hat {h} (x), y) ].
$$
Yet, as standard, we do not have access to the joint distribution for all tasks, but instead  $n$  independent and identically distributed (i.i.d.) samples from each source task and  $m$  i.i.d. samples for the target task. Let  $(x_j^i, y_j^i)$  be the  $i^{\text{th}}$  sample for the  $j^{\text{th}}$  task. To accomplish the above, we use the source data to find a representation, and the target data to find a predictor for the target task. Specifically, in this MTRL procedure, first, we train on all source task data, then freeze the resulting representation. Second, we train over this frozen representation to find a good predictor on the target samples.
We formalize the above as the following two-stage Empirical Risk Minimization (ERM) procedure,
$$
(\hat {\boldsymbol {f}}, \hat {h}) \in \underset {\boldsymbol {f} \in \mathcal {F} ^ {\otimes t}, h \in \mathcal {H}} {\arg \min } \frac {1}{n t} \sum_ {j = 1} ^ {t} \sum_ {i = 1} ^ {n} \ell \left(f _ {j} \circ h \left(x _ {j} ^ {i}\right), y _ {j} ^ {i}\right), \quad (\text {M u l t i - t a k (r e p r e s e n t a t i o n) l e a n i n g}) \tag {2}
$$
$$
\hat {f} _ {0} \in \underset {f \in \mathcal {F} _ {0}} {\arg \min } \frac {1}{m} \sum_ {i = 1} ^ {m} \ell \left(f \circ \hat {h} \left(x _ {0} ^ {i}\right), y _ {0} ^ {i}\right). \quad (\text {T r a n s f e r l e a n n i n g}) \tag {3}
$$
Besides bounding the transfer risk, we seek to understand if the above procedure also yields improved bounds for the source tasks (the problem of MTL).
$$
R _ {\text {s o u r c e}} (\hat {\boldsymbol {f}}, \hat {h}) = \frac {1}{t} \sum_ {j = 1} ^ {t} \mathbb {E} _ {(x, y) \sim P _ {j}} [ \ell (\hat {f} _ {j} \circ \hat {h} (x), y) ]
$$
We make the following regularity assumptions.
A: The loss function  $\ell$  is nonnegative and  $b$ -bounded, i.e.,  $0 \leq \ell(y', y) \leq b < \infty$  for all  $y', y \in \mathcal{V}$ .
B: All functions in  $\mathcal{F}$  are  $L$ -Lipschitz for some  $0 < L < \infty$  w.r.t.  $\| \cdot \|_2$ , i.e.,  $\| f(z_1) - f(z_2) \|_2 \leq L \| z_1 - z_2 \|_2$  for all  $z_1, z_2 \in Dom(f)$  and  $f \in \mathcal{F}$ .
$\pmb{C}$  : Any  $f\circ h\in \mathcal{F}\circ \mathcal{H}$  is  $D$  -bounded over  $\mathcal{X}$  w.r.t.  $\| \cdot \|_{\infty}$  , i.e.,  $\sup_{x\in \mathcal{X}}|f\circ h(x)|\leq D <   \infty$  for all  $f\in \mathcal{F}$  and  $h\in \mathcal{H}$
Unlike Tripuraneni et al. [2020], we assume for some of our theorems that our function is smooth instead of Lipschitz, as leveraging smoothness is key to achieving fast rates [Srebro et al., 2010].
Definition 1 (H-smoothness). The loss function  $\ell: \mathbb{R} \times \mathcal{V} \to \mathbb{R}$  is H-smooth if  $\left|\ell_y'(y_1, y) - \ell_y'(y_2, y)\right| \leq H\left|y_1 - y_2\right|$  for all  $y_1, y_2 \in \mathbb{R}, y \in \mathcal{V}$ .
Task diversity [Tripuraneni et al., 2020]. We benefit from learning a representation for a new target task, so long as there must be information encoded for that task by the source tasks. A typical way to quantify this encoded information is to control the difference between the learned representation and the underlying representation. To this end, we use the framework introduced by Tripuraneni et al. [2020] that quantifies a measure of task diversity. As mentioned in their work, this framework recovers task diversity assumptions in Du et al. [2021] exactly. In our setting, the definitions given in Tripuraneni et al. [2020] simplify considerably.
Definition 2. The tasks  $\{P_j\}_{j=1}^t$  are  $(\nu, \epsilon)$ -diverse over  $P_0$ , if for the corresponding  $\mathbf{f}^* \in \mathcal{F}^{\otimes t}$ ,  $f_0^* \in \mathcal{F}_0$  and representation  $h^* \in \mathcal{H}$ , we have that for all  $h' \in \mathcal{H}$
$$
\inf  _ {f ^ {\prime} \in \mathcal {F} _ {0}} R _ {t a r g e t} (f ^ {\prime}, h ^ {\prime}) - R _ {t a r g e t} (f _ {0} ^ {*}, h ^ {*}) \leq \frac {1}{\nu} \left(\inf  _ {\boldsymbol {f} ^ {\prime} \in \mathcal {F} ^ {\otimes t}} R _ {s o u r c e} (\boldsymbol {f} ^ {\prime}, h ^ {\prime}) - R _ {s o u r c e} (\boldsymbol {f} ^ {*}, h ^ {*})\right) + \varepsilon .
$$
Parameters  $\nu$  and  $\varepsilon$  quantify the similarity between learning the source tasks and the target task. For a detailed analysis of these parameters and the framework introduced by Tripuraneni et al. [2020], see Appendix B.
Local Rademacher Complexity. We now define the measures of complexity that are used in our work. Let  $u, p, n \in \mathbb{N}$ , an input space  $\mathcal{Z}$ , a class of vector-valued functions  $\mathcal{Q}: \mathcal{Z} \to \mathbb{R}^u$ , and a dataset  $\mathbf{Z} = (z_j^i)_{j \in [p], i \in [n]}$  where  $z_j^i \in \mathcal{Z}$ . Define the data-dependent Rademacher width,  $\tilde{\mathfrak{R}}_{\mathbf{Z}}(\cdot)$ , as
$$
\tilde {\mathfrak {N}} _ {\mathbf {Z}} (\mathcal {Q} ^ {\otimes p}) = \mathbb {E} _ {\sigma_ {i, j, k}} \left[ \sup _ {\boldsymbol {q} \in \mathcal {Q} ^ {\otimes p}} \frac {1}{n p} \sum_ {i, j, k = 1} ^ {n, p, u} \sigma_ {i, j, k} (q _ {j} (z _ {j} ^ {i})) _ {k} \right],
$$
where  $\sigma_{i,j,k}$  are i.i.d. Rademacher random variables. Analogously, define the data-dependent Gaussian width,  $\tilde{\mathfrak{G}}_{\mathbf{Z}}(\cdot)$ , as