Title: Optimistic Rates for Multi-Task Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning Theory

Multi-task and Transfer Learning

Classification
Abstract: We study the problem of transfer learning via Multi-Task Representation Learning (MTRL), wherein multiple source tasks are used to learn a good common representation, and a predictor is trained on top of it for the target task. Under standard regularity assumptions on the loss function and task diversity, we provide new statistical rates on the excess risk of the target task, which demonstrate the benefit of representation learning. Importantly, our rates are optimistic, i.e., they interpolate between the standard $O(m^{-1/2})$ rate and the fast $O(m^{-1})$ rate, depending on the difficulty of the learning task, where $m$ is the number of samples for the target task. Besides the main result, we make several new contributions, including giving optimistic rates for excess risk of source tasks (multi-task learning (MTL)), a local Rademacher complexity theorem for MTRL and MTL, as well as a chain rule for local Rademacher complexity for composite predictor classes.

Optimistic Rates for Multi-Task Representation Learning

Abstract

We study the problem of transfer learning via Multi-Task Representation Learning (MTRL), wherein multiple source tasks are used to learn a good common representation, and a predictor is trained on top of it for the target task. Under standard regularity assumptions on the loss function and task diversity, we provide new statistical rates on the excess risk of the target task, which demonstrate the benefit of representation learning. Importantly, our rates are optimistic, i.e., they interpolate between the standard O ( m -1 / 2 ) rate and the fast O ( m -1 ) rate, depending on the difficulty of the learning task, where m is the number of samples for the target task. Besides the main result, we make several new contributions, including giving optimistic rates for excess risk of source tasks (Multi-Task Learning (MTL)), a local Rademacher complexity theorem for MTRL and MTL, as well as a chain rule for local Rademacher complexity for composite predictor classes.

1 Introduction

Transfer learning has emerged as a powerful tool in modern machine learning. The goal is to find a good predictor on a given target task with little data, by extracting knowledge from source task(s). The source tasks can either be explicitly given (supervised learning), or constructed from domain knowledge (unsupervised/self-supervised learning). An example of the supervised learning approach is a widely successful practice in deep learning, wherein a complex model is first trained on a source task with plentiful data, then applied to a target task by extracting representations of data by freezing the last layer of the complex model, and training a simple (linear) classifier on top of it [Bengio et al., 2013, Donahue et al., 2014]. Similarly, the unsupervised approaches of training a (large) neural network on a general objective has seen tremendous success, especially in natural language processing [Brown et al., 2020, Devlin et al., 2019] and computer vision. As before, a simple linear classifier trained on the output of the penultimate layer works well on a target task.

Representation learning , where the goal is to learn representations of data that are useful across multiple domains and tasks, is a common theme in the examples above. We study a popular formalization of this type of machine learning called Multi-Task Representation Learning (MTRL) . As the name suggests, we are given t source tasks each with a small number of data points, say n . We seek to effectively pool the nt data points to extract a common good representation useful for the target task. Therefore, if our tasks are sufficiently 'diverse', we hope to effectively circumvent our data scarcity by sharing knowledge between tasks. As a by-product, since the shared representation

Thanh Nguyen-Tang

can also be used for the source tasks themselves, this procedure is also effective for the related problem of multi-task learning (MTL) .

To study the transfer of knowledge via representations within the MTRL framework, we consider a composite-learning model . This model consists of F and H , a class of hypotheses and representations respectively, where the final end-to-end predictors are given via the composition f ◦ h , for all f ∈ F , h ∈ H . Since we must decide between compositions, we use a two-stage empirical risk minimization procedure. First, the multi-task representation learning stage , the procedure selects one common representation and t predictors, one for each task, by minimizing the average empirical loss on the source tasks. Second, the transfer learning stage , the procedure selects a predictor for the target task that minimizes the empirical loss, w.r.t. m samples, on top of the fixed representation from stage one.

We desire to learn a good predictor - i.e., one that has small excess risk - from F ◦ H for the target task. In this work, we consider the class H to be complex (e.g., deep neural networks), whereas the class F consists of simple functions (e.g., linear predictors). Therefore, the class F ◦ H is complex, so directly learning it would require a large number of samples. Yet, if we can effectively pool all our data from each task, the total number of examples could be sufficiently plentiful to successfully learn a complex representation. Further, a good representation can significantly reduce the statistical burden of learning a good predictor for the target task. This phenomenon is formalized by the prior work of Tripuraneni et al. [2020], which is the work most related to ours. Specifically, they show that if the loss function is Lipschitz and the tasks are sufficiently diverse (see Definition 2), the excess (transfer) risk for the target task is bounded as,

where C ( H ) and C ( F ) denote the complexity measures associated with learning H and F respectively (precise definition to follow in a later section).

While the above rate has the desirable properties described above, it is insufficient on many fronts, which our work seeks to address. First, it is akin to the standard rate of n -1 / 2 , albeit optimal in agnostic settings, for single-task PAC learning. However, it is well-known that a fast rate is achievable under realizability in the standard supervised learning. But, the above guarantee does not capture this possibility. Importantly, the work of Tripuraneni et al. [2020] operates under the assumption that the loss function is Lipschitz - in such settings, it is generally not possible to get fast rates, despite realizability, even for the special case of learning half-spaces [Srebro et al., 2010]. In the single task setting, the literature contains a rich theory of guarantees that interpolate between the slow and fast rate depending on the level of realizability, such guarantees are called 'optimistic rates'. We derive such optimistic rates for MTRL. Consequently, as an important application of our general theory, we show that the (transfer) risk of the MTRL approach on the target task is bounded as,

when using non-negative smooth losses (such as squared loss in linear regression or smoothed ramp loss for classification [Cortes et al., 2021, Srebro et al., 2010]). The values L ∗ source and L ∗ target represent the minimum average risk over the source tasks and target task respectively, hence (1) yields a fast rate under mild realizability, i.e. when they are small.

In the course of proving the above, we extend a number of foundational results in the single-task setting to the multi-task setting. We list them in our contributions below.

We give a general local Rademacher complexity result for transfer learning via multitask representation learning - see Theorem 1.

We also provide a local Rademacher complexity result for multitask learning - see Theorem 3. Our result has two benefits compared to prior work [Yousefi et al., 2018]. First, we perform MTL via MTRL that leads to improved bounds which show the benefit of pooling data as opposed to learning each task separately. Second, we operate under the special, yet fairly natural setting, of hypothesis classes with product-space structure - this simplifies the proof significantly as well as recovers the result for single-task setting with the exact same constants (unlike rates in Yousefi et al. [2018]).

For non-negative smooth losses, we derive optimistic rates (such as Eqn. (1)) for transfer learning and MTL via MTRL. Notably, when we restrict our rates to the single task setting, our proof is significantly simpler and our obtained bound on local Rademacher complexity is smaller compared to the prior work of Srebro et al. [2010].

Finally, we present a chain rule for local Rademacher complexity for composite settings, which allows us to decouple the (local) complexities of representation and predictor classes.

1.1 Our techniques

In this section, we expand on our contributions and provide an overview of techniques and challenges overcome in the context of prior art.

Local Rademacher complexity for transfer learning via MTRL. We build on the work of Tripuraneni et al. [2020], which introduced the notion of task diversity. They showed that task diversity leads to improved rates on excess transfer risk via MTRL, compared to training only on the target task. Their work however yields the standard O ( m -1 / 2 ) rate on excess risk, akin to what is obtained in agnostic settings. We provide a local Rademacher complexity result for transfer learning via MTRL, yielding optimistic rates depending on distributional properties. Our methods are based on tools and techniques from the seminal local Rademacher complexity paper of Bartlett et al. [2005]. A crucial component of our results is relative deviation Bennett concentration inequalities established via log-Sobolev inequalities together with the entropy method [Bousquet, 2002]. We make the crucial observation that the above concentration inequalities, though established for the single-task setting, are general enough to be used to extend the local Rademacher complexity framework to the multi-task and transfer learning setting.

Local Rademacher complexity for multi-task learning via MTRL. Intermediate to our above contribution, we provide a local Rademacher complexity result for MTL via MTRL, yielding optimistic rates. We note that the prior work of Yousefi et al. [2018] already provided a local Rademacher complexity result for MTL, though not via MTRL; importantly, our work establishes the benefits of pooling data compared to learning each task separately. However, in contrast to Yousefi et al. [2018] 1 , our setting is limited to hypothesis classes with a product space structure rather than a general vector-valued hypothesis. This simply means that the tasks are permutation-invariant in our setting, which we argue is a fairly reasonable assumption in the multi-task context. This assumption makes the analysis much simpler as we can reuse concentration inequalities for single task setting whereas Yousefi et al. [2018] had to extend log-Sobolev concentration inequalities to the multi-task setting. Additionally, this yields bounds with better constants than Yousefi et al. [2018] - in the special case of a single task, this recovers the known result with the exact same constants.

Optimistic rates for non-negative smooth losses. An important application of our theorems is providing optimistic rates when using non-negative smooth losses, which is known to enable distribution-free optimistic rates in a single task setting [Cortes et al., 2021, Srebro et al., 2010]. This result shows the provable benefits of representation learning. Further, it can be combined with the chain rule for Gaussian complexity of Tripuraneni et al. [2020]. This allows us to separate the composite class F ◦ H into the complexities of representation class H and predictor class F and also reuse existing complexity bounds in the literature. While we borrow some of the tools for optimistic rates theory in the single task setting from Srebro et al. [2010], the prior proof technique does not directly extend to the multi-task setting. Consequently, we modify the proof, simplifying it significantly as well as getting an improved bound, even in the single task setting. In particular, Srebro et al. [2010] bounds the local Rademacher complexity of the loss class by an ℓ 2 -covering number via Dudley's entropy integral formula, which is then bound by ℓ ∞ -covering number of hypothesis class using smoothness, which in turn is bound by the fat-shattering dimension, which eventually is bound by the (global) Rademacher complexity of the hypothesis class. Some of the steps seemingly do not generalize in the multi-task setting because there are no (well-studied) multi-task analogues, e.g. the fat-shattering dimension. Our proof starts with the above bound on the local Rademacher complexity of the loss class by an ℓ 2 -covering number via Dudley's entropy integral formula but then applies the well-known Sudakov minoration inequality that yields a bound in terms of Gaussian width; this is in turn bound by Rademacher complexity by standard known relations between the two quantities.

Chain rule for local Rademacher complexities. In the application to non-negative smooth losses, the local Rademacher complexity of the loss class is bounded by a non-trivial function of the

(global) Gaussian complexity of the hypothesis class which is further bounded using the chain rule of Tripuraneni et al. [2020]. For general loss functions, there may not be any such non-trivial function. To this end, for Lipschitz losses, we develop a chain rule of local Rademacher complexity, Theorem 5, that similarly allows separating the local complexities of the representation and predictor class.

1.2 Related work

As mentioned before, our work is most related to, and builds on, Tripuraneni et al. [2020]. Other related work includes an early work of Baxter [2000] that gives guarantees on excess risk of transfer learning and multi-task learning via MTRL under a certain generative model for tasks. This was subsequently improved by Maurer et al. [2016], Pontil and Maurer [2013]. These early works give rates of the form O ( t -1 / 2 + m -1 / 2 ) and do not capture the advantage of having a large number of source samples per task. Prior work has extensively studied the special case of linear representations and/or linear predictors, partly in the context of meta learning, for instance see Du et al. [2021], Tripuraneni et al. [2021], Xu and Tewari [2021]. Further, multitask learning has been explored under various notions of task relatedness [Ben-David and Borbely, 2008, Cavallanti et al., 2010].

Regarding optimistic rates, the seminal work of Vapnik and Cervonenkis [1971] provided the first optimistic rates, via a normalized uniform convergence analysis, but was limited to the PAC learning setting. Much later, a line of work of Bousquet et al. [2002], Koltchinskii and Panchenko [2000] derived optimistic rates in more general settings using the technique of localization . In this area, an important result is the local Rademacher complexity theorem of Bartlett et al. [2005], which many works have leveraged to yield improved rates for various problems [Blanchard et al., 2007, Cortes et al., 2013, Ullah et al., 2018]. Importantly, this theorem applies to learning with non-negative smooth losses [Srebro et al., 2010] where the local Rademacher complexity-based technique can be used to give distribution-free optimistic rates. Further, optimistic rates have seen renewed interest owing to the interpolation/benign overfitting phenomenon in deep learning. In certain problems, predictors may interpolate the data (zero training error), yet have small risk. A line of work [Zhou et al., 2020, 2021] shows that in certain linear regression problems, optimistic rates can be used to derive risk bounds that explain the phenomenon, where the usual tool of uniform convergence provably fails.

There has been some work on optimistic rates in the multi-task setting. The work of Yousefi et al. [2018] established a local Rademacher complexity theorem for multi-task learning (see Section 'Our techniques' for detailed comparison). Besides that, the work of Reeve and Kaban [2020] proved optimistic rates for vector-valued hypothesis classes for self-bounding loss functions (which includes non-negative smooth losses).

Notation. Denote ∥·∥ 2 be the Euclidean norm and ∥·∥ ∞ be the infinity norm. We use the convention that [ m ] = [1 , . . . , m ] is the list of contiguous integers starting at 1 and ending at m . For f = ( f 1 , . . . , f t ) let f 2 = ( f 2 1 , . . . , f 2 t ) and denote

2 Problem setup and preliminaries

Let X ⊆ R d and Y ⊆ R denote the input feature space and the output label space respectively. The source tasks and the target task are represented using probability distributions { P j } t j =1 and P 0 , respectively. Let F and F 0 be a class of predictor functions from R k to Y for the source tasks and target task respectively, and H a class of representation functions from R d to R k .

Following Tripuraneni et al. [2020], we assume that marginal distribution of P j over X is the same for all the j from 0 to t , and that there exists a common representation h ∗ ∈ H and task-specific predictors f ∗ j ∈ F for j ∈ [ t ] and f 0 ∈ F 0 such that P j can be decomposed as P j ( x, y ) = P x ( x ) P y | x ( y | f ∗ j ◦ h ∗ ( x )) . Note that this decomposition does not assume that f ∗ j ◦ h ∗ is the optimal in-class predictor; however, we will assume it is to make our optimistic rates more meaningful (although this is not strictly necessary). Also the decomposition implicitly assumes that y depends on x only via f ∗ j ◦ h ∗ ( x ) , and thus any additional noise in y is independent of x . Note that like Tripuraneni et al. [2020], we allow the predictor class for the target task F 0 to be different than on the source tasks F . For example, when performing logistic regression with a linear representation

B ∈ R k × d and linear predictor w ∈ R k , we have P y | x ( y = 1 | f ∗ j ◦ h ∗ ( x )) = σ ( w T B T x ) , where σ is the sigmoid function.

Given a loss function ℓ : R ×Y → R , the goal is to find a good predictor ˆ f ◦ ˆ h from the composite-class F 0 ◦ H with small transfer risk,

Yet, as standard, we do not have access to the joint distribution for all tasks, but instead n independent and identically distributed (i.i.d.) samples from each source task and m i.i.d. samples for the target task. Let ( x i j , y i j ) be the i th sample for the j th task. To accomplish the above, we use the source data to find a representation, and the target data to find a predictor for the target task. Specifically, in this MTRL procedure, first, we train on all source task data, then freeze the resulting representation. Second, we train over this frozen representation to find a good predictor on the target samples.

We formalize the above as the following two-stage Empirical Risk Minimization (ERM) procedure,

Besides bounding the transfer risk, we seek to understand if the above procedure also yields improved bounds for the source tasks (the problem of MTL).

We make the following regularity assumptions.

Assumption 1.

A: The loss function ℓ is nonnegative and b -bounded, i.e., 0 ≤ ℓ ( y ′ , y ) ≤ b < ∞ for all y ′ , y ∈ Y .

B: All functions in F are L -Lipschitz for some 0 < L < ∞ w.r.t. ∥·∥ 2 , i.e., ∥ f ( z 1 ) -f ( z 2 ) ∥ 2 ≤ L ∥ z 1 -z 2 ∥ 2 for all z 1 , z 2 ∈ Dom ( f ) and f ∈ F .

C: Any f ◦ h ∈ F ◦ H is D -bounded over X w.r.t. ∥ · ∥ ∞ , i.e., sup x ∈X | f ◦ h ( x ) | ≤ D < ∞ for all f ∈ F and h ∈ H .

Unlike Tripuraneni et al. [2020], we assume for some of our theorems that our function is smooth instead of Lipschitz, as leveraging smoothness is key to achieving fast rates [Srebro et al., 2010].

Definition 1 ( H -smoothness) . The loss function ℓ : R × Y → R is H -smooth if ∣ ∣ ℓ ′ y ( y 1 , y ) -ℓ ′ y ( y 2 , y ) ∣ ∣ ≤ H | y 1 -y 2 | for all y 1 , y 2 ∈ R , y ∈ Y .

Task diversity [Tripuraneni et al., 2020]. We benefit from learning a representation for a new target task, so long as there must be information encoded for that task by the source tasks. A typical way to quantify this encoded information is to control the difference between the learned representation and the underlying representation. To this end, we use the framework introduced by Tripuraneni et al. [2020] that quantifies a measure of task diversity. As mentioned in their work, this framework recovers task diversity assumptions in Du et al. [2021] exactly. In our setting, the definitions given in Tripuraneni et al. [2020] simplify considerably.

Definition 2. The tasks { P j } t i =1 are ( ν, ϵ ) -diverse over P 0 , if for the corresponding f ∗ ∈ F ⊗ t , f ∗ 0 ∈ F 0 and representation h ∗ ∈ H , we have that for all h ′ ∈ H

Parameters ν and ε quantify the similarity between learning the source tasks and the target task. For a detailed analysis of these parameters and the framework introduced by Tripuraneni et al. [2020], see Appendix B.

Local Rademacher Complexity. We now define the measures of complexity that are used in our work. Let u, p, n ∈ N , an input space Z , a class of vector-valued functions Q : Z → R u , and a dataset Z = ( z i j ) j ∈ [ p ] ,i ∈ [ n ] where z i j ∈ Z . Define the data-dependent Rademacher width, ˜ R Z ( · ) , as

where σ i,j,k are i.i.d. Rademacher random variables. Analogously, define the data-dependent Gaussian width, ˜ G Z ( · ) , as

where g i,j,k are i.i.d. N (0 , 1) random variables. We define the worst-case Rademacher width as ˜ R n ( Q ⊗ p ) = sup Z ∈Z pn ˜ R Z ( Q ⊗ p ) and, analogously, the worst-case Gaussian width as ˜ G n ( Q ⊗ p ) = sup Z ∈Z pn ˜ G Z ( Q ⊗ p ) . The above definitions generalize the standard Rademacher and Gaussian width for real-valued functions and can be derived as a special case of the more general set-based definitions [Wainwright, 2019] (see Appendix A).

We now define local Rademacher width as ˜ R Z ( Q ⊗ p , r ) = ˜ R Z ( { q ∈ Q ⊗ p : V ( q ) ≤ r } ) , where V : Q p → R . That is, the local Rademacher width is simply the Rademacher width restricted by a functional. In our applications, we consider any V satisfying V ( q ) ≤ b 1 np ∑ i,j,k =1 ( q j ( z i j )) k , where b is the uniform bound on the range of q in ℓ 2 -norm. Note this recovers the classical local Rademacher width 2 for real-valued functions and non-product spaces ( p = u = 1 ). We are mostly interested in the local Rademacher width of the loss applied to the pairwise composition between our vector-valued hypothesis class and our representation class. Specifically, if we define such a class as L ℓ ( F ⊗ t ( H )) = { ( ℓ ◦ f 1 ◦ h, . . . , ℓ ◦ f t ◦ h ) | h ∈ H , f ∈ F ⊗ t } , then we seek to bound ˜ R n ( L ℓ ( F ⊗ t ( H )) , r ) . Crucially, our bound will be a sub-root function in r , where we define this type of function below.

Definition 3 (Sub-root Function) . A function ψ : [0 , ∞ ) → [0 , ∞ ) is sub-root if it is nonnegative, nondecreasing, and if r ↦→ ψ ( r ) / √ r is nonincreasing for r > 0 .

Sub-root functions have the desirable properties of always being continuous and having a unique fixed point, i.e., r ∗ = ψ ( r ∗ ) is only satisfied by some r ∗ ; see Lemma 8 in Appendix A.

3 Main Results

In this section, we detail our local Rademacher complexity results for transfer learning (TL) and MTL via MTRL, a bound on the local Rademacher complexity of smooth bounded non-negative losses, and the application of this bound to both TL and MTRL.

Our first result is a local Rademacher complexity result for TL via MTRL.

Theorem 1. Let ˆ h and ˆ f 0 be the learned representation and target predictor, as described in Eqns. (2) and (3) . Under Assumption 1.A, if f ∗ is ( ν, ϵ ) -diverse over F 0 w.r.t. h ∗ and let ψ 1 and ψ 2 be sub-root functions such that ψ 1 ( r ) ≥ R n ( ℓ ◦ F ⊗ t ◦ H , r ) and ψ 2 ( r ) ≥ R m ( ℓ ◦ F 0 , r ) , then with probability at least 1 -4 e -δ , the transfer learning risk is upper-bounded by

where r ∗ 1 and r ∗ 2 are the fixed points of ψ 1 ( r ) and ψ 2 ( r ) , respectively, and c 1 and c 2 are absolute constants. 3

Inequality (4) bounds the transfer risk in terms of task diversity parameters ν and ϵ ; local Rademacher complexity parameters, fixed points r ∗ 1 and r ∗ 1 ; the minimum risk for the target task; and the minimum average risk for the source tasks.

As discussed in Xu and Tewari [2021], there exist settings where the task diversity parameters ν and ϵ are favorable, i.e., ϵ = 0 and ν = Θ(1) , so we will disregard them in the subsequent discussion. The bound is an optimistic rate because it interpolates between √ r ∗ 1 + √ r ∗ 2 and r ∗ 1 + r ∗ 2 , depending on both R target ( f ∗ 0 , h ∗ ) and R source ( f ∗ , h ∗ ) .

The fixed point r ∗ 1 is a function of F ⊗ t ( H ) and encodes its complexity measured with respect to the number of samples for the source tasks (among other class-specific parameters). The same reasoning holds for r ∗ 2 w.r.t. F 0 .

We consider H to be complex and both F and F 0 to be simple . So, if we instead learned the target task directly, then we would pay the complexity of F 0 ( H ) against m samples. In contrast, in our bound, we only pay the complexity of F 0 for m samples and the complexity of F ⊗ t ( H ) for nt samples. Since in many settings nt is much larger than m , we are successfully leveraging our representation to learn the target task.

The dependence on minimum risk is useful in various settings. An instructive regime is when nt ≫ m with R target ( f ∗ 0 , h ∗ ) = 0 and R source ( f ∗ , h ∗ ) > 0 ; in this case, we achieve a bound of √ r ∗ 1 + r ∗ 2 . This demonstrates that data-abundant environments can be leveraged to learn representations that work well in small sample regimes.

The next result uses a decoupling of function classes, the Gaussian chain rule of Tripuraneni et al. [2020], and smoothness to bound r ∗ 1 and r ∗ 2 . We see that under this additional assumption that ℓ is H -smooth, r ∗ 1 and r ∗ 2 decay quickly.

Theorem 2. Under the setting of Theorem 1 along with ℓ being H -smooth and Assumptions 1.B and 1.C, with probability at least 1 -6 e -δ , the fixed points in Inequality (4) ,

where c 3 and c 4 are absolute constants.

Remark 1. Note that in Theorem 2 it is not necessary to apply the chain rule. If the chain rule is not used, Assumptions 1.B and 1.C is not needed. The fixed point r ∗ 2 would be bounded as a function of F ⊗ t ( H ) . In the following, we use the above decomposed form for interpretability.

Note that for constant L, D, H, b, and δ , the terms on the right are always fast, so the rate of decay depends crucially on the behavior of the Gaussian widths for each class. For many function classes √ C ( H ) √ C ( F )

used in machine learning, it is common that G nt ( H ) ≈ nt and G n ( F ) ≈ n , where C ( · ) is some notion of complexity of the function class, which could be, for instance, the VC dimension, pseudo-dimension, fat-shattering dimension, etc. Therefore, it is common that

Recall, from our discussion following Theorem 1, that these fixed points control the order of the rate. Therefore, this theorem interpolates between a rate of 1 / √ m +1 / √ nt and 1 /m +1 /nt , where the fast rate is achieved in a realizable setting.

Linear classes. As an example let us consider a linear projection onto a lower dimensional space along with linear regressors and squared loss. Let X = R d , Y = R , and

If we assume that P x is sub-Gaussian, then standard arguments (see Tripuraneni et al. [2020]) show that ˜ G 2 m ( F 0 ◦ ˆ h ) ≤ O ( k m ) , ˜ G 2 n ( F ) ≤ O ( k n ) , and ˜ G 2 nt ( H ) ≤ O ( k 2 d nt ) . To simplify the bounds let L = 1 , H = 1 , D = 1 , b = 1 , and δ = 0 . 05 . Thus, under the assumptions of Theorem 2, the fixed points are bounded by

r ∗ 1 ≲ ( k m + 1 m ) , and r ∗ 2 ≲ ( k 2 d nt + k n + 1 nt ) , which gives R target ( ˆ f 0 , ˆ h ) -R target ( f ∗ 0 , h ∗ ) ≲ ( √ R target ( f ∗ 0 , h ∗ ) (√ k m ) + k m ) + 1 ν (( √ R source ( f ∗ , h ∗ ) (√ k 2 d nt + k n ) + k 2 d nt + k n )) + ε.

The following result is such a bound on the local Rademacher complexity for a non-negative H -smooth loss and any function class H .

Proposition 1 (Smooth non-negative local Rademacher complexity bound) . Under the setting of Theorem 1 along with ℓ being H -smooth and Assumptions 1.B and 1.C, there exists an absolute constants c 3 and c 4 such that

When we specialize the above bound to the standard single-task non-compositional setting and compare with Srebro et al. [2010], then

The second inequality uses the relation ˜ G n ( F ) ≤ 2 √ log( n ) ˜ R n ( F ) [Wainwright, 2019, p. 155] and that x ↦→ x log eC/x is increasing in x until x = C . The third inequality uses Khintchine's inequality, ˜ R n ( F ) ≥ √ 2 B √ n , where B is the ℓ 2 -bound on the range of F . Thus, the upper bound above is always better than the bound in Srebro et al. [2010] 4 for large enough n . Furthermore, under the additional though well-studied assumption that ℓ (0) ≤ HB 2 [Arora et al., 2022, Shamir, 2015], ours is better for n = Ω(1) . Finally, the Gaussian and Rademacher widths are of the same order for the class of linear predictors, bounded in norm, by a strongly convex regularizer, in general non-Euclidean settings. Hence, in this setting, our bound is smaller by a factor of √ log n in Kakade et al. [2008] see Appendix E for a detailed comparison.

3.1 Multi-Task Learning

Next, we detail some theorems foundational to the results above. For those results, we were able to link MTL with MTRL by using the task diversity assumption. Therefore, we naturally have the following local Rademacher result for MTL.

Theorem 3. Let ( ˆ f , ˆ h ) be an empirical risk minimizer of ˆ R source ( · , · ) as given in Eqn. (2) . Under Assumption 1.A, let ψ be a sub-root function such that ψ ( r ) ≥ R n ( F ⊗ t ( H ) , r ) with r ∗ the fixed point of ψ ( r ) , then with probability 1 -2 e -δ ,

where c is an absolute constant.

As above, we can use our bound on the local Rademacher complexity of a H -smooth loss class to get the following result. This result is similar to Srebro et al. [2010], Theorem 1, yet ours is in an MTL via MTRL setting.

Theorem 4. Under the setting of Theorem 3 along with ℓ being H -smooth and Assumptions 1.B and 1.C, with probability at least 1 -3 e -δ , r ∗ of ψ ( r ) is bounded by

where c is an absolute constant.

3.2 Local Rademacher complexity chain rule

Using the chain rule of Tripuraneni et al. [2021], as we did above, to decouple the complexities of the representation and prediction classes is desirable. Yet, our general local Rademacher complexity Theorems 1 and 3, as stated, do not seem to have this property. Consequently, we develop a local chain rule , which aims to separate the local complexities of learning the representation and predictor. The main result is the following.

Theorem 5. Suppose the loss function ℓ is L ℓ -Lipschitz. Define the restricted representation and predictor classes as follows,

where V is the functional in the local Rademacher complexity description. Under Assumptions 1.B and 1.C and that the worst-case Gaussian width of the above is bounded by the sub-root functions ψ F and ψ H , respectively, there exists an absolute constant c such that

4 Proof Techniques

In this section, we give details on some of the tools we use by giving proof sketches for the results in Section 3. In the next section, we cover some theorems at the root of all our results.

Proof sketch for Theorem 1. Let ˜ f 0 = arg min f ∈F R target ( f, ˆ h ) . By a risk decomposition, we can bound the excess transfer risk by R target ( ˆ f 0 , ˆ h ) -R target ( ˜ f 0 , ˆ h ) + sup f 0 ∈F 0 inf f ′ ∈F { R target ( f ′ , ˆ h ) -R target ( f 0 , h ∗ ) } . We can now use Theorem 3, with t = 1 , to bound R target ( ˆ f 0 , ˆ h ) . Note that R target ( f ∗ 0 , ˆ h ) -R target ( ˜ f 0 , ˆ h ) ≤ 0 . Now inf f ′ ∈F R target ( f ′ , ˆ h ) -R target ( f ∗ 0 , h ∗ ) is less than 1 ν (inf f ′ ∈F ⊗ t R source ( f ′ , ˆ h ) -R source ( f ∗ , h ∗ )+ ε, by the task-diversity assumption. As shown in Tripuraneni et al. [2020], we can bound inf f ′ ∈F ⊗ t { R source ( f ′ , ˆ h ) -R source ( f ∗ , h ∗ ) } with R source ( ˆ f , ˆ h ) . We can then apply Theorem 3 to the remaining R source ( ˆ f , ˆ h ) term. By leveraging task diversity again we can bound the remaining R target ( f ∗ 0 , ˆ h ) as a function of R target ( f ∗ 0 , h ∗ ) .

Proof sketch for Proposition 1. We achieve the following bound on the local Rademacher complexity of the loss class by using a truncated version of Dudley's integral [Srebro et al., 2010, see. Lemma A.3].

We observe that we can use smoothness to move the dependence on the loss to an appropriate scaling, which gives that N 2 ( L ℓ ( F ⊗ t ( H )) , r ) , ε, nt ) ≤ N 2 ( F ⊗ t ( H ) , ε/ √ 12 Hrnt,nt ) (Lemma 18 in the appendix). Next, we apply Sudakov minoration to bound the log covering number by the Gaussian width of our function class: √ log N 2 ( F ⊗ t ( H ) , ε/ √ 12 Hrnt,nt ) ≤ √ 12 Hrnt ϵ ˜ G n ( F ⊗ t ( H )) . With these simplifications, the parameter α from Dudley's integral can either be solved for exactly (as in Inequality 6), so long as α ≤ √ br , or set in such a way that it always holds (like in Inequality 5). Finally, we can optionally apply the Gaussian chain rule from Tripuraneni et al. [2020] to bound ˜ G n ( F ⊗ t ( H )) with L ˜ G nt ( H ) + ˜ G n ( F ) .

Proof sketch for Theorem 2. The right hand side of Inequality 5 is a sub-root function in r , because it is an affine transformation of √ r . Therefore, we just solve for the fixed point of this sub-root function, i.e., solve for r ∗ w.r.t. the equation below.

All that remains of the proof is basic algebraic simplifications. The proof for Inequality 6 is similar.

5 Discussion

The proofs of the above results hinge on the following local Rademacher complexity result.

Theorem 6. Let F be a class of non-negative b bounded functions. Assume that X 1 j , . . . , X n j are n draws from P j for j ∈ [ t ] and all nt samples are independent. Suppose that Var [ f ] ≤ V ( f ) ≤ BP f , where V is a functional from F ⊗ t to R and B > 0 .

Let ψ be a sub-root function with fixed point r ∗ such that B R ( F ⊗ t , r ) ≤ ψ ( r ) for all r ≥ r ∗ , then for K > 1 and δ > 1 , with probability at least 1 -e -δ ,

Yousefi et al. [2018], Theorem 9, showed a result of this form in a more general setting of Bernstein classes and a function class that is not necessarily a product space. In contrast, for simplicity, we do not consider Bernstein classes, though we believe such a generalization is possible, and our function classes are product spaces. By doing so, we achieve better constants than Yousefi et al. [2018] by using the next theorem.

Theorem 7 (Concentration with t functions) . Let F be a class of functions from X to R and assume that all functions f in F are P j -measurable for all j ∈ [ t ] , square-integrable, and satisfy E f = 0 . Let sup f ∈F ess sup f ≤ 1 and Z = sup f ∈F ⊗ t ∑ t j =1 ∑ n i =1 f j ( X i j ) . Let σ > 0 such that σ 2 ≥ sup f ∈F ⊗ t 1 t ∑ t j =1 Var [ f j ( X 1 j ) ] almost surely. Then, for all δ ≥ 0 , we have that

This result also holds for sup f ∈F ⊗ t ∣ ∣ ∣ ∑ t j =1 ∑ n i =1 f j ( X i j ) ∣ ∣ ∣ under the condition that sup f ∈F ∥ f ∥ ∞ ≤ 1 .

Yousefi et al. [2018] also proves something similar to the above from scratch using 'Logarithmic Sobolev' inequalities. We make the observation that within our setting a proof similar to the original proof given by Bousquet [2002] still holds. To clarify this further, Yousefi et al. [2018] state:

Note that the difference between the constants in (2) [Bousquet, 2002, Theorem 2.3] and (3) [Yousefi et al., 2018, Theorem 1] is due to the fact that we were unable to directly apply Bousquet's version of Talagrand's inequality (like it was done in Bartlett et al. [2005] for scalar-valued functions) to the class of vector-valued functions.

We show that within the product space structure, which we consider to be the most natural for MTL, it is possible to achieve the same constants. Concretely, when t = 1 , we realize the single function version of Bousquet [2002] exactly.

The proof of Theorem 6 is similar to the one given in Bartlett et al. [2005], Sec. 3, and generalized to MTL by Yousefi et al. [2018], Appendix B. Indeed, we follow the same steps within these proofs but use our inequality above.

6 Conclusion

We provide the first optimistic rates for transfer learning and multi-task learning via multi-task representation learning. This is achieved by establishing a local Rademacher complexity result for multi-task representation learning. We provide distribution-free optimistic rates for smooth nonnegative losses by bounding the local Rademacher complexity for multi-task loss classes. Besides our contributions to multi-task representation learning and multi-task learning, our work provides several foundational results and improved rates in the standard single-task setting. Directions for future work include: exploring adversarial robustness, active learning, and fine-tuning of the representation within multi-task representation learning.

References

Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Differentially private generalized linear models revisited. In Advances in Neural Information Processing Systems , volume 35, pages 22505-22517, 2022.

Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Annals of Statistics , 33:1497-1537, 2005.

Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research , 12: 149-198, 2000.

Witold Bednorz and Rafał Latała. On the boundedness of bernoulli processes. Annals of Mathematics , 180(3):1167-1203, 2014.

Shai Ben-David and Reba Schuller Borbely. A notion of task relatedness yielding provable multipletask learning guarantees. Machine learning , 73:273-287, 2008.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Gilles Blanchard, Olivier Bousquet, and Laurent Zwald. Statistical properties of kernel principal component analysis. Machine Learning , 66:259-294, 2007.

Olivier Bousquet. A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Mathematique , 334(6):495-500, 2002.

Olivier Bousquet. Concentration inequalities for sub-additive functions using the entropy method. In Stochastic inequalities and applications , pages 213-247. Springer, 2003.

Olivier Bousquet, Vladimir Koltchinskii, and Dmitriy Panchenko. Some local measures of complexity of convex hulls and generalization bounds. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 , pages 59-73. Springer, 2002.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research , 11:2901-2934, 2010.

Corinna Cortes, Marius Kloft, and Mehryar Mohri. Learning kernels using local rademacher complexity. Advances in neural information processing systems , 26, 2013.

Corinna Cortes, Mehryar Mohri, and Ananda Theertha Suresh. Relative deviation margin bounds. In International Conference on Machine Learning , pages 2122-2131. PMLR, 2021.

Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , pages 13089-13099, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019 , pages 4171-4186, 2019.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning , pages 647-655. PMLR, 2014.

Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In 9th International Conference on Learning Representations, ICLR 2021 , 2021.

Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems , 21, 2008.

Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based meta-learning methods. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , pages 5915-5926, 2019.

Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning. In High dimensional probability II , pages 443-457. Springer, 2000.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research , 17(81):1-32, 2016.

Massimiliano Pontil and Andreas Maurer. Excess risk bounds for multitask learning with trace norm regularization. In Conference on Learning Theory , pages 55-76. PMLR, 2013.

Henry Reeve and Ata Kaban. Optimistic bounds for multi-output learning. In International Conference on Machine Learning , pages 8030-8040. PMLR, 2020.

Ohad Shamir. The sample complexity of learning linear predictors with the squared loss. J. Mach. Learn. Res. , 16:3475-3486, 2015.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Optimistic rates for learning with a smooth loss. arXiv: Learning , 2010.

Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning , pages 10434-10443. PMLR, 2021.

Enayat Ullah, Poorya Mianjy, Teodor Vanislavov Marinov, and Raman Arora. Streaming kernel pca with ˜ o ( √ n ) random features. Advances in Neural Information Processing Systems , 31, 2018.

VN Vapnik and A Ya Cervonenkis. On uniform convergence of the frequencies of events to their probabilities. Teor. Veroyatnost. i Primenen , 16(2):264-279, 1971.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

Ziping Xu and Ambuj Tewari. Representation learning beyond linear prediction functions. Advances in Neural Information Processing Systems , 34:4792-4804, 2021.

Niloofar Yousefi, Yunwen Lei, Marius Kloft, Mansooreh Mollaghasemi, and Georgios C. Anagnostopoulos. Local rademacher complexity-based learning guarantees for multi-task learning. Journal of Machine Learning Research , 19(38):1-47, 2018.

Lijia Zhou, Danica J Sutherland, and Nati Srebro. On uniform convergence and low-norm interpolation learning. Advances in Neural Information Processing Systems , 33:6867-6877, 2020.

Lijia Zhou, Frederic Koehler, Danica J. Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. CoRR , abs/2112.04470, 2021.

Appendices

Contents

A Additional preliminaries on Gaussian processes and complexities

A.1 Preliminaries

We recall basic preliminaries on (sub) Gaussian processes [Vershynin, 2018, Wainwright, 2019]. Given a set Θ ⊆ R k , a random process is a collection of random variables { Z θ } θ ∈ Θ defined on the same probability space, indexed by elements of set Θ . The random process induces the following canonical psuedometric on the (abstract) set Θ , defined as d ( θ, θ ′ ) = ∥ Z θ -Z θ ′ ∥ L 2 =

√ E ( Z θ -Z θ ′ ) 2 , (pseudo)metrizing the set T . This allows us to define the covering and packing number of the set Θ with respect to the metric d , denoted as N (Θ , d, ϵ ) and M (Θ , d, ϵ ) respectively.

Definition 4 (Covering number) . An ϵ -cover of a set Θ with respect to a metric d , is a set C ⊆ Θ such that for any θ ∈ Θ , there exists a c ∈ C , such that d ( θ, c ) ≤ ϵ . The ϵ -covering number, N (Θ , d, ϵ ) is the cardinality of the smallest ϵ -cover.

̸

Definition 5 (Packing number) . An ϵ -packing of a set Θ with respect to a metric d , is a set P ⊆ Θ such that for all p, q ∈ P , p = q , d ( p, q ) > ϵ . The ϵ -packing number, M (Θ , d, ϵ ) is the cardinality of the largest ϵ -packing.

These are related as follows.

Lemma 1 (Wainwright [2019]) . For any totally-bounded set Θ and ϵ > 0 ,

A well-studied problem is to control the process uniformly, E [sup θ ∈ Θ Z θ ] in terms of the geometric properties of the set Θ .

A random process is a Gaussian process, if for all finite sets Θ 0 ⊆ Θ , the distribution of the random vector { Z θ } θ ∈ Θ 0 is Gaussian. A well-known result regarding controlling E [sup θ ∈ Θ Z θ ] is the Sudakov minoration inequality.

Lemma 2 (Sudakov minoration [Wainwright, 2019]) . Let { Z θ , θ ∈ Θ } be a zero-mean Gaussian process. Then

An important example is the so-called canonical Gaussian process , Z θ = ⟨ θ, g ⟩ , where g ∼ N (0 , I k ) is the standard Normal vector. In this case, the canonical metric simply becomes the ℓ 2 -norm, d ( θ, θ ′ ) = ∥ θ -θ ′ ∥ 2 . The uniform control of the random process becomes a standard object, known as Gaussian width of the set Θ .

The celebrated Fernique-Talagrand theory of majorizing measures establishes a characterization of the Gaussian width of a set in terms of its metric geometry - see Vershynin [2018] for more details.

A related concept is that of Gaussian complexity , where the goal is to control the process absolutely and uniformly, defined as,

These two are related as follows.

Lemma 3 (Vershynin [2018]) . The following are true about the relationship between Gaussian width and complexity.

An important category, generalizing Gaussian process, is that of sub-Gaussian processes, in which for all θ, θ ′ ∈ Θ , the random variable Z θ -Z θ ′ is d ( θ, θ ′ ) -sub-Gaussian,

The supremum of sub-Gaussian process is upper bound by Dudley's entropy integral formula, as follows.

Lemma 4 (Refined Dudley's entropy integral formula Wainwright [2019]) . Let { Z θ } θ ∈ Θ be a zero-mean sub-Gaussian process indexed on set Θ with diameter B , then

An important example of sub-Gaussian process is the so-called Bernoulli/Rademacher process, where Z θ = ⟨ σ, θ ⟩ , where σ ∈ {-1 , 1 } k , whose co-ordinates are i.i.d. Rademacher random variables. This leads to the notions of Rademacher width ˜ R (Θ) and Rademacher complexity, R (Θ) , defined as follows.

The Rademacher width and complexity are related in the same way as Gaussian width and complexity (akin to Lemma 3).

Lemma 5 (Vershynin [2018]) . The following is true about the relationship between Rademacher width and complexity.

Further, the Gaussian and Rademacher widths are related as follows.

Lemma 6 (Wainwright [2019]) . For a set Θ ⊂ R k ,

Akin to the Fernique-Talagrand majorizing measures theory for Gaussian processes, a complete characterization of the expected suprema for Bernoulli/Rademacher processes in terms of metric and co-ordinate geometry is obtained via (the now proved) Talagrand's Bernoulli conjecture Bednorz and Latała [2014], Ledoux and Talagrand [1991].

Complexity of real-valued function classes. Relatedly, another well-studied concept is Gaussian/Rademacher width/complexity of a set of real-valued functions on a fixed set of inputs. We first define it and then argue how this is simply a special case of the above "set"-view of these. Consider a class of functions Q which map an abstract set Z to R . Given input Z consisting of n points in the domain of q ∈ Q , the Gaussian/Rademacher complexity and width of Q , on Z are defined as follows.

where σ i and g i are i.i.d. Rademacher and standard Normal random variables, respectively. The metric induced on the set Q is the the L 2 distance with respect to the empirical measure on input Z ,

The corresponding covering and packing numbers of Q are denoted as N ( Q , ϵ, Z ) and M ( Q , ϵ, Z ) . To reduce the above from the set-definition, consider the set

In that case, R Z ( Q ) = R ( Q ( Z )) . Further, d Z ( q, q ′ ) = √ n d ( θ q , θ q ′ ) where θ q = 1 n ( q ( z 1 ) , q ( z 2 ) , . . . , q ( z n )) and θ q ′ = 1 n ( q ′ ( z 1 ) , q ′ ( z 2 ) , . . . , q ′ ( z n )) and thus lie in Q ( Z ) .

Complexity of function classes in the multi-task learning setting. We now generalize the above to functions encountered in multi-task representation learning. For input space Z , p, q, n ∈ N , consider a class of vector-valued functions Q : Z → R q , and a dataset Z = ( z i j ) j ∈ [ p ] ,i ∈ [ n ] , where z i j ∈ Z . For Q ⊗ p , the p -fold Cartesian product of Q , we define its data-dependent Rademacher width, ˜ R Z ( · ) , data-dependent Gaussian width, ˜ G Z , ( · ) , data-dependent Rademacher complexity, R Z ( · ) , data-dependent Gaussian complexity, G Z , with respect to input Z as,

where σ i,j,k and g i,j,k are i.i.d. Rademacher and standard Normal random variables, respectively. To reduce the above from the set-based definition, consider the set

In the above, the notation 1 np ((( q j ( z i j )) k ) j ) i denotes a vector in R npq obtained via enumerating i, j and k .

The corresponding metric analogously is,

where θ q = 1 np ((( q j ( z i j )) k ) j ) i and similarly θ q ′ , which lie in Q ⊗ p ( Z ) .

The above can be used to define covering and packing numbers of Q ⊗ p , denoted as N ( Q ⊗ p , ϵ, Z ) and M ( Q ⊗ p , ϵ, Z ) respectively.

The concepts of Dudley's entropy integral formula and Sudakov minoration thus generalize accordingly. A result of the following form, for real-valued functions, appears in Srebro et al. [2010].

Lemma 7 (Refined Dudley's entropy integral) . Consider a class of vector-valued functions Q and input set Z = ( z i j ) j ∈ [ p ] ,i ∈ [ n ] . Define B := sup q ∈Q ⊗ p ∥ q ∥ L 2 ( Z ) =

The equality above follows from the change of variables.

Worst-case complexities of functions. Often, we take the sup over datasets of a specified size, in the above definitions of Rademacher/ Gaussian width/complexity, yielding their worst-case counterparts, denoted as,

The corresponding metric similarly is

which can then be used to define covering and packing numbers of Q ⊗ p , denoted as N ( Q ⊗ , ϵ, np ) and M ( Q ⊗ , ϵ, np ) , respectively.

As before, Dudley's entropy integral formula and Sudakov minoration generalize analogously.

A.2 Sub-root functions and local Rademacher Complexity

We present a key property about sub-root functions from [Bartlett et al., 2005] below.

Lemma 8 (Lemma 3.2. [Bartlett et al., 2005]) . If ψ : [0 , ∞ ) → [0 , ∞ ) is a nontrivial sub-root function, then it is continuous on [0 , ∞ ) and the equation ψ ( r ) = r has a unique positive solution. Moreover, if we denote the solution by r ∗ , then for all r > 0 , r ≥ ψ ( r ) if and only if r ∗ ≤ r .

B Task diversity digression

In this section we make a series of observations regarding the task-diversity definition introduced by Tripuraneni et al. [2020]. Within this work, they introduce two definitions that are used to relate average source tasks performance to target task performance: task-averaged representation difference and worst-case representation difference.

Definition 6 (The task-averaged representation difference w.r.t h ∈ H ) .

Importantly, like the work which introduces these concepts, it is possible to statistically control ¯ d F , f ( ˆ h ; h ∗ ) , where ˆ h results from the first step of the two-tage ERM procedure, see the variational definition Eqn. (2).

The other quantity they introduce is the worst-case representation difference.

Definition 7 (The worst-case representation difference w.r.t h, h ′ ∈ H ) .

Finally, they introduce the following assumption on the two quantities defined above. It has a multiplicative parameter ν and additive parameter ε .

Definition 8 ( ( ν, ε ) -task diversity w.r.t h ∈ H [Tripuraneni et al., 2020]) . For a function class F , we say f ∈ F ⊗ t is ( ν, ϵ ) -diverse over F 0 for a representation h , if uniformly for all h ′ ∈ H ,

We will make a series of observations about these quantities which provide substantial simplification in our setting.

First, observe in the setting of product space,

and

Note, here we are again leveraging product space structure to simplify prior work. Recall in Section 5 we observed that this structure allows us to recover existing single function theorems exactly, unlike Yousefi et al. [2018]. Specifically in the proof of Theorem 9 we 'commute' suprema and summations.

Here we are 'commuting' infima and summations to simplify the framework specified in Tripuraneni et al. [2020] within our setting. Note they work with product spaces but do not make the simplifying observation above. Nonetheless, the definitions they provide are suitable for the more general function classes considered in Yousefi et al. [2018], which are not necessarily product spaces.

Now consider the setting detailed in the preliminaries applied to our approximation ˆ h, h ∗ ∈ H and f ∗ ∈ F ⊗ t with the following ratio

Note every sup and inf will realize a function that minimizes their respective risk.

This ratio is a measure of how similar the performance we can expect when we use our representation learned from the source tasks to and apply it the target task. That is if we seek to control this quantity we do not care per se if the risks are small nor do we care that we are getting good performance in comparison to the ground truth, but how comparable is our performance between source and target.

We seek to control this ratio with some ν

If the risk on our source tasks and target task is similar w.r.t. to the ground truth for all tasks then ν ≈ 1 . That is low (high) average risk on the source tasks leads to low (high) risk on the target task.

This is even more apparent in the realizable setting.

First, imagine a setting in which there are several tasks and on all but a few source tasks, we perform well.

Then 1 ν will be small, which is okay if we do not care about performance across all t tasks, as long as the target task performance is good.

Alternatively, imagine that the task does relatively poorly on the target task but is still small. This could lead to a very large 1 ν which is not desirable.

Therefore we allow for an additive ε to capture this quantity when we get risk on the source task is very small , but we are okay with relatively poor performance on the target task.

That is we care about the following inequality

where ε encodes how much tolerance we have between the performance on the source tasks and the target task. Hence, it is Inequality (10) which we care to control over all functions in H . Therefore, we will assume uniformly for all h ′ ∈ H that:

Although we observe this simplification, to be succinct we will still use that

and

and use this notation in the proofs of Appendix D.3.

C Technical Miscellanea

In this section, we present technical miscellanea that we shall later refer to in obtaining our proofs of the main results in Appendix D.

C.1 Martingale Concentration Inequality

For the sake of notational succinctness, we forgo the compositional model by overloading the notation to just consider a generic function class F : X → R .

Lemma 9 (Theorem 2.1 in Bousquet [2002]) . Let W 1 , . . . , W n be independent random variables in a Polish space W .

Let

be σ -algebras generated by these random variables.

We denote by E k [ · ] the expectation taken conditionally on A k . Let φ ( x ) = (1 + x ) log(1 + x ) -x and ψ ( x ) = e -x -1 + x .

Let ( Z, Z ′ 1 , . . . , Z ′ n ) be a sequence of A -measurable random variables and let ( Z k ) k ∈ [ n ] be a sequence of random variables that are A k measurable, respectively. Assume that there exists u > 0 such that for all k ∈ [ n ] the following inequalities are satisfied

Let σ ∈ R be a real number satisfying σ 2 ≥ 1 n ∑ n k =1 E k [ ( Z ′ k ) 2 ] almost surely and let v = (1 + u ) E [ Z ] + nσ 2 . If the following condition holds

which gives the following bounds for all δ > 0 ,

we obtain, for all λ ≥ 0 ,

The following corollary is immediate as independence is sufficient Bousquet [2003] and we can express the summation over nt random variables as 'double summation' where we first sum over q ∈ [ n ] and then over j ∈ [ t ] .

Corollary 1 (Lemma 9 in block structure) . Assume that W 1 j , . . . , W n j are n independent draws from P j for j ∈ [ t ] with random variables in some polish space, where all nt samples are independent.

̸

Let A be the sigma algebra generated by ( W i j ) j ∈ [ t ] ,i ∈ [ n ] and A q k be the sigma algebras generated by ( W i j ) j ∈ [ t ] ,i ∈ [ n ] ( j,i ) =( k,q ) . We denote by E q k [ · ] the expectation taken conditionally on A q k .

Let Z be a A -measurable r.v., ( ˜ Z q k ) k ∈ [ t ] ,q ∈ [ n ] be a sequence of A -measurable random variables, and ( Z q k ) k ∈ [ t ] ,q ∈ [ n ] be a sequence of random variables that are A q k measurable, respectively. Assume that there exists u > 0 such that for all k ∈ [ t ] and q ∈ [ n ] the following inequalities are satisfied

Let σ ∈ R be a real number satisfying

almost surely and let v = (1 + u ) E [ Z ] + nσ 2 . If the following condition holds

which gives the following bounds for all δ > 0 ,

C.2 Empirical Process Lemmas

The following result is standard symmetrization.

Lemma 10. Let F be a class of functions from X to R and let X denote the set of inputs { X i j } n,t i,j =1 . Assume that all functions f in F are P j -measurable for all j ∈ [ t ] . Then,

where { σ i j } is a sequence of nt independent Rademacher variables.

The result follows by standard symmetrization proof extended to the multi-function setting.

Proof. Let ˜ X 1 j , . . . , ˜ X n j be independent copies of X 1 j , . . . , X n j on probability space P j for all j ∈ [ t ] . Let E ˜ X be the expectation with respect to ˜ X 1 j , . . . , ˜ X n j for all j ∈ [ t ] and E X be the expectation with respect to X 1 j , . . . , X n j for all j ∈ [ t ] . Let σ i j be i.i.d. Rademacher random variables for all j ∈ [ t ]

we obtain, for all λ ≥ 0 ,

and i ∈ [ n ] . Fixing X 1 j , . . . , X n j for all j ∈ [ t ] , and using the copied variables we have that

By Jensen's inequality, we have

If ˜ X i j and X i j have the same distribution, we could exchange f j ( ˜ X i j ) -f j ( X i j ) for f j ( X i j ) -f j ( ˜ X i j ) and the equation would be equivalent. We can do this with Rademacher random variables, by randomly flipping the sign with probability 1/2. This yields the following

where the last inequality follows from triangle inequality.

The next result, a Gaussian chain rule given in Tripuraneni et al. [2020], is crucial to decoupling the complexity of the representation function class and the hypothesis class.

Theorem 8 (Theorem 7 from Tripuraneni et al. [2020]) . Let the function class F consist of functions that satisfy Assumption 1. Then the (empirical) Gaussian width of the function class F ⊗ t ( H ) satisfies,

where G ( F ⊗ t ( H )) = L ˜ G nt ( H ) + ˜ G n ( F ) . Further, if G ( F ⊗ t ( H )) ≤ D then by computing the exact infima of the expression,

Lemma 11 (Lemma A.4 in Bartlett et al. [2005]) . Let F be a class of real-valued functions with range in [ a, b ] . Given a set Z of nt i.i.d. points, with probability at least 1 -e -δ ,

C.3 Analytic Inequalities

Lemma 12. If x - √ xA -B = 0 for A,B > 0 then x ≤ A 2 +2 B .

Proof. As the discriminant of the quadratic in √ x is positive we are guaranteed two real roots x 1 , x 2 .

Lemma 13 (Lemma B.1. from Srebro et al. [2010]) . For any H -smooth non-negative function ℓ : R ↦→ R and any t, r ∈ R we have that

Lemma 14. For k ∈ [ t ] , q ∈ [ n ] ,

Proof. First note that for any k ∈ [ n ] ,

̸

This follows from applying reverse triangle inequality

̸

̸

Finally, we square both sides and rearrange the single summation into a double summation, which completes the proof.

Lemma 15. Consider the function f ( x ) = x + a ln ( b/x ) for a, b ≥ 0 .

The minimizer of the function over 0 < x ≤ b is x ∗ = min( a, b ) .

The corresponding minimum value is a ln ( eb min( a,b ) )

For any c such that a ≤ c ≤ b , a ln ( eb/a ) ≤ c ln ( eb/c ) .

Proof. The first and second claims follow from the fact that the function f decreases from x = 0 to a , and then increases. The third claim follows since the function x ↦→ x ln ( eb/x ) increases till x = b .

Lemma16. Let Z be a random variable taking values in R and let A ∈ R be such that P [ Z ≥ A ] > 0 , then sup | Z | ≥ ess sup | Z | = ∥ Z ∥ ∞ ≥ A .

Proof. The first inequality is standard. The second simply follows from the definition of ess sup :

̸

D Proofs of main results

Theorem 2.1 in Bousquet [2002], Lemma 9

Theorem 9

Multi-Function Bennett inequality

Theorem 10

Multi-Function Analog

of Theorem 2.1 in Bartlett et al. [2005]

Theorem 11

Lemma 19

Theorem 12

MTL

Theorem 16

Loss class

empirical LRC bound

Theorem 14

Target task transfer

risk bound

Theorem 13

task-averaged representation difference bound

Theorem 17

Smooth MTL

Lemma 20

Loss class LRC bound

Theorem 15

MTL via MTRL

Theorem 18

Smooth Loss MTL via MTRL

In this section, we restate the main theorems in our main paper (with precise constants) and give a detailed proof of our results. A graph in Appendix D shows the dependency between our theorems.

Note the relationship between the theorems in the main paper and the theorems in the appendix are as follows:

Theorem 1 becomes Theorem 15

Theorem 2 becomes Theorem 18

Proposition 1 becomes Theorem 16

Theorem 3 becomes Theorem 12

Theorem 4 becomes Theorem 17

Theorem 5 becomes Theorem 19

Theorem 6 becomes Theorem 11

Theorem 7 becomes Theorem 9

In Appendix D.1, we present Theorems 9 to 11 which are generalizations of prior work and also recover the single function setting. In particular, we give a concentration inequality and two risk bounds. In Section D.2, we present Theorem 12 our result for multitask learning with bounded non-negative loss classes. Next, Appendix D.3 contains Theorems 13 to 15 the MTL via MTRL learning setting.

The remaining theorems add the additional assumption of smoothness. We start, in section Appendix D.4, by giving Theorem 16, our bound on the empirical local Rademacher complexity of smooth bounded non-negative loss classes. However, in order to use the above result with our prior theorems we need to bound the local Rademacher complexity constrained on the risk to one

constrained by the empirical risk. So, in Appendix D.5, we introduce Lemmas 19 and 20 to make this conversion. In Appendix D.6 we conclude by invoking our bound on the local Rademacher complexity of smooth bounded non-negative loss classes within our MTL and MTRL theorems with Theorems 17 and 18.

D.1 Concentration Inequalities and Risk Bounds

In this section we forgo the compositional model and will consider a generic function class F .

Theorem 9 (Concentration with t functions) . Let F be a class of functions from X to R and assume that all functions f in F are P j -measurable for all j ∈ [ t ] , square-integrable, and satisfy E f = 0 . Let sup f ∈F ∥ f ∥ ∞ ≤ 1 and

Let σ > 0 such that σ 2 ≥ sup f ∈F ⊗ t 1 t t ∑ j =1 Var [ f j ( X 1 j ) ] almost surely, then for all δ ≥ 0 , we have

Proof of Theorem 9. We prove the result for Z as defined in Equation (19); the proof for the definition of X in Equation (20) is similar. Let

and let f (0) 1 , . . . , f (0) t be the functions that achieve the supremum w.r.t. Z .

or let sup f ∈F ess sup f ≤ 1 and

Next, for each k ∈ [ t ] and q ∈ [ n ] let

̸

The random variable Z q k is essentially removing a f k ( X q k ) term from Z . We define f ( k,q ) 1 , . . . , f ( k,q ) t as the functions which achieve the supremum w.r.t Z q k . That is,

̸

Finally define,

We now prove that the conditions in Corollary 1 are satisfied.

̸

̸

̸

where Inequality 21 follows from f (0) 1 , . . . , f (0) t being the functions which achieve the supremum w.r.t Z and Inequality 22 follows from f ( k,q ) 1 , . . . , f ( k,q ) t achieving the supremum w.r.t Z q k . In addition, we show that E q k [ ˜ Z q k ] is non-negative. Recall that E q k is conditioned on A q k .

Finally, each f ( k,q ) j is a constant w.r.t E q k besides when j = k and i = q so the above reduces to:

̸

̸

Also, we have

̸

̸

̸

Therefore, ∑ t j =1 ∑ n i =1 ( Z -Z q k ) ≤ Z . Finally,

̸

As the samples for the same task are being drawn from the same distribution,

Note that f ( k, 1) k is a fixed function w.r.t. X 1 k , therefore f ( k, 1) k is a fixed function as we integrate over X 1 k . So , E 1 k [ ( f ( k, 1) k ( X 1 k ) ) 2 ] ≤ sup f j ∈F E 1 k [ ( f j ( X 1 k ) ) 2 ] . Applying this reasoning to the t functions gives the desired bound.

Note, crucially, that the product space structure of our function class F ⊗ t allows us to move the suprema outside the summation. Dividing by nt gives the desired bound.

The right-hand side is bounded by σ 2 by hypothesis.

Finally, now we apply Corollary 1 as all the conditions are satisfied.

Theorem 10 (Analog of Theorem 2.1 in Bartlett et al. [2005]) . Let F be a class of functions that map X into [ a, b ] . Given an input X = { X i j } t,n i,j =1 , assume that there is some r > 0 such that for all f 1 , . . . , f t that 1 t ∑ t j =1 Var [ f j ( X 1 j ) ] ≤ r . Then, for every δ > 0 , with probability 1 -e -δ ,

Proof of Theorem 10. Let V + = sup f ∈F ⊗ t ( P f -ˆ P n f ) where ˆ P n f = 1 nt ∑ t j =1 ∑ n i =1 f j ( X i j ) and P f = E ˆ P n f . Let ˜ F = { f -E f ( b -a ) | f ∈ F } be a set of scaled functions. For each ˜ f ∈ ˜ F we have that E ˜ f = 0 and ∣ ∣ ∣ ˜ f ∣ ∣ ∣ ≤ 1 . In addition, for all ˜ f ∈ ˜ F ⊗ t we have that 1 t ∑ t j =1 Var [ ˜ f j ( X 1 j ) ] = 1 ( b -a ) 2 1 t ∑ t j =1 Var [ f j ( X 1 j ) ] ≤ r ( b -a ) 2 . Therefore, the conditions for Theorem 9 are satisfied.

Letting Z = sup ˜ f ∈ ˜ F ⊗ t ∑ t j =1 ∑ n i =1 ˜ f j ( X i j ) = nt ( b -a ) V + as in Equation (20), we thus have with probability 1 -e -δ :

where Inequality 23 is by subadditivity of the square root, Inequality 24 is by application of the AM-GM inequality, and Inequality 25 is by Lemma 10.

Theorem 11. Let F be a class non-negative b bounded functions. Assume that X 1 j , . . . , X n j are n draws from P j for j ∈ [ t ] and all nt samples are independent. Suppose that

where V is a functional from F ⊗ t to R and B > 0 . Given an input X = { X i j } t,n i,j =1 , let ψ be a sub-root function with fixed point r ∗ such that B E ˜ R X ( F ⊗ t , r ) ≤ ψ ( r ) for all r ≥ r ∗ . Then, the following holds: For K > 1 , α > 0 , and x > 1 , with probability at least 1 -e -x .

Remark 2 (Comparison to Yousefi et al. [2018]) . A Theorem of this form was shown by Yousefi et al. [2018], which generalized results from Bartlett et al. [2005] to vector valued functions. The proof of Theorem 11 follows the same steps as the proof of Theorem Yousefi et al. [2018]. Yet, we will use our Theorem 10 which has better constants than Theorem 1 within Yousefi et al. [2018] for when the class that the suprema is indexing over is has product space structure.

The following lemma, and its proof, is a modified version of work in [Yousefi et al., 2018, Lemma B.2 ], which is a generalized version of work within [Bartlett et al., 2005, Lemma 3.8.]. We remove the Bernstein class structure, make B > 0 , and only consider non-negative bounded functions 5 .

The theorem can be generalized to Bernstien classes like Theorem B.3 from Yousefi et al. [2018] with worse dependence on B .

Lemma 17. Let K > 1 , r > 0 and B > 0 . Suppose that for all f ∈ F ⊗ t we have V ( f ) ≤ BP f . Define the re-scaled version of F ⊗ t as

If V + r = sup f ′ ∈F ⊗ t r [ P f ′ -ˆ P n f ′ ] ≤ r BK

Proof of Lemma 17. Let f be any element in F ⊗ t .

Case 1. If V ( f ) ≤ r , then f ′ = f and the inequality V + r ≤ r BK leads to

If all component functions of f are positive then ˆ P n ( f ) is positive and as K/ ( K -1) > 1 for all K > 1 we have

Case 2. If V ( f ) ≥ r , then f ′ = r f /V ( f ) and the inequality V + r ≤ r BK yields

which completes the proof.

Proof of Theorem 11. We modify the proof within Yousefi et al. [2018], which generalises the proof within Bartlett et al. [2005] by extending it to multi-function and Bernstein classes. At a high level we apply the same steps as Yousefi et al. [2018], yet we use our concentration inequality and do not need to consider the more complicated inequality [Yousefi et al., 2018, Lemma B.1] as we do not have the additional Bernstein structure. The proof closely follows from Yousefi et al. [2018] with minor modifications as described and presented for completeness.

Let r ≥ r ∗ be a fixed real number.

Verifying Conditions of Theorem 10. We seek to use Theorem 10 on F ⊗ t r , see Equation (26), therefore we will validate the conditions of the theorem.

Fix f ′ ∈ F ⊗ t r . Thus, f ′ = r f / max( r, V ( f )) for some f ∈ F . If max( r, V ( f )) = r , then f ′ = r f / max( r, V ( f )) = f . Thus, Var [ f ′ ] = Var [ f ] ≤ V ( f ) ≤ r . If max( r, V ( f )) = V ( f ) , then Var [ f ′ ] = r 2 V 2 ( f ) Var [ f ] ≤ r 2 V ( f ) ≤ r . Therefore, 1 t sup f ′ ∈F ⊗ t r ∑ t j =1 E [ f ′ j ( X j ) ] 2 ≤ r . Finally, recall, functions in F are positive and b -bounded.

Thus, by Theorem 10, with probability at least 1 -e -δ and ∀ δ > 0

Solving for P ( f ) gives

Bounding R X ( F ⊗ t r ) . Nowwemust control the Rademacher complexity within the above inequality. This part of the proof is the same for Bartlett et al. [2005], Yousefi et al. [2018].

Observe that ˜ R X ( F ⊗ t r ) = E R X ( F ⊗ t r ) . By assumption we have that V ( f ) ≤ B ( P f ) ≤ Bb, ∀ f ∈ F ⊗ t . Fix some λ > 1 and define k as the smallest integer such that rλ k +1 ≥ Bb . By leveraging the principle that combining function classes on a uniform set of inputs is equivalent to the Minkowski sum of the created image, and utilizing the trait of Rademacher width breakdown under Minkowski sum as detailed by Vershynin [2018], we have,

We thus the following inequalities

As r ≥ r ∗ , we can bound the local Rademacher width with the sub-root function.

Now the sub-root property of ψ gives us that that ψ ( ξr ) ≤ ξ 1 2 ψ ( r ) for any ξ ≥ 1 and, hence,

Pick λ = 4 in the above inequality, which gives ˜ R X ( F ⊗ t r ) ≤ 5 ψ ( r ) /B . Also, r ∗ is the fixed point of ψ , we have for all r ≥ r ∗ that ψ ( r ) ≤ √ r/r ∗ ψ ( r ∗ ) = √ rr ∗ . Taken together, we have

Applying Lemma 17. The remainder of the proof is simple manipulations and applying Lemma 17. Here our proof deviates from [Yousefi et al., 2018]. Using our above bound on the Rademacher complexity gives for any r ≥ r ∗ and δ > 0 , we have with probability at least 1 -e -δ ,

Now, letting A =10(1 + α ) √ r ∗ /B + √ 2 δ/nt and C = b ( 1 α + 1 3 ) δ nt ,

Setting A √ r + C = r BK gives a quadratic, which has both roots bounded by ( ABK ) 2 +2 BKC by Lemma 12 now applying Lemma 17 we have that

Plugging in our values for A and C gives

Applying AM-GM inequality,

which completes the proof.

D.2 Multi-Task Learning

The following theorem is a vector-valued generalization of Theorem 5.2 within Bartlett et al. [2005] with several differences: we do not require δ n ≤ r ∗ , b = 1 , and we solve for constants.

Theorem 12. Let ( ˆ f , ˆ h ) be an empirical risk minimizer as given in Eqn. (2) . Let ψ ( r ) ≥ b E ˜ R X ( F ⊗ t ( H ) , r ) with r ∗ the fixed point of ψ ( r ) . Then, under Assumption 1.A with probability 1 -2 e -δ ,

Proof of Theorem 12. To simplify the notation within the proof let L ∗ = R source ( f ∗ , h ∗ ) .

We follow steps similar to the proof of Theorem 5.2 within Bartlett et al. [2005]. Assume f ∗ j = arg min f ∈F P j ℓ f exists. (As mentioned in Bartlett et al. [2005] if it does not exist we can consider a sequence converging to the infimum.) Note that { f ∗ j ( X i j ) } i ∈ [ n ] ,j ∈ [ t ] are nt independent r.v.. Note that

By Bernstein inequality, with probability 1 -e -δ

By Theorem 11, setting α = 1 11 and B = b , and union bound, with probability 1 -2 e -δ ,

By change of variables we consider K > 0 by adding 1 to K :

Applying AM-GM to √ 2 K √ L ∗ bδ nt ,

These are the terms without K :

Considering only those terms a function of K :

We set K = max ( √ L ∗ , √ bδ nt ) max ( √ r ∗ b , √ bδ nt ) . Terms with K in the numerator are upper-bounded by

Terms with K in the denominator are upper-bounded by

Expanding both expressions gives eight terms where we will choose an appropriate value to maximize the quantity:

.

Adding all of the right-hand side expressions together gives:

Adding the terms without K , Equation (28) gives:

Applying AM-GM again

Rounding up to the nearest integers:

D.3 Multi-Task Learning via Representation Learning

Theorem 13. Let ˆ h be an empirical risk minimizer as in (2). Let ψ ( r ) ≥ b E ˜ R Z ( ℓ ◦ F ⊗ t ◦ H , r ) with r ∗ the fixed point of ψ ( r ) . Then, if Assumption 1.A holds. Then, with probability at least 1 -e -δ ,

where recall that ¯ d F ,f ∗ is defined in Equation (12) .

Proof of Theorem 13.

where the last inequality is due to Theorem 12.

By using our hypothesis which rates the task-averaged representation distance to the worst case representation difference we can bound the excess transfer risk.

Theorem 14. Let ˆ f 0 be an empirical risk minimizer of ˆ R target ( · , ˆ h ) in Eqn. (3) for any representation ˆ h . Let ψ ( r ) ≥ b E ˜ R Z ( ℓ ◦ F 0 , r ) with r ∗ the fixed point of ψ ( r ) . Then if Assumption 1.A holds, with probability at least 1 -δ :

Proof of Theorem 14. Let ˜ f 0 = arg min f ∈F R target ( f, ˆ h ) . By adding and subtracting d F , F 0 ( ˆ h ; h ∗ ) then applying Theorem 12 to R target ( ˆ f 0 , ˆ h ) , we have convenient cancellation of R target ( ˜ f 0 , ˆ h ) . For notational succinctness let

Therefore we have:

Now note that R target ( ˜ f 0 , ˆ h ) = R target ( f ∗ 0 , h ∗ ) + ¯ d F 0 ,f ∗ 0 ( ˆ h, h ∗ ) . After applying subadditivity of the square root √ · and AM-GM we can bound ¯ d F 0 ,f ∗ 0 ( ˆ h, h ∗ ) with Theorem 13. This is concretely

demonstrated with the following chain of inequalities:

Finally, by collecting like terms, we have that R target ( ˆ f 0 , ˆ h ) is bounded by

Theorem 15. Let ˆ h and ˆ f 0 be the learned representation and target predictor, as described in Eqns. (2) and (3) . Let ψ 1 ( r ) ≥ b E ˜ R Z ( ℓ ◦F ⊗ t ◦H , r ) and ψ 2 ( r ) ≥ b E ˜ R Z ( ℓ ◦F 0 , r ) with r ∗ 1 and r ∗ 2 the fixed points of ψ 1 ( r ) and ψ 2 ( r ) , respectively. Then, under Assumption 1.A and that f ∗ is ( ν, ϵ ) -diverse over F 0 w.r.t. h ∗ , with probability at least 1 -4 e -δ , the transfer learning risk is upper-bounded by,

Proof of Theorem 15. The proof is by first applying Theorem 14, then ( ν, ε ) -diversity assumption, and finally Theorem 13. This is demonstrated with the following series of inequalities.

D.4 Empirical Local Rademacher Complexity Bound

The following lemma allows us to discard the composition of a loss so long as we appropriately scale the cover.

Lemma 18. For a non-negative H -smooth loss and any function class F , we have:

We follow the same first few steps as in Srebro et al. [2010] but then bound the maximum by summation to recover an empirical L 2 cover.

Proof of Lemma 18. By Lemma 13 we see that for a non-negative H -smooth function f , we have that ( f ( t ) -f ( r )) 2 ≤ 6 H ( f ( t )+ f ( r ))( t -r ) 2 . Using this inequality, for any sample X = { ( x i j , y i j ) } j ∈ [ t ] i ∈ [ n ] , of nt points, for an f ϵ , we have,

That is, a cover of { f ∈ F ⊗ t : P f ≤ r } at radius ϵ/ √ 12 Hrnt is also a cover of L ℓ ( F ⊗ , r ) at radius ϵ , and we can conclude that,

which completes the proof.

Theorem 16 (Smooth non-negative local Rademacher complexity bound) . Under the setting of Theorem 15 along with ℓ being H -smooth and Assumptions 1.B and 1.C, we have:

and if 16 G ( F ⊗ t ( H )) ≤ D and √ b ≥ 640 √ 3Π( F ⊗ t ( H )) , which holds for large enough samples,

where G ( F ⊗ t ( H )) = L ˜ G nt ( H ) + ˜ G n ( F ) and Π( F ⊗ t ( H )) = √ HG ( F ⊗ t ( H )) log ( eD 16 G ( F ⊗ t ( H )) ) .

Proof of Theorem 16. Let Z denote the dataset of nt points. We first compute sup g ∈F ⊗ t ( H ) sup Z ∥ f ∥ L 2 ( Z ) . Note that,

Therefore, by refined Dudley's entropy integral formula, we have

Now we apply the Gaussian chain rule Theorem 8 to ˜ G nt ( F ⊗ t ( H )) , giving us:

Setting δ = min(16 G ( F ⊗ t ( H )) , D ) and simplifying by bounding the minimum with 16 G ( F ⊗ t ( H )) we have:

Doing the same for δ with min ( √ br, 640 √ 3 G ( F ⊗ t ( H )) √ Hr log ( eD min(16 G ( F ⊗ t ( H )) ,D ) )) we have √ √

The above expression is optimal with respect to the parameters α and δ and clearly gives four different possibilities based on the two minima. Since, under a bounded setting, G ( F ⊗ t ( H ) would typically go down with nt , for large enough nt we have that and

Under these conditions, we have

Under the other three possibilities for the minima some or all the logarithmic factors are identically one and lead to a simplified bound.

Alternatively, for a more simple bound which always holds set α = √ br nt and δ = D √ nt to get

Remark 3 (Standard single function setting.) . Everything in the proof of Theorem 16 up to Equation (31) holds for a non-compositional model and where t = 1 which corresponds to the standard setting. In this context, there is no need to apply the Gaussian chain rule, Theorem 8. Let Q : X → R be a hypothesis class and note that we have n i.i.d. samples.

If we set α = min ( √ br, 10 √ 3 ˜ G n ( Q ) √ Hr ) then we have

and if we set α = √ br n

Using Lemma 5, we also get the following bounds for local Rademacher complexity (as opposed to width),

D.5 Theorems for transition from empirical to population

Lemma 19. Let F be a class of functions that map X into [0 , b ] with b > 0 . Fix α > 0 . For every δ > 0 and r that satisfy

we have with probability at least 1 -e -δ that

The proof is similar to the proof of Corollary 2.2 within Bartlett et al. [2005].

Proof of Lemma 19. For f ∈ { f ∈ F ⊗ t | P f ≤ r } we have Var [ f ] ≤ P f 2 ≤ bP f ≤ br . By Theorem 10, with probability 1 -e -δ , every f ∈ { f ∈ F ⊗ t | P f ≤ r } satisfies, with α = √ 2 ,

D.6 Smooth Learning Bounds

Lemma 20. Under the setting of Theorem 16 for any δ > 0 , the fixed point r ∗ is bounded by,

where c 1 < 1 . 5 × 10 8 and G ( F ⊗ t ( H )) = L ˜ G nt ( H ) + ˜ G n ( F ) . Further, if √ √

where Π( F ⊗ t ( H )) = √ HG ( F ⊗ t ( H )) log ( eD 16 G ( F ⊗ t ( H )) ) , then for any δ > 0 , the fixed the fixed point r ∗ is bounded by, c 2 G ( F ⊗ t ( H ) ) 2 Hb log ( De 16 G ( F ⊗ t ( H )) ) 2 log   2 √ 3 √ b 1920 G ( F ⊗ t ( H )) √ H log ( eD 16 G ( F ⊗ t ( H )) )   2 + 112 b 2 δ 3 nt with c 2 < 8 × 10 8 .

Proof of Lemma 20. Let

Now we have

Note that from Lemma 19, we have that, for any δ > 0 , with probability at least 1 -e -δ ,

Further, we can apply the concentration of Rademacher complexity, Lemma 11, to bound the expected value of the left-hand side as follows. With probability at least 1 -e -δ ,

Hence, with probability 1 -2 e -δ , we get,

We now apply Lemma 16 and upper bound the right-hand side by taking supremum over Z . This gives us,

Plugging the above in Eqn. (32), we get,

Suppose that the local Rademacher complexity is bounded by a multiple of √ r , say √ rE . Now bounding the local Rademacher complexity with Theorem 16, we have that,

where

Note that A √ r + D is sub-root and larger than γ ( r ) . Thus if we solve for a fixed point r ∗ of this expression we have A √ r ∗ + D = r ∗ . By Lemma 12 we have r ∗ ≤ A 2 +2 D .

Therefore, with α = 1 / 8 ,

is bounded by a sub-root function of r with the following fixed point:

We have seen from Theorem 16 that if 16 G ( F ⊗ t ( H )) ≤ D and √ b ≥ 640 √ 3Π( F ⊗ t ( H )) then we can set √ √

In this setting, using Equation (33), the fixed point is bounded by:

796262400 G ( F ⊗ t ( H ) ) 2 Hb log ( De 16 G ( F ⊗ t ( H )) ) 2 log   2 √ 3 √ b 1920 G ( F ⊗ t ( H )) √ H log ( eD 16 G ( F ⊗ t ( H )) )   2 + 112 b 2 δ 3 nt

Alternatively, we can always set E as follows:

therefore we also have that the fixed point is bounded as:

D.7 Smooth MTL and MTL via MTRL

When we add the additional assumption that the loss function is smooth we have the following extensions to Theorems 12 and 15

Theorem 17. Let ( ˆ f , ˆ h ) be an empirical risk minimizer as given in Eqn. (2) . Let ψ ( r ) ≥ b R n ( F ⊗ t ( H ) , r ) with r ∗ the fixed point of ψ ( r ) . Then, under Assumption 1 along with ℓ being H -smooth, with probability 1 -2 e -δ ,

with c < 1 . 5 × 10 8 and G ( F ⊗ t ( H )) = L ˜ G nt ( H ) + ˜ G n ( F ) .

Proof of Theorem 17. Use the fixed point bound from Lemma 20 that always holds and substitute it into Theorem 12.

Theorem 18. Let ˆ h and ˆ f 0 be the learned representation and target predictor, as described in Eqns. (2) and (3) . Let ψ 1 ( r ) ≥ b R n ( ℓ ◦ F ⊗ t ◦ H , r ) and ψ 2 ( r ) ≥ b R n ( ℓ ◦ F 0 , r ) with r ∗ 1 and r ∗ 2 the fixed points of ψ 1 ( r ) and ψ 2 ( r ) , respectively. Then, under Assumption 1 along with ℓ being H -smooth. and that f ∗ is ( ν, ϵ ) -diverse over F 0 w.r.t. h ∗ , with probability at least 1 -2 e -δ , the transfer learning risk is upper-bounded by,

where

where c < 1 . 5 × 10 8 and

Proof of Theorem 18. Use the the fixed point bound from Lemma 20 that always holds and substitute it into Theorem 15.

D.8 Local Rademacher complexity chain rule

Theorem 19. Suppose the loss function ℓ is L ℓ -Lipschitz. Define the restricted representation and predictor classes as follows,

where V is the functional in the local Rademacher complexity description. Under Assumptions 1.B and 1.C and that the worst-case Gaussian width of the above is bounded by the sub-root functions ψ F and ψ H , respectively, there exists an absolute constant c such that

Proof of Theorem 19. Define

Observe that for any ( ℓ ◦ f , h ) ∈ F ⊗ t X ( H ) , we have f ∈ F X ( r ) and h ∈ H X ( r ) . Hence,

Further, by assumption, the composed function ℓ ◦ f is L ℓ L -Lipschitz. We can now apply the (standard) chain rule of Tripuraneni et al. [2021] which gives us,

which completes the proof.

E Detailed comparison with prior works

In this section, we provide a more detailed comparison with prior works.

E.1 Comparison with Srebro et al. [2010]

Firstly, we identify some erroneous or missing though fixable steps in the proof of Srebro et al. [2010].

Dudley's integral formula. The work of Srebro et al. [2010] seems to interchange the concepts of Rademacher width and complexity with the same notation R n ( F ) . The original Dudley's formula and its truncated version proved by Srebro et al. [2010] is for Rademacher (or Gaussian) 'width', yet in their work, it is used to bound Rademacher complexity. It is possible to translate from width to complexity, but this will lead to an additional term of √ br n as detailed below. This additional term only changes their result by constant factors.

Lemma 21. Consider a class of functions F with range in [0 , b ] . Then given input Z of n points, R Z { f ∈ F : ˆ P n f ≤ r } ≤ 2 ˜ R Z { f ∈ F : ˆ P n f ≤ r } + √ br n .

Proof. The proof follows by an application of Lemma 5. In particular, given input Z of n points, for any f ∈ { f ∈ F : ˆ P n f ≤ r } , we have that

Plugging this in Lemma 5 gives the claimed bound.

Centering. Within Srebro et al. [2010] they consider loss with the 'bounded difference' property: ∀ ˆ y, ˆ y, y ′ we have that | ℓ (ˆ y, y ) -ℓ (ˆ y ′ , y ) | ≤ b . To bound the local Rademacher complexity of their loss class which is empirically constrained with ˆ P n ( ℓ ◦ f ) ≤ r , they use Dudley's integral. Note the upper limit of integration in Dudley's integral is √ ˆ P n ( ℓ ◦ f ) 2 and it is claimed that √ ˆ P n ( ℓ ◦ f ) 2 ≤ √ br because ˆ P n ( ℓ ◦ f ) 2 ≤ bP ( ℓ ◦ f ) ≤ br . Yet, this reasoning only follows for b -bounded losses not under this weaker condition of bounded difference. Yet, it is possible to center the process and perform a comparable analysis. To resolve this issue let ˜ f = arg inf f ∈F ˆ P n ( ℓ ◦ f ) and consider ( ℓ ◦ f ) -( ℓ ◦ ˜ f ) . This process is now b -bounded. Centering the process in this way does not affect the downstream results because Gaussian/Rademacher width is shift agnostic due to symmetry.

Missing condition on n . This omission is related to the subsequent discussion about comparison with Srebro et al. [2010]. In the proof of Lemma 2.2, after Eqn. 23, the limits of integration are required to satisfy √ 12 Hr R n ( F ) ≤ √ br . Using the fact that in the setup, in general, R n ( F ) = Θ ( B √ n ) , the above is equivalent to n = Ω ( HB 2 b ) , This condition on n is missing from their main statement. Further, as we detail in Appendix E.1.1, this term, in general, can be unbounded.

Fat-Shattering inequality . Using the notation from Srebro et al. [2010]. Note that fat x is a decreasing function in x . Also x log( n x ) is a decreasing function in x for x ≥ 1 . Therefore fat x log( n fat x ) is an increasing function in x . So therefore, for ε ∈ [ γ, θ ] we have fat γ log( n fat γ ) ≤ fat ε log( n fat ε ) ≤ fat θ log( n fat θ ) . This in inequality is in the wrong direction on page 27 of Srebro et al. [2010].

Disregarding the above issues, the bound obtained on Srebro et al. [2010] on the local Rademacher complexity is,

In contrast ours is,

Worst-case improvement for large enough samples. We first perform a general comparison. Accordingly, we bound the above as,

The above is of the same form as the bound of Srebro et al. [2010] in Eqn. (37), and it is easy to see that ours is better whenever n = e Ω ( b HB 2 ) .

Worst-case improvement for constant samples. Unfortunately, the term b HB 2 can be unbounded, from above, in general, as detailed in Appendix E.1.1. However, a reasonable and standard assumption removes this issue. If we assume that loss at zero is bounded as follows, ℓ (0) ≤ HB 2 , then we show in Lemma 22, that b HB 2 = Ω(1) , thereby, making the requirement to be a constant number of samples. A uniform bound on ℓ (0) features in the characterization of sample complexity of learning with non-negative smooth losses as well as the regime described by the above bound on ℓ (0) has been considered in prior works [Arora et al., 2022, Shamir, 2015].

Improvement for certain hypothesis class. In the above chain of inequalities, we use the bound that ˜ G n ( F ) ≤ R n ( F ) √ log n which is tight in the worst case. However, there are natural situations where the two are of the same order. A prominent example is linear predictors with data bounded in (any) norm ∥·∥ and hypothesis class bounded in the via a (regularization) function R which is strongly convex with respect to the dual norm ∥·∥ ∗ [Kakade et al., 2008].

E.1.1 Understanding the HB 2 b term

No non-trivial lower bound. Firstly, we see that the term is not non-trivially lower bounded, in general. Consider the function ℓ ( f ( x )) = 1 2 ( q -f ( x )) 2 . This function is 1 -smooth and considers B = 1 (which bounds the size of f ( x ) ), however, b here is controlled by the size of q and thus could be unbounded. This means that the is a non-trivial lower bound on HB 2 b in general. The above example also works if relax the definition of b to be "bounded difference", sup f ( x ) ,f ′ ( x ′ ) ∈F ( ℓ ( f ( x )) -ℓ ( f ′ ( x ′ ))) as opposed to a uniform absolute bound.

A lower bound under ℓ (0) bound. We give a lower bound under the assumption of bound on ℓ (0) . Lemma 22. Let F be a class of real-valued functions and let the loss function ℓ : R → R be a non-negative H -smooth function with ℓ (0) ≤ L 0 , and sup z ∈ Range ( F ) ℓ ( z ) ≤ b . Then,

where the first inequality uses smoothness, the second AM-GM inequality, and the third uses the self-bounding property of non-negative smooth losses (Lemma 2.1 in Srebro et al. [2010]).

No non-trivial upper bound. Consider the function ℓ ( z ) = H (1 + sin( z )) ; this is non-negative, H -smooth, and b = 2 H . However, we can define the size of the domain B as arbitrary, without affecting the smoothness and range boundedness. Thus, HB 2 b is unbounded from above, in general. Note that even assuming an upper bound on ℓ (0) doesn't help here.

E.2 Comparison with Denevi et al. [2019], Khodak et al. [2019]

MTRL can be seen as a more specific case of meta-learning in Denevi et al. [2019], Khodak et al. [2019], which has the type of representation learning we study frequently called as 'feature learning.' It is possible to give guarantees for excess transfer risk in this more general setting. For example, the work of Denevi et al. [2019], under a generative model assumption, is restricted to linear predictors and convex losses. In their Thm. 5, they get a rate of O ( 1 √ n + 1 √ t ) , on excess transfer risk. However, for the feature learning setting, as they point out, this rate contains terms with hidden dependence on n and t . Their guarantee for feature learning, with linear representations (which is more restrictive than ours) Corollary 7 , gets a rate of O ( 1 √ n + 1 t 1 / 4 ) .

Another work of Khodak et al. [2019] mainly considers two settings of convex and strongly convex losses, respectively. Interestingly, fast rates in terms of the number of tasks can be obtained. Thm. 5.1 in the work, obtains the following rates, O ( 1 √ n + 1 √ nt ) for convex Lipschitz losses and O ( 1 n + 1 √ nt ) for strongly-convex Lipschitz losses.

Proof. Note that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies multitask representation learning for some (general) function hypothesis classes. The estimators correspond to the composition of a \"complex\" function $h$ (corresponding to the shared representation) and a specific task regressor $f_i$ from a simple hypothesis class. This work aims at bounding the excess transfer risk of the ERM estimator. It extends the bound of Tripuraneni et. al (2020) to optimistic rates. More precisely, the square root bounds of Tripuraneni et. al (2020) are improved to linear bounds when 0 risk solutions exist on every task (e.g. no label noise). To do so, the authors do not need to assume the loss to be Lipschitz, but instead need to assume it is smooth (Lipschitz derivative).\n\n-------\n\nI acknowledge having read the author's rebuttal. The authors answered some of my concerns as explained in the comments below.\n\nSTRENGTHS:\nGetting optimistic rates for multitask learning is a nice improvement wrt previous bounds on transfer excess risk.\nThe obtained bound is quite general. Notably, the setup and results are well put in the context of the literature, easily relating this result to previous one.\n\nWEAKNESSES:\nMy main concern is about the current writing of the paper. First of all, I think it deserves a few careful proofreading, as many typos or grammatical errors can be found. Here are a few examples:\n - l. 108: \"we can reuse of\"\n - l. 118-119:  \"is bound\", \"is bound by\"\n - l. 161: I think there is a j subindex missing in the last sum in both the definition of $Pf$ and $P_n f$\n - l. 192: \"In order for there to be any value to learning\" -> weird phrasing\n - l. 198: I guess it should be $f^*$ and $\\hat{h}$ instead of $f$ and $h'$\n\nAlthough these typos are not significant in many sentences, they bring confusion when they appear in definitions (lines 161 and 198). More generally, I find the mathematical content to be quite technical and hard to follow. Some claims are confusing if not inexact. \nFor example the setting described line 171 *does not* imply that $f_j^{\\star}\\circ h^{\\star}$ is the optimal predictor. The same claim indeed also holds for $\\frac{1}{2}f_j^{\\star}\\circ h^{\\star}$. I think the authors should here introduce the setting similarly to Tripuraneni et al (2020), saying that $f_j^*$ and $h^*$ are in the hypothesis class.\n\nAlso, some notations are heavy/confusing, making the technical content (which is already hard to follow by definition) very hard to get. As an example, the variable $q$ in the equation line 204 represents both an integer and functions. In Theorem 1, it is not clear what $\\psi$ exactly is. I find the whole $r_1^*$ and $r_2^*$ quite cumbersome: I would have preferred to directly state Theorem 1 with the bounds on these quantities provided in Theorem 2. More generally, the whole Section 3 is hard to understand. I think that illustrating the obtained results in the particular case of the linear representational model would help the reader.\n\nAs a last concern, the current seems like an incremental improvement of Tripuraneni et al (2020), with tools leading to optimistic rates, that are mostly inherited from Srebro et al (2010). Even though the authors claim in the introduction that Srebro et al's tools cannot be directly used to MTL, the difference seems to be tenuous and only be due to bounding the Rademacher complexity.\nAlso, the authors might provide more motivation regarding optimistic rates.\n\nQUESTIONS:\nSee weaknesses section above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

comments: SUMMARY:
This paper shows novel statistical rates for generalizing to a target task via multi-task representation learning (MTRL) that attain the optimistic 1/nt + 1/m rate, where n is the number of samples per source task, t is the number of source tasks, and m is the number of samples per target tasks, when the optimal source and target risks achievable by the representation and predictor function classes are small. This is an improvement over the previous state-of-the-art rate of 1/rt(nt) + 1/rt(m), and matches the analogous optimistic rate in the single-task setting. Key to the results are novel technical contributions extending local Rademacher complexity analysis to the multi-task setting.

STRENGTHS:
1. The results are a very significant contribution in my opinion -- the established rates significantly improve over previous state-of-the-art in the near-realizable setting. Indeed, the near-realizable setting is important to consider. All assumptions are reasonable and consistent with prior work. Broadly, multi-task representation learning is an important research area. 
 
2. The analysis is rigorous, there are no mistakes in the proofs to my knowledge. From my understanding, substantial technical innovation is required to achieve the results by extending the local Rademacher complexity framework to the multi-task setting.

3. The paper is very well-written.

WEAKNESSES:
1. The Related Works section should also compare with [XT21].

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-24

comments: SUMMARY:
This paper aims to examine the optimistic rates for multi-task representation learning. The authors illustrate that the rate may be faster than the standard rate, depending on the complexity of the learning tasks. The analysis comprises multiple theoretical contributions.

STRENGTHS:
1. The theoretical analysis sounds solid from my perspective. However, I am not an expert in this area and haven't gone through all the supplementary material. I would like to refer to other reviewers in the discussion period.

2. The authors provide detailed comparisons with other theoretical works and expand on the key contributions, helping to understand the critical points of this work better. However, it is still hard to understand the details for readers not in this field.

WEAKNESSES:
1. The assumption 1 regarding the boundness of the loss function and its gradients is too restrictive, particularly since the feature domain is not bound.

2. Although the paper presents new findings in this field, it lacks a thorough explanation of the significance of these results.

3. No conclusion section.

QUESTIONS:
1. Can you explain the distinction between multitask learning and multitask representation learning? Typically, MTL involves acquiring shared representation layers, which serve as the objective for MTRL.

2. What is the significance of optimistic rates in MTRL, and how does it manifest in real-world situations?

3. Could you offer an intuitive definition of task diversity as it pertains to definition 2?

LIMITATIONS:
1. Assumptions are too strong.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the transfer learning and the multi-task learning setting in a Representation Learning context: multiple source tasks are used to learn a good common representation to facilitate the learning process of a target task (transfer learning) or of the same source tasks (multi-task learning). Under regularity assumptions on the loss function and task similarity, the authors provide optimistic statistical rates for the transfer learning and the multi-task learning setting demonstrating the benefit of representation learning. These optimistic bounds interpolate between the standard -1/2 rate and the fast -1 rate, depending on the difficulty (i.e. the realizability) of the learning tasks. In order to reach such a result, the authors also provide the following intermediate contributions: they give a local Rademacher complexity theorem in the representation learning setting (for both the multi-task learning and transfer learning scenarios) and a chain rule for local Rademacher complexity for composite function classes which allows to decouple the (local) complexities of representation and predictor classes.\n\nSTRENGTHS:\nThe authors address a topic that is interesting in a formal and rigorous way. \n\nThe paper is written in a quite clear way.\n\nWEAKNESSES:\nSome bounds given by the authors, especially those related to the Rademacher complexities, and also Sec. 4 and Sec. 5 should be simplified more in my opinion in order to make them more readable. \n\nThe authors did not provide computational experiments testing the performance of the proposed method in the main body.\n\nThe authors adapt optimistic rates present in literature for the single-task setting in order to get their optimistic bounds for representation learning. I wonder if the theoretical contribution is enough for the venue. I would like to better understand which are the main technical difficulties the authors had to face to adapt the optimistic bounds from the single-task to the multiple-task setting. \n\nSome basic gradient based representation-learning references are missing, such as [1-2] below. \n\nReferences\n\n[1] Denevi et al. \"Online-within-online meta-learning.\"\n\n[2] Khodak et al. \"Adaptive Gradient-Based Meta-Learning Methods.\"\n\nQUESTIONS:\nIn Ass. 1, the Lipschitz and the boundedness assumptions are necessary? Smoothness is for sure necessary to get faster rates.\n\nThe authors should recall in my opinion that m in the first equation represent the number of samples of the target task.\n\nCould you please make a detailed comparison between your optimistic bounds and the non-optimistic bounds in the references [1-2] I mentioned above by only keeping the leading terms w.r.t. the samples/tasks number and the complexity measures?\n\nWhat you call 'transfer learning' looks more similar to meta-learning. In transfer learning usually you only have one source task and one target task and you do not investigate the source task training, but only the transfer knowledge from source to target.\n\nThe authors developed their analysis under a quite general notion of task similarity introduced in previous literature. However such similarity assumption seems to be not well motivated for the representation learning setting in which the natural task similarity assumption is that the target estimators of the tasks all lies in the range of the representation. This natural link is well explained for instance in the reference [1] I mentioned above.\n\nThe paper 'Optimistic Rates for Learning with a Smooth Loss' does not use Rademacher complexity measures in order to give optimistic rates. Did you based the proofs on that? If yes, why are you instead using Rademacher complexities?\n\nLIMITATIONS:\nI do not see any potential negative societal impact related to this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}