Few-Shot Learning via Learning the Representation, Provably

k n2 ) on the target task for the linear representation class ; here d is the ambient input dimension and k ( d ) is the dimension of the representation . This result bypasses the Ω ( 1T ) barrier under the i.i.d . task assumption , and can capture the desired property that all n1T samples from source tasks can be pooled together for representation learning . We further extend this result to handle a general representation function class and obtain a similar result . Next , we consider the setting where the common representation may be high-dimensional but is capacity-constrained ( say in norm ) ; here , we again demonstrate the advantage of representation learning in both high-dimensional linear regression and neural networks , and show that representation learning can fully utilize all n1T samples from source tasks . 1 INTRODUCTION . A popular scheme for few-shot learning , i.e. , learning in a data-scarce environment , is representation learning , where one first learns a feature extractor , or representation , e.g. , the last layer of a convolutional neural network , from different but related source tasks , and then uses a simple predictor ( usually a linear function ) on top of this representation in the target task . The hope is that the learned representation captures the common structure across tasks , which makes a linear predictor sufficient for the target task . If the learned representation is good enough , it is possible that a few samples are sufficient for learning the target task , which can be much smaller than the number of samples required to learn the target task from scratch . While representation learning has achieved tremendous success in a variety of applications ( Bengio et al. , 2013 ) , its theoretical studies are limited . In existing theoretical work , the most natural algorithm is to explicitly look for the optimal representation given source data , which when combined with a ( different ) linear predictor on top for each task can achieve the smallest cumulative training error on the source tasks . Of course , it is not guaranteed that the representation found will be useful for the target task unless one makes some assumptions to characterize the connections between different tasks . Existing work often imposes a probabilistic assumption about the connection between tasks : each task is sampled i.i.d . from an underlying distribution . Under this assumption , Maurer et al . ( 2016 ) showed an Õ ( 1√ T + 1√n2 ) risk bound on the target task , where T is the number of source tasks , n1 is the number of samples per source task , and n2 is the number of samples from the target task.1 Unsatisfactorily , this bound necessarily requires the number of tasks T to be large , and it ∗Alphabetical Order . 1We only focus on the dependence on T , n1 and n2 in this paragraph . Note that Maurer et al . ( 2016 ) only considered n1 = n2 , but their approach does not give a better result even if n1 > n2 . does not improve when the number of samples per source task , n1 , increases . Intuitively , one should expect more data to help , and therefore an ideal bound would be 1√ n1T + 1√n2 ( or 1 n1T + 1n2 in the realizable case ) , because n1T is the total number of training data points from source tasks , which can be potentially pooled to learn the representation . Unfortunately , as pointed out by Maurer et al . ( 2016 ) , there exists an example that satisfies the i.i.d . task assumption for which Ω ( 1√ T ) is unavoidable ( or Ω ( 1T ) in the realizable setting ) . This means that the i.i.d . assumption alone is not sufficient if we want to take advantage of a large amount of samples per task . Therefore , a natural question is : What connections between tasks enable representation learning to utilize all source data ? In this paper , we obtain the first set of results that fully utilize the n1T data from source tasks . We replace the i.i.d . assumption over tasks with natural structural conditions on the input distributions and linear predictors . These conditions depict that the target task can be in some sense “ covered ” by the source tasks , which will further give rise to the desirable guarantees . First , we study the setting where there exists a common well-specified low-dimensional representation in source and target tasks , and obtain an Õ ( dkn1T + k n2 ) risk bound on the target task where d is the ambient input dimension , k ( d ) is the dimension of the representation , and n2 is the number of data from the target task . Note that this improves the dn2 rate of just learning the target task without using representation learning . The term dkn1T indicates that we can fully exploit all n1T data in the source tasks to learn the representation . We further extend this result to handle general representation function class and obtain an Õ ( C ( Φ ) n1T + k n2 ) risk bound on the target task , where Φ is the representation function class and C ( Φ ) is a certain complexity measure of Φ . Second , we study the setting where there exists a common linear high-dimensional representation for source and target tasks , and obtain an Õ ( R̄√Tr ( Σ ) √ n1T + R̄ √ ‖Σ‖ 2√ n2 ) rate where R̄ is a normalized nuclear norm control over linear predictors , and Σ is the covariance matrix of the raw feature . This also improves over the baseline rate for the case without using representation learning . We further extend this result to two-layer neural networks with ReLU activation . Again , our results indicate that we can fully exploit n1T source data . A technical insight coming out of our analysis is that any capacity-controlled method that gets low test error on the source tasks must also get low test error on the target task by virtue of being forced to learn a good representation . Our result on high-dimensional representations shows that the capacity control for representation learning does not have to be through explicit low dimensionality . Organization . The rest of the paper is organized as follows . We review related work in Section 2 . In Section 3 , we formally describe the setting we consider . In Section 4 , we present our main result for low-dimensional linear representation learning . A generalization to nonlinear representation classes is demonstrated in Section 5 . In Section 6 , we present our main result for high-dimensional linear representation learning . In Section 7 , we present our result for representation learning in neural networks . We conclude in Section 8 and leave most of the proofs to appendices . 2 RELATED WORK . The idea of multitask representation learning at least dates back to Caruana ( 1997 ) ; Thrun and Pratt ( 1998 ) ; Baxter ( 2000 ) . Empirically , representation learning has shown its great power in various domains ; see Bengio et al . ( 2013 ) for a survey . In particular , representation learning is widely adopted for few-shot learning tasks ( Sun et al. , 2017 ; Goyal et al. , 2019 ) . Representation learning is also closely connected to meta-learning ( Schaul and Schmidhuber , 2010 ) . Recent work Raghu et al . ( 2019 ) empirically suggested that the effectiveness of the popular meta-learning algorithm Model Agnostic Meta-Learning ( MAML ) is due to its ability to learn a useful representation . The scheme we analyze in this paper is closely related to Lee et al . ( 2019 ) ; Bertinetto et al . ( 2018 ) for meta-learning . On the theoretical side , Baxter ( 2000 ) performed the first theoretical analysis and gave sample complexity bounds using covering numbers . Maurer et al . ( 2016 ) and follow-up work gave analyses on the benefit of representation learning for reducing the sample complexity of the target task . They assumed every task is i.i.d . drawn from an underlying distribution and can obtain an Õ ( 1√ T + 1√n2 ) rate . As pointed out in Maurer et al . ( 2016 ) , the 1√ T dependence is not improvable even if n1 →∞ because 1√ T is the rate of concentration for the distribution over tasks . The concurrent work of Tripuraneni et al . ( 2020 ) studies low-dimensional linear representation learning and obtains a similar result as ours in this case , but they assume isotropic inputs for all tasks , which is a special case of our result . Furthermore , we also provide results for high-dimensional linear representations , general non-linear representations , and two-layer neural networks . Tripuraneni et al . ( 2020 ) also give a computationally efficient algorithm for standard Gaussian inputs and a lower bound for subspace recovery in the low-dimensional linear setting . Another recent line of theoretical work analyzed gradient-based meta-learning methods ( Denevi et al. , 2019 ; Finn et al. , 2019 ; Khodak et al. , 2019 ) and showed guarantees for convex losses by using tools from online convex optimization . Lastly , we remark that there are analyses for other representation learning schemes ( Arora et al. , 2019 ; McNamara and Balcan , 2017 ; Galanti et al. , 2016 ; Alquier et al. , 2016 ; Denevi et al. , 2018 ) . 3 NOTATION AND SETUP . Notation . Let [ n ] = { 1 , 2 , . . . , n } . We use ‖·‖ or ‖·‖2 to denote the ` 2 norm of a vector or the spectral norm of a matrix . Denote by ‖·‖F and ‖·‖∗ the Frobenius norm and the nuclear norm of a matrix , respectively . Let 〈· , ·〉 be the Euclidean inner product between vectors or matrices . Denote by I the identity matrix . Let N ( µ , σ2 ) /N ( µ , Σ ) be the one-dimensional/multi-dimensional Gaussian distribution , and χ2 ( m ) the chi-squared distribution with m degrees of freedom . For a matrix A ∈ Rm×n , let σi ( A ) be its i-th largest singular value . Let span ( A ) be the subspace of Rm spanned by the columns of A , i.e. , span ( A ) = { Av | v ∈ Rn } . Denote PA = A ( A > A ) †A > ∈ Rm×m , which is the projection matrix onto span ( A ) . Here † stands for the Moore-Penrose pseudoinverse . Note that 0 PA I and P 2A = PA. We also define P⊥A = I − PA , which is the projection matrix onto span ( A ) ⊥ , the orthogonal complement of span ( A ) in Rm . For a positive semidefinite ( psd ) matrix B , denote by λmax ( B ) and λmin ( B ) its largest and smallest eigenvalues , respectively ; let B1/2 be the psd matrix such that ( B1/2 ) 2 = B . We use the standard O ( · ) , Ω ( · ) and Θ ( · ) notation to hide universal constant factors . We also use a . b or b & a to indicate a = O ( b ) , and use a b or b a to mean that a ≥ C · b for a sufficiently large universal constant C > 0 . Problem Setup . Suppose that there are T source tasks . Each task t ∈ [ T ] is associated with a distribution µt over the joint data space X× Y , where X is the input space and Y is the output space . In this paper we consider X ⊆ Rd and Y ⊆ R. For each source task t ∈ [ T ] we have access to n1 i.i.d . samples ( xt,1 , yt,1 ) , . . . , ( xt , n1 , yt , n1 ) from µt . For convenience , we express these n1 samples collectively as an input matrix Xt ∈ Rn1×d and an output vector yt ∈ Rn1 . Multitask learning tries to learn prediction functions for all the T source tasks simultaneously in the hope of discovering some underlying common property of these tasks . The common property we consider in this paper is a representation , which is a function φ : X→ Z that maps an input to some feature space Z ⊆ Rk . We restrict the representation function to be in some function class Φ , e.g. , neural networks . We try to use different linear predictors on top of a common representation function φ to model the input-output relations in different source tasks . Namely , for each task t ∈ [ T ] , we set the prediction function to be x 7→ 〈wt , φ ( x ) 〉 ( wt ∈ Rk ) . Therefore , using the training samples from T tasks , we can solve the following optimization problem to learn the representation:2 minimize φ∈Φ , w1 , ... , wT∈Rk 1 2n1T T∑ t=1 n1∑ i=1 ( yt , i − 〈wt , φ ( xt , i ) 〉 ) 2 . ( 1 ) We overload the notation to allow φ to apply to all the samples in a data matrix simultaneously , i.e. , φ ( Xt ) = [ φ ( xt,1 ) , . . . , φ ( xt , n1 ) ] > ∈ Rn1×k . Then ( 1 ) can be rewritten as minimize φ∈Φ , w1 , ... , wT∈Rk 1 2n1T T∑ t=1 ‖yt − φ ( Xt ) wt‖2 . ( 2 ) 2We use the ` 2 loss throughout this paper . Let φ̂ ∈ Φ be the representation function obtained by solving ( 2 ) . Now we retain this representation and apply it to future ( target ) tasks . For a target task specified by a distribution µT+1 over X× Y , suppose we receive n2 i.i.d . samples XT+1 ∈ Rn2×d , yT+1 ∈ Rn2 . We further train a linear predictor on top of φ̂ for this task : minimize wT+1∈Rk 1 2n2 ∥∥∥yT+1 − φ̂ ( XT+1 ) wT+1∥∥∥2 . ( 3 ) Let ŵT+1 be the returned solution . We are interested in whether our learned predictor x 7→ 〈ŵT+1 , φ̂ ( x ) 〉 works well on average for the target task , i.e. , we want the population loss LµT+1 ( φ̂ , ŵT+1 ) = E ( x , y ) ∼µT+1 1 2 ( y − 〈ŵT+1 , φ̂ ( x ) 〉 ) 2 to be small . In particular , we are interested in the few-shot learning setting , where the number of samples n2 from the target task is small – much smaller than the number of samples required for learning the target task from scratch . Data assumption . In order for the above learning procedure to make sense , we assume that there is a ground-truth optimal representation function φ∗ ∈ Φ and specializations w∗1 , . . . , w∗T+1 ∈ Rk for all the tasks such that for each task t ∈ [ T + 1 ] , we have E ( x , y ) ∼µt [ y|x ] = 〈w∗t , φ∗ ( x ) 〉 . More specifically , we assume ( x , y ) ∼ µt can be generated by y = 〈w∗t , φ∗ ( x ) 〉+ z , x ∼ pt , z ∼ N ( 0 , σ2 ) , ( 4 ) where x and z are independent . Our goal is to bound the excess risk of our learned model on the target task , i.e. , how much our learned model ( φ̂ , ŵT+1 ) performs worse than the optimal model ( φ∗ , w∗T+1 ) on the target task : ER ( φ̂ , ŵT+1 ) = LµT+1 ( φ̂ , ŵT+1 ) − LµT+1 ( φ∗ , w∗T+1 ) = 1 2 Ex∼pT+1 [ ( 〈ŵT+1 , φ̂ ( x ) 〉 − 〈w∗T+1 , φ∗ ( x ) 〉 ) 2 ] . ( 5 ) Here we have used the relation ( 4 ) . Oftentimes we are interested in the average performance on a random target task ( i.e. , w∗T+1 is random ) . In such case we look at the expected excess risk Ew∗T+1 [ ER ( φ̂ , ŵT+1 ) ] .

Summary: The paper aims at justifying the success of few shot learning methods that work based on finding a shared representation among a number of tasks. A serious theoretical challenge is that, even if we assume such a representation exists (and belongs to a predefined class of functions with controlled capacity), we would still need to assume something that connects the source tasks with the target task. Previous work has considered "i.i.d. tasks", however, the obtained bounds were not natural in the sense that we don't have the usual decrease in the error as we increase the size of the training set of the source tasks. Under a different set of assumptions, the authors show that, in a sense, one can "fully" exploit the training data from the source tasks.