Task Relatedness-Based Generalization Bounds for Meta Learning

mn ) for meta learning is thus derived by exploiting the proposed task relatedness . Moreover , when investigating a special branch of meta learning that involves representation learning with deep neural networks , we establish spectrally-normalized bounds for both classification and regression problems . Finally , we demonstrate that the relatedness requirement between two tasks is satisfied when the sample space possesses the completeness and separability properties , validating the rationality and applicability of our proposed task-relatedness measure . 1 INTRODUCTION . By leveraging knowledge distilled from the training tasks1 , meta learning ( Thrun & Pratt , 1998 ; Baxter , 2000 ) learns to perform well on a new but related task . One important branch of meta learning achieving great success in practical machine learning applications is representation learning ( Krizhevsky et al. , 2012 ; Bengio et al. , 2013 ; He et al. , 2016 ) , where one first learns a shared feature extractor ( e.g. , deep neural networks ) over the training tasks , and then learns a prediction function for the new task on the top of the features constructed from the extractor , within a few gradient steps ( Finn et al. , 2017 ; Li et al. , 2017 ) or with a feedforward process ( Snell et al. , 2017 ; Sung et al. , 2018 ) . If the feature extractor can capture the common information across tasks , it is possible that utilizing representation learning can help learners generalize well to a new task with much less data . To build up a rigorous framework to support this intuition , the pioneering meta learning theory assumes that both the n training tasks and the new task are i.i.d . generated from the same environment ( Baxter , 2000 ) . Then , analogous to the single task learning whose goal is to select from the hypothesis spaceH a hypothesis h to achieve the minimal expected loss on the task , meta learning expects to choose from the hypothesis space family H a hypothesis spaceH ( ∈ H ) that contains a good solution to any task sampled from the environment . Under the PAC learning framework ( Valiant , 1984 ; Vapnik , 1989 ) , ( Baxter , 2000 ) derives a uniform bound of O ( √ C1 mn + √ C2 n ) on the expected loss of a hypothesis space H over the new task in the environment , according to the empirical loss of H ∗Corresponding author . 1In this work , a task represents a data distribution , or say a probability measure on the sample space . over the n training tasks , where C1 , C2 are different logarithms of the covering number ( Anthony & Bartlett , 2002 ) about H. However , compared with the single task learning whose error bound O ( √ C m ) of any h ∈ H can utilize all m training samples , where C represents certain complexity indicators such as the VC-dimension ( Blumer et al. , 1989 ) or the entropy ( Pollard , 1984 ) ofH , Baxter ’ s meta learning generalization bound of any H ∈ H can not fully utilize all mn training samples ( e.g. , with a bound of O ( 1√ nm ) , without extra terms of O ( 1√ n ) or O ( 1√ m ) ) . Nevertheless , there is still little theoretical research on how the relatedness of these tasks can affect the full utilization of all mn training data in meta learning under Baxter ’ s proposed i.i.d . task environmetn framework . In this paper , we propose to address this problem by studying the task relatedness and provide a new generalization bound . To achieve full utilization of all training sample , our motivation is that different tasks need to be ‘ related ’ enough such that samples from various measures can be assumed to be generated from an ‘ almost ’ identical distribution . The equivalence relation of different data distributions actually corresponds to the measure-preserving isomorphism of their induced measure spaces2 ( see Definition 7 ) . Therefore in this paper , we define a new notion of task relatedness called almost Π-relatedness in Definition 3 according to the existence of a measure-preserving bijective transformation between two measure spaces associated with different tasks . A PAC-style generalization error bound that fully utilize all training data is thus provided in Theorem 3 , by exploiting the proposed task relatedness . We further employ this task relatedness notation to analyze the representation learning with deep neural networks , and establish non-parameter-count-based spectrally-normalized bounds for both classification and regression problems under the meta learning framework . Finally , we demonstrate the rationality of our proposed task-relatedness notion from a theoretical standpoint . Our main contributions are summarized as follows : ( 1 ) We propose a new notion of task relatedness , called almost Π-relatedness , by exploring the existence of a bijective transformation between two tasks in the environment . A novel PAC-style generalization error bound of O ( √ C mn ) is thus derived for general meta learning by exploiting the proposed task relatedness , where C captures the logarithm of the covering number of a hypothesis space family . Such bound thus can fully utilize the whole n ∗m training data in meta learning . ( 2 ) For meta learning that involves representation learning , we bound the covering number in Contribution ( 1 ) with two covering numbers that are both defined over a single task , making our results suitable to be combined with recent works of deep neural network in the single task learning . In particular , we provide the spectrally-normalized bounds of O ( √ C1+nC2 nm ) for classification and re- gression problems in meta learning , where C1 , C2 are certain complexities that are dependent on the matrix norms , but independent of the total number of the parameters , in deep neural networks . ( 3 ) We rigorously demonstrate that , any two tasks are almost Π-related if the sample space is a complete separable metric space , validating the applicability of our proposed task-relatedness measure . 2 RELATED WORK . Meta Learning Theory . The first theoretical analysis for meta learning is performed by ( Baxter , 2000 ) , which gives a bound on the expected loss on the unseen task of any hypothesis space in terms of the empirical training loss . Under this framework , ( Pentina & Ben-David , 2015 ) studies the generalization error bound of the hypothesis space in the case of kernel learning . Although many follow-up works also assume that all tasks are sampled from the same environment , they bound different kinds of generalization error from various perspectives . One important branch is to bound the transfer risk over the new task of a deterministic procedure , such as the linear feature transformation algorithm ( Maurer , 2009 ) , and the Bayes algorithm in PAC-Bayes theory ( Pentina & Lampert , 2014 ; Amit & Meir , 2018 ) . Among them , ( Chen et al. , 2020 ) derive a bound ofO ( 1√ n ) from the algorithm stability perspective , and ( Pentina & Lampert , 2014 ) derives a bound ofO ( 1√ n + 1√ m ) ( we suppress the complexity factor in the numerator for concision , similarly hereinafter ) . Another branch aims to bound the excess risk of the task specific function returned by ERM algorithm over the unseen task . They study meta learning theory from different views , such as multitask representation learning 2A measure space is a triple ( Z , A , P ) , where Z is the set , A is the σ-algebra on Z , P is a measure on A . ( Maurer et al. , 2016 ) and online learning ( Khodak et al. , 2019 ) . Among them , ( Du et al. , 2021 ) and ( Tripuraneni et al. , 2020 ) escape from Baxter ’ s proposed task environment setting , by defining task similarity or task diversity notion , obtaining similar excess risk bounds ofO ( 1√ mn + 1√ m ) in the case of high-dimensional transfer learning . Our work follows Baxter ’ s proposed i.i.d . task environment framework , and gives an improved bound on the generalization error of the hypothesis space . The detailed comparisons between our results and that in ( Baxter , 2000 ) are presented in Appendix C.1 . Task Relatedness . Another related work ( Ben-David & Schuller , 2003 ) also proposes a taskrelatedness concept , by defining the bijective transformation π on the input space X . But in our work , the bijective transformation π is imposed on the sample space Z = X × Y ( where Y is the output space ) . It is not difficult to see that the task relatedness considered in ( Ben-David & Schuller , 2003 ) is actually a special case of our defined task-relatedness measure . We further validate the rationality of our proposed notion by revealing the existence of the bijective transformation π on Z when Z is a complete separable metric space , making our results more applicable . The differences between our proposed task-relatedness notion and that in ( Ben-David & Schuller , 2003 ) can be found in Appendix C.2 . Besides , ( Khodak et al. , 2019 ) and ( Du et al. , 2021 ) also consider taskrelatedness notions such as parameter-closeness or sharing a low-rank subspace for meta learning . To distinguish the notions proposed in this paper with others is truly one of our future directions . 3 PRELIMINARY . In this paper , we use uppercase letters ( e.g. , Z ) to represent different spaces . The boldface z represents a matrix , with zi as its i-th row and z ; j its j-th column . ~z denotes a vector . All detailed proofs of our theoretical results are deferred to the Appendix B for reader ’ s benefits . 3.1 META LEARNING . The formulation of meta learning problem can be summarized as follows . We are given a sample space Z = X × Y , where X is an input space and Y an output space . In this paper , we assume that Z is a complete separable metric space . A loss function is defined as l : Y ×Y → R , and we assume that l has the range [ 0 , 1 ] , or equivalently , with rescaling technique , we assume that l is bounded . An environment ( P , Q ) is a two-tuple , whereP is the set of all probability measures\distributions\tasks on X × Y and Q is a probability measure on P . A hypothesis space family is formulated as H = { H } , where each H ∈ H is a set of hypotheses\functions h : X → Y . In single task learning , the learner needs to choose an optimal hypothesis h∗ ∈ H to minimize the expected loss of h over a probability measure P ( ∈ P ) on the product space X × Y : erP ( h ) = ∫ X×Y l ( h ( x ) , y ) dP ( x , y ) . Similarly , the goal of meta learning is to find a hypothesis space H∗ ∈ H which contains a good solution to any task sampled from the environment , and minimize the following expected loss of H over a measure Q on P ( Baxter , 2000 , Eq . ( 6 ) ) : erQ ( H ) = ∫ P infh∈H erP ( h ) dQ ( P ) . In practice , it is hard to minimize erQ ( H ) directly since we do not know the exact distribution of the environment measure Q . We can only capture the information about the environment ( P , Q ) by observing the training data z generated from the n training tasks Pj ( j = 1 , ... , n ) that are i.i.d . sampled from the environment measure Q . Formally , this is achieved in the following manner : ( 1 ) Sample n times from P according to Q to generate ( i.i.d . ) probability measures P1 , ... , Pn . ( 2 ) Sample m times from X × Y according to Pj to generate { ( x1j , y1j ) , ... , ( xmj , ymj ) } ( 1 ≤ j ≤ n ) . ( 3 ) Denote zij = ( xij , yij ) ∈ Z , and an ( m , n ) -sample will be generated , denoted by z and written as a matrix z = ( zij ) m×n . We then choose to minimize the empirical loss êrz ( H ) over the training data z for meta learning , which is defined as êrz ( H ) = 1n ∑n j=1 infh∈H êrz ; j ( h ) , where z ; j is the j-th column of matrix z . Let P = ( P1 , ... , Pn ) . We also consider the following loss êrP ( H ) as the empirical estimate of the expected loss erQ ( H ) : êrP ( H ) = 1n ∑n j=1 infh∈H erPj ( h ) . Definition 1 ( Baxter , 2000 ) For any hypothesis h : X → Y , define hl as the composition of loss function and hypothesis : hl : X × Y → [ 0 , 1 ] by hl ( x , y ) = l ( h ( x ) , y ) . For any hypothesis space H ∈ H , define Hl = { hl : h ∈ H } as the function space on Z . For any sequence of n hypothesises ( h1 , ... , hn ) , define ( h1 , ... , hn ) l : ( X × Y ) n → [ 0 , 1 ] by ( h1 , ... , hn ) l ( x1 , y1 , ... , xn , yn ) = 1/n ∑n i=1 l ( hi ( xi ) , yi ) . We will use hl to denote ( h1 , ... , hn ) l. ∀H ∈ H , define Hnl = { ( h1 , ... , hn ) l : h1 , ... , hn ∈ H } and Hnl = ⋃ H∈HHnl . Define P ( hl ) = ∫ X×Y hl ( x , y ) dP ( x , y ) and P ( Hl ) = infhl∈Hl P ( hl ) where P is a probability measure on Z , and we have P ( hl ) = erP ( h ) and erP ( H ) = P ( Hl ) . Definition 2 Let ( M , d ) be a pseudo-metric3 space . ∀ > 0 , a subset T̂ is called an -cover of T ⊆ M if ∀t ∈ T , ∃t′ ∈ T̂ such that d ( t , t′ ) ≤ . Let P = P1 × · · · × Pn be the product measure on Zn . For any ( Hnl 3 ) hl , h′l : Zn → [ 0 , 1 ] , define the pseudo-metric dP ( hl , h′l ) =∫ Zn |hl ( ~z ) − h′l ( ~z ) |dP ( ~z ) . Then for any > 0 , the covering number N ( , Hnl , dP ) is defined as min { |T | ∣∣T is an -cover of Hnl under the dP pseudo-metric } , where |T | is the cardinality of T . We hope to use the empirical loss êrz ( H ) or êrP ( H ) to approximate the expected loss erQ ( H ) . Instead of the traditional absolute value function d ( x , y ) = |x−y| , we consider the following metric function ( ν > 0 , x , y ≥ 0 ) dν [ x , y ] = |x−y|x+y+v to measure the distance between x and y , which is first used by ( Haussler , 1992 ) . In Section 4 , we will bound the deviation dν [ êrz ( H ) , erQ ( H ) ] by bounding dν [ êrz ( H ) , êrP ( H ) ] and dν [ êrP ( H ) , erQ ( H ) ] , respectively .

Summary: In this paper, the authors proposed two key concepts: almost PI-Related tasks and almost PI-related environment. Intuitively, if two tasks are PI-related, then if a function space H contains a good solution to task A, it should also contain a good solution to task B.  With these two concepts, the authors went forward and proved that in an almost PI-related environment, the generalization bound has a convergence rate of O(1/\sqrt(mn)). In other words, when tasks are similar, then meta-learning algorithms should be able to utilize all data points across all tasks.