Task Relatedness-Based Generalization Bounds for Meta Learning

mn ) for meta learning is thus derived by exploiting the proposed task relatedness . Moreover , when investigating a special branch of meta learning that involves representation learning with deep neural networks , we establish spectrally-normalized bounds for both classification and regression problems . Finally , we demonstrate that the relatedness requirement between two tasks is satisfied when the sample space possesses the completeness and separability properties , validating the rationality and applicability of our proposed task-relatedness measure . 1 INTRODUCTION . By leveraging knowledge distilled from the training tasks1 , meta learning ( Thrun & Pratt , 1998 ; Baxter , 2000 ) learns to perform well on a new but related task . One important branch of meta learning achieving great success in practical machine learning applications is representation learning ( Krizhevsky et al. , 2012 ; Bengio et al. , 2013 ; He et al. , 2016 ) , where one first learns a shared feature extractor ( e.g. , deep neural networks ) over the training tasks , and then learns a prediction function for the new task on the top of the features constructed from the extractor , within a few gradient steps ( Finn et al. , 2017 ; Li et al. , 2017 ) or with a feedforward process ( Snell et al. , 2017 ; Sung et al. , 2018 ) . If the feature extractor can capture the common information across tasks , it is possible that utilizing representation learning can help learners generalize well to a new task with much less data . To build up a rigorous framework to support this intuition , the pioneering meta learning theory assumes that both the n training tasks and the new task are i.i.d . generated from the same environment ( Baxter , 2000 ) . Then , analogous to the single task learning whose goal is to select from the hypothesis spaceH a hypothesis h to achieve the minimal expected loss on the task , meta learning expects to choose from the hypothesis space family H a hypothesis spaceH ( ∈ H ) that contains a good solution to any task sampled from the environment . Under the PAC learning framework ( Valiant , 1984 ; Vapnik , 1989 ) , ( Baxter , 2000 ) derives a uniform bound of O ( √ C1 mn + √ C2 n ) on the expected loss of a hypothesis space H over the new task in the environment , according to the empirical loss of H ∗Corresponding author . 1In this work , a task represents a data distribution , or say a probability measure on the sample space . over the n training tasks , where C1 , C2 are different logarithms of the covering number ( Anthony & Bartlett , 2002 ) about H. However , compared with the single task learning whose error bound O ( √ C m ) of any h ∈ H can utilize all m training samples , where C represents certain complexity indicators such as the VC-dimension ( Blumer et al. , 1989 ) or the entropy ( Pollard , 1984 ) ofH , Baxter ’ s meta learning generalization bound of any H ∈ H can not fully utilize all mn training samples ( e.g. , with a bound of O ( 1√ nm ) , without extra terms of O ( 1√ n ) or O ( 1√ m ) ) . Nevertheless , there is still little theoretical research on how the relatedness of these tasks can affect the full utilization of all mn training data in meta learning under Baxter ’ s proposed i.i.d . task environmetn framework . In this paper , we propose to address this problem by studying the task relatedness and provide a new generalization bound . To achieve full utilization of all training sample , our motivation is that different tasks need to be ‘ related ’ enough such that samples from various measures can be assumed to be generated from an ‘ almost ’ identical distribution . The equivalence relation of different data distributions actually corresponds to the measure-preserving isomorphism of their induced measure spaces2 ( see Definition 7 ) . Therefore in this paper , we define a new notion of task relatedness called almost Π-relatedness in Definition 3 according to the existence of a measure-preserving bijective transformation between two measure spaces associated with different tasks . A PAC-style generalization error bound that fully utilize all training data is thus provided in Theorem 3 , by exploiting the proposed task relatedness . We further employ this task relatedness notation to analyze the representation learning with deep neural networks , and establish non-parameter-count-based spectrally-normalized bounds for both classification and regression problems under the meta learning framework . Finally , we demonstrate the rationality of our proposed task-relatedness notion from a theoretical standpoint . Our main contributions are summarized as follows : ( 1 ) We propose a new notion of task relatedness , called almost Π-relatedness , by exploring the existence of a bijective transformation between two tasks in the environment . A novel PAC-style generalization error bound of O ( √ C mn ) is thus derived for general meta learning by exploiting the proposed task relatedness , where C captures the logarithm of the covering number of a hypothesis space family . Such bound thus can fully utilize the whole n ∗m training data in meta learning . ( 2 ) For meta learning that involves representation learning , we bound the covering number in Contribution ( 1 ) with two covering numbers that are both defined over a single task , making our results suitable to be combined with recent works of deep neural network in the single task learning . In particular , we provide the spectrally-normalized bounds of O ( √ C1+nC2 nm ) for classification and re- gression problems in meta learning , where C1 , C2 are certain complexities that are dependent on the matrix norms , but independent of the total number of the parameters , in deep neural networks . ( 3 ) We rigorously demonstrate that , any two tasks are almost Π-related if the sample space is a complete separable metric space , validating the applicability of our proposed task-relatedness measure . 2 RELATED WORK . Meta Learning Theory . The first theoretical analysis for meta learning is performed by ( Baxter , 2000 ) , which gives a bound on the expected loss on the unseen task of any hypothesis space in terms of the empirical training loss . Under this framework , ( Pentina & Ben-David , 2015 ) studies the generalization error bound of the hypothesis space in the case of kernel learning . Although many follow-up works also assume that all tasks are sampled from the same environment , they bound different kinds of generalization error from various perspectives . One important branch is to bound the transfer risk over the new task of a deterministic procedure , such as the linear feature transformation algorithm ( Maurer , 2009 ) , and the Bayes algorithm in PAC-Bayes theory ( Pentina & Lampert , 2014 ; Amit & Meir , 2018 ) . Among them , ( Chen et al. , 2020 ) derive a bound ofO ( 1√ n ) from the algorithm stability perspective , and ( Pentina & Lampert , 2014 ) derives a bound ofO ( 1√ n + 1√ m ) ( we suppress the complexity factor in the numerator for concision , similarly hereinafter ) . Another branch aims to bound the excess risk of the task specific function returned by ERM algorithm over the unseen task . They study meta learning theory from different views , such as multitask representation learning 2A measure space is a triple ( Z , A , P ) , where Z is the set , A is the σ-algebra on Z , P is a measure on A . ( Maurer et al. , 2016 ) and online learning ( Khodak et al. , 2019 ) . Among them , ( Du et al. , 2021 ) and ( Tripuraneni et al. , 2020 ) escape from Baxter ’ s proposed task environment setting , by defining task similarity or task diversity notion , obtaining similar excess risk bounds ofO ( 1√ mn + 1√ m ) in the case of high-dimensional transfer learning . Our work follows Baxter ’ s proposed i.i.d . task environment framework , and gives an improved bound on the generalization error of the hypothesis space . The detailed comparisons between our results and that in ( Baxter , 2000 ) are presented in Appendix C.1 . Task Relatedness . Another related work ( Ben-David & Schuller , 2003 ) also proposes a taskrelatedness concept , by defining the bijective transformation π on the input space X . But in our work , the bijective transformation π is imposed on the sample space Z = X × Y ( where Y is the output space ) . It is not difficult to see that the task relatedness considered in ( Ben-David & Schuller , 2003 ) is actually a special case of our defined task-relatedness measure . We further validate the rationality of our proposed notion by revealing the existence of the bijective transformation π on Z when Z is a complete separable metric space , making our results more applicable . The differences between our proposed task-relatedness notion and that in ( Ben-David & Schuller , 2003 ) can be found in Appendix C.2 . Besides , ( Khodak et al. , 2019 ) and ( Du et al. , 2021 ) also consider taskrelatedness notions such as parameter-closeness or sharing a low-rank subspace for meta learning . To distinguish the notions proposed in this paper with others is truly one of our future directions . 3 PRELIMINARY . In this paper , we use uppercase letters ( e.g. , Z ) to represent different spaces . The boldface z represents a matrix , with zi as its i-th row and z ; j its j-th column . ~z denotes a vector . All detailed proofs of our theoretical results are deferred to the Appendix B for reader ’ s benefits . 3.1 META LEARNING . The formulation of meta learning problem can be summarized as follows . We are given a sample space Z = X × Y , where X is an input space and Y an output space . In this paper , we assume that Z is a complete separable metric space . A loss function is defined as l : Y ×Y → R , and we assume that l has the range [ 0 , 1 ] , or equivalently , with rescaling technique , we assume that l is bounded . An environment ( P , Q ) is a two-tuple , whereP is the set of all probability measures\distributions\tasks on X × Y and Q is a probability measure on P . A hypothesis space family is formulated as H = { H } , where each H ∈ H is a set of hypotheses\functions h : X → Y . In single task learning , the learner needs to choose an optimal hypothesis h∗ ∈ H to minimize the expected loss of h over a probability measure P ( ∈ P ) on the product space X × Y : erP ( h ) = ∫ X×Y l ( h ( x ) , y ) dP ( x , y ) . Similarly , the goal of meta learning is to find a hypothesis space H∗ ∈ H which contains a good solution to any task sampled from the environment , and minimize the following expected loss of H over a measure Q on P ( Baxter , 2000 , Eq . ( 6 ) ) : erQ ( H ) = ∫ P infh∈H erP ( h ) dQ ( P ) . In practice , it is hard to minimize erQ ( H ) directly since we do not know the exact distribution of the environment measure Q . We can only capture the information about the environment ( P , Q ) by observing the training data z generated from the n training tasks Pj ( j = 1 , ... , n ) that are i.i.d . sampled from the environment measure Q . Formally , this is achieved in the following manner : ( 1 ) Sample n times from P according to Q to generate ( i.i.d . ) probability measures P1 , ... , Pn . ( 2 ) Sample m times from X × Y according to Pj to generate { ( x1j , y1j ) , ... , ( xmj , ymj ) } ( 1 ≤ j ≤ n ) . ( 3 ) Denote zij = ( xij , yij ) ∈ Z , and an ( m , n ) -sample will be generated , denoted by z and written as a matrix z = ( zij ) m×n . We then choose to minimize the empirical loss êrz ( H ) over the training data z for meta learning , which is defined as êrz ( H ) = 1n ∑n j=1 infh∈H êrz ; j ( h ) , where z ; j is the j-th column of matrix z . Let P = ( P1 , ... , Pn ) . We also consider the following loss êrP ( H ) as the empirical estimate of the expected loss erQ ( H ) : êrP ( H ) = 1n ∑n j=1 infh∈H erPj ( h ) . Definition 1 ( Baxter , 2000 ) For any hypothesis h : X → Y , define hl as the composition of loss function and hypothesis : hl : X × Y → [ 0 , 1 ] by hl ( x , y ) = l ( h ( x ) , y ) . For any hypothesis space H ∈ H , define Hl = { hl : h ∈ H } as the function space on Z . For any sequence of n hypothesises ( h1 , ... , hn ) , define ( h1 , ... , hn ) l : ( X × Y ) n → [ 0 , 1 ] by ( h1 , ... , hn ) l ( x1 , y1 , ... , xn , yn ) = 1/n ∑n i=1 l ( hi ( xi ) , yi ) . We will use hl to denote ( h1 , ... , hn ) l. ∀H ∈ H , define Hnl = { ( h1 , ... , hn ) l : h1 , ... , hn ∈ H } and Hnl = ⋃ H∈HHnl . Define P ( hl ) = ∫ X×Y hl ( x , y ) dP ( x , y ) and P ( Hl ) = infhl∈Hl P ( hl ) where P is a probability measure on Z , and we have P ( hl ) = erP ( h ) and erP ( H ) = P ( Hl ) . Definition 2 Let ( M , d ) be a pseudo-metric3 space . ∀ > 0 , a subset T̂ is called an -cover of T ⊆ M if ∀t ∈ T , ∃t′ ∈ T̂ such that d ( t , t′ ) ≤ . Let P = P1 × · · · × Pn be the product measure on Zn . For any ( Hnl 3 ) hl , h′l : Zn → [ 0 , 1 ] , define the pseudo-metric dP ( hl , h′l ) =∫ Zn |hl ( ~z ) − h′l ( ~z ) |dP ( ~z ) . Then for any > 0 , the covering number N ( , Hnl , dP ) is defined as min { |T | ∣∣T is an -cover of Hnl under the dP pseudo-metric } , where |T | is the cardinality of T . We hope to use the empirical loss êrz ( H ) or êrP ( H ) to approximate the expected loss erQ ( H ) . Instead of the traditional absolute value function d ( x , y ) = |x−y| , we consider the following metric function ( ν > 0 , x , y ≥ 0 ) dν [ x , y ] = |x−y|x+y+v to measure the distance between x and y , which is first used by ( Haussler , 1992 ) . In Section 4 , we will bound the deviation dν [ êrz ( H ) , erQ ( H ) ] by bounding dν [ êrz ( H ) , êrP ( H ) ] and dν [ êrP ( H ) , erQ ( H ) ] , respectively .

Summary: The paper: 1) proposes a new notion of task-relatedness for meta-learning termed "almost $\Pi$-relatedness" that assumes isomorphisms between two tasks in the environment. A PAC-style generalization bound of $\mathcal{O}(\sqrt{\frac{C}{mn}} +  \sqrt{\frac{\ln(1/\delta)}{mn}})$ is shown, improving upon Baxter et. al's. 2) For meta learning that involves representation learning, they bound $C$ with two covering numbers that are both defined over a single task, making their results suitable to be combined with recent works of deep neural network in the single task learning. 3) They demonstrate that any two tasks in the environment are almost $\Pi$-related if the focused sampled space is a complete separable metric space.