Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis

1 Introduction . Learning a task with limited samples is crucial for real-world machine learning applications , where proper prior knowledge is a key component for a successful transfer . Meta-Learning [ 3 ] or learningto-learn ( LTL ) aims to extract such information through previous training tasks , which has recently re-emerged as an important topic . Modern approaches based on MAML [ 2 ] have gained tremendous success by exploiting the capabilities of deep neural networks [ 4–9 ] . However , many theoretical questions still remain elusive . For instance , in the most popular methods for few-shot learning [ 10 ] , the task-specific parameters and meta-parameter are updated in support ( also called meta-train ) and query ( also called meta-validation ) set , respectively . However , the majority of existing theoretical results such as [ 1 , 11–14 ] do not provide a formal understanding of such popular practice . Moreover , modern meta-learning approaches have incorporated over-parameterized deep neural networks , where conducting the theoretical analysis becomes even more challenging . In this paper , we introduce a novel theoretical understanding of the generalization property of metalearning through an information-theoretical perspective [ 15 ] . Compared with previous theoretical results , the highlights of our contributions are as follows : Unified Approach We analyze two popular scenarios . 1 ) The conventional LTL [ 11 ] , where the meta-parameters and task-specific parameters are updated within the same data set ( referred as joint training ) . 2 ) The modern MAML-based approaches where the meta-parameters and task specific parameters are updated on distinct data sets ( referred as alternate training ) , and for which the existing theoretical analysis is rare . Flexible Bounds The proposed meta-generalization error bounds are highly flexible : they are algorithm-dependant , data-dependant , and are valid for non-convex loss functions . 1 ) Specifically , the generalization error bound for joint-training ( Theorem 5.1 ) is controlled by the mutual information ∗Department of Computer Science and Software Engineering , < qi.chen.1 @ ulaval.ca > †Department of Electrical Engineering and Computer Engineering , < changjian.shui.1 @ ulaval.ca > ‡Department of Computer Science and Software Engineering , < mario.marchand @ ift.ulaval.ca > 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . between the output of the randomized algorithm and the whole data set . It can cover the typical results of [ 12 , 1 ] , which can be interpreted with an environment-level and a task-level error . In addition , it reveals the benefit of meta learning compared to single task learning . 2 ) Moreover , the generalization error bound for alternate-training ( Theorem 5.2 ) is characterized by the conditional mutual information between the output of the randomized algorithm and the meta-validation dataset , conditioned on the meta-train dataset . Intuitively , when the outputs of a meta learning algorithm w.r.t . different input data-sets are similar ( i.e . the algorithm is stable w.r.t . the data ) , the meta-generalization error bound will be small . This theoretical result is coherent with the recently-proposed Chaser loss in Bayes MAML [ 16 ] . Non-vacuous bounds for gradient-based few-shot learning Conventional gradient-based metalearning theories heavily rely on the assumption of a Lipschitz loss . However , [ 17 ] pointed out that this Lipschitz constant for simple neural networks can be extremely large . Thus , conventional gradient-based upper bounds are often vacuous for deep few-shot scenarios . In contrast , we propose a tighter data-depend bound that depends on the expected gradient-incoherence rather than the gradient norm ( the approximation of the Lipschitz constant ) [ 18 ] for the Meta-SGLD algorithm , which is a stochastic variant of MAML that uses the Stochastic Gradient Langevin Dynamics ( SGLD ) [ 19 ] . We finally validate our theory in few-shot learning scenarios and obtain orders of magnitude tighter bounds in most situations , compared to conventional gradient-based bounds . 2 Related Work . Conventional LTL The early theoretic framework , introduced by Baxter [ 11 ] , proposed the notion of task environment and derived uniform convergence bounds based on the capacity and covering numbers of function classes . Pentina and Lampert [ 12 ] proposed PAC-Bayes risk bounds that depend on environment-level and task-level errors . Amit and Meir [ 1 ] extended this approach and provided a tighter risk bound . However , their theory applies to stochastic neural networks and used factorized Gaussians to approximate the parameters ’ distributions , which is computationally expensive to use in practice . Jose and Simeone [ 20 ] first analyzed meta-learning through information-theoretic tools , while they applied the assumptions that hide some probabilistic relations and obtained theoretical results substantially different from those presented here . Limited to the space , a more detailed discussion is provided in Appendix F. Gradient based meta-learning In recent years , gradient-based meta-learning such as MAML [ 2 ] have drawn increasing attention since they are model-agnostic and are easily deployed for complex tasks like reinforcement learning , computer vision , and federate learning [ 21–24 ] . Then , Reptile [ 25 ] provided a general first-order gradient calculation method . Other methods combine MAML and Bayesian methods through structured variational inference [ 26 ] and empirical Bayes [ 27 ] . In Bayes MAML [ 16 ] , they propose a fast Bayesian adaption method using Stein variational gradient descent and conceived a Chaser loss which coincides with the proposed Theorem 5.2 . On the theoretical side , Denevi et al . [ 13 ] analyzed the average excess risk for Stochastic Gradient Descent ( SGD ) with Convex and Lipschitz loss . Balcan et al . [ 14 ] studied meta-learning through the lens of online convex optimization , and has provided a guarantee with a regret bound . Khodak et al . [ 28 ] extended to more general settings where the task-environment changes dynamically or the tasks share a certain geometric structure . Other guarantees for online meta-learning scenarios are provided by Denevi et al . [ 29 ] and Finn et al . [ 30 ] . Finally , [ 31 , 32 ] also provided a convergence analysis for MAML-based methods . On meta train-validation split Although the support query approaches are rather difficult to analyze , some interesting works have appeared on the simplified linear models . Denevi et al . [ 33 ] first studied train-validation split for linear centroid meta-learning . They proved a generalization bound and concluded that there exists a trade-off for train-validation split , which is consistent with Theorem 5.2 in our paper . Bai et al . [ 34 ] applied the random matrix theoretical analysis for a disentangled comparison between joint training and alternate training under the realizable assumption in linear centroid meta-learning . By calculating the closed-form concentration rates over the mean square error of parameter estimation for the two settings , they obtained a better rate constant with joint training . However , we aim to provide a generic analysis and do not make such a realizable assumption . We believe an additional excess risk analysis with more assumptions is needed for a similar comparison , which is out of the scope of this article . Moreover , Saunshi et al . [ 35 ] analyzed the train-validation split for linear representation learning . They showed that the train-validation split encourages learning a low-rank representation . More detailed discussion and comparison can be found in Appendix F. Information-theoretic learning for single tasks We use here an information-theoretic approach , introduced by Russo and Zou [ 36 ] and Xu and Raginsky [ 15 ] , for characterizing single-task learning . Characterizing the generalization error of a learning algorithm in terms of the mutual information between its input and output brings the significant advantage of the ability to incorporate the dependence on the data distribution , the hypothesis space , and the learning algorithm . This is in sharp contrast with conventional VC-dimension bounds and uniform stability bounds . Tighter mutual information bounds between the parameters and a single data point are explored in [ 37 ] . Pensia et al . [ 38 ] applied the mutual-information framework to a broad class of iterative algorithms , including SGLD and stochastic gradient Hamiltonian Monte Carlo ( SGHMC ) . Negrea et al . [ 39 ] provided data-dependent estimates of information-theoretic bounds for SGLD . For a recent comprehensive study , see Steinke and Zakynthinou [ 40 ] . 3 Preliminaries . Basic Notations We use upper case letters , e.g . X , Y , to denote random variables and corresponding calligraphic letters X , Y to denote the sets which they are defined on . We denote as PX , the marginal probability distribution of X . Given the Markov chain X → Y , PY |X denotes the conditional distribution or the Markov transition kernel . X ⊥⊥ Y means X and Y are independent . And let us recall some basic definitions : Definition 3.1 . Let ψX ( λ ) def = logE [ eλ ( X−E [ X ] ) ] denote the cumulant generating function ( CGF ) of random variable X . Then X is said to be σ-subgaussian if we have ψX ( λ ) ≤ λ2σ2 2 , ∀λ ∈ R . Definition 3.2 . LetX , Y andZ be arbitrary random variables , and letDKL denote the KL divergence . The mutual information between X and Y is defined as : I ( X ; Y ) def = DKL ( PX , Y ||PXPY ) . The disintegrated mutual information between X and Y given Z is defined as : IZ ( X ; Y ) def = DKL ( PX , Y |Z ||PX|ZPY |Z ) . The corresponding conditional mutual information is defined as : I ( X ; Y |Z ) def= EZ [ IZ ( X ; Y ) ] . Information theoretic bound for single task learning We consider an unknown distribution µ on an instance space Z = X × Y , and a set of independent samples S = { Zi } mi=1 drawn from µ : Zi ∼ µ and S ∼ µm . Given a parametrized hypothesis spaceW and a loss function ` : W×Z → R , the true risk and the empirical risk of w ∈ W are respectively defined as Rµ ( w ) def = EZ∼µ ` ( w , Z ) and RS ( w ) def = ( 1/m ) ∑m i=1 ` ( w , Zi ) . Following the setting of information-theoretic learning [ 36 , 15 , 37 ] , a learning algorithm A is a randomized mapping that takes a dataset S as input and outputs a hypothesis W according to a conditional distribution PW |S , i.e. , W = A ( S ) ∼ PW |S .4 The ( mean ) generalization error gen ( µ , A ) def= EW , S [ Rµ ( W ) −RS ( W ) ] of an algorithm A is then bounded according to : Theorem 3.1 . ( Xu and Raginsky [ 15 ] ) Suppose that for each w ∈ W , the prediction loss ` ( w , Z ) is σ-subgaussian with respect to Z ∼ µ . Then for any randomized learner A characterized by PW |S , for S ∼ µm , we have |gen ( µ , A ) | ≤ √ 2σ2 m I ( W ; S ) . I ( W ; S ) is the mutual information between the input and output of algorithm A ( see definition in Definition A.2 ) . Theorem 3.1 reveals that the less the output hypothesis W depends on the dataset S , the smaller the generalization error of the learning algorithm will be . 4Note that the conditional distribution PW |S is different from the posterior distribution in Bayes learning .

Summary: The authors derive two bounds on the generalization property of meta-learning algorithms using an information-theoretic analysis. The first one considers algorithms based on a *joint-training*, i.e.  when all the data from the tasks is used to update the parameters. The second bound is specific to algorithms learned through an *alternate training*, i.e. the tasks data are split into training and validation, training part is used to update the task-specific parameters and the validation part is used to update the meta-parameters. Then, they analyze two concrete algorithms based on SGLD, a joint-training version and an alternate training version, to derive the bounds for these specific cases. Finally, they empirically estimate the values of their bounds and compare with an empirical estimate of a previous bound based on the lipschitz constant of the network. The empirical estimation is done on synthetic data and the omniglot dataset.