Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis

1 Introduction . Learning a task with limited samples is crucial for real-world machine learning applications , where proper prior knowledge is a key component for a successful transfer . Meta-Learning [ 3 ] or learningto-learn ( LTL ) aims to extract such information through previous training tasks , which has recently re-emerged as an important topic . Modern approaches based on MAML [ 2 ] have gained tremendous success by exploiting the capabilities of deep neural networks [ 4–9 ] . However , many theoretical questions still remain elusive . For instance , in the most popular methods for few-shot learning [ 10 ] , the task-specific parameters and meta-parameter are updated in support ( also called meta-train ) and query ( also called meta-validation ) set , respectively . However , the majority of existing theoretical results such as [ 1 , 11–14 ] do not provide a formal understanding of such popular practice . Moreover , modern meta-learning approaches have incorporated over-parameterized deep neural networks , where conducting the theoretical analysis becomes even more challenging . In this paper , we introduce a novel theoretical understanding of the generalization property of metalearning through an information-theoretical perspective [ 15 ] . Compared with previous theoretical results , the highlights of our contributions are as follows : Unified Approach We analyze two popular scenarios . 1 ) The conventional LTL [ 11 ] , where the meta-parameters and task-specific parameters are updated within the same data set ( referred as joint training ) . 2 ) The modern MAML-based approaches where the meta-parameters and task specific parameters are updated on distinct data sets ( referred as alternate training ) , and for which the existing theoretical analysis is rare . Flexible Bounds The proposed meta-generalization error bounds are highly flexible : they are algorithm-dependant , data-dependant , and are valid for non-convex loss functions . 1 ) Specifically , the generalization error bound for joint-training ( Theorem 5.1 ) is controlled by the mutual information ∗Department of Computer Science and Software Engineering , < qi.chen.1 @ ulaval.ca > †Department of Electrical Engineering and Computer Engineering , < changjian.shui.1 @ ulaval.ca > ‡Department of Computer Science and Software Engineering , < mario.marchand @ ift.ulaval.ca > 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . between the output of the randomized algorithm and the whole data set . It can cover the typical results of [ 12 , 1 ] , which can be interpreted with an environment-level and a task-level error . In addition , it reveals the benefit of meta learning compared to single task learning . 2 ) Moreover , the generalization error bound for alternate-training ( Theorem 5.2 ) is characterized by the conditional mutual information between the output of the randomized algorithm and the meta-validation dataset , conditioned on the meta-train dataset . Intuitively , when the outputs of a meta learning algorithm w.r.t . different input data-sets are similar ( i.e . the algorithm is stable w.r.t . the data ) , the meta-generalization error bound will be small . This theoretical result is coherent with the recently-proposed Chaser loss in Bayes MAML [ 16 ] . Non-vacuous bounds for gradient-based few-shot learning Conventional gradient-based metalearning theories heavily rely on the assumption of a Lipschitz loss . However , [ 17 ] pointed out that this Lipschitz constant for simple neural networks can be extremely large . Thus , conventional gradient-based upper bounds are often vacuous for deep few-shot scenarios . In contrast , we propose a tighter data-depend bound that depends on the expected gradient-incoherence rather than the gradient norm ( the approximation of the Lipschitz constant ) [ 18 ] for the Meta-SGLD algorithm , which is a stochastic variant of MAML that uses the Stochastic Gradient Langevin Dynamics ( SGLD ) [ 19 ] . We finally validate our theory in few-shot learning scenarios and obtain orders of magnitude tighter bounds in most situations , compared to conventional gradient-based bounds . 2 Related Work . Conventional LTL The early theoretic framework , introduced by Baxter [ 11 ] , proposed the notion of task environment and derived uniform convergence bounds based on the capacity and covering numbers of function classes . Pentina and Lampert [ 12 ] proposed PAC-Bayes risk bounds that depend on environment-level and task-level errors . Amit and Meir [ 1 ] extended this approach and provided a tighter risk bound . However , their theory applies to stochastic neural networks and used factorized Gaussians to approximate the parameters ’ distributions , which is computationally expensive to use in practice . Jose and Simeone [ 20 ] first analyzed meta-learning through information-theoretic tools , while they applied the assumptions that hide some probabilistic relations and obtained theoretical results substantially different from those presented here . Limited to the space , a more detailed discussion is provided in Appendix F. Gradient based meta-learning In recent years , gradient-based meta-learning such as MAML [ 2 ] have drawn increasing attention since they are model-agnostic and are easily deployed for complex tasks like reinforcement learning , computer vision , and federate learning [ 21–24 ] . Then , Reptile [ 25 ] provided a general first-order gradient calculation method . Other methods combine MAML and Bayesian methods through structured variational inference [ 26 ] and empirical Bayes [ 27 ] . In Bayes MAML [ 16 ] , they propose a fast Bayesian adaption method using Stein variational gradient descent and conceived a Chaser loss which coincides with the proposed Theorem 5.2 . On the theoretical side , Denevi et al . [ 13 ] analyzed the average excess risk for Stochastic Gradient Descent ( SGD ) with Convex and Lipschitz loss . Balcan et al . [ 14 ] studied meta-learning through the lens of online convex optimization , and has provided a guarantee with a regret bound . Khodak et al . [ 28 ] extended to more general settings where the task-environment changes dynamically or the tasks share a certain geometric structure . Other guarantees for online meta-learning scenarios are provided by Denevi et al . [ 29 ] and Finn et al . [ 30 ] . Finally , [ 31 , 32 ] also provided a convergence analysis for MAML-based methods . On meta train-validation split Although the support query approaches are rather difficult to analyze , some interesting works have appeared on the simplified linear models . Denevi et al . [ 33 ] first studied train-validation split for linear centroid meta-learning . They proved a generalization bound and concluded that there exists a trade-off for train-validation split , which is consistent with Theorem 5.2 in our paper . Bai et al . [ 34 ] applied the random matrix theoretical analysis for a disentangled comparison between joint training and alternate training under the realizable assumption in linear centroid meta-learning . By calculating the closed-form concentration rates over the mean square error of parameter estimation for the two settings , they obtained a better rate constant with joint training . However , we aim to provide a generic analysis and do not make such a realizable assumption . We believe an additional excess risk analysis with more assumptions is needed for a similar comparison , which is out of the scope of this article . Moreover , Saunshi et al . [ 35 ] analyzed the train-validation split for linear representation learning . They showed that the train-validation split encourages learning a low-rank representation . More detailed discussion and comparison can be found in Appendix F. Information-theoretic learning for single tasks We use here an information-theoretic approach , introduced by Russo and Zou [ 36 ] and Xu and Raginsky [ 15 ] , for characterizing single-task learning . Characterizing the generalization error of a learning algorithm in terms of the mutual information between its input and output brings the significant advantage of the ability to incorporate the dependence on the data distribution , the hypothesis space , and the learning algorithm . This is in sharp contrast with conventional VC-dimension bounds and uniform stability bounds . Tighter mutual information bounds between the parameters and a single data point are explored in [ 37 ] . Pensia et al . [ 38 ] applied the mutual-information framework to a broad class of iterative algorithms , including SGLD and stochastic gradient Hamiltonian Monte Carlo ( SGHMC ) . Negrea et al . [ 39 ] provided data-dependent estimates of information-theoretic bounds for SGLD . For a recent comprehensive study , see Steinke and Zakynthinou [ 40 ] . 3 Preliminaries . Basic Notations We use upper case letters , e.g . X , Y , to denote random variables and corresponding calligraphic letters X , Y to denote the sets which they are defined on . We denote as PX , the marginal probability distribution of X . Given the Markov chain X → Y , PY |X denotes the conditional distribution or the Markov transition kernel . X ⊥⊥ Y means X and Y are independent . And let us recall some basic definitions : Definition 3.1 . Let ψX ( λ ) def = logE [ eλ ( X−E [ X ] ) ] denote the cumulant generating function ( CGF ) of random variable X . Then X is said to be σ-subgaussian if we have ψX ( λ ) ≤ λ2σ2 2 , ∀λ ∈ R . Definition 3.2 . LetX , Y andZ be arbitrary random variables , and letDKL denote the KL divergence . The mutual information between X and Y is defined as : I ( X ; Y ) def = DKL ( PX , Y ||PXPY ) . The disintegrated mutual information between X and Y given Z is defined as : IZ ( X ; Y ) def = DKL ( PX , Y |Z ||PX|ZPY |Z ) . The corresponding conditional mutual information is defined as : I ( X ; Y |Z ) def= EZ [ IZ ( X ; Y ) ] . Information theoretic bound for single task learning We consider an unknown distribution µ on an instance space Z = X × Y , and a set of independent samples S = { Zi } mi=1 drawn from µ : Zi ∼ µ and S ∼ µm . Given a parametrized hypothesis spaceW and a loss function ` : W×Z → R , the true risk and the empirical risk of w ∈ W are respectively defined as Rµ ( w ) def = EZ∼µ ` ( w , Z ) and RS ( w ) def = ( 1/m ) ∑m i=1 ` ( w , Zi ) . Following the setting of information-theoretic learning [ 36 , 15 , 37 ] , a learning algorithm A is a randomized mapping that takes a dataset S as input and outputs a hypothesis W according to a conditional distribution PW |S , i.e. , W = A ( S ) ∼ PW |S .4 The ( mean ) generalization error gen ( µ , A ) def= EW , S [ Rµ ( W ) −RS ( W ) ] of an algorithm A is then bounded according to : Theorem 3.1 . ( Xu and Raginsky [ 15 ] ) Suppose that for each w ∈ W , the prediction loss ` ( w , Z ) is σ-subgaussian with respect to Z ∼ µ . Then for any randomized learner A characterized by PW |S , for S ∼ µm , we have |gen ( µ , A ) | ≤ √ 2σ2 m I ( W ; S ) . I ( W ; S ) is the mutual information between the input and output of algorithm A ( see definition in Definition A.2 ) . Theorem 3.1 reveals that the less the output hypothesis W depends on the dataset S , the smaller the generalization error of the learning algorithm will be . 4Note that the conditional distribution PW |S is different from the posterior distribution in Bayes learning .

Summary: This paper extends recent information-theoretic approaches for computing model generalization to the meta-learning setting. They prove algorithm/data-dependent bounds for both the joint-training and support-query meta-learning strategies and instantiate the results for variants of gradient-based meta-learning with noise injection. Finally, they compute generalization bounds on synthetic and Omniglot data.