Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning

Authors: yiqin  lv
qi  wang
dong  liang
zheng  xie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 17:36:36

Abstract: Meta learning is a promising paradigm in the era of large models and 
task distributional robustness has become an indispensable consideration in real-world scenarios.
Recent advances have examined the effectiveness of tail task risk minimization in fast adaptation robustness improvement \citep{wang2023simple}.
This work contributes to more theoretical investigations and practical enhancements in the field.
Specifically, we reduce the distributionally robust strategy to a max-min optimization problem, constitute the Stackelberg equilibrium as the solution concept, and estimate the convergence rate.
In the presence of tail risk, we further derive the generalization bound, establish connections with estimated quantiles, and practically improve the studied strategy.
Accordingly, extensive evaluations demonstrate the significance of our proposal in boosting robustness.

Main paper: 
 introduction: 
 The past few years have witnessed a surge of research interest in meta learning due to its great potential in the academia and industry [2] [3] [4] [5] . By leveraging previous experience, such a learning paradigm can extract knowledge as priors and empower learning models with adaptability to unseen tasks from a few examples [6] .Nevertheless, the investigation of the robustness needs to be more comprehensive from the task distribution perspective. In particular, the recently developed large models heavily rely on the fewshot learning capability and demand robustness of prediction in risk-sensitive scenarios [7] . For example, when the GPT-like dialogue generation system [8] [9] [10] comes into medical consultancy domains, imprecise answers can cause catastrophic consequences to patients, families, and even societies in real-world scenarios. In light of these considerations, it is desirable to watch adaptation differences across tasks when deploying meta learning models and promote task robustness study for meeting substantial practical demands.Recently, Wang et al. [1] proposes to increase task distributional robustness via employing the tail risk minimization principle [11] for meta learning. In circumventing the optimization intractability in the presence of nonconvex risk functions, a two-stage optimization strategy is adopted as the heuristic to solve the problem. In brief, the strategy consists of two phases in iteration, respectively: (i) estimating the risk quantile VaR α [11] with the crude Monte Carlo method [12] in the task space; (ii) updating the meta learning model parameters from the screened subset of tasks. Such a strategy is simple in implementation, with an improvement guarantee under certain conditions, and empirically shows improved robustness when faced with task distributional shifts. Despite these advances, there remain several unresolved theoretical or practical issues in the field.Existing limitations. This paper also works on the robustness of fast adaptation in the task space and tries to fill gaps in [1] . Theoretically, we notice that in [1] (i) there constitutes no notion of solutions, (ii) it lacks an algorithmic understanding of the two-stage optimization strategy, (iii) the analysis on generalization capability is ignored in the tail risk of tasks. Empirically, the use of the crude Monte Carlo might be less efficient in quantile estimates and suffers from a higher approximation error of the VaR α , degrading the adaptation robustness. These bottlenecks may weaken the versatility of the two-stage optimization strategy's use in practice and require more understanding before deployment.Primary contributions. In response to the above-mentioned concerns, we propose translating the two-stage optimization strategy for distributionally robust meta learning [1] into a max-min optimization problem [13] . Intrinsically, this work models the optimization steps as a Stackelberg game, and task selection and the sub-gradient optimizer work as the leader and follower players in decision-making, respectively. The theoretical understanding is from two aspects:1. We constitute the local Stackelberg equilibrium as a solution concept, estimate the convergence rate, and characterize the asymptotic behavior in learning dynamics.2. We derive the generalization bound in the presence of the tail task risk, which connects quantile estimates with fast adaptation capability in unseen tasks.Meanwhile, the empirical influence of VaR α estimators is examined, and we advance meta learners' robustness by comprising more accurate quantile estimators.2 Literature ReviewQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction clearly state the claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
meta learning: 
 Meta learning, or learning to learn, is an increasingly popular paradigm to distill knowledge from prior experience to unseen scenarios with a few examples [6] . Various meta learning methods have emerged in the past decade, and this section overviews some dominant families.The context-based methods mainly use the encoder-decoder structure and represent tasks by latent variables. Typical ones are in the form of the conditional exchangeable stochastic processes and learn function distributions, such as neural processes [14] , conditional neural processes [15] and their extensions [16] [17] [18] [19] [20] [21] [22] [23] [24] . The optimization-based approaches seek the optimal meta initialization of model parameters and update models from a few examples. Widely known are model agnostic meta learning [25] and related variants [26] [27] [28] [29] , such as MetaCurvature [30] , which learns curvature information and transforms gradients in the inner-loop optimization. The metrics-based methods represent tasks in geometry and perform well in few-shot image classification [31] [32] [33] . For example, MetaOptNet [34] proposes to learn embeddings under a linear classifier and achieve SOTA few-shot classification performance. There also exist other methods, e.g., hyper-networks [35, 36] , memory-augmented networks [37] and recurrent models [38] .The robustness concept in meta learning attracts recent attention, particularly when deploying large models in real-world scenarios. Admittedly, previous literature works have investigated the scenarios where the meta dataset's input is corrupted [39, 40] or the model parameter is perturbed [29] . Studies regarding the fast adaptation robustness in task distribution remain limited. Wang et al. [41] explicitly generates task distribution for robust adaptation. Collins et al. [42] employs the worst-case optimization for promoting MAML's robustness to extreme worst cases. With the help of tail risk minimization, Wang et al. [1] proposes two-stage optimization strategies to robustify the fast adaptation. This work centers around [1] but stresses more theoretical understandings and performance improvement points.As for generalization capability, there are a couple of works in meta learning. Chen et al. [43] exploits the information theory to derive the bound for MAML's like methods. From the data splitting perspective, Bai et al. [44] formulates the theoretical foundation and connects it to optimality. In [45] , an average risk bound is constructed with the bias for improving performance. Importantly, prior work [1] ignores the generalization analysis, and meta learner's generalization in tail risk cases has not been studied in the literature.We list all of the assumptions mentioned in this work. These assumptions further serve the demonstration of propositions and theorems in the main paper.Assumption 1. To proceed, we retain most assumptions from [1] for theoretical analysis, including: 1. The meta risk function ℓ(D Q τ , D S τ ; θ) is β τ -Lipschitzsup τ ∈Ωα,τ ℓ(D Q τi , D S τi ; θ) ≤ L max .Assumption 2. The implicit function h(•) is β h -Lipschitz continuous w.r.t. θ ∈ Θ, and ∇ θ F(q, θ) is β q -Lipschitz continuous w.r.t. q ∈ Q α .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of the work in Sec 5.7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 General notations. Let p(τ ) be the task distribution in meta learning. We respectively express the task space and the model parameter space as Ω τ and Θ. We denote the complete task set by T and refer to D τ as the meta dataset.For instance, D τ comprises a collection of data points {(x i , y i )} n+m i=1 in regression. D τ is ususally prepared into the support set D S τ for skill transfer and the query set D Q τ to assess adaptation performance. Take the conditional neural process [15] as an example, D S τ = {(x i , y i )} n i=1 works for task representation with D Q τ = {(x i , y i )} n+m i=1 the all data points to fit in regression. The meta risk function corresponds to a map ℓ : D τ × Θ → R + , evaluating fast adaptation performance. Given p(τ ) and meta learning model parameters θ, we can induce the cumulative distribution of the meta risk function value in the real space as F ℓ (l; θ) := P({ℓ(D Q τ , D S τ ; θ) ≤ l; τ ∈ T , l ∈ R + }), but there is no explicit parameterized form for F ℓ in practice as F ℓ is θ-dependent.When it comes to the tail risk minimization, we commonly use the conditional value-at-risk (CVaR α ) with the probability threshold α ∈ [0, 1). The quantile of our interest is called the value-at-risk (VaR α ) [11] with the definition: VaR α [ℓ(T , θ)] = inf l∈R + {l|F ℓ (l; θ) ≥ α, τ ∈ T }. The resulting normalized cumulative distribution F α ℓ (l; θ) is defined as:F α ℓ (l; θ) = 0, l < VaR α [ℓ(T , θ)] F ℓ (l;θ)-α 1-α , l ≥ VaR α [ℓ(T , θ)].∀θ ∈ Θ, the meta learning operator M θ defines:M θ : τ → ℓ(D Q τ , D S τ ; θ). Accordingly, the tail risk task subspace Ω α,τ := ℓ≥VaRα[ℓ(T ,θ)] M -1 θ (ℓ), with the task distribution constrained in Ω α,τ by p α (τ ; θ). Please refer to Fig. 7 for illustrations of risk concepts.Assumption 1. To proceed, we retain most assumptions from [1] for theoretical analysis, including: 1. The meta risk function ℓ(D Q τ , D S τ ; θ) is β τ -Lipschitzsup τ ∈Ωα,τ ℓ(D Q τ , D S τ ; θ) ≤ L max .This subsection revisits commonly used risk minimization principles in the meta learning field.Expected risk minimization. The standard principle is the expected/empirical risk minimization originated from statistical learning theory [46] . It minimizes meta risk based on the sampling chance of tasks from the original task distribution:EQUATIONWorst-case risk minimization. Noticing that the worst fast adaptation can be disastrous in some risk sensitive scenarios, Collins et al. [42] proposes to conduct the worst-case optimization in meta learning:EQUATIONHowever, as observed from experiments in [42] , such a principle inevitably sacrifices too much average performance for gains of worst-case robustness. Meanwhile, it requires a couple of implementation tricks and specialized algorithms in stabilizing optimization.Expected tail risk minimization (CVaR α ). To balance the average performance and the worst-case performance, Wang et al. [1] minimizes the expected tail risk, or equivalently CVaR α risk measure:EQUATIONDue to no closed form of p α (τ ; θ), Wang et al. [1] introduces a slack variable ξ ∈ R and reformulates the objective as follows:EQUATIONwhere v β := F -1 ℓ (β) denotes the quantile statistics and ℓ(D Q τ , D S τ ; θ)ξ+ := max{ℓ(D Q τ , D S τ ; θ) -ξ, 0}is the hinge risk. The optimization objective involves the integral of quantiles in a continuous interval (α, 1], which is intractable to precisely parameterize with neural networks. The form in Eq. ( 4) utilizes the duality trick [11] , enabling tractable sampling from the complete task space.Before delving deeper into the theoretical issues, we first present DR-MAML [1] as an instantiation to explain the expected tail risk minimization. Example 1 (DR-MAML [1] ). Given p(τ ) and vanilla MAML [25] , the distributionally robust MAML within CVaR α can be written as a bi-level optimization problem:EQUATIONwhere the gradient update w.r.t. the support set ∇ θ ℓ(D S τ ; θ) indicates the inner loop with a learning rate λ. The outer loop executes the gradient updates w.r.t. Eq. ( 5) and seeks the robust meta initialization in the parameter space. Two-stage optimization strategies. Without loss of generality, we further detail the computational pipelines of Example 1 with two-stage optimization strategies. Note that MAML [25] is an optimization-based meta learning method, and the implementation is to execute the sub-gradient descent over a batch of tasks when updating the meta initialization θ meta : On the left side is the two-stage distributionally robust strategy [1] . The contributed theoretical understanding is right-down, with the right-up the empirical improvement. Arrows show connections between components.EQUATIONEQUATIONEQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]  
distributionally robust meta learning as a stackelberg game: 
 Implementing the two-stage optimization strategy in meta learning requires first specifying the stages' order. The default is the minimization of the risk measure w.r.t. the parameter space after the maximization of the risk measure w.r.t. the task subspace. Hence, we propose to connect it to max-min optimization [13] and the Stackelberg game [47] .Max-min optimization. With the pre-assigned decision-making orders, the studied problem can be characterized as:EQUATIONwhere Q α := {q(τ )|T q ⊆ T , τ ∈Tq p(τ )dτ = 1 -α} constitutes a collection of uncertainty sets [48] over task subspace T q , and q(τ ) is the normalized probability density over the task subspace. Note that in the expected tail risk minimization principle, there is no closed form of optimization objective Eq. (4) as the tail risk is θ-dependent. It is approximately interpreted as the max-min optimization when applied to the distribution over the uncertainty set Q α . Proposition 1. The uncertainty set Q α is convex and compact in terms of probability measures.Practical optimization is achieved via mini-batch gradient estimates and sub-gradient updates with the task size B in [1] ; the feasible subsets correspond to all combinations of size ⌈B * (1α)⌉. Also, Eq. ( 7) is non-differentiable w.r.t. q(τ ), leaving previous approaches [49] [50] [51] [52] unavailable in practice.Stackelberg game & best responses. The example computational pipelines in Eq. ( 6) can be understood as approximately solving a stochastic two-player zero-sum Stackelberg game. Mathematically, such a game referred to as SG can be depicted as SG := ⟨P L , P F ; {q ∈ Q α }, {θ ∈ Θ}; F(q, θ)⟩.Moreover, we translate the two-stage optimization as decisions made by two competitors, which are illustrated in Fig. 1 . The maximization operator executes in the task space, corresponding to the leader P L in SG with the utility function F(q, θ). The follower P F attempts to execute sub-gradient updates over the meta learners' parameters via maximizing -F(q, θ).The two players compete to maximize separate utility functions in SG, which can be characterized as:EQUATIONwhere the leader player P L specifies the worst case combinations from the uncertainty set Q α , and the follower P F reacts to the resulting normalized tail risk for increasing fast adaptation robustness.It is worth noting that the update rules in Eq. ( 8) are also called best responses of players in game theory. The above procedures can be deemed the bi-level optimization [53] since the update of the meta learner implicitly depends on the leader's last time decision.The improvement guarantee has been demonstrated when employing two-stage optimization strategies for minimizing the tail risk in [1] . Furthermore, we claim that under certain conditions, there converges to a solution for the proposed Stackelberg game SG. The sufficient evidence is:1. The two-stage optimization [1] results in a monotonic sequence:Model Updates : • • • → {q t-1 , θ t } → {q t , θ t+1 } → • • • (9a) Monotonic Improvement : • • • ≥ F(q t-1 , θ t ) ≥ F(q t , θ t+1 ) ≥ • • • ; (9b) 2. As ℓ ≤ L max , the objective E q ℓ(D Q τ , D S τ ; θ) ≤ L max naturally holds ∀q ∈ Q α and θ ∈ Θ.Built on the boundness of risk functions and the theorem of improvement guarantee, such an optimization process can finally converge [54] . Then, a crucial question arises concerning the obtained solution: What is the notion of the convergence point in the game?To answer this question, we need to formulate the corresponding solution concept in SG. Here, the global Stackelberg equilibrium is introduced as follows.Definition 1 (Global Stackelberg Equilibrium). Let (q * , θ * ) ∈ Q α × Θ be the solution. With the leader q * ∈ Q α and the follower θ * ∈ Θ, (q * , θ * ) is called a global Stackelberg equilibrium if the following inequalities are satisfied, ∀q ∈ Q α and ∀θ ∈ Θ, infθ ′ ∈Θ F(q, θ ′ ) ≤ F(q * , θ * ) ≤ F(q * , θ).Proposition 2 (Existence of Equilibrium). Given the Assumption 1, there always exists the global Stackelberg equilibrium as the Definition 1 for the studied SG.Nevertheless, the existence of the global Stackelberg equilibrium can be guaranteed; it is NP-hard to obtain the equilibrium with existing optimization techniques. The same as that in [55] , we turn to the local Stackelberg equilibrium as the Definition 2, where the notion of the local Stackelberg game is restricted in a neighborhood Q ′ α × Θ ′ in strategies. Definition 2 (Local Stackelberg Equilibrium). Let (q * , θ * ) ∈ Q α × Θ be the solution. With the leader q * ∈ Q α and the follower θ * ∈ Θ, (q * , θ * ) is called a local Stackelberg equilibrium for the leader if the following inequalities hold, ∀q ∈ Q ′ α , infθ∈S Θ ′ (q * ) F(q * , θ) ≥ inf θ∈S Θ ′ (q) F(q, θ), where S Θ ′ (q) := { θ ∈ Θ ′ |F(q, θ) ≤ F(q, θ), ∀θ ∈ Θ ′ }.The nature of nonconvex programming comprises the above local optimum, and we introduce concepts below for further analysis. It can be validated that F(q, θ) is a quasi-concave function w.r.t. q, meaning that for any positive number l ∈ R + , the set {q|q ∈ Q α , F(q, θ) > l} is convex in Q α . As a result, we deduce that there exists an implicit function h(•) : Θ → Q α such that the condition holds h(θ) = q with q = arg max q∈Qα F(q, θ). For the implicit function h, along with ∇ θ F(q, θ), we make the Assumption below. Assumption 2. The implicit function h(•) is β h -Lipschitz continuous w.r.t. θ ∈ Θ, and ∇ θ F(q, θ) is β q -Lipschitz continuous w.r.t. q ∈ Q α .Learning to learn scales with the number of tasks, but the optimization process is computationally expensive [56] [57] [58] [59] [60] , particularly when large language models are meta learners [3, 61, 62] . In training distributionally robust meta learners, estimating the convergence rate allows monitoring of the convergence and designing early stopping criteria to reach a desirable performance, reducing computational burdens [63] . Consequently, we turn to another question regarding the solution concept: What is the convergence rate of the two-stage optimization algorithm?The runtime complexity for the leader's move can be easily estimated from subset selection, while the analysis for the follower is non-trivial. Under certain conditions, we can derive the following convergence rate theorem, where λ is the learning rate in gradient descent w.r.t. θ. EQUATIONwhereT 1 = {τ : ℓ * < VaR * α , ℓ meta ≥ VaR meta α }, T 2 = {τ : ℓ * ≥ VaR * α , ℓ meta < VaR meta α }.For sufficiently large T , the first term can be bounded by a small number due to the convergence, and the second term vanishes since lim T →∞ ℓ meta = ℓ * and lim T →∞ VaR meta α = VaR * α , respectively. Another crucial issue regarding meta learning lies in the fast adaptation capability in unseen cases. This drives us to answer the following question: How does the resulting meta learner generalize in the presence of tail task risk?To this end, we first define R(θ* ) = E pα(τ ) [ℓ * ], R(θ * ) = 1 B B i=1 δ(τ i )ℓ(D Q τi , D S τi ; θ * ), and R w (θ * ) = 1 B B i=1 pα(τi) p(τi) ℓ(D Q τi , D S τi ; θ * ), where τ i ∼ p(τ ). Also note that the support of p α (τ ; θ * ) is within that of p(τ ), namely supp(p α (τ ; θ * )) ⊆ supp(p(τ )). Then we can induce Theorem 4.3 w.r.t. the tail risk generalization. Theorem 4.3 (Generalization Bound in the Tail Risk Cases). Given a collection of task samples {τ i } B i=1 and corresponding meta datasets, we can derive the following generalization bound in the presence of tail risk:EQUATIONwhere the inequality holds with probability at least 1ϵ and ϵ ∈ (0, 1), V[•] denotes the variance operation, and L max is from the Assumption 1.In conjunction with the confidence ϵ and a task batch B of significant size, Theorem 4.3 reveals the generalization bound given the meta-trained parameter θ * . It is also associated with the varianceV τi∼pα(τ ) [ℓ(D Q τi , D S τi ; θ * )]. Besides, we also derive a specific bound in the case of MAML, and details are attached in Appendix Theorem C.1.Theorem 4.3 reveals that an accurate estimate of VaR α yields a precise variance (i.e.,V τi∼pα(τ ) [ℓ(D Q τi , D S τi ; θ * )]), leading to more reliable bounds. Accordingly, this section offers improvements over [1] via utilizing kernel density estimators (KDE) [64] for VaR α 's estimates. Compared to crude Monte Carlo (MC) methods, KDE can handle arbitrary complex distributions, capture local statistics well, and smoothen the cumulative function in a non-parametric way.EQUATIONwhere K : R d → R is a kernel function, e.g., the Gaussian kernel, K(x) = exp(-||x|| 2 /2) exp(-||x|| 2 /2)dx , and h ℓ is the smoothing bandwidth. Once the KDE is built, it enables access to the quantile from the cumulative distribution functions or numeric integrals. The following Theorem 4.4 shows that KDE serves as a reliable approximation for VaR α .Theorem 4.4. Let F -1 ℓ-KDE (α; θ) = VaR KDE α [ℓ(T , θ)] and F -1 ℓ (α; θ) = VaR α [ℓ(T , θ)].Suppose that K(x) is lower bounded by a constant, ∀x. For any ϵ > 0, with probability at least 1ϵ, we can have the following bound:EQUATION)As implied, one can close the distribution approximation gap by adopting a smaller, more flexible bandwidth. Additionally, KDE models offer a smooth estimate of the cumulative distribution function and require no prior assumptions. Remark 1. In addition to smoothness, flexibility, and distribution agnostic traits, KDE in adoption can enhance the studied method's generalization capability. The crude Monte Carlo used in [1] typically incurs an error of approximately O( 1 √ B ) in estimating quantiles [65] . In contrast, that of KDE is no more than O( h ℓ √ B * log B ) from Theorem 4.4. 
empirical findings: 
 Prior sections mainly focus on the theoretical understanding of two-stage distributionally robust strategies. This section conducts extensive experiments on a broader range of benchmarks and examines the improvement tricks, e.g., the use of KDE for quantile estimates, from empirical results.Benchmarks & baselines. We perform experiments on the few-shot regression, system identification, image classification, and meta reinforcement learning, where most of them keep setups the same as prior work [1, 42] . We evaluate the methods from risk minimization principles and corresponding indicators, including expected/empirical risk minimization (Average), worst-case risk minimization (Worst), and tail risk minimization (CVaR α ).MAML mainly works as the base meta learner, and we term the KDE-augmented DR-MAML as DR-MAML+. Then we compare DR-MAML+ with several baselines, including vanilla MAML [25] , TR-MAML [42] , DRO-MAML [66] and DR-MAML [1] .Worst CVaRα The goal of the sinusoid regression [25] is to quickly fit an underlying function f (x) = A sin(x -B) from K randomly sampled data points, and tasks are specified by (A, B). The meta-training and testing setups are the same as that in [1, 42] , where many easy functions with a tiny fraction of difficult ones are included in the training.Result & analysis. As illustrated in Fig. 3 , we can observe that DR-MAML+ consistently outperforms all baselines across average and CVaR α indicators in the 5-shot case. Though the average performance slightly lags behind DR-MAML in the 10-shot case, DR-MAML+ surpasses other baselines in both the Worst and CVaR α indicators. This implies that DR-MAML+ exhibits more robustness in challenging task distributions, e.g., 5-shot case. Furthermore, the standard error associated with our method is significantly smaller than others, underscoring the stability of DR-MAML+.The system identification corresponds to learning a dynamics model from a few collected transitions in physics systems. Here, we consider the Pendulum system and create diverse dynamical systems by varying its mass m and length l, with (m, l) ∼ U([0.4, 1.6], [0.4, 1.6]). A random policy collects transitions for meta training, and 10 random transitions work as a support dataset.Result & analysis. Fig. 4 shows no significant difference between 10-shot and 20-shot cases. DR-MAML+ dominates the performance across all indicators in both cases. Due to the min-max optimization, TR-MAML behaves well in the worst-case but sacrifices too much average performance. Within the studied strategies, DR-MAML+ exhibits an advantage over DR-MAML regarding CVaR α .We perform few-shot image classification on the mini-ImageNet dataset [67] , with the same setup in [42] . The task is a 5-way 1-shot classification problem. And 64 classes are selected for constructing meta-training tasks, with the remaining 32 classes for meta-testing. Result & analysis. In Table 1 , methods within a two-stage distributionally robust strategy, namely DR-MAML and DR-MAML+, show superiority to others across all indicators in both training and testing scenarios, which is similar to empirical findings in [1] . Interstingly, DR-MAML+ and DR-MAML are comparable in most scenarios, and we attribute this to the small batch size in training, which weakens KDE's quantile approximation advantage. Here, we take 2-D point robot navigation as the meta reinforcement learning benchmark in evaluation. The goal is to reach the target destination with the help of a few exploration transitions for fast adaptation, and we retain the setup in MAML [25] . In meta testing, we randomly sample 80 navigation goals and examine methods' navigation performance. Result & analysis. As reinforcement learning methods fluctuate fiercely in worst-case indicators, we only report Average and CVaR α returns in Table 2 . We observe that using studied strategies in DR-MAML enhances the returns. DR-MAML+ benefits from a more reliable quantile estimate and achieves superior performance. The application of distributional robustness to reinforcement learning yields improvements in returns.With the meta trained model, e.g., DR-MAML+ in sinusoid regression, we collect the testing task risk values with different task batch sizes to estimate the VaR α from respectively the crude MC and KDE. As observed from Fig. 5 , the VaR α approximation error decreases with more tasks, and the KDE produces more accurate estimates with a sharper decreasing trend. The above well verifies the conclusion in Theorem 4.3.Here, we summarize two points from the above empirical results and associated theorems. (i) From Theorem 4.2/4.3 and Fig. 3 Improved Robustness in Evaluation: As illustrated in Fig. 6 , DR-MaPLe and DR-MaPLe+ consistently outperform baselines across both average and indicators in cases, demonstrating the advantage of the two-stage strategy in enhancing the robustness of few-shot learning. DR-MaPLe+ achieves better results as KDE quantiles are more accurate with large batch sizes. These results confirm the scalability and compatibility of our method on large models.Learning Efficiency as Limitations: In terms of implementation time and memory cost, we retain the setup the same as that in [1] : use the same maximum number of meta gradient updates for all baselines in training processes, which means given α = 0.5, the tail risk minimization principle requires double task batches to evaluate and screen sub-batches. It can be seen that both DR-MaPLe and DR-MaPLe+ consume more memories, and the extra training time over MaPLe arises from the evaluation and sub-batch screening in the forward pass. Such additional computations and memory costs raise computational and memory efficiency issues for exchanging extra significant robustness improvement in fast adaptation. 
conclusion: 
 To conclude, this paper proposes to understand the two-stage distributionally robust strategy from optimization processes, define the convergence solution, and derive the generalization bound in the presence of tail task risk. Extensive experiments validate the studied improvement tricks and reveal more empirical properties of the studied strategy. We leave computational overhead reduction as a promising topic for future exploration in robust fast adaptation.  
 
 Appendix: This section mainly includes explanations and clarifications on this work.Table 3 : A summary of robust fast adaptation methods. We take MAML as an example, list related methods, and report their characteristics in literature. We mainly report the statistics according to whether existing literature works include the generalization analysis and convergence analysis. The form of meta learner and the robustness type are generally connected.Robustness TypeMAML min θ∈Θ E p(τ ) ℓ(D Q τ , D S τ ; θ) ✓ ✓ -- DRO-MAML [66] max q(τ )∈Q min θ∈Θ E q(τ ) ℓ(D Q τ , D S τ ; θ) ✗ ✗ Uncertainty Set (Not tail risk) TR-MAML [42] min θ∈Θ max τ ∈T ℓ(D Q τ , D S τ ; θ) ✓ ✓ Worst-Case Task DR-MAML [1] min θ∈Θ E pα(τ ;θ) ℓ(D Q τ , D S τ ; θ) ✗ ✗ Tail Task Risk DR-MAML+(Ours) max q(τ )∈Qα min θ∈Θ E q(τ ) ℓ(D Q τ , D S τ ; θ) ✓ ✓ Tail Task RiskPrimary differences: As far as we know, literature work is quite limited regarding fast adaptation robustness in the task space. TR-MAML and DR-MAML are the most recent and typical ones that can handle task distributional shift scenarios well. As reported in Table 3 , TR-MAML only focuses on the worst-case, which considers a bit extreme and rarely occurred cases. DRO-MAML is a new baseline, where the uncertainty set Q is included for robust fast adaptation, hence there exists no theoretical analysis. As for the tail task risk, DR-MAML lacks generalization capability and convergence rate analysis w.r.t. the meta learner. The meta learner in DR-MAML+ is a more specific instantiation of that in DR-MAML. We claim that these theoretical understanding is necessary in the presence of the robust fast adaptation due to its potential applications in large models.Theoretical and empirical insights: In comparison, this work not only contributes to the Stackelberg game for estimates, but also derives the generalization and the asymptotic performance gap in iterations based on a normalized but non-differentiable probability density space. Note that we lean more focus on theoretical understanding and pursuing SOTA performance is not the ultimate purpose of this work. The connections between different quantile estimators and generalization bound highlighted in Theorem 4.3/4.4 and Remark 1 reveal the theoretical advantage of KDEs over crude Monte Carlo methods. This motivates us to replace crude Monte Carlo with KDEs. Such a replacement as a simple implementation trick is supported by rigorous theoretical analysis. The empirical results align with theoretical understanding.In terms of improving the studied strategy, investigations in extensive experiments seem meaningful for practical implementations, and some non-trivial discoveries together with improvement tricks are also reported, such as the relationship between quantile estimate errors and adaptation robustness, the batch size's influence on several benchmarks, etc.In this work, the theoretical and empirical parts are connected in an implicit manner. The generalization capability is empirically examined from experimental results, and the performance gap between DR-MAML+ and DR-MAML can be attributed to the difference in generalization bounds. As for the convergence trait and asymptotic performance, the insight might guide the optimization process in training large models, such as early stopping criteria design.As pointed out in [3, 61, 62] , large language models are few-shot learners. When a large model, such as a large decision-making model in the future, comes into practice, fast adaptation robustness can be a crucial issue as real-world scenarios are indeed risk-sensitive.This work takes the latest work [1] as an example, and the interest is in the theoretical aspect. Most of the assumptions in this work are from [1] . The baselines are typical and latest, while the benchmarks cover diverse downstream tasks. In multimodal few-shot image classification experiments, our contributed points help guide the development of large models in terms of training and robustness enhancement. Our investigations also provide insight into robust policy optimization, particularly when safety is one necessary consideration [70] [71] [72] . [73] is a comprehensive resource that addresses stochastic modeling and optimization methods, but it does not explore solution concepts in game theory or define generalization bounds relevant to meta learning and deep learning. Instead, our work further enriches the stochastic programming theory in meta learning, connects it to the Stackelberg game, and contributes to tail risk generalization bounds, convergence rates, asymptotic properties, and so on. Therefore, the solution concept and the theoretical properties are specific to our meta learning setup, distinctly from the scope covered by [73] .Illustration of VaR α , CDF and others: Fig. 7 illustrates a typical probability distribution, cumulative distribution, and the resulting mean, VaR α , and CVaR α . Given α ∈ [0, 1), VaR α is the α quantile of the risk distribution. Specially, VaR 0.5 coincides with the mean. Upon the definition of VaR α , CVaR α can be define as CVaR α = E p(τ ) ℓ|ℓ ≥ VaR α . That is, CVaR α is the expectation of the risks of the 1α tail of the distribution. Relative to the original probability distribution, CVaR α can be interpreted as a certain distribution shift, which reweighs arbitrary risk exceeding VaR α up to a coefficient 1 1-α . Meaning of the asymptotic performance gap: We plot Fig. 8 to display the gap between the CVaR α value in iterations and that in the convergence. The area difference depicts this gap.Analyzing computational complexity across all meta-learning methods is inherently challenging due to the diversity in methodological approaches within the field. Meta-learning encompasses a wide range of techniques, including gradient-based methods, which rely on iterative updates to model parameters, and non-parametric methods, which may instead focus on instance-based learning or kernel-based approaches. Therefore, the space complexity is specific to the meta-learning method, while this work is agnostic to it. Here, we report the computational complexity for the DR-MAML+ as O B(B -α|M|) while using KDE with the Gaussian kernel, and that of DR-MAML is O B(log(B) -α|M|) .This paper presents work whose goal is to advance the field of robust meta learning. There are many potential societal consequences of our work, which we detail as follows.The fast adaptation robustness is an urgent concern, particularly in large models and risk-sensitive control. This work provides versatile insights for theoretical analysis and performance improvement in the presence of tail task risk, and future explorations can be decision-making scenarios, such as multi-agent policy optimization [74, 75] , and computational/memory cost reduction.For a better understanding of the game theoretical optimization, we take DR-MAML+ and DR-CNP+ as examples and include the Pseudo Algorithms 1/2 in this section. Particularly, the algorithms specify the decision-making orders and highlight the use of KDE modules to build task risk value distributions and estimate the quantile. Without loss of generality, we can also express the process of solving the studied Stackelberg game as:max q∈Qα F(q, θ * (q)) s.t. θ * (q) = arg min θ∈Θ F(q, θ)(13a: Leader's Decision-Making) min θ∈Θ F(q, θ), (13b: Follower's Decision-Making)where the optimization w.r.t. (q, θ) is the computation of the best responses for two adversarial players. As a bi-level optimization in Eq. (14a)/(14b), the exact solution is intractable to obtain in a theoretical sense, and the two-stage distributionallly robust optimization is a heuristic approach.Meaning of the obtained equilibrium. Here, we can interpret the obtained solution (q * , θ * ) from solving Eq. ( 7) as follows. Given the follower's decision θ * and the induced task risk distribution F ℓ (l; θ * ), the leader cannot further raise a proposal of a task subset with a probability 1α to degradde the tailed expected performance. And this explains the meaning of robust fast adaptation solution w.r.t. the tail task risk. B = {ℓ(D Q τi ; θ i )} B i=1 ; Estimate VaR α [ℓ(T , θ)] and set ξ = ξα in Eq. ( 5) with kernel density estimators; Screen the subset L B = {ℓ(D Q τi ; θ i )} K i=1 with ξα for meta initialization updates; # The Follower Player's Decision-Making Execute outer loop via gradient descent to increase adaptation robustness: τi ; z, θ)} K i=1 with ξα for meta initialization updates; # The Follower Player's Decision-Making Execute gradient descent to increase adaptation robustness:EQUATIONθ ← θ -λ∇ θ K i=1 ℓ(D Q τi ; z, θ); end C.Proposition 1. The uncertainty set Q α is convex and compact in terms of probability measures.Proof: We firstly focus on the convexity of Q α . For any {q 1 := q 1 (τ ), q 2 := q 2 (τ )} ∈ Q α , we partition these two task spaces with non-zero sampling probability mass respectively as T 1 ∪ T C and T 2 ∪ T C . As displayed in Fig. 9 , T C denotes the shared subset task between q 1 and q 2 . Below we show that λ 1 q 1 + λ 2 q 2 ∈ Q α with λ 1 + λ 2 = 1. This is true becauseEQUATION)We next demonstrate the compactness of Q α . The distance between two distributions ∀{q 1 , q 2 } ∈ Q α can be defined as:d Qα (q 1 , q 2 ) := τ ∈T q 1 (τ ) -q 2 (τ ) dτ.Since L 1 space is a Banach space, the compactness is equivalent to the closedness and Boundedness of Q α . Considering a sequence {q n (τ ) ∈ Q α } with the resulting limitation is q * (τ ), following the Controlled Convergence Theorem [76] , we know thatEQUATION)Due to the symmetry of the distance, we can have lim n→∞ τ ∈T p * (τ )p n (τ )dτ ≤ 0. Thus, Proof: Note that Θ is compact as a subspace of the Euclidean space. And it is trivial to see that F(q, θ) := E q ℓ(D Q τ , D S τ ; θ) is continuous w.r.t. θ ∈ Θ as ℓ satisfies the β τ -Lipschitz continuity in the Assumption 1.τ ∈T p * (τ )dτ = lim n→∞ τ ∈T p n (τ )dτ = 1 -α. That is, p * (τ ) ∈ Q α , indicating that Q α is a closed set.Here we need to show the continuity of F(q, θ) w.r.t. the collection of probability measures or probability functions Q α . To this end, with ∀θ ∈ Θ fixed, We consider two metric spaces (Q α , d Qα ) and (L, R L ). The map of our interest is g(q) = F(q, •) :Q α → L ⊆ R + .Figure 9 : Partition of the task subspace. Here we take two probability measure {q 1 , q 2 } ∈ Q α for illustration. T 1 ∪ T C and T 2 ∪ T C defines the corresponding task subspaces for q 1 and q 2 with non-zero probability mass in the whole space T .Naturally, we can have the following inequality:EQUATIONEQUATIONwhich implies 3L max -Lipschitz continuity of g(q) w.r.t. ∀q ∈ Q α .According to the Remark in [77] , there always exists the global Stackelberg equilibrium as the Definition 1 when Q α × Θ is compact and F(q, θ) is continuous. This completes the proof of Proposition 2. ■ C.5 Proof of Quasi-concavity for F(q, θ) w.r.t. qIt can be validated that F(q, θ) is a quasi-concave function w.r.t. q, meaning that for any positive number l ∈ R + , the set {q|q ∈ Q α , F(q, θ) > l} is convex in Q α .Proof: According to the conventional definition (i.e., the superlevel set is convex [78] ), for all λ 1 + λ 2 = 1, q 1 , q 2 ∈ {q|F(q, θ) > l}, we can haveF(λ 1 q 1 + λ 2 q 2 , θ) = E λ1q1+λ2q2 ℓ(D Q τ , D S τ ; θ) (18a) = λ 1 E q1 ℓ(D Q τ , D S τ ; θ) + λ 2 E q2 ℓ(D Q τ , D S τ ; θ) (18b) = λ 1 F(q 1 , θ) + λ 2 F(q 2 , θ) (18c) > l.(18d) Thus, λ 1 q 1 + λ 2 q 2 ∈ {q|F(q, θ) > l} and the superlevel set is convex, implying that F(q, θ) is quasi-concave w.r. ≤ 1, and the iteration converges with the rate ||I -λ∇ 2 θθ F(q * , θ * )|| 2 + λβ q β h . Proof: Let the resulting stationary point be [q * , θ * ], we denote the difference terms by q = qq * and θ = θθ * . Then, according to the optimization step, we can have the following equations:EQUATION)Now we perform the first-order Taylor expansion of the θ related function ∇ θ F(q t ; θ) around θ * and can derive:EQUATION)Then we have the following result with the help of Assumption 2:EQUATION)With Eq. ( 19), Eq. ( 20) and Eq. ( 21), we can derive the equation that:θt+1 = θt -λ∇ θ F(q t ; θ t ) (22a) = θt -λ ∇ θ F(q t ; θ * ) + ∇ 2 θθ F(q t ; θ * ) θt (22b) = I -λ∇ 2 θθ F(q t ; θ * ) θt -λ∇ θ F(q t ; θ * ) (22c) =⇒ || θt+1 || 2 ≤ ∥I -λ∇ 2 θθ F(q t ; θ * )∥ 2 || θt || 2 + λ∥∇ θ F(q t ; θ * )∥ 2 (22d) ≤ ∥I -λ∇ 2 θθ F(q t ; θ * )∥ 2 + λβ q β h ||∥ θt ∥ 2 (22e)Thus, when ∥I -λ∇ 2 θθ F(q * ; θ * )∥ 2 < 1λβ q β h , we haveEQUATION)This completes the proof of Theorem EQUATIONwhereT 1 = {τ : ℓ * < VaR * α , ℓ meta ≥ VaR meta α }, T 2 = {τ : ℓ * ≥ VaR * α , ℓ meta < VaR meta α }.Proof. Given a batch of tasks {τ 1 , • • • , τ B } and according to the definition of CVaR α , we haveEQUATIONIn inequality (25f), T 1 = {τ : ℓ * < VaR * α , ℓ meta ≥ VaR meta α }, T 2 = {τ : ℓ * ≥ VaR * α , ℓ meta < VaR meta α }, T 3 = {τ : ℓ * < VaR * α , ℓ meta < VaR meta α }, T 4 = {τ : ℓ * ≥ VaR * α , ℓ meta ≥ VaR meta α }.EQUATIONwhere the inequality holds with probability at least 1ϵ and ϵ ∈ (0, 1), V[•] denotes the variance operation, and L max is from the Assumption 1.Proof. R(θ * ) -R(θ * ) can be decomposed to two parts, i.e.,EQUATIONFor the first part (i.e., R(θ * ) -R w (θ * )), we will adopt the Bernstein's inequality to provide an upper bound. Regarding pα(τi) p(τi) ℓ(D Q τi , D S τi ; θ * ) -R(θ * ) as a random variable with respect to τ i and according to Assumption 1, we know thatEQUATIONThus, following Bernstein's inequality [79] , we know thatEQUATIONwhereEQUATIONEQUATIONSetting ϵ to match the upper bound in inequality (29c) shows that with probability at least 1ϵ, the following bound holds:EQUATIONFor the second part (i.e., R w (θ * ) -R(θ * )), we haveEQUATIONIn summary, we can obtain an upper bound of R(θ * ) -R(θ * ). That is,R(θ * ) -R(θ * ) ≤ R w (θ * ) -R(θ * ) + R w (θ * ) -R(θ * ) ≤ 2( α 1-α L 2 max + V τ ∼pα(τ ) ) ln 1 ϵ B + 2L max ln 1 ϵ 3(1 -α)B + α 1 -α L max B B i=1 δ(τ i ) ≤ 2( α 1-α L 2 max + V τ ∼pα(τ ) ) ln 1 ϵ B + 1 3(1 -α) L max B 2 ln 1 ϵ + 3αB .This completes the proof of Theorem 4.3. ■ C.9 Proof of Theorem 4.4Theorem 4.4 Let F -1 ℓ-KDE (α; θ) = VaR KDE α [ℓ(T , θ)] and F -1 ℓ (α; θ) = VaR α [ℓ(T , θ)].Suppose that K(x) is lower bounded by a constant, ∀x. For any ϵ > 0, with probability at least 1ϵ, we can have the following bound:EQUATIONProof. For any constant M , we firstly notice thatEQUATIONEQUATIONFor any θ and t, we haveEQUATIONwhere K min is the lower bound of the kernel function K(x), i.e., K(x) ≥ K min , ∀x.According to Theorem 3 of [80] , we know that for any ϵ > 0, with probability at least 1ϵ, the following inequality holds:EQUATIONwhereC is a constant. Let M = h ℓ C Kmin √ B * log B .Thus, the Eq. (35f) holds and we complete the proof of Theorem 4.4. ■To gain more theoretical insights into a popular meta-learning method-MAML [25] , we provide the following Theorem C.1. Before proceeding, we introduce some notations. During meta-training, a finite number of task instances are observed by first sampling a task from the distribution p(τ ). Each task D τi comprises a collection of m i data points {(D S i,j , D Q i,j )} mi j=1 , which are distributed over Z with each data point drawn from a distribution D i . For some risk ℓ, define the family of functionsF Z := {ℓ(θ -λ∇ θ ℓ(D S τi ; θ), D Q τi ) : θ ∈ Θ}.For each task D τi , the Rademacher complexity of F on m i samples isEQUATIONwhere the ϵ j 's are Rademacher random variables. LetF i (θ) = E Di ℓ(θ -λ∇ θ ℓ(D S i,j ; θ), D Q i,j ), Fi (θ) = 1 mi mi j=1 ℓ(θ -λ∇ θ ℓ(D S i,j ; θ), D Q i,j). Denote by θ * the optimal model parameter under the two-stage algorithm. Theorem C.1 provides generalization of the algorithm to new tasks. Theorem C.1 (Generalization Bound for MAML in the Tail Risk Cases). For a new task τ B+1 with distribution D B+1 , if D B+1 = B i=1 a i D i , then with probability at least 1δ for any δ > 0, we can haveEQUATIONProof. The proof consists of two parts. We first explore the generalization to new instances of previously-seen tasks. Then we solve the generalization to new tasks.Step 1. For any sample setA = {(D S i,j , D Q i,j )} mi j=1 , define Φ(A) = sup ℓ∈F Z F i (θ) -Fi (θ). Let A and A ′ := {((D S i,j ) ′ , (D Q i,j ) ′ )} mi j=1be two samples that differ by exactly one point. According to the fact supx f (x) -sup x g(x) ≤ sup x (f (x) -g(x)), we know that Φ(A ′ ) -Φ(A) ≤ 1 mi due to the difference in exactly one point. Similarly, we can obtain Φ(A) -Φ(A ′ ) ≤ 1 m , thus Φ(A) -Φ(A ′ ) ≤ 1m . Following McDiarmid's inequality, for any δ > 0, with probability at least 1 -δ 2 , we haveEQUATIONWe next bound the expectation of the right-hand side of inequality (39) as follows:EQUATIONEQUATIONEQUATION= E A,A ′ ,ϵ sup ℓ∈F Z 1 m i mi j=1 ϵ j ℓ(D S i,j , D Q i,j ) -ℓ((D S i,j ) ′ , (D Q i,j ) ′ ) ≤ E A,ϵ sup ℓ∈F Z 1 m i mi j=1 ϵ j ℓ(D S i,j , D Q i,j ) + E A ′ ,ϵ sup ℓ∈F Z 1 m i mi j=1 ϵ j ℓ((D S i,j ) ′ , (D Q i,j ) ′ ) = 2E A,ϵ sup ℓ∈F Z 1 m i mi j=1 ϵ j ℓ(D S i,j , D Q i,j ) = 2R mi (F Z ).Eq. ( 40) uses the law of total expectation. Inequality (41) holds by Jensen's inequality and the convexity of the supremum function. In Eq. ( 42), ℓ(D S i,j , D Q i,j ) := ℓ(θ -λ∇ θ ℓ(D S i,j ; θ),D Q i,j ),where (D S i,j , D Q i,j ) ∈ A. Following inequality (39) , we can know thatEQUATION)Step 2. Since the new distribution D B+1 is the convex combination ofD i , ∀i = 1, • • • , B, we have F B+1 (θ) = B i=1 a i F i (θ).Accordingly, with probability at least 1δ over the choice of samples used to compute F (θ),EQUATIONwhich yields thatEQUATIONIn summary, the two steps complete the proof of Theorem C. Here, we illustrate all meta learning benchmark purposes in Fig. 10 , which includes sinusoid regression, pendulum system identification, few-shot image classification, and meta reinforcement learning. We no longer run experiments on the Omniglot dataset, as most baselines can achieve SOTA performance and cannot tell the difference well from the openreview of [1] . Sinusoid regression: In [1, 42] , a lot of easy tasks and limited challenging tasks are sampled for meta-training, with the tasks from the whole space employed in evaluation. The default range of the phase parameter is B ∈ [0, π], while those of the amplitude are A ∈ [0.1, 1.05] for easy tasks and A ∈ [4.95, 5.0] for challenging tasks. Generally, sinusoid functions with larger amplitudes are hard to adapt from a few support data points. The mean square error works as the risk function to measure the gap between the predicted value f (x) and the actual value. We set the task batch 50 for 5-shot and 25 for 10-shot, and the maximum iteration number is 70000. We refer the reader to TR-MAML and DR-MAML for all of the setups.We retain the neural architectures [42, 1] in for all MAML like methods. In detail, all methods take a multilayer perceptron with two hidden layers and 40 ReLU activation units in each layer. The inner loop is achieved via one stochastic gradient descent step. As for CNP like methods, please refer to the vanilla set-up in [15] (The Github link is attached here: https://github.com/ google-deepmind/neural-processes).As the task space is hugh, there is no way to exactly estimate the risk quantile. Hence, the Oracle quantile in Fig. 5 is roughly computed from the sampled 100 tasks given the pretrained DR-MAML+. The rationale behind this operation is that increasing the population number in statistics reduces the quantile estimate bias.System identification: The pendulum system is a classical environment in the OpenAI gym (environment details are: https://github.com/openai/gym/blob/master/gym/envs/classic_ control/pendulum.py), and it is an actuated joint with one fixed end. The goal of system identification for the pendulum system is to predict the state transition given arbitrary actions with several randomly collected transitions as the support dataset. The observation is a tuple in the form (cos θ, sin θ, θ ′ ), where θ ∈ [π, π]. The action is in the range a ∈ [-2.0, 2.0] and the torque is applied to the pendulum body. The mass m and the length l of the pendulum follows a uniform distribution (m, l) ∼ U([0.4, 1.6] × [0.4, 1.6]), sampled variables configure a Markov decision process as the task. In each batch, there are 16 tasks, and each task comprises 200 data points. Specifically, 10 few-shot data points are randomly sampled to enable system identification per task, denoted as 10-shot. For 20-shot cases, the number of data points in support dataset is 20. And the maximum iteration number is 5000.For all MAML-like methods, the neural architecture used here is a multilayer perceptron with three hidden layers of 128 hidden units each and the activation function is ReLU. The learning rate for both the inner and outer loops is set at 1e-4.The few-shot image classification is mostly described as an N-way K-shot classification, where N classes with K-labeled instances for each are considered. The dataset is organized in the same manner as that in [81, 42, 1] : These include 64 classes for meta-training, with the rest 36 classes for meta-testing. We generate each task in the way: 8 meta-training tasks from the class {6, 7, 7, 8, 8, 9, 9, 10} are randomly generated from 64 meta-training classes; the remaining classes are organized similarly. As a result, each task is constructed from sampling one image from five classes, corresponding to a 5-way 1-shot problem. The task batch is set 4 with a maximum number of iterations of 60000 in meta-training.For all MAML-like methods, the neural architecture used here is a four-layer convolutional neural network for the mini-ImageNet datasets. The inner loop is achieved via one stochastic gradient descent step. We refer the reader to TR-MAML and DR-MAML for all of the setups (The Github link is attached here https://github.com/lgcollins/tr-maml).Meta reinforcement learning: 2D Navigation is a classical meta reinforcement learning benchmark where efficient explorations matter. The task in 2D Navigation is to guide the point robot to take move actions for a purpose of reaching a specific goal location from the step-wise reward. The reward the agent receives from the environment is based on the distance to the goal, and 20 episodes work as the support dataset for navigation fast adaptation. In terms of the task distribution, we sample tasks from a uniform distribution U([-0.5, 0.5] × [-0.5, 0.5]) over goal locations.As for the neural architecture for policy network set-ups, we refer the reader to vanilla MAML (Github link is attached here https://github.com/tristandeleu/pytorch-maml-rl) and CAVIA (The Github link is attached here https://github.com/lmzintgraf/cavia/tree/master/rl). And trust region policy optimization works for policy optimization. This subsection includes the impelementation of KDE for the studied strategy. Here, the example of the hinge loss is illustrated as follows. Due to the page limit in the main paper, we include additional experiments and corresponding results in this section.In addition to MAML, we apply a similar modification to CNP, which results in TR-CNP, DRO-CNP, DR-CNP, and DR-CNP+ (DR-CNP with KDE for VaR α estimates). We report the meta testing results on sinusoid regression and pendulum system identification benchmarks.As illustrated in Table 5 /6, all methods achieve comparable average performance in sinusoid and pendulum system identification. Regarding CVaR α , DR-CNP's improvement is relatively marginal over others except DR-CNP+. Compared to MAML, CNP seems more sensitive to quantile estimate accuracies when meeting with the studied strategies. We can attribute the performance differences of the two methods to the cumulative quantile estimation errors using the crude MC. Even though the quantile estimation error in Fig. 5 difference is tiny in each step, the cumulative error indeed affects the converged equilibrium a lot. This reflects the advantage of the KDE's used in DR-MAML+ when the task batch size cannot be set larger in practice.We also investigate the task risk value distributions in pendulum system identification. To this end, we visualize one run testing results for all methods in Fig. 11 . It seems DR-MAML+'s result is more skewed to the left than others.Fig. 12 displays all methods' performance w.r.t. the average and CVaR α returns along the metatraining process. We exclude TR-MAML in visualization due to its worse performance and unstable training properties. We can find that the DR-MAML exhibits a fast performance rise at the early stage but its capability to continuously improve diminishes over time. DR-MAML+ consistently outperforms other baselines in most cases. The above suggests that the KDE module achieves performance gains over the crude MC when implemented with the two-stage distributionally robust strategy for meta RL scenarios.To reveal the impact of confidence levels on model performance, we perform a sensitivity analysis with respect to confidence levels. Since only DR-MAML and DR-MAML+ are influenced by the confidence levels during the distributionally robust optimization across all baselines, we only compare the performance of the two methods to highlight the differences between them. As shown in Fig. 13/14, we can observe that in both sinusoid 5-shot and 10-shot tasks, as the confidence level varies, DR-MAML+ exhibits more stable performance than DR-MAML, indicating that DR-MAML+ has a lower sensitivity to confidence levels. It can be illustrated that the crude Monte Carlo used in DR-MAML is more unstable in terms of quantile estimation than the kernel density estimator used in DR-MAML+. This can be due to the fact that the crude Monte Carlo method is more likely to get stuck in the local optimal solution. In addition, it can be seen from Fig. 13 /14 that the performance of our developed DR-MAML+ is better than DR-MAML in most cases. DR-MAML+ exhibits lower mean squared errors than DR-MAML in the average, worst, and CVaR α indicators, demonstrating the advantages of more accurate quantile estimation in improving robustness. We demonstrate the adaptation risk landscape of meta-trained MAML [25] , TR-MAML [42] , DRO-MAML [66] , DR-MAML [1] and our DR-MAML+ in Fig. 15 . The adaptation risk landscape shows the superiority of our method in optimizing within the expected tail risk minimization. Compared to other methods, DR-MAML+ exhibits smoother and smaller risk profiles, illustrating its robustness even in challenging tasks.This work employs Pytorch [85] as the default deep learning toolkit when implementing the developed methods. As for baselines, TR-MAMAL follows the standard implementation as work [42] and runs with Tensorflow [86] . Others are implemented with Pytorch. All experimental results are computed by NVIDIA RTX6000 GPUs and A800 GPUs.NeurIPS Paper ChecklistThis section presents theoretical insights into two-stage optimization strategies. We perform analysis from the algorithmic convergence, the asymptotic tail risk robustness, and the cross-task generalization capability in meta learning. 


Decision: This paper investigates tail risk minimization in meta-learning, providing theoretical analysis and a practical enhancement to the DR-MAML method. Reviewers generally found the paper technically sound with a good presentation and valuable theoretical contributions, including the Stackelberg game equivalence, convergence rate analysis, and generalization bounds. 
There were few initial concerns regarding the limited scope and incremental nature of the contribution (6Ej4), the fact that the practical benefits were also seen as tangential to the main theoretical contribution (ci3Y), and limited experimental evaluation that lacking comparisons to SOTA methods and larger models (6Ej4). These were subsequently addressed by authors during rebuttal stage by clarifying the broader applicability of their theoretical analysis and providing additional analysis / experiments. 
Overall, The paper presents a solid theoretical investigation of tail risk minimization in meta-learning and demonstrates its practical benefits. The authors effectively addressed initial concerns regarding scope, evaluation, and applicability to large models, leading to a consensus among reviewers on acceptance.

Reviews: review: paper_summary: paper_summary: The paper provides theoretical investigations for better understanding of an existing method in literature that focuses on minimizing expected tail risk. Equivalence of the algorithm is shown to a Stackerlberg game, which allows to study its convergence rate and asymptotic bounds on performance and generalization are provided. Finally, a practical heuristic is suggested to obtain performance improvements over the base method.
strength_weakness: strength_weakness: Clarity:  
- It is not clear how the expression for the expected tail risk minimization is obtained in (3); the reviewer had to refer to the corresponding reference to understand it and thus, including it would help make the paper self-contained.
- Figure 1 is not clear; the font size needs to be increased.
questions: questions: - What is the overhead induced by searching for the appropriate value of hyperparameter $\alpha$, compared to other baselines?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes an enhancement to the previous work termed DR-MAML by reformulating it as a Stackelberg game. Theoretical investigations regarding its solution concept, convergence rate, and generalization bound are provided. Numerical experiments demonstrate the improved robustness of the proposed method.
strength_weakness: strength_weakness: 1. The contribution of this work is incremental and exclusive to DR-MAML, which limits its scope and broader applicability. 
2. The writing is hard to parse, and several notions are pretty vague. For instance, in lines 95-96, $F_{\ell}^{\alpha}$, $\Omega_{\alpha,\tau}$, and $p_{\alpha}$ are defined through illustration in a figure instead of mathematical expressions. 
3. Experiments are limited and lack comparison to SOTA methods. It is recommended comparing DR-MAML+ with popular meta-learning methods (such as MetaCurvature and MetaOptNet) on open-source benchmark datasets including tieredImageNet, CUB-200-2011, and MetaDatasets. In addition, the improvement in Table 1 and Table 2 are rather marginal (no more than 0.6%). 
4. There is no comparison of time and space complexity, and the scalability to large NNs like ResNet is unknown.
5. The last statement of Theorem 4.1 is imprecise. According to Eq. (22), this convergence rate solely holds true when $t$ approaches infinity. 
6. The paper contains numerous typos. For instance, in line 4, the citation is not compiled. In line 95, "the resulling" should be corrected to "resulting." Additionally, in lines 96, 130, 240, 241, 244, 246, 254, 265, 283, 295, 303, 319, 328, 332, and 335-337, the cross-references are in the equation style instead of the correct format.
questions: questions: See above.
limitations: limitations: No explicit discussion on limitations are provided.
paper_summary: paper_summary: This work investigates the tail risk minimization in meta-learning from theoretical and practical perspectives. Overall, this work is well-written, novel and theoretically enriches TR-MAML[1]/DR-MAML[2].


In the realm of large models, meta-learning plays a crucial role due to the pressing concern of distributional robustness across various tasks, particularly in risk-sensitive applications. 
Here, I express my positive attitude based on this work's completeness, novelty, and workload.

[1] Collins, L., Mokhtari, A., & Shakkottai, S. (2020). Task-robust model-agnostic meta-learning. Advances in Neural Information Processing Systems, 33, 18860-18871.

[2] Wang, Q., Lv, Y., Xie, Z., & Huang, J. (2024). A simple yet effective strategy to robustify the meta learning paradigm. Advances in Neural Information Processing Systems, 36.
strength_weakness: strength_weakness: No concrete weakness.
questions: questions: With complete proofs and empirical evaluations, I do not have too many concrete questions except for some discussions. 

(1) Both group DRO and tail risk minimization handle the distribution shift in robust optimization; what are the advantages or disadvantages between them in meta-learning? 

(2) What difficulties will we encounter if we adopt the developed strategy in the optimization of large models?
limitations: limitations: None

Comments: title: Thanks for the update
Thanks for feedback
Thanks for precious feedback
Acknowledgement of Rebuttal
Thanks for the rebuttal and update the review
comment: Your suggestions help improve our manuscript a lot, and we express gratitude for your support.
Thank you for the elaborations and further experiments. I have no other concerns and I will update my rating.
We're happy to see most of the concerns are addressed, and we express sincere gratitude for the helpful feedback and suggestions. The following answers other questions: 

1. We apologize for not clearly explaining that this work is agnostic to meta-learning methods. 

- Eq. (3) is not specifically for DR-MAML but is a general form of the risk function $\ell(\mathfrak{D}\_{\tau}^{Q},\mathfrak{D}\_{\tau}^{S};\theta)$ used in typical meta-learning methods. **Eq. (4)** is an example of Eq. (3) and is specific to DR-MAML. In this case, the risk function becomes $\ell(\mathfrak{D}\_{\tau}^{Q};\theta-\lambda\nabla\_{\theta}\ell(\mathfrak{D}\_{\tau}^{S};\theta))$, implying the implementation of MAML. **Algorithm 2 on Lines 661-662** shows the application of another method CNP, where the loss function is specifically $\ell(\mathfrak{D}\_{{\tau}}^{Q};z,\theta)$, with $z=h\_{\theta\_1}(\mathfrak{D}\_{\tau}^{S})$.

- The theoretical insights in Section 4 are considered in a general form of meta learning with risk function $\ell(\mathfrak{D}\_{\tau}^{Q},\mathfrak{D}\_{\tau}^{S};\theta)$, not limited to MAML cases.
Besides, we take MAML as a specific example for conducting theoretical analysis in Appendix Theorem C.1 on Lines 779-782.

2. The practical implementation time/space comparisons. We keep the setup the same as that in [1]:  use the same maximum number of meta gradient updates for all baselines in training processes (this means given $\alpha=0.5$, the tail risk minimization principle requires double task batches to evaluate and screen sub-batches).  For practical implementation, we take vanilla MaPLe, DR-MaPLe (MC augmented tail risk minimization), and DR-MaPLe+ (KDE augmented tail risk minimization) in tieredImageNet few-shot classification as the example to report. The Table below reports the overall training time and memory (the vanilla MaPLe serves as the anchor point, and + means additional costs from the two-stage operation).

| -                        | MaPLe | DR-MaPLe | DR-MaPLe + |
|--------------------------|------|---------|----------|
| Implementation time      |   2.1h  |    +1.7h  |    +1.7h      |
| Memory     |   41.57G  |    +36.84G  |    +36.84G      |

Despite the more complex quantile estimations, DR-MaPLe+ does not show significantly higher time and space consumption compared to DR-MaPLe. It can be seen that both DR-MaPLe and DR-MaPLe+ consume more memories, and the extra training time over MaPLe arises from the evaluation and sub-batch screening in the forward pass. Such additional computations and memory costs bring significant robustness improvement, which can be crucial in risk-sensitive fast adaptation. We'll also include this point as a potential limitation in the Appendix.

[1] Wang, Qi, et al. "A simple yet effective strategy to robustify the meta learning paradigm." NeurIPS 2023.


---

*Feel free to let us know if these questions have been answered well, and we’re happy to engage in further discussions. We’ll incorporate these precious suggestions into the manuscript. It would be appreciated if you could update the scores after the concerns are addressed. Your help means a lot to us.*
Thank you for providing the detailed response and additional experiments, which addressed most of my concerns. 

1. Regarding the scope, could you please elaborate more on why this paper is "meta-learning method agnostic." The CVaR objective Eq. (3) of this work is specific to DR-MAML [1], and is not applicable to generic meta-learning approaches. Moreover, the theoretical investigations in Section 4 are also tailored to DR-MAML [1]. 

2. Could you please also provide the practical implementation time (throughput) and space (GPU memory) comparisons?
Thanks for recognizing our primary theoretical contributions. 

---

You're right; the scope of this work is restricted to a theoretical understanding of tail risk minimization in meta learning. 

The practical benefits of quantile estimate are indeed inspired by the hypothesis in [1], and we (i) *establish rigorous theoretical analysis in* **Theorem 4.4** and **Remark 1**; (ii) *empirically conduct large-model few-shot learning experiments and verify the advantage of KDE over MC with larger batch size, complementing the theory*. 

Ref: [1] Wang, Qi, et al. "A simple yet effective strategy to robustify the meta learning paradigm." NeurIPS 2023.

---

*Finally, we're encouraged by your positive comments, and your valuable suggestions help improve the manuscript greatly. Many thanks.*
I thank the authors for responding to the reviewers comments and feedback, and I appreciate the additional evaluation on larger models. 

However, I am inclined to agree with the other reviewers regarding the limited scope of the work. Additionally, the practical benefits of the proposed method is largely tangential to the main theoretical contribution about the Stackerlberg game equivalence, and arises from improving the risk estimates using more sophisticated density estimation, something that is already noted in the original work it builds from [1]. For these reasons, I keep my score as is.

[1] Wang, Qi, et al. "A simple yet effective strategy to robustify the meta learning paradigm." NeurIPS 2023.
Thank you for your helpful suggestions and kindness. Your comments help improve the manuscript a lot.
After reading the rebuttal and other reviewers' comments, I add further reviews. 

(1) Overall, this is a little niche but comprehensive theoretical paper about the tail risk minimization for meta learning. 

(2) The extra effort made by the authors is impressive in the era of large models. These new results on tail risk minimization for MaPLe are inspiring and well answer my lasting question about the role of CLIP like models in few-shot prediction. I am happy to see the benefits of tail risk minimization in large-model augmented meta-learning and looking forward to the open-sourced code.

For the above, I thank the authors for the rebuttal and tend to raise my score as the bonus.