Title: Online Constrained Meta-Learning: Provable Guarantees for Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta-learning; generalization
Abstract: Meta-learning has attracted attention due to its strong ability to learn experiences from known tasks, which can speed up and enhance the learning process for new tasks. However, most existing meta-learning approaches only can learn from tasks without any constraint. This paper proposes an online constrained meta-learning framework, which continuously learns meta-knowledge from sequential learning tasks, and the learning tasks are subject to hard constraints. Beyond existing meta-learning analyses, we provide the upper bounds of optimality gaps and constraint violations produced by the proposed framework, which considers the dynamic regret of online learning, as well as the generalization ability of the task-specific models. Moreover, we provide a practical algorithm for the framework, and validate its superior effectiveness through experiments conducted on meta-imitation learning and few-shot image classification.

Online Constrained Meta-Learning: Provable Guarantees for Generalization

Abstract

Meta-learning has attracted attention due to its strong ability to learn experiences from known tasks, which can speed up and enhance the learning process for new tasks. However, most existing meta-learning approaches only can learn from tasks without any constraint. This paper proposes an online constrained metalearning framework, which continuously learns meta-knowledge from sequential learning tasks, and the learning tasks are subject to hard constraints. Beyond existing meta-learning analyses, we provide the upper bounds of optimality gaps and constraint violations of the deployed task-specific models produced by the proposed framework. These metrics consider both the dynamic regret of online learning and the generalization ability of the task-specific models to unseen data. Moreover, we provide a practical algorithm for the framework and validate its superior effectiveness through experiments conducted on meta-imitation learning and few-shot image classification.

1 Introduction

In the setting of learning from multiple tasks, a shared prior is extracted from commonalities of existing tasks, resulting in improved learning efficiency and prediction accuracy for task-specific models. Conventional approaches includes multi-task learning [11], lifelong learning [46, 45, 34]. These approaches extract the prior by vanilla learning approaches on the tasks and do not explicitly involve the task-specific learning in the prior extraction, as illustrated by [28]. In contrast, metalearning or learning-to-learn [51, 23, 47] learns a meta prior, usually a meta parameter, by evaluating and optimizing the meta-objective, which is defined by the performance of the task-specific model adapted from the prior. Meta-learning has been extended to accelerate reinforcement learning [23, 34], where the tasks are modeled by Markov decision processes.

Online meta-learning considers that the meta parameter is learned from tasks that are sequentially revealed. Papers [24, 1, 16] are shown to be no-regret with respect to the static comparator, i.e., the gap between the aggregate performance of the meta parameter sequence and that of any fixed meta parameter is sublinear in the number of revealed tasks. The online-MAML [24] and the MOML [1] adopt one-step gradient descent to adapt the task-specific parameter, and paper [16] adapts it by the ridge regression. A direct evaluation of meta-learning is the performance of the task-specific model adapted from the meta parameter on each individual task. The methods in [4, 3, 34] consider dynamic regret [56, 4], which compares the task-specific parameter adapted from the meta parameter with the optimal counterpart for each task. The dynamic regrets of all the algorithms are shown to be bounded when there are sufficiently many tasks.

In many applications of meta-learning, such as few-shot learning in computer vision [38] and robotics [43, 5, 39], a task-specific model is expected to be quickly adapted by using a small amount of

training data of a new task and generalize to the unseen data of the task. The generalization ability of the task-specific adaptation has been considered in [13, 20, 21, 31]. Most of them consider the task-specific model to be adapted using a small number of samples from the data distribution, and the generalization error is quantified by the optimality gap between the produced task-specific model and the task-specific model adapted using the best meta-parameter. In contrast, in this paper, we explore a generalization error metric, which is defined as the optimality gap between the produced task-specific model and the optimal task-specific model, based on the entire data distribution. By directly quantifying the performance of the task-specific model on the entire data (including unseen data), this metric identifies the improvement in the model's generalizability for each specific learning task when the meta-parameter is employed. Despite its significance, this generalization metric is often overlooked in most meta-learning analyses. Generalization metrics that are similar to ours are evaluated in [15, 18] and discussed in [20].

Typically, meta-learning handles learning tasks as unconstrained optimization problems. Many real-world applications such as robot learning in [14, 54, 2, 29, 42], nevertheless, are subject to safety constraints that should be rarely violated. For this case, constrained meta-learning aims to learn meta-prior for constrained learning tasks, which are formulated as constrained optimization problems. As a result, in constrained meta-learning, besides the generalization with respect to the objective risk, the generalization with respect to constraint violation is also required to be considered during the task-specific adaptation. More specifically, in the constrained learning task, each constraint is also defined by the expectation over a data distribution, and only a small amount of training data sampled from the distribution is observable during the task-specific adaptation. Therefore, the adaptation can only guarantee the constraint satisfaction over the training data, while the constraint violation over the whole data distribution is required to be quantified.

Main contribution. In this paper, we develop the first approach for the online constrained metalearning problem. In particular, we consider a sequential task setting of constrained learning tasks. At each round, a learning task with hard constraints is revealed and is required to be solved, and only a small number of data for the revealed task is provided. To address the problem, we use the Follow-the-Perturbed-Leader (FTPL) approach to learn a meta-regularization at each round from the revealed tasks and apply the meta-regularization to solve the new task. The overall contributions are summarized as follows. (i) We apply constrained optimization with biased meta-regularization to adapt the task-specific parameter by using a small amount of data. Then, we optimize the metaobjective function, which is defined by the performance of the task-specific parameter on validation data. (ii) To handle the non-convex meta-objective function, we use the Follow-the-PerturbedLeader (FTPL) to update the meta-regularization from all the available tasks for the new task. (iii) We theoretically analyze and prove the upper bounds that (a) quantify both the optimality gaps on the objective functions and the constraint violations of task-specific models; (b) quantify the dynamic regrets where the comparator is dynamic and always optimal for each task; (c) quantify the generalization errors of task-specific models to the unseen data, where only a small number of training data are given in each task. The upper bounds show that the optimality gap between the optimal meta-parameter and that learned by our method decreases with a rate of O (1 / √ T ) . (iv) We develop a practical algorithm for the online constrained meta-learning that can be used effectively on large-scale problems, and empirically verify the efficacy of the proposed algorithm by conducting experiments on meta-imitation learning and few-shot robust image classification.

Table 1 compares the challenges of interests between this paper and previous works [4, 15, 24, 1, 18]. In particular, we consider the hard constraints in each learning task, the dynamic regret in the online meta-learning problem, and the generalization ability of the taskspecific models to unseen data, si-

multaneously. Even if the constraints are removed, this paper provides a more general theoretical analysis than most papers, as paper [15] quantifies the empirical loss over known training data, and does not consider the generalization, and papers [24, 1] consider a static regret.

Related works. Optimization-based meta-learning algorithms usually learn an optimization parameter as the meta prior, and each algorithm consists of a meta-algorithm, which learns the

meta parameter, and an inner-task or within-task algorithm, which adapts to a task-specific model based on the meta parameter. Based on the type of the within-task approach, the methods can be categorized into meta-initialization approaches [41, 23, 24, 4, 35, 33] and meta-regularization approaches [17, 15, 47, 32]. In meta-initialization approaches, the within-task algorithm uses the meta parameter as the initial point and takes a few optimization steps to minimize the empirical risk of the new task. For example, the MAML [23] algorithm takes one-step gradient descent as the within-task algorithm. In meta-regularization approaches, the within-task algorithm completely solves an optimization problem, where the meta parameter serves as the bias of the regularization term for the empirical risk. In our constrained meta-learning problem, the task-specific parameter should satisfy its given constraints. In the meta-initialization approaches, as the within-task algorithm only takes a few optimization steps, even if we do optimization to reduce the constraint violation, the solution is far from feasible. On the other hand, the meta-regularization approaches fully solve the within-task. To prioritize constraint satisfaction, we combine the meta-regularization approach with hard constraints as the within-task algorithm for constrained meta-learning.

Notations. Denote the l 2 norm of vectors and the spectral norm ( 2 -norm) of matrices by ∥ · ∥ . Consider multiple data distributions D = {D 0 , D 1 , . . . , D m } and multiple training datasets D tr = {D tr 0 , D tr 1 , . . . , D tr m } . We use D tr i ∼ D i to represent that each data z in the dataset D tr i is i.i.d sampled from the distribution D i , and D tr ∼ D to represent D tr i ∼ D i for all 0 ≤ i ≤ m . We use |D tr i | to denote the number of data in the dataset D tr i . A notation checklist is attached in Appendix A.

2 Problem Formulation

2.1 Constrained learning tasks and the sequential setting

At round t , an agent aims to solve a constrained learning task T t , which is formulated as the following constrained optimization problem:

where each data point is denoted by z = ( x, y ) ∈ Z with x ∈ X ⊆ R d x being the input and y ∈ Y ⊆ R d y being its corresponding output. The loss function ℓ 0 : R d ×Z → R is the performance metric of a model parameterized by θ ∈ Θ ⊆ R d and the constraint functions ℓ i : R d × Z → R , i = 1 , . . . , m evaluate the constraint metric of the model. Each performance or constraint metric ℓ i is computed on the data distribution D i,t , which is a joint distribution over input-output pairs z = ( x, y ) . The optimal solution of Problem (1) is denoted as θ ∗ t , and the feasible set of Problem (1) is denoted as K t ≜ { θ ∈ Θ | E z ∼D i,t [ ℓ i ( θ, z )] ≤ c i,t , ∀ i = 1 , . . . , m } . The constrained learning task T t is characterized by its data distributions D t = {D 0 ,t , D 1 ,t , . . . , D m,t } and constraint limits { c 1 ,t , . . . , c m,t } . In general, the data distributions D t are unknown to the agent. Instead, the training datasets D tr t = {D tr 0 ,t , D tr 1 ,t , . . . , D tr m,t } are sampled i.i.d from D t and available to the agent.

Consider that a set of constrained learning tasks are revealed sequentially , i.e., task T t is revealed at round t . In each round t , the agent updates a model once D tr t is obtained. The updated model is deployed and works on T t , i.e., predicting the outputs y of all requested inputs x . In the next round, the procedure repeats.

Since the data distributions D t are unknown, the agent cannot completely solve Problem (1) and obtain θ ∗ t . A common solution is to replace the data distribution D i,t in Problem (1) by the sampled dataset D tr i,t and solving the problem for each task T t . However, when the sample number in D tr i,t is small, separately solving T t with D tr i,t leads to limited performance. Similar to [4, 15], in this paper, the learning algorithm is expected to improve the performance of the solution for task T t by exploiting the correlation between the task T t and all previous tasks {T 1 , · · · , T t -1 } .

2.2 Metrics of a learning algorithm

Consider a learning algorithm produces a parameter sequence { θ 1 , · · · , θ T } for the task sequence {T 1 , · · · , T T } . At round t , the optimality gap R 0 ,t of θ t is defined as R 0 ,t ( θ t ) = max { E z ∼D 0 ,t [ ℓ 0 ( θ t , z ) -ℓ 0 ( θ ∗ t , z )] , 0 } . The violation for the i -th constraint denoted by R i,t is defined as: R i,t ( θ t ) = max { E z ∼D i,t [ ℓ i ( θ t , z )] -c i,t , 0 } , i = 1 , . . . , m. The task-averaged optimality gap (TAOG) denoted by ¯ R 0 , [1: T ] and the task-averaged constraint violation (TACV) denoted

by ¯ R i, [1: T ] of the algorithm after T rounds are defined as

Under the sequential task setting, the goal of a learning algorithm is to minimize the TAOG and the TACV of the parameter sequence { θ 1 , · · · , θ T } . The definitions of the TAOG and the TACV consider both the generalization ability of θ t to the unseen data of each revealed task T t , and a dynamic notion of the optimality regret. First, the optimality gap R 0 ,t and the constraint violation R i,t are defined by the expectations over the inaccessible data distribution D i,t , instead of the empirical risk over the given dataset D tr i,t that is used in online meta-learning analysis [4, 35]. Thus, the TAOG and the TACV can be used to quantify the generalization errors. It is more challenging to minimize them, as θ t is expected to generalize to the unseen data in D i,t while only the training datasets D tr i,t are given, and their size could be small. Second, unlike the fixed comparator used in the FTML algorithm [24], the comparator θ ∗ t is dynamic and always optimal for each task T t . As pointed out in [4, 34, 56], one cannot achieve the dynamic regret sublinear in T for either online learning or online meta-learning.

2.3 Task dissimilarity

As mentioned in Section 2.1, we expect that the performance produced by the learning algorithm, i.e., the TAOG and the TACV, are improved as the similarity and correlation among the task sequence {T 1 , · · · , T T } are higher. Similar to [4, 15, 34], we define the dissimilarity between tasks by the distance between the optimal parameters of tasks. Given the optimal task-specific parameters { θ ∗ 1 , · · · , θ ∗ T } for the task sequence {T 1 , · · · , T T } , the average distance between the parameter ϕ and the optimal task-specific parameters is defined by D ist ( ϕ, T 1: T ) ≜ √ 1 T ∑ T t =1 1 2 ∥ θ ∗ t -ϕ ∥ 2 . The dissimilarity of {T 1 , · · · , T T } is defined as S ∗ ( T 1: T ) ≜ min ϕ D ist ( ϕ, T 1: T ) . When the disclosure of {T 1 , · · · , T T } is stationary, i.e., the task T t at each t is sampled from the same probability distribution p ( T ) , the task dissimilarity of the task distribution p ( T ) is defined as S ∗ ( p ( T )) ≜ √ min ϕ E T t ∼ p ( T ) [ 1 2 ∥ θ ∗ t -ϕ ∥ 2 ] .

3 Online Constrained Meta-Learning Algorithm

In this section, to handle the sequential constrained learning tasks formulated in Section 2.1, we propose the online constrained meta-learning framework, which contains the meta-parameter update in Section 3.2 and the task-specific adaptation from the meta-parameter in Section 3.3.

3.1 Online constrained meta-learning setting.

Online constrained meta-learning aims to handle the problem given in Section 2.1, where the sample numbers of the datasets for each task, i.e., |D tr i,t | , are limited and the agent is required to quickly adapt to a new task once the task is revealed. At round t , the task-specific parameter θ t is updated by a within-task algorithm, and the meta parameter ϕ t is updated by a meta-algorithm. In particular, in the first step, the agent adapts the meta parameter ϕ t to the task-specific parameter θ t for task T t , by using a within-task algorithm A lg with the training datasets D tr t = {D tr 0 ,t , · · · , D tr m,t } and the constraint limits { c 1 ,t , . . . , c m,t } . Next, the agent uses all available tasks T 1 , . . . , T t to update the meta parameter to ϕ t +1 by a meta-algorithm, which will be used in the task-specific adaptation for the next task T t +1 at round t +1 . Our task-specific adaptation algorithm A lg and meta-algorithm are discussed in the following.

3.2 Task-specific adaptation.

At round t , the task-specific parameter θ t is adapted from the meta parameter ϕ t to task T t by the within-task algorithm A lg . In the meta-initialization approaches [41, 23, 24], the within-task algorithm A lg only takes a few optimization steps toward reducing the empirical risk and the constraint violation, and thus the solution is far from being feasible for Problem (1). To prioritize constraint satisfaction, we employ the meta-regularization approach with hard constraints. In particular, the

task-specific adaptation from the meta-parameter ϕ t is defined by constrained optimization with the ϕ t -biased regularization, i.e., θ t = A lg ( λ, ϕ t , D tr t ) is defined by

where the regularization weight λ > 0 . By the algorithm in (3), the task-specific parameter θ t satisfies the constraints over the training datasets D tr i,t . The objective function includes the empirical loss defined on the training dataset D tr 0 ,t and is regularized by a biased term λ 2 ∥ θ -ϕ t ∥ 2 [47, 15], which penalizes the deviation from the meta parameter ϕ t . The feasible set of the optimization problem in (3) is denoted as K tr t ≜ { θ ∈ Θ | 1 |D tr i,t | ∑ z ∈D tr i,t ℓ i ( θ, z ) ≤ c i,t , ∀ i = 1 , . . . , m } . In the following sections, we consider that, the numbers of the data sampling from D 0 ,t , i.e., |D tr 0 ,t | , are the same for all t , and is denoted as |D tr 0 | ; the numbers of the data sampling from D i,t , i.e., |D tr i,t | are the same for each i and t , and is denoted as |D tr + | .

3.3 Meta-parameter update.

After the model parameterized by θ t is deployed, the agent obtains the dataset D val t = {D val 0 ,t , · · · , D val m,t } by sampling from the data distribution D t = {D 0 ,t , · · · , D m,t } . The performance evaluation of the model with parameter θ is defined as L val ( θ, D val 0 ,t ) = 1 |D val 0 ,t | ∑ z ∈D val 0 ,t ℓ 0 ( θ, z ) . Here, we consider that |D val 0 ,t | is the same for all t and is denoted as |D val 0 | .

In online meta-learning, at each round t , the meta-parameter is updated by using an online learning algorithm to the meta-objective functions on all revealed tasks {T 1 , · · · , T t } . In our meta-learning problem, as the agent has collected the validation data {D val 1 , · · · , D val t } , we can evaluate the performance of the task-specific parameter θ t for each task, then the meta-objective function at round t is defined as ∑ t t ′ =1 L val ( A lg ( λ, ϕ, D tr t ′ ) , D val 0 ,t ′ ) , where A lg is defined in (3).

Papers [4, 24] solve online meta-learning and update the meta-parameter by using the Follow-theLeader (FTL) [26] to their meta-objective functions. However, the FTL requires the objective function to be strongly convex, and cannot be used in our problem, where the meta-objective function is non-convex. Therefore, in this paper, we use the Follow-the-Perturbed-Leader (FTPL) [50] to solve the problem, which is the first time that the FTPL is applied to online meta-learning. At round t , the FTPL optimizes the meta-objective function with perturbed terms over all revealed tasks to obtain the meta parameter ϕ t +1 for task T t +1 . In particular, at round t , the meta parameter ϕ t +1 is obtained by solving the following optimization problem:

where σ t ∈ R d is the random perturbed vector and its components { σ t,j } d j =1 is i.i.d sampled from the exponential distribution with a parameter η > 0 at each t . As A lg ( λ, ϕ, D tr t ′ ) included in (4) is the optimal solution of the constrained optimization in (3), Problem (4) is a constrained bilevel optimization problem [55].

3.4 Algorithm statement.

The online constrained meta-learning algorithm is formally stated in Algorithm 1. Note that line 3 is a deterministic convex optimization problem, which has been widely studied and solved by [10, 6]. The optimization problem (4) in line 6 is a deterministic constrained bilevel optimization problem, which can be solved by the algorithm in [55]. The implementation details are included in our proposed practical algorithm (Algorithm 2 in Appendix B).

4 Theoretical Results

In this section, we derive the upper bounds of the TAOG and the TACV produced by Algorithm 1, which characterize the regrets of the generalization errors of the models deployed for the sequential tasks. First, we study the generalization error bound produced by our within-task algorithm (3) for a single constrained learning task (Section 4.1). Next, we show the upper bounds of the TAOG and the TACV when the meta-parameter can be arbitrarily selected (Section 4.1). Finally, we obtain the theoretical bounds of the TAOG and the TACV, when the meta-parameter is not arbitrarily selected bu produced by Algorithm 1 (Section 4.2).

Before the result statements, we introduce some definitions and the required assumptions about the constraint qualifications for the optimization problems in (1) and (3). The constraint qualification assumptions are usually used in constraint optimization analyses [7, 19].

We consider an optimization problem with inequality constraints: min x g ( x ) s.t. h i ( x ) ≤ 0 , i ∈ I ≜ { 1 , . . . , m } and denote it as (P). Denote the feasible set of (P) as K ≜ { x | h i ( x ) ≤ 0 , ∀ i ∈ I } and I ( x ) ≜ { i ∈ I | h i ( x ) = 0 } for x ∈ K .

Definition 1. The Linear Independence Constraint Qualifications (LICQ) holds for (P), if vectors {∇ h i ( x ) | i ∈ I ( x ) } are linearly independent for any x ∈ K .

Definition 2. Slater's condition (SC) holds for (P) with the margin C > 0 , if there exists ¯ x such that h i (¯ x ) -C ≤ 0 , ∀ i ∈ I .

Assumption 1 (Constraint qualifications) . (i) There exists a compact set with diameter B > 0 such that the following properties hold for each t : for the given data distribution D t , the feasible set K t of the optimization problem in (1) is included in the compact set; with probability 1 , for D tr t sampled from D t , the feasible set K tr t of the optimization problem in (3) is included in the compact set.

(ii) There exists C > 0 such that the following properties hold: for the given data distribution D t , the SC holds for the optimization problem in (1) with the margin C ; with probability 1 , for D tr t sampled from D t , the SC holds for the optimization problem in (3) with the margin C .

(iii) For the given data distribution D t , the LICQ holds for the optimization problem in (1) .

Remark 1. A sufficient condition for part (i) of Assumption 1 is that Θ is a compact set.

We also require the following assumptions on the loss function and the constraint functions, which are standard in the analysis of meta-learning problems [4, 15, 24, 20].

Assumption 2 (Function properties) . (i) For any z ∈ Z , the loss function ℓ 0 ( · , z ) and the constraint functions ℓ 1 ( · , z ) , · · · , ℓ m ( · , z ) are twice continuously differentiable. (ii) For any z ∈ Z , ℓ 0 ( · , z ) is L 0 -Lipschitz, i.e., ∥ ℓ 0 ( w,z ) -ℓ 0 ( u, z ) ∥ ≤ L 0 ∥ w -u ∥ for any w,u ∈ Θ . (iii) For any z ∈ Z , ℓ 0 ( · , z ) is ρ -smooth, i.e., ∥∇ ℓ 0 ( w,z ) -∇ ℓ 0 ( u, z ) ∥ ≤ ρ ∥ w -u ∥ for any w,u ∈ Θ . (iv) For any z ∈ Z and 1 ≤ i ≤ m , ℓ i ( · , z ) is L c -Lipschitz, i.e., ∥ ℓ i ( w,z ) -ℓ i ( u, z ) ∥ ≤ L c ∥ w -u ∥ for any w,u ∈ Θ . (v) For any z ∈ Z and 0 ≤ i ≤ m , ℓ i ( · , z ) is convex. (vi) For any t ∈ { 1 , · · · , T } , z ∈ Z , w ∈ Θ , and 1 ≤ i ≤ m , ℓ i ( w,z ) -c i,t are bounded by M .

4.1 Generalization of constrained learning with biased regularization

We begin with the generalization error bound on a single constrained learning task T t , which is produced by the within-task algorithm (3) with an arbitrarily given meta-parameter ϕ t . As stated in (3), the algorithm replaces all the expectations in (1) by the sample averages oven the given datasets and adds a regularization term with the bias ϕ t to the objective function.

Proposition 1. Suppose that Assumptions 1 and 2 are satisfied. For any given metaparameter ϕ t and regularization weight λ > 0 , the following bounds hold for task

Proposition 1 shows the generalization error bounds of the loss function ℓ 0 and the constraint violation for each ℓ i . The coefficients of the notations O and the proofs are shown in Propositions 3 and 4 of Appendix E. As the objective function in (3) includes the biased regularization term λ 2 ∥ θ -ϕ t ∥ 2 , where the bias serves as the prior estimation for the solution, the generalization error bound includes a term of λ 2 ∥ θ ∗ t -ϕ t ∥ 2 , which is decreasing as the estimation ϕ t is more accurate. The objective function and the constraint functions include the sample averages over the training dataset D tr t . As the numbers of data points ( |D tr 0 | and |D tr + | ) are larger, the sample average approximation is more accurate, and then the generalization error is smaller.

The generalization error bounds shown in Proposition 1 motivate learning a good meta-parameter ϕ t by meta-learning. Next, we study the generalization error bounds for the task sequence {T 1 , · · · , T T } , when the meta-parameter ϕ is arbitrary and fixed. Proposition 2 shows the upper bounds of the TAOG and the TACV for any given meta-parameter ϕ , i.e., the task-specific parameter θ t in (2) is computed by θ t = A lg ( λ, ϕ, D tr t ) for all t .

Proposition 2. Suppose that Assumptions 1 and 2 are satisfied. Consider the task sequence {T 1 , · · · , T T } , for any given meta-parameter ϕ , choose the regularization parameter λ = 2 √ d ( ρ B + L 0 √ ln |D tr 0 | ) D ist ( ϕ, T 1: T ) √ |D tr 0 | . Then, the following bounds hold:

The coefficients of the notations O and the proofs are shown in Proposition 5 of Appendix E. Following the result in Proposition 2, as the average distance between the meta-parameter ϕ and the optimal task-specific parameters { θ ∗ 1 , · · · , θ ∗ T } , i.e., D ist ( ϕ, T 1: T ) , becomes small, the upper bound of the TAOG is small. By minimizing the distance D ist ( ϕ, T 1: T ) over ϕ , we can get the optimal upper bound of the TAOG:

when the meta-parameter is selected by ϕ ∗ = arg min ϕ D ist ( ϕ, T 1: T ) . Note that the optimal upper bound is not achievable since the optimal task-specific solution θ ∗ t is not achievable for each t . In the next section, we show that the sequence of meta-parameters ϕ t produced by Algorithm 1 produce a comparable performance as the optimal meta-parameter ϕ ∗ .

4.2 Generalization of the online constrained meta-learning algorithm

Consider the meta-parameter is updated by (4) in Algorithm 1. As we apply the FTPL algorithm to the meta-objective function for the online constrained meta-learning problem, the upper bounds of the TAOG and the TACV produced by Algorithm 1 are shown in Theorem 1 and Corollary 1. As the algorithm to compute θ t is stochastic, which depends on the task sequence and the sampling of the training data D tr t and evaluation data D val 0 ,t , we denote the expectation of the optimality gap and the constraint violations R i,t ( θ t ) as E [ R i,t ( θ t )] , and the expectations of the TAOG and the TACV are denoted as E [ ¯ R i, [1: T ] ] = E [ 1 T ∑ T t =1 R i,t ( θ t ) ] for all i = 0 , . . . , m .

Theorem 1. Suppose that Assumptions 1 and 2 are satisfied. Suppose that Θ is included in a compact cube with edge of length D l , i.e. ∥ ϕ ∥ ∞ ≤ D l for any ϕ ∈ Θ . Choose the regularization parameter λ = 2 √ d ( ρ B + L 0 √ ln |D tr 0 | ) S ∗ ( T 1: T ) √ |D tr 0 | and the perturbed parameter η = 1 L 0 √ dT . For the task

Corollary 1. If the task T t at each t is sampled from a fixed distribution p ( T ) , we choose λ = 2 √ d ( ρ B + L 0 √ ln |D tr 0 | ) S ∗ ( p ( T )) √ |D tr 0 | , then S ∗ ( T 1: T ) in the upper bound of E [ ¯ R 0 , [1: T ] ] shown in Theorem 1 can be replaced by the fixed constant S ∗ ( p ( T )) .

The coefficients of the notations O and the proofs are shown in Appendices E and F. In the online constrained meta-learning problem, as stated in Algorithm 1, after the task-specific parameter θ t for task T t is deployed, the agent is usually able to sample the dataset D val 0 and |D val 0 | ≫|D tr 0 | . In this case, we can ignore the term √ ln |D val 0 | |D val 0 | in the upper bound of E [ ¯ R 0 , [1: T ] ] in Theorem 1. We compare the upper bound of the TAOG by Algorithm 1 with the unachievable optimal upper bound shown in (5), where the term √ ln |D val 0 | |D val 0 | is ignored. The only additional term included in Theorem 1 is O ( 1 √ T ) . It implies that the regret of the optimality gap between the meta-parameter ϕ t updated by Algorithm 1 and the optimal meta-parameter ϕ ∗ used in (5) is sublinear with respect to the task number T .

In particular, when the constraint is ignored from Problem (1), the upper bound of the TAOG reduces to E [ ¯ R 0 , [1: T ] ] ≤ O ( S ∗ ( p ( T )) √ ln |D tr 0 | |D tr 0 | + 1 √ T ) , which has the same order as the upper bound shown in [18] with respect to the task dissimilarity S ∗ ( p ( T )) , the data number |D tr 0 | , and task number T .

Corollary 1 considers that the revealed tasks are sampled from a static task distribution. In this case, when T is sufficiently large, Algorithm 1 degenerates into an offline constrained meta-learning algorithm. If we further ignore the constraints in Problem (1), the bound in Corollary 1 of the TAOG holds the same order as the upper bound shown in [15].

5 Applications

5.1 Meta imitation learning

Meta-imitation Learning [25, 40] has been developed as a powerful tool for a learner to finish a new task by observing the demonstrations of an expert. However, existing methods do not consider any constraint, e.g., collision avoidance, for the new task. In this experiment, we apply the constrained meta-learning algorithm to a constrained meta-imitation learning problem where the learner learns to write letters in a cluttered environment [30, 29]. At each round t , the expert writes a different letter in a free space. The learner can observe the locations and velocities of a few points of the letter and is asked to write the same letter in a cluttered environment where the obstacle is known. The learner learns the policy by using constrained learning, where the loss function is defined by the tracking error of the learned policy, and the constraint is defined by a collision metric. Note that the expert performs in the free space, and the learner performs in the cluttered environment. The additional obstacle makes the problem more difficult. To the best of our knowledge, it is the first time to study

0

20

40

60

80

100

Round (task index)

0

10

20

30

40

50

60

70

Test error of loss function

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

0

20

40

60

80

100

Round (task index)

0

1

2

3

4

Constraint voilation on test data

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

0

20

40

60

80

100

Round (task index)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Collision probability

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

the problem. We provide the details of the problem formulation and the implementation setting in Appendix C.1.

We compare the proposed method with two benchmarks: (a) starting from scratch; (b) online MAML [24] with constraint penalty. In (a), the optimization problem in (3) is solved from a random initial parameter. In (b), we add a weighted penalty term for constraint violation to the loss function of the online MAML. For each task, the three methods share the data points.

Few-shot imitation learning. Consider that the learner collects the locations and velocities on a few points (few-shot data) from the expert for each task. Fig. 1 compares the proposed constrained meta-learning with the selected benchmarks. Fig. 1 Left shows that our approach has the fastest learning rate, i.e., its test error is smaller than 20 at around T = 10 , while the online MAML achieves that at around T = 30 , and the test error of starting from scratch is always above 20 . Moreover, our approach has the best learning accuracy, i.e., the steady-state test error of our approach is much smaller than those of the other two methods. In terms of constraint violation, our approach achieves the lowest collision probability in Fig. 1 Right and the lowest constraint violation metric in Fig. 1 Middle.

Improve full-shot imitation learning by meta-learning.

We consider that the learner can collect the locations and velocities on sufficient points (full-shot data) from the expert for each task, and test the proposed method and the selected benchmarks. In this experiment, we show that meta-learning can improve the performance of constrained learning, even if suf-

ficient data can be accessed by the learner. As shown in Fig. 2, our method can achieve superior performance than that in the two benchmark approaches (only our method can completely draw the shape of the letter). Fig. 3 shows that our method achieves a comparable adaptation time to online-MAML while outperforming online-MAML in terms of test error and collision avoidance. Both our method and online-MAML much outperform starting from scratch in terms of test error, collision avoidance, and adaptation time.

0

20

40

60

80

100

Round (task index)

0

10

20

30

40

50

60

70

Test error of loss function

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

0

20

40

60

80

100

Round (task index)

0

1

2

3

4

Constraint voilation on test data

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

0

20

40

60

80

100

Round (task index)

0.0

0.1

0.2

0.3

0.4

Collision probability

Starting from scratch

Online MAML with constraint penalty

Constrained meta-learning

Starting

from scratch

MAML with

constraint penalty

Constrained

meta-learning

0

2

4

6

8

Adaptation time (s)

5.2 Few-shot image classification with robustness

Meta-learning has been widely used for few-shot learning. Few-shot learning aims to train a model with only a few data samples, which is widely solved from the perspective of meta-learning by learning prior knowledge from multiple tasks. Consider a problem of few-shot image classification with robust learning [57]. In particular, after it is updated by a few data points, we require that the task-specific model has high accuracy on clean test data and is also robust to adversarial attacks on the test data. Robust learning for a single task can be formulated by a constrained optimization problem [12]. Correspondingly, in this experiment, the robust few-shot image classification is formulated as a constrained meta-learning problem and is solved by the proposed algorithm. The details of the problem formulation and the implementation setting are shown in Appendix C.2.

To evaluate the performance of the proposed algorithm, we test the taskspecific model on two sets of each dataset: (a) the clean test dataset; (b) the corrupted test dataset which is obtained by adding perturbations on the clean test dataset through the Projected Gradient Descent (PGD) method [37]. Correspondingly, we show the test accuracy on (a) as clean accuracy; the test accuracy on (b) as PGD accuracy. Moreover, we use Balance Score (Bscore) [57] as a metric to evaluate the comprehensive performance of the model, which is defined as B-score = 2 × (CA × PA)/(CA + PA) with CA and PA denoting clean accuracy and PGD

accuracy, respectively. We compare the proposed constrained meta-learning (CML) with several benchmarks: (i) MAML [23] with constraint penalty (CP); (ii) ProtoNet [49] with CP; (iii) BOIL [44] with CP; (iv) MAML with MOML [57]; (v) ProtoNet with MOML; (vi) BOIL with MOML. As the baseline methods for few-shot learning, MAML, ProtoNet, and BOIL, can only deal with unconstrained meta-learning, at (i)(ii)(iii), we add a weighted penalty term of the robustness constraint violation to their loss functions. At (iv)(v)(vi), MAML, ProtoNet and BOIL fit into a multi-objective meta-learning (MOML) framework [57], which regards the clean accuracy and the PDG accuracy as two competing objectives. Since all baseline methods are offline meta-learning, for comparisons, we use the similar way shown in Algorithm 2 of Appendix B to adapt them to the online setting. Specifically, in each round, we sample a batch of tasks from the revealed tasks, and use the gradientbased method on their meta-objective functions over the data of sampled tasks. The modification from the offline baseline methods to their online versions are exactly following our approach in terms of optimization of the online meta-objective.

0

50

100

150

200

Round (task index)

0.35

0.40

0.45

0.50

0.55

0.60

Clean accuracy

Dataset: mini-ImageNet (5-way 5-shot)

CML (ours)

Protonet with MOML

MAML with MOML

BOIL with MOML

0

50

100

150

200

Round (task index)

0.25

0.30

0.35

0.40

0.45

0.50

PGD accuracy

Dataset: mini-ImageNet (5-way 5-shot)

CML (ours)

Protonet with MOML

MAML with MOML

BOIL with MOML

0

50

100

150

200

Round (task index)

0.30

0.35

0.40

0.45

0.50

0.55

B-score

Dataset: mini-ImageNet (5-way 5-shot)

CML (ours)

Protonet with MOML

MAML with MOML

BOIL with MOML

We test the algorithms on two few-shot learning datasets, CUB [53] and mini-ImageNet [52]. Due to the page limit, the results on the CUB are shown in Appendix C.2. Table 2 shows that our method significantly improves the PGD accuracy and the B-score than the benchmarks and keeps the clean accuracy comparable. Fig. 4 shows that our method outperforms the benchmarks in terms of both test accuracy and learning speed, i.e., its test accuracy is larger than 0 . 5 at T = 30 , while the benchmarks take at least 50 tasks to achieve so.

6 Conclusion

In this paper, we develop an online constrained meta-learning framework, which learns the meta-prior from learning tasks with constraints. We theoretically quantify the optimality gaps and constraint violations produced by the proposed framework. Both two metrics consider the generalization ability of the task-specific models to unseen data. Moreover, we propose a practical algorithm for the constrained meta-learning framework. Our experiments on meta-imitation learning and few-shot image classification demonstrate the superior effectiveness of the algorithm.

Acknowledgments and Disclosure of Funding

This work is partially supported by the National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like to thank the reviewers for their constructive and insightful suggestions.

References

Durmus Alp Emre Acar, Ruizhao Zhu, and Venkatesh Saligrama. Memory efficient online meta learning. In International Conference on Machine Learning , pages 32-42. PMLR, 2021.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pages 22-31. PMLR, 2017.

Pierre Alquier, Massimiliano Pontil, et al. Regret bounds for lifelong learning. In Artificial Intelligence and Statistics , pages 261-269. PMLR, 2017.

Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradientbased meta-learning. In International Conference on Machine Learning , pages 424-433. PMLR, 2019.

Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-based meta-reinforcement learning for flight with suspended payloads. IEEE Robotics and Automation Letters , 6(2):1471-1478, 2021.

Dimitri Bertsekas. Convex optimization theory , volume 1. Athena Scientific, 2009.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

Aude Billard, Yann Epars, Sylvain Calinon, Stefan Schaal, and Gordon Cheng. Discovering optimal imitation strategies. Robotics and Autonomous Systems , 47(2-3):69-77, 2004.

Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics , 9:323-375, 2005.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Rich Caruana. Multitask learning. Machine learning , 28(1):41-75, 1997.

Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. Advances in Neural Information Processing Systems , 33:16722-16735, 2020.

Jiaxin Chen, Xiaoming Wu, Yanke Li, Qimai Li, Liming Zhan, and Fu-lai Chung. A closer look at the training strategy for modern meta-learning. Advances in Neural Information Processing Systems , 33:396-406, 2020.

Siddharth Choudhary, Luca Carlone, Carlos Nieto, John Rogers, Henrik I Christensen, and Frank Dellaert. Distributed mapping with privacy and communication constraints: Lightweight algorithms and object-based models. The International Journal of Robotics Research , 36(12):12861311, 2017.

Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning , pages 1566-1575. PMLR, 2019.

Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Incremental learningto-learn with statistical guarantees. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 , volume 34, pages 457-466. AUAI, 2018.

Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around a common mean. Advances in Neural Information Processing Systems , 31, 2018.

Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. Advances in Neural Information Processing Systems , 32, 2019.

Yuri M Ermoliev and Vladimir I Norkin. Sample average approximation method for compound stochastic optimization problems. SIAM Journal on Optimization , 23(4):2231-2263, 2013.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Generalization of model-agnostic metalearning algorithms: Recurring and unseen tasks. Advances in Neural Information Processing Systems , 34:5469-5480, 2021.

Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via PAC-bayes and uniform stability. Advances in Neural Information Processing Systems , 34:2173-2186, 2021.

Anthony V Fiacco. Introduction to sensitivity and stability analysis in nonlinear programming . Academic press, 1983.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135. PMLR, 2017.

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning , pages 1920-1930. PMLR, 2019.

Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning , pages 357-368. PMLR, 2017.

James Hannan. Approximation to bayes risk in repeated play. Contributions to the Theory of Games , 3(2):97-139, 1957.

Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint arXiv:1802.04376 , 2018.

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):5149-5169, 2021.

Yanlong Huang. Ekmp: Generalized imitation learning with adaptation, nonlinear hard constraints and obstacle avoidance. arXiv preprint arXiv:2103.00452 , 2021.

Yanlong Huang, Leonel Rozo, Joao Silvério, and Darwin G Caldwell. Kernelized movement primitives. The International Journal of Robotics Research , 38(7):833-852, 2019.

Yu Huang, Yingbin Liang, and Longbo Huang. Provable generalization of overparameterized meta-learning trained with SGD. arXiv preprint arXiv:2206.09136 , 2022.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , pages 4882-4892. PMLR, 2021.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Theoretical convergence of multi-step model-agnostic meta-learning. Journal of Machine Learning Research , 23:29-1, 2022.

Vanshaj Khattar, Yuhao Ding, Javad Lavaei, and Ming Jin. A CMDP-within-online framework for meta-safe reinforcement learning. In International Conference on Learning Representations , 2023.

Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient-based meta-learning methods. Advances in Neural Information Processing Systems , 32, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security , pages 99-112. Chapman and Hall/CRC, 2018.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10657-10665, 2019.

Thomas Lew, Apoorva Sharma, James Harrison, Andrew Bylard, and Marco Pavone. Safe active dynamics learning and control: A sequential exploration-exploitation framework. IEEE Transactions on Robotics , 2022.

Jiayi Li, Tao Lu, Xiaoge Cao, Yinghao Cai, and Shuo Wang. Meta-imitation learning by watching video demonstrations. In International Conference on Learning Representations , 2022.

Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 , 2017.

Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems. Advances in Neural Information Processing Systems , 35:33444-33456, 2022.

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through metareinforcement learning. In International Conference on Learning Representations , 2018.

Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. BOIL: Towards representation change for few-shot learning. In International Conference on Learning Representations , 2021.

German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks , 113:54-71, 2019.

Anastasia Pentina and Christoph Lampert. A PAC-bayesian bound for lifelong learning. In International Conference on Machine Learning , pages 991-999. PMLR, 2014.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems , 32, 2019.

Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations , 2017.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems , 30, 2017.

Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory , pages 845-861. PMLR, 2020.

Sebastian Thrun and Lorien Pratt. Learning to learn . Springer Science & Business Media, 2012.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems , 29, 2016.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. California Institute of Technology , 2011.

Siyuan Xu and Minghui Zhu. Meta value learning for fast policy-centric optimal motion planning. Robotics Science and Systems , 2022.

Siyuan Xu and Minghui Zhu. Efficient gradient approximation method for constrained bilevel optimization. Proceedings of the AAAI Conference on Artificial Intelligence , 37(10):1250912517, 2023.

Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In International Conference on Machine Learning , pages 449-457. PMLR, 2016.

Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multiobjective meta learning. Advances in Neural Information Processing Systems , 34:21338-21351, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of online meta-learning with constraints. After a formalization of the problem, the paper proposes an algorithm in the case where the loss function is convex, using Follow-the-Perturbed-Leader (FTPL) to update the meta-objective.\nThen the paper theoretically prove upper bounds on the regret and the violation of the constraints for their constrained learning setting.\nFinally, the paper presents experimental results on two different applications, meta imitation learning with collision avoidance and robust few-shot image classification.\n\nSTRENGTHS:\n### Originality\n\n- The paper presents a detailed theoretical analysis in the constrained setting.\n- The algorithm is a combination of iMAML [1] using FTPL instead of FTL for the online setting.\n\n### Quality\n\n- The paper presents and proves theoretical upper bounds in the constrained setting with their proposed algorithm, with a lot of theoretical developments in the appendix. It shows the soundness of their approach.\n\n### Clarity\n\n- The formalization of the problem is clear.\n\n### Significance\n\n- Results on the different benchmarks and applications presented show strong performance compared to baselines.\n\n[1]: Rajeswaran, A., Finn, C., Kakade, S. M., & Levine, S. (2019). Meta-learning with implicit gradients. Advances in neural information processing systems, 32.\n\nWEAKNESSES:\n### Clarity\n\n- The relation of the proposed algorithm to previous work is quickly dismissed. The authors introduce online meta-learning algorithms in l.23-33, and in the related works section (l.88 - 98), the paper presents only optimization-based meta-learning algorithm. However, there is no discussion to clearly explain the differences with their proposed algorithm.\n- The way the tasks are set up in the applications are not very well described. Specifically in the robust few-shot classification benchmarks, it is not clear how the benchmark is adapted for this *online* setting, since these datasets are more commonly used for few-shot meta-learning. \n\n\n### Quality\n\n- If I understand correctly, the approach presented is a combination of iMAML with FTPL for the online setting and with the constraint penalty. Thus, it would make sense to add other meta-regularization algorithm baselines in the benchmarks, such as iMAML at least.\n- In the robust few-shot classification application presented, the other baselines are not designed for *online* meta-learning. The comparison seems unfair.\n- The authors state that their approach speeds up the adaptation to new tasks (l.357), but we can see in Figure 3 (right) that it is the opposite, their algorithm takes more time to adapt.\n\nQUESTIONS:\nI would like the authors to describe how the methods are evaluated on the robust few-shot image classification benchmarks:\n- The models are trained on how many tasks ? \n- Are the tasks different between meta-training and meta-testing ? If so, how many meta-testing tasks are considered ? Otherwise, I would like to see results of online meta-learning methods on this benchmark.\n- It's not totally clear to me why the algorithm would be specific to the constrained case.\n\nLIMITATIONS:\nThe paper does not include a discussion of the limitations of their approach. The algorithm presented seems to be more costly to run than online MAML.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
In this paper, a novel online constrained meta-learning framework is presented. The framework is designed to facilitate continuous learning from sequential tasks while ensuring that these tasks adhere to strict constraints. In addition to existing analyses of meta-learning, this study goes further by presenting the upper bounds for optimality gaps and constraint violations that arise from the proposed framework. 
This framework takes into account the dynamic regret of online learning and the generalization capability of the task-specific models. Finally, the paper offers a practical algorithm to implement the framework, and its superior effectiveness is validated through experiments conducted in the domains of meta-imitation learning and few-shot image classification.

STRENGTHS:
- This paper consider that the meta-objective is non-convex.
- Study the dynamic regrets.
- Two elaborate applications demostrate the superior effectiveness of the algorithm.

WEAKNESSES:
- The bound is scaled with $\mathcal{O}(\frac{1}{\sqrt{T}})$. This seems to ignore the size of every training datasets.
- The comparion with existing online meta learning bounds is necessary.

QUESTIONS:
- In section 2.1, the proposed constrained optimization paradigm in the paper requires that different tasks satisfy certain constraints on errors under any task-adaptive parameters. Intuitively, this may seem contradictory to improving the performance of specific tasks. Task-adaptive parameters should ideally be focused solely on specific tasks, so why is there a need to ensure the performance of other tasks simultaneously? Is it more reasonable to constrain the meta-parameters?

LIMITATIONS:
It should compare with existing online meta learning bounds.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the theory of biased-regularization meta-learning under the sequential task setting. Despite there having been previous works in this area, this paper distinguishes itself by introducing the concept of the Online Constrained Meta-Learning problem and presenting a straightforward solution. It applies constrained optimization with biased meta-regularization, utilizing Follow-the-Perturbed-Leader (FTPL) FTPL to handle the non-convex meta-objective function, providing theoretical analysis and proofs of upper bounds, and developing a practical algorithm for large-scale problems. Empirical experiments validate the effectiveness of the proposed algorithm in meta-imitation and meta-reinforcement learning.\n\nSTRENGTHS:\n- This paper first studies the online constrained meta-learning, which is rarely concerned. To this end, this paper gave the formal problem formulation of constrained sequential learning. \n\n- The author distinguishes the assumptions proposed by this paper and follows others. It makes it easier to examine the assumptions applied in this paper.\n\n- A solid work that follows the setting of learning with biased regularization. The author gives a detailed discussion of different cases.\n\nWEAKNESSES:\n- Of a particular relevant missing work, [1] can also be deemed as the constrained (conditional) meta-learning. The author should further discuss this work since [1] also shows the generalization results. \n\n- More implications are needed to explain the results, i.e., Corollary 1. \n\n- The bounds of the derived results have not been examined. For instance, the second term in RHS of Prop 3 is the order of $\\mathcal{O}(d^2\\mathcal{B} L_0^2 \\ln |\\mathcal{D}^{tr}_0|)$, which can be a dominate term.  Furthermore, the diameter $\\mathcal{B}$ and model size $d$ can also be large enough. This result may not be applied to overparameterized settings.\n\n- The relationship between $\\mathcal{D}_t$ and $\\mathcal{D}$ is vague. Suppose $\\mathcal{D}$ refers to any $t$ in $\\mathcal{D}_t$. The statement of Propositions and Theorems should clarify this point.\n\n- From the Proofreading of Appendix 4, I found the bounds in Proposition 3 & 4 depend on many terms. However, in Prop.1 & 2, the author removes the small terms (in Big-O notation) without discussing the orders of these terms in what limits. \n\n——Minors——\n- Confusing statement about “Problem (1), (3)”. Since the author refers to Equations (1) and (3) as “Algorithms” and “Problems” simultaneously, it may be better to define the “Problems” officially.\n\n- Adding a notation cheatsheet in the appendix may be more readable.\n\nQUESTIONS:\n- Why define $D(\\phi, \\mathcal{T}_{1:T})$, $\\mathcal{S}^*(\\rho(\\mathcal{T}))$ in square root form?\n\n- What will happen if we only learn the same task, i.e., $\\mathcal{S}^*(\\rho(\\mathcal{T})) \\to 0$, then the coefficient will be infinitely large $\\lambda \\to \\infty$. Do the results still hold true in such a degenerate case?\n\n- Is Assumption 1 should hold $\\forall t$? If it is, please make it clear.\n\n- As mentioned above, what does big-O notation mean in the results?\n\nLIMITATIONS:\nN/A pure theoretical work.\n\n\n**References**:\n\n[1] Denevi, Giulia, Massimiliano Pontil, and Carlo Ciliberto. \"The advantage of conditional meta-learning for biased regularization and fine-tuning.\" Advances in Neural Information Processing Systems 33 (2020): 964-974.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an online constrained meta-learning algorithm that is able to sequentially learn a  sequence of tasks that are subject to hard (and stochastic) constraints. The authors also theoretically quantify the optimality gaps and constraint violations produced by the proposed method, by considering the dynamic regret of online learning and the generalization ability of the task-specific models. They also validate the effectiveness of the proposed method on numerical experiments on meta-imitation learning and few-shot image classification.\n\nSTRENGTHS:\nThe authors validate their proposed method, both theoretically and experimentally.\n\nThe authors present a new meta-learning method aiming at facing the more challenging situation in which the tasks are stochastically subject to constraints.\n\nWEAKNESSES:\nThe statements and the notation of the paper could be simplified and made more intuitive, less heavy.\n\nWhy can the constrained meta-learning framework be interesting in practical application? The authors do not well motivate the setting they consider. This is a very important aspect in my opinion. Could you please describe some examples of possible applications in which the proposed constrained setting can be useful/necessary?\n\nQUESTIONS:\nThe constraints are random variables. This seems to be problematic. Which is the main trick you use to deal with it?\n\nThe authors use the notion of variance of the task optimal parameters to measure the similarity among the tasks, as in [14] and the paper [A] I mention here below. Could you please make a clearer comparison between your rates and those obtained in these other work, by just looking at the main important constants and leading terms w.r.t. the number of tasks and within-task points? Consider also the different assumptions.\n\n[A] Denevi et al., 'Online-Within-Online Meta-Learning'.\n\nLIMITATIONS:\nI do not see any potential negative societal impact related to this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}