SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siyuan Xu & Minghui Zhu
University Park, PA 16801
{spx5032, muz16}@psu.edu
Meta-learning has attracted attention due to its strong ability to learn experiences from known tasks, which can speed up and enhance the learning process for new tasks. However, most existing meta-learning approaches only can learn from tasks without any constraint. This paper proposes an online constrained meta-learning framework, which continuously learns meta-knowledge from sequential learning tasks, and the learning tasks are subject to hard constraints. Beyond existing meta-learning analyses, we provide the upper bounds of optimality gaps and constraint violations of the deployed task-specific models produced by the proposed framework. These metrics consider both the dynamic regret of online learning and the generalization ability of the task-specific models to unseen data. Moreover, we provide a practical algorithm for the framework and validate its superior effectiveness through experiments conducted on meta-imitation learning and few-shot image classification.
In the setting of learning from multiple tasks, a shared prior is extracted from commonalities of existing tasks, resulting in improved learning efficiency and prediction accuracy for task-specific models. Conventional approaches include multi-task learning [11], lifelong learning [46, 45, 34]. These approaches extract the prior by vanilla learning approaches on the tasks and do not explicitly involve the task-specific learning in the prior extraction, as illustrated by [28]. In contrast, meta-learning or learning-to-learn [51, 23, 47] learns a meta prior, usually a meta parameter, by evaluating and optimizing the meta-objective, which is defined by the performance of the task-specific model adapted from the prior. Meta-learning has been extended to accelerate reinforcement learning [23, 34], where the tasks are modeled by Markov decision processes.
Online meta-learning considers that the meta parameter is learned from tasks that are sequentially revealed. Papers [24, 1, 16] are shown to be no-regret with respect to the static comparator, i.e., the gap between the aggregate performance of the meta parameter sequence and that of any fixed meta parameter is sublinear in the number of revealed tasks. The online-MAML [24] and the MOML [1] adopt one-step gradient descent to adapt the task-specific parameter, and paper [16] adapts it by the ridge regression. A direct evaluation of meta-learning is the performance of the task-specific model adapted from the meta parameter on each individual task. The methods in [4, 3, 34] consider dynamic regret [56, 4], which compares the task-specific parameter adapted from the meta parameter with the optimal counterpart for each task. The dynamic regrets of all the algorithms are shown to be bounded when there are sufficiently many tasks.
In many applications of meta-learning, such as few-shot learning in computer vision [38] and robotics [43, 5, 39], a task-specific model is expected to be quickly adapted by using a small amount of
training data of a new task and generalize to the unseen data of the task. The generalization ability of the task-specific adaptation has been considered in [13, 20, 21, 31]. Most of them consider the task-specific model to be adapted using a small number of samples from the data distribution, and the generalization error is quantified by the optimality gap between the produced task-specific model and the task-specific model adapted using the best meta-parameter. In contrast, in this paper, we explore a generalization error metric, which is defined as the optimality gap between the produced task-specific model and the optimal task-specific model, based on the entire data distribution. By directly quantifying the performance of the task-specific model on the entire data (including unseen data), this metric identifies the improvement in the model's generalizability for each specific learning task when the meta-parameter is employed. Despite its significance, this generalization metric is often overlooked in most meta-learning analyses. Generalization metrics that are similar to ours are evaluated in [15, 18] and discussed in [20].
Typically, meta-learning handles learning tasks as unconstrained optimization problems. Many real-world applications such as robot learning in [14, 54, 2, 29, 42], nevertheless, are subject to safety constraints that should be rarely violated. For this case, constrained meta-learning aims to learn meta-prior for constrained learning tasks, which are formulated as constrained optimization problems. As a result, in constrained meta-learning, besides the generalization with respect to the objective risk, the generalization with respect to constraint violation is also required to be considered during the task-specific adaptation. More specifically, in the constrained learning task, each constraint is also defined by the expectation over a data distribution, and only a small amount of training data sampled from the distribution is observable during the task-specific adaptation. Therefore, the adaptation can only guarantee the constraint satisfaction over the training data, while the constraint violation over the whole data distribution is required to be quantified.
Main contribution. In this paper, we develop the first approach for the online constrained meta-learning problem. In particular, we consider a sequential task setting of constrained learning tasks. At each round, a learning task with hard constraints is revealed and is required to be solved, and only a small number of data for the revealed task is provided. To address the problem, we use the Follow-the-Perturbed-Leader (FTPL) approach to learn a meta-regularization at each round from the revealed tasks and apply the meta-regularization to solve the new task. The overall contributions are summarized as follows. (i) We apply constrained optimization with biased meta-regularization to adapt the task-specific parameter by using a small amount of data. Then, we optimize the meta-objective function, which is defined by the performance of the task-specific parameter on validation data. (ii) To handle the non-convex meta-objective function, we use the Follow-the-Perturbed-Leader (FTPL) to update the meta-regularization from all the available tasks for the new task. (iii) We theoretically analyze and prove the upper bounds that (a) quantify both the optimality gaps on the objective functions and the constraint violations of task-specific models; (b) quantify the dynamic regrets where the comparator is dynamic and always optimal for each task; (c) quantify the generalization errors of task-specific models to the unseen data, where only a small number of training data are given in each task. The upper bounds show that the optimality gap between the optimal meta-parameter and that learned by our method decreases with a rate of  $\mathcal{O}(1 / \sqrt{T})$ . (iv) We develop a practical algorithm for the online constrained meta-learning that can be used effectively on large-scale problems, and empirically verify the efficacy of the proposed algorithm by conducting experiments on meta-imitation learning and few-shot robust image classification.
Table 1 compares the challenges of interests between this paper and previous works [4, 15, 24, 1, 18]. In particular, we consider the hard constraints in each learning task, the dynamic regret in the online meta-learning problem, and the generalization ability of the task-specific models to unseen data, si
Table 1: Challenges of interests
This paper	[4]	[15]	[18]	[1, 24]
Constraint	✓	×	×	×	×
Online learning	✓	✓	×	✓	✓
Dynamic regret	✓	✓	N/A	✓	×
Generalization	✓	×	✓	✓	×
multaneously. Even if the constraints are removed, this paper provides a more general theoretical analysis than most papers, as paper [15] quantifies the empirical loss over known training data, and does not consider the generalization, and papers [24, 1] consider a static regret.
Related works. Optimization-based meta-learning algorithms usually learn an optimization parameter as the meta prior, and each algorithm consists of a meta-algorithm, which learns the
meta parameter, and an inner-task or within-task algorithm, which adapts to a task-specific model based on the meta parameter. Based on the type of the within-task approach, the methods can be categorized into meta-initialization approaches [41, 23, 24, 4, 35, 33] and meta-regularization approaches [17, 15, 47, 32]. In meta-initialization approaches, the within-task algorithm uses the meta parameter as the initial point and takes a few optimization steps to minimize the empirical risk of the new task. For example, the MAML [23] algorithm takes one-step gradient descent as the within-task algorithm. In meta-regularization approaches, the within-task algorithm completely solves an optimization problem, where the meta parameter serves as the bias of the regularization term for the empirical risk. In our constrained meta-learning problem, the task-specific parameter should satisfy its given constraints. In the meta-initialization approaches, as the within-task algorithm only takes a few optimization steps, even if we do optimization to reduce the constraint violation, the solution is far from feasible. On the other hand, the meta-regularization approaches fully solve the within-task. To prioritize constraint satisfaction, we combine the meta-regularization approach with hard constraints as the within-task algorithm for constrained meta-learning.
Notations. Denote the  $l_{2}$  norm of vectors and the spectral norm (2-norm) of matrices by  $\| \cdot \|$ . Consider multiple data distributions  $\mathcal{D} = \{\mathcal{D}_0, \mathcal{D}_1, \dots, \mathcal{D}_m\}$  and multiple training datasets  $\mathcal{D}^{tr} = \{\mathcal{D}_0^{tr}, \mathcal{D}_1^{tr}, \dots, \mathcal{D}_m^{tr}\}$ . We use  $\mathcal{D}_i^{tr} \sim \mathcal{D}_i$  to represent that each data  $z$  in the dataset  $\mathcal{D}_i^{tr}$  is i.i.d sampled from the distribution  $\mathcal{D}_i$ , and  $\mathcal{D}^{tr} \sim \mathcal{D}$  to represent  $\mathcal{D}_i^{tr} \sim \mathcal{D}_i$  for all  $0 \leq i \leq m$ . We use  $|\mathcal{D}_i^{tr}|$  to denote the number of data in the dataset  $\mathcal{D}_i^{tr}$ . A notation checklist is attached in Appendix A.
2.1 Constrained learning tasks and the sequential setting
At round  $t$ , an agent aims to solve a constrained learning task  $\mathcal{T}_t$ , which is formulated as the following constrained optimization problem:
$$
\theta_ {t} ^ {*} = \underset {\theta \in \Theta} {\operatorname {a r g m i n}} \mathbb {E} _ {z \sim \mathcal {D} _ {0, t}} \left[ \ell_ {0} (\theta , z) \right] \text {s . t .} \mathbb {E} _ {z \sim \mathcal {D} _ {i, t}} \left[ \ell_ {i} (\theta , z) \right] \leq c _ {i, t}, i = 1, \dots , m, \tag {1}
$$
where each data point is denoted by  $z = (x,y) \in \mathcal{Z}$  with  $x \in \mathcal{X} \subseteq \mathbb{R}^{d_x}$  being the input and  $y \in \mathcal{Y} \subseteq \mathbb{R}^{d_y}$  being its corresponding output. The loss function  $\ell_0: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}$  is the performance metric of a model parameterized by  $\theta \in \Theta \subseteq \mathbb{R}^d$  and the constraint functions  $\ell_i: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}, i = 1, \ldots, m$  evaluate the constraint metric of the model. Each performance or constraint metric  $\ell_i$  is computed on the data distribution  $D_{i,t}$ , which is a joint distribution over input-output pairs  $z = (x,y)$ . The optimal solution of Problem (1) is denoted as  $\theta_t^*$ , and the feasible set of Problem (1) is denoted as  $K_t \triangleq \{\theta \in \Theta \mid \mathbb{E}_{z \sim D_{i,t}}[\ell_i(\theta,z)] \leq c_{i,t}, \forall i = 1, \ldots, m\}$ . The constrained learning task  $T_t$  is characterized by its data distributions  $D_t = \{D_{0,t}, D_{1,t}, \ldots, D_{m,t}\}$  and constraint limits  $\{c_{1,t}, \ldots, c_{m,t}\}$ . In general, the data distributions  $D_t$  are unknown to the agent. Instead, the training datasets  $D_t^{tr} = \{D_{0,t}^{tr}, D_{1,t}^{tr}, \ldots, D_{m,t}^{tr}\}$  are sampled i.i.d from  $D_t$  and available to the agent.
Consider that a set of constrained learning tasks are revealed sequentially, i.e., task  $\mathcal{T}_t$  is revealed at round  $t$ . In each round  $t$ , the agent updates a model once  $\mathcal{D}_t^{tr}$  is obtained. The updated model is deployed and works on  $\mathcal{T}_t$ , i.e., predicting the outputs  $y$  of all requested inputs  $x$ . In the next round, the procedure repeats.
Since the data distributions  $\mathcal{D}_t$  are unknown, the agent cannot completely solve Problem (1) and obtain  $\theta_t^*$ . A common solution is to replace the data distribution  $\mathcal{D}_{i,t}$  in Problem (1) by the sampled dataset  $\mathcal{D}_{i,t}^{tr}$  and solving the problem for each task  $\mathcal{T}_t$ . However, when the sample number in  $\mathcal{D}_{i,t}^{tr}$  is small, separately solving  $\mathcal{T}_t$  with  $\mathcal{D}_{i,t}^{tr}$  leads to limited performance. Similar to [4, 15], in this paper, the learning algorithm is expected to improve the performance of the solution for task  $\mathcal{T}_t$  by exploiting the correlation between the task  $\mathcal{T}_t$  and all previous tasks  $\{\mathcal{T}_1,\dots ,\mathcal{T}_{t - 1}\}$ .
2.2 Metrics of a learning algorithm
Consider a learning algorithm produces a parameter sequence  $\{\theta_1, \dots, \theta_T\}$  for the task sequence  $\{\mathcal{T}_1, \dots, \mathcal{T}_T\}$ . At round  $t$ , the optimality gap  $R_{0,t}$  of  $\theta_t$  is defined as  $R_{0,t}(\theta_t) = \max \left\{\mathbb{E}_{z \sim \mathcal{D}_{0,t}}\left[\ell_0(\theta_t,z) - \ell_0(\theta_t^*,z)\right], 0\right\}$ . The violation for the  $i$ -th constraint denoted by  $R_{i,t}$  is defined as:  $R_{i,t}(\theta_t) = \max \left\{\mathbb{E}_{z \sim \mathcal{D}_{i,t}}\left[\ell_i(\theta_t,z) - c_{i,t}, 0\right], i = 1, \ldots, m\right\}$ . The task-averaged optimality gap (TAOG) denoted by  $\bar{R}_{0,[1:T]}$  and the task-averaged constraint violation (TACV) denoted
by  $\bar{R}_{i,[1:T]}$  of the algorithm after  $T$  rounds are defined as
$$
\bar {R} _ {0, [ 1: T ]} = \frac {1}{T} \sum_ {t = 1} ^ {T} R _ {0, t} \left(\theta_ {t}\right), \quad \bar {R} _ {i, [ 1: T ]} = \frac {1}{T} \sum_ {t = 1} ^ {T} R _ {i, t} \left(\theta_ {t}\right), i = 1, \dots , m. \tag {2}
$$
Under the sequential task setting, the goal of a learning algorithm is to minimize the TAOG and the TACV of the parameter sequence  $\{\theta_1,\dots ,\theta_T\}$ . The definitions of the TAOG and the TACV consider both the generalization ability of  $\theta_{t}$  to the unseen data of each revealed task  $\mathcal{T}_t$ , and a dynamic notion of the optimality regret. First, the optimality gap  $R_{0,t}$  and the constraint violation  $R_{i,t}$  are defined by the expectations over the inaccessible data distribution  $D_{i,t}$ , instead of the empirical risk over the given dataset  $D_{i,t}^{tr}$  that is used in online meta-learning analysis [4, 35]. Thus, the TAOG and the TACV can be used to quantify the generalization errors. It is more challenging to minimize them, as  $\theta_{t}$  is expected to generalize to the unseen data in  $D_{i,t}$  while only the training datasets  $D_{i,t}^{tr}$  are given, and their size could be small. Second, unlike the fixed comparator used in the FTML algorithm [24], the comparator  $\theta_t^*$  is dynamic and always optimal for each task  $\mathcal{T}_t$ . As pointed out in [4, 34, 56], one cannot achieve the dynamic regret sublinear in  $T$  for either online learning or online meta-learning.
2.3 Task dissimilarity
As mentioned in Section 2.1, we expect that the performance produced by the learning algorithm, i.e., the TAOG and the TACV, are improved as the similarity and correlation among the task sequence  $\{\mathcal{T}_1,\dots ,\mathcal{T}_T\}$  are higher. Similar to [4, 15, 34], we define the dissimilarity between tasks by the distance between the optimal parameters of tasks. Given the optimal task-specific parameters  $\{\theta_1^*,\dots ,\theta_T^*\}$  for the task sequence  $\{\mathcal{T}_1,\dots ,\mathcal{T}_T\}$ , the average distance between the parameter  $\phi$  and the optimal task-specific parameters is defined by  $\mathcal{D}ist(\phi ,\mathcal{T}_{1:T})\triangleq \sqrt{\frac{1}{T}\sum_{t = 1}^{T}\frac{1}{2}\|\theta_t^* - \phi\|^2}$ . The dissimilarity of  $\{\mathcal{T}_1,\dots ,\mathcal{T}_T\}$  is defined as  $\mathcal{S}^{*}(\mathcal{T}_{1:T})\triangleq \min_{\phi}\mathcal{D}ist(\phi ,\mathcal{T}_{1:T})$ . When the disclosure of  $\{\mathcal{T}_1,\dots ,\mathcal{T}_T\}$  is stationary, i.e., the task  $\mathcal{T}_t$  at each  $t$  is sampled from the same probability distribution  $p(\mathcal{T})$ , the task dissimilarity of the task distribution  $p(\mathcal{T})$  is defined as  $\mathcal{S}^{*}(p(\mathcal{T}))\triangleq \sqrt{\min_{\phi}\mathbb{E}_{\mathcal{T}_t\sim p(\mathcal{T})}\left[\frac{1}{2}\|\theta_t^* - \phi\|^2\right]}$ .
3 Online Constrained Meta-Learning Algorithm
In this section, to handle the sequential constrained learning tasks formulated in Section 2.1, we propose the online constrained meta-learning framework, which contains the meta-parameter update in Section 3.2 and the task-specific adaptation from the meta-parameter in Section 3.3.
3.1 Online constrained meta-learning setting.
Online constrained meta-learning aims to handle the problem given in Section 2.1, where the sample numbers of the datasets for each task, i.e.,  $|\mathcal{D}_{i,t}^{tr}|$ , are limited and the agent is required to quickly adapt to a new task once the task is revealed. At round  $t$ , the task-specific parameter  $\theta_t$  is updated by a within-task algorithm, and the meta parameter  $\phi_t$  is updated by a meta-algorithm. In particular, in the first step, the agent adapts the meta parameter  $\phi_t$  to the task-specific parameter  $\theta_t$  for task  $\mathcal{T}_t$  by using a within-task algorithm  $\mathcal{A}lg$  with the training datasets  $\mathcal{D}_t^{tr} = \{\mathcal{D}_{0,t}^{tr},\dots,\mathcal{D}_{m,t}^{tr}\}$  and the constraint limits  $\{c_{1,t},\ldots ,c_{m,t}\}$ . Next, the agent uses all available tasks  $\mathcal{T}_1,\ldots ,\mathcal{T}_t$  to update the meta parameter to  $\phi_{t + 1}$  by a meta-algorithm, which will be used in the task-specific adaptation for the next task  $\mathcal{T}_{t + 1}$  at round  $t + 1$ . Our task-specific adaptation algorithm  $\mathcal{A}lg$  and meta-algorithm are discussed in the following.
3.2 Task-specific adaptation.
At round  $t$ , the task-specific parameter  $\theta_t$  is adapted from the meta parameter  $\phi_t$  to task  $\mathcal{T}_t$  by the within-task algorithm  $\mathcal{A}lg$ . In the meta-initialization approaches [41, 23, 24], the within-task algorithm  $\mathcal{A}lg$  only takes a few optimization steps toward reducing the empirical risk and the constraint violation, and thus the solution is far from being feasible for Problem (1). To prioritize constraint satisfaction, we employ the meta-regularization approach with hard constraints. In particular, the
task-specific adaptation from the meta-parameter  $\phi_t$  is defined by constrained optimization with the  $\phi_t$ -biased regularization, i.e.,  $\theta_t = \mathcal{A}lg(\lambda, \phi_t, \mathcal{D}_t^{tr})$  is defined by
$$
\underset {\theta \in \Theta} {\operatorname {a r g m i n}} \frac {1}{| \mathcal {D} _ {0 , t} ^ {t r} |} \sum_ {z \in \mathcal {D} _ {0, t} ^ {t r}} \ell_ {0} (\theta , z) + \frac {\lambda}{2} \| \theta - \phi_ {t} \| ^ {2} \text {s . t .} \frac {1}{| \mathcal {D} _ {i , t} ^ {t r} |} \sum_ {z \in \mathcal {D} _ {i, t} ^ {t r}} \ell_ {i} (\theta , z) \leq c _ {i, t}, i = 1, \dots , m, \tag {3}
$$
where the regularization weight  $\lambda > 0$ . By the algorithm in (3), the task-specific parameter  $\theta_t$  satisfies the constraints over the training datasets  $\mathcal{D}_{i,t}^{tr}$ . The objective function includes the empirical loss defined on the training dataset  $\mathcal{D}_{0,t}^{tr}$  and is regularized by a biased term  $\frac{\lambda}{2} \| \theta - \phi_t \|^2$  [47, 15], which penalizes the deviation from the meta parameter  $\phi_t$ . The feasible set of the optimization problem in (3) is denoted as  $\mathcal{K}_t^{tr} \triangleq \left\{ \theta \in \Theta \mid \frac{1}{|\mathcal{D}_{i,t}^{tr}|} \sum_{z \in \mathcal{D}_{i,t}^{tr}} \ell_i(\theta, z) \leq c_{i,t}, \forall i = 1, \dots, m \right\}$ . In the following sections, we consider that, the numbers of the data sampling from  $\mathcal{D}_{0,t}$ , i.e.,  $|\mathcal{D}_{0,t}^{tr}|$ , are the same for all  $t$ , and is denoted as  $|\mathcal{D}_0^{tr}|$ ; the numbers of the data sampling from  $\mathcal{D}_{i,t}$ , i.e.,  $|\mathcal{D}_{i,t}^{tr}|$  are the same for each  $i$  and  $t$ , and is denoted as  $|\mathcal{D}_+^{tr}|$ .
3.3 Meta-parameter update.
After the model parameterized by  $\theta_{t}$  is deployed, the agent obtains the dataset  $\mathcal{D}_t^{val} = \{\mathcal{D}_{0,t}^{val},\dots ,\mathcal{D}_{m,t}^{val}\}$  by sampling from the data distribution  $\mathcal{D}_t = \{\mathcal{D}_{0,t},\dots ,\mathcal{D}_{m,t}\}$ . The performance evaluation of the model with parameter  $\theta$  is defined as  $\mathcal{L}^{val}(\theta ,\mathcal{D}_{0,t}^{val}) = \frac{1}{|\mathcal{D}_{0,t}^{val}|}\sum_{z\in \mathcal{D}_{0,t}^{val}}\ell_0(\theta ,z)$ . Here, we consider that  $|\mathcal{D}_{0,t}^{val}|$  is the same for all  $t$  and is denoted as  $|\mathcal{D}_0^{val}|$ .
In online meta-learning, at each round  $t$ , the meta-parameter is updated by using an online learning algorithm to the meta-objective functions on all revealed tasks  $\{\mathcal{T}_1,\dots ,\mathcal{T}_t\}$ . In our meta-learning problem, as the agent has collected the validation data  $\{\mathcal{D}_1^{val},\dots ,\mathcal{D}_t^{val}\}$ , we can evaluate the performance of the task-specific parameter  $\theta_{t}$  for each task, then the meta-objective function at round  $t$  is defined as  $\sum_{t^{\prime} = 1}^{t}\mathcal{L}^{val}(\mathcal{A}lg(\lambda ,\phi ,\mathcal{D}_{t^{\prime}}^{tr}),\mathcal{D}_{0,t^{\prime}}^{val})$ , where  $\mathcal{A}lg$  is defined in (3).
Papers [4, 24] solve online meta-learning and update the meta-parameter by using the Follow-the-Leader (FTL) [26] to their meta-objective functions. However, the FTL requires the objective function to be strongly convex, and cannot be used in our problem, where the meta-objective function is non-convex. Therefore, in this paper, we use the Follow-the-Perturbed-Leader (FTPL) [50] to solve the problem, which is the first time that the FTPL is applied to online meta-learning. At round  $t$ , the FTPL optimizes the meta-objective function with perturbed terms over all revealed tasks to obtain the meta parameter  $\phi_{t+1}$  for task  $\mathcal{T}_{t+1}$ . In particular, at round  $t$ , the meta parameter  $\phi_{t+1}$  is obtained by solving the following optimization problem:
$$
\phi_ {t + 1} = \underset {\phi \in \Theta} {\operatorname {a r g m i n}} \sum_ {t ^ {\prime} = 1} ^ {t} \mathcal {L} ^ {v a l} \left(\mathcal {A} l g \left(\lambda , \phi , \mathcal {D} _ {t ^ {\prime}} ^ {t r}\right), \mathcal {D} _ {0, t ^ {\prime}} ^ {v a l}\right) - \sigma_ {t} ^ {\top} \phi , \tag {4}
$$
where  $\sigma_t \in \mathbb{R}^d$  is the random perturbed vector and its components  $\{\sigma_{t,j}\}_{j=1}^d$  is i.i.d sampled from the exponential distribution with a parameter  $\eta > 0$  at each  $t$ . As  $\mathcal{A}lg(\lambda, \phi, \mathcal{D}_{t'}^{tr})$  included in (4) is the optimal solution of the constrained optimization in (3), Problem (4) is a constrained bilevel optimization problem [55].
Algorithm 1 Online Constrained Meta-Learning Framework
Require: Regularization weight  $\lambda > 0$ ; Initial meta parameter  $\phi_1$ ; Perturbed parameter  $\eta$ .