Title: Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
Venue: NeurIPS 2024
Accepted: yes
Keywords: Meta-learning

reinforcement learning
Abstract: Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel optimization framework for meta-RL (BO-MRL) to learn the meta-prior for task-specific policy adaptation, which implements multiple-step policy optimization on one-time data collection. Beyond existing meta-RL analyses, we provide upper bounds of the expected optimality gap over the task distribution. This metric measures the distance of the policy adaptation from the learned meta-prior to the task-specific optimum, and quantifies the model's generalizability to the task distribution. We empirically validate the correctness of the derived upper bounds and demonstrate the superior effectiveness of the proposed algorithm over benchmarks.

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

Abstract

Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel optimization framework for meta-RL (BO-MRL) to learn the meta-prior for task-specific policy adaptation, which implements multiple-step policy optimization on one-time data collection. Beyond existing meta-RL analyses, we provide upper bounds of the expected optimality gap over the task distribution. This metric measures the distance of the policy adaptation from the learned meta-prior to the task-specific optimum, and quantifies the model's generalizability to the task distribution. We empirically validate the correctness of the derived upper bounds and demonstrate the superior effectiveness of the proposed algorithm over benchmarks.

1 Introduction

Meta-learning [58, 15, 25] aims to extract the shared prior knowledge, known as meta-prior, from the similarities and interdependencies of multiple existing learning tasks, in order to accelerate the learning process, increase the efficiency of data usage, and improve the overall learning performance in new tasks. Meta-learning has been extended to solve RL problems, known as meta-RL [15, 5], and shows its promise to overcome the challenges of traditional RL algorithms, including scarce real-world data [3, 44, 65], limited computing resources, and slow learning speed [54, 63].

Meta-learning methods can be generally categorized into optimization-based, model-based (black box methods), and metric-based methods [27, 5]. The optimization-based meta-learning approach [25] is compatible with any model trained by an optimization algorithm, such as gradient descent, and thus is applicable to a vast range of learning problems, including RL problems. Specifically, it formulates meta-learning as a bilevel optimization problem. At the lower-level optimization, the task-specific model is adapted from a shared meta-parameter by an optimization algorithm. At the upper-level optimization, the meta-parameter is to maximize the meta-objective, i.e., the performance of the model adapted from the meta-parameter over training tasks. The existing methods, including MAML and its variants [15, 38, 12], take a one-step gradient ascent as the lower-level policy optimization algorithm, which limits its data inefficiency and leads to sub-optimality.

During the meta-test, MAML conducts one-time data collection, i.e., collecting data using one policy (the meta-policy), and adapts the policy by one step of policy gradient to the new task. However, the collected data is only used in one policy gradient step, which may not sufficiently leverage the data and potentially fail to achieve a good performance. To mitigate the issue, a typical practice is to implement the data collection and the policy gradient alternately multiple times [15]. However, the

environment exploration is usually costly and time-consuming during the meta-test in applications of meta-RL [44, 6, 36]. As a result, the low data efficiency limits the optimality of task-specific policies. In contrast, in this paper, we collect data by meta-policy for one time and utilize multiple policy optimization steps to improve the data efficiency.

The optimality analysis of MAML is studied in [12, 60] with a metric of optimality on the metaobjective , where the error of the meta-objective is defined by the expectation of the optimality gap between the task-specific policy adapted from the learned meta-parameter and the policy adapted from the best meta-parameter [60, 14, 26]. However, the best meta-parameter is shared for all tasks. Even if the meta-objective error is close to zero, i.e., the learned meta-parameter is close to the best one, the model adapted from the learned meta-parameter might be far from task-specific optimum for some tasks. In contrast, we aim to design a meta-RL algorithm that can fit a stronger optimality metric, called near-optimality under all-task optimum , where the comparator, i.e., the policy adapted from the best meta-parameter, is replaced by the task-specific optimal policy for each task. This metric offers a more strict comparator for the model adapted from the learned meta-parameter, i.e., when the metric achieves zero, the policy adaptation produces the optimal policy for every task. A similar metric is studied by [42]. It assumes that the task-specific optimal expert policy for each task is accessible and serves the supervision for policy adaptation during meta-training, which alleviates the analysis difficulty caused by the optimal policy comparator. However, the expert policy supervision is not accessible in a standard meta-RL problem. The metric under all-task optimum is also studied by [9, 10, 65] in the context of supervised meta-learning.

Main contribution. We develop a bilevel optimization framework for meta-RL, which implements multiple-step policy optimization on one-time data collection during task-specific policy adaptation. The overall contributions are summarized as follows. (i) We develop a universal policy optimization algorithm, which performs multiple optimization steps to maximize a surrogate of the accumulated reward function. The surrogate is developed only using one-time data collection. It includes various widely used policy optimization algorithms, including the policy gradient, the natural policy gradient (NPG) [30], and the proximal policy optimization (PPO) [52] as the special cases. Then, to learn the mete-prior, we formulate the meta-RL problem as a bilevel optimization problem, where the lower-level optimization is the universal policy optimization algorithm from the meta-policy and the upper-level optimization is to maximize the meta-objective function, i.e., the total reward of the models adapted from the meta-policy. (ii) We derive the implicit differentiation for both unconstrained and constrained lower-level optimization problems to compute the hypergradient, i.e., the gradient of the meta-objective, and propose the meta-training algorithm. In contrast to [60], we do not require to know the closed-form solution of the lower-level optimization. (iii) We derive upper bounds that quantify (a) the optimality gap between the adapted policy and the optimal task-specific policy for any task, and (b) the expected optimality gap over the task distribution. Since the proposed framework incorporates several existing meta-RL methods, such as MAML, as a special case, the analysis also provides the theoretical motivation for them. (iv) We conduct experiments to validate the theoretical bounds and verify the efficacy of the proposed algorithm on meta-RL benchmarks.

Table 1 compares the solved theoretical challenges of meta-RL between this paper and previous works [12, 57, 60, 42]. Specifically, paper [60] derives the optimality on the meta-objective under the assumption of bounded hypergradient. Papers [12, 57] consider the convergence of the meta-objective. The near-optimality under all-task optimum is considered in [42]. However, it assumes the optimal expert policies of the training tasks are available in meta-training, such that it can learn to approach the expert policies, while the other methods do not require the expert policies and learn from the explorations of the environments. In this paper, we show the convergence and optimality guarantee on the meta-objective, and, more importantly, the optimality guarantee under the all-task optimum comparator. It is noted that the optimality on the meta-objective is an immediate result from [60].

2 Related works.

Categorization of meta-RL. Meta-RL methods can be generally categorized into (i) optimizationbased meta-RL, (ii) black-box (also called context-based) meta-RL. Optimization-based meta-RL approaches, such as MAML [15] and its variants [55, 38], usually include a policy adaptation algorithm and a meta-algorithm. During the meta-training, the meta-algorithm aims to learn a meta-policy, such that the policy adaptation algorithm can achieve good performance starting from the meta-policy. The learned meta-policy parameter is adapted to the new task using the policy adaptation algorithm during the meta-test. Black-box meta-RL [11, 59, 49, 47, 68] aims to learn an end-to-end neural network model. The model has fixed parameters for the policy adaptation during the meta-test, and generates the task-specific policy using the trajectories of the new task takes. In optimization-based meta-RL, the task-specific policy is adapted from a shared meta-policy over the task distribution. The learned meta-knowledge is not specialized for each task, and its meta-test performance on a task depends on a general policy optimization algorithm applied to new data from that task. In contrast, the end-to-end model in black-box meta-RL typically includes specialized knowledge for any task within the task distribution, and uses the new data merely as an indicator to identify the task within the distribution. As a result, the optimality of optimization-based methods is usually worse than black-box methods, especially when the task distribution is heterogeneous and the data scale for adaptation is extremely small. On the other hand, the policy adaptation algorithms in the meta-test of optimization-based methods can generally improve the policy starting from any initial policy, not only the learned meta-policy. Therefore, it is robust to sub-optimal meta-policy and can deal with tasks that are out of the training task distribution [16, 62]. In contrast, due to the specialization of the learned model, black-box methods cannot be generalized outside of the training task distribution. In this paper, we focus on the category of optimization-based meta-RL and compare the proposed algorithm with the existing optimization-based meta-RL approaches in terms of both experimental results and theory.

Bilevel optimization in meta-RL. Bilevel optimization has been widely studied empirically [45, 21, 17, 18, 53, 29] and theoretically [20, 22, 29]. It has been applied to many machine learning problems, including meta-learning [35, 48], hyperparameter optimization [45, 17, 18], RL [24, 34], and inverse RL [39, 40, 41]. Since the overall objective function in bilevel optimization is generally non-convex, theoretical analyses of bilevel optimization mainly focus on the algorithm convergence [20, 29, 64], rarely on the optimality. This paper formulates meta-RL as a bilevel optimization problem. The key theoretical contribution of this paper is to derive upper bounds on the near-optimality under all-task optimum, i.e., the expected optimality of the solutions of the lower-level optimization compared with that of the task-specific optimal policies. The near-optimality under all-task optimum is unique to meta-learning and has not been studied in the literature on bilevel optimization.

3 Problem statement

MDP. A Markov decision process (MDP) M ≜ {S , A , γ, ρ, P, r } is defined by the bounded state space S , the discrete or bounded continuous action space A , the discount factor γ , the initial state distribution ρ over S , the transition probability P ( s ′ | s, a ) : S × A × S → [0 , 1] , and the reward function r : S × A × S → [0 , r max ] .

Policy and value function. A stochastic policy π : S → P ( A ) is a map from states to probability distributions over actions, and π ( a | s ) denotes the probability of selecting action a in state s . For a policy π , the value function is defined as V π ( s ) ≜ E [ ∑ ∞ t =0 γ t r ( s t , a t , s t +1 ) | s 0 = s, π ] . The action-value function is defined as Q π ( s, a ) ≜ E [ ∑ ∞ t =0 γ t r ( s t , a t , s t +1 ) | s 0 = s, a 0 = a, π ] . The advantage function is defined as A π ( s, a ) ≜ Q π ( s, a ) -V π ( s ) . The accumulated reward function is J ( π ) ≜ E s ∼ ρ [ V π ( s )] . Define the discounted state visitation distribution of a policy π as ν π ( s ) ≜ E s 0 ∼ ρ [(1 -γ ) ∑ ∞ t =0 γ t P ( s t = s | π )] . In this paper, we consider parametric policy π θ , parameterized by θ . The optimal parameter θ ∗ can maximize the accumulated reward function, i.e., θ ∗ ≜ argmax θ J ( π θ ) . If θ ∗ is not unique, denote the set of the optimal solutions by Θ ∗ .

Meta-reinforcement learning. Meta-RL aims to solve multiple RL tasks. Consider a space of RL tasks Γ , where each task τ ∈ Γ is modeled by a MDP M τ ≜ {S , A , γ, ρ τ , P τ , r τ } . Correspondingly, the notations V π τ , Q π τ , A π τ , ν π τ , θ ∗ τ , Θ ∗ τ and J τ are defined for task τ . The RL tasks follow a probability distribution P (Γ) . Meta-RL aims to learn a meta-policy π ϕ parameterized by a meta parameter ϕ ,

such that it can adapt to an unseen task τ new ∼ P (Γ) with a few iterations and a small number of new environment explorations. In specific, during the meta-training, several tasks can be i.i.d. sampled from P (Γ) , i.e., { τ j } T j =1 ∼ P (Γ) , and the tasks' MDPs {M τ j } T j =1 can be explored. The meta-learner applies a meta-algorithm to update the meta parameter ϕ by using the data collected from the sampled tasks. During the meta-test, a new task τ new is given, one time of a within-task algorithm A lg with data collected from τ new is applied, the meta-parameter ϕ is adapted to the task-specific parameter θ ′ τ new and the task-specific policy π θ ′ τnew is tested on the task τ new .

Optimality Metric. Consider a meta-RL algorithm that produces a meta-parameter ϕ , and the take-specific parameter π θ ′ τ is adapted from the meta-parameter ϕ on a task τ , denoted as π θ ′ τ = A lg ( π ϕ , τ ) . We define the task-expected optimality gap (TEOG) as the metric to evaluate the algorithm, i.e., E τ ∼ P (Γ) [ J τ ( π θ ∗ τ ) -J τ ( A lg ( π ϕ , τ ))] , where θ ∗ τ is the optimal parameter for task τ . First, the TEOG considers the expected error over the task distribution P (Γ) , reflecting the generalizability of the produced meta-parameter. Second, the TEOG adopts the comparator of the optimal task-specific policy π θ ∗ τ for any task τ (all-task optimum comparator), and evaluates the optimality gap J τ ( π θ ∗ τ ) -J τ ( A lg ( π ϕ , τ )) . In contrast, [60, 14, 26] adopts the comparator of the policy adapted from the optimal meta-parameter π ϕ ∗ , and evaluates the optimality gap J τ ( A lg ( π ϕ ∗ , τ )) -J τ ( A lg ( π ϕ , τ )) . The latter only considers the optimality on the meta-objective, i.e., how well the trained meta-objective can approach the optimal meta-objective. However, even if the error of the meta-objective is approaching zero, i.e., the learned meta-policy is close to the best candidate, the performance of the model adapted from the optimal meta-policy might still be lacking. This is because policy optimization usually requires thousands of value/policy iterations to converge; when tasks are heterogeneous, even if it starts from the best meta-policy, one time of A lg with one time of value estimate may not be sufficient. In contrast, if our metric is zero, the policy adapted from the meta-parameter to any task is optimal for the task.

Policy distance and task variance. To find the solution for a new task within a few iterations of policy optimization, it is crucial that the meta-policy π ϕ can benefit from learning on correlated tasks. Similar to [4, 9, 31], we measure the correlation of tasks in the task distribution P (Γ) by its variance, defined by the minimal mean square of the distances among the optimal task-specific policies, i.e., V ar ( P (Γ)) ≜ min θ min θ ∗ τ ∈ Θ ∗ τ E τ ∼ P (Γ) [ D 2 τ ( π θ , π θ ∗ τ )] . Here, D τ ( π θ , π θ ∗ τ ) is the distance metric between π θ and π θ ∗ τ on the task τ and is defined by D τ ( π θ , π θ ′ ) ≜ √ E s ∼ ν π θ τ [ d 2 ( π θ ( ·| s ) , π θ ′ ( ·| s ))] , where d ( π θ ( ·| s ) , π θ ′ ( ·| s )) is the distance of the policies π θ and π θ ′ on the state s .

̸

Note that the distance metrics D τ ( · , · ) and d ( · , · , s ) can be custom-defined, leading to multiple policy update algorithms, as shown in Section 4. Here, we introduce several examples of d ( · , · , s ) and D τ ( · , · ) , which are commonly used as the distance metrics in RL literature [51, 30, 37]. For policies π θ and π θ ′ , we apply (i) the KL-divergence of the action probability distribution, i.e., d 2 1 ( π θ , π θ ′ , s ) ≜ D KL ( π θ ( ·| s ) ∥ π θ ′ ( ·| s )) , which is similar to the definition in [31]; (ii) The KLdivergence with the other order, i.e., d 2 2 ( π θ , π θ ′ , s ) ≜ D KL ( π θ ′ ( ·| s ) ∥ π θ ( ·| s )) ; (iii) the Euclidean distance of the parameters, i.e., d 2 3 ( π θ , π θ ′ , s ) ≜ ∥ θ -θ ′ ∥ 2 . Correspondingly, for i = 1 , 2 , and 3 , we define D τ,i ( π θ , π θ ′ ) ≜ √ E s ∼ ν π θ τ [ d 2 i ( π θ , π θ ′ , s )] . Note that the distance metrics (i)(ii) are not symmetric, i.e., D τ ( π θ ′ , π θ ′′ ) = D τ ( π θ ′′ , π θ ′ ) , and (iii) is symmetric.

In the subsequent sections, we present algorithms based on the generalized distance definitions of D τ ( · , · ) and d ( · , · , s ) . Moreover, we conduct analyses for the introduced distance metrics, from D τ, 1 to D τ, 3 , to provide comprehensive insights into their respective performances.

4 Meta-Reinforcement Learning Framework

In this section, we develop a meta-RL algorithm by bilevel optimization, where the lower-level optimization is the within-task algorithm that adapts the parameter from the meta-parameter and the upper-level optimization is the meta-algorithm that obtains the meta-parameter. The proposed algorithm has two distinctions compared with existing algorithms. First, it uses one time of a universal policy optimization algorithm as the lower-level within-task algorithm. Second, we derive the hypergradient by the implicit differentiation, where the closed-form solution of the lower-level optimization is not required.

Within-task algorithm. Consider the policy optimization from the meta policy as the within-task algorithm A lg . Specifically, given the meta-parameter ϕ and a task τ , the task-specific policy π θ ′ τ = A lg ( π ϕ , λ, τ ) is defined by θ ′ τ = argmax θ E s ∼ ν π ϕ τ ,a ∼ π θ ( ·| s ) [ Q π ϕ τ ( s, a ) ] -λD 2 τ ( π ϕ , π θ ) . When the action space A is discretized and the policy is tabular, i.e., the probabilities of actions are independent between different states, the above problem can be solved by π θ ′ τ ( ·| s ) =

for all states s ∈ S . When the policy is parameterized by an approximation function, in both continuous and discrete action space A , π θ ′ τ = A lg ( π ϕ , λ, τ ) is computed by θ ′ τ =

In (1) and (2), λ > 0 is a tuning hyperparameter and the distance metric D τ can be arbitrarily chosen. Considering the explorations for the task τ are limited, A lg only needs to evaluate the Q π ϕ τ by Monte-Carlo sampling on a single policy π ϕ , where the data sampling complexity is exactly the same as the one-step gradient descent in MAML [15]. Therefore, we denote A lg , i.e., collecting data on the meta-policy and solving the optimal solution of (1) and (2) as the one-time policy adaptation. More details about the data sample complexity and the computational complexity of (1) and (2) are clarified in Appendix F. On the other hand, one gradient step is usually not sufficient to identify a good policy. Therefore, A lg is to solve the optimal solution of (1) or (2). As shown in Section 5.4, the objective function of (1) or (2) is an approximation of the true objective function J τ ( π ) .

Note that the objective function in (1) and (2) can reduce to that of multiple widely used policy optimization approaches: (i) PPO in [51, 52] when D τ = D τ, 2 ; (ii) a variant of the PPO [60, 37], when D τ = D τ, 1 ; (iii) the proximally regularized policy update, i.e., the policy optimization regularized by Euclidean distance of the policy parameter [51], when D τ = D τ, 3 . Moreover, (iv) if we approximate the expectation in (2) by its first-order approximation and also select D τ = D τ, 3 , the within-task algorithm (2) also can be reduced to one-step policy gradient, as shown in Appendix H; (v) if we use the first-order approximation of the expectation in (2), the second-order approximation of the term D 2 τ ( π ϕ , π θ ) , and select D τ = D τ, 2 , the within-task algorithm (2) is reduced to the natural policy gradient (NPG).

Meta-algorithm. The performance of the meta-parameter ϕ is evaluated by the meta-objective function, which is defined as the expected accumulated reward after the parameter is adapted by the within-task algorithm, i.e., E τ ∼ P (Γ) [ J τ ( A lg ( π ϕ , λ, τ ))] . In the meta-algorithm, we maximize the meta-objective to obtain the optimal meta-parameter ϕ ∗ , i.e.,

As (1) and (2) provide multiple choices of the within-task algorithms when selecting different D τ , the meta-algorithm (3) provides the algorithms to learn the corresponding meta-priors. For example, (3) takes on the role of the meta-PPO algorithm when D τ = D τ, 1 or D τ, 2 , i.e., (3) learns the meta-initialization for PPO. It is a meta-NPG algorithm with the corresponding approximation and D τ . Moreover, when A lg ( π ϕ , λ, τ ) in (2) reduces to the one-step policy gradient shown in (iv) of the last paragraph, (3) represents a precise formulation of MAML in [15]. More details about the formulation and its relations with MAML are shown in Appendix G and H.

Hypergradient computation. Simlar to [29, 64], the meta-algorithm in (3) aims to solve a bilevel optimization problem. In previous works [60], they apply the policy optimizations that have known closed-form solutions as the lower-level within-task algorithms. As a result, the bilevel optimization problem is reduced to a single-level problem. In contrast, in this paper, as we consider a universal policy optimization, its closed-form solution cannot be obtained. To address the challenge, we compute ∇ ϕ A lg ( π ϕ , λ, τ ) and the hypergradient by deriving the implicit differentiation on A lg ( π ϕ , λ, τ ) . As shown in Section 4, the optimization problem A lg ( π ϕ , λ, τ ) is unconstrained in (2), but is constrained in (1) due to ∑ a ∈A π ( a | s ) = 1 . Therefore, we derive the implicit differentiation for both unconstrained and constrained optimization problems. The following proposition shows the hypergradient computation for the tabular policy. Its proof is shown in Appendix J.1.

Proposition 1 (Hypergradient for the tabular policy ) . For the tabular policy in the discrete state-action space, consider any meta-parameter ϕ and the within-task algorithm (1). Let

The computation of ∇ ϕ Q π ϕ τ ( s, · ) is shown in Appendix C. A sufficient condition of M ( s ) being non-singular is that d is locally strongly-convex at π = π θ ′ τ , shown in Appendix J.1. Moreover, when d = d 1 or d = d 2 (correspondingly, D τ = D τ, 1 or D τ = D τ, 2 in (1)), the matrix M ( s ) = λ ∇ 2 π ( ·| s ) d 2 ( π ϕ ( ·| s ) , π ( ·| s )) is always non-singular for any ϕ and M ( s ) is always diagonal, and thus it is easy to compute M -1 ( s ) . The hypergradient computation ∇ ϕ J τ ( π θ ′ τ ) for D τ = D τ, 1 and D τ, 2 is shown in Appendix K.1 and L.1.

The following proposition shows the hypergradient computation for the policy with function approximation. Its proof is shown in Appendix J.2.

Proposition 2 (Hypergradient for the policy with function approximation ) . When a policy is represented by a function approximation, in both the discrete and continuous action spaces, for any meta-parameter ϕ and the within-task algorithm in (2). Let π θ ′ τ = A lg ( π ϕ , λ, τ ) .

A sufficient condition of ∇ ϕ J τ ( π θ ′ τ ) being existent is the objective function of (2) is locally strongly concave at θ = θ ′ τ , as proven in Appendix J.2. The computation of ∇ ϕ Q π ϕ τ ( s, · ) is shown in Appendix C. Note that we need to compute the inverse of the Hessian when computing the hypergradient in Proposition 2. Similar to several widely used RL algorithms, such as TRPO [51] and CPO [1], we apply the conjugate gradient algorithm [23] to compute the inverse of the Hessian, which has demonstrated high efficiency across a wide range of applications of RL and meta-learning [51, 29, 15]. More clarifications about the computation efficiency of the Hessian inverse are shown in Appendix E.

With the hypergradient computations in Proposition 1 and Proposition 2, we apply the stochastic gradient ascent (SGD) to solve the optimization problem in (3). The meta-training of the bilevel optimization framework for meta-RL (BO-MRL) is formally stated in Algorithm 1. The state-action value function in lines 3 and 5 can be estimated by many approaches, including Monte-Carlo sampling used in MAML [15] and vine in [51]. We also propose a practical algorithm of Algorithm 1, as shown in Algorithm 2 in Appendix D, which includes more implementation details of the algorithm and several mechanisms to improve Algorithm 1.

5 Theoretical Results

In this section, we quantify the performance of Algorithm 1, where the softmax policies and several distance metrics introduced in Section 3 are adopted. For convenience, we denote A lg (1) as A lg in (1) and (2) when D τ = D τ, 1 , and denote A lg (2) and A lg (3) in an analogous way. In Section 5.1, we

introduce the softmax policy and necessary assumptions. In the following three sections, we consider two cases of Algorithm 1, including (i) Algorithm 1 with the within-task algorithm A lg (1) and A lg (2) for the tabular softmax policy; and (ii) Algorithm 1 with the within-task algorithm A lg (3) for the softmax policy with function approximation. For the algorithms in (i) and (ii), we study the existence of hypergradient in Section 5.2, derive the convergence guarantees in Section 5.3, and derive the near-optimality under the all-task optimum, i.e., derive the upper bounds of TEOG, in Section 5.4.

5.1 Softmax policy and assumptions

We apply the softmax policies, which are commonly applied in [66, 37, 60], and use the following assumptions on the task τ .

Softmax policies. Consider the softmax policies ˆ π θ parameterized by θ for (i) the tabular policy and (ii) the policy with function approximation. In particular, the tabular policy in a discrete state-action space is defined by ˆ π θ ( ·| s ) ∝ exp( θ ( s, · )) , where θ ∈ R |S|×|A| is a tabular map. The policy with function approximation is defined by ˆ π θ ( ·| s ) ∝ exp( f θ ( s, · )) , where f θ is a function approximation model S × A → R with the parameter θ ∈ R n .

Assumption 1 (Upper bound of advantage function) . For any task τ ∈ Γ and any softmax policy ˆ π θ , | A ˆ π θ τ ( s, a ) | ≤ A max for any a ∈ A and any s ∈ S .

Since the reward r τ ≤ r max is bounded, it is easy to show that | A ˆ π θ τ ( s, a ) | ≤ r max 1 -γ and Assumption 1 always holds. But we still keep Assumption 1 here, since there usually exist A max such that A max ≪ r max 1 -γ . We also have the following assumption and show its remark.

Assumption 2 (Sufficient state visit) . For any task τ ∈ Γ , there exists a constant ϵ > 0 , such that for all bounded parameters ϕ , ν ˆ π ϕ τ ( s ) ≥ ϵ for all s ∈ S .

Remark 1. Here are two sufficient conditions for Assumption 2: (i) For any task τ ∈ Γ , the MDP M τ is ergodic [43, 56]; or (ii) the initial state distribution ρ τ has ρ τ ( s ) > 0 for any s ∈ S .

The proof of Remark 1 is shown in Appendix O. Note that (i) of Remark 1 is a mild condition and is assumed in recent studies on RL algorithm analysis [61, 46].

For the policy with function approximation, we require the following additional assumptions on the approximate function f θ , which are standard or weaker than those in the analysis of meta-learning and meta-RL problems [9, 12, 13, 14].

Assumption 3 (Property of the approximate function) . For any state-action pair ( s, a ) ∈ S × A , (i) the approximate function f θ ( s, a ) are cubic differentiable. (ii) f θ ( s, a ) is L 1 -Lipschitz, i.e., ∥ f θ 1 ( s, a ) -f θ 2 ( s, a ) ∥ ≤ L 1 ∥ θ 1 -θ 2 ∥ for any θ 1 , θ 2 ∈ R n . (iii) ∇ θ f θ ( s, a ) is L 2 -Lipschitz, i.e., ∥∇ θ f θ 1 ( s, a ) -∇ θ f θ 2 ( s, a ) ∥ ≤ L 2 ∥ θ 1 -θ 2 ∥ for any θ 1 , θ 2 ∈ R n , (iv) ∇ 2 θ f θ ( s, a ) is L 3 -Lipschitz, i.e., ∥ ∥ ∇ 2 θ f θ 1 ( s, a ) -∇ 2 θ f θ 2 ( s, a ) ∥ ∥ ≤ L 3 ∥ θ 1 -θ 2 ∥ for any θ 1 , θ 2 ∈ R n .

5.2 Existence of hypergradient.

An essential prerequisite for using Algorithm 1 is that the hypergradients in Propositions 1 and 2 exist. As shown in Section 4, for the tabular policy, when i = 1 or 2 , the hypergradient ∇ ϕ J τ ( A lg ( i ) (ˆ π ϕ , λ, τ )) exists for any ϕ . For the policy with function approximation, we derive the following sufficient condition of the hypergradient being existent. Its proof is shown in Appendix M.

Proposition 3 (Existence of hypergradient for the policy with function approximation) . In both discrete and continuous action space, consider the softmax policy with function approximation shown in Section 5.1. Suppose that Assumptions 1 and 3 hold. If λ > (6 L 2 1 + 2 L 2 ) A max , ∇ ϕ J τ ( A lg (3) (ˆ π ϕ , λ, τ )) always for any ϕ .

5.3 Convergence guarantee

We begin with the convergence guarantee of Algorithm 1 for the tabular policy. The following notations are used in the theorem: B i , C i , G i , K i , M i ( i = 1 and 2 ), where K i ≜ 2( B i +2 C 2 i ) r 2 max (1 -γ ) 4 ,

Theorem 1 (Convergence guarantee for tabular softmax policy ) . Consider the tabular softmax policy in the discrete action space. Suppose that Assumptions 1 and 2 hold. Let { ϕ t } T t =1 be the sequence of meta-parameters generated by Algorithm 1 with λ ≥ 2 A max and the step size α = min { ( r max B i (1 -γ ) 2 + 2 γr max C 2 i (1 -γ ) 3 ) -1 , 1 G i √ T } . Then, the bound: 1 T ∑ T t =1 E [ ∥∇ ϕ E τ ∼ P (Γ) [ J τ ( A lg ( i ) (ˆ π ϕ t , λ, τ ))] ∥ 2 ] ≤ K i T + M i √ . holds for i = 1 or 2 .

The first expectation comes from the random sampling in line 2 of Algorithm 1. The proofs of Theorem 1 are shown in Appendices K.2 and L.2.

The following theorem shows the convergence guarantee for the policy with function approximation. The notations are used in the theorem: B 3 , C 3 , G 3 , K 3 , M 3 , where K 3 ≜ 2( B 3 +2 C 2 3 ) r 2 max (1 -γ ) 4 ,

Theorem 2 (Convergence guarantee for softmax policy with function approximation ) . In both discrete and continuous action space, consider the softmax policy with function approximation. Suppose that Assumptions 1, 2, and 3 hold. Let { ϕ t } T be the sequence of meta-parameters generated by Algo-

t =1 rithm 1 with λ > (6 L 2 1 +2 L 2 ) A max and the step size α = min { ( r max B 3 (1 -γ ) 2 + 2 γr max C 2 3 (1 -γ ) 3 ) -1 , 1 G 3 √ T } . Then, the bound 1 ∑ T t =1 E [ ∥∇ ϕ E τ ∼ P (Γ) [ J τ ( A lg (3) (ˆ π ϕ t , λ, τ ))] ∥ 2 ] ≤ K 3 + M 3 √ . holds.

The first expectation arises from the random sampling in line 2 of Algorithm 1. The proof of Theorem 2 is shown in Appendix M. Theorems 1 and 2 show that the convergence rate of Algorithm 1 is O ( 1 √ T ) and the constants in the notation O are only related to the discount factor γ , the reward bound r max , the bound of the advantage function A max , and the Lipschitz constants of f θ .

5.4 Near-optimality under all-task optimum

Before the derivation of the optimality analysis, we first introduce two intermediate Lemmas.

Lemma 1. Suppose that Assumptions 1, 2 hold. For any task τ , any bounded parameters θ and θ ′ , and i = 1 or 2 , we have J τ (ˆ π θ ′ ) -J τ (ˆ π θ ) ≥ E s ∼ ν ˆ π τ ,a ∼ ˆ π ′ ( ·| s ) [ A ˆ π θ τ ( s,a ) 1 -γ ] -2 γA max (1 -γ ) 2 ϵ D 2 τ,i (ˆ π θ , ˆ π θ ′ ) .

Lemma 2. Consider the softmax policy with function approximation shown in Section 5.1. Suppose that Assumptions 1, 2, and 3 hold. For any task τ , and any softmax policies parameterized by bounded θ and θ ′ , we have J τ (ˆ π θ ′ ) -J τ (ˆ π θ ) ≥ E s ∼ ν ˆ π θ τ ,a ∼ ˆ π θ ′ ( ·| s ) [ A ˆ π θ τ ( s,a ) 1 -γ ] -4 γA max L 2 1 (1 -γ ) 2 ϵ D 2 τ. 3 (ˆ π θ , ˆ π θ ′ ) .

The proofs of Lemmas 1 and 2 are shown in Appendix N.1. Given Lemma 1, when λ = 2 γA max (1 -γ ) ϵ , the within-task algorithm A lg (1 , 2) (ˆ π, λ, τ ) in (1) is actually designed to maximize the right-hand side of the inequality, where ˆ π ′ is the decision variable. Similarly, Given Lemma 2, when λ = 4 γA max L 2 1 (1 -γ ) ϵ , A lg (3) (ˆ π θ , λ, τ ) in (2) maximizes the right-hand side of the inequality, where ˆ π θ ′ is the decision variable. In other words, for each i = 1 , 2 , and 3 , the within-task algorithm A lg ( i ) is to maximize a lower bound of J τ (ˆ π θ ) , denoted as ¯ J τ (ˆ π θ ) . This idea, referred to as the minorization-maximization (MM) [28], is widely used in [51, 33]. The design of A lg ( i ) enables us to connect the accumulated reward of the policy after the policy adaptation with that of the optimal policy ˆ π θ ∗ τ for task τ , i.e., ¯ J τ ( A lg ( i ) (ˆ π ϕ , λ, τ )) ≥ ¯ J τ (ˆ π θ ∗ τ ) , which is a key intermediate result for the optimality analysis.

The final preparatory step is that we borrow the analysis of the meta-training error from [60]. In particular, its theoretical result is encapsulated in the following assumption.

Assumption 4. (Bounding error of meta-objective using gradient) Let F ( i ) ( ϕ ) ≜ E τ ∼ P (Γ) [ J τ ( A lg ( i ) (ˆ π ϕ , λ, τ ))] . For both the tabular policy and the policy with functional approximation, there exists a concave positive non-decreasing function h i : [0 , + ∞ ) → [0 , + ∞ ) , such that max ϕ ′ F ( i ) ( ϕ ′ ) -F ( i ) ( ϕ ) ≤ h i ( ∥∇ ϕ F ( i ) ( ϕ ) ∥ 2 ) .

Assumption 4 assumes the optimality gap of ˆ π ϕ on the meta-objective is upper bounded by an increasing function of its gradient. A sufficient condition of Assumption 4 is provided by [60]. Combine the Assumption 4 and the convergence analysis in Theorems 1 and 2, we can bound the error of the meta-objective, i.e., max ϕ F ( i ) ( ϕ ) -F ( i ) ( ϕ t ) . This result is referred to as the optimality of the meta-objective shown in Table 1. Finally, we derive the upper bounds of the TEOG for both the tabular policy and the policy with function approximation.

Theorem 3 (Optimality guarantee for softmax tabular policy ) . Consider the tabular softmax policy for the discrete state-action space. Suppose that Assumptions 1,2 and 4 hold. Let { ϕ t } T t =1 be the sequence of meta-parameters generated by Algorithm 1 with λ = 2 A max (1 -γ ) ϵ and the step size α shown in Theorem 1. Then, the following holds for i = 1 or 2 : 1 T ∑ T t =1 E t [ E τ ∼ P (Γ) [ J τ (ˆ π θ ∗ τ ) -J τ ( A lg ( i ) (ˆ π ϕ t , λ, τ ))] ] ≤ h i ( K i T + M i √ T ) + 2(1+ γ ) A max (1 -γ ) 2 ϵ V ar i ( P (Γ)) , where ˆ π θ ∗ τ is the optimal softmax policy for task τ and the constants K i and M i are shown in Theorem 1.

Theorem 4 (Optimality guarantee for softmax policy with function approximation ) . In both discrete and continuous action space, consider the softmax policy with function approximation. Suppose that Assumptions 1,2, 3 and 4 hold. Let { ϕ t } T t =1 be the sequence of meta-parameters generated by Algorithm 1 with λ = (6 L 2 1 +2 L 2 ) A max (1 -γ ) ϵ and the step size α shown in Theorem 2. The following holds: 1 T ∑ T t =1 E t [ E τ ∼ P (Γ) [ J τ (ˆ π θ ∗ τ ) -J τ ( A lg (3) (ˆ π ϕ t , λ, τ ))] ] ≤ h 3 ( K 3 T + M 3 √ T ) + ((6+4 γ ) L 2 1 +2 L 2 ) A max (1 -γ ) 2 ϵ V ar 3 ( P (Γ)) , where ˆ π θ ∗ τ is the optimal softmax policy for task τ and the constants K 3 and M 3 are the same as Theorem 2.

The proofs of Theorems 3 and 4, as well as the selection of the hyperparameter λ in these two theorems, are shown in Appendix N.2. The theorems derive the upper bounds of the TEOGs between the parameter adapted by one-time policy adaptation from the produced meta-parameter ϕ t and the task-specific optimal parameter θ ∗ τ . It is shown that, with at most T iterations, we can achieve the upper bounds in the order of O ( h i ( 1 √ T ) + V ar ( P (Γ))) . In other words, there exists a t ≤ T with E τ ∼ P (Γ) [ J τ ( A lg (ˆ π ϕ t , λ, τ ))] ≥ E τ ∼ P (Γ) [ J τ (ˆ π θ ∗ τ )] -O ( h i ( 1 √ T ) + V ar ( P (Γ))) . As the number of iterations T increases, or the variance of the task distribution V ar ( P (Γ)) reduces, the optimality of the meat-parameter ϕ t improves. The second term V ar ( P (Γ)) in the upper bounds of Theorems 3 and 4 corresponds the intuition of meta-learning, which is that, if the variance of a task distribution is smaller, the meta-policy learned from the task distribution is more helpful for new tasks in the task distribution, then the performance is better. Moreover, this term shows that the learned meta-policy achieves a better performance than the meta-policy ϕ center defined by arg min ϕ E τ ∼ P (Γ) [ D 2 τ,i ( π ϕ , π θ ∗ τ )] , which is the center of all the task-specific optimal policies π θ ∗ τ .

The order of our upper bounds are comparable to O ( T -1 4 + V ar ( P (Γ)) that is shown in [31]. On the other hand, compared with [31], in this paper, the constants in the notation O only consist of γ , r max , A max , and the Lipschitz constants of f , and do not rely on |A| and |S| . As a result, our upper bounds are tighter when handling high-dimensional problems or continuous spaces.

Monotonic improvement of the within-task algorithm. Another benefit from Lemmas 1 and 2 and the idea of MM used by the within-task algorithm is that, the policy update by the within-task algorithm monotonically improves, i.e., J τ ( A lg ( i ) (ˆ π θ , λ, τ )) ≥ J τ (ˆ π θ ) for i = 1 , 2 and 3 and any θ and any task τ . Therefore, multiple times of A lg always perform better than one-time A lg .

6 Experiments

6.1 Verification of theoretical results

We conduct an experiment to verify the optimality bounds of Algorithm 1 shown in Theorems 3 and 4. We consider two scenarios of the Frozen Lake environment in Gym: two task distributions with a high task variance and a low task variance. More details of the setting and the hyperparameter selection are shown in Appendix A. We consider the within-task algorithm A lg ( i ) for all i = 1 , 2 and 3 , where the results of i = 2 and 3 are shown in Appendix A.



0

1

2

3

4

Number of policy adaptation steps

-0.5

0.0

0.5

1.0

1.5

Expected Accumulated reward

High task variance (



(1)

applied)

BO-MRL (ours)

Random initialization

MAML

Optimal task-specific policies

0

1

2

3

4

Number of policy adaptation steps

0.0

0.5

1.0

1.5

Expected Accumulated reward

Low task variance (



(1)

applied)

BO-MRL (ours)

Random initialization

MAML

Optimal task-specific policies

No

adaptation

One-time

of



(1)

One-step of

policy gradient

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Expected Optimality Gap

Upper bound for

one-time



(1)

High task variance (

(1)

applied)

No

adaptation

One-time

of



(1)

One-step of

policy gradient

0.00

0.05

0.10

0.15

Expected Optimality Gap

Upper bound for

one-time



(1)

Low task variance (

(1)

applied)



0

1

2

3

Number of policy adaptation steps

-160

-140

-120

-100

-80

-60

-40

Accumulated reward

Half-cheetah, goal velocity

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

ProMP

E-MAML

0

1

2

3

Number of policy adaptation steps

0

100

200

300

400

500

600

Accumulated reward

Half-cheetah, moving direction

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

ProMP

E-MAML

0

1

2

3

Number of policy adaptation steps

40

60

80

100

120

Accumulated reward

Ant, goal velocity

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

0

1

2

3

Number of policy adaptation steps

0

100

200

300

400

500

600

700

Accumulated reward

Ant, moving direction

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

ProMP

E-MAML

We compare our algorithm with MAML [15] and the random initialization. Figure 1 shows that, for Algorithm 1 with the within-task algorithm A lg (1) , it outperforms the baseline methods. For all scenarios, the expected optimality gap of the one-time policy adaptation is smaller than the upper bounds shown in Theorems 3 and 4, which verify our theoretical analysis. Moreover, in Figure 1, the expected optimality gap of the policy adaptation is better (smaller) but close to the upper bound, while that of the other policy adaptation approach, the policy gradient, is worse (larger) than the upper bound. It shows that the derived upper bound is tight.

6.2 High-dimensional Experiment

To evaluate the proposed practical algorithm, Algorithm 2 in Appendix D, we conduct experiments on high-dimensional locomotion settings in the MuJoCo simulator, including Half-Cheetah with goal directions and goal velocities, Ant with goal directions and goal velocities. We compare the proposed algorithm with several optimization-based meta-RL algorithms, including MAML, E-MAML [55], and ProMP [50]. For the fairness of the comparison, all the methods share the same data requirement and task setting. More details of the task setting, the hyperparameter selection, and the supplemental results are shown in Appendix B.

Figure 2 shows that the proposed algorithm with the within-task algorithms A lg ( i ) outperforms the baseline methods in all four experimental settings. For example, we achieve about 25% of performance improvement in Half-cheetah direction and Ant direction experiments. Moreover, compared with the baseline methods, the proposed algorithm achieves more policy improvement when more policy optimization steps are given. For example, our approach achieves about 10% of performance improvement in the second policy optimization step, while those of baseline methods are almost 0% .

7 Conclusion

This paper develops a bilevel optimization framework for meta-RL, which implements multiple-step policy optimization on one-time data collection during task-specific policy adaptation. Beyond existing meta-RL analyses, we provide upper bounds of the expected optimality gap over the task distribution. Our experiments validate the bounds derived from our theoretical analysis and show the superior effectiveness of the proposed framework.

Acknowledgments and Disclosure of Funding

This work is partially supported by the National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like to thank the reviewers for their constructive and insightful suggestions.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. In 2020 IEEE International Conference on Robotics and Automation , pages 2725-2731. IEEE, 2020.

Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradientbased meta-learning. In International Conference on Machine Learning , pages 424-433. PMLR, 2019.

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023.

Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-based meta-reinforcement learning for flight with suspended payloads. IEEE Robotics and Automation Letters , 6(2):1471-1478, 2021.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems . Cambridge University Press, 2011.

Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning , pages 1566-1575. PMLR, 2019.

Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. Advances in Neural Information Processing Systems , 32, 2019.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. In International Conference on Learning Representations , 2017.

Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of debiased model-agnostic meta-reinforcement learning. Advances in Neural Information Processing Systems , 34:3096-3107, 2021.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradientbased model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics , pages 1082-1092. PMLR, 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Generalization of model-agnostic metalearning algorithms: Recurring and unseen tasks. Advances in Neural Information Processing Systems , 34:5469-5480, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In International Conference on Learning Representations , 2018.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning , pages 1165-1173. PMLR, 2017.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 , 2016.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR, 2020.

Magnus R Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving. Journal of research of the National Bureau of Standards , 49(6):409, 1952.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actorcritic. SIAM Journal on Optimization , 33(1):147-180, 2023.

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(9):5149-5169, 2021.

Yu Huang, Yingbin Liang, and Longbo Huang. Provable generalization of overparameterized meta-learning trained with SGD. Advances in Neural Information Processing Systems , 35:1656316576, 2022.

Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning. Artificial Intelligence Review , 54(6):4483-4541, 2021.

David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician , 58(1):30-37, 2004.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , pages 4882-4892. PMLR, 2021.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Vanshaj Khattar, Yuhao Ding, Javad Lavaei, and Ming Jin. A CMDP-within-online framework for meta-safe reinforcement learning. In International Conference on Learning Representations , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems , pages 1008-1014, 2000.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10649-10657, 2019.

Thomas Lew, Apoorva Sharma, James Harrison, Andrew Bylard, and Marco Pavone. Safe active dynamics learning and control: A sequential exploration-exploitation framework. IEEE Transactions on Robotics , 2022.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems , 32, 2019.

Hao Liu, Richard Socher, and Caiming Xiong. Taming MAML: Efficient unbiased metareinforcement learning. In International conference on machine learning , pages 4061-4071. PMLR, 2019.

Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems. Advances in Neural Information Processing Systems , 35:33444-33456, 2022.

Shicheng Liu and Minghui Zhu. Learning multi-agent behaviors from distributed and streaming demonstrations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Shicheng Liu and Minghui Zhu. Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis. In The Twelfth International Conference on Learning Representations , 2023.

Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems , 32, 2019.

Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1451-1458, 2012.

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through metareinforcement learning. In International Conference on Learning Representations , 2018.

Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning , pages 737-746. PMLR, 2016.

Shuang Qiu, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. On finite-time convergence of actor-critic algorithm. IEEE Journal on Selected Areas in Information Theory , 2(2):652-664, 2021.

Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation to new environments via policy-dynamics value functions. In Proceedings of International Conference on Machine Learning , pages 7920-7931, 2020.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning , pages 5331-5340. PMLR, 2019.

Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In International Conference on Learning Representations , 2019.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1723-1732. PMLR, 2019.

Xingyou Song, Yuxiang Yang, Krzysztof Choromanski, Ken Caluwaerts, Wenbo Gao, Chelsea Finn, and Jie Tan. Rapidly adaptable legged robots via evolutionary meta-learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 3769-3776. IEEE, 2020.

Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01118 , 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Yunhao Tang. Biased gradient estimate with drastic variance reduction for meta reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21050-21075. PMLR, 17-23 Jul 2022.

Sebastian Thrun and Lorien Pratt. Learning to learn . Springer Science & Business Media, 2012.

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 , 2016.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. On the global optimality of modelagnostic meta-learning. In International conference on machine learning , pages 9837-9846. PMLR, 2020.

Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:1761717628, 2020.

Zheng Xiong, Luisa M Zintgraf, Jacob Austin Beck, Risto Vuorio, and Shimon Whiteson. On the practical consistency of meta-reinforcement learning algorithms. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems , 2021.

Siyuan Xu and Minghui Zhu. Meta value learning for fast policy-centric optimal motion planning. Robotics Science and Systems , 2022.

Siyuan Xu and Minghui Zhu. Efficient gradient approximation method for constrained bilevel optimization. Proceedings of the AAAI Conference on Artificial Intelligence , 37(10):1250912517, 2023.

Siyuan Xu and Minghui Zhu. Online constrained meta-learning: Provable guarantees for generalization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. Varibad: a very good method for bayes-adaptive deep rl via meta-learning. Proceedings of ICLR 2020 , 2020.

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via metalearning. In International Conference on Learning Representations , 2019.

Appendix for "Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator"

Experimental Supplements

All experiments are executed on a computer with a 5.20 GHz Intel Core i12 CPU.

A Experimental Supplements of Verification of Theoretical Results.

Experimental settings. In Section 6, we use the Frozen Lake environment in Gym [7] and consider a task distribution P (Γ) with high task variance and a task distribution P (Γ) with low task variance. In each distribution, there are 20 tasks. The tasks are characterized by the different settings of holes in the lake, where the holes are generated by random sampling. In the task distribution with high variance, the probability of the appearing hole in each grid is 0 . 3 ; in the task distribution with low variance, its probability is 0 . 1 . We set γ = 0 . 8 , the reward is 1 when reaching the goal, and the reward is -1 when reaching the holes. When deriving the upper bound in Theorems 3 and 4, we approximately regard T be sufficiently large, and O ( h i ( 1 √ T )) be close to 0 . The Lipschitz of the tabular policy is 1 , i.e., L 1 = 1 ; the Lipschitz of the derivative and the second-order derivative of the tabular policy are both 0 , i.e., L 2 = 0 and L 3 = 0 .

Selection of hyper-parameters. We consider the tabular softmax policy and use Monte Carlo sampling to evaluate the Q-value. For the task distribution with high task variance, we set λ = 0 . 5 for A lg (1) , λ = 0 . 5 for A lg (2) , and λ = 0 . 04 for A lg (3) . For the task distribution with low task variance, we set λ = 0 . 25 for A lg (1) , λ = 0 . 25 for A lg (2) , and λ = 0 . 02 for A lg (3) . There is a clarification about the hyper-parameter selection and the verified bound shown in Appendix N.3.

Supplemental results. Figures 3 and 4 show the results of the proposed algorithm with A lg (2) and A lg (3) . It shows that, for all scenarios, the expected optimality gap of the policy adaptation A lg (2) or A lg (3) is smaller than the upper bound shown in Theorems 3 and 4, which verify our theoretical analysis.

0

1

2

3

4

Number of policy adaptation steps

0.0

0.5

1.0

1.5

Expected Accumulated reward

High task variance (



(2)

applied)

BO-MRL (ours)

Random initialization

MAML

Optimal task-specific policies

No

adaptation

One-time

of



(2)

One-step of

policy gradient

0.00

0.25

0.50

0.75

1.00

1.25

Expected Optimality Gap

Upper bound for

one-time



(2)

High task variance (



(2)

applied)

0

1

2

3

4

Number of policy adaptation steps

0.0

0.5

1.0

1.5

Expected Accumulated reward

Low task variance (



(2)

applied)

BO-MRL (ours)

Random initialization

MAML

Optimal task-specific policies

No

adaptation

One-time

of



(2)

One-step of

policy gradient

0.00

0.05

0.10

0.15

Expected Optimality Gap

Upper bound for

one-time



(2)

Low task variance (



(2)

applied)

No

adaptation

One-time

of



(3)

One-step of

policy gradient

0.0

0.5

1.0

1.5

Expected Optimality Gap

Upper bound for

one-time



(3)

High task variance (



(3)

applied)

No

adaptation

One-time

of



(3)

One-step of

policy gradient

0.0

0.2

0.4

0.6

Expected Optimality Gap

Upper bound for

one-time



(3)

Low task variance (



(3)

applied)

B Experimental Supplements of Locomotion.

Experimental settings. We consider locomotion tasks HalfCheetah with goal directions and goal velocities, Ant with goal directions and goal velocities. We follow the problem setups of [67, 15]. In the goal velocity experiments, the moving reward is the negative absolute value between the agent's current velocity and a goal velocity, which is chosen uniformly at random between 0 . 0 and 2 . 0 for the cheetah and between 0 . 0 and 3 . 0 for the ant. In the goal direction experiments, the moving reward is the magnitude of the velocity in either the forward or backward direction, chosen at random for each task τ in P . For the Half-cheetah, the total reward = moving reward - ctrl cost. For the ant, the total reward = healthy reward + moving reward - ctrl cost - contact cost. The horizon is H = 200 , with 20 rollouts per policy adaption step for all problems except the ant direction task, which used 40 rollouts per step.

Selection of hyper-parameters. We apply the proposed practical algorithm of Algorithm 1, Algorithm 2 in Appendix D. We consider the policy as a Gaussian distribution, where the neural network produces the means and variances of the actions. The neural network policy has two hidden layers of size 64, with tanh nonlinearities. We use Monte Carlo sampling to evaluate the Q-value. At the lower-level task-specific policy adaptation, the optimization number by Adam is 50. The models are trained for up to 500 meta-iterations. For the TRPO in meta-parameter optimization, we use the KL-divergence constraint as δ = 1 e -3 .

For the experiment of Half-Cheetah with goal velocities, we set λ = 0 . 5 for A lg (1) , λ = 0 . 4 for A lg (2) . For the experiment of Half-Cheetah with goal directions, we set λ = 0 . 5 for A lg (1) , λ = 0 . 5 for A lg (2) . For the experiment of Ant with goal velocities, we set λ = 0 . 5 for A lg (1) , λ = 0 . 5 for A lg (2) . For the experiment of Ant with goal directions, we set λ = 0 . 5 for A lg (1) , λ = 0 . 5 for A lg (2) .

Comparison setting. We compare the proposed algorithm with several optimization-based meta-RL algorithms, including MAML, E-MAML [55], and ProMP [50]. The experiment results of E-MAML, ProMP, and MAML-TRPO come from [67, 15]. We do not compare the proposed algorithm with black-box meta-RL algorithms, as they are based on the task context and even can achieve good performance without adaptation.

Supplemental results. Figure 5 shows that the proposed algorithm with both within-task algorithms A lg ( i ) outperform the baseline methods in four experimental settings. The accumulated rewards of proposed algorithms increase fast and stop at points with better performance than the baseline methods.

0

60

120

180

240

300

360

420

480

Number of meta-training iterations

-250

-225

-200

-175

-150

-125

-100

-75

Accumulated reward

Half-cheetah, goal velocity

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

0

60

120

180

240

300

360

420

480

Number of meta-training iterations

0

100

200

300

400

500

600

Accumulated reward

Half-cheetah, goal direction

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

0

60

120

180

240

300

360

420

480

Number of meta-training iterations

-50

-25

0

25

50

75

100

125

Accumulated reward

Ant, goal velocity

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

0

50

100

150

200

250

300

350

400

Number of meta-training iterations

100

200

300

400

500

600

Accumulated reward

Ant, goal direction

BO-MRL with



(1)

BO-MRL with



(2)

BO-MRL with



(3)

MAML-TRPO

Algorithm supplement

C Computation of ∇ ϕ Q π ϕ τ ( s, a )

In the computation of meta-objective shown in Propositions 1 and 2, we need to compute ∇ ϕ Q π ϕ τ ( s, a )

where the state-action visitation probability σ ( s,a ) τ,π θ initialized at ( s, a ) ∈ S × A is defined by

For the tabular softmax policy in discrete state-action space shown in Section 5.1,

where ⊙ is the element-wise product, ϕ ( s ′ , · ) is the vector which includes ϕ ( s ′ , a ′ ) for all a ′ ∈ A as the elements, and A ˆ π ϕ τ ( s, · ) is the vector which includes A ˆ π ϕ τ ( s, a ) for all a ∈ A as the elements. Equivalently,

For the softmax policy with the function approximation,

Proof. As shown in [60],

By Lemma 4, from (12), we can obtain (4); from (14), we can obtain (6).

D Practical algorithm

In Sections 4 and 5, we develop a theoretically guaranteed algorithm with Assumptions 1, 2, and 3. In this section, we develop a practical instantiation of Algorithm 1 and evaluate its performance in high-dimensional experiments in Section 6.

Algorithm 2 states the practical algorithm of Algorithm 1. Compared with Algorithm 1, Algorithm 2 considers and overcomes the following limitations of Algorithm 1: (a) evaluating the exact expectation in (1) and (2) is costly and the approximation error could influence the task-specific policy adaptation if using sampling, especially in the meta-RL problem where the sampling data is limited; (b) the optimization problems in (1) and (2) have no closed-form solution; (c) the computation of the gradients of the meta-objectives shown in Propositions 1 and 2 is time-consuming; (d) the gradient-based approach to optimize the meta-objective is not stable in RL problems.

In the beginning of Algorithm 2, we first sample a batch of tasks { τ i } N i =1 ∼ P (Γ) . On each task τ i , we sample the trajectories of the meta-policy π ϕ t as B τ i , and evaluate the state-action value function Q π ϕt τ i ( · , · ) for each τ i . Next, since the number of the sampling state-action pairs in B τ i is limited, if we directly use the sampling average to approximate the expectation in (2), the approximation error will be very large when π ϕ ( a | s ) is small. Therefore, we solve the following optimization problem as the within-task algorithm instead of (2):

where h ( x ) = 2 1+ e -2( x -1) . The function h avoids the term π θ ( a | s ) π ϕ ( a | s ) is optimized to very large. We use Adam [32] to solve the problem in (7). Next, the computation of the gradients of the metaobjectives shown in Proposition 2 is time-consuming, since the computation complexity of the term

Algorithm 2 Practical Algorithm of BO-MRL

Require: Regularization weight λ > 0 ; initial meta-parameter ϕ 0 ; learning rate α .

1: for t = 1 , · · · , T do

2: Sample a batch of tasks { τ i } N i =1 ∼ P (Γ) with the MDP M τ i i.i.d.

3: On each task τ i , sample the trajectories of the meta-policy π ϕ t as B τ i .

4: Evaluate the state-action value function Q π ϕt τ i ( · , · ) for each τ i .

5: For each task τ i , compute the task-specific policy π θ ′ τ i by solving A lg ( λ, ϕ t , τ i ) defined in (7) by Adam.

6: Compute ∇ ϕ J τ i ( π θ ′ τ i ) in (8) by conjugate gradient method

7: Update meta-parameter by the TRPO with the gradient 1 N ∑ i ∇ ϕ J τ i ( π θ ′ τ i ) and the sampling trajectories { B τ i } N i =1 .

8: end for

9: Return ϕ T

-∇ θ π θ ( a | s ) ∇ ⊤ ϕ Q π ϕ τ ( s, a ) is very high. So, we omit the term, and compute ∇ ϕ J τ ( π θ ′ τ ) as

Finally, since the gradient-based approach is not stable in RL problems, we optimize meta-parameter by the TRPO with the gradient 1 N ∑ i ∇ ϕ J τ i ( π θ ′ τ i ) and the sampling trajectories { B τ i } N i =1 , similar to [15].

E Discussion about computational complexity of hyper-gradient

In Algorithms 1 and 2, we compute the inverse of the Hessian matrix when computing the hypergradient by Proposition 2 and (8). The computation of the inverse of the Hessian matrix is not time-consuming and does not increase the processing time much. Here are the two reasons.

First, we apply the conjugate gradient algorithm to compute the inverse of the Hessian and its computation complexity is not high. According to our experiment of Half-cheetah, the computation time of the hyper-gradient with the inverse of Hessian for a three-layer neural network is about 0 . 3 second in each meta-parameter update, where we use only the CPU to compute the hyper-gradient. This approach has demonstrated high efficiency across a wide range of applications, including several widely used RL algorithms, such as TRPO [51] and CPO [1], which compute the inverse of the Hessian in each policy update iteration. The detail is shown in Appendix C of [51]. They usually compute thousands times of the Hessian inverse for a single RL task. In the simplest meta-RL method, MAML [15], the authors use the TRPO to update the meta-parameter, as shown in Section 5.3 of [15], the inverse of the Hessian is also computed. Therefore, the computational complexity of the hyper-gradient in our proposed method is comparable to many existing RL and meta-RL approaches, which are shown efficient.

Second, the biggest computational bottleneck in the meta-RL framework is not the hyper-gradient computation. According to our experiment, the percentage of the computation time in the metaparameter update, including the computation time of the hyper-gradient computation, is less than 5% , where we use only the CPU to compute the hyper-gradient. The percentage of computation time in the data collection and the Q value computation by Monte-Carlo sampling is more than 70% , although the state-action data points are collected in the MDP simulator Gym and the data collection is very fast. In real-world applications, the state-action data points are even harder to collect and

data collection consumes a longer time. Therefore, the computational time of the hyper-gradient computation has a relatively small impact on the mete-RL framework.

F Data sampling complexity and computational complexity of one-time policy adaptation

The one-time policy adaptation in our algorithm is defined as solving the optimal solution of the optimization problem in (1) or (2) by multiple optimization iterations. The definition of the one-time policy adaptation follows many widely used RL algorithms, such as TRPO [51] and CPO [1], which evaluate the Q-values for the current policy and solve the optimal solution for an optimization problem to obtain the next policy in each policy optimization iteration. For example, TRPO solves the optimization problem in (14) of [51] in each iteration.

In the one-time policy adaptation, we only need to evaluate the Q-function for one policy π ϕ by Monte-Carlo sampling, which requires the agent to explore the MDP using one policy π ϕ , then solve the optimization problem in (1) or (2) by multiple optimization iterations with the fixed Q-function. The data sampling complexity is exactly the same as the one-step gradient descent in MAML, which uses Monte-Carlo sampling to evaluate the Q-function and compute the policy gradient based on the Q-function.

The multiple optimization steps in the one-time policy adaptation are different from the multi-step policy gradient update in MAML. In our algorithm, the multiple optimization steps in a one-time policy adaptation only need to evaluate the Q-function for one policy π ϕ , which requires the agent to explore the MDP using only π ϕ . In MAML, the Q-function for a new policy needs to be evaluated in each policy gradient update, and then multiple Q-functions are evaluated for multiple policies, which requires the agent to explore the MDP using multiple policies. Instead, the one-time policy adaptation in our algorithm corresponds to a one-step policy gradient update in MAML, as they use the same number of data points.

Moreover, we would like to claim that the computation complexity for the one-time policy adaptation in our algorithm and that of the one-step policy gradient update in MAML is comparable, although our algorithm requires multiple optimization iterations. As mentioned in Appendix E, the computation time in the data collection and the Q value computation takes more than 70% of total computation time, which is much longer than other parts of the algorithm, including the multiple optimization iterations in police adaptation ( 15% of total computation time). This happens although the state-action data points are collected in the MDP simulator Gym and the data collection is very fast. In real-world applications, the state-action data points are even harder to collect and the consuming time of data collection is much longer. Therefore, the computational time of the multiple optimization iterations has a relatively small impact on the mete-RL framework. Therefore, the computation time of our algorithm and that of MAML is comparable.

From the statement in the above paragraphs, both the data sampling complexity and computational complexity of the one-time policy adaptation in our algorithm and the one-step policy gradient update in MAML are similar. Thus, we define solving the optimal solution of the optimization problem in (1) or (2) as a single policy adaptation step.

G Algorithm details with the first-order approximation

As we mentioned in Section 4, we can approximate the first term E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ π θ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] in (2) by its first-order approximation as the within-task algorithm, similar to the implementations in TRPO [51] and PPO [52]. In particular, the within-task algorithm is reduced to the following formulation,

Here, we use the first-order approximation to replace the first term of (2). In particular, G ( ϕ ) ⊤ ( θ -ϕ ) is the first order approximation of E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ π θ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] , where G ( ϕ ) = ∇ θ E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ ∇ θ π θ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] | θ = ϕ = E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ ∇ ϕ π ϕ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] . Under the simplified within-task algorithm A lg , the hypergradient of the meta-objective function

∇ ϕ J τ ( π θ ′ τ ) can be computed by

where

The computation of ∇ ⊤ ϕ θ ′ τ is derived in Section J.3.

H Connection between the proposed algorithm and MAML

As we claim in Section 4, when we approximate the first term E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ π θ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] in (2) by its first-order approximation and also select D τ = D τ, 3 , the within-task algorithm (2) is reduced to the policy gradient ascent. In particular, the term E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ π θ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] is approximated by ( θ -ϕ ) ⊤ E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ ∇ π ϕ ( a | s ) π ϕ ( a | s ) Q π ϕ τ ( s, a )] , then the within-task algorithm A lg ( π ϕ , λ, τ ) becomes to

Solve the optimization problem, we have

which is policy gradient ascent. Thus, when we select (11) as the within-task algorithm, the metaalgorithm (3) is reduced to the algorithm that can learn the initialization parameter for the policy gradient ascent.

As shown in [15], MAML also learns the initialization parameter ϕ for the policy gradient ascent. However, MAML ignores that the sampled trajectories with policy π ϕ also depend on ϕ . Specifically, MAMLfirst uses the sampled trajectories to approximate Q π ϕ τ ( s, a ) by (Monte Carlo sampling on the REINFORCE algorithm), then computes the policy gradient and does one step of gradient ascent for the task-specific adaptation. Next, it computes ∇ ϕ J τ ( θ ′ τ ) to update the meta-parameter ϕ . When it computes ∇ ϕ θ ′ τ , it treats Q π ϕ τ ( s, a ) as a given data point that is independent with ϕ , and then ignore the ∇ ϕ Q π ϕ τ ( s, a ) . In contrast, our reduced meta-algorithm takes it into account and provides a precise formulation to learn the meta-initialization for the policy gradient algorithm.

Since the proposed meta-RL framework can include MAML as a special case, our analysis in Section 5 also provides the theoretical motivation for MAML.

Analysis and Proof

I Auxiliary Results

Lemma 3 (Policy gradient [56, 2]) . Let π θ be the parameterized policy with the parameter θ . It holds that

Lemma 4 (Policy gradient of the softmax policy) . Consider the softmax policy ˆ π θ parameterized by θ . For a discrete state-action space and the tabular policy, ˆ π θ ( a | s ) = exp( θ ( s,a )) ∑ a ′ ∈A exp( θ ( s,a ′ )) , ∀ ( s, a ) ∈ S × A . It holds that

where ⊙ is the element-wise product, θ ( s, · ) is the vector which includes θ ( s, a ) for all a ∈ A as the elements, A ˆ π θ τ ( s, · ) is the vector which includes A ˆ π θ τ ( s, a ) for all a ∈ A as the elements. Equivalently,

For the softmax policy with function approximation, the policy π θ is defined by π θ ( a | s ) = exp( f θ ( s,a )) ∫ A exp( f θ ( s,a ′ )) da ′ , ∀ ( s, a ) ∈ S × A . It holds that

Proof. For the discrete state-action space and the tabular policy, (12) is shown in Lemma C.1 of [2]. For the softmax policy with function approximation, from Lemma 3, we have

J Proofs of the computation of hypergradient

J.1 Proofs of Propositions 1

Proofs of Propositions 1. Consider the within-task algorithm in discrete space:

Here, d can be selected from d 1 to d 3 defined in Section 3, corresponding to the selection of D τ from D τ, 1 to D τ, 3 .

Here,

Since

The above optimization problem is formally defined by the following problem,

With Assumption 2, the problem is equivalent to that, for any s ∈ S ,

Consider a s ∈ S , the Lagrangian of the above maximization problem is

where µ is the Lagrangian multiplier. The optimality condition of π ( ·| s ) is that,

Here, Q π ϕ τ ( s, · ) denotes a vector include Q π ϕ τ ( s, a ) for each a ∈ A , and π ( ·| s ) denotes a vector include π ( a | s ) for each a ∈ A .

Then, we have

Note that the optimization problem (15) depends on ϕ , and π = A lg ( π ϕ , λ, τ ) is a function of ϕ , we have

i.e., µ is a function of ϕ .

Also, we have

With (17) and (18), we can compute ∇ ϕ A lg ( π ϕ , λ, τ ) , where A lg ( π ϕ , λ, τ ) is continuously differentiable as shown in [64]. We do derivative of (17) and (18) with respect to ϕ , we have [ ∇ ϕ A lg ( π ϕ , λ, τ ) , ∇ ϕ µ ( ϕ )] ⊤ =

where π = A lg ( π ϕ , λ, τ ) .

Solve the equation, we have

where M ( s ) = λ ∇ 2 π ( ·| s ) d 2 ( π ϕ ( ·| s ) , π ( ·| s )) . It is easy to show that ∇ 2 π ( ·| s ) d 2 ( π ϕ ( ·| s ) , π ( ·| s )) is non-singular for any ϕ for any selected d = d 1 , d = d 2 , or d = d 3 .

From the policy gradient theorem in Lemma 3,

where ∇ ϕ π θ ′ τ ( ·| s ) = ∇ ϕ A lg ( π ϕ , λ, τ )( ·| s ) is shown in (19).

J.2 Proofs of Propositions 2

Proofs of Propositions 2. First, we have

From the policy gradient theorem in Lemma 3,

We have

Next, we compute ∇ ϕ θ ′ τ , where

The optimization problem is equivalent to

Similar to the derivation from (15) to (16) with Assumption 2, we have that, when θ = A lg ( π ϕ , λ, τ ) ,

Then, we have

By using implicit differentiation, if the matrix E s ∼ ν π ϕ τ [ -∫ A ∇ 2 θ π θ ( a | s ) Q π ϕ τ ( s, a ) da + λ ∇ 2 θ d 2 ( π ϕ ( ·| s ) , π θ ( ·| s )) ] is invertible, i.e., E s ∼ ν π ϕ τ [ -∫ A π θ ( a | s ) Q π ϕ τ ( s, a ) da + λd 2 ( π ϕ ( ·| s ) , π θ ( ·| s )) ] is strongly convex at θ = θ ′ τ , we have

This is equivalent to

J.3 Proofs of hypergradient of the algorithm in Section G

Deviation of (10). As θ ′ τ = argmin θ -1 λ θ ⊤ E s ∼ ν π ϕ τ ,a ∼ π ϕ ( ·| s ) [ ∇ ϕ π ϕ ( a | s ) π ϕ ( a | s ) A π ϕ τ ( s, a )] + D 2 τ ( π ϕ , π θ ) , by the implicit differentiation theorem in bilevel optimization analysis,

Also, we have and

Then, we can get ∇ ⊤ ϕ θ ′ τ .

K Proofs of convergence when D τ = D τ, 1

K.1 Gradients of ∇ ϕ J τ ( π θ ′ τ ) when D τ = D τ, 1

From Proposition 1,

where

∇ ⊤ ϕ π θ ′ τ ( ·| s ) = ( M ( s ) -1 -M ( s ) -1 1 1 ⊤ M ( s ) -1 1 ⊤ M ( s ) -1 1 ) ( ∇ ⊤ ϕ Q π ϕ τ ( s, · ) -λ ∇ ⊤ ϕ ∇ π ( ·| s ) d 2 1 ( π ϕ , π, s ) ) | π = π θ ′ τ , where

Then,

and

Also,

Then, plugging these equations into (20), we have

where

Then, we simplify the computation of ∇ ⊤ ϕ J τ ( π θ ′ τ ) , we have ∇ ⊤ ϕ J τ ( π θ ′ τ ) =

When the tabular policy is the softmax policy, we have ˆ π ϕ ( a | s ) = exp( ϕ ( s,a )) ∑ a ′ ∈A exp( ϕ ( s,a ′ )) , then

̸

Here, 1 ( s ′ , a ′ ) denote the column vector where the element is 1 if s = s ′ and a = a ′ , otherwise is 0 , for each pair ( s, a ) ∈ S × A ; ˆ π ϕ ( ·| s ′ ) is the column vector, where the element is ˆ π ϕ ( a | s ′ ) if s = s ′ , 0 if s = s ′ , for each pair ( s.a ) ∈ S × A .

So, we have

K.2 Convergence guarantee when D τ = D τ, 1

K.2.1 Auxiliary lemmas

Lemma 5. Suppose that Assumption 2 holds. Let π θ ′ τ = A lg ( π ϕ , λ, τ ) where D τ = D τ, 1 , for any s ∈ S and a ∈ A , we have

Proof. From (16), when D τ = D τ, 1 , we have π θ ′ τ = A lg ( π ϕ , λ, τ ) and

For any s ∈ S , the Lagrangian of the above maximization problem is

where µ is the Lagrangian multiplier. The optimality condition of π ( ·| s ) is that,

Solve the equation,

Let µ 1 ( s ) = -V π ϕ τ ( s ) + µ ( s ) , we have

Then,

We derive the summation of all a ∈ A ,

We have

From (26), we have

So, we have then

Lemma 6. Suppose that Assumption 2 holds. Let π θ ′ τ = A lg ( π ϕ , λ, τ ) where D τ = D τ, 1 , we have

Proof. As shown in (25),

Since ∑ s ∈S ν ˆ π θ ′ τ τ ( s ) = 1 and ∑ a ∈A ˆ π θ ′ τ ( a | s ) = 1 for all s ∈ S , we have ∥∇ ϕ J τ (ˆ π θ ′ τ ) ∥ ≤

1 1 -γ max a,s ∥ ( A ˆ π θ ′ τ τ ( s, a ) -c τ ( s ))( ˆ π θ ′ τ ( a | s ) λ ˆ π ϕ ( a | s ) ∇ ⊤ ϕ Q ˆ π ϕ τ ( s, a ) + 1 ⊤ ( s, a ) -ˆ π ϕ ( ·| s ) ⊤ ) ∥ .

From (23), for any s ∈ S and a ∈ A , we have

Also, for any s ∈ S and a ∈ A ,

From the computation of ∇ ϕ Q ˆ π ϕ τ ( s, a ) shown in (5) of Appendix C,

From Lemma 5,

Also, since ∑ a ∈A ,s ∈S σ ( s,a ) τ, ˆ π ϕ ( s ′ )ˆ π ϕ ( a ′ | s ′ ) = 1 , we have

Therefore, we have

Lemma 7. Suppose that Assumption 2 holds. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 1 , for any s ∈ S we have

and

Proof. From (19), for any s ∈ S ,

From the computations of M ( s ) -1 , ∇ ⊤ ϕ ∇ ˆ π ( ·| s ) d 2 1 (ˆ π ϕ , ˆ π, s ) | ˆ π =ˆ π θ ′ τ , and ∇ ϕ Q ˆ π ϕ τ ( s, · ) in (21) (27), we have

and

From (22)(24)

The last inequality comes from Lemma 5. So, we have

Therefore,

Also, we have

From Lemma D.4 in [2], we have

Moreover, we have

So,

Lemma 8. Suppose that Assumptions 1 and 2 hold. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 1 , we have

where C = 1 λ -A max ( γA max 1 -γ + 2 λ + 2 A max ) and B = 1 λ -A max ( 8 r max (1 -γ ) 3 + λ + A max λ -A max ( (2 -γ ) A max 1 -γ +2 λ +2)) .

Proof. From Lemma 7, we have bounded ∑ a ∈A ∥∇ ϕ ˆ π θ ′ τ ( a | s ) ∥ and ∑ a ∈A ∥∇ 2 ϕ ˆ π θ ′ τ ( a | s ) ∥ . Borrow the result from Lemma D.2 in [2].

K.2.2 Convergence guarantee

Theorem 5. Consider the tabular softmax policy for the discrete state-action space shown in Section 5.1, and the within-task algorithm A lg in (1). Suppose that Assumptions 1 and 2 hold. Let { ϕ t } T t =1 be the sequence generated by Algorithm 1 with D τ = D τ, 1 , λ > A max , and the step size selected as

Then, where

and

Proof. As the smoothness constant of J τ (ˆ π θ ′ τ ) , i.e., J τ ( A lg (ˆ π ϕ , λ, τ ) is obtained in (8), the smoothness constant of E τ ∼ P (Γ) [ J τ ( A lg (ˆ π ϕ , λ, τ ))] is the same, i.e.,

Moreover, from Lemma 6, we have

From the convergence theorem of SDG with smoothness and bounded gradient shown in [19], let the step size

we have

Since E τ ∼ P (Γ) [ J τ ( A lg (ˆ π ϕ T , λ, τ )) -J τ ( A lg (ˆ π ϕ 0 , λ, τ ))] ≤ r max 1 -γ , we have

where and

Corollary 1. Suppose all assumptions and conditions in Theorem 5 hold, and we set λ ≥ 2 A max , then

where B ≜ 16 r max λ (1 -γ ) 3 + 24 1 -γ + 12 λ , C ≜ 6 1 -γ , and G ≜ 4 A max (1 -γ ) 2 .

Proof. Since λ ≥ 2 A max , we have 1 λ -A max ≤ 1 A max and 1 λ -A max ≤ 2 λ . Then, simplify the inequality in Theorem 5.

L Proofs of convergence when D τ = D τ, 2

L.1 Gradients of ∇ ϕ J τ (ˆ π θ ′ τ ) when D τ = D τ, 2

From Proposition 1, we have

where where

Then,

Also,

Specially, A ˆ π θ ′ τ τ ( s, · ) ⊤ M ( s ) -1 1 1 ⊤ M ( s ) -1 1 ⊤ M -1 1 = 0 , because we have

Then,

Here, 1 ( s ′ , a ′ ) denote the column vector where the element is 1 if s = s ′ and a = a ′ , otherwise is 0 , for each pair ( s, a ) ∈ S × A .

L.2 Convergence guarantee when D τ = D τ, 2

L.2.1 Auxiliary lemmas

Lemma 9. Suppose that Assumption 2 holds. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 2 , we have

Proof. As shown in (33)

As shown in proof of Lemma 6 in (27),

we have that

Lemma 10. Suppose that Assumption 2 holds. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 2 , for any s ∈ S , we have

and

Proof. As shown in 30, we have ∇ ⊤ ϕ ˆ π θ ′ τ ( ·| s ) =

where the computations of M ( s ) -1 and ∇ ⊤ ϕ ∇ ˆ π ( ·| s ) d 2 2 (ˆ π ϕ , ˆ π, s ) are shown in (31) (32) and (24), then

Therefore,

Then,

From (27), we have

Then,

And

Moreover, since we have

Then,

From (34),

From (27)

From Lemma D.4 in [2], we have then

Therefore,

Lemma 11. Suppose that Assumptions 1 and 2 hold. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 2 , we have

where C = 2 γ λ (1 -γ ) A max +4 and B = ( 2 γ λ (1 -γ ) A max +4) 2 + 16 r max λ (1 -γ ) 3 +2 .

Proof. Similar to the proof of Lemma 8 by using Lemma 10.

L.2.2 Convergence guarantee

Theorem 6. Consider the tabular softmax policy for the discrete state-action space shown in Section 5.1, and the within-task algorithm A lg in (1). Suppose that Assumptions 1 and 2 hold. Let { ϕ t } T t =1 be the sequence generated by Algorithm 1 with D τ = D τ, 2 and the step size selected as

Proof. Similar to the proof of Theorem 5, by using the gradient bound in Lemma 9 and the smoothness in Lemma 11.

Corollary 2. Suppose all assumptions and conditions in Theorem 6 hold, and we set λ ≥ 2 A max , then

where B ≜ 16 r max λ (1 -γ ) 3 + 18 (1 -γ ) 2 , C ≜ 4 1 -γ , and G ≜ 2 A max (1 -γ ) 2 ) .

Proof. Since λ ≥ 2 A max , we have 1 λ ≤ 1 2 A max . Then, simplify the inequality in Theorem 5.

M Proofs of convergence when D τ = D τ, 3

Lemma 12. Suppose that Assumptions 1, 2, and 3 hold. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 3 . If λ > (6 L 2 1 +2 L 2 ) A max , then ∇ ϕ J τ ( A lg (3) (ˆ π ϕ , λ, τ )) exists for any ϕ , and

Then, where

and

Proof. From Proposition 2, we have where

When D τ = D τ, 3 , and the policy with function approximation is defined by ˆ π θ ( a | s ) ≜ exp( f θ ( s,a )) ∫ A exp( f θ ( s,a ′ )) da ′ , ∀ ( s, a ) ∈ S × A , from Lemma 4,

where ∇ ⊤ ϕ θ ′ τ =

First, we have

and

For the term ∇ ϕ θ ′ τ , consider ∇ θ ′ τ ˆ π θ ′ τ ( a | s ) and ∇ 2 θ ′ τ ˆ π θ ′ τ ( a | s ) , we have

Then,

Then,

So,

Since E s ∼ ν ˆ π ϕ τ [ ∫ A ∇ 2 θ ′ τ ˆ π θ ′ τ ( a | s ) A ˆ π ϕ τ ( s, a ) da ] is a diagonal matrix, the above shown its largest absolute eigenvalue is smaller than (6 L 2 1 + 2 L 2 ) A max . Then, the smallest eigenvalue of E s ∼ ν ˆ π ϕ τ [ -∫ A ∇ 2 θ ′ τ ˆ π θ ′ τ ( a | s ) A ˆ π ϕ τ ( s, a ) da + λI ] is larger than λ -(6 L 2 1 +2 L 2 ) A max . Therefore, if λ > (6 L 2 1 +2 L 2 ) A max ,

Moreover, if λ > (6 L 2 1 +2 L 2 ) A max , the objective function in the optimization problem A lg (ˆ π ϕ , λ, τ ) is strongly concave. Then, from [64], the solution is unique and ∇ ϕ J τ ( A lg (3) (ˆ π ϕ , λ, τ )) exists. From (6),

Then,

Combine (37), we have

So we have

Therefore, we have

Lemma 13. For a softmax policy parameterized by ϕ ,

Proof. From 37,

From 38,

Borrow the result from Lemma D.2 in [2],

Lemma 14. Suppose that Assumption 2 holds. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 3 , for any s ∈ S , we have and

Proof. First consider ∇ ϕ ˆ π θ ′ τ ( a | s ) , we have

Then,

Then,

Next, we consider ∇ 2 ϕ ˆ π θ ′ τ ( a | s ) . From (42), we have

Therefore,

From (40) and (43)

Then,

Next, we consider ∇ 2 ϕ θ ′ τ . We have

where M = E s ∼ ν ˆ π ϕ τ [ -∫ A ∇ 2 θ ′ τ ˆ π θ ′ τ ( a | s ) A ˆ π ϕ τ ( s, a ) da + λI ] and N = E s ∼ ν ˆ π ϕ τ [ ∫ A ∇ θ ′ τ ˆ π θ ′ τ ( a | s ) ∇ ⊤ ϕ Q ˆ π ϕ τ ( s, a ) da + λI ] . Also, we have M -1 N = ∇ ϕ θ ′ τ .

Similar to (37)(38), we can derive the upper bound of ∥∇ 3 ϕ ˆ π ϕ ∥ , then

So, from (38)(39)(40)(41), we have

Simplify the inequality by γ < 1 and 1 -γ < 0 ,

Then,

Lemma 15. Suppose that Assumptions 1, 2, and 3 hold. Let ˆ π θ ′ τ = A lg (ˆ π ϕ , λ, τ ) where D τ = D τ, 3 , we have

Proof. Similar to the proofs of Lemma 8 and Lemma 11 by using Lemma 14.

Theorem 7. In both discrete and continuous action space, consider the softmax policy with function approximation shown in Section 5.1, and the within-task algorithm A lg is defined in (2) with D τ = D τ, 3 . Suppose that Assumptions 1, 2, and 3 hold. If λ > (6 L 2 1 + 2 L 2 ) A max , then ∇ ϕ J τ ( A lg (3) (ˆ π ϕ , λ, τ )) exists for any ϕ .

Let { ϕ t } T t =1 be the sequence generated by Algorithm 1 with λ > (6 L 2 1 +2 L 2 ) A max and the step size

Then, the following bound holds:

where and

Proof. Similar to the proof of Theorem 5, by using the gradient bound in Lemma 12 and the smoothness in Lemma 15.

N Optimality of one-time policy adaptation

N.1 Important Lemmas

Lemma 16. Suppose that Assumptions 1, 2 hold. For any task τ , and any policies π and π ′ , the following bound holds:

where

Here, we define D TV ( π ( ·| s ) || π ′ ( ·| s )) ≜ 1 2 ∑ a ∈A | π ( a | s ) -π ′ ( a | s ) | in a discrete action space or D TV ( π ( ·| s ) || π ′ ( ·| s )) ≜ 1 2 ∫ a ∈A | π ( a | s ) -π ′ ( a | s ) | da in a continuous action space, and D max TV ( π || π ′ ) ≜ max s ∈S D TV ( π ( ·| s ) || π ′ ( ·| s )) .

Proof. Let P π τ is a matrix where P π τ ( i, j ) = E a ∼ π ( ·| s i ) P τ ( s j | s i , a ) and P π ′ τ is a matrix where P π ′ τ ( i, j ) = E a ∼ π ′ ( ·| s i ) P τ ( s j | s i , a ) . Let G = (1 + γP π τ + ( γP π τ ) 2 + . . . ) = (1 -γP π τ ) -1 , and similarly ˜ G = (1 + γP π ′ τ +( γP π ′ τ ) 2 + . . . ) = (1 -γP π ′ τ ) -1 . Let ρ be a density vector on state space and r τ is a reward function vector on state space, thus r ⊤ τ ρ is a scalar meaning the expected reward under density ρ . Note that J τ ( π ) = r ⊤ τ Gρ τ , and J τ ( π ′ ) = r ⊤ τ ˜ Gρ τ . Here, ρ τ is the initial state distribution for task τ . Let ∆ = P π ′ τ -P π τ .

Follow the proof in Appendix B in [51], we have

Left multiply by ˜ G and right multiply by G ,

Left multiply by G and right multiply by ˜ G

Substituting the right-hand side in (45) into ˜ G in (46), then

So we have

Note that r ⊤ τ G = v π τ ⊤ , where v is the value function on the state space. We also have Gρ τ = 1 1 -γ ν π τ , where ν π τ is the state visitation distribution vector. So,

Consider the first term γ 1 -γ v π τ ⊤ ∆ ν π τ , similar to Equation (50) in [51], we have

Since we have ∑ a π ( a | s ) A π τ ( s, a ) = 0 , we have

Combine (47) and the above equation, we have the following for the second term:

Then we need to show

First, by Hölder's inequality,

Similar to (48), each element in the vector γv π τ ⊤ ∆ is ∑ a ( π ′ ( a | s ) -π ( a | s )) A π τ ( s, a ) , then we have

From the Lemma 3 of [1], we have

Therefore, we have

Then the bounds hold.

Lemma 17. Suppose that Assumptions 1, 2 hold. For any task τ , any bounded parameters θ and θ ′ , and i = 1 or 2 , the following bound holds for both i = 1 and 2 :

and

Proof. The proof follows similar lines of Theorem 1 in [51] and Corollary 1 and 2 in [1]. For the sake of self-containedness, we provide the complete proof.

We show the first inequality. The second inequality follows a similar way. From Lemma 16,

From Assumption 2, ν ˆ π θ ( s ) ≥ ϵ for any s ∈ A . Also, D TV (ˆ π θ ( ·| s ) || ˆ π θ ′ ( ·| s )) ≥ 0 for any s ∈ A .

Then, we have

From Jensen's inequality, we have

From the above three inequalities, we have

From [8], we have and

Therefore, and

Lemma 18. Consider the softmax policy with function approximation shown in Section 5.1. Suppose that Assumptions 1, 2, and 3 hold. For any task τ , and any softmax policies parameterized by bounded θ and θ ′ , the following bound holds:

and

Proof. From (36), for any θ ∈ R n ,

Then,

From the mean value theorem, we have

where ϕ ( a ) = δ ( a ) θ +(1 -δ ( a )) θ ′ and 0 ≤ δ ( a ) ≤ 1 . So,

From (49), we have

We use the same way to show another inequality.

N.2 Proof of Theorems 3 and 4

Proof of Theorem 3. When the requirement of Theorem 1, λ ≥ 2 A max , is satisfied, From Assumption 4 and Theorem 1, for both i = 1 and 2 ,

where the constants K i and M i are shown in Theorem 1. The last inequality sign comes from that h i is a concave function and Jensen's inequality.

Let ˆ π θ ′ τ ( ϕ ) = A lg ( i ) (ˆ π ϕ , λ, τ ) for any meta-parameter ϕ . From the definition of the within-task algorithm, we have

This is equivalent to

when λ ≥ 2 γA max (1 -γ ) ϵ , from the second inequality in Lemma 17 and the above inequality,

From the second inequality in Lemma 17,

From the last two inequalities,

i.e.,

Then,

Let ϕ ∗ = arg max ϕ E τ ∼ P (Γ) [ J τ ( A lg ( i ) (ˆ π ϕ , λ, τ ))] , we have

Therefore,

Since we have

Note that in the above analysis, we need λ ≥ 2 A max and also λ ≥ 2 γA max (1 -γ ) ϵ . So, we select we select λ = 2 A max (1 -γ ) ϵ to satisfy the requirement. When λ = 2 A max (1 -γ ) ϵ , we have

From (50) and (51) we have

Proof of Theorem 4. Similar to the above proof of Theorem 3. The difference is using two inequalities in Lemma 18 instead of those in Lemma 17 and using Theorem 2 for convergence instead of Theorem 1.

The requirement of Theorem 2 is λ > (6 L 2 1 + 2 L 2 ) A max , and the requirement of Lemma 18 is λ ≥ 4 γA max L 2 1 (1 -γ ) ϵ . Therefore, we select λ = (6 L 2 1 +2 L 2 ) A max (1 -γ ) ϵ . Then, the bound is

N.3 Clarification of A max

In all the proofs in Sections N.1 and N.1, we can replace as A max to A ′ max , where A ′ max is defined by the maximum advantage function value of policy ˆ π ϕ ′ , where ϕ ′ = arg min ϕ E τ ∼ P (Γ) [ D 2 τ,i (ˆ π ϕ , ˆ π θ ∗ τ )] . It is easy to see A ′ max ≤ A max . For simplification of the assumption statements, theorem statements, and convenience of the proofs, we keep A max in the proofs and Theorems 3 and 4. We actually can make the bound in Theorems 3 and 4 tighter by replacing A max to A ′ max . In the verification of the theoretical results of Section 6, we select λ based on A ′ max and verify the tighter bounds by the experiments.

O Proofs of Remarks

Proof of part (i) of Remark 1 . If the MDP M τ is ergodic, there exists a policy ˆ π such that ν ˆ π τ ( s ) ≥ ϵ 0 . As ϕ is bounded, the probability (or probability density) of each action of the softmax policy is larger than 0 and lower bounded by a ϵ 1 > 0 . Therefore, the action probability of the policy ˆ π ( a | s ) can be upper bounded by ˆ π ϕ ( a | s ) /ϵ 1 for any a . Therefore, ν ˆ π ϕ τ ( s ) ≥ ϵ 0 /ϵ 1 .

Proof of part (ii) of Remark 1 . If the initial state distribution ρ τ has ρ τ ( s ) > 0 for any s ∈ S . Since S is bounded, ρ τ ( s ) ≥ ϵ 2 for any s ∈ S . Then, ν ˆ π ϕ τ ( s ) ≥ (1 - γ ) ϵ 2 .

P Limitations

In this paper, we provide several theorems, where the hyper-parameter selection, e.g., λ , is provided by the theorems. The theoretical analysis usually chooses hyper-parameters, which are sometimes conservative. In practice, we can tune them to improve the performance.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction, including the main contribution statement and related works, accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide the limitations in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs are provided in Appendix

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all details of the information needed to reproduce the main experimental results in the experiment section and in Appendix A and B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the data and code with sufficient instructions in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all training details in Appendix A and B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide it in the section of the experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information in the beginning of the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It is followed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There is no potential societal consequence.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a bilevel optimization framework for meta-reinforcement learning (Meta-RL) named BO-MRL, which aims to enhance policy adaptation through a universal policy optimization algorithm. The framework is designed to improve data efficiency by implementing multiple policy optimization steps on a single data collection during task-specific adaptation. The authors provide theoretical guarantees, including upper bounds on the expected optimality gap over a task distribution, which measure the distance between the adapted policy and the task-specific optimal policy. Empirical validation is conducted on several benchmarks, demonstrating the superior effectiveness of the proposed algorithm compared to existing methods.\n\nSTRENGTHS:\n### S1. Theoretical Contributions\n\nThe paper provides a solid theoretical foundation for the proposed bilevel optimization framework, including convergence guarantees and upper bounds on the expected optimality gap. This strengthens the understanding of the method's performance and its theoretical underpinnings.\n\n### S2. Empirical Validation\n\nThe empirical results presented in the paper are comprehensive and robust, showing significant improvements over state-of-the-art methods on various benchmarks. This demonstrates the practical applicability and effectiveness of the proposed framwork.\n\n### S3. Novel approach\n\nThe introduction of a universal policy optimization algorithm within a bilevel optimization framework is a novel and effective approach to tackling the challenges in Meta-RL. This method addresses both the optimality and data-inefficiency issues commonly faced by existing methods.\n\nWEAKNESSES:\n### W1. Complexity of Implementation\n\nThe proposed framework, while theoretically sound and empirically validated, may be complex to implement in practice. The bi-level optimization and the need for hypergradient computation require careful tuning and expertise, which might limit its accessibility to a broader audience.\n\n### W2. Limited discussion on practical implications\n\nThe paper could benefit from a more detailed discussion on the practical implications and potential limitations of the proposed method in real-world applications. This includes considerations such as computational resources, scalability to large-scale environments, and adaptability to varying task complexities.\n\n### W3. Comparison with Broader Range of Methods\n\nWhile the paper compares the proposed method with several state-of-the-art Meta-RL algorithms, it would be beneficial to include a broader range of methods, especially those outside the immediate Meta-RL context, to provide a more comprehensive evaluation of its performance.\n\nQUESTIONS:\n### Q1. Hyperparameter Sensitivity\n\nHow sensitive is the proposed method to the choice of hyperparameters? \n\n### Q2. Scalability\n\nCan the authors provide more insights into the scalability of the proposed method? How does the computational complexity scale with the number of tasks and the size of the state-action space?\n\n### Q3. Practical Deployment\n\nWhat are the practical considerations for deploying the proposed method in real-world scenarios? Are there any specific challenges or requirements that practitioners should be aware of?\n\nLIMITATIONS:\nThe authors have adequately addressed the theoretical limitations and the assumptions underlying their framework. However, a more detailed discussion on the potential negative societal impacts and how to mitigate them would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper proposes an optimization framework to learn the meta-prior for task-specific policy adaptation.

STRENGTHS:
1. The proposed method claims to be data-efficient in terms of data collection developed only using one-time data collection. Also, this paper aims to solve the RL problem as a bilevel optimization problem.
2. The proposed method considers both unconstrained and constrained optimization problem cases.
4. the upper bounds of the optimality between the adapted policy and the optimal task-specific policy are provided.

WEAKNESSES:
1. the proposed method heavily depends on the minimization problems of eq. 1 or eq. 2, which minimizes the distance between the policy and the predefined policy and finds the optimal policy for the additional task at the same time. My concern and question is that given the same problem that we want to adapt a pre-defined policy \pi to a new task L while keep the original task J, what is the performance difference if i just train the \pi to minimize L+J ? 
2. can you plot the algorithm deployment performance for the example in Fig. 2? since sometimes if you only do \min_\theta ||\pi_\theta - K(x)||, where K(x) is a predefined controller, the \theta converges fast, but the achieved \pi_\theta does not work well in some examples.

QUESTIONS:
see the weaknesses

LIMITATIONS:
yes, the author has provided that
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a bilevel optimization algorithm for Meta-RL, which unlike MAML, implements multiple-step policy optimization on one-time data collection. In addition, the paper provides an upper bound on the expected optimality gap over the task distribution, that quantifies the model’s generalizability to a new task from the task distribution. Experiments show the advantages of the proposed framework over existing meta-learning approaches.\n\nSTRENGTHS:\n* The paper proposes a practical algorithm, supported by a theoretical upper bound of the optimality of the proposed algorithm.\n* The paper is well-written and easy to follow and understand.   \n* The appendix is well-organized and contains all the proofs and discussions about complexity and its relation to MAML.\n* Experiments were performed to verify the theoretical results and show the advantage of the proposed algorithm over MAML.\n\nWEAKNESSES:\n* Comparison to state-of-the-art: the paper compares the proposed algorithm to MAML, EMAML, and ProMP, which were proposed 5 years ago. Since meta-RL is a very active research field, and even MAML has newer and more advanced variants, it is hard to be convinced that the paper compares the proposed algorithm to the most relevant approaches. In addition, the authors did not explain why they chose to compare to those specific baselines, and if the contribution of newer MAML variants can also be incorporated into their algorithm. \n\n* The related work section can be improved. Specifically, the section is a single short paragraph that covers only bilevel optimization algorithms for meta-RL. An extended related work section on meta-RL (even in the appendix) that covers the different types of meta-RL algorithms and places the paper with respect to the various approaches could help the reader understand the paper's contribution.\n\nQUESTIONS:\n* Please address the aforementioned weaknesses.\n\n* In Line 178: does NPG mean Natural Policy Gradient? If yes, maybe state it explicitly. \n\n* Is it possible to formulate a similar bound for other meta-RL algorithms, such as RL2 [1] or VariBAD [2]? \n\n* Lines 32-33 explain that even if the learned meta-parameter $\\phi$ is close to the best one, the model adapted from the learned $\\phi$ might be far from the task-specific optimum for some tasks, since the best meta-parameter is shared for all tasks and learned from the prior distribution that can be with high variance. The proposed algorithm was designed to adapt a stronger optimality metric, where the model is adapted from the task-specific optimal policy for each task. Could you provide a real experiment (even a synthetic experiment) that demonstrates this phenomenon?\\\n\\\nThe existing experiments show that the proposed approach performs better than MAML, but it is still not entirely clear that this leads to the improved performance. A synthetic example can help fully explain this important failure case of existing meta-learning approaches. \n* Perhaps I missed the following point when reading the paper, but during task-specific policy adaptation, the algorithm performs multiple-step policy optimization on one-time data collection, which speeds up the learning process - should it decrease performance (theoretically) compared to one-step policy optimization on one-time data collection?\n\n[1] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv:1611.02779, 2016.\n\n[2] Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-adaptive deep RL via meta-learning. Journal of Machine Learning Research, 22(289):1–39, 2021.\n\nLIMITATIONS:\nThe authors addressed the limitations of the proposed algorithm in section P.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}