Efficient Safe Meta-Reinforcement Learning: Provable Near-Optimality and Anytime Safety
Authors: Siyuan Xu
Minghui Zhu
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
Primary area: reinforcement_learning

Abstract: This paper studies the problem of safe meta-reinforcement learning (safe meta-RL), where an agent efficiently adapts to unseen tasks while satisfying safety constraints at all times during adaptation. We propose a framework consisting of two complementary modules: safe policy adaptation and safe meta-policy training. The first module introduces a novel one-step safe policy adaptation method that admits a closed-form solution, ensuring monotonic improvement, constraint satisfaction at every step, and high computational efficiency. The second module develops a Hessian-free meta-training algorithm that incorporates safety constraints on the meta-policy and leverages the analytical form of the adapted policy to enable scalable optimization. Together, these modules yield three key advantages over existing safe meta-RL methods: (i) superior optimality, (ii) anytime safety guarantee, and (iii) high computational efficiency. 
Beyond existing safe meta-RL analyses, we prove the anytime safety guarantee of policy adaptation and provide a lower bound of the expected total reward of the adapted policies compared with the optimal policies, which shows that the adapted policies are nearly optimal. Empirically, our algorithm achieves superior optimality, strict safety compliance, and substantial computational gains—up to 70\% faster training and 50\% faster testing—across diverse locomotion and navigation benchmarks.

Efficient Safe Meta-Reinforcement Learning: Provable Near-Optimality and Anytime Safety
Siyuan Xu Minghui Zhu
School of Electrical Engineering and Computer Science
The Pennsylvania State University
University Park, PA 16801 (spx5032, muz16) @ @psu.edu
Abstract
This paper studies the problem of safe rentrreeinoor learning (safe meta- RL), where an agent efficiently adapts to unseen tasks while satisfying safety constraints at all times during adaptation. We propose a framework consisting of complementary modules: safe policy adaptation and safe meta-policy training. The first module introduces a novel one-step safe policy adaptation method that admits closed-form solution, ensuring monotonic improvement, constraint satisfaction at every step, and high computational efficiency. The second module develops Hessian-free meta-training algorithm that incorporates safety constraints on the meta-policy and leverages the analytical form of the adapted policy enable scalable optimization. Together, these modules yield three key advantages over existing safe meta-RL methods: (i) superior optimality. (ii) anytime safety guarantee, and (iii) high computational efficiency. Beyond existing safe meta-RL RL analyses, we prove the anytime safety guarantee of policy adaptation and provide lower bound of expected total reward of the adapted policies compared with the optimal policies, which shows that the adapted policies are nearly optimal. Empirically, our algorithm achieves superior optimality, strict safety compliance, and substantial computational gains up to 70% faster training and 50% faster testing-att across diverse locomotion and navigation benchmarks.
Introduction
Reinforcement learning (RL) 47 has achieved significant successes in various domains, from video games (37 46 26 to robotics [28]
27 35 36). The RL problem is formulated
Markov expected total reward. Safe RL (88 55
decision process (MDP) and aims to maximize
15 57
addresses additional safety requirements, such collision avoidance for robots [51 21 and operation restrictions in financial management [[] Typically, the safe RL problem is formulated as constrained MDP (CMDP) [4), which aims to maximize the expected total reward while ensuring that the expected safety costs are below given thresholds. As noted [15], the goals of reward maximization and constraint enforcement are not completely aligned, aggravating the challenge of the inherent trade-off between exploration and exploitation.
Meta-reinforreemet learning (meta-RL) [5] aims to extract common knowledge from multiple existing RL tasks, accelerating the learning process and increasing the data efficiency RL algorithms. Safe meta-RI. (24 55 [59 integrates safe RL and meta-RI. and inherits the benefits of both. On the other hand, existing safe meta-RI. methods face three new challenges: optimality, computational efficiency, and anytime safety. Meta-CRPO [24] considers online safe meta-RL problem. In each round, it computes the task-specific policy by CRPO (55) and updates the meta-policy that has the minimal average distance the task-specific policies of all previous tasks. However, the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Comparison with existing safe meta-RL methods
Theoretical results
Experimental results
Safety
Bounded optimality gap
Methods
Efficiency
Optimality
Constraint violation
Target policy
Positive
Safety for final policy
Low
Low
12
Positive
Safety for adapted policy
Low
Medium
Ours
Zero
Anytime safety
High
High
meta-training does not optimize the performance of the task-specific policy adaptation, and the policies adapted from the learned meta-policy may be sub-optimal for new tasks. Meta-CPO [2]
optimizes the policies adapted from the meta-policy by constraint policy optimization (CPO) 2 Nevertheless, its computational complexity is high in both the meta-training and meta-test stages. Specifically. during the meta-training, meta-CPO solves a constrained bilevel optimization problem [52] where the constraints are present at both the upper and lower levels. The computation involved, particularly the inverse of the Hessian, is computationally expensive. During the meta-test, each policy adaptation step solves a nonconvex constrained optimization problem.
In applications of (safe) meta-RL 38 6) during the meta-test, the agent collects the rewards/costs of state-action pairs by exploring a new, unknown CMDP and optimizes the policy based on the collected data. Therefore, it important to guarantee anytime safety, i.e., the safety constraints must be satisfied for every policy used for the exploration. However, the anytime safety is overlooked in all cxisting safe meta-RL algorithms [2]. Specifically, during the meta-test, they start with the meta-policy and repeatedly adapt the most recent policy into
by the policy adaptation algorithm, which generates a sequence of policies. Except for the final policy in the sequence, each policy, including the initial meta-policy, is used to explore the environment and collect data. Meta-CRPO [24] only quantifies the safety constraint violation of the final convergent policy in the sequence, neglecting that of intermediate policies for data collection. Meta- CPO [12] applies the CPO [2] as the policy adaptation algorithm, which can quantify the safety constraint violation of policies that have undergone least one adaptation step. However, the safety of the meta-policy is ignored. Moreover, both meta-CRPO and meta-CPO provide positive upper bounds of the constraint violation, which do not guarantee zero violation of the safety constraints.
1.1 Main contribution
This paper develops a safe meta-RL framework consisting of two modules: safe policy adaptation and safe meta-policy training. Specifically, we propose novel safe policy adaptation method with three key properties for a single policy adaptation step: (a) it guarantees monotonic improvement, (b) it ensures safety of the adapted policy, and (c) it admits a closed-form solution. For the meta- policy training, we impose safety constraints on the meta-polle meta-policy, derive the meta- vecta-gradien, simplify its computation by leveraging the closed-form expression of the adapted policy, and develop Hessian-free meta-training algorithm.
Leveraging these three properties of the safe policy adaptation method, the proposed algorithms offer three key advantages over existing safe meta-RL methods. (i) Superior optimality. Our safe meta-policy training algorithm maximizes the expected accumulated reward of the policies adapted from the meta-policy, and then improves the optimality of meta-CRPO (24 and naive transfers from meta-RI., which do not consider the task-specific safe policy adaptation in the meta-training. (ii) Anytime safety guarantee during the meta-test. With the imposed safety constraint on the meta- policy, the safe meta-policy training produces safe initial meta-policy. Moreover, as mentioned by (b), the safe policy adaptation guarantees safety for each step when the initial policy is safe. By integrating these two modules, anytime safety is achieved. (iii High computational efficiency in both the meta-test and meta-training stages. As mentioned by (c), we derive the close-formed solution for the policy adaptation. It makes the meta-test much more efficient than those in meta- meta-CRPO [24] and meta-CPO [1],, which solve constrained optimization problems. In the meta-tran veca-training, the close-formed solution of the policy adaptation used to derive Hessian-free meta-gradient and reduces the computation complexity of the proposed algorithm to approach that in the single-level optimization, making it more efficient than meta-CPO [12 and many meta-RI. algorithms [16 30] with the bi-level optimization steps and the computation of Hessian and Hessian inverse. We conduct

---

experiments on seven scenarios including navigation tasks with collision avoidance and locomotion tasks to verify these advantages of the proposed algorithms.
Another major contribution of the paper is that it the first to derive a comprehensive theoretical analysis regarding near optimality and anytime safety guarantees for safe meta-RI. First, we establish the theoretical basis of the algorithm design that guarantees anytime safety, i.c., zero constraint violation for any policy during the policy adaptation. Second, we derive lower bound of the expected accumulated reward of the adapted policies compared to that of the task-specific optimal policies, which shows the near optimality of the proposed safe meta-RL framework. Finally, we demonstrate a trade-off between the optimality bound and constraint violation when the allowable constraint violation varies, which enables the algorithm to be adjusted to prioritize either safety or optimality.
Table I compares both the theoretical and experimental results between this paper and previous works [2 First, we study anytime safety and provide a zero constraint violation guarantee. In previous works, they only provided positive upper bounds for the constraint violation, and the upper bounds only work for the final policy [24] or the adapted policies [12]. Second, although (24) provides an upper bound of the optimality gap, the experimental optimality is the worst. On the other hand, [12] does not provide an optimality bound. In contrast, our method exhibits high optimality and provides a near-optimality guarantee, outperforming existing approaches in terms of both experimental and theoretical outcomes. Third, our method is more efficient than the existing methods [24] 12].
Related works. Due the space limit, we include section of related works in Appendix A
1.2 Notations
Denote the la norm of vectors and the spectral norm (2-norm) of matrices by 1- I|. Denote the Kullback-Leibler divergence (KL-divergence) of probability distributions and q defined on the same sample space X' by (Pl())l4) & Sxx In p(da)
q(dx)) p(dr).
2 Problem Statement
{_1} is defined by the state space S, the CMDP. A CMDP M A 7,,,, (G))==
action space A, the discount factor 7, the initial state distribution p over S, the transition probability P(s'|s,a) a): : S x A x S + [0, 1). the reward function r S x A x S [0, max) J, cost functions where the i-th cost function defined C x A S + [0, c c,naa] for constant d, which is the limit of constraint i. The state space S could be either a discrete space or bounded continuous space. The action space A could either discrete or continuous.
-,... P. and the
Policy. A stochastic policy x : S + ((A) is a mapping from states to probability distributions over action. When A is discrete, x{a)s) denotes the probability of choosing action in state s; when A is continuous, x (als) denotes the probability density. Denote the policy space as II. In addition, exp(fa(e.o)) softmax policy parameterized by 0 E R is denoted as To. where Te(a)s) JA (xp(Jo(ssa))daa"
V(saa) a) E S x A, for continuous action space A. or (a(a))) A space A, and fa S x A + continuous function for any 0.
exp(fa(e.a)) Evee exp(fo(s,a')) for discrete action
E -= r(st, at 8(41) Safe RL. For policy x, the value function is defined as V* E
s, T). The action-value function is defined Q* a) A - / dr 8241)|80 8,00
a, n]. The advantage function defined as A (s, A (*(ssa) - V*(s). The accumulated
reward function is (T)) 2 Ee-p [V*(s]] Similarly, for each = I.... Pp we define (T((s) 2 D (2t,, Gt. 9++1)|ss
x]. Q (s,a) E ( (st, at se+1)(s0 = 8 a, m]. (s, A QT, (s, a) Vg((s, and (T) A Egup [Z(())] The discounted state visitation distribution is defined as vz (s) & (1 - ED P s|T)]. The safe RL problem to maximize the accumulated reward function while the accumulated cost functions satisfy the constraints, i.c., solving the problem max (T) s.l. Je, (T) S de, Vi = -I...., ,P.
Safe meta-RL RL with anytime safety. Safe meta-RL targets multiple safe RL tasks. Consider space of safe RL tasks I, where each task I is modeled by CMDP M, & (S.A, 7, Pri Pr,
3

---

ferr Pi [ds, Following the notions in the above subsections, the notations pr, Pr. Tr; Ci,7 di,r, as well as V. V c.. QT. Qu A7, A J,. Je..r. and V. are defined for task T. Consider a set of safe RL tasks in I following a probability distribution P(T). Safe meta-RI. aims to learn the meta-prior from PPTT) which can be used to train policy for an unseen task Tnew ~ P(T) by a small number of new environment explorations with anytime safety. In specific, during the meta-training tasks can be sampled from P(T'), i.e., (T)) ~ P(TT) and the tasks' CMDPs [Mr, can be explored. During the meta-test, a new task Tnew is given, and the agent explores the CMDP Mr..e and produces the task-specific policy. Note that we consider the meta-training to be an offline stage, e.g. done in simulated environments, the safety constraints may be violated. In contrast, the policies deployed to practical environments during the meta-test. Any policy used to explore M or used to exccute the task Tnew should satisfy the safety constraints.
3 Safe Meta-RL Framework
The proposed safe meta-RL framework aims to learn a meta-policy Ti such that it can adapt to new tasks with anytime safety guarantee. The framework includes two modules: safe policy adaptation, by which the task-specific policy nt' for task is adapted from the meta-policy Tg, and safe meta-training, which identifies the meta-policy Té-
Considering that the amount of data collection is limited, we expect that the task-specific policy T² is adapted from the meta-policy Te by a few safe policy adaptation steps and can achieve good performance and guarantee safety on the new tasks. To achieve this goal, we design the one-step safe policy adaptation in Sc. which achieves significant policy improvement, guarantees safety, and holds high computational efficiency. In Section (.2) the meeta-poliyy training is to optimize the -pec-ipec policy, which is adapted from the meta meta-policy Th by one-step safe policy adaptation.
3.1 One-step safe policy adaptation
Since data collection is limited when a new task is revealed, performing numerous policy adaptation steps to solve the original RL problem becomes impractical, as each step requires collecting a batch of data using the corresponding policy. Accordingly, we define one-step policy adaptation as the policy adaptation that only needs to collect the data by a single policy and derive the method for one-step safe policy adaptation in the remainder of this section.
We derive the optimization problem to achieve one-step safe policy adaptation from the meta-policy, For task T, the policy n² is adapted from the meta-policy To by the safe policy adaptation A*;
argmax EE Eowu t FEll
(s,a) [A7
[A] T
-AE A. [DEz (T(-1s) (+ss))
(s,
s.t. Jeut (TØ) +
+ da E Banero oo [DKL (1(1ss][ S drr + dees (1)
where i= 1, Pp.
Ae,. : Ax,,) and A 2 (8.,..... &., are the hyper-parameters of A The safe policy adaptation A' in problem (T) is inspired by the derivation of CPO 2 where both problem ①I) and CPO aim to approximate the original safe RL problem. Specifically, the objective and constraint functions of problem ( serve as upper bounds of the true objective and constraint functions J, and (T) of the safe RL problem. More details about the upper bounds will be discussed in Lemma Section 5. More importantly, problem I only needs to collect state-action data points and evaluate A for single policy Ta. which keeps the same requirement of data collection as one-step of gradient ascent in MAML [66]. Therefore. A is the one-step safe policy adaptation. On the other hand, considering single gradient ascent in MAML is usually insufficient to identify a policy with good performance and safety, A is to completely solve (
The existence of the solution, the safety, and the monotonic improvement are guaranteed for A Specifically, when setting A O, given that the meta-policy ne is safe for task T, i.e., Je. (TT) < di.,, Vi 1, ....... Pp. for an appropriate hyper-paue parameter A, we have following properties: (i) the feasibility set of problem ( is not empty; (ii) x" is safe for task t, i.e., Je,,(TT) < di,r, (4,r,,ii Vi 1.... P/. (iii) the performance of x-' is better than the meta-policy no. i.e. J-(m²) J+(T)). The
complete statements and proofs of property (i) are shown in Proposition[]on Section properties (ii) and (iii) under selected hyper- paperr-rrameter A are shown in Section Moreover, when the requirement

---

of the constraint satisfaction is not strict, setting 6c, = O for all i in problem may overly restrict the policy update step. To enhance the algorithm's flexibility, we set S & maa as an allowable constraint violation in problem ①I
As mentioned in the above properties (ii) and (iii), both CPO and problem 1 can achieve policy improvement and safety guarantee. However, the computational complexity of directly solving CPO or the constrained optimization problem ( is high. CPO 2 and meta-CPO [12] solve an approximate problem to mitigate the issue, but the computational complexity is still high, meanwhile the safety constraint violation cannot be avoided in theory and also usually appears in practice. In contrast, the safe policy adaptation in (I) is designed to have the closed-form solution under certain Lagrangian multipliers, and then can be efficiently solved, which will be discussed in Section 4.1]
Note that problem ① for the first time, simultaneously offers two key advantages: (a) constraint satisfaction guarantee for single policy optimization step (policy optimization using data collected on a single policy), which enables anytime safety in each policy adaptation step during the meta-test, and (b) the closed-form solution, which significantly reduces the computational complexity of the meta-policy training. The details of the two benefits to the safe meta-RL problem will be discussed in Sections 4.1 and 5 Consequently, it is particularly well-suited for the safe meta-RL problem formulation. As the existing safe policy optimization algorithms, such as hrimal-dull-base algorithm in RCPO [ J. PPO-Lagrangian [43] and CRPO [55] used by meta-CRPO do not hold any of these two benefits, and therefore (I) cannot be replaced by these algorithms. Moreover, although some prior works [6] 34 also derive closed-form solutions of safe policy optimization, safety cannot be guaranteed in each step. Instead, safety is only guaranteed for the final convergent policy, where the trust region size is reduced to O.
3.2 Safe meta-policy training
We obtain the optimal meta-policy TΦ- by solving the following optimization problem:
max E. (T A, r)],, s.t. Jeir (TT) S di,- + Sc,, Vi = 1,
1....,a7 . and VT E I. (2)
Here, [ (A A, T)) is the meta-objective function and is defined by the expected accumulated reward after the parameter is adapted by the policy adaptation, which evaluates the optimality of the meta-policy TΦ. We choose the constraints (T&) S di,r + de,, Vi = I, 1,... .P for any task 7 (similar to problem (T). we set 8e, as the allowable error). There are two reasons to set the constraints. First, as shown in Proposition [ Jeur (T+)) < di,- + de,, Vi = 1, ... P is sufficient condition for that the safe policy adaptation algorithm A A, A, r) has a solution. and further assure the safe meta-policy training (2) is well-defined. Second, the exploration of the CMDP by the meta-policy Te should be safe for each task T to guarantee the initial policy of the policy adaptation is safe. As mentioned in Section 3.1 A A, A,,) is guaranteed to be safe for task when n is safe, and iterative policy adaptation using A is guaranteed to be safe. Therefore, the anytime safety of the policy adaptation is guaranteed. Its formal statement is shown in Section
4 Algorithm
This section introduces the efficient algorithmic solutions to solve problems and 2 respectively.
4.1
Closed-form solution for safe policy adaptation
Based on the design of problem ( we can derive its closed-fore -fom solution under certain Lagrangian multipliers, and then solve the Lagrangian multipliers to obtain the overall solution. We first derive the closed-form solution of problem
and show its existence in the following proposition.
Proposition 1. Suppose that the softmax policy n satisfies Je.. (T&) S +de, Vi L, p. the
solution πt of the optimization problem [1] exists. Under certain mild constraint qualifications, there exists Lagrangian multipliers T with 0 S u',, < oc, such that x (-Iss x exp(fo(s.-)
7 (AT - E' "e A (s, for any s E S. where n 2 X + (1-7) E. ua,rdae
The complete statement of Proposition that includes the sufficient condition for the existence of {ur 1==1 as well as the proof of the proposition are shown in Appendix F.2. F.2.1 Proposition I shows
5

---

that, when the meta- ceeta-poicyyaao vs is softmax, the closed-fome solution of the policy adaptation T is also softmax. The approximate function fo for the meta-policy FΦ is adapted to fo + 1j (A,
" Ac,i)) of '². With this computation, the approximate function of x² can be directly obtained, which much simpler than solving problem (I). More importantly, it can significantly reduce the computational complexity of the meta vreta-gradient, which will be discussed in Section 4.2
In addition, the closed-form solution in Proposition I iiplies the safe policy adaptation (1) can be reduced to the policy adaptation for an unconstrained MDP under the penalized reward function. Specifically, when we define a comprehensive reward function F, - rr Ei= " Ci.r. then the term A Di= Acuir is the advantage function of T for Fr. This implies that problem ①I is equivalent to an unconstrained policy optimization problem, where the reward r- penalized by the negative costs -c,r and the weights of the cost penalty are given by the Lagrangian multiplier u
Proposition 2. Suppose that the assumption in Proposition 7 holds. Ler T" (u _ [ui.-.. Up)) be the policy with n" (-1s) x exp( -)+ Then, the Lagrangian multipliers {u 12
Ei=1, u ) (A"" -s.-)-E;= Li=1 u,A i (i in Proposition 7 iss the solution of the dual problem of
-)).
①1 i.e.
min E
snng To [(A7 - -
- -DEL
Ei= 2:
NER 20
am0m"
where 11" x x Ei Uide, and d', & - (dd,r + 6e, - Ja.r (To)).
Proposition 2 shows the derivation of the Lagrangian multiplier wo... Its proof is shown in Appendix
F.2.2 With ".,.. Note that problem (3) is the dual problem of (T). which is always convex. As a
result, we can apply convex optimization approaches [9] to solve problem (3). and then the solution
of safe policy adaptation (T) can be obtained immediately by Proposition]] ! We provide an optional
algorithm for solving problem (3) and its computational complexity analysis in Appendix E.
Algorithm 1 Safe meta- policy training algorithm
Require: Initial meta-policy Tag allowable constraint violation 6c, defined in Problems and I: for 0,... N do
2: Sample task T with the CMDP M, from the task distribution PPTT)
3: Evaluate Jeg.r (TT.., A,"" (..-) and A (+.)) by sampling data using the meta-policy næ., on task
4: Jeg.r (T...) S d4,r + de;, Vi I.... .p then
5:
Solve the task-specific policy x² and the Lagrangian multipliers u...r( with meta-policy Taa
6:
Evaluate Q7 (.,-) by sampling data using the task task-specific policy x on task T
7:
Compute the meta-gradient VaJ- ) by (4)
8:
Take step of TRPO 44 with using V VoJ-(T)) towards maximize J+("") to obtain Φn+1
9:
else
10:
Choose any i, E {I.... , p} such that Jc,,, (TT...) > din.r + deen
11:
Compute the policy gradient VoJei,,. (To..) x E swx" don aamton (-la) Volon (s, A
Ton (s,a)],
12:
Take a step of TRPO with using VaJa (TT..) towards minimize Jc,, v (TT) to obtain Φ
13:
end if
14:
end for
15:
Return Man
4.2 Safe meta-policy training algorithm
To solve the optimization problem (2) for meta-training, we first consider the computation of the meta-
gradient, i.e., V PPI) (AA (TTr A, A, T))]- The following proposition provides the computation of V (AA(( A, A, r)). Notice that the Lagrangian multipliers fut, in Propositions 1 and aare solved by problem (3), and thus depend on the meta-policy TT. We denote the solved Lagrangian multipliers with Ty as ue,,- (TT) in the following sections.
Proposition 3. Suppose the assumption in Proposition holds. Let π- = AT Under certain conditions, we have that VoJ, (T') exists and Voli(T)) 1. (anerr"" (1ne) -1Q7" (s, n( VoQT (s,a) + Valo(s,a)) (s,
where (TT) A x (1-7) 7) ΣL= We (To) and Q7aa A Q - "e

---

We show the computations of V V. VoQui (.) and Vo (Tp) in Appendices F.3.2 and
The complete statement of Proposition Ihhat includes the sufficient condition of the existence of <z++(x"), as well as the proof of the proposition are shown in Appendix F.3.1 In Proposition the gradient Vaua, ( w.r.t d is the gradient of the solved Lagrangian multipliers, i.e. the optimal solution of problem ( We apply the implicit gradient theorem for constrained optimization
F3.
in [2] to show the existence and the computation of V Voug, (TT). which shown in Appendix
F33.3
We further simplify the computation of the meta-gradient
(T')( E ES (owar(lo)
a) + (TTTT)))) 1 Q7 (s, (s.
(4)
where TTTT A X (1-7) 7) Yi u ua (o and V Q V -E 1 Vo
In 4 take V Fou (TT) (mT)) O in Proposition 3 approximately. On one hand, the computation complexity of V (To) is high, approximation,
shown in Appendix F3.3 On the other hand, under this only omit the small change of the Lagrangian multiplier = (TT) around the meta-policy Tg. i.e, we keep the penalty to constraint violation but treat the weight of the penalty to constraint violation unchanged over small neighbor of a Therefore, the omitted term is higher-order term with a smaller impact on the meta-gradient. Note that, the meta-gradients in many meta-learning approaches include the Hessian computation, such as supervised meta-learning approaches, like MAML and iMAML 16 42 53 meta-RL [16 30 and safe meta-RL approach meta-CPO [12]. In contrast, thanks the closed- form solution (shown in Proposition of the policy adaptation problem ① the meta-gradient in 4 does not include the computations of Hessian and inverse of Hessian w.r.t. Ø which holds a comparable computational complexity as the policy gradient, and therefore is more computationally efficient than the above meta-learning approaches.
The safe meta-policy training algorithm aims solve the optimization problem in 2 and is stated in
Algorithm| [ To handle the constraint imposed on the meta-policy T& in problem (2). we use the idea similar CRPO [55). Specifically, we first check the constraint violation in line 4 If the constraints are not violated, we maximize the meta-objective; otherwise, we minimize the constraint functions. Under this procedure, we always have Je, (To... S dg,r + de,, Vi = L.... when computing the task-specific policy π- = A A, A, T), and therefore the solution of .t always exists, according Proposition[] To stabilize the training, we use the TRPO for the policy update in lines and T2 which only needs the gradient information.
5 Theoretical Results
In this section, we introduce the theoretical results of the safe meta-RL framework. Note that problem (2) is constrained bilevel optimization problem, and the convergence and optimality analysis of solving the problem and obtaining TΦ- are widely studied in [52] 7 31.. So we analyze the performance of the solved meta-policy T&- in our theoretical results. In particular, we introduce the necessary assumptions and notations, derive the performance guarantee for safe policy adaptation A in Section 5.1 and then derive the optimality and safety guarantee of the safe RL. framework in Section 5.2 We introduce an assumption and several notations used in the theoretical results.
Assumption 1. The feasible set of problem (2) is not empty and bounded.
Assumption I supposes problem 2 is well defined and its optimal meta- meta-parameter 6* exists. Since the reward r- S ymaz and Ci,r S cemar, then |A (8, a)l S m /(1 -7) and | e,. (8, a)l mmea /(1 77 are upper bounded. We denote Amar A max+er,=en (s5(s,a) and A Amar xen |A2, (A2,-(s, a)] for each 1,....,. ,.
5.1 Monotonic improvement and anytime safety for policy adaptation
We first introduce a key lemma and show its proof in Appendix F.4.1 Here, we define Dry (T(]s)(-)) (-1s) & I (T(ass) (a|s))
Lemma 1. For any task т, and any policies ा and E II with max,es (rv(T||T) s a Egwey Drv (T(())) || (-1s))]]] we have
(g a)
2 maz
(x'' S (x) + E, (-wr
+
(1 - 71² Eonny [D (DKI(T'(-1)) (111)))]7]](())
1 1-7
7

---

A7 a)
27a maz
J,(x') 2 Jz( E Exwey aa (116)
(1 7)- Enwey [D ( (+1))
1 -
The inequalities also holds when A7, A. Amar and Jr are replaced by A A Age and Jor all == 1....","" .P.
The right-hand sides of the inequalities in Lemmal are the objective function and constraint functions of problem ① The first inequality in Lemma T (applied (=')= shows that the constraint function of problem (I is the upper bound of the accumulated cost Je..r- Therefore, the constraint functions in problem (T) limit the upper bound of Je..((T') to be below the constraint limit, which also applies to Je,. (TT) (TT itself. The second inequality in Lemma (applied to JJ) shows that the objective function of problem (1 the lower bound of J- J-(').. Then, AA' in problem ( to maximize the lower bound of J=(')), which guarantees monotonic improvement. We formalize the results in Proposition and show the proofs in Appendix [.422]
1,.... ,P Proposition 4. Suppose TH satisfies that & is bounded and Je, (Ta) S di,r + &d, Vi=,,
There exists constant a such that, when π. (,, with X 2 2naAmun and do; Z 2yaa
1-y (1-7)= for each i = -,..",, PD, then ( S di,r + de, for each i, and J+(=²) 2 (T
With this proposition, we can derive the properties of monotonic improvement and anytime safety guarantee for the policy adaptation, which is stated in Corollary [ As Assumption | Jssumme the feasible set of problem ② is bounded, we fix the constant a in Proposition
3 for the bounded set.
Corollary 1. Suppose that Assumptions holds. Let X 2
27aAmmn and de; "[1a]] for each I-y
2ya
Let TH [k+1] = [m(TT) A, T,) with 8c, 0 for k E N, where T = TT. being the solution of problem Then, for all l E N. Meur((ii S da, for each and J. (R+1)) (TTg)
When a new task т E is given, we start from the meta-policy Tor, iteratively implement A", and generate a policy sequence (T[H))=oo As indicated in Corollary 1 the constraints are satisfied for each policy in the policy sequence, which shows the anytime safety of the policy adaptations.
5.2 Near-optimality and safety guarantee after one-step policy adaptation
In [.] we show the policy is always monotonically improved from ag- and satisfies the safety constraints during policy adaptation. On the other hand, πT- is learned from the task distribution TP(T), which should be a good initial policy for the task sampled from IP))). In this section, we consider the policy that is obtained from using only one step of policy adaptation from wa- and compare its optimality with the task-specific optimal policy to verify the optiaopy of the proposed safe meta-RL framework. We start by introducing several definitions.
Definitions. Define the optimal policy मद for task T.. A argmax, wen (+( s.t. V S di.r. Define the e-conservatively optimal policy o. T[,[e] which is optimal for under conservative safety constraints, i.e., क. [_,cc] A argmax (TT s.t. (T) S di,--c c, where the conservative constant 0, and π. To, We define the variance of task distribution P(T)) as Var(P(T)) & which the minimal mean square of the distances
ming B B (Dxx(][((ss))] among the optimal task specificc policies πī. and the minimal point is denoted as Φ. Similarly, the task variance under the conservative safety constraints is defined as Var (IP(T)) & min& Ey-P(r) E [D [o ([s)))[] and the minimal point is denoted as T[e), The radius of IP()) IT (+s))]] where the set E is defined by E a {c 2 0: T exists for all 7 E T}. Note that the task variance Var(t) and the radius R is the inherent property of P(T), which measures the similarity of tasks sampled from P(T). For example, if the reward function r and cost o; among tasks are similar, optimal policies -_, are close, then Vark and Rt are close O. With the definitions, the near-optimality and safety guarantee the safe meta-RL is shown in Theorem[] I
is defined R((P(L)) maxreT,ce E a Jatel (TT,e
awu,
Theorem 1. Suppose that Assumptions holds. Let X = 2 Ama
dez 2ya.mma and de, = (1-7)²
4ya maz R(P(T)) - e for all 1,..", ,., where e chosen from
R(P(T)) Let di be 0, (1-7)²
Aradmar
8

---

the solution of problem 2 The solution of A gr, A, T) exists, and we have
47 Var (P(()).
R PeNBET)(J+(A A, ()), 2 B, BANETT)J-
(5)
(1 (1-
4ya
Je,, (T A. 7)) dir - (1 (1- R(P(T)) -c for any T TTT
(6)
Theorem D is proven in Appendix F.4.3 The theorem derives (i) the lower bound of the expected accumulated reward of the policy adapted by one time of A from the meta- partmmeteeeeer Tg. with the comparison to the task-specific (conservatively) optimal policy [e It also derives (ii) the upper
bound of the constraint violation for each task т. We further discuss the tightness of the derived bounds in Appendix G Next, we explore two specific cases of 8c, to illustrate Theorem [
Case 1 (Safety guaranteed), When dc. = 0, the safe constraint is strictly satisfied, i.e., Jenr (T*)
di.- I for any т, but the optimality comparator J- J-(mT with 4-yo A R(P(T)) in 5 is
conservatively optimal (e-conservatively optimal).
Case 2 (Near-optimality). When 8c, = (1-})² R(P(T)), the optimality comparator J, (T
(TT] ) in (5) is all-task optimum, but the constraint is violated at most 4ya.m
(1-7)² R(P(T)).
As shown in Cases ] and 2 there is a trade-off between the optimality of accumulated reward and the safety constraint satisfaction when the allowable constraint violation thresholds &c, vary. In particular, when 6,, is increased, the optimality is improved while the constraint violation increases. As indicated by the optimality-safety trade-off, in the implementation of the proposed algorithm, choose a large &c, when the constraint satisfaction is not required to be strict, and a small & s 0 when the constraint satisfaction is prioritized. The reason for the trade-off is that the constraint function in problem ( approximate the true constraints Je,rr (TT) - di S 0 for any π by only knowing the information (the advantage functions A² at single policy Tg and therefore are more conservative than the true constraints, which leads to loss of optimality. To the best of our knowledge, as anytime safety cannot be guaranteed in the existing framework [24] 12), it is the first time to show the trade-off between optimality and safety, and is also the first to provide an optimality bound with the anytime safe guarantee. Moreover, when choosing e 0, Theorem is reduced to the results in [54] for the unconstrained meta-RL.
Next, we delve into the optimality bound. Consider fixing 8c, and c and then fixing the upper bound of the constraint violation Je,r (TT). Theorem shows that, the performance of meta-RL is improved when the variance of the task distribution Var (IP()) )) is reduced, as π approach the spec-fie optimal policy [el It corresponds the intuition of meta-learning. which that, when the variance of task distribution is smaller, the tasks are more similar, and then the experience learned from the task distribution works better for new tasks sampled from the task distribution.
Experiments
Our experiments aim validate three claimed benefits of the proposed algorithms for safe meta-RI.: (i) superior optimality, i,e, the accumulated rewards of the proposed algorithms can exceed those of baselines; (ii) anytime safety, i.e. all the learned meta-policy and the adapted policies should satisfy the safety constraint; (iii) high computational efficiency for both the meta-training and meta-test.
We conduct experiments on four high- high-ddmeesiionl locomotion scenarios, including Half-Cheetah, Humanoid, Hopper, Swimmer, and three navigation scenarios with collision avoidance, including Point-Cirede Car- Circ--Crreeeeee Hazard, and Point- Button in Gym and Siiiety-Gymmssimm libraries [10] 23 We compare the proposed method with three benchmarks: (a) MAML [6] with constraint penalty; (b) meta-CPO 12] (c) meta-CRPO (24). In (a), we add weighted penalty term for constraint violation to the loss function of the MAMI. Note that (c) is originally designed for online safe meta-RI., where tasks are revealed sequentially during the meta-training. So, provided before the meta-training and it does not have the meta-training stage (Figures [ and do not have meta-training for meta-CRPO). For the faimess of the comparison, all the methods have the same data requirements and task settings. More details about the settings of the tasks, algorithm implementation, and hyper-parameteer are shown in Appendices D. and D.2
(3) with all training tasks

---

Sata mwiss-d (curs)
VAM with perilty
11.5
Nurbor rote-2uRing Bonttrrs
Nombor middd---ate aunttnggg Samtiro
Nurter of polcy adappation stape
Wirter acipeation rteps
Pa Caceerrtteeee
Nox oocumrttooo
2
200 30
230 Number
Numbor teratora
Seratora
polcy
Figure 1: Average accumulated reward (columns and 3, higher is better) and maximal accumulated cost (columns and 4, higher is worse) across validation/test tasks during the meta training (columns and 2) and the meta-test (columns and 4) in Half- CheetCh (row 1) and Point-Ciree (row 2). The accumulated reward and cost during meta-training are computed on the policy adapted one step from the meta-policy. The black dashed line is the constraint of the accumulated cost (helow the line means satisfaction).
Figures and present the experimen- tal results on Half-Cheetah and Point- Circle tasks. Due to page limitations, the results for the other four scenarios
nens- (Oum)) Malainng with parsity
Sith nebs Ril-Bi (our
Mea-cro
Wormed pnd compAraaton
Normal pod con conpopitatoon
MAMI pwrtty
Mata-Dorrd
are deferred to Appendix D.3
1.5
Performance on optimality and Safety. Figure illustrates that the proposed safe meta-RL RL algorithm sub- stantially outperforms all baselines in terms of optimality, achieving approx- imately 50% higher accumulated re- wards than the best-performing baseline during both the meta-training and meta-test phases. Moreover, as depicted in the fourth column of Figure [ our method ensures anytime safety during meta-testing, ie.. the maximal accumulated costs consistently remain below the prescribed safety thresholds, whereas all baselines experience constraint violations at various adaptation stages.
1.
0.0
Meli--ood
Hets-teningg
Figure 2: iteration) and meta-test.
Normalized computation time of the meta-training (per
Efficiency. Figure demonstrates that the proposed algorithm achieves remarkable computational efficiency, reducing the meta- peta-training time by about 70% and the meta-testing time by about 50% relative to meta-CPO. This efficiency gain stems from the closed-form safe policy adaptation and Hessian-free meta-gradient which avoid costly second-order computations common previous meta-RI. methods.
Trade-off' Verification. Finally, we empirically examine the optimality-safety trade-off in Appendix D.4, confirming our theoretical analysis that relaxing the safety tolerance slightly improves the achiev- able reward, while strict constraint satisfaction preserves anytime safety with minimal optimality degradation.
7 Conclusion
This paper presents an efficient framework for safe meta-RL that achieves provable anytime safety and near-optimality. By integrating closed-form one-step safe policy adaptation with a Hessian- free safe meta veeaa-policy training scheme, the proposed method ensures zero constraint violation for every exploration policy, guarantees monotonic performance improvement, and significantly reduces computational cost. We provide first formal analysis establishing both an optimality bound and an explicit safety-optimality trade-off, offering tunable balance between strict safety enforcement and reward maximization. Empirically, our approach outperforms existing safe meta-RI RI. methods in both optimality and efficiency across diverse locomotion and navigation tasks.
10

---

Acknowledgements
This work is partially supported by the National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like to thank the reviewers for their constructive and insightful suggestions.
References
[1] Naoki Abe, Prem Melville, Cezar Pendus, Chandan K Reddy, David L Jensen, Vince P Thomas, James J Bennett, Gary Anderson, Brent Cooley, and Melissa Kowalczyk. Optimizing debt collections using constrained reinforcement learning. In Proceedings of ACM SIGKDD international conference on Knowledge discovery and data mining, pages 75 75-84, 2010.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31, 2017.
[3] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506 2021.
[4] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
[5] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint Stti::2301.00022. 2023.
[6] Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-based meta-reinforreemnnt learning for flight with suspended payloads. IEEE Robotics and Automation Letters, 6(2):1471-1478, 2021.
[7] Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. Implicit differentiation for fast hyperparameter selection non-smooth convex learning. Journal of Machine Learning Research, 23(149):1 43, 2022.
[8] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997.
[9] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[10] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1:06::1006.055440 2016.
[11] Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dur approach to constrained markov decision processes. arXiv preprint arXi::2101.10895 2021.
[12] Minjae Cho and Chuangchuang Sun. Constrained meta-reinforcemmnt learning for adapt- able safety guarantee with differentiable convex programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20975-20983, 2024.
[13] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1-51, 2018.
[14] Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
[15] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. International Conference on Artificial Intelligence and Statistics, pages 3304-3312, 2021.

---

[16] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnosee meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1 1126-11 1135, 2017.
[17] Giorgio Giorgi and Cesare Zuccotti. A tutorial on sensitivity and stability in nonlinear program- ming and variational inequalities under differentiability assumptions. Technical report, DEM Working Paper Series, 2018.
[18) Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan, Yifan Dai, and Bo Cheng. Integrated decision and control: toward interpretable and computationally efficient driving intelligence. IEEE Transactions on Cybernetics, 53(2):859 003999-77722 2022.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018.
[20] Sandy Huang, Abbas Abdolmaleki, Giulia Vezzani, Philemon Brakel, Daniel Mankowitz, Michael Neunert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, et al. A con- strained multi-objective reinforcement learning framework. In Conference on Robot Learning, pages 883-893. PMLR, 2022.
[21] Yanlong Huang. Ekmp:
Generalized imitation learning with adaptation, nonlinear hard con- straints and obstacle avoidance. arXiv preprint arXiv:2103.00455 2021.
[22] Alfredo N Jusem. On the convergence properties of the projected gradient method for convex optimization. Computational & Applied Mathematics, 22:37-52, 2003.
[23] Jiaming Ji, Borong Zhang. Jiayi Zhou, Xuehai Pan, Weidong Huang. Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
[24] Vanshaj Khattar, Yuhao Ding, Javad Lavaei, and Ming Jin. CMDD-within-onllnn onniine framework for meta-safe reinforcement learning. In International Conference on Learning Representations, 2023.
[25] Konwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao. Sanjiban Choudhury, and Steven Z Wu. Learning shared safety constraints from multi-task demonstrations. Advances in Neural Information Processing Systems, 36, 2023.
(26) Dennis Lee, Haoran Tang. Jeffrey Zhang, Huazhe Xu, Trevor Darrell, and Pieter Abbeel. Modular architecture for starcraft ii with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 14, pages 187-193, 2018
[27] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5947):5986, 2020.
[28] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39): 40, 2016.
[29] Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in Neural Information Processing Systems, 32, 2019.
[30] Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- reinforcement learning. In International Conference on Machine Learning, pages 4061-4071, 2019.
[31] Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang, Towards gradient- based bilevel opti- mization with non-convex followers and beyond. Advances in Neural Information Processing Systems, 34:8662 8675,8675550 2021.
[32] Shicheng Liu and Minghui Zhu. Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis. In The Twelfth International Conference on Learning Representations.
12

---

[33] Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems, Advances in Neural Information Processing Systems. 35:33444-33456 2022.
[34] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668, PMLR, 2022.
[35] Gabriel B Margolis, Tao Chen, Kartik Paigwar, Xiang Fu, Donghyun Kim, Sang Bae Kim, and Pulkit Agrawal. Learning to jump from pixels. In Annual Conference on Robot Learning, 2021.
[36] Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43(4):572-587,
2024.
[37] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(75400:529-533 2015.
[38] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference on Learning Representations, 2018.
[39] Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control, 68(3) 2022.
[40] David W Peterson. A review of constraint qualifications in finiteddimensional spaces. Siam Review, 15(3):639-6544 1973.
[41] Nicholas Polosky, Bruno Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference Machine Learning, pages 17801-17810, 2022.
[42] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems, 32, 2019.
[43] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. 2019.
[44] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024 02438, 2015.
[46] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schritwieser,, loannis Antonoglou, Veda Panneershelvam, and Mare Lanctot. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484448) 2016.
[47] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[48] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization.
In International Conference on Learning Representations, 2018.
[49] Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes, In International Conference on Machine Learning, pages 9797-9806. 2020.
[50] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. On the global optimality of model- agnostic meta-learning. In International Conference on Machine Learning, pages 9837-9846, 2020.
13

---

[51] Siyuan Xu and Minghui Zhu. Meta value learning for fast policy-cenne centric optimal motion planning. Roboties Science and Systems, 2022.
[52] Siyuan Xu and Minghui Zhu. Efficient gradient approximation method for constrained bilevel optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):12509 2023.
[53] Siyuan Xu and Minghui Zhu. Online constrained meta-learning: Provable guarantees for generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[54] Siyuan Xu and Minghui Zhu. Meta-reinforcement learning with universal policy adaptation: Provable near-optimality under all-task optimum comparator. arXiv preprint arXiv:2410.09728, 2024.
[55] Tengyu Xu, Yinghin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 11480-1 1441, 2021.
[56] Tsung-Yon Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-bass constrained policy optimization. In International Conference Learning Representations, 2019.
[57] Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning, pages 25636-25655, 2022..
[58] Ming Yu, Zhuoran Yang. Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[59] Zhenyuan Yuan, Siyuan Xu, and Minghui Zhu. All-time safety and sample-efficient meta update for online safe meta reinforcement learning under markov task transition. Machine Learning, 1 114(8):173, 2025.
[60] Jesse Zhang, Brian Cheung. Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adaptation for reinforcement learning in safety-critical settings. In International Conference on Machine Learning, pages 11055-11065, PMLR, 2020.
[61] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338-15344 2020.
14

---

Appendix for "Efficient Safe Meta-Reinforcement Learning: Provable Near-Optimality and Anytime Safety"
A Related Works
Safety metries in safe RL. Safe RL aims to handle the safety requirements in the practical applica- tions of RL. Safe RL typically applies two categories of safety metrics. The first metric is used in
CMDP and is applied in [48 13 15 I 56 41 33 32 It introduces costs associated with state-action pairs based on MDP, and the agent is defined as safe when the expected accumulated costs satisfy given safety constraints. The second metric is stay in the safety region [49] 57 39], which is stricter than the first metric. Specifically, the agent is safe when it remains in desired safe set for any sampled trajectory. In this paper, we consider the anytime safety during policy adaptation, where each policy is required during the exploration of an unknown MDP. It is naturally infeasible to guarantee anytime safety under the second safety metric, as the action to remain in the safety region unknown before the exploration. In contrast, the agent could be safe under the first safety metric even if it visits some undesired states. As a result, we consider the first safety metric.
Solutions of CMDPs. The solutions of the CMDPs can be categorized into (i) penalty function [8]
(ii) primal-dual approaches [48] 13 58 15 M. (iii) trust-region approaches 22 56 6
34]. Existing
works theoretically establish the safety guarantee for both primal-dual approaches [13
58 13 and
trust-region approaches [2]. The primal-dur approaches update the dual variables and the policy simultancously. Therefore, they gradually reduce the total cost below the required threshold by multiple policy optimization steps and can only establish the safety guarantee for the final convergent policy and cannot guarantee anytime safety during policy optimization. Therefore, they cannot meet the anytime safety requirement during policy adaptation in the safe meta-RL, problems, i.e., the safety constraints are satisfied during each step of policy adaptation. In contrast, trust-region approaches constrain the policy within a safe policy set, potentially ensuring safety for every policy during the policy optimization process. However, the computational complexity of existing trust- region approaches is high, especially when applied to the safe meta-RL problem. The safety policy adaptation in this paper belongs to the category of trust-region approaches. On the other hand, we propose a novel safe policy adaptation method to address the computational inefficiency issue.
Cautious adaptation and safe meta-RL. Cautious adaptation [60] and safe meta-RI. both consider to learn prior knowledge to improve the safety level of the adaptations in new environments. On the other hand, cautious adaptation considers the out-of-distribution exploration with the prior learned safety knowledge. The safe meta-RL focuses on in-distribution few-shot learning with safety constraints. Therefore, the safe meta-RL requires less exploration data during adaptation than cautious adaptation, but is limited to in-distribution tasks and less generalizable than cautious adaptation.
Safe meta-RL v.s. httpiiiiiskk/uuttoojjeeetteiiiii safe RL methods. Safe meta-RL. RL, multi- task safe RL (25), and multi-objective safe RL (20] all consider the multiple tasks in the safe RL setting. However, the biggest difference between meta-safe RL and
safe RL is that the agent in meta-safe RI. is required to adapt to a new and unknown environment under few-shot data collection. Therefore, the policy adaptation algorithm is the most important part of meta-safe RL. This paper designs a novel policy adaptation algorithm that holds several benefits for the few-shot policy adaptation that the existing methods do not hold. In contrast, the safe RL learns the policies for multiple tasks during the training stage, where the policy adaptation is not required. Therefore, the multi-task/mi pillii-assk/uutt-oojcctiie can borrow the existing policy optimization methods and do not need to design a new one.
mask/multi-objecee
B Discussion of the relations between CPO by problem
2 and the safe policy adaptation
The safe policy adaptation A in ①1 is inspired by the derivation of CPO, the first optimization problem in Section 5.3 of (2), and replaces the term / /DxL (IT-[s)) | (-1s) in the objective and the
constraint functions of the optimization problem by DrL (TT(|s) Il ( ( Similarly, we derive the
inequalities in Lemma replace the term max, D (T'(ss) [ (s) in Theorem in [44) and replace
the term / /Eewer [D D (+]s))] in Corollary 2 by Reweg (r))ss)) in the right-hand side of the inequalities.
15

---

The modification from 2 to the safe policy adaptation A holds two benefits: (i) performance guarantee and (ii) computational efficiency. First, as Corollary in [2] enables the feasibilityy, the monotonic improvement, and the constraint satisfaction to hold for the solution of the first optimization problem in Section 5.3 of [2) Lemma enables the feasibility, the monotonic improvement and the constraint satisfaction to hold for the safe policy adaptation A².. Second, the modification to the safe policy adaptation A enables us to derive its closed-form solution, which significantly reduces the computational complexity of the meta-safe RL algorithm, mentioned in Section 4.1 On the other hand, one cannot derive the closed-form solution for the first optimization problem in Section 5.3 of 22). and the computational complexity is high. Paper [2] solves an approximate problem to mitigate the issue, but the computational complexity is still high, meanwhile, the safety constraint violation cannot be avoided in theory and also usually appears in practice.
C Comparisons between the proposed safe policy adaptation method and existing hagrangian-based safe RL algorithms
The Lagrangian-bas based policy optimization algorithm, such RCPO 48 PPO-Lagrangian 43] and
CRPO [55] used in meta-CRPO 24], has been widely used to solve safe RL. However, although both the proposed method in Section and the primal-dual method in RCPO, PPO-Lagrangian, and CRPO, are Lagrangian-based safe policy optimization algorithms, they are different. The primal-dual method is much worse than the proposed method and is not suitable for this safe meta-RL problem.
The method in Section 4.1 1 is to solve the safe policy adaptation problem in ①. As mentioned in Section 3.I the safe policy adaptation (T) holds several benefits similar to CPO, including the safety guarantee for a single policy optimization step (using data collected on a single policy) and the monotonic improvement. Morcover, we derive the closed-form solution under certain Lagrangian multipliers for the optimization problem (T). Based on the derived closed-form solution of (T) (shown in Proposition]] Tp.
use the method shown in (3) and 7 to solve the safe policy adaptation problem in which significantly reduces the computational complexity during the meta-training.
In contrast, RCPO and PPO- #PO-Lagrangian do not hold any of the benefits shown in CPO and the proposed algorithm. First, RCPO and PPO-Lagrangian use the gradient ascent steps on the Lagrangian, which do not have the safety guarantee and the monotonic improvement in each policy optimization step, and therefore cannot guarantee anytime safety in the meta-test stage. Moreover, there is no closed-form solution for the policy optimization step in RCPO and PPO- Lagrannggn,, which leads the high computational complexity during the meta-tran veea-training.
D Experimental Supplements
All experiments are executed on computer with a 5.20 GHz Intel Core i12 CPU.
D.1 Task settings
We conduct experiments on totally seven scenarios, which include four high-dimensional locomotion scenarios (Half- -half-Cheetah, Humanoid, Hopper, and Swimmer) in Gym library [10), and three navigation scenarios with collision avoidance (Point- -irele, Gymnasium library (23]. The scenarios are visually illustrated in Figure 3 We use the task setups
Cirr-C-i-eeaainee and Buint-Buttoo) in Safety-
similar those used in previous works on meta-RLaan and safe meta-RI. 12 16. 24 We provide the details of the task setups as follows.
Half-Cheetah. Half-Cheetah (Figure 3 a)) has 17-dimensional state space and a 6-dimensional action space. In the experiment of Chee-Ch the reward is the negative absolute value between the agent's 's current velocity and goal velocity, where the goal velocity characterizes the task. The task distribution is defined by the distribution of the goal velocity, which is a uniform distribution from 0.0 to 2.0. The cost is defined by heheetah ho S dr, i.e. the cost is positive when its head is higher than hg-
Humanoid. Humanoid (Figure b)) has 376-dimensional observation space and a 17-dimensional action space. the experiment of Humanoid, the reward is set as by sin + u2 cos 0, where u and Uy are the velocities along the r-axis and y-axis, and 0 is the walking direction of the humanoid. So
16

---

(a) Half-Cheetah
(b) Humanoid
(c) Hopper
(d) Swimmer
(e) CarCCirele-Hazne
(f) Point-Button
(g) Point-Circle
Figure 3: High High-dimennsooal locomotion tasks and navigation tasks with collision avoidance.
the reward is the velocity along the direction O. The task is characterized by the walking direction 6, which is sampled uniformly from to π n/2. The cost is defined by the control cost of the humanoid robot, i.e., E, ef, where is the torque imposed on each component.
Hopper. Hopper (Figure c) has 12-dimensional state space and 3-dimensional action space. In the experiment of Hopper, the reward is the negative absolute value between the agent's current velocity and goal velocity, where the goal velocity characterizes the task. The task distribution is defined by the distribution of the goal velocity, which is a uniform distribution from 0.0 to 1.0. The cost defined by the control cost of the robot.
Swimmer.
Swimmer (Figure d) has a 8-dimensional state space and a 2-dimensional action space. In the experiment of Swimmer, for different tasks, add Gaussian noise to the state transition, and the variance is uniformly sampled from 0.0 to 0.5 for different tasks; we use the reward defined the negative absolute value between the agent's current velocity and a goal velocity, which is uniform distribution from 0.0 to 1.0, we used the cost defined by the control cost of the swimmer robot, i.e., Σ; of, where c; the torque imposed on each component and the weight is sampled uniformly from 0.5 to 1.
Point-Circle (Figure e) has 28 28-dimensional state space and 2-dimensional action space. In the experiment Point-Circle, a positive reward given when the agent runs in a circle, and positive cost is given when the agent does not stay within the safe region. The setting of the safe region characterizes the task. The task distribution is defined by the distribution of the circle radius and the wall distance. The circle radius is uniform distribution from 1.0 to 1.5 and the wall distance is uniform distribution from 0.55 to 0.75.
Point-Circle.
Circle-Hazard. Car-Cirele-azzard (Figure f) has 60-dimensional state space and 2- dimensional action space. In the experiment of Can-Cirrle-Hazaane positive reward given when the agent runs in circle, and positive cost is given when the agent does not stay within the safe
17

---

the distribution of the position of Table 2: Hyper-parameter setting in A
scenario
X, A
d-
1.0
10.0
0.0
Humanoid
5.0
20.0
0.0
Hopper
1.0
5.0
0.0
Swimmer
0.2
5.0
0.0
Point-Circle
0.5
10.0
0.0
Car-Cirele--Fee
0.5
10.0
0.0
Point-Button 0.5
10.0
0.0
region or collides with Hazards. The setting of the safe region and the hazards characterize the task. The task distribution is defined by the distribution of the circle radius, the distribution of the positions. and the distribution of the number of hazards. The circle radius is a uniform distribution from 0.7 to 1.0 and the number of hazards a uniform distribution from the hazard is uniform distribution over the safety space.
Point-Button. Point-Button (Figure glg) has 56-dimensional state space and 2-dimensional action space. In the experiment of Point-Button, a positive reward is given when the agent touches goal button, and positive cost is given when it does not stay within the safe region and touches any no-goal button or hazards. The setting of the buttons and the hazards characterize the task. The task distribution is defined by the distribution of the number and the positions of buttons and the number and the positions of hazards. Both the number of buttons and the number of hazards is uniform distribution from 10, and the distributions of positions of buttons and hazards are uniform distributions over the safety space.
D.2 Algorithm settings
We apply Algorithm 4 We consider the policy Gaussian distribution, where the neural network produces the means and variances of the actions. The neural network policy has two hidden layers of size 64, with tanh nonlinearities. The horizon is 200, with 40 rollouts per policy adaptation step for all problems the high-dimensional locomotion scenarios. The horizon is 500, with 10 rollouts per policy adaptation step for all problems in the navigation scenarios. The discount factor y 0.99. In each iteration, we sample 10 tasks from the task distribution. Therefore, for each meta-training iteration, the number of the sampled state-action pairs is 50k or 80k. The models are trained for up to 300 meta-iterations in the meta-training. Therefore, the overall number of sampled state-action pairs from 15M or 24M. The policy is tested on 20 tasks and is adapted by 20 iterations for each task in the meta-test. For the TRPO in meta parameter optimization, we use the KL-divergence constraint
le 3. We set A = de, in the safe policy adaptation A" in problem IT. Table 2
shows the setting of X and d- in A for each scenario.
We compare the proposed method with three benchmarks: (a) MAML. [16 with constraint penalty, (b) meta-CPO 11), and meta-CRPO (24. For all methods, we run each algorithm 5 times, including meta-training and meta-test, and show the mean and standard deviation of the evaluation quantities.
D.3 Supplemental results
Figures and show the experimental results in Humanoid, Hopper, Car-Circle-Hazanr and Point- Button. Note that meta-CRPO is not designed for offline optimization of meta-policy, and then there is no meta-training result for the approach. Due to the high dimension the Humanoid tasks, the meta-training of meta-CPO is too slow (10 times slower than the proposed method) in Humanoid tasks. It is extremely consumnng to run the meta-training of meta-CPO multiple times on humanoid tasks and draw its figure. So the result of meta-CPO is not shown in Fig
Figure shows that the proposed safe meta-RL algorithm significantly outperforms all the baseline methods regarding the optimality, i.e. the accumulated reward during both the meta-training and the meta-test in all the scenarios. Moreover, it shows that the proposed algorithms achieve anytime safety during the meta-test, i.c., the maximal accumulated costs always satisfy the constraints, while
18

---

the baselines cannot achieve it. Figure S shows that our algorithm is much more efficient than the baselines in both the meta-trame meta-training and meta-test stages.
Sale meto RL (ours)
Meta CPO
Meta CRPO
MAML with penally
Huroraikteen smbiig
Mtx 900 da Cooa
1000
300
Namber of mosatta nin laratona
Nusterofnelateeee Deralora
Number aripestion
Numter
Nurit pollcy araptaion step
Number pollyy adapation Stepe
Swat Daderraeeaaaaarrrg
Svicers Mota yoneey
Swirenn Mera
Se Rermenrrato
MO RCCUT LAMIN Terwerd
Narrter of marts martetaiing Partooss
14 umber of medetter eg tondloe
Cond
scuuettee sevel
Max www.maaeeee 0o1.
humbbr of polly adepation Aass
Narber of poliy scaptalion shgp Po Rocssertraeleeee.e
Pratt-Sdteeellsaacceee. nirg
Print Set.eeetttttttteeeeeee nirg
Pren Mertie
Aoarag sntrmbiee reage
25
12
Maa
Narier www.tttttggggg........... Intlicro
200
Namierr Foo
Figure 4: Average accumulated reward (columns and 3, higher is better) and maximal accumulated cost (columns and 4, higher is worse)
all validation/test tasks during the meta-training (columns and and the meta-test (columns 3 and 4) in Humanoid (row 1), Hopper (row 2), Swimmer (row 3), Car-Cirele- Hazard (row 4), Point-Borte Botton (row 5). The accumulated reward and cost during meta-training are computed on the policy adapted one step from the meta-policy. The black dashed line is the constraint of the accumulated cost (below the line means satisfaction).
D.4 Experimental results the trade-off between optimality and constraint satisfaction
To investigate the influence of the allowable constraint violation constant & experiments, we conduct the experiments with & 0.0, 1.0, 2.0 and 3.0, two environments, including Half- cheetah and Car- Ci--Cicee-Haarrd The results are shown in Figure 6
As stated in Section 5.2, the theoretical result shows trade-off between the optimality and the safety constraint satisfaction when the allowable constraint violation thresholds 8e, vary. In particular, when 8c, is increased, the optimality is improved while the constraint violation increases. This statement is verified by Figure 6 Specifically, especially in Car-Cirole-Hazard, when the allowable constraint violation threshold d., varies from 0.0 to 3.0, the performance is improved but the constraint violation increased in both the meta-training and the meta-test. Therefore, as indicated in both theoretical results in Section 5.2 and the experimental results in Figure 6 choose large de, when the
19

---

Hogpar
Swmme
ot plan) INBaCTO
2.0
Heaoor
Autiredi (han)
Naraltee: comprnatinn
Nommalmee compunttion
1.0
Misu copeo
Miex paen
1450
Mria-tst
0.0
Motb-Faaiing
Math
Mete-niiiing
West-roll
CorCrcal
Poinn Button
Sels iutefi. (pec
Miaccho
MeaaCRFD
VAMk perily
Vita-rttro
Villa Ael
Mate DWiFTn
Modi--ant
Figure 5: Normalized computation time of the meta- mecta-raining and the meta-test in Humanoid, Hopper, Swimmer, Car-Circle-Hazaree and Poitt-Botton.
Sate meta RL (ours)
Meta CRO
Meta GRPO
MMM with perally
Harl Cheeeet
Half Auf-Chersnneeee-aae
.00
3,
Pu pobly adagaation staps
No odeyaation stips
Ca Mrr
Cer Hogerieee
Sooe +0
The
Namberr meeovinin teeiioos
Namtee mart-inins Freeioon
T5
Figure 6: Average accumulated reward (columns and 3, higher is better) and maximal accumulated cost (columns 2 and 4, higher worse) across validation/test tasks during the meta-training (columns and and the meta-test (columms 3 and 4) in Half-Cheetah (row 1) and Car-Circle-Hazze reward and cost during meta- eeta-training are computed on the policy adapted one step from the meta- meta-policy. The black dashed line is the constraint of the accumulated cost (below the line means satisfaction).
(row 2). The accumulated
constraint satisfaction is not required to be strict, and small 8c, > 0 when the constraint satisfaction is prioritized.
D.5 Selection of
For the hyper-parameter Àc; we X Ac; and them such that, the KL divergence of initial policy and the adapted policy n' solved from the safe policy adaptation problem (I) is close to 0.03. If the KL divergence too large, the objective and constraint functions of problem I are not good approximations to the accumulated reward/cost functions, as indicated by Lemma If the KL
divergence is too small, the policy adaptation step of problem ① is too small.
E
Algorithm Supplement
E.1 An optional algorithm for solving problem
Algorithm 2 states the algorithm for the safe policy adaptation. We apply the projected gradient descent (PGD) to solve the optimization problem (3) to obtain the Lagrangian multipliers { K then the closed-form solution of problem (I) is immediately obtained. The gradient of the objective function T(u) of problem (3) w.r.t u (used in line of of Algorithm 2 can be stated as
V V.,L(u) - E Ex-nTo [eav [A, (s, a)] + (1 7) DEL (T (-1s)))] + d',r. (7)
20

---

where x and d' are defined in Proposition and then the gradient step is projected to R The computation in 7 derived based on the dual method shown Proposition 6.1.1 in [8] which is simplified compared with direct computation by the chain rule. The derivation is shown in Appendix E.2.3 As the optimization problem (3) is the dual problem of (I) and is always convex, the PGD method Algorithm can guarantee convergence
the global optimum [2 Due to the low dimensionality of the decision variables of problem ( (the dimension of the Lagrangian multipliers
( is the constraint number p) and the simplicity of gradient computation, the computational
complexity of Algorithm 2 is much lower than directly solving problem
Algorithm 2 Safe policy adaptation algorithm
Require: Meta-policg Re; Advantage functions A," and A step size B.
I: u; = 0 for all E 1, .P
.P
2: for = 1, 1........................ .N do
3: Compute n7" (-1s) exp(fo(x.))
Ei m,de;) (177)))) - (s,
4: Ug ma u, - vv,, [(u)} for each i 1, where Lu) is shown in
5: end for
uG,.7 us all 1,... .p
7: x" (+1s) x exp(fo(s,)) (A -)) (A7 Ei (s.
8: Return (
E.2 An alternative algorithm implementation
When the proposed algorithms are applied to sigh-dimensional continuous state and action spaces, we provide Algorithms 3 and an alternative algorithm implementation of Algorithms [ and 2 Compared with Algorithms I and 2 Algorithms and avoid approximating A and A during the meta-training, since it is costly to approximate the value functions V and Va by neural networks and use GAE 45 to estimate the advantage functions A," * and A for each sampled task. Instead, Algorithms 3 and 4 only require to approximate Q ah and Q which can
be estimated by Monte-Carke Carlo sampling.
More specifically, line of Algorithm replace
x (-(ss) x cxp(ff((s)) +A+
Σ uide,) -1
Σ u (s,
in line of Algorithm by
*'|ss & ( (1-7) Cndo.)- - Vi " (s)
(8)
These two equations are equivalent, where the Q function replaces the A function. Similarly, line TO of Algorithm 3 is also equivalent to line of Algorithm
Line 11 in Algorithm ils equivalent to line of of Algorithm ! where the Q function also replaces the function. The left problem is how to solve the optimization problem and obtain the the Lagrangian multipliers w (TT..) only using the Q functions.
We show the solution next. The gradient of the objective function L(u) in problem (3) w.r.t u is
n; [(u)
-E Eg [Bawx* 8) - (s, as) (1 7),, DeL (T* (+++)) + d
as shown in Notice that the value of V., L(u) is the constraint function in the optimization problem
E S.w [aax a9) (8, a)] (1 A., DxL. (T((1s)]
+ d',,,
when ग = T Moreover, the constraint function in problem ① is already designed as replacement of Je,. (T)) + dzr + dc, and it is cheaper to compute than - -Je, (T)) + di,- de, for arbitrary ı in problem I However, in the problem of approximating V.., L(),. thanks to the derived closed-form
21

---

Algorithm 3 Safe policy adaptation algorithm with the first-order approximation
Require: Meta-policy Rei Advantage functions Q and Quiei step size 3.
I: u; = 0 for all E :P
2: 1,
N do
3: Compute x (-1s) exp(fo(8,.) (1+ DE (0. Eiimm((ir(s))))
ior 1,
5: Us +- max u, (.,L(u)) where (u, is shown in
6: end for
7: end for
8: "e,, u, for all 1,... P
9: x" (-Is) x exp(fals,) (A (1 - de) Mar (s,
Return -E=1.
Algorithm 4 An alternative algorithm of meta- meta-training
Require: Initial meta policy Tágg I: for 0...... N do
Sample task т with CMDP M, from the distribution P(T))
3: Evaluate Jeg.r (TT.). Q, (.,-) and Q (-,-) for the current veta--olicy Ta., task 7 Jezs (TT..) I de.r + dec, Vi I, ,P then Obtain the task task-specific policy x' and the Lagrangian multipliers uf;, by Algorithm with the meta-policy
5:
6:
Evaluate Q (:--) for the task-specific policy x² on task
7:
Compute the meta-gratn mota-gradient VxJ)(x²)) = --yEemey' _awar( Volen (s, a) (s, as)
8:
Take step of TRPO [4] with using VoJ.(TT) towards maximize J, (TT) obtain 0.+1
else
10:
Choose any in E {1, .P} such that Jein (To.) ) > din.t + dmin
11:
Compute the policy gradient VaJoi,, (TT..) x E swo aanan aamttn (11)) Volon (s, a)Q on (s,
12:
Take step of TRPO 44 with using VaJoi, x (To,)) towards minimize Jo rr (Tx) obtain 6m41
end if
14:
end for
T² as TT" shown in 8). using the original one -Je,-(TT)) + di,r + 8c, becomes cheaper. So, we directly use -Ja, + di + 8e.- Therefore, we have
V u. L(u) ~ (1-
+ di,- + 8c.).
(9)
Next, we use the first-order approximation to approximate - -Jo.r(TT)) ) + dir + do, Assume the policy n² is parameterized by To. then
V L(u)= -Je, di de,
(T)) Φ) + Je,, (TT)) + di,r dc,
E Eurnto ( V In (a)s) - o) (a.((TT) + dir Sa,
Then.
V., M(, -E - aam (+1s) (1s)) In (t.
( di,r 5 )
(10)
In this way, we replace all the estimations of the A function with the estimations of the Q functions, without the requirement extra data collection.
E.3 Action sampling in algorithm implementation
Algorithms
and we need to sample actions from
π (-18) x exp(/o(8,.) + (QT -
" "+Coir(s,)))
(11)
22

---

When the action space is discrete (no matter whether the state space is discrete continuous). it is trivial to do the sampling. When the action space high-dimensional and continuous, is not easy to do the sampling. Here, we show two solutions. In the implementation of Algorithms 2 and [ we apply the second solution.
E.3.1 The first solution
Similar to many widely used RL algorithm implementations, such as 44 we also consider the policy parameterized by a Gaussian distribution, i.e.,
exp (fo(s,a) = Ai exp (- [-(a-gal (s))))
(a
(a(als)
Jar exp (fo(s,a')) da'
28
where fo (a())) 253
and g is a neural network with the input s. So the policy is a softmax
policy.
For the policy in 11.. we have
(als) A2 exp (- (a - {))
17 9Q (x))2
28%
262
Here, (s, E Qa. (s, a) is approximated by C(s) are neural networks with the input Then,
(a- (+)) 26g
+ C(s) where gQ(s) and
(a
752
(s) +
63
92
(Q(ss) 2
+52
n53 +8
π T" (als) Ag exp
(12)
63,6
2 70 +
n +52+58
62
i.e. the (als) is Gaussian with the mean
90 + n5 55g
IQ (s()) and the standard deviation
62.8 480
is
This can be sampled by many code libraries directly.
We can also treat the approximate function (و=ي)(
as A (s, - L' (s, a) and used in Algorithms and 11. which take TT (-Is) x exp(ff(s,)
26
(s, Ei=r (8, )))
E.3.2 The second solution
In the second solution, we also consider the policy parameterized by a Gaussian distribution, i.c.,
exp (fo(s,a)) (a(a)s) Ja. exp (Fo(s,a')) da' = Ai exp
(a (s(s))
28
where fo {x())) and 9ø is neural network with the input s. 28g
We use the policy parameterized by 0 to approximate the policy π (-((ss) x exp(so(s,)) n -Q (s. Ei_u, (r, -)), by minimizing the expected KL-divergence, ie..
DEL no (-Is) exp(fo(s,)) ' (s,-) Eiau
min (0s =Bann Eo
8
Z (s)
As shown in 19 the problem is equivalent to ming 10s)
E ( In me (als) (f)(s. + (QQ7((s)) - -" u.
This optimization problem can be restated as
min sou (11a) 0
To (-Is)
[In (a|s)
(p u (0i
(-(s)
23

---

Therefore, we do not need more data to approximate the expectation E2, sNw" a Similarly, we
can also use to approximate π (-Is) x exp(fols..) (A (1 -2) Ei= udd,)) (A7 E
(A7 (s,
A4,7 Fe ())..
F Analysis and Proof
F1 Auxiliary results
Lemma 2 (Policy gradient
Let no be the parameterized policy with the parameter
holds that
VoJ-(TO)
E " (
Vo In (sa
1 y
1 EE anww am))())s)
[Vo In To (T as).
Lemma (Policy gradient of the softmax policy). For the softmax policy ao as To(a|s) exp(fe(c.a)) En'es explJo(s.a"]] (in discrete action space A) or TT(als) & (o(a_a))))) action space A), v(s,a) E x A. It holds that
Ja exp(fs(s.a))dan (in continuous
1
VOJ+(TT)) = - Eenuço a-mo((1)) (V) (s)
(13)
Proof. We prove under the discrete action space A. The proof under the continuous action space A similar.
From Lemma we have
V ( 1- E smmee awwro()ss) [V (
1
exp(fo(s. a))
E ana))(-1s)
Voln Ev'e exp(
A5 (s,
1
E To (1
Vofa(s, V In Σ exp(fo(s,a')) (s,
Na'EAA
Here, Vo In (ENEE
is independent with a, then VoJr(TO)
1
Vofa(s,a) Voonn Σ exp((o((, A7 (8.
1
E B (anr((-l)) (A
y
E RN Vo Σ exp(fo(s. A7 (3..0)
y
AoEd
Since Be A50 Beawoo( (0 (s, V. Then.
1 1-9
VOJr(TO)
E ( (2
F.2 Proofs of closed-form solution of safe policy adaptation
E.2.1 Proof of Proposition
We provide the complete statement of Proposition]] D as the following Proposition 5
24

---

Proposition 5. When the softmax policy T satisfies (TT) S + Vi be. P. the solution nO of the optimization problem (T) exists. Suppose an appropriate constraint qualification (to be stipulated) holds at n², there exists (W )i=z with Wersr 2 0, such that
Σ Wa - Aeir(s.)1) 4s E,
T- (-1s) & exp
Le.,
<a(s)) exp (o((s,a) (A+ Ei+E ug (Ar*(s, - -E= ug A (s,
Enedexp exp (fo(s, + n (A7 (s, - Di=1 ua, A (8, a'))
in discrete action space A. or
x x'(als)= exp (fols, + (A Ei=1 war de.) (A e (5, a) - Ei= U Acc
exp (So(s,a)) 1 (Ar (8, (,, Ei=1 w Aci (8, (8, a')) da
in continuous action space A. where (1 7) Ei= W Wes.
There are many constraint qualifications where each of them assures the validity of the proposition, including but not limited to Mangasarian-Froomvee constraint qualification (MFCQ), linear inde- pendence constraint qualification (LICQ), and Slater's condition (SC) [7]. Refer to for more validated constraint qualifications,
The assumption that one constraint qualification holds xt is mild. For example, if there exists policy π such that Vi
Aci, (8, 1-7
Japt (TT) +
E aruyy assi(-1s)
Xe; BNNg [DR (=((1))]1m < di, + teg
(14)
A (s,a)
then the Slater's condition holds. Note that when и TT. have Jear (TT)
E
1-7
Ae. an To [DKE (T(-ss)]](p((ss)) s di,r + de. It usually exists Th such that 14 holds or the Th itself can assure (14) holds. Next, we prove the proposition.
Proofs of Proposition
5
The optimization problem ① can be restated as
argmin #ETT
E
[Ar 1 (s. a)] + A s~. [De (m())))l=m(+sss),
smu.
(-1)(-10)
E 8rup- o (~xx(-1)
[A (s, + No E Eangoo
uo [DE (T(((s)) di 1,,..
where the constants X₂ - (1-7)Ae, and d' & (1 (1- (dd,r + 5 Je.,r (T²)).
First, we consider the discrete state-action space x A. Considering the probability at each state- action pair (als) as the decision variable, the minimization is taken over the probability simplex (T(+1s): : < T(ass) < 1, EnE. (als))
=1]. Then the optimization problem is formally stated as
argmin E sower
(alss) (s, ADxr (T(
LaEd
s.t. E S
Σ
a(als) (5, a) + (T( S d i = 1,...,P
aEA
(15)
Σ ralss)= =1 for any S,
aEA
<(als) < 1 for any A, A S.
({ass) < for any a E A, s E.
25

---

Since T E II we have d,,r = =1 (d,, + dc. - -e.rr (Tτ)) 2 0. the solution of 115 exists.
According Theorem in [77 and theorems in 88 1 since the constraint qualification holds, the Karush-Kuhn-Tuucer (KKT) conditions hold at T², i.e., there exists Lagrangian multipliers (
1p u (s) for all (s a) and uz(s, a) for all (s, S A, such that
- 20,i . P. ai 2 0, u3 (s. 2.,
V(s.a)
(16)
EE 5nupt
Σ LaEA
n" (als) (s, + DzL (- (x
-/ir 50, vi 1, p.
2 T( (s, 1, ES
(17)
Σ
('|as)) = 1, VoES,
(18)
aEA
B Bawer
Σ (als)) (8, + Xx, Dxr. (T (-16) ((()) de.r = 0,
LEEA
" (((s,a)(T = 0, ((s, a) ES A,
(19)
u5(s, ((s E ESxA A
(20)
VxL(TT, 1 up ui u²)= = 0,
(21)
where
MT, (W2, uhj ui, uz)) A E
Σ - ADRx (
LEEA
+
Σ Weyer
E
Σ
(xt( -
LEEd
+ Σ
(s)
(alss)
-1 + ΣΣ
ui ()(()))))))))))))))))))))))))))) 1) ()(ss))
#ES
AoEAA
aES #ES
(22)
Note that problem:
16
constitute the KKT condition for the following optimization
argmin E
Σ x(als) LaEA
-A77
Σ U Agt
Aai (0,00)
smp,
D
+ A
Σ No DEL (T(())))
Σ
(23)
s.t. Σ ([als) for any E,
<als)) for any A,
(als) < 0 for any E A,
i.e., the KKT condition for the optimization problem (23 holds at nt with Lagrangian multipliers u5(s), ui (s,a) a) and uz (s, a). Here, (uo are constants for the problem.
a)] and E "~we." [ENEA (als)) a)] are lin- ear; the term E BanyTo [DEl (T(+|s)] n ( (-1s))] is convex, the optimization problem (25) is convex. Moreover, since all the constraint functions are affine, the Slater's condition holds naturally for the optimization problem (23), as shown
Since the terms -E, s-ue" -Ba-uz (als)
99. Therefore, the strong duality holds. Then, n² the
optimal solution for 223
26

---

In 23 we can omit the term Di=1 Uz da and keep the solution unchanged. Next, we borrow the conclusion of Proposition 3.1 in 229 we have x² (a|s)
exp (fo(s,a) (A+ Ei= "
(s,a) Ei=1 Wh, (cir(,))s)
EuEr exp (fo(s, + E " Ner
(A7*(s, (s,a') - -EE Vant (cir(s,
i.e,
P
(-1s) x exp fo(s,-) (A+
Σ
u 2a, - Σ Eua
for all 8 E S. Since Xa = - >) the proof is done.
E2.2 Proof of Proposition
Proof of Proposition 2 For the Lagrangian multiplier variables u, un, M1, U2, denote the solution of min, [(T, u, uo u1 u2) n [u,,uovan,02) (L is shown in (2 i.e.,
n (unon = arg min L(T, u, uo, u1, u2).
From the proof of Proposition Then, we have {u", uū. ui. u
we have the strong duality for the optimization problem 15 holds.
arg max (n,No.Ms .22 L(T (2)) uo. u1, U2), s.t. 0, u1 2
0, U2 2
0.
(24)
Next, from the above optimization problem, we ug #1, Ug as ug (w), uf (u), 2zz (u)),, wi (u), (u) are the solution of dual variable (Lagrangian multiplier solution) of the following problem:
24) where
P
argmin "NN,
Σ <a|ss)
-177
Us Aar 1a: (8.0)
LoEA
A + (1-11)) -1-7 Σ uido:
DEL (T(1))))))))))
Σ undler
+
i=
im1
(25)
s.t. Σ x (als)) for any E,S,
<als) < for any a E A, S,
T(ass) < 0 for any A.
We have
" arg max (u
(( (u), (1 (w)) u, uu), (u), uz(u)), s.t. u 2 0.
(26)
Similar to solution of (33) we have the solution of (55 is π", where π (-lss x exp(fo(s.-) -
(Ei= A, (A, (s.
El A (s,-))) Moreover, from the strong duality of the opti-
mization problem (25 (linear inequality constraints), we have
n (M. = arg min L(TT, u, (u)) ui(w), uz(4)) = n.
(27)
Therefore,
arg max,
[(T", I
((() s.t. u uz
Moreover, we know
-als)))-1 Σ (s)()(s) Σ (als) -1)
ΣΣ ((()(s.))(=(<sa)) (((4)(s, 00
aES
27

---

Form 26 and 22
we have
g +
p
Σ us (s, + (A+ Σ vidors
max swer (arx"
IE aNDT To DEL (T"
Σ ui(1 (c(1~))(d17)
() + dcs Jear (To))
04 O, 1, P. Then, the proof is done.
F.2.3 Deviation gradient w.r.t the dual variables derive the gradient of L w.r.t. the dual variables a for 7 Let
T") & E [A7 (s,
U
a20
P
- +A+ (1-)) Σ unde;) E BNue (DEL (1 (1)) Σ und
where d', - (1 - (diir + de; - Je,,7 (TT)). Then,
v.I(u)= = VIL(u, + Vux" V2
Consider v2L(u, TT). From
we have
(fun)())) (w),, arg min L(T, u, ug(u), (u), vz(u)) n²
where L is shown 22 and un (u), uf (u). us (u) are the solution of dual variable of (25) Then
V L(T (u), of (w), uz(u)) 0.
Moreover, we know
Σ (s(())(s)
Σ (alss) -1) -1) ΣΣ ui(u)ss.)\==(sso>> 1) -3)
AaE
Thus,
T2L(u,x*) = LI , uz(u)) = 0.
Then, we have
V.I(u) (.I(u)=VZ((uxx*)
Therefore,
V v,, L(u) EE smue -B BaNry af [A (s, a)] + (1 - M),, DEL (11) + d
E3
Meta-Gradient
F33.1
Computation of meta-gradient
Proposition 6. Let = A A, A, T). Suppose all the assumptions in Proposition 5 hold. Sup- pose the LICQ and the strict complementary slackness condition (SCSC) T7 52 for the optimization problem 33.1 holds at n². Then, V (TT) exists and
VoJ+(TT)
1 1- E E aar Voalalll
+n)(TT) - -V (8. a) + Vole(s, O7 (s,a)),
where ((TT)) A x (1 (1-7) Σ. ue (To)des and O²* & Q²
28

---

Proof. For any meta-policy TΦ. the objective function of the optimization problem (3.1 is strongly concave and the constraint function is convex. The LICQ and the SCSC hold According to Theorem 2 in 52 (=r=(m") exists.
We have
(+1s) exp(fo(s.)
E U Achi
? i=1
is equivalent to
n' (-(s)) x (
' E 2 (arr(())))
From Lemma
we have
1
VØJ+(m")
7 B snupp a (v))) (T( ( (s, + fs(s, 07 (8,a))
1
y B Bonup" a (Von((() -07
+ Volo O7 (s.
F33.2 Computation of VoQi
We have
V VoQT Td (8,
7
E (araara [olo (8, Q² (s'
(28)
1 1-7
where the state- action visitation probability ((sa) initialized (s, A defined by
o (8,0)
(1-7)
Σ
P Ps = s',a To so ~ pr()),a)).
t=
Proof. As shown in
V&Qt (s, V ((1 7) -r (8, + (xNPP:(la)) [V." (s')) [V In (a'ls') ²= (s', a')
E
(nuu)
By Lemma from 13 we can obtain 288
F.3.3 Gradient of Lagrangian multipliers
We show the existence and the computation of V V (T²) in the following proposition.
Proposition 7, Let (Ts, T). Suppose all the assumptions in Proposition (5) hold. Sup- pose the LICQ and the strict complementary slackness condition (SCSC) [I7] [2] for the optimization problem 33. holds at n². Then, the Lagrangian multipliers W is unique for any given Ta. V aT) exists. For i [I....,P), if (TT) 0, then Vou T (To) Let Ve (T)) be
the vector includes all all i E (1.... .p] with uz (TT) > 0,
Vou (Tx) -V. b) L ())
where L(u, Φ) = E[AT" (x, Ei= u AA, (s, 17"h Eserro DDe (17(
Ei u,di, dc, Jenr (T+)).
Proof. For any meta-policy næ. the objective function of the optimization problem 33.1 is strongly concave and the constraint function is convex. The LICQ and the SCSC hold m² According to Theorem 2 in [52), the Lagrangian multipliers u (TT) is unique for any given T and Voua, (T)) exists. The computation is shown in [22 For all i E {1,-.. .P} with u (T²) = 0, we have Vou (Tx) = 0.
29

---

F.4
Optimality and constraint satisfaction analysis
E.11
Lemmas for optimality and safe analysis
Lemma 4. For any task T, and any policies x and n' E 11, the following bound holds:
1
E
J (,)) (C)
E
+ (29)
snupe a~d (+1s)
a (1ss)
where
C7(T') = 47 maxx,a (s)
DTV (alla') Bawup (Drv (T
Here, define Drv (1))
EEEd x'(ass)l and ())((=)))) &
max.es Drv (T([))( (-16))
The inequalities 29 also holds for each i = 1, :P. when A and AT are replaced by A and As" A. maxs,a A (5, a) is replaced by maxs,a (8..(8,0)) Jr is replaced by Jeg.-
Proof. The proof follows similar lines of Theorem 1 in [4] and Corollary 1 and 2 in [2]. For the sake of self-comtainedness, we provide the complete proof.
Let PT is matrix where P7 (i, = [. (s, a) and pr is a matrix where P (i,j)= (s. a). Let G = (1 +1+771 ypx + (T +...) (1 - -P and similarly C= (1+yps + (? +...)= (1 p1 ) Let p be a density vector on state space and r+ is a reward function vector on state space, thus r. p is a scalar meaning the expected reward under density Note that J, (T) =r, Gpr, and J, = r. Gpr. Here, Pr is the initial state distribution for task 7. Let A PF P7.
Follow the proof in Appendix B in 44 we have
G-1 - 71- = AA
Left multiply by G and right multiply by G.
G GAG GG
(30)
Left multiply by G and right multiply by G
G = GGA + GG.
(31)
Substituting the right-hand side in (30) into G in 31 then
G = G GGGGGGGaa- + GAGAG.
So we have
JT( - -T] =T, (, +
+
(32)
Note that G of where v the value function on the state space. We also have Gpy where vi is the state visitation distribution vector. So,
12,
72 of AGAV..
(x) G)p: =
AV.
Consider the first term
7 1イ T AV similar to Equation (50) in 44 we have
1ア (P=' - )リ アリ AVT
Σ 1 (s) Σ Σ nT' (als) =(als)) (s'ls,a) (s')
(33)
Σ (s) Σ (alss) - x(gls))
r(s)+ + E (5)) u(s)
Σ (s) Σ (al(s) - ((als))44(ssaa)
30

---

Since we have Σ. [((s))))47ss, = 0, we have
70 Av Σ (s) Σ [s).))
T
= E
(s,(s)))
Combine 32 and the above equation, have the following for the second term:
2
A = J- (+(
E g-OP a~z (-1s)
(A a))
Then need show
72 T
AGA
(T)()
1-7 U7
First,
22
AGAATT
Ur
nt (GAV.)
T
A)
+
+
72
17 A)
(GAuT)
5z
8/5
8/5"
By Hölder's inequality,
2
AGAUT
S -- 111 111
7
||
Similar to 33
each element in the vector yu
T A is E. (al(s) (alss) (8, a), then we have
ye
A
max
(
S 2max (7 DTV (F||=²).
SES
5,
From the Lemma 3 of we have
| S
Rxwey (Drv(
1 1-
Therefore, we have
2
o AGAV
S C (T')
1 ?
4y max (8, a)
DTV (T|| (T|[T')EEsns EEnrr [Prv((((ss)]
(1 7)2
Then the bounds hold.
Lemma 5 (Restatement of Lemma D. For any task and any policies a and n' E II with
max,es Drv [T| I a Eawuy [Drv (-1s) || (+(s)))) the following bound holds:
(A a)) 2ya.Ams
-
E
(1 (1- (1 Enwry [ ( (11s)((1)))))
a~x (-1s)
and
2yaAmaz
1 1
E a
(s,
(1 71- Enaer (TTss
(-1s)
These two inequalities also holds for each 1,... P, when A and A are replaced by A and Add Aaa is replaced by Amaz Jy is replaced by Jern.
31

---

Lemma 5 is variant of Theorem in and Corollary 1 and 2 in 22 The difference that, the inequalities in Lemma 5 replace the term max, D (a (-1))][ in Theorem
T(|
in 44 and replace the term E E [DkL(T [=<]] in Corollary 1 and in by Ex-uy (+
in the right-hand side of the inequalities.
Proof. We show the first inequality. The second inequality can be proven similarly, From Lemma 4
J= (=(
E
[A]
soNy amot (-1a)
4y maxx,a A7 DTV
[Drv (((ss)]
(1 (1-
We have DTv (T [1(²² a.lignu (=(ss(( Therefore, we have
4ya max,, (7 a) Bonuy ([)((((()s]]((((()))) E (s,
J+(T)-J, (T)
71
anen' (-le)
From Jensen's 's inequality, we have
Egnuy DDvv (=(())) s Eawny [Div (T(+1))11((+1s)]]
From the above inequalities, have
[A7 ()) 5 4y0rAmaa
J+(T') - (x
E sop
(1 -- Bawuy [ (34)
a>x'(-1s)
From
14), we have
( 1
DKI(w (T())
Therefore,
(A7 a)) + 2ycrA
J+(x') (m(m) <
E sov a~x (:15)
(1 (1- Eanny (<<((T(()ss]]
F.4.2 Proof of Propostion
Before we prove Proposition 4 we first show a lemma.
Lemma There exists a constant a such that, for any bounded Φ, maxses DTv(To||TT) S AymAmes
Sooe [Drv (T [|TT()))))] where nt = A, with X 2 1-7 and Ao. >
2:yo Alnai
(1-7)) for each
27a Proof. Consider x. = A A, with X > 2ax and Aoe [1-7}= for each i. From
Lemma I we have
π. (-ls) & exp(ff(s..) ( (1
U - (AA Ei (st (),
27aamma max
270 maar
When A
and Ae, 2 (1-7)=
1
1 1-7
1
(++ (1-11) 7)
-1 5
201y
Amaz + E5
A maz
1=1
32

---

Then.
(A + (1 --) Σ
P
Wen Aes) (Ar
Eiar A (s.
1-7
2017
Next, we denote A + (1-7) -i=1 ver de; as Anews denote (A7' 2 Aci (ne ) and denote Amax > Wa, Amar A Then, we have W Anow (8, 2xry
Consider a is sufficiently large, then
1-1 could be sufficiently small. From Bay
n' (-Is) exp(fo(s,)) + X (s,
we have
exp (fo(s, a) + A Anew (s,
exp (Sols, a) + Anewls,
T (-(s)) = Exed exp (fo(s, + Anew (8,
EOEA exp (fols,a')
(+)(s) me (s, (s, [T(-)s) X Anew(s)))
Therefore,
Drv(Ta((ls)]IT (-1s) =
Σ aEA
- [T(aass -J (a|s)] = new
Σ aEA
Annm (T)
For policy with bounded parameter o ma(als) is non-trivial larger than 0. Then EnEA () is non-trivial larger than if there exists a with (s, a)l > 0. Anew (s, a)l = for any E S and a E A, then a could any constant. If there is s E and A Anew (s,a)) 0, since Asewrs a) is continuous, there exists closed set S, such that EacA Aneuo To (a(ass) is non-trivial larger than O. Therefore, there exists a constant a such that
max Drv (To](=²) S aE s~ ze (To T² (+|s))]}. #ES
Next, we prove Proposition 4
Proof of Propostion From Lemma
1 and Lemma 6 we have
A A a)
2 maz
(x(x) S Jr (TT) + Eswuy (wa(1s)
(1s()1s)
1- ?
+
(1 1--7 7)2
E Banno (
27a4max man
Since Ao, 2 (1-7)3 we have
(TT)(
A (8, a)
27a maa
S Jei (To) +
E goae amx (-18)
+
E Bann DKL (TT (-1s)]
(1 71- 7)2
A (s,
S Jeur (To) +
E swa
de, Banz'o [DL (TT (-1s)]
1-7
(-1a)
amx
< di,r dc.
Also, we have
AF AF
27a max
(T) 2 (TT) + E Exwez B ((1)))
E Banuto [KI[[(([))][(((H)))
1 1-7
7)2
Since A > 1-y we have
maz
A t (s, a)
(T( 2 (TT) + E Exver ( (-))(s)s)
anieo zo [DKu(T((o)((a<(11))
1-7
33

---

For the solution n² of problem
we have
Ar a)
A
2+(7)
E soa, amit (-1s)
E
BNN DEL (TT (15)) (T()
A7 a (8, a)
max
E
an [Dx ( + (T(
felle swaty t (~<)(1)s)
7
A7 ( a)
X
E grulle a
E s DKL (Ta((ba)] + (T J (TT).
where 11 is the feasible set of problem
① The last inequality comes from T E 11.
E.4.3 Proof of Theorem 1
Recall the notations defined in Section 5.2 and used in this section: the optimal task-specific policy π² for task J as
argmax,en (+T s.t. (er, S diyi
the conservative task task-specific optimal policy =2.[-]- which is optimal for under conservative safety constraints, i.e.,
[e] argmax (x) s.t. Jeur (T) S der c,
where the conservative constant é 2 0; the task variance
Var(P(T)) - ming Er s Dxxn (1s)((((
the task variance under the conservative safety constraints
Var (P())) 2 ming E smujo (H)]
and its minimal point
abl e arg ming B.EEeTY [DII (ew.:
the radius of the task distribution P(T)
R(P(T)) A max,er.ce E "Nu," Tale [D (T (+1))) (+s))),
where the set defined by {e 2 0: T exists for all r E r).
We also define
Red (P(T)) A maxter Talel (12)111)))))) (+1a))
sowa,
We first show some lemmas for the proof Theorem I
Lemma For any 6, the policy T.. belongs the set fo E II (T))) =ale [KII(((H))]Tgg
A. (e.a)
21aAma max (1-7)) snyv
s dir e
as>x(-ls)
4rya Amzz
R(P(T)) for all i 1,....
Proof. From the second inequality in Lemma
Jecr (TT,te) 2
1
E aale a~xi
Vale A
(s,a)
27aamaas Amax
(1 11-7) 7)2
E
snu, smo,ate) Fate]
(1) (1s))
(11))
34

---

Since (1) S d2,t - e, we have
Jen +
E
all (s.
snuy, aldl
amoma (-1a)
2700 Amer
(1 -7)
18
aakl
( s d
Then,
Aesi (s, a)
+ 2 mman
Jeir (T +
E
1-8
(1 7)2
E
omer
sile
D (1
srug-
a (11)
470:mar
< di,, - 6 + (1 3 awe sid
470: mar
S di,7 6 +
R (IP(L))
(1 7)2
AyaAmar
S di,r c+
R(P(T)).
(1 7)2
Lemma 8. We have
The E E I (T) S de,r - 4ya.maa Amax
Ja.r
(1 7)= ((P(T)) fol 1, , and ErJ.
Proof. From the second inequality in Lemmal 1 Jes, (T3 2
27a max
(F + 1-7
smp, old a~x (-1)(
Jild Acur (s)
(1 1 7)2
E
snu,
Jate)
(x (11)))
Since (1),)) S di,r - E, we have
Jeur (T
E swy skl
A (s. (s,
a (e)())) (-18)
27yxA max
< di,r +
(1
7)2
E
aal
(+s)))
Also, from
33 and the proof of Lemmal I
we have
1
T as
1-7
alel
arot' (-1()
Σ ur
S(c)
Σ
(m (e][)) (als)) Fale) (s)
2 maz
Diaa (-1s) (1)
1 1-7
Amaz max
E
35

---

Then.
(TT(t)) + 1-7
A (s,
3le
aruge
amx (1)
20x Amax
$ d 6+
(1
7)² E
Gael
[ (1aa)) (11s))
4ya A maa
S dir +
(P(())
7)2
47a Aaax
S di ++
R(P(T)).
(1 7)
Here, we assume 7 2 0.5, which is commonly used.
Theorem 2. Suppose that Assumptions 1 holds. Let π- (3ld)
A, with À
2:yoA maz
1-7 Xa (1-7)²
27a. and 8c, (1-7)² R(P(T)) - c. We have
2
4yamaax
(n-Rr))(+(TT(e)) (T S
(1 (1- Var
Proof. From Lemma 7 we have that T..edl TT where B f E IT (T
E
swan awo (Dxu((((b)l)aaa((())) s di, + d., Amoly (s,a) +de
8ru, sle] am)((11))
Also, x (Blee) E I1 Therefore, from the definition of A in problem I we have
AA (s.a)) (s.a))-ADe -AD (T" (3)lo)), x&(i)
A - (s. (s. -DD (T) Tz()))
E sauy 3(c) awo (g (-1s)
E
s(1
snap (-1
where we use DKL (m (=15),=2(11)) represent Eg-w?? D (T (2(1))))()))))
From the second inequality Lemma T and the above inequality,
Jr (x (+(T)(1)))) - (T(()))) 2 1-7
E swa akl
A (0l (s.
DxL(TT (6lt)), mate))
1 1-
amx (c1)))()1
A Aalle (s, 1- (TI
dd(c]
amei" (115)
From the first inequality in Lemmal
1
( (s. + myx
Jr (TT (x(()) S
E ale)
(1 (1- DKL DKI(TT (T0))) T3m)).
amit
(-1
From the last two inequalities,
2 mar
A
( (T(
+ +
D ( (aa)
7--
i.e.,
270x AryxAmas maa
(x( (T A,),7)
+
DKL(TT Tga)).
1- (1 7)2
36

---

Then.
E (
J, (A A, A,)]
27aAmar max
BEWE(T) ( (a T3t)))
1-7
1
a Amaz
X
Var
7)2
Moreover, from Lemma 8
Tao II - E II: Je,, (T) S d,r + dc, for all i 1,.., , a r}.
From the definition of o¹, we have
E ( A,A, 2 max E
*ene
2 E ( A,,,),
Then, we have
E (1 ( A,
x Ey (F - Jr (A A, A))
2
X
+ 1 1-9
Var (P(T))
(1 21
47a max
Var (P()))
(1 7)-
Proof of Theorem 7
Theorem iss proven by combining Theorem
2 with Corrolary
G Discussion on the Tightness of the Derived Lower Bound in Theorem
Notice that the meta-safe RL aims to extract common knowledge from multiple existing RL tasks. The tasks in the task distribution are usually correlated, and their near-optimal policies usually produce similar actions scenarios with collision avoidance, although optimal policies under different environments should produce different actions when meeting different obstacles, the actions in free space should be similar. As Var(P((T)) is defined as ming E,-r(ry snee" (=]
a large part of the state space. For example, in the experiments of navigation
(=I((1s) (-1s) T (-Is))] which computes the expectation over the entire state space, it could be small, especially when the state space is large. Moreover, Var(PPT)) & ming E,,=(ry>E. ..,, - [ ( (- [=)((=]) is the minimal average distance between policies. According to our experiment, 0.05 is a sufficiently large number for the quantity for the KL divergence between two policies and the value of Var(P(T)) under this KL divergence is about 1 4 x 0.05.
Next, we roughly evaluate the quantity of the optimality bound 4-yam Ama (1-7)² Var (P(()) and show it can be relatively tight. We set y = 0.99, the reward/cost r e [o, 1] and c E [o, 1]. The max advantage function value maax < 1 is generally not very small. However, indicated in the proof of Lemma
F.4.1 we actually can replace max by max,, A,* (s, a), which is computed on the meta-policy and usually much smaller than
set it as 0.05. Then, the optimality bound of 4 is about 25. Note that, the quantity of the total reward/cost is about Σ 7" the bound is about 25% of the total reward/cost, which is relatively tight.
(1-7)² Var (IP(T))
100. Therefore,
standard in RL and safe RL derive lower bound with an order of quantity similar to the optimality bound 4 mas
(1-7)² Var (P(T)) in this paper. For example, in TRPO 44 and CPO [2) the A lower bounds also include the term [1-3]7 DKL- Moreover, for conciseness and broad applicability, we have to derive general optimality bound, which makes the bound looser. When the used condition and target problem are specified, tighter optimality bound can be derived. For example, when
37

---

large task distribution is specified, i.., Var(F(T)) is large, can also make some inequalities (such as those lines 1573 to 1585) tighter. However, when the target problem is required be specified, the generality of the optimality bound will be lost and it will become harder to understand and less intuitive.
H Limitations and Future Works
In this paper, consider the safety metric of CMDP, i.c., the expected accumulated costs satisfy the given safety threshold. This metric is generally less rigorous than the safe control research, where safety is defined as persistently satisfying certain state constraints. A future work is establishing safe meta-RL. algorithm with the rigorous safety metric. Another limitation that we assume the solution of problem (2) exists, i.e., there exists policy such that it is safe for all tasks as the initial policy for policy adaptation steps. A future work is to release this assumption and identify safe task-specific meta-policy for each given task.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction, including the main contribution statement and related works, accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitations in the Appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
39

---

Justification:
All proofs are provided in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all details of the information needed to reproduce the main experimental results in the experiment section and in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
40

---

Answer: [Yes]
Justification: We provide open access to the data and code with sufficient instructions in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp cc/public/guu httpggidddeee/eebbbnnnnneeee
ssiiisolccyyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all training details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide it in the section of the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
41

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Justification: We provide the information at the beginning of the Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: pttppp////aarrpo.ccc/////////c/a
Answer: [Yes]
Justification: the NeurlPS Code of Ethics is followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There is no potential societal consequence.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
42

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
43

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
44

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //errri////upp...// wwwptrripppccoooeeeeeeeeeeee /c//oonfreennes/22255 for what should or should be described.
45


Reviews:
[{"review_id": "1jaSo8gCyt", "reviewer": "Reviewer_NE4p", "summary": "The paper studies the safe meta-reinforcement learning (safe meta-RL) problem where anytime safety is ensured during the meta-test, which means that all policies output during the meta-test phase satisfy constraints. It proposes an algorithm similar to meta-CPO, but importantly the KL divergence constraint for policy updates has a reverse order, which leads to a closed-form policy update rule and can be efficiently implemented. As in meta-CPO and CPO, the proposed algorithm has monotonic policy improvement guarantee and anytime safe guarantee.", "questions": "It would be nice if the authors provide experimental results in highly stochastic environments. Currently all experiments seem to be done in deterministic environments.", "limitations": "The CMDP formulation used in the paper is pointed out to be insufficient for safety-critical problems, and there is some efforts to fix it, e.g., Anytime-Constrained Reinforcement Learning by Jeremy McMahan and Xiaojin Zhu (2023). It would be nice to mention this limitation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strength\n\nConsistent performance improvement over previous algorithms and particularly meta-CPO, which is closes to the proposed algorithm in my opinion. The writing is clear and easy to understand.\n\n# Weakness\n\nChanging the order of KL has a limited novelty. However, this change introduces computational advantages and no approximation error in contrast to meta-CPO, which is a highly related algorithm.\n\nTheoretical analysis do not consider sample error, which would be a serious issue in stochastic environments.\n\nThe proposed idea and the closed form of policy updates are related to MPO (Maximum a Posteriori Policy Optimization by Abdolmaleki et al. 2018). Also, some existing research suggest the benefit of using the KL order used in this paper, stating that it confers stability to noise and function approximation error. For example, Theoretical Analysis of Efficiency and Robustness of Softmax and Gap-Increasing Operators in Reinforcement Learning by Kozuno et al. (2019) and Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning by Vieillard et al. (2020). I recommend the authors to cite those papers.", "was_revised": true, "final_justification": "While the theoretical analysis and empirical results are solid, the core idea of using KL regularization in meta-RL is already proposed in meta-CPO. The algorithm employs a different order of KL than the one used in meta-CPO, and this order of KL has been already used in, for example, MPO. From these, I set my score to 5.", "rebuttal": "We are grateful and indebted to the time and effort invested in evaluating our manuscript, as well as to all the suggestions and reference recommendations that have made our manuscript a stronger and more valuable contribution.\n\n>**Weakness 1: Changing the order of KL has a limited novelty.**\n\n**Answer:**\nThe main contribution of the paper in terms of the algorithm design in (1) is not changing the order of KL.\nInstead, the algorithm design (i) guarantees monotonic improvement, (ii) ensures safety, and simultaneously (iii) provides a closed-form solution for a single safe policy adaptation step, which is applied to the meta-policy training and simplifies its computation by leveraging the closed-form expression of the adapted policy and developing a Hessian-free meta-training algorithm. \n\nSpecifically, the proposed algorithms offer three key advantages over existing safe meta-RL methods.\n(i) Superior optimality. Our safe meta-policy training algorithm in (2) maximizes the expected accumulated reward of the policies adapted from the meta-policy. In contrast, the meta-training of meta-CRPO learns the meta-policy by minimizing the distance between the meta-policy and the task-specific policy, which does not consider the optimality and the safety of the task-specific policies adapted from the meta-policy.\n(ii) Anytime safety guarantee during the meta-test. The safe meta-policy training produces a safe initial meta-policy by imposing the safety constraint. The safe policy adaptation imposes a constraint on the upper bound of the total cost, and thus is guaranteed to produce a safe policy for each iteration when the initial policy is safe. By integrating these two modules, anytime safety is achieved. In contrast, meta-CRPO employs CRPO, a primal-dual-based method, for policy adaptation, which does not have any safe guarantee in a single policy adaptation step. Thus, anytime safety cannot be guaranteed in meta-CRPO.\n(iii) High computational efficiency in both the meta-test and meta-training stages. In the meta-test, the derivation of the closed-form solution of Problem (1) makes it efficient to be solved. In contrast, the meta-CRPO and meta-CPO require to solve a constrained optimization problem, which is more computationally expensive than solving problem (1). In the meta-training, the closed-form solution of the policy adaptation (1) is used to derive a Hessian-free meta-gradient and reduces the computation complexity of the proposed algorithm to approach that in the single-level optimization.\nIn contrast, the meta-CPRO requires that the task-specific optimal policies have been learned, which is impractical when the number of training tasks is large.\nThe meta-CPO uses bi-level optimization to learn the meta-policy, which requires the computation of Hessian and Hessian inverse.\nIn Section 6, Figures 1, 2, 4 and 5, we conduct experiments on seven scenarios including navigation tasks with collision avoidance and locomotion tasks to verify these advantages of the proposed algorithms.\n\n**The novelties of the techniques.**\nIn order to derive the algorithm in (1), which guarantees monotonic improvement, (ii) ensures safety, and simultaneously (iii) enables a closed-form solution, we apply the following techniques.\nFirst, we derive the closed-form solution of the safe policy adaptation by applying the KKT analysis on the softmax policy. Second, we derive an upper bound of the constraint violation by analyzing the accumulated cost gap between different policies. The safe policy adaptation imposes a constraint on the upper bound of the total cost, and thus is guaranteed to produce a safe policy for each iteration by the majorization-minimization theorem. Third, to derive a lower bound of the expected accumulated reward of the adapted policies, we define the variance of the task distribution and then derive the lower bound by constructing a virtual sub-optimal meta-policy and quantifying the expected accumulated reward under the sub-optimal meta-policy.\n\n**The theoretical contribution and novelties of the proposed method.** The paper is the first to derive a comprehensive theoretical analysis regarding near optimality and anytime safety guarantees for safe meta-RL.\nFirst, we establish the theoretical basis of the algorithm design that guarantees anytime safety, i.e., zero constraint violation for any policy used for exploration. Second, we derive a lower bound of the expected accumulated reward of the adapted policies compared to that of the task-specific optimal policies, which shows the near optimality of the proposed safe meta-RL framework. Finally, we demonstrate a trade-off between the optimality bound and constraint violation when the allowable constraint violation varies.\nIn meta-CPRO, the optimality bound is provided, but anytime safety is not guaranteed.\nMeta-CPO provides neither an optimality bound nor anytime safety. \n\n>**Weakness 2: Theoretical analysis does not consider sample error, which would be a serious issue in stochastic environments.**\n\n**Answer:**\nTheoretically, we assume that the constrained policy optimization problem in (1) can be solved exactly, ensuring both monotonic policy improvement and zero constraint violations. This assumption is standard in the literature, such as CPO and TRPO, and also neglects sampling errors when establishing theoretical guarantees, as accounting for such errors would render the analysis intractable, especially the anytime safety property. Under this widely accepted assumption, the anytime safety property demonstrated in our paper, which represents one of the most stringent safety criteria, is theoretically achievable. \n\nIn the experiment, with a sampling size of 2000, the sampling error is sufficiently small to have a negligible impact on performance.\nIn our Swimmer experiment, we additionally add a large Gaussian noise to the state transition (up to Gaussian noise with variance 0.5), and therefore the environment is highly stochastic.\nAs shown in Figure 4, our experiments on the highly stochastic Swimmer environment demonstrate that the constraints are consistently satisfied, confirming the robustness of our approach in realistic scenarios.\n\n>**Weakness 3: The proposed idea and the closed form of policy updates are related to MPO (Abdolmaleki et al. 2018). Also, some existing research suggests the benefit of using the KL order used in this paper, stating that it confers stability to noise and function approximation error. For example, Theoretical Analysis of Kozuno et al. (2019) and Vieillard et al. (2020). I recommend the authors to cite those papers.**\n\n**Answer:**\nThanks for pointing out the connection with MPO and for suggesting these valuable references.\nMPO and [30] (Liu et al., 2019) have derived the closed-form solution for policy optimization.\nIn this paper, we extend the solution to the constrained policy optimization problem, and simultaneously guarantee the closed-form solution, policy improvement, and the strict constraint satisfaction. \nThe proposed algorithm is the first to provide both the safety guarantee and a closed-form solution.\n\nWe will cite the papers and add the following discussion to the related works section:\n\n“We note that the closed-form policy update considered in our work is closely related to MPO (Abdolmaleki et al., 2018) and its extensions (Liu et al., 2019), which derived similar formulations for unconstrained policy optimization. Furthermore, the KL order adopted in our framework has been shown to improve robustness against noise and function approximation errors (Kozuno et al., 2019; Vieillard et al., 2020). Building upon these insights, our work extends the closed-form approach to the constrained policy optimization setting, providing the first algorithm that simultaneously achieves strict constraint satisfaction, guaranteed policy improvement, and a closed-form solution, thereby offering both theoretical safety guarantees and practical efficiency.”\n\n>**Question: It would be nice if the authors provide experimental results in highly stochastic environments. Currently all experiments seem to be done in deterministic environments.**\n\n**Answer:** We appreciate the reviewer’s suggestion. \nIn our paper, the swimmer experiment adds a large Gaussian noise to the state transition (up to Gaussian noise with variance 0.5), and therefore the environment is highly stochastic. Please refer to Appendix D4.1 for more experiment setting details on the Swimmer experiment. As shown in Figure 4, the proposed method still works well and significantly outperforms meta-CPO and meta-CRPO in both meta-training and meta-test. \n\nIn other environments, the current MuJoCo environments also already include a degree of randomness, such as variations in initial states, actuation noise, and sensor noise, which introduce stochasticity into the dynamics.\n\n>**Limitations:** The CMDP formulation used in the paper is pointed out to be insufficient for safety-critical problems, and there is some efforts to fix it, e.g., Anytime-Constrained Reinforcement Learning by Jeremy McMahan and Xiaojin Zhu (2023). It would be nice to mention this limitation.\n\n**Answer:** Thanks for pointing out the safety-critical metric and the reference recommendations.  We will cite the paper and incorporate the following discussion into the section of related works:\n\n``While Constrained Markov Decision Processes (CMDPs) are a common framework for incorporating constraints into reinforcement learning, they can be insufficient for safety-critical applications where violations must be avoided at all times. Recent advances, such as Anytime-Constrained Reinforcement Learning (McMahan and Zhu, 2023), introduce stricter formulations that enforce constraints at every timestep rather than in expectation. These approaches provide stronger safety guarantees and highlight potential limitations of standard CMDPs, suggesting promising directions for future work.\""}, {"review_id": "rv8xXFC3EO", "reviewer": "Reviewer_LfmJ", "summary": "The paper studied safe meta-reinforcement learning, with the objective of ensuring anytime safety during the meta-test phase, such that safety constraints are satisfied at every step of the adaptation process. The paper proposes a safe meta-RL framework consisting of two modules: safe policy adaptation and safe policy training. The paper provided theoretical guarantees, demonstrating the possible trade-off between optimality and anytime safety. The experimental results justify the theoretical guarantee.", "questions": "Please see the weaknesses above. \n\nAdditional questions:\n1) The key function $f$ used in the softmax policy parameterization is not clearly defined or explained.", "limitations": "No", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths \n* The considered problem is meaningful, especially in safety-critical applications where violations cannot be tolerated even during the learning phase. The paper successfully proposes a framework that is both theoretically grounded and practically relevant. The experiments are promising to support the theory. \n\n\nWeaknesses\n* The algorithm and its the performance of  heavily depends on the choice of hyperparameters such as $\\lambda,$  $\\alpha$, and $\\delta$, which are difficult (even impossible) to determine in practice. \n\n* Theorem 1 is a bit confusing, as it establishes the trade-off between constraint violation and optimality. This seems to contradict the claim of zero violation under anytime safety. It would be better to discuss optimality in the case of anytime safety $\\delta=0.$ Theorem 1 seems only to provide the distance between the regularized objective and the epsilon-optimal policy and it is more convincing to provide the performance analysis w.r.t. the original objective. Theorem 1 is also not ideal as the optimality gap could be large. Besides, it would be helpful if the paper elaborated on how $R$ scales with the size and structure of the state and action spaces. Moreover, no sample complexity analysis is provided. The paper also did not provide the sample complexity analysis. \n\n* Though the safe policy adaptation step admits a closed-form solution, solving the dual problem for the Lagrange multipliers and computing the meta-gradient still involve non-trivial computations. The actual computational advantage over prior methods may not be as significant as claimed\n\n* The anytime safety seems straightforward by assuming initial policy is safety based on the formulation in (1). It is unclear how it can be adapted to the setting when the initial policy is also unsafe. \nThe formulation (1) and the algorithm CPRO seems both from the previous CMDP papers. The paper would benefit from better emphasizing its unique contributions beyond this integration.", "was_revised": true, "final_justification": "The response addressed my concerns, and I increased my ratings.", "rebuttal": "We are grateful and indebted to the time and effort invested in evaluating our manuscript and for all the suggestions that have made our manuscript a better and stronger contribution.\n\n>**Weakness 1: The algorithm and its performance heavily depend on the choice of hyperparameters such as $\\lambda$, $\\alpha$ and $\\delta$; these are difficult to determine in practice.**\n\n**Answer.** Our method only introduces two practical hyperparameters: $\\lambda$ and $\\delta$. The constant $\\alpha$ appears only in the theoretical analysis as a problem-dependent parameter and is not tuned in practice. \n\nFor the constraint violation threshold $\\delta$, we can simply set $\\delta = 0$ to guarantee anytime safety, which requires no tuning. \n\nThe hyperparameter $\\lambda$ controls the trust-region size. We select $\\lambda$ so that the KL divergence between the initial policy $\\pi$ and the adapted policy $\\pi'$ from problem (1) satisfies $\\mathrm{KL}(\\pi \\,\\|\\, \\pi') \\approx 0.03$. Practically, we adjust $\\lambda$ using a simple backtracking rule: if $\\mathrm{KL}(\\pi \\,\\|\\, \\pi') > 0.03$, we increase $\\lambda$ (e.g., multiply by 2); if $\\mathrm{KL}(\\pi \\,\\|\\, \\pi') < 0.03$, we decrease $\\lambda$ (e.g., divide by 2), until $\\mathrm{KL}(\\pi \\,\\|\\, \\pi') \\in [0.02, 0.05]$. This procedure is standard and efficient in the existing policy optimization algorithms, such as PPO and TRPO.\n\nNote that existing methods for meta-RL and safe meta-RL, such as MAML and meta-CPO, also require tuning a similar hyperparameter, i.e., the stepsize for task-specific policy adaptation, which is analogous to the tuning of $\\lambda$ in our method. Therefore, our approach does not introduce any additional tuning complexity compared to these existing methods.\n\n>**Weakness 2:**\n\n>**Weakness 2.1. Theorem 1 is a bit confusing, as it establishes the trade-off between constraint violation and optimality. This seems to contradict the claim of zero violation under anytime safety. It would be better to discuss optimality in the case of anytime safety $\\delta=0$.**\n\n**Answer 2.1:**\nTheorem 1 characterizes the optimality gap under a general constraint violation threshold $\\delta \\geq 0$. For the special case $\\delta = 0$, the theorem still provides an optimality analysis while guaranteeing zero constraint violations, consistent with the anytime safety property. \nThe theorem simply indicates that allowing a nonzero $\\delta$ can lead to a smaller optimality gap. \nIt does not imply that near-optimality cannot be guaranteed when $\\delta = 0$. \n\n>**Weakness 2.2. Theorem 1 seems only to provide the distance between the regularized objective and the epsilon-optimal policy and it is more convincing to provide the performance analysis w.r.t. the original objective.**\n\n**Answer 2.2:**\nTheorem 1 directly provides the distance between the original objective of the safe policy optimization, i.e., the expected accumulated reward $\\mathbb{E}\\_{\\tau \\sim \\mathbb{P}(\\Gamma)}[J\\_\\tau(\\mathcal{A}^s (\\pi\\_{\\phi^\\*}, \\Lambda, \\Delta, \\tau))]$ of the adapted policy $\\mathcal{A}^s (\\pi_{\\phi^*}, \\Lambda, \\Delta, \\tau))$, and the epsilon-optimal policy. \n\n>**Weakness 2.3. Theorem 1 is also not ideal as the optimality gap could be large.**\n\n**Answer 2.3:**\nIn the manuscript, we have a section (Appendix G) about the discussion on the tightness of the derived lower bound in Theorem 1. In summary, as $\\mathcal{V}ar(\\mathbb{P}(\\Gamma))$ measures the minimal average KL divergence between optimal policies and a meta-policy, it can be small, especially in large state spaces; for example, we observe $\\mathcal{V}ar(\\mathbb{P}(\\Gamma)) \\approx \\frac{1}{4} \\times 0.05$ in our experiments. With $\\gamma=0.99$ and a reduced $A^{\\max}=0.05$, the bound $\\frac{4\\gamma \\alpha A^{\\max}}{(1-\\gamma)^2} \\mathcal{V}ar^\\epsilon (\\mathbb{P}(\\Gamma))$ is around $25$, which is about 25% of the total reward/cost (about 100), indicating reasonable tightness. It is standard in RL and safe RL to derive a lower bound with an order of quantity similar to the optimality bound $\\frac{4\\gamma \\alpha A^{max}}{(1-\\gamma)^2} \\mathcal{V}ar^\\epsilon ( \\mathbb{P}(\\Gamma))$ in this paper. For example, in TRPO and CPO, the lower bounds also include the term $\\frac{A^{max}}{(1-\\gamma)^2}D_{KL}$. Moreover, for conciseness and broad applicability, we have to derive a general optimality bound, which makes the bound looser. When the used condition and target problem are specified, a tighter optimality bound can be derived. For example, when a large task distribution is specified, i.e., $\\mathcal{V}ar ( \\mathbb{P}(\\Gamma))$ is large, we can also make some inequalities (such as those in lines 917 to 923) tighter. However, when the target problem is required to be specified, the generality of the optimality bound will be lost and it will become harder to understand and less intuitive.\n\n>**Weakness 2.4. Besides, it would be helpful if the paper elaborated on how $R$ scales with the size and structure of the state and action spaces.**\n\n**Answer 2.4:** The optimality bound is independent of the dimensions of the state and action spaces. Instead, the term $\\mathcal{V}ar(\\mathbb{P}(\\Gamma))$ reflects the variability of the task-specific optimal policy parameters within the task distribution. \n\n>**Weakness 2.5. Moreover, no sample complexity analysis is provided.**\n\n**Answer 2.5:** In each meta-training iteration, we sample 10 tasks from the task distribution. In each task, we sample 40 trajectories, and the horizon of each sampled trajectory is 200. Therefore, for each meta-training iteration, the number of sampled state-action pairs is 80k. We train the model from 300 to 500 iterations in the meta-training. \n\n>**Weakness 3:\nThough the safe policy adaptation step admits a closed-form solution, solving the dual problem for the Lagrange multipliers and computing the meta-gradient still involves non-trivial computations. The actual computational advantage over prior methods may not be as significant as claimed.**\n\n**Answer:** The computational complexity of the proposed method, including solving the dual problem for the Lagrange multipliers and computing the meta-gradient, has significantly reduced the computational complexity in baseline methods in both the meta-test and meta-training stages. \n\nIn the meta-test, the derivation of the closed-form solution of Problem (1) makes it efficient to be solved through the dual problem. The dual problem in (3) is always convex and low-dimensional. The dimension of the problem (3) is the number of constraints $c_i$. It is extremely efficient in solving the problem, such as $0.1$ seconds. \nIn contrast, the meta-CRPO and meta-CPO require solving a constrained optimization problem, which is more computationally expensive than solving problem (1). \n\nIn the meta-training, the closed-form solution of the policy adaptation (1) is used to derive a Hessian-free meta-gradient and reduce the computation complexity of the proposed algorithm to that in the single-level optimization. The computational complexity is on a scale of simple policy gradients.\nIn contrast, the meta-CPRO requires that the task-specific optimal policies have been learned, which is impractical when the number of training tasks is large.\nThe meta-CPO uses bi-level optimization to learn the meta-policy, which requires the computation of the Hessian and the Hessian inverse. The Hessian inverse for the large neural network is significantly computationally complex.\n\nIn Section 6, we conduct experiments on seven scenarios, including navigation tasks with collision avoidance and locomotion tasks, to verify the computational efficiency of the proposed algorithms. Figures 1 and 5 show that our algorithm is much more efficient than the baselines, saving about 70% of the computation time for meta-training and 50% for meta-testing compared to meta-CPO. \n\n\n>**Weakness 4: The anytime safety seems straightforward by assuming the initial policy is safety based on the formulation in (1). It is unclear how it can be adapted to the setting when the initial policy is also unsafe.**\n\n**Answer:** In the proposed safe meta-RL algorithm, we aim to learn a meta-policy from the training tasks, such that the policy adaptation from the meta-policy to the new task can achieve anytime safety.\nTo achieve anytime safety, the meta-policy, which is the starting policy for the meta-test, is designed to be safe by being learned from the optimization problem (2). \nIn Section 3.2, the optimization problem (2) is solved for the meta-training stage. The constraints in the optimization problem of (2) ensure that the solved meta-policy is safe, i.e., the starting policy for the meta-test is safe. To solve the optimization problem (2), in Algorithm 2, we prioritize the minimization of the constraint violation of the meta-policy (shown in lines 10-12 in Algorithm 2). Therefore, if there exists a safe policy for all tasks in the task distribution, the meta policy solved by (2) is always safe. As shown in the last column of Figures 1 and 4, the cost of the initial policy is always below the constraint threshold for all scenarios.\n\nMoreover, it is non-trivial even if the initial policy is safe. To ensure anytime safety, the safe policy adaptation in (1) should be safe for any safe initial policy. To achieve this, we first derive Lemma 1 to upper bound the accumulated cost of the optimized policy by the initial policy, and then design the constraint set of (1) correspondingly. Meanwhile, the closed-form solution of (1) is also required for the safe meta-RL algorithm. The proposed (1) is the first algorithm that simultaneously offers the two key properties."}, {"review_id": "8txV6WW3UE", "reviewer": "Reviewer_9YLt", "summary": "In this paper, the authors develop a safe meta-RL framework consisting of two modules: safe policy adaptation and safe meta-policy training. They introduce a novel safe policy adaptation method, which guarantees monotonic improvement, ensures safety, and provides a closed-form solution for a single safe policy adaptation step. For the meta-policy training, the authors impose safety constraints on the meta-policy by leveraging the closed-form expression of the adapted policy and developing a Hessian-free meta-training algorithm. The authors validate their safe meta-RL algorithm on Mujoco safe tasks and collision avoidance tasks.", "questions": "1. I have one clarification question for the authors about the one-step policy adaptation module. If the state action data points need to be collected to evaluate $A_\\tau^{\\pi_\\phi}$, there could be constraint violations in task $\\tau$ while collecting those state action pairs. Then, how is zero constraint violation guaranteed? \n\n2. In Theorem 1, eq.(6), the upper bound is finite, and will only be zero when $\\delta_{c_i}$ is zero. How realistic is this assumption, as zero constraint violation depends on this.\n\n3. Is there a reason that the regret guarantee do not depend on the number of tasks in Theorem 1. I was expecting the total number of tasks to appear in the analysis and have a positive effect on the regret guarantee.\n\n4. How is the proposed method different from the zero constraint violation methods in the following papers? I think it would be good to add a discussion to compare these methods.\n[1] Liu, Tao, et al. \"Learning policies with zero or bounded constraint violation for constrained mdps.\" Advances in Neural Information Processing Systems 34 (2021): 17183-17193.\n[2] Liu, Tao, et al. \"Learning policies with zero or bounded constraint violation for constrained mdps.\" Advances in Neural Information Processing Systems 34 (2021): 17183-17193.\n[3] Xu, Siyuan, and Minghui Zhu. \"Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-Task Optimum Comparator.\" Neural Information Processing Systems. 2024.", "limitations": "1. I think ensuring zero-constraint violation only happens under some strict assumptions. The authors should clarify on that.\n2. Dependence on the number of tasks for the regret guarantees is not shown.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The method is able to achieve zero-constraint violation for various safe RL tasks\n2. The theoretical analysis seems well developed.\n\nWeaknesses:\n1. It is not very clear how the zero-constraint violation is achieved if there will be some exploration done by the policy adaptation module.\n2. The dependence of regret guarantess for the meta-safe RL algorithm are not shown with respect to the number of tasks as shown in the Meta-CRPO paper.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful and indebted to the time and effort invested in evaluating our manuscript, as well as to all the suggestions and reference recommendations that have made our manuscript a stronger and more valuable contribution.\n\n>**Weakness 1: It is not very clear how the zero-constraint violation is achieved if there will be some exploration done by the policy adaptation module.**\n\n>**Question 1: I have one clarification question for the authors about the one-step policy adaptation module. If the state action data points need to be collected to evaluate $A_\\tau^{\\pi_\\phi}$, there could be constraint violations in the task \n$\\tau$ while collecting those state action pairs. Then, how is zero constraint violation guaranteed?**\n\n**Answer:** In the CMDP setting of this paper, the constraint satisfaction is defined as that the expected accumulated cost is below a threshold, i.e., $J\\_{c_i,\\tau}(\\pi) = \\mathbb{E}\\_t[ \\sum_t c_i] \\leq d_{i,\\tau}$.\nThe advantage function $A_\\tau^{\\pi_\\phi}$ can be evaluated based on sampling a lot of trajectories by the policy $\\pi_\\phi$, but the policy $\\pi_\\phi$ still satisfies the constraint and is defined as safe as long as the expected accumulated cost, i.e., the average cost over all sampled trajectories, is below a threshold. \n\nIn some papers [48] [56] [37], the safety is defined by that the agent cannot visit some states. In this case, the advantage function $A_\\tau^{\\pi_\\phi}$ is hard to evaluate under the safety definition in these papers.\n\n\n>**Weakness 2: The dependence of regret guarantees for the meta-safe RL algorithm is not shown with respect to the number of tasks as shown in the Meta-CRPO paper.**\n\n>**Question 3: Is there a reason that the regret guarantee do not depend on the number of tasks in Theorem 1. I was expecting the total number of tasks to appear in the analysis and have a positive effect on the regret guarantee.**\n\n>**Limitation 2: Dependence on the number of tasks for the regret guarantees is not shown.**\n\n**Answer:** The theoretical analysis in this paper does not derive the regret analysis for online meta-policy training, but derives the optimality analysis for the meta-test, i.e., the optimality analysis and the anytime guarantee during the safe policy adaptation.\n\nOur work considers the safe meta-RL training with a form of the offline optimization problem, where the meta-policy is learned from the sampled tasks from a fixed task distribution, and focuses on the optimality analysis of the safety policy adaptation from the learned meta-policy. \nWe assume we can obtain the solution of the meta-training, no matter how many tasks are used during the meta-training.\nSince the optimization problem is offline, the theoretical analysis does not include the regret in an online optimization problem.\n\nIn contrast, Meta-CRPO addresses online meta-RL, where tasks are revealed sequentially. So they provide the regret bound that depends on the number of revealed tasks.\n\n>**Question 2: In Theorem 1, eq.(6), the upper bound is finite, and will only be zero when $\\delta_{c_i}$ is zero. How realistic is this assumption, as zero constraint violation depends on this.**\n\n**Answer:**\nWe do not directly assume zero constraint violation. \nInstead, based on Corollary 1, the proposed method is designed to and has been shown to achieve the zero constraint violation while achieving good optimality. Theorem 1 provides the optimality analysis under different tolerance settings of the constraint violation, which includes the special case that the zero constraint violation is strictly required.\n\nSpecifically, the parameter $\\delta_{c_i}$ in Theorem is a user-defined tolerance hyper-parameter that controls the maximum allowed violation of constraint $c_i$. \nOur optimality in (6) holds for any $\\delta_{c_i} \\geq 0$. \nWhen strict constraint satisfaction is required, we can set $\\delta_{c_i} = 0$, and the optimality analysis naturally reduces to the zero-violation case. \nOtherwise, for a practical tolerance $\\delta_{c_i} > 0$, Theorem 1 quantifies the trade-off between optimality and the small, controlled violation bounded by $\\delta_{c_i}$. Therefore, the statement in Theorem 1 includes the optimality analysis that encompasses both scenarios where the constraint satisfaction is strict and the more common settings. \n\n>**Question 4: How is the proposed method different from the zero constraint violation methods in the following papers? I think it would be good to add a discussion to compare these methods. [1] Liu, Tao, et al. \"Learning policies with zero or bounded constraint violation for constrained mdps.\" Advances in Neural Information Processing Systems 34 (2021): 17183-17193. [2] Xu, Siyuan, and Minghui Zhu. \"Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-Task Optimum Comparator.\" Neural Information Processing Systems. 2024.**\n\n**Answer:** Thanks for the reference recommendations. \nPaper [1] considers an online policy optimization problem and derives the sub-linear regret of the constraint violation.\nThe paper is not a safe meta-RL setting and does not have a meta-training process. In the setting of paper [1], it is theoretically unachievable to achieve the anytime safety, which is defined as the constraint violation being smaller than $0$ for any turn. Therefore, although paper [1] achieves a very strong constraint satisfaction metric, it is different from the anytime safety in our paper.\nPaper [2] considers an unconstrained policy optimization problem, and no constraint satisfaction guarantee is provided. \nIn our paper, we achieve anytime safety in the setting of safe meta-RL by combining the proposed safe meta-training and the safe policy adaptation.\n\nWe will cite these papers and include the above discussion in our modified version.\n\n>**Limitation 1: I think ensuring zero-constraint violation only happens under some strict assumptions. The authors should clarify on that.**\n\n**Answer:** \nThe zero constraint violation is developed based on (i) our CMDP constraint setting (lines 91-96) and (ii) Assumption 1 (line 274). The assumption for the theoretical analysis only includes Assumption 1, which is not overly strict.\n\n(i) As we mentioned in the answer to weakness 1, this paper considers the constraint violation on the CMDP setting, i.e., the constraint satisfaction is defined as that the expected accumulated cost is below a threshold, i.e., $J\\_{c_i,\\tau}(\\pi) = \\mathbb{E}\\_t[ \\sum_t c_i] \\leq d_{i,\\tau}$. In this setting, the zero constraint violation is achieved in an expected metric in the proposed method.\n\n(ii) Assumption 1 requires the existence of at least one policy that satisfies the constraints across all tasks. This assumption only requires the task distribution that not all feasible policies are mutually exclusive with respect to constraint satisfaction. This assumption is not overly conservative and is actually a minimal requirement if one wants to achieve anytime safety. If the assumption is violated, for any initial policy, anytime safety would be violated at the beginning of policy adaptation."}, {"review_id": "2n0LJH9KGq", "reviewer": "Reviewer_5CEM", "summary": "This paper presents a safe Meta RL algorithm. In particular, the paper considers a meta-training stage where the idea is to find a policy that will be safe across the tasks, where the policy will maximize the reward across the tasks drawn from a probability distribution. The policy is then adapted to the new task. The policy is also guaranteed to be safe by considering an advantage-based policy update. The paper validates the theory across diverse simulation experiments.", "questions": "Please see weakness", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The meta safe policy is very important. The paper thus considers an important problem.\n\n2. The proposed algorithm has a theoretical basis, and the paper shows theoretical justifications as well. \n\n3. The empirical results show that the proposed approach has practical significance.\n\nWeaknesses:\n\n1. (2) says that the policy has to satisfy the constraints of all the tasks, which seems to be very conservative. In particular, consider the grid-world scenario where some locations have obstacles that one wants to avoid that locations; now consider the other task where the obstacles are placed at the complementary set of locations, then there will be no policy. Hence, it is not clear what type of policies one can expect to get solving (2).\n\n2. In order to solve (1), one again needs to ensure that $\\pi_{\\phi}$ is safe. Without knowing the task,  ensuring that $\\pi_{\\phi}$ is not clear. In particular, if the Meta test task is different from the Meta-training task, it is not clear whether solving (2) will ensure that $\\pi_{\\phi}$ will be safe for new task.\n\n3. Convergence rate results have not been provided.", "was_revised": true, "final_justification": "This paper considers an important and interesting problem in meta CMDP.  The paper's idea is sound, and it has interesting simulation results. However, one criticism is that in Theorem 1, it does not specify $\\epsilon$ that can balance between the sub-optimality gap and the constraint satisfaction. I mean, if $\\epsilon$ is high, then one can show that the policy is feasible; however, the sub-optimality gap is constant, and vice versa. Hence, I am fully into accepting this paper. Some sort of trade-off result would have been good. The authors have also accepted the above.", "rebuttal": "We are grateful and indebted to the time and effort invested in evaluating our manuscript, as well as to all the suggestions and reference recommendations that have made our manuscript a stronger and more valuable contribution.\n\n>**Weakness 1: (2) says that the policy has to satisfy the constraints of all the tasks, which seems to be very conservative. In particular, consider the grid-world scenario where some locations have obstacles that one wants to avoid that locations; now consider the other task where the obstacles are placed at the complementary set of locations, then there will be no policy. Hence, it is not clear what type of policies one can expect to get solving (2).**\n\n**Answer:** Thanks for the question about Assumption 1 and providing the example. Here, we claim that, although Assumption 1 may be violated, it is not overly conservative and is realistic for the safe meta-RL setting.\n\nAssumption 1 requires the existence of at least one policy that satisfies the constraints across all tasks. This assumption only requires the task distribution that not all feasible policies are mutually exclusive with respect to constraint satisfaction. This assumption is not overly conservative and is actually a minimal requirement if one wants to achieve anytime safety. If the assumption is violated, for any initial policy, anytime safety would be violated at the beginning of policy adaptation. \n\nIn the CMDP setting of this paper, the constraint satisfaction is defined as that the expected accumulated cost is below a threshold, i.e., $J\\_{c_i,\\tau}(\\pi) \\leq d\\_{i,\\tau}$. In the provided example, when setting the constraint violation threshold $d_{i,\\tau}>0$, it is possible that there exists a policy $\\pi$ such that $J_{c_i,\\tau}(\\pi) \\leq d_{i,\\tau}$ for both the environments with complementary cost sets. In this case, Assumption 1 is satisfied. By solving (2), an initial meat-policy $\\pi$ with  $J_{c_i,\\tau}(\\pi) \\leq d_{i,\\tau}$ can be obtained.\n\nIf we simultaneously consider (i) the strict case that the constraint satisfaction requires $J_{c_i,\\tau}(\\pi) \\leq 0$, i.e., there are strictly no costs, and (ii) given two tasks with complementary sets of obstacles, Assumption 1 is violated and the optimization problem does not have a solution. \nIn this case, anytime safety is theoretically unachievable for any algorithm. \nOn the other hand, the safe meta-RL is to extract the useful information (meta-policy) from correlated task samples. If the distances between the tasks are too far, such as when the constraint set is nearly complementary, then the learned meta-policy is not useful for the meta-test. So the above extreme setting is less meaningful in the safe meta-RL setting, and then Assumption 1 should not be viewed as overly conservative, even though the strict case described above does not satisfy it.\n\n\n>**Weakness 2: In order to solve (1), one again needs to ensure that $\\pi_\\phi$ is safe. Without knowing the task, ensuring that $\\pi_\\phi$ is not clear. In particular, if the Meta test task is different from the Meta-training task, it is not clear whether solving (2) will ensure that $\\pi_\\phi$ will be safe for the new task.**\n\n**Answer:** Thanks for the question. The question is intuitive for our anytime safety result. \n\nOur meta-training process is explicitly designed to extract constraint information from a diverse set of correlated training tasks. Although the test task may be new, it often shares structural similarities with the training tasks—especially in terms of the constraints. Since the training tasks collectively cover the relevant constraint space, the meta-policy learns to satisfy a combination of the constraints seen during training. This means that even if the test task is new, its constraints are still within the span of the training constraints, enabling the meta-policy to maintain safety. \n\nFrom a theoretical analysis point of view, we assume the training task and the test task follow the same distribution. Therefore, when the safe behavior is learned from the meta-policy, the initial policy will satisfy the constraints.\n\nIn practice, when the set of sampled training tasks is sufficient, the empirical constraint violation during meta‑test adaptation is almost zero.\nAs shown in Figures 1 and 4, anytime safety is achieved during the meta-test after the meta-policy is learned from the training tasks.\n\nIf we assume the test tasks are shifted from the distribution, due to the initial policy violating the safety constraints, anytime safety cannot, in general, be guaranteed without additional mechanisms. Characterizing constraint violations under such shifts and designing new adaptation algorithms that retain safety is an important direction for future work.\n\n>**Weakness 3: Convergence rate results have not been provided.**\n\n**Answer:** The convergence rate of stochastic constrained optimization has been extensively studied in general settings, such as in [a] and [b]. As these results already provide well-established convergence guarantees for a broad class of constrained optimization problems, we do not repeat such an analysis here. Instead, our focus is on the unique challenges of safe meta-RL, where we derive a dedicated theoretical framework that emphasizes optimality bounds and anytime safety guarantees.\nIt is the first to derive a comprehensive theoretical analysis regarding near optimality and anytime safety guarantees for safe meta-RL. \nSpecifically, first, we establish the theoretical basis of the algorithm design that guarantees anytime safety, i.e., zero constraint violation for any policy during the policy adaptation. Second, we derive a lower bound of the expected accumulated reward of the adapted policies compared to that of the task-specific optimal policies, which shows the near optimality of the proposed safe meta-RL framework. \nFinally, we demonstrate a trade-off between the optimality bound and constraint violation when the allowable constraint violation varies, which enables the algorithm to be adjusted to prioritize either safety or optimality.\n\nTable 1 compares both the theoretical and experimental results between this paper and previous works\n[24, 12]. First, we study anytime safety and provide a zero constraint violation guarantee. In previous\nworks, they only provided positive upper bounds for the constraint violation, and the upper bounds\nonly work for the final policy [24 ] or the adapted policies [ 12 ]. Second, although [24 ] provides an\nupper bound of the optimality gap, the experimental optimality is the worst. On the other hand, [12]\ndoes not provide an optimality bound. In contrast, our method exhibits high optimality and provides\na near-optimality guarantee, outperforming existing approaches in terms of both experimental and\ntheoretical outcomes. Third, our method is more efficient than the existing methods [24, 12].\n\n[a] Dentcheva et al., Optimization with Stochastic Dominance Constraints, 2003.\n\n[b] Geiersbach et al., Sample-Based Consistency in Infinite-Dimensional Conic-Constrained Stochastic Optimization, 2025."}]

Author rebuttal:

Meta-review: The authors' rebuttal has led to a significant improvement of the manuscript.
After extensive discussions, all reviewers agree that the paper can be accepted.
Decision: Accept (poster)