Title: Efficient Safe Meta-Reinforcement Learning: Provable Near-Optimality and Anytime Safety
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning
Abstract: This paper studies the problem of safe meta-reinforcement learning (safe meta-RL), where an agent efficiently adapts to unseen tasks while satisfying safety constraints at all times during adaptation. We propose a framework consisting of two complementary modules: safe policy adaptation and safe meta-policy training. The first module introduces a novel one-step safe policy adaptation method that admits a closed-form solution, ensuring monotonic improvement, constraint satisfaction at every step, and high computational efficiency. The second module develops a Hessian-free meta-training algorithm that incorporates safety constraints on the meta-policy and leverages the analytical form of the adapted policy to enable scalable optimization. Together, these modules yield three key advantages over existing safe meta-RL methods: (i) superior optimality, (ii) anytime safety guarantee, and (iii) high computational efficiency. 
Beyond existing safe meta-RL analyses, we prove the anytime safety guarantee of policy adaptation and provide a lower bound of the expected total reward of the adapted policies compared with the optimal policies, which shows that the adapted policies are nearly optimal. Empirically, our algorithm achieves superior optimality, strict safety compliance, and substantial computational gains—up to 70\% faster training and 50\% faster testing—across diverse locomotion and navigation benchmarks.


Peer reviews:
comments: SUMMARY:
This paper presents a safe Meta RL algorithm. In particular, the paper considers a meta-training stage where the idea is to find a policy that will be safe across the tasks, where the policy will maximize the reward across the tasks drawn from a probability distribution. The policy is then adapted to the new task. The policy is also guaranteed to be safe by considering an advantage-based policy update. The paper validates the theory across diverse simulation experiments.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The meta safe policy is very important. The paper thus considers an important problem.

2. The proposed algorithm has a theoretical basis, and the paper shows theoretical justifications as well. 

3. The empirical results show that the proposed approach has practical significance.

Weaknesses:

1. (2) says that the policy has to satisfy the constraints of all the tasks, which seems to be very conservative. In particular, consider the grid-world scenario where some locations have obstacles that one wants to avoid that locations; now consider the other task where the obstacles are placed at the complementary set of locations, then there will be no policy. Hence, it is not clear what type of policies one can expect to get solving (2).

2. In order to solve (1), one again needs to ensure that $\pi_{\phi}$ is safe. Without knowing the task,  ensuring that $\pi_{\phi}$ is not clear. In particular, if the Meta test task is different from the Meta-training task, it is not clear whether solving (2) will ensure that $\pi_{\phi}$ will be safe for new task.

3. Convergence rate results have not been provided.

QUESTIONS:
Please see weakness

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors develop a safe meta-RL framework consisting of two modules: safe policy adaptation and safe meta-policy training. They introduce a novel safe policy adaptation method, which guarantees monotonic improvement, ensures safety, and provides a closed-form solution for a single safe policy adaptation step. For the meta-policy training, the authors impose safety constraints on the meta-policy by leveraging the closed-form expression of the adapted policy and developing a Hessian-free meta-training algorithm. The authors validate their safe meta-RL algorithm on Mujoco safe tasks and collision avoidance tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method is able to achieve zero-constraint violation for various safe RL tasks\n2. The theoretical analysis seems well developed.\n\nWeaknesses:\n1. It is not very clear how the zero-constraint violation is achieved if there will be some exploration done by the policy adaptation module.\n2. The dependence of regret guarantess for the meta-safe RL algorithm are not shown with respect to the number of tasks as shown in the Meta-CRPO paper.\n\nQUESTIONS:\n1. I have one clarification question for the authors about the one-step policy adaptation module. If the state action data points need to be collected to evaluate $A_\\tau^{\\pi_\\phi}$, there could be constraint violations in task $\\tau$ while collecting those state action pairs. Then, how is zero constraint violation guaranteed? \n\n2. In Theorem 1, eq.(6), the upper bound is finite, and will only be zero when $\\delta_{c_i}$ is zero. How realistic is this assumption, as zero constraint violation depends on this.\n\n3. Is there a reason that the regret guarantee do not depend on the number of tasks in Theorem 1. I was expecting the total number of tasks to appear in the analysis and have a positive effect on the regret guarantee.\n\n4. How is the proposed method different from the zero constraint violation methods in the following papers? I think it would be good to add a discussion to compare these methods.\n[1] Liu, Tao, et al. \"Learning policies with zero or bounded constraint violation for constrained mdps.\" Advances in Neural Information Processing Systems 34 (2021): 17183-17193.\n[2] Liu, Tao, et al. \"Learning policies with zero or bounded constraint violation for constrained mdps.\" Advances in Neural Information Processing Systems 34 (2021): 17183-17193.\n[3] Xu, Siyuan, and Minghui Zhu. \"Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-Task Optimum Comparator.\" Neural Information Processing Systems. 2024.\n\nLIMITATIONS:\n1. I think ensuring zero-constraint violation only happens under some strict assumptions. The authors should clarify on that.\n2. Dependence on the number of tasks for the regret guarantees is not shown.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied safe meta-reinforcement learning, with the objective of ensuring anytime safety during the meta-test phase, such that safety constraints are satisfied at every step of the adaptation process. The paper proposes a safe meta-RL framework consisting of two modules: safe policy adaptation and safe policy training. The paper provided theoretical guarantees, demonstrating the possible trade-off between optimality and anytime safety. The experimental results justify the theoretical guarantee.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n* The considered problem is meaningful, especially in safety-critical applications where violations cannot be tolerated even during the learning phase. The paper successfully proposes a framework that is both theoretically grounded and practically relevant. The experiments are promising to support the theory. \n\n\nWeaknesses\n* The algorithm and its the performance of  heavily depends on the choice of hyperparameters such as $\\lambda,$  $\\alpha$, and $\\delta$, which are difficult (even impossible) to determine in practice. \n\n* Theorem 1 is a bit confusing, as it establishes the trade-off between constraint violation and optimality. This seems to contradict the claim of zero violation under anytime safety. It would be better to discuss optimality in the case of anytime safety $\\delta=0.$ Theorem 1 seems only to provide the distance between the regularized objective and the epsilon-optimal policy and it is more convincing to provide the performance analysis w.r.t. the original objective. Theorem 1 is also not ideal as the optimality gap could be large. Besides, it would be helpful if the paper elaborated on how $R$ scales with the size and structure of the state and action spaces. Moreover, no sample complexity analysis is provided. The paper also did not provide the sample complexity analysis. \n\n* Though the safe policy adaptation step admits a closed-form solution, solving the dual problem for the Lagrange multipliers and computing the meta-gradient still involve non-trivial computations. The actual computational advantage over prior methods may not be as significant as claimed\n\n* The anytime safety seems straightforward by assuming initial policy is safety based on the formulation in (1). It is unclear how it can be adapted to the setting when the initial policy is also unsafe. \nThe formulation (1) and the algorithm CPRO seems both from the previous CMDP papers. The paper would benefit from better emphasizing its unique contributions beyond this integration.\n\nQUESTIONS:\nPlease see the weaknesses above. \n\nAdditional questions:\n1) The key function $f$ used in the softmax policy parameterization is not clearly defined or explained.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the safe meta-reinforcement learning (safe meta-RL) problem where anytime safety is ensured during the meta-test, which means that all policies output during the meta-test phase satisfy constraints. It proposes an algorithm similar to meta-CPO, but importantly the KL divergence constraint for policy updates has a reverse order, which leads to a closed-form policy update rule and can be efficiently implemented. As in meta-CPO and CPO, the proposed algorithm has monotonic policy improvement guarantee and anytime safe guarantee.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\nConsistent performance improvement over previous algorithms and particularly meta-CPO, which is closes to the proposed algorithm in my opinion. The writing is clear and easy to understand.\n\n# Weakness\n\nChanging the order of KL has a limited novelty. However, this change introduces computational advantages and no approximation error in contrast to meta-CPO, which is a highly related algorithm.\n\nTheoretical analysis do not consider sample error, which would be a serious issue in stochastic environments.\n\nThe proposed idea and the closed form of policy updates are related to MPO (Maximum a Posteriori Policy Optimization by Abdolmaleki et al. 2018). Also, some existing research suggest the benefit of using the KL order used in this paper, stating that it confers stability to noise and function approximation error. For example, Theoretical Analysis of Efficiency and Robustness of Softmax and Gap-Increasing Operators in Reinforcement Learning by Kozuno et al. (2019) and Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning by Vieillard et al. (2020). I recommend the authors to cite those papers.\n\nQUESTIONS:\nIt would be nice if the authors provide experimental results in highly stochastic environments. Currently all experiments seem to be done in deterministic environments.\n\nLIMITATIONS:\nThe CMDP formulation used in the paper is pointed out to be insufficient for safety-critical problems, and there is some efforts to fix it, e.g., Anytime-Constrained Reinforcement Learning by Jeremy McMahan and Xiaojin Zhu (2023). It would be nice to mention this limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}