Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The proposed method concerns safe reinforcement learning as a constrained MDP problem. In particular, the authors consider safe meta RL, a method to deal with dynamically shifting environments. Technically, the method is realized using a so-called ‘CMDP-within-online’ framework, where a constrained learning task is performed ‘within’, and the resulting policies are input for the meta learner. Theoretical regret bounds and estimation errors (regarding sub-optimality) are given, and the expected theoretical learning rates are derived. 

strength_and_weaknesses: ++Important problem, clearly fitting into ICLR
++Strong technical contribution

—Experimental evaluation is very sparse. 

clarity,_quality,_novelty_and_reproducibility: I think this is a very good paper, rather clear and of high quality. I vote for acceptance, but would encourage the authors to expand on the experimental evaluation, for instance adding one environment, and making the distributional shift in environments clearer. I list a few questions to the authors below.

Perhaps relevant to add to the literature is the recent paper listed below that deals with robust MDP learning against shifting environments

Suilen et al. Robust anytime learning of Markov Decision Processes. NeurIPS 2022.


Questions

— how is it justified that the ‘meta-initialization’ policy is ‘fair’ that is, has full support over the state/action space? I will necessarily not be safe, if I’m not mistaken. Please discuss this requirement.

— with long sequences of learning tasks, how does the approach perform in sparse reward settings which usually require a lot of data?

— should the safety constraints also be adapted with shifting environments, or what is the motivation to have them static?

— how do the individual tasks compare to a safe policy improvement problem (Thomas et al), where one aims to (safely, or better, reliably) find a policy that outperforms the behavior policy?

summary_of_the_review: Strong paper, experimental evaluation should slightly be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the meta-learning problem regarding various safe RL tasks that are modeled by constrained Markov decision processes (CMDPs).  The authors proposed a CMDP embedded online learning framework to generalize learnt safe policies to other unseen CMDP environments. In theory, the authors proved regret bounds in terms of within-task optimality gap and constraint violations in both static and dynamic regret metrics. Finally, some comparison experiments are provided to show the effectiveness of the proposed method.  
strength_and_weaknesses: Strengths:

- The studied meta learning problem is motivated by applications of learning a safe policy. This is a useful generalization of meta-learning to  constrained Markov decision processes, which allows to adapt the learnt policy to new constraints quickly. 

- The propsoed CMDP embedded online learning framework is practical in sense that it only relies on the inexact solution to sub-CMDP  tasks and data trajectories from previous tasks. 

- When the task similarity is bounded, the authors provide theoretical guarantees on both task-within optimality gap and constraint violation in static regret bound, under some assumptions on policy initializations. For practical use, the authors generalize this result to a dynamic regret bound, which is more useful since it does not assume boundedness of the task-similarity. 

Weaknesses:

- The provided CMDP embedded online learning framework builds on a simplified version of the existing CRPO algorithm. Although this simplification permits convenience in analysis, it brings some restriction to the applicability of the proposed algorithm. 

- In the meta-learning problem with multiple tasks, the static regret seems to be not very useful since the bounded ask similarity is assumed and environments associated with different tasks are different. Otherwise, this becomes a single task problem which doesn't have to necessitate meta-learning.

- The authors mentioned the hardness of bounding regrets in meta-learning by citing Kwon et al. (2021). This seems to be the key challenge to establish the theoretical guarantees. Although a series of assumptions are made, it is not very clear if they are necessary to address this challenge. In addition, the challenge from constraint seems to be unrevealed yet.

- The proposed CMDP embedded framework relies on DualDICE to estimate state-action distributions. However, DualDICE is an existing offline RL algorithm for the standard MDPs. It is more natural to think of some off-line CMDP algorithms that respect the constraints more effectively. 

clarity,_quality,_novelty_and_reproducibility: Overall, the paper has a clear delivery except for some ambiguity. Since the paper builds on several existing algorithms, mixing notation from different sources make it hard to follow the main results. More efforts need to be made to explain the proposed algorithm well. Some assumptions or techniques are introduced in the middle of the paper, while it is less discussed how do they address the key challenge. In addition, many references are mixed in the content, which is a distraction to readers.   

The paper presents a new meta learning framework for safe RL, which is useful to generalize across different safe learning tasks. To achieve this, the proposed method combines several existing techniques in a clever way. However, the analysis and assumption mostly build on existing results, the novelty in which requires justification. The experiment utilizes a simple RL environment which might be too simple for meta-learning to generalize.  

Here are some other questions for consideration: 

- It is useful to summarize an abstract meta algorithm at the beginning instead of delaying Algorithm 1 to the end of the paper.

- Whey we can't use the original CRPO results? Assuming exact action-value function makes the proposed algorithm and theory impractical. 

- How does Assumption 1 address the hardness results from Kwon et al. (2021)? How are they related to your constrained learning problems?

- Assumption 2 need to be clarified: 'o-minimal structure'.

- Please check your citation formats. They are not standard. 

- It is useful to verify the effectiveness of the proposed method in other environments, e.g., humanoid? 
summary_of_the_review: All claims of the paper are supported by proofs or references. Most of them are well-supported, but some cited results are vague or modified in an ambiguous way. The proposed meta learning method combines existing RL algorithms for safe RL and theory builds on existing results and assumptions. The experiments are based on some basic RL environments. 


============================

POST-REBUTTAL. Thank you for your response. Since most of my concerns are addressed, the score has been updated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. It obtains task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the task-averaged optimality gap and constraint satisfaction improve with task-similarity in a static environment, or task-relatedness in a dynamic environment.

Furthermore, it enables the learning rates to be adapted for every task and extends our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of the approach.

strength_and_weaknesses: Strength: Propose the a novel CMDP-within-online framework where the within-task is CMDP and the meta learner aims to learn the meta-initialization and learning rate. It shows that the task-averaged regrets for optimality gap (TAOG) and constraint violations (TACV) diminish with respect to both the number of steps for the within-task algorithm M and the number of tasks T.  It adapts the learning rates for each task
to a dynamic environment.

Weakness: 

1. the notation $\nu^*_t$ in page 2 is unclear. What is $\pi^*$? It makes harder to understand this paper.

2. The primal approach explained in Section 2.1 is confusing. Why the sub-optimality gap in (2) holds by running CRPO? What is CRPO? Is CRPO an online or offline algorithm? There is no explanation about CRPO and no comments about the "carefully chosen parameters". It is not helpful for people who is not familiar with CRPO and makes the paper not self-contained. Those observations make the paper hard to read.

3. Lemma 1 relies on $D^*$ is known and Theorem 3.1 relies on approximation error of DualDIce. The whole algorithm/analysis seems to be a composition of existing analysis (DualDice+CRPO), which might lack generality. It is hard to understand what is the key contribution of this paper.

4. Theorem 3.3 replies on the assumptions of two regret upper bound $U^{init}_T$ and $U^{sim}_T$. There is no explanation regrading why those assumptions are reasonable. With those assumptions, the hardness of analysis is largely diminished.

5. In the experiment, only frozen lake and acrobot are considered. How about the your algorithm for Mujoco environments?
clarity,_quality,_novelty_and_reproducibility: The writing of the paper is somewhat unclear. 
summary_of_the_review: Please address the questions I mentioned above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces the CMDP-within-online framework for meta-safe RL.

Strengths:
* highly innovative work
* safe RL is very important
* The proposed CMDP embedded online learning framework is practical in the sense that it only relies on the inexact solution of sub-CMDP tasks and data trajectories from previous tasks.

Weaknesses:
* Sparse experiments (although a few environments were added in the rebuttal)
* Theoretical analysis is not always clear
justification_for_why_not_higher_score: * Sparse experiments (although a few environments were added in the rebuttal)
* Theoretical analysis is not always clear
justification_for_why_not_lower_score: * highly innovative work
* safe RL is very important
* The proposed CMDP embedded online learning framework is practical in the sense that it only relies on the inexact solution of sub-CMDP tasks and data trajectories from previous tasks.
Decision: Accept: notable-top-25%