Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the safe reinforcement learning problem where safety constraint violations are bounded. Therefore, the agent's goal is to manage exploration and safety maintenance efficiently.

To achieve that, the authors propose a cautious RL scheme that uses Dirichlet-Categorical models of transition probabilities. They also empirically validate the different performance aspects of the proposed algorithm.
strength_and_weaknesses: **Strengths of paper:**
1. The problem of safety constraint violations in RL is important and has many real-world applications.

2. The authors empirically validated the performance of the proposed algorithm.

**Weakness of paper:**
1. The assumption of knowing which states are safe and which are unsafe (even locally) is very strong. 

2. The motivational example is unsuitable for the considered problem setup in the paper as the state space can be very large or even continuous for the robotic problem.

3. The proposed approach will become computationally inefficient (or even infeasible) for a large number of states or actions (continuous state and action space).

4. It is unclear why authors do not use tighter concentration results for variance like Bernstein inequality.

5. No theoretical guarantee: No theoretical guarantee of how the proposed method will work compared to the optimal policy. 


**Question and other comments.** 

Please address the above weaknesses. 

I have a few more questions/comments:
1. What is the initial belief used for transition probabilities?

2. Page 3, the line before Section 3: Change $0$ in $\alpha_a^{i0}$ to something else as it gives the impression that transition is happening from state $i$ to state $0$.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The paper is well organized, but the presentation has minor details that could be improved, as discussed earlier in Strength And Weaknesses.

**Quality:** 
The experimental evaluation is the bare minimum to support the main claims.

**Novelty:** 
The main ideas of the paper have limited novelty.

**Reproducibility:** 
The code is unavailable, which makes it difficult to reproduce the empirical results.
summary_of_the_review: This paper significantly overlaps with my current work, and I am very knowledgeable about most of the topics covered by the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a risk-aware bayesian reinforcement learning method to tackle the problem of safe exploration. Specifically, the authors assume that the agent maintains a Dirichlet-Categorical model of the MDP, and propose a method to derive an approximate bound on the confidence that the risk is below a certain level. Experiments in tabular environments demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:

1. The problem of safe exploration is important. 

2. The proposed method is theoretically sound. The authors leverage a method from [1] to approximate the expected value and variance of the agent’s belief about the risk, and prove its convergence. 

Weaknesses:

1. The novelty of the proposed method is unclear to me. Bayesian RL and risk-aware safe exploration have been widely studied in previous work. The authors may want to provide a detailed discussion on the novelty of the proposed method.

2. The experiments are insufficient. First, some important baselines are missing. The authors may want to compare to more baselines. Second, it would be more convincing if the authors could evaluate their proposed method on continuous control tasks, such as Mujoco [1].  

[1] Todorov et al. "Mujoco: A physics engine for model-based control." international conference on intelligent robots and systems. IEEE, 2012.
clarity,_quality,_novelty_and_reproducibility: Clarity and Reproducibility: This paper is well-organized and easy to follow.

 Quality and Novelty: The novelty of the proposed method is unclear to me. Moreover, the experiments are insufficient.
summary_of_the_review: The proposed method is theoretically sound. However, the novelty of the proposed method is unclear to me, and the experiments are insufficient.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors investigates a well founded probabilistic approach to safe RL, with primary contributions pertaining to safe training. Transition between states in the MDP is (during learning) modelled explicitly using a Dirichlet-Categorical model. Given a prior on transition probabilities (which can be fully informative) then the RL agent updates the belief of transition probabilities based on observed transition using Bayesian inference. The authors further derive suitable approximations of the risk associated with selecting action a in state q, based on the probability of transition and the cost of bad states. With the reasonable assumption that the agent can observe unsafe states m steps away (as transitions in the MDP), the approach is evaluated in slippery Gridworld and slippery PacMan showing impressive results. The paper ends with a well suited discussion on what the consequences for a learning agent when (proper) probabilistic safety guarantees are imposed, e.g. when the prior of the MDP transitions is weak. 
strength_and_weaknesses: Strength:
* A proper probabilistic treatment on prior, safety and risk for RL given a Markov Decision Process.
* Tractable and useful approximations of expected risk (similar to traditional expected reward). Furthermore, a clear probabilistic safety requirement is used to bound the risk (using Cantelli Inequality), with a high interpretability.
* An excellent analysis and discussion on the results, and what safe RL entails.
* Good exposition of related works.
* The approach can be added on top of other RL methods (with discrete state action spaces?).

Weaknesses:
* No comparison to any other approach to safe RL. (However, given how formally well founded the presented approach is, this is more a valuable addition than a necessity.)
* Figure 2 (which I think is great and really important) can be improved:
  * I suggest that you draw red and yellow boundaries around where respective areas are in (b)-(d).
  * Make sure that all four figures have the same xtick and font size.
  * When referring to the figure, (y,x) is used which was confusing at first. E.g. the stand-still state in (b) is referred to as (13,1). I suppose that what is used is (row, column), but for figures/images (x,y) is standard.
  * For clarity, I suggest that you invert the yticks, without changing the figure, such that the starting location is (0,0). A $(-\hat{y},\hat{x})$ coordinate system for figures is only standard within UX/GUI/Graphics. It is better to use a $(\hat{x},\hat{y})$ coordinate system.
  * I understand that it does not fit, but I would have liked to have this figure in the main paper.
* The slippery PacMan experiment could be extended (different $P_\text{max}$, two ghosts, a little bit larger).
clarity,_quality,_novelty_and_reproducibility: High clarity and quality. High novelty, as far as I can tell, but I might have missed some parts of the safe RL literature.
Most details is in the paper to reproduce, I think. But I strongly encourage the authors to release source code upon acceptance.
summary_of_the_review: The presented approach is (as far as I know) well suited in the literature, well founded, well presented, a highly suitable approach to safe RL (for training and otherwise) (in the applicable context of the approach) and sufficient empirical evaluation with interesting results which also motivate the method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work introduces Risk-aware Cautious RL (RCRL) algorithm that allows agents to be trained and keep them safe. By keeping a Dirichlet-Categorical model for each environment state, PCRL approximates the risk associated with the agent’s behavior originating from local action selection. To evaluate the performance of the PCRL algorithm authors have conducted experiments at a small grid-world environment ($20 \times 20$) and a simplified version of PacMan (one ghost, 2 foods). 

strength_and_weaknesses: **Strength**

- The problem of keeping agents safe during the training is of high interest and challenge. RCRL is able to estimate the risk associated with the agent’s behavior.
- A theoretical analysis is also provided.

**Weaknesses**
- The idea of keeping a separate  Dirichlet-Categorical model for each environment state is quite limited. The scalability of the RCLR algorithm is under question.
- Despite its merits the novelty of the RCLR algorithm is limited.
- Some parts of this work are hard to be followed by the reader, especially due to the notation.
- Empirical analysis is limited as experiments have been conducted in a small grid-world environment and on an over-simplified version of PacMan. 
clarity,_quality,_novelty_and_reproducibility: The first point that should be addressed by the authors is the scalability of the proposed RCLR algorithm. Is it feasible the application of the RCLR algorithm to environments with large discrete state spaces or even continuous environments? Also, the complexity of the RCLR algorithm should be presented, and its connection to the hyper-parameter $m$ (back-propagation steps). 

The main novelty of the proposed work should be also highlighted. Is the main contribution the keeping of a separate  Dirichlet-Categorical model for each environment state or the estimation of the risk associated with the agent’s behavior? In an abstract point, the way under which RCLR estimates the risk associated with the agent’s behavior can be seen as an MCTS. 

As regards the clarity of the paper, some parts should be revised. Especially, the notation makes it hard for the reader to understand easily the idea behind the RCLR algorithm.

The empirical analysis is another limitation of this work. Specifically, it would be really interesting the examination of the RCLR algorithm on more challenging high-dimensional tasks. Also, comparisons should be conducted with more baselines apart from the vanilla Q-learning algorithm. The standard deviation should be also provided in Table 1. Finally, it is not clear the number of runs at PacMan (I guess that the presented results are coming by only one run).
summary_of_the_review: As aforementioned, the paper presented some interesting ideas but it has many weaknesses in its current version.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies safe reinforcement learning (RL) where safety violation must be bounded during training. They propose a new approach that balances the trade-off between efficient progress in exploration and safety guarantee. Specifically, the proposed approach updates Dirichlet-Categorical models of the state transition probabilities that describes the agent's behavior within the environment via Bayesian inference. They then approximate the agent's belief in terms of risks. They provide theoretical guarantees on the convergence on as well as empirically showing the performance of the proposed approach.
strength_and_weaknesses: ### Strength
**Problem Settings.** The problem settings are important and interesting.

**Theoretical Analysis** I think Theorem 3.1 is indeed a nice theorem though it is not clear to me how novel this theorem is.

### Weakness
**Paper writing.** First of all, the presentation of this paper should be improved in terms of notations. For example, $P$ or $Q$ are used in two meanings
+ P: Transition probability (in Section 2) and lowest risk level (in Section 3.3)
+ Q: finite set of states (in Section 2) and Q-function (in Section 3.4).

Such inconsistency makes this paper quite hard to follow. Also, the notations are far from standard as an RL paper, which gives the reader unnecessary and unessential burden. As an instance, I personally want the authors to avoid using $Q$ in other meanings than Q-function. State space is typically denoted as $S$ or $X$ in RL papers.

**Related work.** It would be better to write related work section in more organized manner. The current related work section is just a list of relevant papers, and it is hard to follow the story of the related work.

**Missing citations.** As a particularly relevant papers, I would like to list the following paper. I think it may be better to compare the authors' method with this existing work (the source code is released).
- As, Yarden, et al. "Constrained Policy Optimization via Bayesian World Models." International Conference on Learning Representations. 2021.

**Experiments.**  I don't think the experiment has been fully conducted. The benchmark task is very simple (i.e., grid world and PacMan), and the baseline methods are vanilla Q-learning. There are a lot of existing safe RL algorithms or benchmark tasks, so I think the authors should have compared with recent ones. For example, representatives of the benchmark tasks for safe RL is Safety-Gym or MuJoCo, and the notable safe RL baselines are PPO-Lagrangian (Here are just examples). At least, I do not think that the authors' claims are supported by the current experiments.

### Minor comments
- Since Moldovan and Abbeel (2012) is on safe exploration without ergodicity assumption, position of its citation seems weird to me. It would be more reasonable to cite it in the end of the next sentence.
clarity,_quality,_novelty_and_reproducibility: **Clarity.** This paper is hard to follow due to inconsistency of mathematical notations. Also, related work section should be rewritten in an organized manner.

**Quality.** The quality of the theoretical analysis seems ok to me (though I am not fully confident), but that of experiments should be improved in terms of the selection of both benchmark problems and baselines.

**Novelty.** As far as I know, the proposed method is new. However, compared to recent work, I am not convinced whether the proposed method is better than existing ones. I don't think this paper has not supported the advantages of the proposed method neither empirically nor theoretically

**Reproducibility.** The source code is not attached and experimental conditions are not fully written in the paper (including appendix), I need to say that the reproducibility is low for now.
summary_of_the_review: Though the problem setting and proposed method is interesting, I have several concerns as I listed in the Weakness. Hence, I recommend rejection for now.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a safe reinforcement learning (RL) method, where the agent maintains Dirichlet-Categorical models of state transitions, and the proposed method of approximating the confidence that the risk is below a certain level is used to control exploration.  The safe RL is certainly important, and the paper proposes a theoretically well grounded method for this important subject, which is the main strength of the paper.

The major weakness pointed out by reviewers is its scalability.  While the effectiveness of the proposed method is empirically demonstrated in tabular environments, it is not at all clear how the proposed method can be applied with functional approximations, which are needed for most practical tasks.  
justification_for_why_not_higher_score: The proposed method has significant limitation in its applicability, and it is unclear how it can be applied beyond toy tasks in tabular environments.
Decision: Reject