ROSARL: Reward-Only Safe Reinforcement Learning

Authors: Geraud Nangue Tasse, Tamlin Love, Mark Nemecek, Steven James, Benjamin Rosman
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Deep Reinforcement Learning, Safety, Safe AI, Safe RL

Abstract:
An important problem in reinforcement learning is designing agents that learn to solve tasks safely in an environment. A common solution is to define either a penalty in the reward function or a cost to be minimised when reaching unsafe states. However, designing reward or cost functions is non-trivial and can increase with the complexity of the problem. To address this, we investigate the concept of a Minmax penalty, the smallest penalty for unsafe states that leads to safe optimal policies, regardless of task rewards. We derive an upper and lower bound on this penalty by considering both environment diameter and solvability. Additionally, we propose a simple algorithm for agents to estimate this penalty while learning task policies. Our experiments demonstrate the effectiveness of this approach in enabling agents to learn safe policies in high-dimensional continuous control environments.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730470589740

thread_type:
official_review

<thread_object>
creator: reviewer_lxye

<content>
summary:
The paper tackles the problem of solving tasks safely using reinforcement learning. The main idea of the paper is to introduce a term to appropriately penalise unsafe behaviour, the analysis of the paper seeks to bound the magnitude of this term to minimise the probability of arriving at an unsafe goal state while achieving the task. To develop this term, the authors establish several objects required to quantify the complexity and "size" of the problem. These two measures are then used to construct the additional reward term to adhere to the safety criterion. It is then shown theoretically that adding a reward term that adheres to certain inequality constraints ensures a high degree of safe behavior.

strengths:
The paper addresses an important challenge with an intriguing idea. Beyond supplying solely theoretical results, the paper provides a practical method suggested by the theoretical analysis. The results suggest that the method leads to a notable reduction in safety costs which is one of the main stated goals of the paper. 

The intuition of the paper is largely clear and most of the non-technical parts are nicely written and overall the concepts well-explained. The authors have included a practical example midway through the paper which is useful for discussing the ideas in a concrete setting and explaining some of the concepts.

weaknesses:
===Analysis===

A1. Some parts of the paper aren't written with a high level of rigour, for example, the experiments of Section 5 are divided into parts depending on whether the "theoretical assumptions are satisfied" but there is no clear statement as to what exact assumptions this refers to. Additionally, there are a few typos in the mathematical expressions e.g., the argument of the min and max operator should be $s_T$ not $s$. There are other examples and some more technical concerns I have such as the existence of $D$ - I have given more details of this concern below. Other more minor  points  are as follows:

   **In Definition 2 the case of equality is not covered.

   **Theorem 1 is a bit pointless - perhaps the authors could consider writing it as a note within the text.

A2. I am finding it difficult to understand how the object in Definition 3 could be computed. Although the restriction to proper policies ensures the goal state is reached in a finite number of time-steps, in a given environment, for any deterministic policy I may be able to increase the number of time steps it takes to reach the goal by for example asking the agent to go back on itself before going forward again. In this case, it is unclear whether such a maximum exists (although the function T may be upper bounded on the set of proper policies, its image is not a closed set).

A3. I found it slightly troubling that many of the key results and characterisations are derived using the solvability parameter $C$ an then because calculating this is impractical as the authors acknowledge, the algorithm put forward uses a replacement for $C$ without rigorous explanation as to whether the resulting calculations are good proxies. Indeed, although some intuition is provided as to why all is not lost by omitting $C$ in the calculation, the effect of this omission on the computations and overall behaviour is not properly studied.

===Experiments===

B1. A possible concern is that although from the empirical evaluation the method does reduce the total costs by an appreciable amount, this seems to come at a heavy price in the returns. Is there a configurable parameter that can control this trade-off. I would also like to see a comparison to the baselines that allow this trade-off to be configured, specifically considering a range of values for the parameter that controls this trade-off.

B2. Overall, I did not find the experimental results to be extremely compelling, firstly the results themselves suggest that the method reduces costs at a significant price to the rewards but also, the environments are quite simple and the method has not been tested on a range of complex environments e.g. safety gym.

questions:
1. On lines 267-268, is the method considering proper policies, if so why do the expressions have $R_{\rm MAX}$ and $R_{\rm MIN}$ and not $DR_{\rm MAX}$ and $DR_{\rm MIN}$ respectively?

2. How can we be sure that introducing the minmax penalty preserves the original solution of the MDP (and hence, we are still solving the original problem)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730553356043

thread_type:
official_review

<thread_object>
creator: reviewer_sfyw

<content>
summary:
The paper discusses a reward-only safe RL as an alternative to reward-shaping RL, and constraint-based RL for safe RL. The main idea is to compute the minmax penalty, which will be used to re-cast the problem so that a safe policy can be computed using standard RL methods. The method is evaluated on classical methods such as TRPO Lagrangian, CPO and a newer method called Saute RL with TRPO backbone. The environment is quite challenging and coming from the safety gym.

strengths:
1. This is a different and possibly a new method to view safe RL 
2. The paper is quite well-written, however, I believe the algorithm is not described in the best way (see weaknesses) 
3. The authors present novel theoretical results and then discuss their strengths and limitations. In particular, they discuss convergence, computational complexity, applicability of the results

weaknesses:
1. The algorithm is not well-described. It’s hard to understand the mathematical formulation of the problem and the solution. 
2. Experimental results are limited in scope and some of the conclusions are slightly misleading. 
            a. For example, Saute RL solves a problem with probability one constraints and cannot be directly compared to TRPO Lagrangian which solves the problem with constraints on average. Especially, when the environment is highly stochastic.
            b. There are other baselines that the authors could compare to, to make their case. I recommend looking into implementations in https://github.com/PKU-Alignment/safety-gymnasium
            c. There’s no ablation study
3. Presentation of the paper focuses on the theoretical results, which are hard to judge without an explicit mathematical problem formulation. 
4. The experimental part is hard to evaluate due to the lack of details, for example the algorithm is not even in the main paper (it’s in appendix).

questions:
1. The authors look for a minimax penalty. Could the authors comment on the similarities and the differences with Lagrangian methods, where the algorithm performs an automatic search for a Lagrangian coefficient that weighs the constraint violation values.
2. What is the problem that this algorithm aims to solve? I would like to see a precise mathematical formulation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649855299

thread_type:
official_review

<thread_object>
creator: reviewer_pefw

<content>
summary:
This paper introduces Reward-Only Safe RL (ROSARL), a novel safe RL agent that focuses on learning appropriate penalties for unsafe states rather than using explicitly constructed constraints or human-designed penalties. The key technical contribution is the concept of a "Minmax penalty" - the smallest penalty for unsafe states that guarantees safe optimal policies regardless of task rewards. The authors derive theoretical upper and lower bounds on this penalty based on environment diameter and solvability. A practical algorithm is designed for estimating the minimax penalties during under model-free RL setting. The authors empirically demonstrate that the resulting algorithm yields stronger and safer performance on grid world and PILLAR environments, compared to selected baseline methods like CPO and TRPO-Lagrangian.

strengths:
- The theoretical proofs are clearly and correctly stated.
- The motivation is clear and well-written.
- The proposed practical algorithm is simple (in a good) that can be integrated with existing RL methods.

weaknesses:
- The practical algorithm uses a simplified estimate that does not explicitly consider solvability. Moreover, the algorithm leverages value function estimates as approximations without theoretical guarantees. More analysis for addressing the gap between the theoretically constructed objectives and the actual practical objective used would significantly improve the paper.
- There are two main issues with the experimental study. Firstly, only two environments are used for empirical evaluation of the proposed algorithm, whilst one of them is grid world. Secondly, there are limited analysis of failure cases, which I believe might contain interesting insights. 
- Some minor issues. It would be useful to include some crucial baseline comparisons, such as PID-Lagrangian (Stooke et al. 2020). Also, there is no analysis or ablation studies on the computational complexity of the proposed algorithm, hindering its practical applicability.

questions:
- Why does the practical algorithm work despite ignoring solvability? What are the theoretical guarantees?
- What happens when the environment has delayed consequences for unsafe actions rather than immediate terminal states?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700475989

thread_type:
official_review

<thread_object>
creator: reviewer_ndzj

<content>
summary:
This paper addresses the difficulty of reward/cost engineering in Safe RL by proposing a minimax penalty. Particularly, it uses notions of environment diameter and solvability to bound on both sides and estimate the minimum penalty for unsafe states that leads to optimally safe policies independent of task rewards.

strengths:
- well-motivated
- extensive experimentation 
- good theoretical justification
- good study of success/failure cases of proposed approach and baselines

weaknesses:
- consider adding up/down arrows to indicate if higher/lower values are good in the graphs (Average Returns(↑) and Failure Rate(↓))
- consider citing some works on feasibility/reachability in RL

Minor errors:
-Definition 1 typo “sThen”

questions:
- what are the limitations of this approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732404204686

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated paper with clarifications and additional experiments (Part 1)

comment:
We would like to thank all the reviewers for their time and effort spent reviewing our paper and providing useful feedback. A number of reviewers have expressed concerns about:
   * Not knowing what our precise mathematical problem definition is. 
   * The lack of complex environments and lack of ablation experiments.
   * The use of only 2 environments in the experiments in the main paper.


We believe these are extremely important points that are at the source of their other concerns, and hence merit a general response.
We hope the following clarifications and additional experiments convincingly clarifies all these concerns. We have also marked in red all the main changes in the main paper, and moved the related works section to the appendix for additional space (we also include a brief version in the main paper).


# Our precise mathematical problem definition
   * We have moved the safety definition to the background section to make our problem setting clearer in the background and given a precise mathematical problem formulation: 
     * Where $\widehat\Pi$ is the set of all safe policies, the goal of an agent in this work is to learn an optimal safe policy $\widehat\pi^*$ that maximises the value function $V^{\widehat\pi^*}(s) = \max_{\widehat\pi\in\widehat\Pi} V^{\widehat\pi}(s) $ for all $s \in \mathcal{S}$. 
   * **Hence, the optimal safe policy according to our definition is the one that maximises rewards *only* over safe policies, similarly to prior works.** 
   * Hence, it is optimal to sacrifice returns by taking the longer paths to the goal, if that ensures the policy is safer. This is especially true in stochastic environments. It is also especially true in environments with dense rewards like safety-gym where the rewards incentivise the agent to go straight to the goal ignoring all the hazards along the way. Also note how when the environment is not noisy, the policy that maximises rewards is also a safe policy. Figures 4 and 5 are perfect examples of these, and demonstrate how TRPO Minmax indeed achieves that desired behaviour.

# The lack of complex environments and lack of ablation experiments 
   * **The original submission included ablations in 4 complex safety-gym environments (see Figures 13-22 in pages 19-25)**. We investigated the effect of removing safe goals termination on the performance of all the algorithms, and the effect of different cost thresholds on the baselines.
   * We have now also added several experiments using Safety-Gymasium (as requested by Reviewer SFyW) and with additional baselines using OmniSafe (as requested by Reviewer PeFw). Instead of PID-Lagrangian mentioned by Reviewer PeFw which is an older algorithm, we included P3O instead. Precisely:
     * We evaluate the performance of trained models in the Pillar domain **(see Table 2 in page 26)**. We also include the training curves. **We observe that only TPRO-Minmax prioritises minimising the probability of unsafe transitions, consistently achieving the lowest cost while trading off the rewards. Interestingly, by using Algorithm 2 with OmniSafe’s implementation of TRPO, TPRO-Minmax achieves the lowest cost, highest success rate, and highest returns across all noise levels**.
     * We also investigate the effect of termination on the performance of trained models in 2 Safety-Gymnasium default environments (PointGoal1 and PointPush1). See **Tables 3-7 in pages 30-31**. We also include the training curves. Consistent with previous results, we observe that our approach consistently achieves the lowest cost while trading off the rewards.
   * We are also happy to run any additional ablation and any of the baselines included in OmniSafe that the reviewers believe is necessary, time permitting.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a safe RL algorithm, Reward-Only Safe RL (ROSARL), that learns penalties for unsafe states (behaviors) rather than using predefined penalties. The main technicality is the concept of minmax penalty, i.e., the smallest penalty for unsafe states that guarantees safe optimal policies (independent of task rewards). The authors derive theoretical bounds on this penalty based on environment diameter and solvability.They propose a method for estimating the minimax penalties in a model-free RL setting. Finally, they empirically evaluate their algorithm and compare it with CPO and TRPO-Lagrangian. 

(+) The paper addresses an important challenge with a novel idea. 
(+) The motivation behind the work is well-explained. The paper is well-written, although reviewers think the algorithm could be explained better and parts of the paper, especially in the experimental section are not written rigorously. 
(+) The paper has a good balance of theory and algorithmic implementation. 

(-) The reviewers found the scope of the experiments limited and some conclusions from them slightly misleading. They also feel comparison to newer baselines would be necessary. Overall, They did not find the experimental results compelling.
(-) The practical algorithm is obtained using a rough estimate from the theoretical results. There is not much in the paper to identify the gap between the theoretically constructed objectives and the one used by the practical algorithm.
(-) The reviewers are not satisfied with the way that the algorithm is explained. Moreover, they found parts of the paper, especially in the experimental section not written rigorously enough. 

I see this as a borderline paper. Thus, I strongly recommend that the authors revise their work using the reviewers' comments and prepare it for an upcoming venue.