Risk-aware Bayesian RL for Cautious Exploration
Authors: Rohan Mitta
Hosein Hasanbeig
Daniel Kroening
Alessandro Abate
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
Bayesian inference
Safe learning
Risk
Safety Specification

Abstract: This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL), such that the safety constraint violations are bounded at any point during learning. Whilst enforcing safety during training might limit the agent's exploration, we propose a new architecture that handles the trade-off between efficient progress in exploration and safety maintenance. 
As the agent's exploration progresses, we update Dirichlet-Categorical models of the transition probabilities of the Markov decision process that describes the agent's behavior within the environment by means of Bayesian inference. We then propose a way to approximate moments of the agent's belief about the risk associated with the agent's behavior originating from local action selection. We demonstrate that this approach can be easily coupled with RL, we provide rigorous theoretical guarantees, and we present experimental results to showcase the performance of the overall architecture.

Under review as a conference paper at ICLR 2023
RISK-AWARE BAYESIAN REINNORCEMENT LEARNING FOR CAUTIOUS EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper addresses the problem of maintaining safety during training in Rein- forcement Learning (RL), such that the safety constraint violations are bounded at any point during learning. Whilst enforcing safety during training might limit the agent's exploration, we propose a new architecture that handles the trade-off between efficient progress in exploration and safety maintenance. As the agent's s exploration progresses, we update Dirichlet-Catee Categorical models of the transition probabilities of the Markov decision process that describes the agent's 's behaviour within the environment by means of Bayesian inference. We then propose a way approximate moments of the agent's 's helief about the risk associated to the agent's behaviour originating from local action selection. We demonstrate that this ap- proach can be easily coupled with RL, we provide rigorous theoretical guarantees, and we present experimental results showcase the performance of the overall architecture.
INTRODUCTION
Traditionally, RL is principally concerned with the policy that the agent generates by the end of the learning process. In other words, the agent's policy during learning overlooked to the benefit of learning how to behave optimally. Accordingly, many standard RL methods rely on the assumption that the agent selects each available action at every state infinitely often during exploration (Sut-) ton et al. 2018; Puterman 2014). A related technical assumption that is often made is that the MDP ergodic, meaning that every state is reachable from every other state under proper action selection (Moldovan & Abbeell 2012). These assumptions may sometimes be reasonable, e.g., in virtual environments where restarting is always an option. However, in safety-critical systems, these assumptions might be unreasonable, as we may explicitly require the agent to never visit certain un- safe states. Indeed, in a variety of RL applications the safety of the agent is particularly important, e.g. expensive autonomous platforms robots that work in proximity of humans. Thus, researchers are recently paying increasing attention not only to maximising a long-term task-driven reward, but also enforcing avoidance of unsafe training.
Related Work The general problem of Safe RL has been an active area of research in which nu- merous approaches and definitions of safety have been proposed (Brunke et al.] 2021 Garcia & Fernandez, 2015 Pecka & Svoboda 2014). In (Moldovan & Abbeel| 2012). safety is defined in terms of ergodicity, with the goal of safety being that an agent is always able to return to its cur- rent state after moving away from it. In (Chow et al.] 2018a), safcty is pursued by minimising a cost associated with worst-case scenarios, when cost is associated with lack of safety. Similarly, (Miryoosefi et al. 2019) defines the safety constraint in terms of the expected sum of a vector of measurements to in target set. Other approaches (Li & Belta 2019 Hasanbeig al. 2019a b 2020 Cai et al.) 2021 Hasanbeig et al., 2022) define safety by the satisfaction of temporal logical formulae of the learnt policy, but do not provide safety while training such policy. Many existing approaches have been concerned with providing guarantees on the safety of the learned policy some- times under the assumption that a backup policy is available (Coraluppi & Mareus, 1999 Perkins
& Barto 2002 Geibel & Wysotzki) 2005 Mannucci et al., 2017: Chow et al. 22018b Mao et al.)
2019). These methods are applicable systems if they can be trained on accurate simulations, but for many other real-world systems we instead require safety during training.
There has also been much research done into the development of approaches maintaining safety during training. For instance, (Alshickh et al,, 2017 Jansen et al., (2019; Giacobbe et al.) (202]) leverage the concept of shield that stops the agent from choosing any unsafe actions. The shield

---

Under review as a conference paper at ICLR 2023
assumes the agent has to observe the entire MDP (and opponents) to construct safety (game) model, which will be unavailable for many partially-kowwn MDP tasks. The approach in (Garcia & Fernandez 2012) assumes a predefined safe baseline policy that is most likely sub-optimal, and attempts to slowly improve it with a slightly noisy action-selection policy, while defaulting to the baseline policy whenever measure of safety is exceeded. However, this measure of safety assumes that nearby states have similar safety levels, which may not always be the case. Another common approach is to use expert demonstrations
attempt to learn how behave safely (Abbeel et al.) 2010), or even to include an option to default to an expert when the risk is too high ('oorey & Taylor] 2012). Obviously, such approaches rely heavily on the presence and help of an expert, which cannot al always be counted upon. Other approaches on this problem (Wen & Topcu| 2018 Cheng et al. 2019; Turchetta et al., 2016) are either computationally expensive or require explicit, strong assumptions about the model of agent-environment interactions. Crucially, maintaining safety in RL by efficiently leveraging available data is an open problem (Taylor et al. (021).
Contributions We tackle the problem of synthesising a policy via RL that optimises a discounted reward, while not violating safery requirement during learning. This paper puts forward a cautious RL scheme that assumes the agent maintains a Brriclet-Categoriis porate higher-order information from the Dirichlet distributions, in particular we compute approxi- mations of the (co)variances of the risk terms. This allows the agent to reason about the contribution of epistemic uncertainty to the risk level, and therefore to make better informed decisions about how to stay safe during learning. We show convergence results for these approximations, and propose novel method to derive an approximate bound on the confidence that the risk is below a certain level. The new method adds functionality to the agent that prevents it from taking critically risky actions, and instead leads the agent to take safer actions whenever possible, but otherwise leaves the agent to explore normal. The proposed method is versatile given that it can be added on to general RL training schemes, in order to maintain safety during learning.
model of the MDP. We incor-
2 BACKGROUND
2.1 PROBLEM SETUP
Definition 2.1 finite MDP with rewards (Sutton et al. 2018 is a tuple M = (Q, A, qo, P. Re) where Q {q' q² "²"g"".... "²....."" is finite set of states, A is finite set of actions, without loss of generality qu is the initial state, P(g'lq, a) is the probability of transitioning from state q to state of after taking action a, and Re(g, a.) is real-valued random variable which represents the reward obtained after taking action a in state q. A realisation of this random variable (namely a sample, obtained for instance during exploration) will be denoted by re(q,a). a).
An agent is placed at qu E Q at time step t = 0. At every time step t E No, the agent selects an action E A, and the environment responds by moving the agent to some new state 9--1 according to the transition probability distribution, i.e., 4++1 ~ P(- at). The environment also assigns the agent reward re(gt; az). The objective of the agent is to learn how to maximise the long term reward. In the following we explain these notions more formally.
Definition 2.2 A policy π assigns a distribution over A at each state: T(alq) is the probability of selecting action in state q. Given a policy म. we can then define state-value function
Ux 0x(4) = E Σ
Yrre(g, az)
qo =
where [-] denotes the expected value given that actions are selected according to and 0 < 7 1 is a discount factor.
Specifically, this means that the sequence qo, 40, 91. C1.... is such that an T(-1qx) and 9n+1 ~ P(-1q,, ).). The discount factor y is pre-determined
ppper-parameter that causes immediate re- wards to be worth more than rewards in the future, as well as ensuring that this sum is well-defined, provided the standard assumption of bounded rewards. The agent's s goal to learn an optimal policy, namely one that maximises the expected discounted return. This is actually equivalent to finding a policy that maximises the state-value function V, (q) at every state (Sutton et al. 2018).
Definition 2.3 A policy n is optimal if, at every state q, ux (=())= v. (q) max, Ux (g)-
2

---

Under review as a conference paper at ICLR 2023
Definition 2.4 Given a policy n, can define state-action-vaiee function Vz(9, a) E" 1Et=o Prre(q, at) qa = 9, au = a] similarly to the state-value function. This allows us to reinterpret the state-value function (g) E. Ux (q, ())(aa))) and thus we see that optimal deterministic policy a must assign zero probability to any action a that doesn't I maximise the state-action value function.
2.2 SANCHLET-CATEGORICAA MODEL OF THE MDP
We consider model for an MDP with unknown transition probabilities (Ghavamzadeh al. 2015). The transition probabilities for a given state-action pair are assumed to be described by a categorical distribution over the next state. We maintain Dirichlet distribution over the possible values of those transition probabilities: since the Dirichlet distribution is conjugate to the categorical distribution, we can employ Bayesian inference update the Dirichlet distribution, as new observations are made while the agent explores the enviromment.
Formally, for each state-action pair (q',a), we have Dirichlet distribution Pa P ..... PoN Dir(a a² " ......))). The random variable pd represents the agent's belief about the transition probability ((' a). At the start of learning, the agent will be assigned prior Dirichlet distri- bution for each state-action pair, according to its initial belief about the transition probabilities. At every time step, as the agent moves from some state o² some state d by taking action a, it will generate an event q¹ S q², which constitutes new data for the Bayesian inference. From Bayes' rule:
Pr(p'- = =xalo' gt) x Pr(g" 4 o pp =
xx = =xa)
II
(1)x)) (0
ra
jtk
which immediately yields
Pr(pl xala" 9 g²) = Dir(a oa 1..., 0 + 1, 0 I.N.
Thus, the posterior distribution is also Dirichlet distribution. This update is repeated at each time step: the relevant information to the agent's posterior belief about the transition probabili- ties is the starting prior Dir(a a2 ...aa and the "transition counts" c², keeping track of the number of times that q² S q² has occurred. The agent's 's posterior is then (pi, .?..,.... p% :...::N): ): from this distribution, can distill the expected value P of each random variable phl as well as the covariance of any two p and p (therefore also the variance of a single p')):
.....
~
Dir(a at² ,...a?
Couly' p01l a (fik a a
P = Ely'
( + 1)
where a IN an and 8öh is the Kronecker delta.
3 RISK- BAYESIAN RL FOR CAUTIOUS EXPLORATION
In this section we propose a new approach to Safe RL, which will specifically address the problem of how to learn an optimal policy in an MDP with rewards, while avoiding certain states classified as unsafe during training. The agent is assumed to know which states of the MDP are safe and which unsafe, but instead of assuming that the agent has this information globally, namely for all states of the MDP, we find it more reasonable that the agent observes states within an area around itself. This closely resembles real-world situations, where systems may have sensors that allow them to detect close-by danger areas, but not necessarily know about danger zones that are far away from them. In particular, we assume that there is an observation "boundary" O, such that the agent can observe all states that are reachable from the current state within O steps and distinguish which of those states are safe or unsafe. The rest of this section is structured follows:
1. In Section 3.1 we define the risk r. (a) over m steps of taking an action a at the current state, denoted as gf We then introduce random variable 1 (a) representing the agent's s belief about the risk;
2. In Section 3.2 we leverage method from (Casella & Berger 2021 to approximate the pected value and variance of the random variable Rs (a). We provide convergence results on the approximations of the expectation and variance of RTm(a)::
3

---

Under review as a conference paper at ICLR 2023
3. In Section 3.3
show how the Cantelli Inequality (Cantelli 1929) allows us estimate
a confidence bound on the risk r²"(a))
4. In Section 3.4] we prescribe a methodology for incorporating the expectation and variance of risk into the action selection during the training of an RL agent.
3.1 DEFINITION AND CHARACTERISATION OF THE RISK
Given the observation boundary O, reason about the risk incurred over the next steps after taking a particular action a in the current state q", for any m S O. However, note that there is a dependence between the agent's s estimate of such risk and the action selection policy. In order to solve this dilemma we severe the dependency between the risk that we calculate and the actions selected generating that risk by fixing policy over the m-step horizon, and calculating the risk given that policy, Similar to temporal-difference learning schemes, this is done by assuming best-case action selection, namely, the m-step risk r (a) at state qe after taking action is defined assuming that after selecting action a, the agent will select subsequent actions to minimize the expected risk going forward. Assuming that the agent is state q", we define the agent's approximation of the m-step risk R (a) by "expected safest policy" over m steps, as follows:
of that estimate to inform its
the risk given the
R (a is observed and unsafe);
if q² is observed and unsafe otherwise;
Ra+1(a)
(2)
R E P.
if qt is observed and unsafe minaen R (a) otherwise.
Rn++1
(3)
We terminate this iterative process n m and once have calculated (a) (c = k) for all actions a. Note that, despite the use of progressing indices n, this is an iterative back-propagation that leverages the expected values of agent's belief about the transition probabilities, i.e., pt. Thus, RT (a) is the agent's approximation of the expectation of the probability of entering an unsafe state within m steps by selecting action a state q", and thereafter by selecting actions that it currently believes will minmmize the probability of entering unsafe states over the given time horizon.
The term phj Elph*] is used as point estimate of the true transition probability t P( la a). The value of Rm only relics on states which the agent believes are reachable from of within m steps. In particular so long the horizon is less than the observation boundary O, the agent is able to observe all states which are relevant to the calculation of R. (a), so specifically, (q' unsafe) 1(q" observed and unsafe) for all relevant states q² (see Appendix G for more details).
3.2 APPROXIMATION OF EXPECTED VALUE AND COVARIANCE OF THE RISK
Let x denote the vector of variables xyd where i.j range from 1 to N and ranges over A, i.e., X ((rf)) ou We assume that these indices are ordered lexicographically by ai,a; This because and a will be used to signify a state-action pair (q a). and j will be used to signify potential next state q'. Introduce a set of functions (we shall see they take the shape of polynomials) of [x] defined, for each state q², as follows:
ii == I(g is observed and unsafe);
if q is observed and unsafe otherwise;
9k ++ (a)[x) = EnT 2 [z
if g² is observed and unsafe
g [x]
[ (arg min, R 1 (a)) [x] otherwise.
Then can write the risk (of selecting action a in state qº, rm
steps) defined above as (m(a) (a) = get (a)(t), where t = ((12) )i.j=1,.... and VaEA) is a vector of all "true" transition prob- abilities 1 = P(g'lg',a). We can similarly write the agent's approximation of the risk as R.T = g.t (a)[p). where similarly p ((P2) Jij=I.... and 1).. We refer to the actions spec- ified by these argmin operators the agent's expected safest action in each state over the next
4

---

Under review as a conference paper at ICLR 2023
steps. Now, crucially,
also define
random variable Re ge (a) where
((rr)).j= and Aa Since the p are random variables representing the agent's beliefs about the true transition probabilities t2. we in fact have that this random variable _ee (a) represents the agent's 's beliefs about the true risk r (a). In the following. we show that R. (a) can be viewed approximation of E R (a)], and
provide and justify approximation of Var RR." (a)]-
These approximations can be used by the agent to reason about the true risk r2 (a).
In order to construct approximations of the expectation and variance of Re (a), we make use of the first-order Taylor expansion of 9:" (a)(x) around x p. following method in (Casella & Berger 2021). The Taylor expansion is
0g. (a) 0uyy
9. (a)
(a x
9. g.
(x)
remainder,
(4)
PE
where the partial derivatives are also evaluated p. Now we can turn equation f approximation by dropping the remainder and reasoning over the random variables for x, namely:
into statistical
05 Ozh
-e xa (a)
(Pb pb)
(5)
DEA
We can then take the expectation of both sides, obtaining
Eig." (a) [p]] E[g. (a)
P
(0b xb-)
duy
BEA
Og (a)
= =
Bi(p) 1j - Pb = (p)
Ony
1-=1bEA
where the above steps follow since the only random term the right-hand side is p for which B(p!) pp Recall that g. (a) [p] =R" and (a) [p] R.th (a). Thus, we now have R (a) as an approximation of the expectation of RTH (a). For the approximation of the variance of the agent's 's believed risk, which is again random variable, we can write:
Var(ge (a)lp)) & El(g." (a)lp) de (a)(
27
N
0g (a)
2 E
(wb
-pb))
(from equation 5
Onh
PEA
0g.
0g.
Cou(p pb-))
dxhy
Ougt
ij,,t== bi Aa
N
0g (a)
0g. (a)
Coulp pb) = V:T (a),
(6)
Oxyy
dxh
i=l 1E
where VTM (a) is the approximation for the variance of RTh (a), i.e., R Var(R. and the last line follows from the fact that the covariance between two transition probability beliefs p and piz is always O, unless they correspond to the same starting state-action pair (g", b). In other words, Cov(pg 35 pt 0 unless j and b = by- Next, we show consistency of the estimate in the limit (see Appendix for the proof).
Theorem 3.1 Under Q-learning convergence assumptions [aakinns] 1989,, namely that reachable state-action pairs are visited infinitely often, the estimate of the mean of the believed risk distribu- tion RTT (a) converges the true risk r" (a), and it does with the variance of the believed risk distribution V Var (a)(p) (p) approaching the estimate of that variance V.m (a). Specifically.
R (a) T (a)
NN(0, 1) in distribution.
(a)
5

---

Under review as a conference paper at ICLR 2023
3.3 ESTIMATING CONFIDENCE ON THE APPROXIMATION OF THE RISK
So far we have shown that when the agent in the current state qf, for each possible action a, approximations of the expectation and variance of its belief R (a] about the risk r" (a) can be formally obtained: we denote these two approximations by R (a) and Vin(a), respectively. We describe method for combining these approximations to obtain a bound on the level of confidence that the risk r²" (a) is below a certain threshold.
We appeal to the Cantelli Inequality, which is a one-sided Chebychev bound (Cantelli 1929). Hav- ing computed R
(a) and V." (a). for a particular confidence value < Ć 1 we can define
[m)C Pa = R (a) + 1-
From the Cantelli Inequality we then have
Pr(R. (a) S Pa) 2 C.
Specifically, Pa is the lowest risk level such that, according its approximations, the agent be at least 100 x C % confident that the true risk is below level Pa- The agent can therefore leverage Pa when attempting perform safe exploration (please refer to Appendix for more details).
3.4 RISK-AWARE BAYESIAN RL FOR CAUTIOUS EXPLORATION (RCRL)
We propose setup for Safe RL that leverages the expectation and variance of the risk to allow an agent to explore the environment safely, while attempting to learn an optimal policy. In order to pick the most optimal yet safe action at each state, we propose a double-learner architecture, referred to as Risk-aware Cautious RL (RCRL.) and explained next.
The first learner is an optimistic agent that employs Q-learning (QL) maximize the expected cumulative return. The second learner is a pessimistic agent that maintains Dirichlet-Categorica model of the transition probabilities of the MDP. In particular, this agent is initialized with prior Pri that encodes any information the agent might have about the transition probabilities. For each state-action pair (g', a) we have Dirichlet distribution P Pd² ..... Pa Dir a ...a, .....,) As the agent explores the environment, the Dirichlet distributions are updated using Bayesian inference.
For each action a available in the current state qf, the pessimistic learner computes the approxima- tions RTT( (a) and (a) a) of its belief R.t (a) of the risk over the next m steps of taking action a in is hyper-parameter that, discussed, should be set at most the obser- vation boundary O. The pessimistic learner is also initialized with two hyper-parameters Pmxx and C(n): Pmux represents the maximum level of risk that the agent should be prepared to take, whereas C(n) is a decreasing function of the number of times n that the current state has been visited, which satisfies C(0) and lim ((n) O. From Section 3.3 action the value
q. The "risk horizon"
the agent can then compute, for each
VTm (a)
Pa = Ru (a) +
1-C(n)
which can thus define a set of safe actions: these are all the actions that the agent believes have risk less than Prnars with confidence at least C(n), namely
Asafe = {a E A|Pa S Pmaa).
In case there are no actions a such that P. < Pmax. the agent instead allows
Asafe {a E AM (a) min RTT (a))-
(8)
Finally, the agent selects an action a" from the set of safe actions using softmax action selection (Sut- ton al. 2018) according to the Q-values of those actions, with some temperature >> 0:
/a)/t
Pr(a" = EnEAm elg oo)/t
The pseudo-code for the full algorithm is available in Appendix B
In summary, effectively have two agents learning to accomplish two tasks. The first agent per- forms Q-learning to learn an optimal policy for the reward. The second agent determines the best approximation of the expected value and variance of each action, enabling it to prevent the first agent from selecting actions that cannot guarantee to be safe enough (with at least a given confi- dence). When instead the pessimistic agent cannot guarantee that any action is safe enough, it forces

---

Under review as a conference paper at ICLR 2023
Table Total successes and failures. Gridworld: different priors and acceptable risks Pmax. averaged over 10 agents. PacMan: varying risk horizon m. single agent.
Experiment
Setup
# Successes
# Failures
Total Episodes
Prior 1,
Pmaxx 0.33
404.3
54.2
500
Prior I, mmx 0.01
506.0
417.9
1500
Prior 2, Piaxx 0.01
384.6
0.5
500
Gridworld
Prior 3, mmx 0.01
407.4
1 14.4
500
Prior 3, max 0.0033
421.3
1.1
500
Native Q-Learning
414.6
990.5
1500
Risk Horizon m
234
77
311
PacMan
Risk Horizon m =
207
68
275
Native Q-Learning
0
1500
1500
the optimistic learner to go into "safety mode", i.e., to forcibly select the actions that minimize the expected value of the risk, as per equation [ From an empirical perspective, implementing this con- cept of a "safety mode" allows for continued progress, and pairs extremely well with the definition of the risk: namely, when the agent deems that a state too risky, it will go into this "safety mode" until it is back in state with sufficiently safe actions.
Finally, note that ((n) represents the level of confidence that the agent requires in an action being safe enough for it to consider taking that action. When the agent starts exploring and C(n) is at its highest, the agent only explores actions that it is very confident in. However, it may need to take actions that it is less confident in order to find an optimal policy. Thus, as it continues exploring, C(n) reduced, allowing the agent to select actions upon which it is not as confident. However, in the limit, when C(n) + O, we have that Pa = R (a). which means that the agent never takes an action if its approximation of the expected value of the risk R (a) is more than the maximum allowable risk P Pmaz.
EXPERIMENTS
Gridworld - We first evaluated the performance of RCRL on a Slippery Gridworld Bridge example. The states of the MDP consist of 20 20-grid, as depicted in Figure 2a (Appendix C). The agent is initialized at in the bottom-left corner (green). The agent's 's task is to get to the goal region without ever entering an unsafe state. particular, upon reaching a goal state, the agent given reward of 1 and the learning episode is terminated; at every other state it receives a reward of O, and upon reaching an unsafe state the learning episode terminates with reward of the 4 neighbouring states, or stay in its current position; thus, the
each time step
the agent might move into agent has access to 5 actions at each state, A = (right, up, left, down, stay). If the agent selects action A, then it has a 96% chance of moving in direction a, and 4% chance of "slipping", namely moving to another random direction. If any movement would ever take the agent outside of the grid, then the agent will just remain in place. The agent is assumed to have an observation boundary O = 2 steps. Note that due to the slipperiness of the grid and the narrow passage to reach the goal state, minimizing the risk is not aligned with maximizing the expected reward.
We tested RCRL with 5 different combinations of a prior Pri and maximum acceptable risk Pmax The following additional hyper-parameters of the algorithm were kept constant: the maximum number of steps per episode max sstepssspssssss = 400, the maximum number of episodes max_epiee maareepisodes = 500 (although this was increased to 1500 in two cases when the agent did not converge to near- optimal policy within the first 500, cf. Table []: the learning rate = 0.85; the discount factor (Appendix B). Recall that prior consists of Dirichlet y 0.9; and the risk horizon m iN for every state-action pair (g', a)). We considered three distribution p P.. ~ Dir(a ...., priors:
Prior completely uninformative: in this case we assigned a value of every a. This yields distribution that is uniform over its support.
assigned a value of 12 to the corresponding moving in the correct direction, and a value of to all other a's. This gives a distribution in between Prior 1 and Prior 3 in both degree of bias and concentration.
Prior 2 weakly informative:

---

Under review as a conference paper at ICLR 2023
Prior 3 highly informative: we assigned value of 96 to the a corresponding to moving in the correct direction, and a value of 1 to all other a's. This gives a distribution that is highly concentrated, and for which the mean values of the transition probability random variables are the true transition probabilities of the MDP, and hence unbiased.
We tested the algorithm with all three priors and a maximum acceptable risk of Paux 0.01 and repeating each experiment 10 times to take averages. On average, the agent with the highly informa- tive prior (Prior 3) entered unsafe states 14.4 times, and always converged to near-optimality within about 200 steps, successfully crossing the bridge 407.4 4 times. For the other 78.2 episodes, the agent reached the episode limit within crossing the bridge or entering an unsafe state. The agent with Prior 2 interestingly only entered unsafe states an average of 0.5 times per experiment, and converged to near-optimal policy within about 300 episodes, successfully crossing the bridge 384.6 times. On the other hand, the agent with Prior only crossed the bridge less than 30 times. We therefore increased the total number of episodes to 1500 and tried again, yet still over half the time it did not converge to near-optimal policy (Appendix
We then tested Prior I with a more lenient maximum acceptable risk of Pmaz = 0.33, and found that the agent this time managed to converge to near-optimality within around 200 episodes, entering unsafe states 54.2 times and successfully crossing the bridge 404.3 times. We also tested Prior 3 with stricter P Pmexr 0.0033 and found out that it entered unsafe states only 1.1 times and succeeded 421.3 times, converging to near-optimality within 150 episodes (Appendix (A).
Finally, we tested native Q-learning, without any safe learning scheme. This native scheme had almost no successful crossings of the bridge in the first 500 episodes, so we ran it for 1500 episodes and found that it only converged to a near-optimal policy about half the time, on average entering unsafe states 990.5 times and successfully crossing the bridge 414.6 times.
Table summarizes the number of successes and failures for each agent. To understand better the rate of convergence near-optimality, Figure I [ (A) displays the number of steps taken by the agent to cross the bridge at every successful episode (it displays 400 if the agent never managed to cross the bridge) averaged over the 10 experiments. On each graph we display for comparison the theoretical least number of steps it could cross the bridge in, which is 22. Note that because the grid-world is slippery, even an optimal policy would have fluctuations above the 22-steps line.
Discussion The first result of note is how poorly Prior 1 performs with Pmaz = 0.01. It mostly fails to converge to near-optimal behaviour even with 1500 steps as can be seen in Figure [p (Ap- pendix A). in fact seeming to converge slower than native Q-learning. This occurs because the maximum allowable risk is set too low for the given prior. In particular, there are two main issues with this. The first issue is a type of degenerate behaviour specific to our algorithm and to the com- pletely uninformative prior with overly strict P Pmaa:: given that the agent starts with information on the transition probabilities, it is unable to tell which actions are safe and which are unsafe. In particular, with P mea at 1%. the first time the agent arrives at any state ge from which can observe some unsafe state, it immediately goes into safety mode as it judges that the risk of every action is above 1%. Since it has no information on which action is safest, it randomly selects an action (as- suming the Q- Q-values were initialized to 0). If that randomly-selected selected action does not take the agent closer to a risky state, then after updating the agent's beliefs about the transition probabilities for that action, it will believe that action is the safest one from that state. Thus every time it encounters that state again, it will always select that action, never attempting any other actions. This behaviour can be seen in Figure 2b (Appendix C). The state (13, 1) has been visited significantly more often than any other state. This has occurred because the first time the agent encountered that state, it chose action stay, and as above, from then always chose stay in state (13, 1). This would cause the agent to remain in (13, 1) until it slipped off of that state.
The second issue with having such strict Pmux could involve any prior. In this case Pmux is set so low that actions that may be optimal are simply never tested, as the agent's initial belief about those actions causes the expected risk associated with them to always be greater than Pmmx- This should not be viewed as an undesirable consequence of the algorithm, but rather as the algorithm working as intended. With the maximum allowable risk level Pmux set so low, the agent judges that certain actions are riskier than acceptable and therefore does not take them. However, this does raise more general question about the nature of safe learning in general: ensuring safety while learning necessarily means avoiding actions we believe are too dangerous, so if we want any guarantees on safety, then we must accept that the agent may be unable explore the entire state space.
8

---

Under review as a conference paper at ICLR 2023
The second result of note is that Prior performs much less safely than Prior 2 does at Pmax 0.01. This seems counter intuitive at first, given that Prior is more accurate and more confident than Prior 2. However, the explanation is quite simple. Prior 3 (initially) causes the agent's 's expected belief to correctly predict that there is only a 1 1% chance of moving to an unsafe state on a particular step if the agent selects the action to move away from it. On the other hand, Prior 2 causes the agent's expected belief to predict there is a 6.25% chance of this happening. Thus, Prior 3 (correctly) evaluates the risk of moving within 1 step of a risky state as much lower than Prior 2 does. It likely that at some points in the experiments, the agent with Prior 3 chose to move within 1 step of an unsafe state where an agent starting with Prior 2 (with the same experiences) would have rejected that action as too risky. The agent with Prior 3 would then be at risk of slipping into an unsafe state. In Figure 2c and 2d (Appendix C. we can see exactly this happening, where Prior 3 regularly visits state (13, 8), which is adjacent to the unsafe state (12, 8). Prior 2 instead regularly moves one more state to the right before moving up to row 13, since (12, 9) is safe.
Prior 3 with P Pmax_ = 0.0033 shows how we can make use of a highly accurate prior to guarantee even less risk, and in this case the agent almost never enters unsafe states, while converging faster than any other setup to near-optimality.
The final result is that the rate of convergence of the native Q-eaarning agent is much slower on this MDP than the other agents (excluding Prior 1 with the inappropriate Pmax = 0.01). As in Figure (Appendix A Q-learning took between 300 and 1500 episodes to converge when it did, and occasionally failed to converge, compared to 150-300 episodes for the four other agents to converge in all 10 experiments. This was even the case for the agent with the completely uninformative prior, with max = 0.33. This is a key result: it shows that not only can RCRL keep the agent safe during learning when possible, it may also direct the agent to explore more fruitful areas of the state-space. In this case study in particular, the native Q-learning agent entered unsafe states so often initially that it took many episodes before it was able access the bridge and find the reward at the other side. Conversely, since the safe agents mostly avoided "sinking" situations, they were able to explore much more of the state space on each episode.
PacMan We also evaluated the performance of RCRL on a PacMan example. Figure 3a (Ap- pendix D) depicts the initial state of the environment, where the agent (PacMan) must get both yellow dots (food) without getting caught by the ghost. Note that because both the agent and the ghost move through the maze, the PacMan MDP has about 10 times more states than the Gridworld, and up to 5 times more possible next states at any given state. Upon picking up the second piece of food, the agent is given a reward of I and the learning episode stops. Every other state incurs reward of O and if the ghost catches PacMan, the learning episode stops with reward O. The agent has access to four actions each state, A (right, up, left, down), and will move the direc- tion selected, or if that direction moves into a wall, then it will stay still. The ghost will with 90% probability move in the direction that takes it closest to the agent's next location, and with 10% probability will move in random direction. For this setup,
assumed an observation boundary O = 3 and compared two values of the risk horizon, m = 2, 3. We therefore kept constant the other parameters and typee-parametent the learning rate μ 0.85; the discount factor 7 = 0.9; the max- imum number of steps per episode mar_steps = 400; the maximum acceptable risk Pmaaz = 0.33; the prior, which set to be completely uninformative prior as in the Gridworld example; the maximum number of episodes, which we set as 1500 or the number of episodes before the total rate of successful episodes exceeded 75%.
As in Table T the agent with a risk horizon of m 2 steps exceeded a success rate of 75% after 311 episodes, having failed 77 times. The agent with the larger risk horizon of m = 3 only took 275 steps to exceed that success rate, and only failed 68 times. Figures 3b 3c ( (Appendix (D) display the number of steps taken by the agent to win (or 400 if they lose) for each agent, as well as the running average number of steps over the previous 50 episodes.
Discussion The improvement in performance from m = 2 to 3 is likely due to the increased foresight of the agent leading potentially avoiding entering a state from which entering a dangerous state is unavoidable. However, may also be simply due to the fact that increasing the risk horizon leads to an overall increase in risk estimates, which will naturally cause more actions to be considered too risky and may reduce the number of failures. In other words, we may have been in a situation where decreasing the maximum acceptable risk Pmaz would have led to similar improvements, and the increase in risk horizon was behaving functionally more like a decrease in P Pmaz- Both risk-aware agents compare very favourably against the Native Q-Learning agent, which did not succeed once in 1500 episodes.
move away from excessively risky scenarios further in advance,
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel, Adam Coates, and Andrew Y. Ng. Autonomous helicopter aerobatics through ap- prenticeship learning. The International Journal of Robotics Research, doi: 10. 1011177022833641137/9999999
2010.
Mohammed Alshickh, Roderick Bloem, Ruediger Ehlers, Bettina Könighofer, Scott Nickum, and Ufuk Topeu. Safe reinforcement learning via shielding, 2017.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning based control to safe reinforcement learning. CoRR, abs/2108.062666 2021.
Mingyu Cai, Hosein Hasanbeig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. Modular deep reinforcement learning for continuous motion planning with temporal logic. IEEE Robotics and Automation Letters, 6(4):7973-7980, 2021.
Francesco Paolo Cantelli. Sui confini della probabilita. In Atti del Congresso Internazionale dei Matematici: Bologna del 10 de settembre di 1928, pp. 47-60, 1929.
George Casella and Roger L. Berger. Statistical inference. Brooks/Cole Cengage Learning, 2021.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3387-3395, 2019.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Mareo Pavone. Coss-Const Reinforcement Learning with Percentile Risk Criteria. Journal of Machine Learning Research 18, pp. 1-51, 201 8a.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman Guzman, and Mohammad Ghavamzadeh. Iyapunov- based approach to safe reinforcement learning. In Advances in neural information processing systems, pp. 8092-8101, 2018b.
Stefano P. Coraluppi and Steven I. Marcus. Risk-sensitive and minimax control of discrete- time, finite-state Markov decision processes. Automatica, 35(2):301-35) 1999. doi: 10.1016/ s0005- 109180900099)0053-....
Garcia and F. Fernandez. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515-564, Dec 2012. ISSN 1076-9757. doi: 10. 1613/jair.3761. URL http:t/d:/ doi http:///ax..oo..o//.0........
3761
Javier Garcia and Fernando Fernandez. A Comprehensive Survey Safe Reinforcement Learning.
Journal of Machine Learning Research 16, 2015.
Geibel and F. Wysotzki. Risk-sensitive reinforcement learning applied to control under con- straints. Journal of Artificial Intelligence Research, 24:81 1-088 Jul 2005. doi: 10. 10.161 10.1613//jir... 1666.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian Reinforcement Learning: A Survey. Foundations and Trends in Machine Learning, 8(5-)):35 483, 2015. ISSN 1935-8237, 1935-8245. doi: 10.1561/2200000049, URL URL http:://txxi..oo/// 04436
arXiv: 1609.04436.
Mirco Giacobbe, Hosein Hasanbeig, Daniel Kroening, and Hjalmar Wijk. Shiclding Atari games with bounded prescience. In Proceedings of the 20th International Conference Autonomous Agents and MulriAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2021.
Hosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Certified reinforcement learning with logic guidance. arXiv preprint arXiv: 1=XX:00902.00775 2019a.
Hosein Hasanbeig, Alessandro Abate, and Daniel Kroening. constralc- neural fitted Q- iteration. In Proceedings of the 18th International Conference on Autonomous Agents and Multi- Agent Systems, pp. 2014,2014, International Foundation for Autonomous Agents and Multiagent Systems, 2019 lb..
10

---

Under review as a conference paper at ICLR 2023
Hosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Deep reinforcement learning with temporal logics. International Conference on Formal Modeling and Analysis of Timed Systems, pp. 1-22. Springer, 2020.
Hosein Hasanbeig, Daniel Kroening, and Alessandro Abate. LCRL:
Certified policy synthesis via logically-constrained reinforcement learning. In International Conference on Quantitative Eval- uation of Systems, pp. 217-231. Springer, 2022.
Nils Jansen, Bettina Könighofer, Sebastian Junges, Alexandru C. Serban, and Roderick Bloem. Safe reinforcement learning via probabilistic shields, 2019.
Xiao Li and Calin Belta. Temporal logie guided safe reinforcement learning using control barrier functions, 2019.
Tommaso Mannucci, Erik-Jan van Kampen, Cornelis De Visser, and Qiping Chu. Safe exploration algorithms for reinforcement learning controllers. IEEE transactions on neural networks and learning systems, 1964):106--1081 2017.
Hongzi Mao, Malte Schwarzkopf, Hao He, and Mohammad Alizadeh. Towards safe online rein- forcement learning computer systems. 33rd conference on neural information processing systems (NeurIPS 2019), 2019.
Sobhan Miryoosefi, Kianté Brantley, Hal Daume III,, Miro Dudik, and Robert E Schapire. Reinforce- ment learning with convex constraints. Advances in Neural Information Processing Systems, 32, 2019.
Teodor Mihai Moldovan and Pieter Abbeel. Safe Exploration in Markov Decision Processes. 17X5v:1205.4810 [cs], July 2012. URL 1205.4810.
abs/aabb....... 4810 arXiv:
Martin Pecka and Tomas Svoboda. Safe exploration techniques for reinforcement learning-an overview. In International Workshop Modelling and Simulation for Autonomous Systems, pp. 357-375. Springer, 2014.
Theodore J Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 3(Dec):803-83,, 2002.
Martin L. Puterman. Markov decision processes: Discrete stochastic dynamic programming John Wiley & Sons, 2014.
E. Slutsky. Über Stochastische Asymptoten und Grenzwerte. Metron Nr. 3-89 (1925)., 1925.
Richard S. Sutton, Francis Bach, and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press Ltd, 2018.
Andrew Taylor, Victor D Dorobantu, Sarah Dean, Benjamin Recht, Yisong Yue, and Aaron D Ames. Towards robust data-driven control synthesis for nonlinear systems with actuation uncer- tainty. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 6469-6476 IEEE, 2021.
Lisa Torrey and Matthew E. Taylor. Help an agent out: Student/teacher learning in sequential decision tasks. In Proceedings of the Adaptive and Learning Agents workshop (at AAMAS-12), June 2012.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov deci- sion processes with Gaussian processes. In Advances in Neural Information Processing Systems, pp. 4312 4320- 2016.
Christopher John Cornish Hellaby. Watkins. Learning from delayed rewards. PhD thesis, 1989.
Min Wen and Ufuk Topcu. Constrained cross-entropy method for safe reinforcement learning. In Advances in Neural Information Processing Systems, pp. 7450-7460, 2018
11

---

Under review as a conference paper at ICLR 2023
APPENDICES
A
APPENDIX A. GRIDWORLD: AVERAGE NUMBER OF STEPS TO SUCCEED
460
420
data oot
010 0801 0065 0400000
350-
optimaa
10000 5100 Valion 01.000
200
502
600
632
3003
17000
(a)
(b)
M000 1180 30000 0000 22000001
-00
-200
325
300
303
525
320
200
303
500
(c)
(d)
prior 3. De.. 0,0033
G-Leaming Only
200
400
250
250
200
32a
300
420
520
200
600
833
3003
1200
1460
sole 1am1A
solk namria
(e)
(
Figure 1: The number of steps it takes the agent to the bridge for every episode where it crosses. Averaged over 10 experiments. Results for Q-learning only and for RCRL across different priors and values of risk Pm... As Q-learning converges, it approaches the lower bound the optimal number of steps per episode.
12

---

Under review as a conference paper at ICLR 2023
B APPENDIX B. RISK-AWARE CAUTIOUS RL PSEUDO CODE
Algorithm 1: Risk-aware Cautious RL (RCRL)
input: Pri, C(n), Paaxy mar sleps, max episodes, H4, 7,
initialize Q(g, a) for each state-action pair (g,a), a);
121
initialize ruu__sleps = 0:
(3)
initialize num-episodes 0;
while num_episodes max max_episodes do
of q":
(4)
rum_epissoes + reumepisooees + 1;
while num_steps < maz max steps and qe is not unsafe do
calculate 12 (a) as in
calculate V.-T (a) as
681
calculate Pa as in
Asafe = {a A Pa :
191
Asafe = 0 then
Asafe {a A (a) ming (a));
(10)
end
choose action a* according to 9
pass action environment and receive next state d' and reward re( a²)) update belief as in section
(121
update Qq a²) ) - (1 p (gQ a²) + (re(q", a²) + max Q(d', a'))
(14)
of d':
1161
num sleps +- num sleps 1;
end
end
13

---

Under review as a conference paper at ICLR 2023
C
APPENDIX C. GRIDWORLD EXPERIMENT DETAILS
Pres.
90
12
(a)
(b)
stata count: prier 2,P 3.01
count prior Pass:
12
10
20 20
13
20
(c)
(d)
surrounded by the obser- vation area (white line). Labels denote target (yellow), unsafe (red) and safe states (blue), and initial state (4o, green). (b) For single experiment, number of state-visitations for Prior P 0.01.
Figure 2: (a) Slippery Gridworld setup: agent is represented
(c-d) Number of state-visitations, for Priors 2 and 3 at Pmax = 0.01.
14

---

Under review as a conference paper at ICLR 2023
D APPENDIX D. PACMAN EXPERIMENT DETAILS
HL
:
(a)
rSA m002
290
300
Sirueiing suerage
320
300
150
200
166
302
250
03030
(b)
(c)
Figure 3: PacMan Setup: agent (PacMan) starts position (1,3). Food is denoted by yellow dots, and the ghost starts in the top right corner. (b-c) Number of steps taken to win (i.e. eat both foods without being caught by the ghost) episodes where the agent does win (or 400 if the agent is caught). for risk horizon 2 and 3. The orange line denotes the running average number of steps to win over the previous 50 episodes.
15

---

Under review as a conference paper at ICLR 2023
E APPENDIX E. CONVERGENCE RESULTS FOR THE APPROXIMATIONS OF THE EXPECTED VALUE AND VARIANCE OF THE RISK
Theorem E.1 Under Q-learming convergence assumptions [Watkins] 1989, namely that reachable state-action pairs are visited infinitely often, the estimate of the mean of the believed risk distribu- tion R (a) converges to the true risk re (a), and it does with the variance of the believed risk distribution Varlg. (a) approaching the estimate of that variance Van (a). Specifically,
RR (a) (a)) - -N(00, 1) in distribution
(m)(a)
Proof.
Let us first rewrite the expressions in equation 6 in vector form, first introducing the following covariance matrix for p:
Cov(ph) p Coulph) Phy Cou(pb, Pb.
Cou(pb) pb?) 12 12
11,
L =
NN NN). Coulpes Phyt
Recall that the variables y are ordered lexicographically by (i, a, j). Here we wrote by for the first action in A and by for the last one, assuming |A| M. Using matrix E, we can rewrite equation 6 for the approximate variance as
"g (a) Ban 0g." th (a) Suy 12
Var (R. (V (a)l)) E(Vg EVV"
<a)
(10)
0g (a)
where Vg."" (a) the gradient vector of g.t gaa(a) (a) evaluated at p.
the following, we employ the 'Delta Method' as described in (Casella & Berger 2021 to allow us derive convergence result for the approximations for the mean and variance of 1 (a) that we defined above. Let us introduce a semi-vectorised representation of equation 6 where we still leverage the fact that covariances across different state-action pairs are 0, i.c.
Coo(p) Pb. Cov(pi) .pb
51) Cou(pi) p Cou(p) .bh))
Cou(pe &N ppb)
the variance-covariance matrix for ((x))) (b²))==....... Since E is built by listing the Σj along the diagonal for = L,.., N and E A, with zeros elsewhere, have that equation can be rewritten as
89 (a) Bitt 0g (a)
Var(R (a)) ~
L. (a)lp)) E (Viig" (a)
i= P
0g (a)
(11)
where Vig.t (a) is the gradient vector (V (a)(p) restricted to entries 0g. for j = 1,... N. Dzy
We refer to this approximation for the variance of RT (a) (mT((a (se (ar
Consider the random vector X (Xz)ij=l,... and aE A (with the previously discussed lexico-
graphic order on the X where each N J== follows Categorical distribution with probabilities
16

---

Under review as a conference paper at ICLR 2023
t ie. realisation of the vector X represents the result of taking transition from every state-action pair. Wherever X it represents transition q² + q¹. X then has means t and covariances
if and
Cov(XX Xgg
0
otherwise
We can then write the variance-covariance matrix for X as
Cov(Xp) , x
Cou(X x2 Cou(X X12 x11
Cov(X bi x
01
Xx
CN bar X
we observe independent random samples X(), x ( and denote the sample means as X = (X;) or X = + X (k) then for the function ge g((a) [x] we have,
Ig. (a)
Ouy
1E=1EA
This is direct result from the first-order Taylor expansion around and therefore the derivatives are evaluated at t.. In vector notation, we have
ghth (a) R
9" (V
(a) t).
where
(ay
0z @g.((a)
02,
0g (a) N
xx=t
From the 'Multivariate Delta Method' theorem (Casella & Berger 2021). as long as
(Vg." th (t) Σxx (Vg. (c) 0,
which we will prove later in Lemma 1 and Lemma we have the following convergence:
Vn (a) - 9.hh (a)[tt) (N in distribution.
(12)
Note that this is equivalent to
vn De (a) X - geth (a)(t))
NO, 1) in distribution,
(13)
where T = V
In the following we define (()) and (n)) to be what Po and Σ would have been had the agent started with it's prior about the transition probabilities and then witnessed exactly the transitions repre- sented by the random sample (1). (2) (()) Formally, suppose that the agent's 's starting prior was, for each state-action pair (g', b), that pb ill .p? ....PP NN ~ Dir(at) 22 ..... at Then we can Dir(of) n ag + n ,..., ag + +
consider the random variables p (n). pbb (m) ....PP (...)
(.NN Since x8) is the count of the number of times X was in random sample, this new distribution is exactly the result of performing Bayesian inference on the prior given the random sample as our new data. We then let
0 + +nxi
(m) := E pb
(m)
Σv
ab + n Eb
17

---

Under review as a conference paper at ICLR 2023
and we also define (n) the covariance matrix of the Pb (m) over all i, namely
Cou(ph (1)) .pp Pb Cou(p) (m), (),(())) Cou(ye
(1))))
(m)) ,Pb (00)
(2(m)) Coc(p) Pbb ()2(())
((n)
Cou(p (a)) pp p.
From Lemmal
we have
vn ge (a)lp(m) --d [a][x)
in probability,
(14)
and this allows the well-known Slutsky's Theorem (Slutky 1925 equation 13 to show that
equation
14 and
Vn (1 (4) - (ae((a)(t) (a)(t) NO 1) in distribution.
(15)
We must make one more modification to this result. Let
(r(m))² (a)((()))) I(n)
(VoT
We would like to show that n(r(1))² + 7 in probability. do this, first note that p(n) + t in prob- ability, so since ded (a) has continuous derivatives we have that (Vye (a)[p("))) + (a)
in probability. Next we note that (n(n) - Exx in probability. This because for the (i, b), (s, 62, eeneryy we have 0ff or bi + b2, and otherwise we have
-n(a if + mx)))(a +x
(Cou(pg .Pb
.pb (((()))=
E k= (at ) Ex (af* +x
nnaa + -x8, (a + +
(n Ei a ) + 1+ Σi= a
1/ = Coc (x )
Therefore we have that the products converge in probability:
( (1)"())))))) mI(") (a)(())
(a) (xxx)) Exx (o](t)) =
Since ?2 always positive, and the square root function is therefore continuous at 72, we have that (x) - т, and so Virta, + probability. Now we can finally apply Slutsky's Theorem to obtain our final result, which is
(g (a) 9. (a)(t)) - (0, in distribution. T(m)
(16)
Recall that g (a)[t] is the actual risk in the current state qf, 9.th (a) [(m)] is the agent's approxima- tion of the expectation of the risk given it's beliefs, and (7(1))2 the agent's approximation of the variance of the risk given it's beliefs (both, in this case, assuming it has seen exactly n transitions from each state). So indeed our estimate of the mean of the believed risk distribution converges to the true risk with enough data, and it does so with the variance of the believed risk distribution approaching our estimate of that variance.
18

---

Under review as a conference paper at ICLR 2023
Lemma
Civen the definition of the polynomial 9² (a)(x), we have the following:
V ge (a)[p(m)
- -º -00 (a)[ii)
0 in probability
Proof.
As required for the convergence results in Theorem 3.1 one can see that all of the coefficients in ge (a)[x] [x] are either 0 or This means that we can rewrite it as a sum of terms the form
II KJ,5
for exponents n.. This means that we can write
ghhh (a)lp())
- (a)
as sum of terms of the form
/n
n
(x)) my )
II 3
pb
II
r
Substituting in the definition of p And can simplify this by leveraging that Σx= (n.(x*)
(x)
to this expression yields
a + +28
II J.J.b
Σk=1
a sk + nx(p
n1, get
II bib
ag +
n2 k=
ni
II (xx) 4.7.b
71
Now, the a. are constants, as is and the values of X are all bounded between 0 and 1. Thus to show that this expression converges that some term the denominator dominates all terms in the numerator. Let M = Eijs ni The expression above equal to
probability, we will write as one quotient, and show
n. -Ilgo (x2 n E k== og
I (ag) + +xttt
vn
T
H (n+ ER= agtt
n
Now on the numerator of the inner quotient, there are only two terms order n M One is an
n
n II
R.
that comes from the product on the left, and one is a
"
n
from the product on the right, and these cancel each other out. This means the numerator is entirely of order M-1 less. On the other hand, the denominator of the inner quotient contains the term
19

---

We need to show that there is some possible value of X such that 8 * 0. Now the value of X is determined by the values of X == )X for each state-action pair b)). Furthermore, these X are independent, and the value of st, depends only on the value of X So if there is some state-action pair (g', b) such that two possible values of X yield two distinct values of si both with nonzero probability, then we can fix the values of the x for all j and all (h, b/) & (i, b) to be some values that occur with non-zero probability, which would fix the value of si and would be Under review as a conference paper at ICLR 2023
nM Therefore, even after multiplying by the v on the outside, which would mean the highest order term on the numerator could be as high as M- the nM in the denominator still dominates and the expression as whole will converge in probability. Since
Te [p 9-' (a)
was a sum of expressions of that form, and they all converge to probability, we get the result we desired, which is that
vn -1"" 0 in probability
T
Lemma The defined variable 72 = (v Xx is strictly greater than zero, namely 72 >>>00 0.
Proof.
Note that the covariance matrix can be written as Exx = E[(( - t)( t (recall t is the mean vector for X). So we have
2 EK (x) ()(X (V) (a)(t)
= (((V5.. (a)(t)) (X -)-
where we note that s == (V) - t) is a real-valued random variable, so ST = s. Thus prove 7² > we simply have show that for some value of X that occurs with non-zero probability. Now,
0g.
(X)) - +//
Oxyt!
x==
0gth (a) Oxryy
Σ
(X - (//)
stale-action pains (g*
possible nexl states q²
So let sf. Estates q²
0g. (a) ON
(X5² ). then = Estatte action pairs (a' 8/-
x
could
our two distinct values of si to find two distinct values of s. Both cannot be 0, done.
Now, the value of X is characterized by picking one j s.t. X = 1, and setting all other X = for 1 # j. This means that to find two different values of some sg. we just need to find states ,',4', g' and an action b such that the derivatives (n."(a)
x and (g."
8x x are distinct. Then
setting X = would yield a different value of sJ from setting X = So long as the events X = 1 and x, = both have nonzero probability, we would be done.
In order to show that such states q q', g' and such an action exist, we must introduce vectors A that will effectively keep track of each state's contribution towards g. (a)[t) at the nth step of the risk backpropagation. First, define the -V matrix P% [x] for n 0, 1,..., m such that
if = j and q² is unsafe and observed if / j and q² unsafe and observed otherwise
(P*
Den
20

---

Under review as a conference paper at ICLR 2023
where where dnn arg miny (b). Define P Pm [x] as
if i j and q' is unsafe and observed
(P (Pn-1(x))ss
0 xyy
if + j and q¹ is unsafe and observed otherwise
Then the P [x] represent the transition probabilities used in the calculation of ge (a) Specifi-
cally, we have that
9 [x] is the kth entry of the vector (xt-1xx)))..Pxxxx(1/ for < m
Ok (a)
xx] is the kth entry the vector [x]] (P..)))))))))..(((((())))
So the risk at current state qr, (a (t), is the eth entry of the vector
(tt)
where 9d is the vector with entries (g")x 1 (1* is observed and unsafe). We can now define the vectors n for n S m by
AP (P- (P)(P ((())).(PP(())))))
if o' is safely reachable from of in exactly steps otherwise
Where in this case state qrn is defined be safely reachable from the current state 800 q" in exactly n steps
- for actions bao and there are states qet ("' , such that each "0,
dag == arg min, R. Rm-k-1(b) (b) determined by the agent's expected safest policy, and the states ""...""...."""" " are safe (note that n'n can still be unsafe)
The purpose of these A is just restrict attention the states step of the backpropagation that actually influence ge (a)( It is easy to see that
(Pmm (t)
-. (a] for every 0, 1,..,,
(17)
Now we will be able to argue that if g (a)[t] is not equal to 1, there are states o' g o and
an action such that ey and ti are both non-zero (so there is positive probability of the events
X = 1 and X = 1) and such that (g] ang
0g" D2y
x=t
So assume that g.th (a) is not equal O or Let nn be the largest index such that A no contains an steps
entry (Ate), ) that is equal to 0 and such that gf is safely reachable from dº in exactly so (Ate); is 0 that came from (P [t)) (P))
Since (a)[t] (at qº0 gr1 ...., ,..,.... qmm--n of be path along which q is safely reachable. Then let q² -0-1 -o and For brevity, write b' bam-np-1 -mp-
is not
no m. and since of is safely reachable in m no steps, let de
have that is safe, and ty mo
Now since of is safely reachable in m (no 1) steps, (Ano+1); cannot be equal (since no was maximal), so there must be some state q¹ such that ty 0 and Aç > O, (in order for the term A to contribute some positive value to A,ot)). Finally, let be the probability of safely entering q² m (no + 1) steps (i.e., the sum over paths that safely reach q¹ the probability of taking that path by choosing the actions specified by the agent's 's expected safest policy). Then by the chain rule,
(a) duly
A1 =p ( 1x Aço + + ((Pho-ix))--. (P((x)))))
0
Ox
x=t
((P'g (1))-1)))
since clearly
cannot be negative. On the other hand,
21

---

Under review as a conference paper at ICLR 2023
09"
1 (Amo)) ig (P)--1x)))..(((((()))))))
-p
duy xx=
((P no (
= 1 110++ 16
0
Oxy
since only one of ty and (()-)111))--(((((01))
can nonzero if increasing the value of tit x=t
could increase the value (Ata); = ((P mg
[t)).. (P(tt)))) from 0 to greater than O, then tj
must have been 0 since (())a-))t))...(((((t))))))))) is sum of products of values from t, all of which are non-negative.
Hence we have found states g', o,, de and action V/ such that the derivatives
8a." (a) 8x
and
0g (a) Ban
distinct. Hence the claim.
x=t
The only detail left note is that we assumed that _-th [a][t] is not either equal to 0 or 1. This assumption is reasonable to make, because if did not hold, then either our agent would be doomed to enter unsafe state within steps, there is chance of entering unsafe state within m steps, according to the agent's ' expected safest actions. Since what matters to us is how the agent manages risk, situations involving risk risk O are irrelevant.
22

---

Under review as a conference paper at ICLR 2023
F APPENDIX F. CONFIDENCE BOUND ON THE RISK
To estimate confidence bound on the risk, we appeal the Cantelli Inequality, which is a one- sided Chebychev bound (Cantelli 1929), and states that for a real-valued random variable R with we have
expectation [[] and variance ar [ for
Var R
Pr E + X) 2 1
Var[It] + X2
If we let C == 1 VarlM
[or[R]+X then rearranging we get that À
VarlRCC I-C
Thus for variable R
that represents some sort of risk, and for some value of C 1, we can say
P)
where P : [[) + V Varl In words, "there is at least C chance that the risk is most P." Alternatively, "we least % confident that the risk is most P."
23

---

Under review as a conference paper at ICLR 2023
G APPENDIX G
To understand what exactly R. (a) is an approximation of, consider instead calculating this risk using the true transition probabilities t5, We would get
0 = 1( is observed and unsafe)
(18)
if ge is observed and unsafe otherwise
(a): N
m++
=
f== r3
(19)
if q² is observed and unsafe
x+1 %
(20)
r r--1 min (a)) otherwise
Note that crucially still take the minimum risk action according to the agent's 's approximation (a). In this case, the term r. (a) is the true probability of entering an unsafe state after select- ing action in the agent's current state ge and thereafter selecting the actions that the agent currently believes will minimize the probability of entering an unsafe state over the horizon m. R (a) is the agent's ' approximation of r (a).
We will later justify the use of RTth (a) as an approximation of of (a), but for now us consider why it makes sense to define m-step risk as (a). This because the action that minimizes believed risk is the action that the agent would choose it was trying to behave as safely as possible, what I will call going into 'safety mode'. Consider the motivating example of pilot learning fly a remote control helicopter by incrementally expanding the set actions they feels safe taking. They start by generating just enough lift begin flying, then immediately land back down again. They repeat this process few times until they feel that they have a good understanding of how the helicopter responds this limited range of inputs. Then they take risk (by either flying bit higher, or attempting to move horizontally) and once again immediately land. As they repeat this process of taking small risks and landing to remain safe, they begin to expand their comfort zone. At some point after taking risk, they will feel comfortable just coming back to hovering position rather than landing, once they have become confident that they can hover safely. This suggests that a natural process for learning to operate in the face of risks is to repeatedly take small risks followed by going into safety mode until back in a confidently safe state. Thus, when calculating how risky an action is, it makes sense to consider the probability of entering unsafe state given that after the action the agent will enter safety mode. r. (a) does exactly this.
As mentioned carlier, the other reason for defining the risk r-th (a) in this way is that it makes it possi- ble for the agent to attempt to calculate the risk without having reason about the inter-dependency between the calculated risk and the agent's future actions. However, it does more than this. We will see in the next section that it in fact allows the agent to view RT (a) as (an approximation of) the expected value of a random variable for the believed risk, where we can also approximate the variance of that random variable, allowing for deeper reasoning about action-selection for Safe RL.
24


Reviews:
[{"review_id": "u2COOjvug1", "reviewer": "Reviewer_BCMN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies safe reinforcement learning (RL) where safety violation must be bounded during training. They propose a new approach that balances the trade-off between efficient progress in exploration and safety guarantee. Specifically, the proposed approach updates Dirichlet-Categorical models of the state transition probabilities that describes the agent's behavior within the environment via Bayesian inference. They then approximate the agent's belief in terms of risks. They provide theoretical guarantees on the convergence on as well as empirically showing the performance of the proposed approach.", "summary_of_the_review": "Though the problem setting and proposed method is interesting, I have several concerns as I listed in the Weakness. Hence, I recommend rejection for now.", "strength_and_weaknesses": "### Strength\n**Problem Settings.** The problem settings are important and interesting.\n\n**Theoretical Analysis** I think Theorem 3.1 is indeed a nice theorem though it is not clear to me how novel this theorem is.\n\n### Weakness\n**Paper writing.** First of all, the presentation of this paper should be improved in terms of notations. For example, $P$ or $Q$ are used in two meanings\n+ P: Transition probability (in Section 2) and lowest risk level (in Section 3.3)\n+ Q: finite set of states (in Section 2) and Q-function (in Section 3.4).\n\nSuch inconsistency makes this paper quite hard to follow. Also, the notations are far from standard as an RL paper, which gives the reader unnecessary and unessential burden. As an instance, I personally want the authors to avoid using $Q$ in other meanings than Q-function. State space is typically denoted as $S$ or $X$ in RL papers.\n\n**Related work.** It would be better to write related work section in more organized manner. The current related work section is just a list of relevant papers, and it is hard to follow the story of the related work.\n\n**Missing citations.** As a particularly relevant papers, I would like to list the following paper. I think it may be better to compare the authors' method with this existing work (the source code is released).\n- As, Yarden, et al. \"Constrained Policy Optimization via Bayesian World Models.\" International Conference on Learning Representations. 2021.\n\n**Experiments.**  I don't think the experiment has been fully conducted. The benchmark task is very simple (i.e., grid world and PacMan), and the baseline methods are vanilla Q-learning. There are a lot of existing safe RL algorithms or benchmark tasks, so I think the authors should have compared with recent ones. For example, representatives of the benchmark tasks for safe RL is Safety-Gym or MuJoCo, and the notable safe RL baselines are PPO-Lagrangian (Here are just examples). At least, I do not think that the authors' claims are supported by the current experiments.\n\n### Minor comments\n- Since Moldovan and Abbeel (2012) is on safe exploration without ergodicity assumption, position of its citation seems weird to me. It would be more reasonable to cite it in the end of the next sentence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SPuVk9a5hxz", "reviewer": "Reviewer_HHjA", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigates a well founded probabilistic approach to safe RL, with primary contributions pertaining to safe training. Transition between states in the MDP is (during learning) modelled explicitly using a Dirichlet-Categorical model. Given a prior on transition probabilities (which can be fully informative) then the RL agent updates the belief of transition probabilities based on observed transition using Bayesian inference. The authors further derive suitable approximations of the risk associated with selecting action a in state q, based on the probability of transition and the cost of bad states. With the reasonable assumption that the agent can observe unsafe states m steps away (as transitions in the MDP), the approach is evaluated in slippery Gridworld and slippery PacMan showing impressive results. The paper ends with a well suited discussion on what the consequences for a learning agent when (proper) probabilistic safety guarantees are imposed, e.g. when the prior of the MDP transitions is weak. ", "summary_of_the_review": "The presented approach is (as far as I know) well suited in the literature, well founded, well presented, a highly suitable approach to safe RL (for training and otherwise) (in the applicable context of the approach) and sufficient empirical evaluation with interesting results which also motivate the method.", "strength_and_weaknesses": "Strength:\n* A proper probabilistic treatment on prior, safety and risk for RL given a Markov Decision Process.\n* Tractable and useful approximations of expected risk (similar to traditional expected reward). Furthermore, a clear probabilistic safety requirement is used to bound the risk (using Cantelli Inequality), with a high interpretability.\n* An excellent analysis and discussion on the results, and what safe RL entails.\n* Good exposition of related works.\n* The approach can be added on top of other RL methods (with discrete state action spaces?).\n\nWeaknesses:\n* No comparison to any other approach to safe RL. (However, given how formally well founded the presented approach is, this is more a valuable addition than a necessity.)\n* Figure 2 (which I think is great and really important) can be improved:\n  * I suggest that you draw red and yellow boundaries around where respective areas are in (b)-(d).\n  * Make sure that all four figures have the same xtick and font size.\n  * When referring to the figure, (y,x) is used which was confusing at first. E.g. the stand-still state in (b) is referred to as (13,1). I suppose that what is used is (row, column), but for figures/images (x,y) is standard.\n  * For clarity, I suggest that you invert the yticks, without changing the figure, such that the starting location is (0,0). A $(-\\hat{y},\\hat{x})$ coordinate system for figures is only standard within UX/GUI/Graphics. It is better to use a $(\\hat{x},\\hat{y})$ coordinate system.\n  * I understand that it does not fit, but I would have liked to have this figure in the main paper.\n* The slippery PacMan experiment could be extended (different $P_\\text{max}$, two ghosts, a little bit larger).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "SERPdk8qQz", "reviewer": "Reviewer_Fqdj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces Risk-aware Cautious RL (RCRL) algorithm that allows agents to be trained and keep them safe. By keeping a Dirichlet-Categorical model for each environment state, PCRL approximates the risk associated with the agent’s behavior originating from local action selection. To evaluate the performance of the PCRL algorithm authors have conducted experiments at a small grid-world environment ($20 \\times 20$) and a simplified version of PacMan (one ghost, 2 foods). \n", "summary_of_the_review": "As aforementioned, the paper presented some interesting ideas but it has many weaknesses in its current version.", "strength_and_weaknesses": "**Strength**\n\n- The problem of keeping agents safe during the training is of high interest and challenge. RCRL is able to estimate the risk associated with the agent’s behavior.\n- A theoretical analysis is also provided.\n\n**Weaknesses**\n- The idea of keeping a separate  Dirichlet-Categorical model for each environment state is quite limited. The scalability of the RCLR algorithm is under question.\n- Despite its merits the novelty of the RCLR algorithm is limited.\n- Some parts of this work are hard to be followed by the reader, especially due to the notation.\n- Empirical analysis is limited as experiments have been conducted in a small grid-world environment and on an over-simplified version of PacMan. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Qpeo3z0cn", "reviewer": "Reviewer_G8rL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the safe reinforcement learning problem where safety constraint violations are bounded. Therefore, the agent's goal is to manage exploration and safety maintenance efficiently.\n\nTo achieve that, the authors propose a cautious RL scheme that uses Dirichlet-Categorical models of transition probabilities. They also empirically validate the different performance aspects of the proposed algorithm.", "summary_of_the_review": "This paper significantly overlaps with my current work, and I am very knowledgeable about most of the topics covered by the paper.", "strength_and_weaknesses": "**Strengths of paper:**\n1. The problem of safety constraint violations in RL is important and has many real-world applications.\n\n2. The authors empirically validated the performance of the proposed algorithm.\n\n**Weakness of paper:**\n1. The assumption of knowing which states are safe and which are unsafe (even locally) is very strong. \n\n2. The motivational example is unsuitable for the considered problem setup in the paper as the state space can be very large or even continuous for the robotic problem.\n\n3. The proposed approach will become computationally inefficient (or even infeasible) for a large number of states or actions (continuous state and action space).\n\n4. It is unclear why authors do not use tighter concentration results for variance like Bernstein inequality.\n\n5. No theoretical guarantee: No theoretical guarantee of how the proposed method will work compared to the optimal policy. \n\n\n**Question and other comments.** \n\nPlease address the above weaknesses. \n\nI have a few more questions/comments:\n1. What is the initial belief used for transition probabilities?\n\n2. Page 3, the line before Section 3: Change $0$ in $\\alpha_a^{i0}$ to something else as it gives the impression that transition is happening from state $i$ to state $0$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2Ofyvi_Qj8b", "reviewer": "Reviewer_qaTz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a risk-aware bayesian reinforcement learning method to tackle the problem of safe exploration. Specifically, the authors assume that the agent maintains a Dirichlet-Categorical model of the MDP, and propose a method to derive an approximate bound on the confidence that the risk is below a certain level. Experiments in tabular environments demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed method is theoretically sound. However, the novelty of the proposed method is unclear to me, and the experiments are insufficient.", "strength_and_weaknesses": "Strengths:\n\n1. The problem of safe exploration is important. \n\n2. The proposed method is theoretically sound. The authors leverage a method from [1] to approximate the expected value and variance of the agent’s belief about the risk, and prove its convergence. \n\nWeaknesses:\n\n1. The novelty of the proposed method is unclear to me. Bayesian RL and risk-aware safe exploration have been widely studied in previous work. The authors may want to provide a detailed discussion on the novelty of the proposed method.\n\n2. The experiments are insufficient. First, some important baselines are missing. The authors may want to compare to more baselines. Second, it would be more convincing if the authors could evaluate their proposed method on continuous control tasks, such as Mujoco [1].  \n\n[1] Todorov et al. \"Mujoco: A physics engine for model-based control.\" international conference on intelligent robots and systems. IEEE, 2012.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject