Safe Policy Optimization with Local Generalized Linear Function Approximations
Authors: Akifumi Wachi
Yunyue Wei
Yanan Sui
Venue: NeurIPS 2021 Poster (2021)
Keywords: Safe Reinforcement Learning
Constrained Markov Decision Process

Abstract: Safe exploration is a key to applying reinforcement learning (RL) in safety-critical systems. Existing safe exploration methods guaranteed safety under the assumption of regularity, and it has been difficult to apply them to large-scale real problems. We propose a novel algorithm, SPO-LF, that optimizes an agent's policy while learning the relation between a locally available feature obtained by sensors and environmental reward/safety using generalized linear function approximations. We provide theoretical guarantees on its safety and optimality. We experimentally show that our algorithm is 1) more efficient in terms of sample complexity and computational cost and 2) more applicable to large-scale problems than previous safe RL methods with theoretical guarantees, and 3) comparably sample-efficient and safer compared with existing advanced deep RL methods with safety constraints.

Safe Policy Optimization with Local Generalized Linear Function Approximations
Akifumi Wachi IBM Research
Yunyue Wei
Tsinghua University
aki ikifumi. wachi @ibm. com
weiyy200mails. tsinghua, tsinghua.edu.com eeu.ghua.edu.cr edu.
Yanan Sui
Tsinghua University ysui ssui@tsinghua. edu. cn
Abstract
Safe exploration is a key to applying reinforcement learning (RL) in safety-critical systems. Existing safe exploration methods guaranteed safety under the assumption regularity, and it has been difficult to apply them to large-scale real problems. We propose novel algorithm, SPO-LF, that optimizes an agent's policy while learning the relation between a locally available feature obtained by sensors and environmental reward/safety using generalized linear function approximations. We provide theoretical guarantees on its safety and optimality. We experimentally show that our algorithm 1) more efficient in terms of sample complexity and com- putational cost and 2) more applicable to large-scale problems than previous safe RL methods with theoretical guarantees, and 3) comparably sample-efficte and safer compared with existing advanced deep RI. methods with safety constraints.
Introduction
Applying reinforcement learning (RL) to applications with unknown safety constraints is challenging as RL inherently exploratory process and requires agents to learn the whole environment first. Though there has been a surge of attempts to incorporate safety in RL [3, 6, 11, 16), it is essentially difficult for most of the algorithms to guarantee safety, especially in the exploration phase. If single mistake could lead to catastrophic failures, conventional algorithms cannot be directly applied.
To guarantee safety while learning environment, safe exploration problems have been actively studied as a sub-field of safe RL [17]. A mainstream safc-exploration method is to utilize a Gaussian process (GP, [24]) under problem settings in which the agent observe only the safety function value of its current state. Since the agent cannot get any information on the neighboring states, is necessary assume desirable properties for the function. Specifically, Sui al. [31] assumed that a safety function Lipschitz continuous and has regularity that can be captured by appropriate kernel functions. However, these assumptions do not hold in many environments. For example, in the context of autonomous driving, the degree of safety varies steeply depending on the presence or absence of other vehicles and pedestrians. Also, previous work on GP-based safe exploration suffers from inconsistency between their theoretical guarantees and computational cost. Previous work has proved the completeness of the predicted safe region [34] and the optimality of the acquired policy [36], after a large number of samples. However, the computational cost of GP is known to be large. Hence, it is hard achieve theoretically guaranteed performance in a large environment.
Based on the fact that robots are equipped with sensors, more reasonable to formulate the problem assuming that robots obtain "feature vectors" for inferring the degree of safety of the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Non-smooth point
Eai
Non-smooth point
Safety function
Nid Dellece
Safety function
Safety
Feature
State
State
Feature Safety
Feature Saafftyy X
SSfeety
Safety X
Feature X. Safety X
(a) Existing safe exploration.
(h) Proposed safe exploration.
Figure 1: (a) Previous GP-based safe exploration under regularity assumptions, which fails to handle non-smooth changes. (b) Our proposed catture-based safe exploration. In this problem setting, an agent predicts safety function value while using feature vector obtained by far-sighted observations.
(observable) states of their neighbors. This problem formulation is realistic for real-world robots, and also allows us to deal with safety that changes steeply, which previous research could not handle (see Figure 1). Under this new problem setting, aim propose theoretically guaranteed and empirically efficient safe RL algorithm that is applicable to real, large-scale problems.
Related Work
RL with (generalized) linear function approximation. Function approximation, especially that based deep neural networks (DNNs), has been of the main reasons for the success of RL in recent years, and it has attracted more attention in both theory and practice [10, 32). Given the difficulty of theoretical analysis of DNNs, linear MDPs [19, 22] characterized by linear transition kernels and reward are frequently used due to their tractability. The main interest of the theoretical analysis in existing studies has been the efficiency of RL algorithms. In fact, under the assumption of linear MDPs, recent studies (e.g., Jin et al. [18], Yang and Wang (39]) have proposed algorithms with theoretical guarantee regarding sample efficiency. Wang al. [38] then extended Jin et al. [18] to generalized linear model (GLM) settings and presented practical and provably efficient algorithm under strictly weaker assumptions.
Safe RL. A constrained MDP (CMDP, [4]) is a popular model for formulating safe RL problems. In the CMDP formulation, agent needs maximize the expected cumulative reward while ensuring that the expected cumulative cost is less than a threshold. There has been a number of attempts solve CMDPs [26], represented by constrained policy optimization (CPO [2]), reward constrained policy optimization [33]. Lagrangian-bate based actor-critic [7. 9, 13], primal-dual policy optimization [23, 39], and interior-point policy optimization [21]. As recent, closely related work, Ding et al. [14] proposed an algorithm with theoretical guarantees regard to regret and constraint violation for CMDPs with linear structures of transition, reward, and safety. In the previous papers mentioned above, however, safety constraint is defined over a horizon; hence, they deal with less strict safety constraints than our paper that requires the agent satisfy safety constraints at every time step. There has been research aimed at guaranteeing safety at every time step in both state-less and stateful settings. For state-less settings, SAFHOPT [31] and SAFE-LUCB [5] were respectively proposed for Bayesian optimization problems and linear stochastic bandit problems. For stateful settings, Turchetta et [34] proposed GP-based SAFEMDP algorithm that extends SAFEOPT to stateful settings. Under the assumptions of the deterministic transition and regularity of safety, while Turchetta et al. [34] aimed to fully explore the safe region, Wachi et al. [37] proposed a GP-based algorithm for maximizing the cumulative reward under safety constraint.
Our Contributions
We model new class of safe exploration problems where features are locally available within limited sensor range. This is a realistic formulation for some real-world robots with sensors, and it allows us to get rid of the regularity and Lipschitz assumptions that previous safe exploration

---

algorithms relied on. We develop safe policy optimization with a local feature (SPO-LF), an RL algorithm that maximizes the cumulative reward under safety constraints while predicting reward and safery function values on the basis of local features via GL.Ms. We analyze the theoretical properties of SPO-LF and prove that an agent will achieve near-optimal policy with high probability while satisfying safety constraints at every time step. Finally, we evaluate our algorithm in two experiments. In the first experiment using Gym-Minn Miiiiiriirr [12], we show that our algorithm is 1) more efficient in terms of sample complexity and computational cost and 2) more applicable to large-scale problems GP-based methods. In the second experiment using Safety-Gym [25], we show that our algorithm achieves comparable performance in terms of the reward to several notable safe RL baselines (e.g., CPO (2]) without executing even a single unsafe action.
than previous, highly-theoreticn
2 Problem Formulation
We consider a safety-consttainee MDP, defined as tuple
M = (S,A, Hr r, o, b),
where S is a finite set of states {8}, Ais a finite set of actions {a). f: SxA-x S the deterministic state transition, and H E N is the finite horizon. Also, r S + [0. and g S [0, 1] are bounded reward and safety functions. We assume that both r and g are not known a priori.' At every time step t N, the agent must be in safe state. Specifically, for state st, the safety function value 9(s)) must be above a threshold h E (0, 1]; that is, the safety constraint is represented as g(st) 2 h.
In real applications, robot is equipped with sensors and can infer the values of reward safety functions by using such measurements as "hints." " To formulate this mechanism, M contains two additional functions. First, Φ S - Id is an priori unknown function for mapping a state 8 to d-dimensional feature vector, where Bd := {æ E Rd ||||| S. 1}. For convenience, let Φ, == ((s) denote the feature vector for a state Second, we define a function sb : S + S for returning a set of states within the sensor range from a certain state. Note that, while conventional methods for solving MDPs with linear function approximation assume the prior availability of the feature map for whole state space, we consider the problem where features are locally obtained upon observation.
Near-sighted and far-sighted observations. On the basis of the two functions & and ab. we model two types of observation: near-sighted observation and far-sighted observation. Near-sighted observation is associated with direct observations of reward and safety, which is commonly used in typical RI. settings including previous GP-based safe exploration studies. Using near-sighted observations, the agent obtains a set of environmental information that contains noisy observations of the reward and safety function values along with the feature vector for the current state. That is, the agent obtains 8r, &a,, Mr. yg ). where y/- and yg are noisy observations of reward and safety, which are represented as y, = r(s) + ni and y = g(s) + ni, with independent zero-mean noises n. and n²" Far-sighted observation is more related to feature vectors. In this observation scheme, an agent obtains features for (ss). ). For convenience, let V, C S denote the set of states that have been observed at least once until time t; that is, V,, == v (s)) U (b((s)) U...U U (s.s). needs to predict the reward and safety function values by improving the internal GI Gs on the basis of near-sighted observations and applying them the features obtained by far-sighted observations.
Intuitively, the agent
Goal. The quality of a policy π. S - A is evaluated in terms of the cumulative reward achieved by the agent under the safety constraint. The problem is formally written as
maximize: (M,)) = E ETT((()4)) π
subject to: g(s,) 2 h.
There are a number of previous studies solutions to CMDPs, but most of them violate safety constraints during training (e.g.. (2]) In contrast, we require the agent to learn a policy in M without even single constraint violation. Without assumptions, it is intractable to solve the problem mentioned above. In the following, we list assumptions in this paper.
Generalized linear function approximation. reward and safety functions. In this paper, we are concerned with generalized linear models (GLMs) regarding the reward and safety functions that are characterized by feature vector.
The first assumption is on the structure of the
For simplicity, we assume that the state transition function, f is known priori. Note that it is not difficult extend our setting to priori unknown, f by referring Biyik et al. [8].
3

---

Assumption 1. The reward and safety functions are respectively represented using GLMs that are characterized by unknown coefficient vectors 0, 0 E R and fixed. strictly increasing link link functions Pr, Hr.Hg" : R -> [0, 1] such that
r(s) 1r ur 0:) and g(8) Mg (3(
Note that linear and logistic models are the special cases of GLM with (xx) = x and ju(x) i+e-- respectively. To predict the coefficients, we obtain maximum likelihood estimators (MLEs) for reward and safety functions as in Filippi et al. [15]. Let 0, and 0g be the MLEs for reward and safety, respectively. Here, O, and 0g are the unique solutions of the following equations.
(r(sr)- - (0 0r)) ds. =0 and Σ (o(o,) - 16 0g)) Φn, =0 T=0
(
r=0
The optimal solutions of the above equations can be found efficiently using such standard algorithms as Newton's method. We additionally make an assumption on the properties of the link functions.
Assumption 2.
Let denote either g- The link function ooo is twice differentiable, and the first and second order derivatives are respectively bounded by Lo and Mo. Also, the link function po satisfies E inf]jo, (all0-0gllssm 1.ll.s [o
[ 0o) O.
Prior knowledge on safety. Without prior information on the states known to be safe, an agent cannot guarantee safety at the first step. Hence, we make the following assumption.
Assumption 3. The agent is initially in a set of states, So S S, that is known to be safe a priori.
Sub-Gaussian noise. The final assumption is on the properties of the observation noise, which has been commonly made in previous work (e.g.. Abbasi -aakki- al. [1]). Assumption 4. Let be either
g. The noise ni is sub-Gaussian with fixed (positive) parameters { [ Fi-1] I c//00/2, where {Fi] increasing sequences of
ooi that is, for all t, we have E
sigma fields such that nę is Fi "-measurabll with ng [[-1==1 =0.
3 Preliminary: Defining True Safe Space
In this section, we ask and describe what the true safe space is. Since the agent observes noisy measurements reward and safety functions, it is impossible to know the exactly true reward and safety functions. This is particularly critical when dealing with safety; thus, aim to conservatively estimate the safety function value up to a certain confidence
R2z0- As discussed in Turchetta et al. [34], to identify the safety of a state, we incorporate the safety constraint, reachability, and returnability. Under our settings, however, an agent is allowed to observe the feature vector for multiple neighboring states; hence, the existing definition of the true safe space cannot be directly used. Hereinafter, we explain how to represent the true safe space in M.
First, we simply consider set of states that satisfy the safety constraint with margin of e. Given the set of states X, the following region can be identified as safe based on the far-sighted observation.
Yx(X) = X (s S Es' E X, & E vp(s') g(s) 6 2 h).
Second, we define reachability. The agent may not be able to reach state that is in Y-(-) due to unsafe states in the middle of the pathway. Given set X, the states in v that are reachable from X in one step are given by Yxeach(()
fs' E v Es E X, A: s' f(s, a)). Hence, an n-step reachable operator is given by Y (X) = Yreach (Y (Y (X)). where Y (X) = Yreach( (X). Finally, the set containing all the states that are reachable from X along an arbitrary long path in V defined as Vrench (X) lim,- (X)) (X). Finally, even after arriving at state with reachability, the agent may not have even single option from among the viable actions for moving
another safe state. Hence, before moving to state 8, we should consider whether or not there is at least one viable path from 8. The set of states from which the agent can return to a set X through another set of states X in one step is given by Yreum (X,, X) X (s E X 3a E A: f(s a) E X). Thus,
with y1 Y2umm (X,X) = Y Yretum (X,, X), an restep returnability operator given by Y (X.X) Yreum (X, y--1 (X, X)). The set containing all the states from which the agent can return to X along yn an arbitrary long path in X is defined as Yretum (X, (X, lim Yetum (X,X).

---

To guarantee safety, all the above requirements must be satisfied. As such, we define 2(-):
(x = Yx(X) n (X)) n Freurn (X)),,,
The safe set after repeating such expansions n-times is Zx(X) Ze 22 with Z: Ze (X). Finally, the true safe space, (X) is obtained by taking the limit in terms of n;; that is, lim Z (X).
Ze (X)
4 SPO-LF Algorithm
Our SPO-LF algorithm is outlined in Algo- rithm On the basis of near-sighted and
Algorithm SPO-LF with ETSE
far-sighted observations, an agent first caleu-
1: Co(s) +- (o) for all E So
2: loop
lates the upper and lower bounds of reward and safety function values and predicts safe re-
V. < V.-1 (((ss)(ss)
4: Qi(s) - - Q(())nc())
gion (lines 3 8). The agent then optimizes
5: h(s) (i)) mmx Ci(s)
its policy such that the cumulative reward is maximized while guaranteeing safety. For this purpose, the agent chooses the next state within the pessimistically identified safe space though this choice is basically based on the optimism
St f (
7:
St + { m(()2
X + 1-),, X (3)
9:
J (*) (4)
10:
if arg max seYnaaaa J Js(s) S XT then
in the face of uncertainty (OFU) principle in terms of reward (lines 9 11). However, we
11:
8:+1 t- mng EE (Yxan((a)))) J' (s)
12:
else
observed a problem in that this pure approach
13:
J (8x) + (5)
often causes the agent to get stuck in a reward-
14: while s1 # arg max Exx (s) do
poor state. As work-around for this propose,
we additionally propose an algorithm, called event-triggered safe expansion (ETSE) that en- courages the agent to focus on the exploration of safety necessary (lines 12 17). Note that the proofs of the lemmas are in Appendix C.
15:
8141 + arg max, max (({e)) (s)
16:
end while
17:
end
18:
end loop
4.1 Confidence Intervals
Since the reward and safety function values need to be "estimated" on the basis of the observations, we aim to achieve a provably safe and optimal RL algorithm by calculating their upper and lower bounds. Hence, we first derive theoretically guaranteed confidence intervals in terms of reward and safety. Since the accuracy of the predictions depends on the availability of the feature vector, we obtain the confidence bounds according to whether a state is inside or outside V,, Hereinafter, let the design matrix be W, = Σt= Φx, $T. Also, the weighted L2-norm of & associated with W- (Φ W 06)/22 Finally, the maximum and minimum singular values of matrix are denoted as Amaax (.) and (-)
is given by ||
Confidence intervals inside 1t, For all in L,, the agent has an observed feature vector, hence, the reward and safety function values can be estimated more accurately by leveraging it. On the basis of Theorem I in Li et al. [20], we have the following lemma.
Lemma 1. (Thm. in (20]). Let do > O be given and B. 3hads v /lo(3/06), Assume Xmin (W.) 2 31202Ma50 (d² + log 8, ). Then, with a probability of at least 1 8,, the MLE satisfies
o (s) - no ( 0.) S 3o 11 I|
The symbol o denotes either or gi that is, Lemma respectively holds for reward and safety.
Confidence intervals outside v,,, For outside L the agent does not know even the feature vector. To obtain the confidence intervals outside 1,, we should incorporate any possible feature vector; hence, confidence bounds are rather loose presented below.
Lemma 2. Let a denote either r or Assume Amin (W)) 2 512 - 22220?20 ogM3c (l + +og 5o Then, with a probability of al least do. for any 8 E \ V, we have
0 < o(s) I no (10, 1) + Bo Amax \max(W;" (W.
5

---

The next lemma is on the relation between the upper bound defined above and the threshold, h. Lemma 3. Assume Amin (W)) 2 512 o² M2c c- (d² + log 5. Then, with a probability at least 1 1- 6g we have My (g(100g))) - Bg Amax ( ( h.
Efficiency metric of exploration. Because 3, and Bg are constants independent from 8, the efficiency of the exploration of the reward and safety functions depends solely on I| within V and Amax ( outside vr, Hence, useful actions for exploring the reward also contribute the exploration of safety. This is an advantage in leveraging feature vectors in safe RL problems, and it leads to more efficient policy optimization compared with previous work such Wachi and Sui [36] that requires the agent to explore the two functions separately.
Assumption on min (W,). As discussed in Li al. [20], the assumption on Amin (Wa) is satisfied under mild conditions. However, since we consider safe exploration problem, it is extremely important to know how to ensure that the assumption on Amin (W;) is valid. This paper presents two methods; the first is to provide (typically small number of) data for initializing GLMs, and the second is to prepare sufficiently large Su and prohibit the agent from going outside of So, until the condition on Auin (W) holds. In experiment, simply took the first approach.
4.2 Prediction of Safe Space
We now define two types of predicted safe space where safety is identified either optimistically or pessimistically, as discussed in Turchetta al. [35]. First, we define the lower and upper bounds of safety. By Lemma I and 2, the confidence bounds on safety are represented as
{
s E Mr, ((0gg) + +,y Amax (W, (W;1)] otherwise.
[1 0g) + Agg ||
Qc(s)
(1)
We consider the intersection of Q2 up to iteration f, which is defined as ==(s)=Qo(s)) (C- where Co(s) [h, 20] for all s So- The lower and upper bounds on Cs(s) are denoted by 4, (s) == min (s) and uz(s) == max (s),, respectively.
Predicted pessimistic safe space. We now define the predicted pessimistic safe space. We first consider a set of states such that the safety constraint is satisfied with high probability. It is formulated using 4; that is,
ST = {s S ((()) 2 h.
The desired pessimistic safe space, X, is a subset of S; and also satisfies the reachability and returnability constraints; that is,
X = S. n (X,_, Froum ( (),)))
(2)
Predicted optimistic safe space. To optimize a policy, we need to incorporate the reward not only in X but also in region that contains all the states that may turn out to be safe even with small probability. We consider the set of states that may satisfy the safety constraint even with a small probability, which is represented
St : {s E (8) h..
Note that, by Lemmas 2 and 3, for all states in S' Vr, the upper and lower confidence bounds in terms of safety satisfy (s))) < h < (s).. Hence, all states in S Vr, are identified as pessimistically unsafe but optimistically safe. Similarly to the pessimistic safe region, again incorporate reachability and returnability, The resulting optimistic safe region given by
X St Yreech (xtx) Y Vrcum x,,t).
4.3 Safe Policy Optimization based on OFU Principle
As discussed in Section 4.1, efficiently explore reward and safety simultaneously by focusing on | and Amax (W,1). Hence, it turns out that we only need to balance exploration and

---

exploitation in terms of reward. For balancing 1) the exploration of reward and safety and 2) the exploitation of reward, we solve Bellman equation on the basis of the OFU principle:
(*, = max R()) + +s) where R(s):= pr, ($] 0,) ) + 1. Ilw if 8 E Vt: (4r (1 II) + A, (W.") otherwise.
(4)
=
Note that R(s) represents the upper confidence bound with regard to reward. The next state to visit should be pessimistically safe and reachable from 8, with one step; that is,
arg max sE(2, ()xxa((=)))
(s)...
Problem with this approach. We observed that the above algorithm often causes the agent to get stuck state with (relatively) small reward. The reason for this problem is as follows. When the state with the maximum J" is outside X,, the agent must execute the "second-best" action. The second-best solution often the one where the agent tries to take the stay action and then visit $441 then, the agent continues stay at the current state.
Event-triggeree safe expansion. To address the above problem, we propose an algorithm called event-triggered safe expansion (ETSE). We first define
= arg max J*)ss, sEYmal(=e))
meaning the state that the agent will visit if there no safety constraint. The ETSE algorithm
is triggered only when $ * 1). Intuitively, if is within X, the agent can simply execute the next optimal action visit $141: ; hence, the above problem arises only if 8 # nj is satisfied. This algorithm focuses on expanding the pessimistic safe region by visiting the states with large uncertainty.
J*())) max
[1 llw + + (s(+1)] Sx41EX,
On the basis of (5), the agent plans the trajectory so as to gain a lot of information, and the ETSE con- tinues until the agent arrives the state with the highest uncertainty. Once the ETSE finishes being applied, the agent aims to handle the exploration and exploitation dilemma by solving (4).
Theoretical Analysis
In this section, we present the theoretical results obtained for our SPO-LF. As preliminary for the results, let us define the time step t', which can be used as an indicator that the agent has sufficiently understood the models of reward and safety. We now present the following lemma regarding t".
Lemma 4. Let C1 and C2 be positive, universal constants. Also, let t" denote the smallest integer satisfying
{
BB
Amin (E)) O1 Vid C2V/log(65 max
with Σ
== E J. Then, for all > t", the following inequalities hold.
1+
t
1n, 0g) - u 0,) S and 1 I S Hog
T=2+1
Finally, we present two main theorems. The first theorem is on the satisfaction of the safety constraint, and the second on the near-optimality of the acquired policy.
Theorem 1. Assume that the noise of safety is
and Amin (W;) 2 512 M2 (df + log 6-1) holds. Suppose that the next state chosen from X, Then, the fol- lowing statement holds with probability of at least 1 dg:
VI 0, g(8r) h.
The proof is in Appendix E. The agent chooses the next state to visit within X (C ST), and St is characterized with 2(s). Since g(s) is greater than I(s) with probability of at least 5g. algorithm guarantees that the agent will execute only safe actions.
Unlike Turchetta et al. [34] or Wachi and Sui [36], this paper does not provide a theory on the completeness (i.e., convergence to the true safe region) of the predicted safe region. This is because our algorithm explores safety as result of balancing exploration and exploitation for reward, which contributes to its -ffpleeefficiency
7

---

Theorem 2. Ser =====
= t² + Zooo) and
(++
ev
Vmax
8g + (,(66)))
2dd
where Vmax is the upper bound of the value function. Assume that the noise of reward and safety is Sab-Gaussian and H > T Then, with high probability,
( 2 (*+)) =V:
-ie. the algorithm is ev-close to the optimal policy-Jee for all but T time steps while guaranteeing
safety with probability of at least 1 8g-
The proof is in Appendix E. An outline of the proof is as follows. At each iteration, the agent encounters out of the following and the agent can execute an action to purely optimize its policy. In the other event, 8/41 # is satisfied, and the agent focuses on exploring the safe regions. The worst case scenario for the sample efficiency is when the second event continues to occur; that is, the agent is required to explore the whole safe space. However, after a sufficiently large number of time steps greater than 7"* the agent can achieve an sy-close policy by deciding the next action on the basis of the OFU principle in fully- exlly-explored safe region. Note that, in the case of a short horizon, the agent cannot achieve the near-optimal policy; hence, we assume H > T
events. In one event, the ETSE algorithm is not triggered,
6 Experiments
We evaluate the performance of our SPO-LF in two experiments. In the first experiment using Gym- MiniGrid [12], we show that our algorithm is better than existing ones backed by theory including [36] in terms of sample efficiency and scalability. In the second experiment based on Safety-Gym [25], we compare our algorithm with advanced deep RL algorithms incorporating constraints. We also demonstrate the effectiveness of the ETSE algorithm. For future research, our code is open-sourced."
6.1 Grid World
Settings. We constructed simulation environment based on Gym-MiniGrid [12]. We considered a 25 x 25 grid in which each grid was associated with randomly generated feature vector with the dimension = Coefficients for reward and safety were also randomly generated; thus, true reward and safety function values were allocated to each grid along with feature vector. At every time step, an agent chose the next action from five candidates: stay, up, right, down, and left. The agent predicted the reward and safety functions while learning the relationship between the feature vector and the two functions with GLMs. In this simulation, we allowed the agent to 1) observe the feature vector in the 7 7 neighboring grids that in the front of the agent and 2) obtain the feature vector and noisy measurements of the reward and safety values at the current state. As prior information, the agent received 10 samples of (Ø, g). initialize the internal GLM model regarding safety. We considered a linear model with (x) = x for both reward and safety. Finally, we set 7 = 0.999, 5, dg 0.05, and h 0.1, and optimized policy with policy iteration.
We compared our SPO-LF with the following six baselines. The first is called ORACLE, and has non-exploratory agent who knows the true reward and safety functions priori. The second is called UNSAFE GLM, which a safety-agnostic agent maximizes cumulative reward on the basis of GLMs. The third is called RANDOM, in which agent randomly chooses the next action. The last three baselines are all stepwise approaches as in Wachi and Sui [36]. The fourth baseline is STEP SAFE GLM, in which agent first expands the safe region and then optimizes the cumulative reward in the certified safe region while learning the reward and safety function structures via GLMs. The fifth is GP (FEATURE), which is GP-based stepwise safe exploration and optimization where the inputs of GP are d-dimensional feature vectors. The final baseline is called GP (STATE), a stepwise approach proposed in Wachi and Sui [36] under the assumption that the reward and safety functions have regularity with regard to state. We used the reward and the number of unsafe actions as evaluation metries.
phttps://giteee http::///ggicuuu..oooatttttteee com/akiiumiii wachi wachi-4/sphhlf
8

---

1.0
1.0
0.8
0.8
Rowerd
0.8
SPO-L wis ETSE
SPOLL WTEE
0.8
Ureate GLM
SPOLE 50/
SPD-1 p0V SPO-U (O/7
Random
GP (Nstar)
SPO-T FoV 10
0.2
(oste)
022
crour crou- 13
Onacke
Orcde
0.0
0
100
200
360
400
0.0
0
100
200 Time step
300
400
Time step
(a)
(b)
Figure 2: (a) Comparison of performance of SPO-LF and baselines. Standard error is shown as shaded area. (b) Comparison of performance of the SPO-LF with different sizes of field of view.
Results. Figure 2(a) compares the performance with regard to the cumulative reward of each algorithm. SPO-LF, UNSAFE GLM, STEP SAFE GLM, and GP (FEATURE) achieved near-optimal policy. The convergence of SPO-
Computation time [sac]
+00
LF was faster than STEP SAFE GLM and GP (FEATURE) but slower than UNSAFE GLM. This is because SPO-LF could not always execute the optimal action for maximizing the cumulative reward to guarantee safety. Also, without ETSE, the agent often got stuck in state with a non- optimal reward. Figure 2(b) compares the performance of SPO-LF with different sizes of field of view (FOV) from 1 1 1 to 13 x 13. For a larger FOV, the convergence of the algorithm got faster. Given that SPO-LF (FOV x 1) corresponding to previous work (e.g.. [34, 36)) dealt with only near-sighted observations, it is useful to incorporate far-sighted observations as in our new framework. Figure 3 compares the computational time for different environment sizes. SPO-LF is more scalable and applicable to large-scale problems given that the GP-based algorithms did not run in a reasonable amount of time for large environments such as that with 150 x 150 states. In the GLM- methods, most of the computational cost when obtaining confidence bounds is for calculating Amaa (W. This computational complexity is (dd), which does neither depend on the number of states nor samples. Finally, the average number of unsafe actions was 3.4 for UNSAFE GLM, 49.0 for RANDOM, and 0.0 for the other methods.
SPONF
agoette (S)
200
100
10000 20000
30000
Number of states
Figure 3: Computational time of GLM- based SPO-LF and GP-based algo- rithm (i.e., Wachi and Sui [36]).
6.2 Safety-Gym
Settings. The second experiment was based on Safety-Gym [25] with continuous state and action spaces. The objective of this experiment was compare our SPO-LF with several notable safe RL algorithms including CPO [2], PPO-Lagrangian, and TRPO-Lagrangian. The last two baselines are ones that introduce that Lagrangian method into a safety-agnoste algorithm (i.e., PPO [29] and TRPO [28]) to incorporate constraints (for more details, see Section 5 in Ray et al. [25]). In this experiment, as shown Figure 4(a), one goal (in green) and multiple hazards (in blue) were randomly placed, and the agent (in red) needed to reach the goal without visiting hazards. For encouraging the agent to reach the goal, we set two kinds of reward: 1) a reward of 1.0 when arriving at the goal and 2) a reward of 0.01 Agual when getting closer to the goal, where A Agaul is the amount of decrease in distance from the goal. To implement SPO-LF, we discretized the environment into 50 x 50 grids and used a computationally inexpensive policy iteration algorithm proposed in Scherrer et al. [27] for policy optimization. The feature vector ø was constructed by LiDAR observations (i.e., the distance from the goal hazards). Since reward and safety function values vary steeply according to the existence of the goal or hazards, we use the sigmoid function μ(z) = i+e-= as the link function. As for implementing the three baselines, our experiments largely depended on the "safety- starter-agent" repository (https: : //t///gtttt//tt////////// com/ hwtp////aaa/atttyy..eaaao starter- agents git). For

---

of see
80
PPO-Lagiann Logiangin TRRC- agganiin
GPO
S7O4F
Lagrongian
005000
Num actiona
Num
action
(a) Environment
(h) Reward
Number of unsafe actions
Figure 4: Experiment with Safety- Gymm (a) Example screen capture from simulation environment with goal and hazards. (b) Average reward over episodes. (c) Number of unsafe actions per cpisode. Note that our SPO-LF did not execute even single unsafe action.
the initialization, we provided all the agents with reward and safety samples as prior information. For more detailed experimental settings, see Appendix F.
Results. Figures 4(b) and 4(c) show the experimental results. We used the reward and the number of unsafe actions as evaluation metrics and calculated the mean and standard error after running each method 200 times. As shown Figure 4(b), our SPO-LF exhibited comparable performance the baselines with regard to the sample efficiency. However, our algorithm did not execute even a single unsafe action, while the other three methods optimized their policies while making mistakes.
Conclusion
We formulated new problem characterized by constrccon MDPs with local feature and then proposed the SPO-LF algorithm for safely optimizing policy an priori unknown environment. SPO-LF efficiently and safely optimizes policy by leveraging feature information while expanding the safe space as necessary by using the ETSÉ algorithm. Theoretical analysis showed that our algorithm obtains near-optimal policy while guaranteeing safety, with a high probability. Our experiments showed that our algorithm 1) achieved better efficiency and scalability than previous safe exploration methods with theoretical guarantees and 2) behaved more safely than existing ad advanced deep RL methods with constraints.
We consider that our proposed method compensates for the shortcomings of safe RL methods with theoretical guarantees (e.g.. [34], [36]) and advanced deep RL methods (e.g., [2]), which potentially sets out research direction to bridge the gap between two distinct methods. However, there are several neglected problems we as the community should address future work. One of the biggest problems is that our proposed algorithm (that is more scalable than tighly-theoretical ones) is still far from practical in real applications with continuous state and action spaces. It would be an interesting direction to develop scalable algorithm as advanced deep RL algorithms while maintaining theoretical safety guarantee.
Finally, we believe that safety is an essential requirement for applying RL in many real problems and have found any negative societal impact of algorithm. However, need remain that any sequential decision-mis -akinng algorithms are vulnerable to misuse and ours is no exception.
Acknowledgments
We deeply appreciate the anonymous reviewers for constructive comments. This work is partially funded by Tsinghua GuoQiang Research Institute.
References
[1] Abbasi-yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. In Neural Information Processing Systems (NeurlPS).
10

---

[2] Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In International Conference on Machine Learning (ICML).
[3] Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topcu, U. (2018). Safe reinforcement learning via shielding. In AAAI Conference on Artificial Intelligence (AAAI).
[4] Altman, (1999). Constrained Markov decision processes, volume CRC Press.
[5] Amani, S., Alizadeh, M., and Thrampoulidis, C. (2019). Linear stochastic bandits under safety constraints. In Neural Information Processing Systems (NeurIPS).
[6] Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. (2017). Safe model-based reinforee- ment learning with stability guarantees. In Neural Information Processing Systems (NeurIPS).
[7] Bhatnagar, S. and Lakshmanan, K. (2012). An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3):688-708.
[8] Biyik, E., Margoliash, Alimo, S. R., and Sadigh, D. (2019). Efficient and safe exploration in deterministic markov decision processes with unknown transition models. In American Control Conference (ACC). IEEE.
[9] Borkar, V. S. (2005). An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54(3):207-213.
[10] Bradtke, S. and Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning, Machine learning, 22(1):33-57.
[11] Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. (2019). End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI Conference Artificial Intelligence (AAAI).
[12] Chevalier-Boisverr, M., Willems, L., and Pal, S. (2018). Minimalistic gridworld environment for OpenAl gym. https: /ttps://giiie //g:://iitttt com/ www.///wmmm//////mommo/mmm/ gym cym--inigrrd.
[13] Chow, Y.. Ghavamzadeh, M., Janson, L., and Pavone, M. (2017). Risk-constrainee reinforcement learning with percentile risk criteria. Journal of Machine Learning Research (JMLR), 18(1):6070- 6120.
[14] Ding, D., Wei, X., Yang, Z., Wang, Z., and Jovanovic, M. (2021). Provably efficient safe explo- ration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics (AISTAT).
[15] Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. In Neural Information Processing Systems (NeurIPS).
[16] Fulton, N. and Platzer, A. (2018). Safe reinforcement learning via formal methods: Toward safe control through proof and learning, In AAAI Conference on Artificial Intelligence (AAAI).
[17] Garcia, J. and Fernández, F. (2015), A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1):1437-1480.
[18] Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory (COLT).
[19] Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandee approach to personalized news article recommendation. In International Conference World Wide Web (WWW).
[20] Li, L., Lu, Y., and Zhou, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning (ICML).
[21] Liu, Y., Ding, J., and Liu, (2020). Ipo: Interior-point policy optimization under constraints. In AAAI Conference Artificial Intelligence (AAAI).

---

[22] Osband, I., Van Roy, B., and Wen, Z. (2016). Generalization and exploration via randomized value functions. In International Conference on Machine Learning (ICML).
[23] Paternain, S., Calvo-Fullana, M., Chamon, L. F., and Ribeiro, A. (2019). Safe policies for reinforcement learning via primal-dual methods. arXiv preprint arXiv:
[24] Rasmussen, C. E. (2004). Gaussian processes in machine learning. In Advanced Lectures Machine Learning, pages 63-71. Springer.
[25] Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking safe exploration in deep reinforce- ment learning. OpenAI.
[26] Satija, H.. Amortila, P., and Pineau, (2020). Constrained markov decision processes via backward value functions. In International Conference on Machine Learning (ICML).
[27] Scherrer, B., Ghavamzadeh, M., Gabillon, V., and Geist, M. (2012). Approximate modified policy iteration. In International Conference on Machine Learning (ICML).
[28] Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International Conference Machine Learning (ICML).
[29] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
[30] Strehl, A. L. and Littman, M. L. (2005). A theoretical analysis of model-based interval estimation. In International Conference on Machine Learning (ICML).
[31] Sui, Y., Gotovos, A., Burdick, W., and Krause, A. (2015). Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning (ICML).
[32] Sutton, R. S. (1988). Learning predict by the methods of temporal differences. Machine learning, 3(1):9-44.
[33] Tessler, C., Mankowitz, D. J., and Mannor, S. (2018). Reward constrained policy optimization. arXiv preprint urXiv:1805.11074
[34] Turchetta, M., Berkenkamp, F., and Krause, A. (2016). Safe exploration in finite Markov decision processes with Gaussian processes. In Neural Information Processing Systems (NeurIPS).
[35] Turchetta, M., Berkenkamp, F., and Krause, A. (2019). Safe exploration for interactive machine learning. In Nearul Information Processing Systems (NeurIPS).
[36] Wachi, A. and Sui, Y. (2020). Safe reinforcement learning in constrained Markov decision processes. In International Conference Machine Learning (ICML).
[37] Wachi, A., Sui, Y., Yue, Y., and Ono, M. (2018). Safe exploration and optimization of constrained MDPs using Gaussian processes. In AAAI Conference on Artificial Intelligence (AAAI).
[38] Wang. Y., Wang, R., Du, S. S., and Krishnamurthy, A. (2019). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representation (ICLR).
[39] Yang, L. and Wang, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernlss and regret bound. In International Conference on Machine Learning (ICML).
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see the second paragraph in Conclusion.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see the third paragraph in Conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes] Assumptions are listed in Section 2.
(b) Did you include complete proofs of all theoretical results? [Yes] Proof sketches are in main paper and full proofs are in the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We attached the source-code as supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
The source code is attached as a supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "m5JmUwcBXz", "reviewer": "Reviewer_E6K1", "summary": "This paper studies the problem of safe exploration in reinforcement learning contexts. It develops a new algorithm which, by making several assumptions, can stay safe while exploring and achieves good sample efficiency. The proposed approach incorporates sensor measurements (\"far-sighted observations\") and estimates of safety based on those observations, as well as the idea of an optimistic and pessimistic safe set, to stay safe during exploration. It also introduces a novel way to ensure that the algorithm does not get stuck in a safe area by detecting this case and more aggressively expanding the set of considered exploration actions.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nThe approach is novel. The paper draws on unexpected sources to develop new theoretical guarantees on the algorithm's performance. ESTE is a clever workaround to getting stuck during exploration.\n\n# Significance\nA number of techniques have tackled this problem in the past with varying degrees of success (Sui et al. 2018, Wachi and Sui 2020, Berkenkamp et al. 2016 are a few notable examples). This paper is significant in the field because it seems to provide a comprehensive framework for dealing with safe RL problems - as long as the problems satisfy this (fairly relaxed) set of assumptions, the technique should apply broadly - even if the safety function has discontinuities, which is common in real world scenarios.\n\n# Quality\nThis paper builds up a new safe exploration technique with logical assumptions and clearly shows the approach's effectiveness. So much of the paper hinges on these assumptions, so it is good that they are stated clearly.\n\nIt's unclear why the approach was not compared directly to the algorithms used in prior work. Many of these works even have open-source code available, so the experiments could have been true head-to-head rather than \"stepwise approaches as in [36]\" (line 310) which implies a re-implementation, or a similar-but-not-identical algorithm.\n\nFrom my reading of the paper, it seems that the ESTE version should perform not worse than the non-ESTE version in all cases. However, in Figure 2, the ESTE version of the algorithm did seem a bit slower than the non-ESTE version, and it's unclear why this would be the case. Perhaps the difference is random deviation, however, since the experiment is fully controlled, it should be possible to do a direct, deterministic, head-to-head comparison between ESTE and non-ESTE versions of the algorithm on identical problems. \n\nIt is odd that Algorithm 1 gives the next state directly, rather than the action to be taken. This makes it difficult to see how it would extend to a situation with non-deterministic transition functions.\n\nIs one of the baselines algorithmically equivalent to SPO-LF but with no sensor data? This would be an interesting comparison to make, to see how vital the sensor data is.\n\n# Clarity\nThe paper is written and organized superbly.\n\nMinor edits/questions:\n- L160: Should $Z_\\epsilon^n$ be just $Z^n$? or does $Z_\\epsilon(X) = Z(X, \\epsilon)$?\n- L204: this follows from $g(s) = ◇(s) \\geq h$? It would be nice for this to be made explicit\n- L204: this only holds for $s \\in S \\setminus \\Psi$, correct?\n- L58: resprcively", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The assumptions are clearly stated, and this gives some idea of the limitations of the algorithm. However, further limitations were not really discussed. Line 183 gives a worst-case scenario for sample efficiency. However, some other limitations are that this paper uses deterministic state transitions, and that it uses a discrete MDP.\n\nSome relevant questions:\n- Is there really no way for an agent to get \"stuck\" or accidentally visit an unsafe state as long as the assumptions hold?\n- Are these assumptions valid for most robotics problems? Are there some situations that are more likely to invalidate the assumptions than others?\n- How would the technique deal with limited sensor information (finite range, obstacles, a LIDAR that was not 360 degrees, etc.)?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "HrfN32OVVEt", "reviewer": "Reviewer_N2r5", "summary": "This paper proposes SPO-LF, a safe learning algorithm for sequential decision making. The key idea is to predict a pessimistic safe space and an optimistic safe space based on the confidence intervals, and constrains the exploration in the safe space. The proposed algorithm is evaluated in the mini-grid and the safety-gym environments, compared with several baselines, and claimed to be able to handle large-scale problems, and behave more safely than the CMDP-based deep RL algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Safe exploration is critical to apply reinforcement learning in real-world systems, such as robotics and self-driving cars. This paper tries to address this important challenge by extending the dynamic programming approach to ensure zero unsafe actions during training while still learning near-optimal policies.\n\nMy major concern of this paper is its unsupported claims that \"our algorithm is more applicable to large-scale problems\" and \"behaves more safely than existing advanced deep RL methods with constraints\". It appears to me that SPO-LF only works for environments with discrete state and action spaces. Discretizing the state and the action space, as did in the safety-gym evaluation, is not scalable to large scale continuous control problems. For this reason, comparisons with CPO, PPO-Lagrangian or TRPO-Lagrangian are not fair because the baseline algorithms are solving a much more challenging continuous-space problem.\n\nHere are a few clarification questions:\n1) Equations below line 121. Why is it necessary to have \\phi_{s_\\tau} right before \"=\" on the left hand side of equations?\n2) It would be great to give some intuitions of Assumption 2 \"The link function satisfies ...\" What's its physical meaning? How strong is this assumption?\n3) I had some trouble understanding line 223-225. What is the physical meaning of C_t?\n4) The ETSE seems to be a heuristic for exploration whose goal is to reduce uncertainty. I could misunderstand here, but ETSE can potentially make the entire algorithm unsafe because it needs to explore the states with large uncertainty. If this is the case, I am surprised to see that the number of unsafe actions is still zero for both the grid world and safety-gym evaluations. Did I miss anything here?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not find where the limitations are discussed. Are there any limitations of the proposed method? \nI did not find any potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "4seIQh5ND0y", "reviewer": "Reviewer_XEQB", "summary": "This work proposes a novel safe RL algorithm based on locally available far-sighted features based on GLM assumption.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of shared representations for safety/reward is interesting.\nThe paper is well written.\n\nMajor concerns:\n1) in line 155, Y_return(X,bar{X}) is defined by bar{X} \\cup ---.  Is it obvious that from state in bar{X} to bar{X} is feasible?  is there any action choice that makes the agent stays in the same state?  also in line 159, is Y_epsilon(X) needed in the definition of Z? (removing this term does not seem to affect the set)\n\n\nMinor concerns:\n1) in line 89, the definition of psi should be psi:S -> 2^S ?\n2) this work considers finite sets of state and action, and also assumes that the transition is known; therefore, saying that existing GP-based method is not scalable while the proposed work is scalable is a bit misleading; what if the work is extended to unknown transition?\n\n-----------\nReview updated after the initial response", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The reviewer wanted to see the discussions on discretization accuracy vs the usefulness of local feature for continuous control problems.  It would be possible to discuss limitations more.\n\nSocial impact is discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer believes there is no significant ethical concern in this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance, two strongly. The reviewers recognize the importance of the topic (safe exploration in reinforcement learning) and the significance of the submission within this specific area. Any concerns were addressed clearly by the authors, and so the AC recommends acceptance. 

Note: There appears to be some confusion surrounding a claim "our algorithm is more applicable to large-scale problems than previous safe RL methods with theoretical guarantees". The point of confusion appears to be what constitutes a previous safe RL method with theoretical guarantees. If CPO is included (it is safety-inspired and has some theoretical guarantees [though not strong guarantees of safety]) then this may not be true. However, when comparing to algorithms with very strong (high-confidence) safety guarantees, the author's claim is clearly true. The AC hopes that the authors do not dismiss the reviewers concerns about this line, but rather aim to clarify this line to ensure that other readers do not have the same misconception about the authors' intent with the statement.
Decision: Accept (Poster)