Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation
Authors: Yannick Hogewind
Thiago D. Simão
Tal Kachman
Nils Jansen
Venue: ICLR 2023 poster (2023)
Keywords: safety
reinforcement learning
safe reinforcement learning
constrained Markov decision process
partially observable Markov decision process
MDP
POMDP

Abstract: We address the problem of safe reinforcement learning from pixel observations. Inherent challenges in such settings are (1) a trade-off between reward optimization and adhering to safety constraints, (2) partial observability, and (3) high-dimensional observations. We formalize the problem in a constrained, partially observable Markov decision process framework, where an agent obtains distinct reward and safety signals. To address the curse of dimensionality, we employ a novel safety critic using the stochastic latent actor-critic (SLAC) approach. The latent variable model predicts rewards and safety violations, and we use the safety critic to train safe policies. Using well-known benchmark environments, we demonstrate competitive performance over existing approaches regarding computational requirements, final reward return, and satisfying the safety constraints. 

Published as a conference paper at ICLR 2023
SAFE REINFORCEMENT LEARNING FROM PIXELS USING A STOCHASTIC LATENT REPRESENTATION
Yannick Hogewind, Thiago D. Simão, Tal Kachman & Nils Jansen Radboud University, Nijmegen
fyannick ck.n hogewind, thiago. s simao, nils. jansen nl
tal. tal.kaahhaa .kachman@donders ru.nl
ABSTRACT
We address the problem of safe reinforcement learning from pixel observations. Inherent challenges in such settings are (1) a trade-off between reward optimiza- tion and adhering to safety constraints, (2) partial observability, and (3) high- dimensional observations. We formalize the problem in constrained, partially observable Markov decision process framework, where an agent obtains distinct reward and safety signals. To address the curse of dimensionality, we employ a novel safety critic using the stochastic latent actor-critic (SLAC) approach. The latent variable model predicts rewards and safety violations, and we use the safety critic to train safe policies. Using well-known benchmark environments, we demonstrate competitive performance over existing approaches regarding com- putational requirements, final reward return, and satisfying the safety constraints.
INTRODUCTION
As reinforcement learning (RL) algorithms are increasingly applied in the real-worid (Mnih et al., 2015; Jumper et al., 2021; Fu et al., 2021), their safety becomes ever more important with the increase of both model complexity, and uncertainty. Considerable effort has been made to increase the safety of RL (Liu et al., 2021). However, major challenges remain that prevent the deployment of RL in the real-world (Dulac-Arnold et al., 2021). Most approaches to safe RL are limited to fully observable settings, neglecting issues such as noisy or imprecise sensors. Moreover, realistic environments exhibit high-dimensional observation spaces and are largely out of reach for the state- of-the-art. In this work, we present an effective safe RL approach that handles partial observability with sigh-dimensional observation spaces in the form of pixel observations.
In tandem with prior work, we formalize the safety requirements using a constrained Markov de- cision process (CMDP; Altman, 1999). The objective is to learn a policy that maximizes reward while constraining the expected return of a scalar cost signal below a certain value (Achiam et al., 2017). According to the reward hypothesis, it could be possible to encode safety requirements di- rectly in the reward signal. However, as argued by Ray et al. (2019), safe RL based only on scalar reward carries the issue of designing a suitable reward function. In particular, balancing the trade- off between reward optimization and safety within a single reward is a difficult problem. Moreover, over-engineering rewards to complex safety requirements runs the risk of triggering negative side effects that surface after integration into broader system operation (Abbeel & Ng, 2005; Amodei et al., 2016). Constrained RL addresses this issue via a clear separation of reward and safety.
Reinforcement learning from pixels typically suffers from sample inefficiency, as it requires many interactions with the environment. In the case of safe RI,, improving the sample efficiency espe- cially crucial each interaction with the environment, before the agent reaches safe policy, has an opportunity to cause harm (Zanger et al., 2021). Moreover, to ensure safety, there is an incentive to act pessimistic with regard to the cost (As et al., 2022). This conservative assessment of safety, in turn, may yield a lower reward performance than is possible within the safety constraints.
Our contribution. We propose Safe SLAC, extension of the Stochastic Latent Actor Critic ap- proach (SLAC; Lee et al., 2020) to problems with safety constraints. SLAC learns a stochastic latent variable model of the environment dynamics, to address the fact that optimal policies in partially ob-

---

Published as a conference paper at ICLR 2023
servable settings must estimate the underlying state of the environment from the observations. The model predicts the next observation, the next latent state, and the reward based on the current ob- servation and current latent state. This latent state inferred by the model is then used to provide the input for an actor-critic approach (Konda & Tsitsiklis, 1999). This algorithm involves learning a critic function that estimates the utility of taking a certain action in the environment, which serves as supervision signal for the policy, also named the actor. The SLAC method has excellent sample efficiency in the aagety-agnostic partially observable setting, which renders a promising candidate to adapt to high-dimensional settings with safety constraints. At its core, SLAC is an actor-critic ap- proach, carrying the potential for natural extension to safety with a safety critic. We extend SLAC in three ways to create our safe RL approach under partial observability (Safe SLAC); (1) the latent variable model also predicts cost violations, (2) we learn a safety critic that predicts the discounted cost return, and (3) we modify the policy training procedure to optimize
feeey-coontrained objec- tive by use of a Lagrangian relaxation, solved using dual gradient descent on the primary objective and Lagrange multiplier to overcome the inherent difficulty of constrained optimization.
We evaluate Safe SLAC using a set of benchmark environments introduced by Ray et al. (2019). The empirical evaluation shows competitive results compared with complex late-of-the-aa approaches.
2 RELATED WORK
Established baseline algorithms for safe reinforcement learning in the fully observable setting in- clude constrained policy optimization (CPO; Achiam et al., 2017), as well as trust region policy optimization ('aaOn-Lagranniann (Ray et al., 2019), a cost-constrained variant of the existing trust region policy optimization (TRPO; Schulman et al., 2015). While TRPO-Lagrangian uses an adap- tive Lagrange multiplier to solve the constrained problem with primal-dual optimization, CPO solves the problem of constraint satisfaction analytically during the policy update.
The method closest related to ours is Lagrangian model-based agent (LAMBDA; As et al., 2022), which also addresses the problem of learning a safe policy from pixel observations under high partial observability. LAMBDA uses the partially stochastic dynamics model introduced by (Hafner al., 2019). The authors take a Bayesian approach on the dynamics model, sampling from the posterior over parameters obtain different instantiations of this model. For each instantiation, simulated trajectories are sampled. Then, the worst cost return and best reward return are used to train critic functions that provide gradient the policy. LAMBDA shows competitive performance with base- line algorithms, however, there are two major trade-offs. First, by taking a pessimistic approach, the learned policy attains lower cost return than the allowed cost budget. A less pessimistic approach that uses the entirety of the allowed cost budget may yield constraint constraint-satiifying policy with a higher reward return. Second, the LAMBDA training procedure involves generating many samples from their variable model to estimate the optimistic/pessiiitic temporal difference updates.
While the reinforcement learning literature has numerous safety perspectives (Gareía & Fernández, 2015; Pecka & Svoboda, 2014), we focus on constraining the behavior of the agent on expectation. A method called shielding ensures safety already during training, using temporal logic specifications for safety (Alshiekh et al., 2018; Jansen et al., 2020). Such methods, however, require extensive prior knowledge in the form of a (partial) model of the environment (Carr et al., 2023).
3 CONSTRAINED PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
In reinforcement learning, an agent learns to sequentially interact with an environment maxi- mize some signal of utility. This problem setting is typically modeled as a Markov decision pro- cess (MDP: Sutton & Barto, 2018), in which the environment composed of a set of states S and a set of actions A. At each timestep i, the agent receives the current environment state E S and executes an action a; E A. according to the policy क; ar ~ T(aa] { This action results in a new state according to the transition dynamics 8141 p((++1 St, ar) and scalar reward signal r: = r(s;, 4z) E R, where r is the reward function. The goal is for the agent to learn an optimal policy x² such that the expectation of discounted, accumulated reward in the environment under that policy is maximized, i.e. 7' = arg max, E |E,- rrr with y E [0, 1). We use pa to denote the distribution over trajectories induced in the environment by policy m.
2

---

Published as a conference paper at ICLR 2023
In partially observable Markov decision process (POMDP; Kaelbling al., 1998), the agent cannot observe the true state s, of the MDP and instead receives some observation Iı E X that provides partial information about the state, sampled from the observation function 2t X(s)). In this setting, learning policy л is more difficult than in MDPs since the true state of the underlying MDP is not known and must be inferred from sequences of observations to allow the policy to be optimal. Therefore, the optimal policy is function of a history of observations and actions T (a. Titt 01.--1). practice, representing such policy can be infeasible, so the policy often conditioned on compact representation of the history.
While the reward hypothesis states that desired agent behavior can plausibly be represented single scalar reward signal, in practice it can be difficult to define a reward function that balances different objectives (Vamplew et al., 2022; Roy et al., 2022). The same is true for safety: as ar- gued by Ray et al. (2019), a useful definition of safety is constraint on the behavior, stating the problem as a constrained MDPs (CMDP; Altman, 1999), with analogous definitions for constrained POMDPs (CPOMDP, Isom et al., 2008; Lee et al., 2018; Walraven & Spaan, 2018). A scalar cost variable 6₂ E R at each time step of the MPPO)MDP, according to a cost function G = c(8,, at), serves measure of safety violation. The objective of the reinforcement learning problem then changes to constrain the accumulated cost under a given safety threshold d E R:
arg max I
Σ r'r
S.
Σ <d
Next, we review RL algorithms for POMDPs and CMDPs, laying the foundations for the safe RL algorithm we propose in Section 5 dedicated to CPOMDPs with unknown env inviromment dynamics.
4 STOCHASTIC LATENT ACTOR-CRITIC
Soft actor-critic (SAC: Haarnoja et al., 2018a) is an RL approach based on the maximum entropy framework. Besides the traditional reinforcement learning objective of maximizing the reward, it also aims to maximize the entropy of the policy. The sampling of states that allow high-entropy action distributions is maximized. This approach can be robust to disturbances in the dynamics of the enviromment (Eysenbach & Levine, 2022). In practice, it can be challenging to determine weight for the entropy term in the objective priori, instead, the entropy is constrained to minimum value (Haarnoja et al., 201 (0).. A stochastic policy is particularly interesting to our work since deterministic policy might be suboptimal in the CPOMDP setting (Kim et al., 2011).
Stochastic latent actor-critic (SLAC; Lee et al., 2020) is an RL. algorithm that addresses partial observability and nigh-dimensional observations by combining probabilistic sequential latent vari- able model with an actor-critic approach. The sequential latent variable model aims to infer the true state of the environment by considering the environment as having an unseen true latent state z and latent dynamics p(z Z, a.). This model generates observations X; ~ p(x, Zz) and rewards I, ~ p(r, Z,, a, 1)) By approximating these latent dynamics in the latent variable model using approximate variational inference, the learned model can infer the latent state from previous obser- vations and actions as {2, Xo.e, a.:r). Consequently, SLAC can be viewed as an adaption of SAC to POMDPs, using the inferred latent state input for the critic in SAC.
Equation 2 describes the latent variable model M, which is parameterized by th This model infers a low-dimensional latent state representation z from a history of high-dimensional ob- servations x and actions a gathered through interaction with the environment. This model is trained by sampling trajectories from the environment and inferring latent state from each trajectory according to the distributions given in Equation 3. The model uses this inferred latent state to predict the next observation and reward according to the distributions in Equation 3.
Zi p(z1).
Ze+1 Pa= (Ze+1
Zr, az).
(0o xi). Zc+1 que (Ze+1 xe+1, Ze, a.).
(2)
Xt Pa= (x: zz).
P (r, aq, Z(41).
Hereinafter, we use M to refer to the functions of the model (Par,Pe>, Pyr, 9uo, qy=), and use eb without the superscript to simplify the notation. Furthermore, following Lee al. (2020), in our
3

---

Published as a conference paper at ICLR 2023
implementation decompose the latent variable 2 in two parts (see Appendix A for more details). The parameters the model vo are trained to optimize the objective in Equation 4, in which DKL is the Kullback-Leible divergence, asymmetric measure of the difference between two distributions commonly used in variational inference (Kingma & Welling, 2014):
log Pol (x++1 (DKI (Z441 Nx+1, Ze, a:) Ps (Z++1 Ze; at))
((41) log Pa (re+1 2(+1)
JM(()) 21
(4)
The resulting inferred latent state, which captures information about previous observations and ex- pected reward in future time steps, serves as the input for the critic in SAC.
In practice, the probability distributions in Equations and 3 are set to follow Gaussian distributions with diagonal covariance matrices, where the means and variances are the output of neural networks, optimized using the reparametrization trick (Kingma & Welling, 2014). The neural network that models dv [z] x treats lower-dimensional feature vector, extracted from the high-dimensional observation using a convolutional neural network. For further details, we refer the interested reader to Lee al. (2020).
We can solve this problem taking a Lagrangian relaxation of the constrained problem and optimizing the unconstrained dual problem through dual gradient descent (Geibel & Wysotzki, 2005; Stooke et al., 2020). SAC-Lagrangian uses this approach, to bring SAC to the safety-constrained CMDP setting, where it may constrain the step-wise cost (Ha et al., 2020) cost return (Yang al., 2021). In the next section, we present our approach to adapt SLAC for the CPOMDP setting.
5 SAFE SLAC
The Safe SLAC model is an adaption of SLAC for the safety-constrained setting under partial ob- servability. There are three steps to this adaption. First, adapt the latent variable model to include information related to safety in the latent state. Second, we train safety critic to predict the ex- pected cost return resulting from a given action. Third, we constrain the policy objective and use its Lagrangian relaxation to train the actor.
Latent variable model with cost prediction. We include notion of safety in the latent state by changing the generative part of the model to predict the cost at the next time step, in addition to predicting the observation and reward. This introduces the conditional probability distribution c, ~ Pae (cr Ze, ar, 1z++1) to the generative component of the model, analogous to the component that predicts reward. Using the simplified notation of the parameters (t)), the resulting model objective is:
log Pv (xe+ 1/2241) log Pe (Fe+1 (x41)) log Pv (Ce+ (2e+1) ] (5)
(+b)
Xt+1. Ze, az) (0 Zz, a:))
21:41009 T
While the other probability distributions in the model are Gaussian variables with diagonal co- variance matrix, we follow As et al. (2022) and use a Bernoulli distribution for the conditional cost distribution, since the cost is binary in the environments we consider.
Safety critic. As an integral part of Safe SLAC, we train a safety critic. First, the reward critic Q" with parameters 0 remains unchanged from SLAC and estimates the expected discounted reward return for a given latent state and action pair under the current policy, as well as the expected entropy of the policy after taking the given action the environment. To this end, Q" optimizes the soft Bellman residual in Equation 6.
(o) = E
(Q a,) (rr + 7 + (V((++1)))
(6)
Vol2++1) a, 11
E
(% (2 ar+1) a log x (ar+1 a1:)))
(7)
In addition to the reward critic, train the safety critic Q with parameters Ç that predicts the expected discounted cost return for a given latent state and action. The loss function for the safety critic, given in Equation is similar Equation 6 but does not include entropy term.
4

---

Published as a conference paper at ICLR 2023
/ (Q(( (cr +V 21:h 12049
Jq (5)
E
(8)
a. 11
Safe SLAC policy objective. SLAC follows the maximum entropy framework objec- tive (Haarnoja et al., 2018b), using the latent state as input for the critics. We adjust this to be constrained safe behavior. The policy objective in safety-indifferent SLAC maximizes both the entropy of the policy, which ensures sufficient exploration, and the soft cetion-value estimate as predicted by the reward critic Qú-
(9) Analogously to the safety critic term in SAC-Lagrangian (Ha et al., 2020), we obtain the objective for the safety-constrained policy by including term to minimize the expected cost return as estimated by the safety critic Qg. weighted by the Lagrange multiplier A:
Jx (0) Zu
E
E B14100k
[a log (a,+1 X1r++1, (irr+1,817)) 1,, ar+1]]]
Ja (6)
IE Piir+i 1
Ja log (ar+1 Xi a (0 + XQ (10)
As we discussed in Section 3, it is not practical condition the policy the full history. To circumvent this problem, Lee et al. (2020) use a truncated history of observations and actions as input for the policy; however, we find empirically that using the latent state as input for the actor yields better results in our setting. As such, in our training procedure the policy takes as input the current latent state z. which updated after each environment step. When using the latent state as input for the policy, occurrences of nh in Equations 6, 9 and 10 and Algorithm can instead be read as modeling the distribution To (ar+1 2,+1) and T ke), respectively. the gradient steps, we infer the latent state from truncated history of observations and actions that is sampled from the replay buffer, rather than from the full history.
Note that Safe SLAC carries the same limitation as SLAC. Overall, even in the unconstrained case, conditioning the policy on the latent variable is not guaranteed to result in the optimal policy of the underlying POMDP (see Lee et al., 2020). To find an optimal policy, we must condition the policy in the history or a belief state that provides sufficient statistics of the history. which may lead the agent take actions to gather information (Cassandra et al., 1994; Thrun, 1999).
The entropy and expected cost-return are constrained, therefore, a and À are adjusted dynamically to incentivize the policy to adhere to the constraints. is tuned using the procedure and hyperpa- rameters as in (Haarnoja et al., 2018b). Next, we describe how to adjust À during training.
5.1 SAFETY LAGRANGE MULTIPLIER LEARNING
We use on-policy data to adjust the Lagrange safety multiplier. Previous work in the fully observable setting has adjusted the Lagrange multiplier by estimating the cost return as the mean prediction of the safety critic over states sampled from the replay buffer(Yang et al., 2021). It has been shown for Q-learning (Thrun& & Schwartz, 1993) and policy gradient methods (Fujimoto al., 2018), that using function approximators such as neural networks to estimate state and action value functions can result in biased value estimates. While bias in value estimation can result in suboptimal policy learning, the induced bias is doubly important if the safety critic not only serves to provide gradient to the agent, but its magnitude is also used to determine the Lagrange multiplier. We have found in our experiments that for the given environments, this bias of safe policies and we were unable to mitigate this using the techniques proposed by Fujimoto et al. (2018). We attribute this to the high degree of partial observability in the environments, the off-policy nature of the learning algorithm, and the use of O-step temporal difference error for value updating. Consequently,
sufficient to prevent the learning
adjust the Lagrange multiplier À based on the real undiscounted cost return incurred in each training episode. This is expressed in the following loss, where À is changed to minimize the incurred cost return less the allowed cost budget:
T
Ja(A) = E
A
Σ t=
d
(II)
prope
5

---

Published as a conference paper at ICLR 2023
Algorithm Safe SLAC
Input: Empty replay bufter D. environment &, and initialized parameters v. 01, 02, ç, and Hyperparameter: Number of warmup interactions W, and warmup training steps W
Output: Optimized parameters
1: Sample W, transitions from & using a stochastic policy and store these in D.
2: for i=1 to W; do
3: Sample experience from
4: Update according to Equation 5
5: end for
6: while not converged do
7: for each environment step at time / do
8: ag (a -1:--)
9: Obtain (X++1, Ce+1) by executing in E
+ D U (Xe+1; a,, (+41,0++1)
11: Update by minimizing Equation 11
end for
13:
for each gradient step do
14:
(x a1,,, 11 C1:+11~~
15:
Update 81, fg according Equation 6
16:
Update &i according to Equation
Update according to Equation 10
18:
Update { according to Equation 8
O1 f v01 + 1
20:
Oo + v02 + (1 v2
< vc (1 D)C
22:
end for end while
where p is trajectory and d is the budget. Intuitively, the weight of the safety critic in the policy loss increased when the policy unsafe and decreased when the policy is safe. In practice, we found this procedure to be stable and effective for sufficiently low learning rate. We stress that the evaluation episodes are not used for policy learning, including adjustment of the Lagrange multiplier.
5.2 TRAINING PROCEDURE
Safe SLAC is described in Algorithm 1. The training procedure starts initializing the replay buffer by taking actions under a stochastic policy for 60k environment steps, with actions sampled from a truncated diagonal Gaussian distribution. then trains an initial latent variable model for 30k steps on sequences uniformly sampled from the replay buffer, minimizing the objective in Equation 5. After this warm-up procedure, we alternate between collecting experiences and training the latent variable model, actor, and critics.
To improve stability during training, we maintain target networks for the critics, which are updated using exponential averaging (Lines 19 to 21). The neural networks that parameterize the model, actor, and critics have the same architecture as SLAC, with hyperparameters as detailed in Ap- pendix B. Except for the safety Lagrange multiplier, we optimize all parameters using the Adam optimizer (Kingma & Ba, 2014).
Whereas Lee et al. (2020) indicate that the optimal additive noise added to the observation prediction during latent variable model training is of low variance for most environments, we find better results with large variance (0.4). This is consistent with their hypothesis that strong noise yields better results for environments with large differences between subsequent observations, as is the case in environments.
We follow both SLAC and LAMBDA in using two action repeats on these environments, meaning that transform the base environment that each action executed in it is performed twice before the agent receives a new observation. We aggregated both the reward and cost incurred in the transitions induced by these repeated actions to scalar values by summing them.

---

Published as a conference paper at ICLR 2023
PointGoall
PointGoal2
DoggoGoall
CarGoall
PointPushl
PointButtonl
Six SG6 environments used in experiments. Top row shows the first-person camera pixel image that our agent observes. Bottom row gives a top-down overview of the environment. In all environments, the agent (red) must reach the goal (green) while avoiding the hazards (blue and purple). In PointPushl, the agent must push the yellow box into the green goal to receive reward.
Figure 1:
EMPIRICAL EVALUATION
We evaluate our approach on a set of six SafetyGym benchmark environments introduced by Ray et al. (2019) as SG6, shown in Figure 1. For all environments, we the standard cost return budget 25 per episode. As training progresses, we frequently evaluate the performance of the policy in 10 evaluation episodes. The mean reward and cost return during these evaluation episodes are shown in Figure 2 for three different random seeds per environment. We calculate the normalized reward and cost return using the performance in the last 5 evaluations, proposed by Ray et al. (2019). We compare our results with the final results from basic algorithms, including the unconstrained proximal policy optimization (PPO; Schulman et al., 2017), using sensor observations, and with LAMBDA trained on pixel observations, as reported by As et al. (2022); see Section 2 for descrip- tion of these algorithms. Figures and compile the results obtained in the SG6 set. Appendix D presents qualitative analysis of the results.
Figure shows that Safe SLAC converges on safe policy for nearly environments and attains a competitive reward return. Safe SLAC reaches higher reward return than LAMBDA compared to other baselines such as PointGoall and CarGoall environments. PointGoal2, we also reach a higher reward return than LAMBDA, but only after 2 million environment steps. The results shown for LAMBDA in PointGoal2 used training for only million steps; we reach the same performance as LAMBDA using approximately 40% more samples and continue improve after this.
Safety and reward tradeoff. In the DoggoGoall environment, we observe that while the reward return of Safe SLAC is significantly better than for the other algorithms, the mean cost return slightly larger than the allowed limit and is only better than CPO. This indicates that the tradeoff between reward and safety is balanced too far towards reward. Adjusting the learning rate for the safety Lagrange multiplier from the default 2 10-6 to * 10-6, we obtain the results indicated in Figure 3 as Safe SLAC+, which increases the reward return in PointGoal2. From this effect, we conclude that our policy makes meaningful tradeoff between optimizing reward and cost return, and that the current training procedure for the Lagrange multiplier is sensitive to the learning rate.
Challenges strong partial-observabiiity Like LAMBDA. our approach fails to make substan- tial improvement in reward return in PointPushl, although the policy is safe. We attribute this to the strong partial observability due to the pixel observations: while the box is pushed, the agent cannot observe its surroundings and so must rely entirely its belief latent state. This is supported by the fact that the baselines that use sensor observations are able to attain a much higher reward return.
Normalized metrics. To summarize the performance of our algorithm, we normalize the mean reward return and cost return for each environment the end of training by dividing them by the performance of unconstrained PPO, as proposed by Ray et al. (2019). The resulting normalized metrics, shown in Figure 3, indicate that Safe SLAC improves over the baselines and competitive
7

---

Published as a conference paper at ICLR 2023
median of Safe SLAC 5th 95th percentile Safe SLAC
Final CPO (sensor)
Final LAMBDA
Final TRPO Lagrangion (sensor)
Final PPO (sensor)
Budget
Reward Return
Cost return
CarGoall
20
100
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
x10%
x10
PointPushl
100
0.0
0.5
2.0
0.0
2.0
100
x10
DoggnGoall
40
2010
Test reward/cost return
20
100
0.0
0.5
2.0
0.0
2.0
110"
x10
PoiorGoal2
100
10-
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
x10#
x10
100
PointButonn
10-
0.0
0.5
1.5
2.0
0.0
1.0
1.5
2.0
X10P
x10
PoinCroal
20
50
0.4
1.0
0.4
1.0
X1IP
x10"
Environment steps
Figure 2: Evaluation reward and return. Note that for PointGoal2, the final results for LAMBDA are calculated after million environment steps, whereas we use 2 million.
with LAMBDA. We exclude DoggoGoall from this figure, Safe SLAC is not safe for standard learning rate as discussed above. The normalized reward return so large for Safe SLAC that obscures the performance in other environments. The same figure with DoggoGoall included can be seen in Figure 5 (Appendix C).
History as actor input. As discussed in Section 5, we deviate from the suggestion by Lee et al. (2020) to use the truncated history as input for the actor and instead use the latent state inferred from the full history select actions. In Figure inputs on the PointGoal2 environment. While both variants converge to a safe policy, the reward performance of the latent state policy is higher than that of the history policy.
compare the performance of these two policy
8

---

Published as a conference paper at ICLR 2023
PointGoall
Safe SLAC
Safe SLAC+
PointCioa22
Ensieomenn
Lagrangian
CarGiall
LAMBDA
1.0
PointPushi
0.5
Mean
0.0
0.0
0.5
1.0
1.5
0.0
0.2
0.4
0.6
Task Objective
Salety
Normalized Reward
Normalized
Figure 3: Normalized reward return and normalized cost return performance of Safe SLAC com- pared to LAMBDA and the baselines. "Safe SLAC" indicates our approach evaluated using single of typerparameters for all environments and after million environments steps PointGoal2, to match LAMBDA. "Safe SLAC+" shows results using an adjusted learning rate on the DoggoGioal environment and million steps on PointGoa12.
Latent policy
Mean reward return
15
Mean cost return
150
History policy
Budget
100
10
50
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
10
x10
Environment steps
Figure 4: Comparison of latent state and truncated history as actor input on the PointGoal2 environ- ment. While both are safe at the end of training, the latent state policy shows higher reward return.
Safe SLAC may be less pessimistic than LAMBDA. For PointGoal2, LAMBDA was trained only with IM environment steps. For this computational budget, Safe SLAC is able to find safe policy, however, we notice final reward return from LAMBDA. Nevertheless, with more time Safe SLAC clearly outperforms LAMBDA, whose reward return in this cnvironment plateaus before reaching million environment steps (Figure 5 from As et al., 2022). In summary, Safe SLAC can keep improving while LAMBDA stagnates as we use more of the allowed cost budget.
not converged yet, and the reward return slightly lower than the
Computational requirements. Our approach is less complex than LAMBDA, as it does not re- quire costly sampling of Bayesian world models, generating synthetic experience using the latent variable model. Thus, we conjecture that Safe SLAC also has lower computational requirements. As an example, training our approach for I million base environment steps on PointGoall takes a total of 10 hours and 14 minutes. For contrast, LAMBDA takes 13 hours and 20 minutes (measured on the same hardware). Nevertheless, this improvement might also be due implementations details.
7 CONCLUSION
This paper proposes Safe SLAC, a constryconttre RL approach for partially observable settings, which uses stochastic latent variable model combined with safety critic. Empirically, shows competitive performance with state-of-the-art methods while having less complex training proce- dure. Safe SLAC often derives policies with higher reward returns, while still satisfying the safety constraint. Future work includes using distributed critic increase the control over the safety violations (Yang et al., 2022) and improving the optimization of the Lagrange multiplier (Stooke al., 2020), which is currently sensitive to its learning rate seen in the DoggoGoall experiments.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was funded by the ERC Starting Grant 101077178 (DEUCE) and the NWO grant NWA. 160. 8.238 (erimaVera). We thank Toshiki Watanabe for their reimplementation of SLAC in PyTorch, which served as the basis for our code.
REPRODUCIBILITY STATEMENT
The code for our implementation of Safe SLAC is available GitHub at https: g/thtub. com/ lava- lab/ safe- slac. The repository contains implementations for the Safe SLAC ac- tor, critics, model and training procedure. This includes the necessary code interface with the pub- licly available SaferyGym environments using pixel observations, as well as instructions to install the required dependencies and reproduce our experiments. In addition, the default hyperparameters are specified in configuration file.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Exploration and apprenticeship learning in reinforcement learning. In ICML, volume 119 of ACM International Conference Proceeding Series, pp. 1-8. ACM, 2005.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 2017.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI, pp. 2678. AAAI Press, 2018.
Eitan Altman. Constrained Markov Decision Processes: Stochastic Modeling. Routledge, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in AI safety. arXiv preprint arXiv:1606.06555 2016.
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via Bayesian world models. In ICLR. OpenReview.net, 2022.
Steven Carr, Nils Jansen, Sehastian Junges, and Ufuk Topcu. Safe reinforcement learning via shield- ing under partial observability. In AAAI, 2023.
Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In AAAI, pp. 1023-1028 AAAI Press The MIT Press, 1994.
Gabriel Dulac-Amold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Mach. Learn., (10(9))44199244)) 2468. 2021.
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In ICLR. OpenReview.net, 2022.
Justin Fu, Andrea Tacchetti, Julien Pérolat, and Yoram Bachrach. Evaluating strategic structures in multi-agent inverse reinforcement learning. Artif. Intell. Res., 71:925-951, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, volume 80 of Proceedings of Machine Learning Research, pp- 1582-1591. PMLR, 2018.
Javier Gareía and Fernando Fernández. A comprehensive survey safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015.
Peter Geibel and Fritz. Wysotzki. Risk-sensitive reinforcement learning applied to control under constraints. J. Artif. Intell. Res., 24:81-108; 2005.
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In CoRL, volume 155 Proceedings of Machine Learning Research, pp. 1110-11 200 PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arviv: / 2.09959, 2018b.
Danijar Hafner, Timothy P. Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 2019.
Joshua D. Isom, Sean P. Meyn, and Richard D. Braatz. Piecewise linear dynamic programming for constrained POMDPs. In AAAI, pp. 291-296. AAAI Press, 2008.
Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe re- inforcement learning using probabilistic shields (invited paper). In CONCUR. volume 171 of LIPIcs, pp. 3:1-3:16. Schloss Dagstuhl Leibt----ebibirtrrre für Informatik, 2020.
John Jumper, Richard Evans, Alexander Pritzel. Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, al. Highly accurate protein structure prediction with Alphafold. Nature, 596(7873::833-58,, 2021.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):9-134, 1998.
Dongho Kim, Jaesong Lee, Kee-Eung Kim, and Pascal Poupart. Point-based value iteration for constrained POMDPs. In IJCAI, pp. 1968-1974. ICCA/AAAL, 2011.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1511.0728% 2014.
Diederik Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR, 2014.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In NIPS, pp. 1008-1014. The MIT Press, 1999.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. NeurIPS, 2020.
Jongmin Lee, Geon-hyeong Kim, Pascal Poupart, and Kee-Eung Kim. Monte-Carlo tree search for constrained POMDPs. In NeurlPS, pp. 7934-7943, 2018.
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free rein- forcement learning: A survey. ICCAI, pp. 4508 4515. ijcai.org, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. RUsu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. 2015.
Nat., (1..51875400:5599-55553
Martin Pecka and Tomás Svoboda. Safe exploration techniques for reinforcement learning An Sciennc,e overview. In MESAS, volume 8906 of Lecture Notes in Computer Science, p 357-375. Springer, 2014.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning, 2019. URL https: cdn. http:://ccc....aii.eeeeeemee openai com/ sa waffe--soosrr pdf.
Julien Roy, Roger Girgis, Joshua Romoff, Pierre-Lue Bacon, and Christopher J. Pal. Direct behavior specification via constrained reinforcement learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 18828-18843, PMLR, 2022.
11

---

Published as a conference paper at ICLR 2023
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. ICML, volume JMLR Workshop and Conference Proceedings, pp- 1889-1897. JMLR.org, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 9133-9143, PMLR, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Sebastian Thrun. Monte Carlo POMDPs. In NIPS, pp. 1064-1070. The MIT Press, 1999.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. Proceedings of the Fourth Connectionis! Models Summer School, pp. 1-9, 1993.
Peter Vamplew, Benjamin J. Smith, Johan Källström, Gabriel Oliveira Ramos, Roxana Rad- ulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Pieter J. K. Libin, Richard Dazeley, and Cameron Foale. Scalar reward is not enough: response to Silver, Singh, Precup and Sutton (2021). Auton. Agents Multi Agent Syst., 36(2):41, 2022
Erwin Walraven and Matthijs T. J. Spaan. Column generation algorithms for constrained POMDPS. J. Artif. Intell. Res., 62:489-533, 2018.
Qisong Yang, Thiago D. Simāo, Simon H. Tindemans, and Matthijs T. Spaan. WCSAC: Worst- case soft actor critic for safety-constrained reinforcement learning. In AAAI, pp. 10639-10646. AAAI Press, 2021.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Safety-constrained reinforcement learning with a distributional safety critic. Mach. Learn., pp. 1-29, 2022.
Moritz A. Zanger, Karam Daaboul, and J. Marius Zöllner. Safe continuous control with constrained model-based policy optimization. In IROS, pp. 3512-3519, IEEE, 2021.
12

---

Published as a conference paper at ICLR 2023
A LATENT VARIABLE MODEL
Following the observations from Lee et al. (2020, Appendix B), we implemented a model composed of two latent variables z and z Equations and provide the detailed description of this model.
z p(zi).
(12a)
2 Poiaoa (z}
2/)
(12b)
2/41 Puee 22/11 2 az).
(12c)
2 2i41 Po (2it1
2i+1 zi ar).
(12d)
Xx Pe (xt
2, z2).
(12e)
It ~ per (It bi bi
Bes 2i41 2(+1).
(12f)
z 00 [z] x1).
(13a)
z ~ Puie2 (z} 2t)-
(13b)
111 ~ qaal (2
X(+1; zi az).
(13c)
2/1 Paie (2z+1
2,+1> 27 ar).
(13d)
Notice that this factorization the generative and variational models share some parts, namely Equations 13b and 13d the same as Equations 12b and 13d, respectively. As Lee al. (2020)
shows, this allow us to compute the KL-divergence (gp X Ze, Zt, ar) Pa (2
Z,, a:)) based only on the z', which simplifies the training process. We refer the reader to Lee et al. (2020) for further details of the implementation.
B HYPERPARAMETERS
Table 1: Hyperparameters used for safe SL.AC
Parameter
Value
Action repeat
2
Image size
64 x 64 x
Image reconstruction noise
0.4
Length of sequences sampled from replay buffer
10
Discount factor
0.99
Cost discount factor
0.995
11 size
32
2² size
200
Replay buffer size
2 105
Latent model update batch size
32
Actor-critic update batch size
64
Latent model learning rate
1 * 10-1
Actor-critic learning rate Safety Lagrange multiplier learning rate Initial value for a Initial value for À Warmup environment steps Warmup latent model training steps Gradient clipping norm
2 10
2e -
4 10 3
2 10
60 103
30 103
40
Target network update exponential factor
5 10
13

---

Published as a conference paper at ICLR 2023
C NORMALIZED RESULTS INCLUDING DOGGOGOALI
PuintCcall
SLAC
PuinsGool2
SLAC+
CPO
Carkkall
Lagrangion
LAMRDA
PointBatton
Namalzze Motrie
PoincPushi
DegnoGeall
Objective
Sufety
Normalized Reward Retum
Normalized Cost Retuen
Metric
Figure 5: Normalized reward return and normalized cost return performance Safe SLAC com- pared LAMBDA and the baselines. "Safe SLAC" indicates approach evaluated using single set of hyperparameters for all environments and after million environments steps on PointGoal2, to match LAMBDA. "Safe SLAC+" shows results using adjusted learning rate the DoggoGoall environment and 2 million steps on PointGoal2.
D QUALITATIVE ANALYSIS OF THE RESULTS
(a) CarGoal1
(b) DoggoGoall
(c) PointButton
(d) PointGoal1
(e) PointGoal2
(f) PointPush1
Figure 6: Sample from the roll-outs of the final policies in each environment.
Using the final policy computed by Safe SLAC, collected few trajectories each environment to make qualitative analysis. Figure 6 shows one time-step for each environment. For each en- vironment, we have three videos showing the observation, the latent variable model reconstruction, and overview of the environment.
We observe that SLAC can effectively solve the PointGoall, PointGoal2, and CarGoall tasks. In these environments, the agent has exploration strategy of turning 180 degrees in place when the target not visible. In PointGoal2, we also see the agent carefully crossing spaces between hazards. As expected, the agent mostly moves forward in these environments, while the agents using sensor observations also move backward." 2
In the PointButtonl environment, observe that the agent reach the target few times but might fail when the goal is not in the field vision of the agent.
In the DoggoGoall environment, the agent can also reach the goal. However, the spaces between the hazards can be small compared to the robot's size. We suspect this makes this task considerably more complex than the remaining tasks in terms of safety.
Finally, in the PointPush1 environment, the agent goes to the box. However, does not move the box the goal position. This is due to the strong partial observability of this environment since the agent cannot observe the goal when the box is between the agent and the goal position.
Videos are available at phttps:///peae com wwt..//w.//ccce.ooooooooo faty-
wttp:/ttwwh.ccc.ooaaaaaaaaa http:/laaalaab/aaaaaa safe- sereesslactse http://aafsssaactsaleeeeeeee rollouts.
14


Reviews:
[{"review_id": "tnM6_G3244", "reviewer": "Reviewer_kqqD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\n\nThis paper extends the stochastic latent actor-critic method to safe reinforcement learning by a safety critic. The experiments show the competitive performance of the proposed methods over existing approaches on benchmark pixel input tasks. ", "summary_of_the_review": "\n\nGenerally, this paper is a combination of existing methods to me and is not novel. The paper takes a lot of content on introducing existing methods (constrained POMDP, SLAC). Using the Lagrange multiplier is also not novel for safe RL [1,2].\n\n\n\n\n\n[1] Geibel, P. and Wysotzki, F. Risk-sensitive reinforcement learning applied to control under constraints. CoRR, abs/1109.2147, 2011. U\n\n[2] Stooke A, Achiam J, Abbeel P. Responsive safety in reinforcement learning by pid lagrangian methods[C]//International Conference on Machine Learning. PMLR, 2020: 9133-9143.\n\n\n\n", "strength_and_weaknesses": "\n\nStrength:\n\nThe presentation of the paper is very clear. \n\nThe experimental results are also sufficient to show the effectiveness of the proposed methods.\n\n\n\nWeakness:\n\nThe main weakness is that the proposed method is not novel. The author claimed that 3 contributions of the method. They are minor to me. In my view, this paper extends SLAC to safe RL case by introducing the cost mechanism to the prediction target and discounted cost return, respectively. None of them are novel.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "bFm9ua5Ip7x", "reviewer": "Reviewer_asV6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for safe reinforcement learning from pixel observations. This is framed as a POMDP problem where the agent receives separate cost signals from the environment in addition to rewards. At a high-level, this work adapts the Stochastic Latent Actor-Critic framework (which learns a latent representation along with a dynamics model in the latent space) to handle safety constraints. Particularly, soft constraints since the agent can violate them at every time step for a cost. Thus the resulting learned policy becomes \"safe\" by minimizing some notion of expected cumulative cost in addition to maximizing the expected return. In this work, safety is considered through the addition of a separate safety critic in the policy objective that predicts the expected cost return given a latent state and an action. ", "summary_of_the_review": "This paper presents a straightforward set of modifications to the SLAC framework onto the safety-constrained setting. This would make implementation and benchmarking fairly simple since the environments used are also standard in this setting.", "strength_and_weaknesses": "The trained policy is empirically evaluated on a set of challenging, fairly-recent safety-gym environments (Ray et al. 2019). The method seems to learn safe, competitive policies in most of the environments (with DoggoGoal1and PointPush as exceptions). However, it is difficult to understand whether these policies are any good. A potential quick fix here is to plot a line similar to the budget line on the cost return plots but for the policy. This line would represent the reward return for the worst policy that still solves the problem. Later in the paper, there is a discussion on computational requirements where the running time of the method is compared to that of LAMBDA (As et al. 2022). However, it is not clear to the reader whether or not this difference was due to implementation details or computational complexity.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "INszl_pZkUz", "reviewer": "Reviewer_4rUn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends SLAC [1] to make it applicable to constrained POMDPs. The proposed approach combines model-free RL with a latent representation that is learned by a generative model. Constraints are satisfied by learning a constraint critic and optimizing a Lagrangian, which boils down to extending the regular SAC objective by another critic component that is weighted by a Lagrange multiplier. The method performs on par with the state-of-the-art on the safety-gym benchmark\n\n---\n\n[1] Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model, https://arxiv.org/pdf/1907.00953.pdf", "summary_of_the_review": "This paper is an iteration over SLAC, which modifies that work with a constraint critic, allowing application to constrained POMDPs. The method consists mostly of existing approaches combined in a new form, which has the benefit that each component has a history of related work supporting it. That being said, the method does not have a clear edge over the closest related work (LAMBDA). Thus, I see this paper as a new approach for attaining state-of-the-art performance in a benchmark for safe-RL. I believe this is still a contribution worthy of acceptance, since it will widen researchers' knowledge of which methods work and which do not.", "strength_and_weaknesses": "**strengths**\n\n* The paper proposes a method for safe RL in POMDPs which is competitive with the state-of-the-art.\n* The method mostly consists of previously established work that is combined in a new way. I see this as a strength.\n\n**weaknesses**\n\n* The performance gap to previous work is not clear. As such, the paper is not an improvement over the current state of this setup. This is my main reason for the score I've given to the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Pjv7LbeN3", "reviewer": "Reviewer_sCDz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an algorithm for solving safety-constrained decision problems with unknown dynamics that are also partially observable, which is one of the most difficult type of decision problems. The approach is to learn a safety-constrained POMDP model that has latent state, and also includes a safety critic. All components of the model are estimated simultaneously from experienced transitions, by modifying the training objective from only maximizing cumulative discounted reward to also maintain cumulative costs, in the sense of safety violations, under a specified threshold. Empirical evaluation in several environments shows computational advantages with respect to existing baseline algorithms.", "summary_of_the_review": "This paper appears to be a good contribution to the field of learning and solving POMDP problems, with the further complication of safety constraints. The explanations of the algorithm and the mathematical notation can be improved.", "strength_and_weaknesses": "The paper attempts to solve one of the most difficult decision problems possible, and achieves favorable empirical results in comparison to existing algorithms. Moreover, it does it by learning a latent state model in the form of a POMDP, which makes the paper very suitable for this conference. Although the verification is in simulation and on relatively simple navigation problems, the dimensionality of the observation space is high, because observations are images, so solving this kind of problems is far from trivial, especially in the partially observable setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster