Constrained Policy Optimization via Bayesian World Models
Authors: Yarden As
Ilnura Usmanova
Sebastian Curi
Andreas Krause
Venue: ICLR 2022 Spotlight (2022)
Keywords: Reinforcement learning
Constrained Markov decision processes
Constrained policy optimization
Bayesian model-based RL

Abstract: Improving sample-efficiency and safety are crucial challenges when deploying reinforcement learning in high-stakes real world applications. We propose LAMBDA, a novel model-based approach for policy optimization in safety critical tasks modeled via constrained Markov decision processes. Our approach utilizes Bayesian world models, and harnesses the resulting uncertainty to maximize optimistic upper bounds on the task objective, as well as pessimistic upper bounds on the safety constraints. We demonstrate LAMBDA's state of the art performance on the Safety-Gym benchmark suite in terms of sample efficiency and constraint violation.

Published as a conference paper at ICLR 2022
CONSTRAINED POLICY OPTIMIZATION VIA BAYESIAN WORLD MODELS
Yarden As ETH Zurich
Ilnura Usmanova, Sebastian Curi ETH Zurich
Andreas Krause ETH Zurich
ABSTRACT
Improving sample-effiieney and safety are crucial challenges when deploying reinforcement learning in high-stakes real world applications. We propose LAMBDA, a novel model-based approach for policy optimization in safety critical tasks modeled via constrained Markov decision processes. Our approach utilizes Bayesian world models, and hamnesses the resulting uncertainty to maximize optimistic upper bounds the task objective, well as pessimistic upper bounds on the safety constraints. We demonstrate LAMBDA's 's state of the art performance the Safety- benchmark suite in terms of sample efficiency and constraint violation.
INTRODUCTION
Task objective
Safety the end of t'taiinnn Safety during truining
A central challenge in deploying reinforcement learning (RL) agents in real-world systems is to avoid unsafe and harmful situations (Amodei et al., 2016). We thus seek agents that can learn efficiently and safely by acting cautiously and ensuring the safety of themselves and their surroundings.
Normalized metrics
1.0
0.5
common paradigm in RL for modeling safety critical envi- ronments are constrained Markov decision processes (CMDP) (Altman, 1999). CMDPs augment the common notion of the reward by additional cost function, c.g., indicating unsafe state-action pairs. By bounding the cost, one obtains bounds on the probability of harmful events. For the tabular case, CMDPs can be solved via Linear Programs (LP) (Altman, 1999). Most prior work for solving non-tabular CMDPs, utilize model-free algorithms (Chow et al., 2015; Achiam et al., 2017: Ray et al., 2019; Chow al., 2019). Most notably, these methods are typically shown to asymptotically converge to constraint- satisfying policy. However, similar model-free approaches in unconstrained MDPs, these approaches typically have very high sample complexity, i.e., require large number of - poten- tially harmful interactions with the environment.
0.0
LAMBDA
CEM-MPC
PPO-L Lagrangian
TRPO-L Lagrangian
CPO
Figure I: Normalized perfor- mance and safety metries, aver- aged across tasks of the Safety- Gym (Ray et al., 2019) SG6 benchmark. LAMBDA achieves constraint satisfaction in all tasks of the SG6 benchmark while sig- nificantly improving performance and sample efficiency. See Ap- pendix H for further details on normalization.
promising alternative to improve sample efficiency to use model-based reinforcement learning (MBRL.) approaches (Deisenroth & Rasmussen, 201 Chua et al., 2018; Hafner et al., 2019a: Janner et al., 2019). In particular, Bayesian approaches to MBRL quantify uncertainty in the estimated model (Depeweg et al., 2017), which can be used to guide exploration, e.g., via the celebrated optimism in the face of uncertainty principle (Brafman & Tennenholtz, 2003; Auer & Ortner, 2007; Curi et al., 2020a). While extensively explored in the unconstrained setting, Bayesian model-based deep RL approaches for solving general CMDPs remain largely unexplored. In this paper, we close this gap by proposing LAMBDA, Bayesian approach to model-based policy optimization in CMDPs. LAMBDA learns safe policy by experiencing unsafe as well as rewarding events through model -eneeetee trajectories instead of real ones. Our main contributions are as follows:
"ooreeponneecce to: yardas@ettz..co
1 Equal contribution.

---

Published as a conference paper at ICLR 2022
(a) PointGoall
(b) PointGool2
(c) CarGoall
(d) PointPushl
(c) PointButton
(f) DoggoGoall
Figure 2: Safety-Gym SG6 benchmark tasks. In our experiments, we use first-person-view images of
size 64x64 64 pixels as observations. Green objects represent goals that should be reached by the robot. Apart from the yellow box, that should be pushed to the goal area in the PointPushl task, hitting all other types of objects is considered an unsafe behavior. In Safety-Gym, stochasticity is achieved by performing random number of simulation steps before exposing the agent with a new observation.
We show that is possible to perform constrained optimization by back- back-propagating gra- dients of both task and safety objectives through the world-model. We the Augmented Lagrangian (Nocedal & Wright, 2006; Li et al., 2021) approach for constrained optimiza- tion.
We hamnsss the probabilistic world model to trade-off between optimism for exploration and pessimism for safety.
We empirically show that LAMBDA successfully solves the SG6 benchmark tasks in Safety-Gy -yym (Ray al., 2019) benchmark suite with frst-person-vied observations illus- trated Figure 2. Furthermore, we show that LAMBDA outperforms other model-based and model-free methods in this benchmark.
2 RELATED WORK
Interpretations of safety in RL research We first acknowledge that there exist different defi- nitions for safety (Gareía et al., 2015). One definition reversible Markov decision processes (Moldovan & Abbeel, 2012) that, informally, define safety as reachability between states and use backup policies (Eysenbach al., 2017) to ensure safety. Another definition adopts robust Markov decision processes (Nilim & Ghaoui, 2005; Tamar et al., 2013; Tessler et al., 2019), which try to maximize performance under the worst-case transition model. Finally, in this work we use non- tabular CMDPs (Altman, 1999) that define the safety requirements bounded by predefined threshold. Similarly, Achiam al. (2017); Dalal al. (2018); Ray et al. (2019); Chow et al. (2019); Stooke et al. (2020); Bharadhwajj et al. (2021): Turchetta et al. (2021) use non-tabular CMDPs as well, but utilize model-free techniques together with function approximators to solve the constrained policy optimization problem. As further demonstrate in our experiments, is possible to achieve better sample efficiency with model-based methods.
cost function that should be
Safe model-based RL A successful approach in applying Bayesian modeling to low-dimensional continuous-contrr problems is to use Gaussian Processes (GP) for model Icarning. Notably, Berkenkamp et al. (2017) use GPs to construct confidence intervals around Lyapunov functions which are used to optimize policy such that it is always within a Lyapunov-stable region of attrac- tion. Furthermore, Koller et al. (2018); Wabersich Zeilinger (2021) use GP models to certify the safety of actions within model predictive control (MPC) scheme. Likewise, Liu et al. (2021) apply problems together with ensembles of neural networks
MPC for high-dimensional continuous-comta (NN), the Cross Entropy Method (CEM) (Kroese al., 2006) and rejection sampling maximize expected returns of safe action sequences. However, by planning online only for short horizons and not using crities, this method can lead to myopic behaviors, later show in experiments.
2

---

Published as a conference paper at ICLR 2022
Lastly, similarly to this work, Zanger et al. (2021) NNs and constrained model-based policy op- timization. However, they do not use model uncertainty within an peiimistic-pssiimist framework but rather to limit the influence of erroncous model predictions on their policy optimization. Even though using accurate model predictions can accelerate policy learning, this approach does not take advantage of the epistemic uncertainty (e.g., through optimism and pessimism) when such accurate predictions are rare.
Exploration, optimism and pessimism Curi et al. (2021) and Derman et al. (2019) also take a Bayesian optimistic-pessimiiii perspective to find robust policies. However, these approaches do not use CMDPs and generally do not explicitly address safety. Similarly, Bharadhwaj et al. (2021) apply pessimism for conservative safety critics, and use them for constrained model-free policy optimization. Finally, Efroni al. (2020) lay theoretical foundation for the exploration- exploitation dilemma and optimism in the setting of tabular CMPDs.
3 PRELIMINARIES
3.1 SAFE MODEL-BASE BBSSED REINFORCEMENT LEARNING
Markov decision processes We consider episodic Markov decision process with discrete time steps E {O....,7)) T}. We define the environment' ' state as 8, E R" and an action taken by the agent, as ar E R Each episode starts by sampling from the initial-state distribution 80 ~ p(so). After observing 8o and at each step thereafter, the agent takes an action by sampling from a policy distribution a x--(s)). The next state is then sampled from an unknown transition distribution 8141 ~ s(. az). Given state, the agent observes reward, generated by r, ~ p(-1st, az). We define the performance of a pair of policy æ and dynamics p as
T Σ
J(T = Bainn, 1mp.sompp
rc
Constrained Markov decision processes To make the agent adhere human-defined safety con- straints, we adopt the constrained Markov decision process formalism of Altman (1999). In CMPDs, alongside with the reward, the agent observes cost signals of generated by of p(-1s az) where i = ..... C' denote the distinct unsafe behaviors we want the agent to avoid. Given ef, we define the constraints as
J' (T.P) = BaINR,SL p
Σ 1==
c
S d ViE f1, C).
(2)
where d' are human-defined thresholds. For example, a common cost function is c'( 1,, E where H' is the set of harmful states (e.g., all the states in which robot hits obstacle). In this case, the constraint (2) can be interpreted as a bound on the probability of visiting the harmful states. Given the constraints, we aim to find policy that solves, for the true unknown dynamics p²,
max sell
(T, s.t. J'(TTpp) d', Vi (I....,C).
Model-based reinforcement learning Model-based reinforcement learning revolves around the repetition of an iterative process with three fundamental steps. First, the agent gathers observed transitions (either by following an initial policy, or an offline dataset) of (8141, 8t, az) } into a dataset D. Following that, the dataset is used to fit statistical model p(s (st, ag, 0) that approximates the true transition distribution p*. Finally, the agent uses the statistical model for planning, either within an online MPC scheme or via offline policy optimization. In this work, we consider the cost and reward functions as unknown and similarly to the transition distribution, we fit statistical models for them as well. Modeling the transition density allows us to cheaply generate synthetic sequences of experience through the factorization p(8 # 81 ay H --1, 0) = III p($ ar, 0) whereby H is predefined sequence horizon (see also Appendix 1). Therefore, as already shown em- pirically (Deisenroth & Rasmussen, 2011; Chua et al., 2018; Hafner et al., 2019b), MBRL achieves superior sample efficiency compared to its model-free counterparts. We emphasize that in safety- critical tasks, where human supervision during training might be required, sample efficiency is par- ticularly important since it can reduce the need for human supervision.
3

---

Published as a conference paper at ICLR 2022
Learning a world model Aiming to tighten the gap between RL and real-world problems, relax the typical full observability assumption and consider problems where the agent receives an observation o, ~ p(-(st) instead of 8, each time step. To infer the transition density from ob- servations, we base our world model on the Recurrent State Space Model (RSSM) introduced in Hafner et al. (2019a). To solve this inference task, the RSSM approximates the posterior distri- bution 8 ~ qp(-1or:s- H,, -r-1:r+H-1) via an inference network parameterized by 6. We utilize the inference network to filter the latent state as new observations arrive, and use the in- ferred latent state as the policy's s input. Furthermore, the RSSM models the predictive distribution p(( P( 1 + 1. f) as differentiable function. This property allows us to perform the constrained policy optimization by backpropagating gradients through the model. We highlight that the only requirement for the world model is that models p(87:r+ + 18(87-1- ar- 1:-+ 0) as a differentiable function. Hence, it is possible to use other architectures for the world model, as long as p( + 8y-1, a2 -1: 0) is differentiable (c.g.. see GP-models in Curi et al. (2020b)).
3.2 BAYESIAN VIEW ON MODEL-BASE REINFORCEMENT LEARNING
The Bayesian predictive distribution A common challenge in MBRL is that policy optimization can "overfit" by exploiting inaccuracies of the estimated model. A natural remedy is to quantify uncertainty in the model, to identify in which situations the model is more likely be wrong. This is especially true for safety-critical tasks in which model inaccuracies can mislead the agent into taking unsafe actions. A natural approach perspective, where we adopt prior on the model parameters, and perform (approximate) Bayesian inference obtain the posterior. Given such a posterior distribution over the model parameters P(()D), we marginalize over 9 to get a Bayesian predictive distribution. Therefore, by maintaining such posterior, pl D(S 8, a,
uncertainty quantification is take Bayesian
determined by the law of total probability
p(8 8 -- 1
E0)p(()D) [p(*r:++H 8--1, - - 0)]
(4)
Such Bayesian reasoning forms the basis of celebrated MBRL algorithms such as PETS (Chua et al., 2018) and PILCO (Deisenroth & Rasmussen, 2011), among others.
Maintaining a posterior over model parameters Since exact Bayesian inference is typically infeasible, rely on approximations. In contrast the popular approach of bootstrapped ensem- bles (Lakshminarayanal et al., 2017) widely used in RL research, we use the Stochastic Weight Serraiing-Gausiian (SWAG) approximation (Maddox et al., 2019) of p(0 Briefly, SWAG con- structs posterior distribution over model parameters by performing moment-matching over iterates of stochastic gradient decent (SGD) (Robbins, 2007). The main motivation behind this choice the lower computational and memory footprint of SWAG compared to bootstrapped ensembles. This consideration crucial when working with parameter-rich world models (such as the RSSM). Note that approach is flexible and admits other variants of approximate Bayesian inference, such as variational techniques (Graves, 2011; Kingma & Welling, 2014).
LAGRANGIAN MODEL-BASED AGENT (LAMBDA)
We first present our proposed approach for planning, i.e., given world model representing CMDP, how to use to solve the CMPD. Following that, we present our proposed method for finding the optimistic and pessimistic models with which the algorithm solves the CMDP before collecting more data.
4.1 SOLVING CMPDS WITH MODEL-BASED CONSTRAINED POLICY OPTIMIZATION
The Augmented Lagrangian We first consider the optimization problem in Equation (3). To find a policy that solves Equation (3), we take ad advantage of the Augmented Lagrangian with proximal relaxation method as described by Nocedal & Wright (2006). First, we observe that
J(T) if अ feasible TENI 1-00
max min HEll X
J(T)
max
(5)
otherwise
is an equivalent form of Equation (3). Hereby X are the Lagrange multipliers, each corresponding to safety constraint measured by (Tr). In particular, if is feasible, we have J'()) & d' Vi
4

---

Published as a conference paper at ICLR 2022
[I and so the maximum over X' is satisfied if X' Vi (I..... C). Conversely, if is infeasible, at least one can be arbitrarily large to solve the problem Equation (5). Particularly, Equation (5) is non-smooth when transitions between the feasibility and infeasibility sets. Thus, in practice, we use the following relaxation:
C Pk Ex-- xb)
J(m) Ex(T(m)
max min xell A20
where Pa is a non-decreasing penalty term corresponding to gradient step k. Note how the last term in (+) encourages A' to stay proximal to the previous estimate Mf. and as consequence making (+)
a smooth approximation of the left hand side term in Equation (5). Differentiating (*) with respect
to X and substituting back leads to the following update rule for the Lagrange multipliers:
X + ())(())) d²) if X + (N - d') 2 0
Vi E (1,.... о) A =
{0
(6)
otherwise
We take gradient steps of the following unconstrained objective:
J(T; (TT) Σ
V (TT; (T: Mk),
where
(T) d') (J'(T) d')² (A
if + (m(((T) -d') 20 (8)
V' (T
otherwise.
3pp
Task and safety crities For the task and safety critics, we use the reward value function (s;) and the cost value function each constraint v'() Vi E [1.....C), We model v(s)) and 1 (s;) as neural networks with parameters & and v¹ respectively. Note that we omit here the dependency of the critics to reduce the notational clutter. As in Hafner al. (2019b), we use TD(A) (Sutton & Barto, 201 8) to trade-off the bias and variance of the critics with bootstrapping and Monte-Carlo value estimation. To learn the task and safety critics, we minimize the following loss function
Lug (v²) [....((srr
1 2H
Σ (u)(sr) - Vx(si)) (V(ss))
which uses the model to generate samples from p(8 87 11-11ar-1:44 -1 0). We denote o as the neural network that approximates its corresponding value function and Vx as the TD(A) value as presented Hafner et al. (2019b). Similarly, Vž is the TD(A) value of the 7th constraint Vi {....., C). Note that, while we show here only the loss for the task critic, the loss function for the safety critics is equivalent to Equation (9).
Policy optimization We model the policy Gaussian distribution via neural network with parameters & such that ne (x)) = =N(a;; NN (st). NN (s+)). We sample a sequence 8 ~ p(s 5--1, - f), utilizing og to generate actions on the fly (see Appendix I). To estimate J (T:; AR. pR),, we compute V^ and VA for each state in the sampled sequence. This approx- imation leads to the following loss function for policy learning
++ Σ
L (E) ppe
Va(si)
Σ 1=1
(10)
We approximate J (((rg) in ( Na. m4) (recall Equation (8))
11
1 F
J ('(TT) Ro E Erg ule, 1.a. Lx H
Σ
vi(+s)
(11)
1.0)
Moreover, the first expression in the policy's loss (10) approximates J(zz)- We backpropagate
through p( n 18--1 --1. using path-wise gradient estimators 2020).
(Mohamed et al.,
5

---

Published as a conference paper at ICLR 2022
Sample models,
Estimate she objjetive/sonetriinng for each model.
largest estimate as an upper bound.
generate trajectories.
08
0:
0.
02
Figure 3: Posterior sampling: We sample j = I,..., NN models 0j ~ ~>(0(D) (e.g., in this illus- tration, N = 5). For each model, we simulate trajectories that are conditioned the same policy and initial state. Objective and constraints: For a given posterior sample O3. we use the simulated trajectories to estimate JTT, Po,) and J' Po,) Vi E {1,...,C) with their corresponding critics. Upper bounds: Choose largest estimate for each of PT,, and (p,, Vi E {I,.... C} among their N realizations.
4.2 ADOPTING OPTIMISM AND PESSIMISM TO EXPLORE IN CMDPS
Optimistic and pessimistic models As already noted by Curi et al. (2020a), greedily maximizing the expected returns by averaging over posterior samples in Equation (4) is not necessarily the best strategy for exploration. In particular, this greedy maximization does not deliberately leverage the epistemic uncertainty guide exploration. Therefore, driven by the concepts of optimism in the face of uncertainty and upper confidence reinforcement learning (UCRL) (Auer et al.. 2009; Curi al., 2020a), we define set of statistically plausible transition distributions denoted by and let Po E "P be particular transition density in this set. Crucially, we assume that the true model p" is within the support of P. and that by sampling @ ~ p() the conditional predictive density satisfies pe P. These assumptions lead to the following constrained problem
p(5 (t. aze 0)
max max (T) s.t. max J' Po.) S d', Vi E (1,....C). Pa:
(12)
TEll PoEP
The main intuition here is that jointly maximizing ( pe) with respect to and pa can lead the agent to try behaviors with potentially high reward due optimism. On the other hand, the pes- simism that arises through the inner maximization term is crucial to enforce the safety constraints. Being only optimistic can easily lead to dangerous behaviors, while being pessimistic can lead the agent to not explore enough. Consequently, we conjecture that optimizing for an optimistic task ob- jective J(x, po) combined with pessimistic safety constraints J J'(x, (o, allows the agent to explore better task-solving behaviors while being robust model uncertainties.
Estimating the upper bounds We propose an algorithm that estimates the objective and con- straints in Equation (12) through sampling, illustrated in Figure 3. We demonstrate our method in Algorithm 1. Importantly, we present Algorithm only with the notation of the task critic. However, use it to approximate the model's upper bound for each critic independently, i.e., for the costs as well as the reward value function critics.
A Algorithm 1 Upper confidence bounds estimation via posterior sampling
Require: N. p)0(())) D), , p(s (( - ---1:+H-11 0). s, 1, π (a,, (st). # Set of objective estimates, under different posterior samples.
1: Initialize V {
2: for to N do
3: 8~ p()(D
# Posterior sampling (e.g., via SWAG).
4: 8r:++H ~ pp(rrr+HHle 1, 11
# Sequence sampling, see Appendix
5: Append V +- V - LI=r (x()))
6: end for
7: return max V.
The LAMBDA algorithm Algorithm 2 describes how all the previously shown components in- teract with each other to form a model-based policy optimization algorithm. For each update step, sample batch of B sequences with length L from replay buffer to train the model. Then, sample N models from the posterior and use them to generate novel sequences with horizon H from every state in the replay buffer sampled sequences.

---

Published as a conference paper at ICLR 2022
Algorithm 2 LAMBDA
1: Initialize D by following random policy or from an offline dataset.
2: while not converged do
3: for u to U update steps do
4: Sample sequences {(rr' 1.-1 Opt:r4 Por (()441) ~ D uniformly.
5: Update model parameters 0 and Φ.
# #.gg,, Hafner et al. (2019a) for the RSSM.
6:
Infer 8y'p'+L ~ qo 4-(-0or:+t, -1r''tb-- Compute Σ (s.),), Σ V (8x) via Algorithm 1. Use each state in +. as an initial state for sequence generation.
7:
8:
Update &ú and gi* via Equation (9) with _
S (s,), and Eier Vi(s))
9:
Update & according Equation (10) with Σ (x())) and Σte Vi(ss).
10: Update X' via Equations (6) and (11).
11:
end for
12:
for to do Infer 8, 4o(-10r, 0,-1, 8t-1).
13:
14:
Sample a₁ (-))).
15:
Take action at, observe It, cf O++1 received from the environment.
16:
end for
Update dataset D - DU fOur, anT, FIT, c.r).
18:
end while
EXPERIMENTS
We conduct our experiments with the SG6 benchmark as described by Ray et al. (2019), aiming to answer the following questions:
How does model-based approach compare to model-free variants in terms of perfor- mance, sample efficiency and constraint violation?
What is the effect of replacing our proposed policy optimization method with an online planning method? More specifically, how does LAMBDA's policy compare to CEM-MPC Liu et al. (2021)?
How does our proposed optimism-pessimisee pessimism formulation compare to greedy exploitation in terms of performance, sample efficiency and constraint violation?
We provide an open-source code for our experiments, including videos of the trained agents at https://g htt.://ww....ooyyyaarree/aaaaaaa
SG6 BENCHMARK
Experimental setup In all of our experiments with LAMBDA, the agent observes 64x64 pixels RGB images, taken from the robot's point-of-view, as shown in Figure 2. Also, since there is only safety constraint, we let Je(T) (T). We measure performance with the following metrics, as proposed in Ray et al. (2019):
Average undiscounted episodic return for episodes: j(T) 1 Σ
E E rt
Average undiscounted episodic cost return for E episodes: j((T) = Σ E Ct
Normalized sum of costs during training, namely the cost regret: for a given number of >-T total interaction steps T. we define pe(=) -- as the cost regret.
We compute (Tr) and Je(TT) by averaging the sum of costs and rewards across E = 10 evaluation episodes of length Tep = 1000, without updating the agent's 's networks and discarding the interac- tions made during evaluation. In contrast to the other metrics, to compute pe(x), accumulated during training and not evaluation episodes. The results for all methods are recorded once the agent reached 1M environment steps for PointGoall, PointGoal2, CarGoall and 2M steps
the costs
7

---

Published as a conference paper at ICLR 2022
LAMBDA
CEM-MPC
PPO-Lagrangias
LRPO-Lagrangian
CPO
PointGoall
Goint
CarGoall
PointButtonI
PointPush1
DoggoCoall
SG6 (avcrage)
0
2
0.0
0.2
0.4
0.6
0.0
0.5
1.0
1.5
J(x)
Jo(x)
Pc(x)
Figure 4: Experimental results for the SG6 benchmark. As done in Ray et al. (2019), we normalize the metrics and denote J((T) (TT) the normalized metries (see also Appendix H). We note that LAMBDA achieves better performance while satisfying the constraints during test time. Furthermore, LAMBDA attains similar result to the baseline in terms of the cost regret metric. We also note that CPO performs similarly LAMBDA solving some of the tasks but fails to satisfy the constraints in all tasks.
for PointButtonl, PointPush2, DoggoGoall enviromments respectively. To reproduce the scores of 'TRPO-Lagrangian, PPO-Lagrangian and CPO, we follow the experimental protocol of Ray et al. (2019). We give further details on our experiment with CEM-MPC at the ablation study section.
Practical aspects on runtime The simulator's 's integration step times are 0.002 and 0.004 seconds for the Point and Car robots, making each learning task run roughly 30 and 60 minutes in real life re- spectively. The simulator's integration step time of the Doggo robot is 0.01 seconds thereby training this task in real life would take about 167 minutes. These results show that in principle, the data ac- quisition time in real life can be very short. However in practice, the main bottleneck is the gradient step computation which takes roughly 0.5 seconds GPU. In total, takes about 18 hours train an agent for 1M interaction steps, assuming we take 100 gradient steps per episode. In addition, for the hyperparameters: in Appendix C, get total of 12M simulated interactions used for policy and value function learning per episode.
single unit of Nvidia GeForceRTX2080T
Results The results of our experiments are summarized in Figure As shown in Figure 4. LAMBDA is the only agent that satisfies the safety constraints in all of the SG6 tasks. Further- more, thanks its model-based policy optimization method, LAMBDA requires only 2M steps to successfully solve the DoggoGoall task and significantly outperform the other approaches. In Ap- pendix we examine further LAMBDA's 's sample efficiency compared the model-free baseline algorithms. Additionally, in the PointGoal2 task, which is denser and harder to solve in terms of safety, LAMBDA significantly improves over the baseline algorithms in all metrics. Moreover, in Figure we compare LAMBDA's ability trade-off between the average performance and safety metrics across all of the SG6 tasks. One main shortcoming of LAMBDA is visible in the Point- Push1 environment where the algorithm fails to learn to push box the goal area. We attribute this failure to the more strict partial observability of this task and further analyse in Appendix D.
5.2 ABLATION STUDY
Unsafe LAMBDA As our first ablation, we remove the second term in Equation (10) such that the policy's loss only comprises the task objective (i.e., with only optimistic exploration). We make the following observations: (1) Both LAMBDA and unsafe LAMBDA solve the majority of the SG6 tasks, depending on their level of partial observability; (2) in the PointButtonl, PointGoall and CarGoall, LAMBDA achieves the same performance of unsafe LAMBDA, while satisfying the safety constraints; (3) LAMBDA is able reach similar performance
unsafe LAMBDA in the PointGoal2 task which strictly harder than the other tasks in terms of safety, as shown in Figure 5. We note that partial observability is present in all of the SG6 tasks due to the restricted field of view
8

---

Published as a conference paper at ICLR 2022
Average rewnnti relurn
20
0.25
Averae cost relurn
15-
200
Cret: regret.
0.20
0.15
10-
100
0.10
0.05
0.0
0.2
0.4
0.6
0.8
0.4
0.6
0.8
1.0
0.0
0.2
0.1
Training steps x10"
Training steps
x10*
Training steps x10"
CPO (10M stepe, proprin)
PPO Lagrangian (10M steps, proprio)
TRPO Lagrangian (10M stepe, proprio)
LAMBDA
Unsafe LAMBDA
Figure 5: Learning curves of LAMBDA and its unsafe version on the PointGoal2 environment. As
shown, LAMBDA exhibits similar performance in solving the task while maintaining the safety constraint and significantly improving over the baseline algorithms.
of the robot: i... the goal and obstacles are not always visible the agent. In the DoggoGoall task, where the agent does not observe any information about the joint angles, LAMBDA is still capable of learning complex walking locomotion. However, in the PointPushl task, LAMBDA struggles to find a task-solving policy, due to the harder partial observability of this task (see Appendix D). We show the learning curves for this experiment in Appendix
Ablating policy optimization Next, we replace our actor-critic procedure and instead of perform- ing policy optimization, we implement the proposed MPC method of Liu al. (2021). Specifically, for their policy, Liu et al. (2021) suggest to use the CEM to maximize rewards over action sequences while rejecting the unsafe ones. By utilizing the same world model, we able directly compare the planning performance for both methods. As shown in Figure 11, LAMBDA performs consid- erably better than CEM-MPC. We attribute the significant performance difference the fact that the CEM-MPC approach does not use any critics for its policy, making its policy short-sighted and limits its ability address the credit assignment problem.
Optimism and pessimism compared to greedy exploitation Finally, we compare our upper bounds estimation procedure with the more common approach of greedily maximizing the expected performance. More specifically, instead of employing Algorithm 1, we use Monte-Carie sampling to estimate Equation (4) by generating trajectories with N sampled models and taking the average trajectory over the samples, By doing so, we get an estimate of the mean trajectory over sampled models together with the intrinsic stochasticity of the environment. We report our experiment results in Appendix G and note that LAMBDA is able to find safer and more performant policies than its greedy version.
6 CONCLUSIONS
We introduce LAMBDA, Bayesian model-based policy optimization algorithm that conforms to human-specified safety constraints. LAMBDA uses its Bayesian world model to generate trajectories and estimate optimistic bound for the task objective and pessimistic bounds for the constraints. For policy search, LAMBDA uses the Augmented Lagrangian method to solve the constrained optimization problem, based on the optimistic and pessimistic bounds. In our experiments we show that LAMBDA outperforms the baseline algorithms in the Safety- -ym benchmark suite terms of sample efficiency as well as safety and task-solving metrics. LAMBDA leams its policy directly from observations in an end-to-end fashion and without prior knowledge. However, we believe that introducing prior knowledge with respect to the safety specifications (c.g., the mapping between state and its cost) can improve LAMBDA's performance. By integrating this prior knowledge, LAMBDA can potentially learn policy that satisfies constraints only from its model-gae generated experience and without to a notable open question on how to integrate this prior knowledge within the world model such that the safety constraints satisfied during learning and not only the end of the training process.
violating the constraints in the real world. This leads

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme grant agreement No 815943, the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545 and the Swiss National Science Foundation, under grant SNSF 200021 172781.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization, 2017.
E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, 2016.
Peter Auer and Ronald Ortner. Logarithmic online regret bounds for undiscounted reinforcement learning. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Pro- cessing Systems, volume 19. MIT Press, 2007. URL https: /ptettps//prrceeed ngs neurips.
co/paper/2006, http:///////...oooooaaaaaaaaaaa. Pwwpaddss366---ppppppppp pdf.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for re- inforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 21. Curran Asso- ciates. Inc., 2009. URL https: http:///ppooeeeegg.eeeeeee. wwwpuurpssccppppp.eeeeee cc/ppaper/200//iil/ pdf.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees, 2017.
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety critics for exploration, 2021.
Ronen I. Brafman and Moshe Tennenholtz. R-max a general polynomial time algorithm for near- optimal reinforcement learning. J. Mach. Learn. Res., 31mll):2133233 March 2003. ISSN 1532-4435. doi: 10.1 10.166//15324433765888883 1532443037 012803365508877...
URL https: :/ts::///ol/ http://c....r/..1111.e1. 1266//
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. CoRR, abs/1512.01629, 2015. URL http:
01629.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov- safe policy optimization for continuous control, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, abs/1805. 14, 2018. URL http://///// / aaaiiviiii. org/ abs/1805 12114.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus), 2016.
Sebastian Curi, Felix Berkenkamp, and Andreas Krause, Efficient model-based reinforcement learn- ing through optimistic policy search and planning. 2020a.
Sebastian Curi, Silvan Melchior, Felix Berkenkamp, and Andreas Krause. Structured variational inference in partially observable unstable gaussian process state space models. Learning for Dynamics and Control, pp. 147-157. PMLR, 2020b.
Sebastian Curi, Ilija Bogunovic, and Andreas Krause. Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning, 2021.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces, 2018.
Mare Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient ap- proach to policy search. Proceedings of the 28th International Conference on International Conference Machine Learning, ICML'11, pp. 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 978 0781450306195.
Stefan Depeweg, José Miguel Kernández-Lobatee Finale Doshi- Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning, 2017.

---

Published as a conference paper at ICLR 2022
Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A bayesian approach to robust reinforcement learning, 2019.
Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploiaatinn in constrained mdps, 2020.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. 2017.
Javier García, Fern, and Fernandez. comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015. URL http: jmlr....... .org/ papers/v16. http///gariaaaaaaaaaaaaa html.
Alex Graves. Practical variational inference for neural networks. In J. Shawe- Taylor, R. Zemel, P. Bartlett, Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
URL https::/ppooceeiins..eeee neurips httpr/ps..cpppppr//////LLLL. pdf.
3d4 aeb.cceeeea11eeeeeaaaaaaaaaaaaa
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications, 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp.
2019a.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to trol: Learning behaviors by latent imagination. CoRR, abs/1912.01603, 2019b. URL http: / //arxiv.org/ abs 1912.1122 01 603.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning-based model predictive control for safe exploration. 2018 IEEE Conference on Decision and Control (CDC), pp. 6059-6066. IEEE, 2018
Dirk P. Kroese, Sergey Porotsky, and Reuven Y. Rubinstein. The cross-entropy method for tinuous multi-extremal optimization. Methodology and Computing in Applied Probability, (3):883-407, Sep 2006. ISSN 1573-7713. doi: 10. 10.1007/s1 10.107//110099-0--95553 URL https:
00000/00000010000000000000000000 006- 0009999----0999--
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
Jinggi Li, David Fridovich-Keil, Somayeh Sojoudi, and Claire J. Tomlin. Aug- mented lagrangian method for instantaneously constrained reinforcement learning prob- lems. 2021. URL / eecs. berkeley. edu/ -so acdu/=sojcdu/AAuue /Augmented
Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Con- strained mosel-based reinforcement learning with robust cross-entropy method, 2021.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning, 2019.
12

---

Published as a conference paper at ICLR 2022
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti- mation in machine learning, 2020.
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. CoRR, abs/1205.4810 2012. URL http::/aaxii..rrrreeee/// www.or/////a//aeeee 4810.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML'10 10, pp. 807,814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
Arnab Nilim and Laurent Ghaoui Robust Control of Markov Decision Processes with Uncertain Transition Matrices. Operations Research, 53())780-779 October 2005. reper. a/ imm/oropre,
doi: 10.
URL
v53y200515p 798 html.
Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer, New York, NY, USA, second edition, 2006.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22: 400-407, 2007.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning, 2013.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control, 2019.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction, 2021.
Kim P. Wabersich and Melanie N. Zeilinger. A predictive safety filter for learning- -baed control of constrained nonlinear dynamical systems, 2021.
Moritz Zanger, Karam Daaboul, and J. Marius Zöllner. Safe continuous control with constrained model-basee policy optimization, 2021.
13

---

Published as a conference paper at ICLR 2022
A
LEARNING CURVES FOR THE SG6 BENCHMARK
Average rocand retura
Averagg coet rotuen
0.05
201
100
CarGoal1
0.06
Coal
x100
X10P
Aveesuo not miton
reword
12
PaintPoshl
Cust:
0.04
0
K/4
0.02
x100
x10P
x10
Avenge reward neturn
Average roe retrams
0.08
20
150
Coel. regrat
0.06
DoggoGooll
100
10
50
0.04
0.02
Averago rewent petar
Avenge end atum
150
Clal agget
PaintGoo22
100-
0.8
02.2
11.6
0.88
300
Averog preaed mttun
Averagi returrs
15.
0.20
PointButoul
10
Ouod
100
0.10
0.05
0.5
1.0
2,0
0.0
0.5
1.0
1.5
2.0
0.5
1.0
2.0
x10
Average rewent rearr
Avernge roturn
0.08
150-
Del magat
PointGroll
10
50
0.04
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0..5
0.8
1.0
0.0
10.2
0.4
0.6
0..
1.0
Training stepp
x110
stepp
Training stege
x10
LAMBDA
CPO (progerko)
PPO Lagisngion (proprio)
Lagrangaan (proprio)
Figure 6: Benchmark results of LAMBDA. Solid red lines indicate the threshold value d 25. Dashed lines correspond to the benchmark results for the baseline algorithms after 10M training steps for all tasks except the DoggoGoall, which is trained for 100M environment steps, in Ray et al. (2019). Shaded areas represent the 5% and 95% confidence intervals across 5 different random seeds.
14

---

Published as a conference paper at ICLR 2022
B SAMPLE EFFICIENCY
We record LAMBDA'S 's performance (zr at the end of training and find the average number of steps (across seeds) required by the baseline model-free methods to match this performance. We demon- strate LAMBDA'S 's sample efficiency with the ratio of the amount of steps required by the baseline methods and amount of steps the end of LAMBDA's 's training. As shown in Figure 7, in the major- ity of tasks, LAMBDA is substantially more sample efficient. In the PointPushl task LAMBDA is outperformed by the baseline algorithms, however we assign this the partial observability of this task, as further analyzed in Appendix D. It is important to note that by taking LAMBDA'S 's perfor- mance at the end of training, we take conservative approach, convergence can occur with many less steps, shown in Figure 6.
Required amount of steps to reach LAMBDA's performance
PPO-I agrangian
XX
TRPO-Lase TRPO-Lagrangian
Steps ratio
CPO
10x
X
X
1x
PointGooal
Goi1t
CarGioal
PointButton|
PointPushi
DoggoGoall
Figure 7: LAMBDA solves most of the tasks with significantly less interactions with the envi- ronment, compared to the baseline model-free methods. Green check marks and red "x" indicate constraint satisfaction after reaching the required number of steps.
15

---

Published as a conference paper at ICLR 2022
C HYPERPARAMETERS
In this section we specify the most prominent hyperparameters for LAMBDA, however we encour- age the reader visit https: /github.. com/ www.dyadddaaaaaa mbda holds many specific (but important) implementation details. Table summarizes the hyperparameters used in our algo- rithm.
World model As mentioned before, for our world model we use the RSSM proposed in Hafner et al. (2019a). Therefore, most of the architectural design choices (e.g., specific details of the con- volutional layers) are based on the work there.
Cost function We exploit our prior knowledge of the cost function's 's structure in the Safety-Gym tasks. Since know a-priori that the cost function is implemented neural network approximation for it is modeled as binary classifier. As training progresses, the dataset becomes less balanced as the agent observes less unsafe states. To deal with this issue, we give higher weight to the unsafe interactions in the binray classification loss.
indicator function,
Posterior over model parameters sumption of fixed dataset D, such that weight averaging takes place only during the last few training epochs. To adapt SWAG to the dataset's distributional shifts that occur during training in RL, use exponentially decaying running average. This allows within early stages of learning, as opposed to the original design of SWAG that normally uses the last few iterates of SGD construct the posterior. Furthermore, use cyclic learning rate (lz- mailov et al., 2019; Maddox et al., 2019) to help SWAG span over different regions of the weight space.
supervised learning, SWAG typically used under the as-
maintain posterior
Policy The policy outputs actions such that each element in is within the range [-1, 1]. How- ever, since we model Ie (e(aa)))) 8;) as N (aa;; NN (s:), NNŞ (s:)). we perform squashing of the Gaus- sian distribution into the range [-1, 1], as proposed in Haarnoja et al. (2019) and Hafner et al. (2019b) by transforming it through tangent hyperbolic bijector. We scale the actions in this way so that the model, which takes the actions as an input is more easily optimized. Furthermore, to improve numerical stability, the standard deviation term NN! (sx) is passed through a softplus function.
Value functions To ensure learning stability for the critics, we maintain a shadow instance for each value function which is used the bootstrapping Equation (9). We clone the shadow instance such that it lags U update steps behind its corresponding value function. Furthermore, we stop gradient on Vx when computing the loss function in Equation (9).
General Parameter update for all of the neural networks is done with ADAM optimizer (Kingma & Ba, 2014). We use ELU (Clevert et al., 2016) as activation function for all of the networks except for the convolutional layers in the world model in which ReLU (Nair & Hinton, 2010).
16

---

Published as a conference paper at ICLR 2022
Table 1: Hyperparameters for LAMBDA. For other safety tasks, we recommend first tuning the initial Lagrangian, penalty and penalty power factor at different scales and then fine-tune the safety discount factor to improve constraint satisfaction. We emphasize that it is possible improve the performance of each task separately by fine-tuning the hyperparameters on per-task basis.
Name
Symbol
Value
Additional
World model
Batch size
B
32
Sequence length
L
50
Learning rate
le-4
SWAG
Burn-in steps
500
Steps before weight averaging starts.
Period steps
200
Use weights every 'Period steps' to update weights running average.
Models
20
Averaging buffer length.
Decay
0,8
Exponential averaging decay factor.
Cyclic LR factor
5.0
End cyclic Ir period with the base LR times this factor.
Posterior samples
N
Safety
Safety critic learning rate
2e-4
Initial penalty Initial Lagrangian
14g
5e-9
A/
le-6
Penalty power factor
le-5
Multiply Pe by this factor each gradient step.
Safety discount factor
0.995
General
Update steps
U
100
Critic learning rate
8e-5
Policy learning rate
8e-5
Action repeat
Repeat same action for this amount of steps.
Discount factor
0.99
TD(A) factor
0.95
Sequence generation horizon
H
15
17

---

Published as a conference paper at ICLR 2022
D POINTPUSHI ENVIRONMENT WITH A TRANSPARENT BOX
Figure 8: PointPushl with partially transparent box. When the color of the box is solid LAMBDA struggles in solving the task due to occlusion of the goal by the box. By changing the transparency of the box, we make the PointPush1 task less partially observable and thus easier to solve.
In all of our experiments with the PointPushl task, we provide the agent image observations in which the box is observed as a solid non-transparant object. As previously shown, we note that in this setting, LAMBDA and unsafe LAMBDA fail learn the task. We maintain that this failure arises from the fact that this task is significantly harder in terms of partial observability, the goal is occluded by the box while the robot pushes the box.' Furthermore, in Ray et (2019), the authors eliminate issues of partial observability by using what they term "pseudo-LiDAR" which is not susceptible to occlusions as it can see through objects. To verify our hypothesis, we change the transparency of the box such that the goal is visible through it, shown in Figure 8 and compare LAMBDA'S performance with the previously tested PointPush1 task. We present the learning curves of the experiments in Figure 9. As shown, LAMBDA is able safely solve the PointPushl task if
returr
150
0.10
Codl rughtt
0.08
Average reward
10
100
0.06
50
0.01
0.02
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Training strpe
x10
Training stegs
X10
Training stops
X10
CPO (10M steps, propeio)
PPO Lagrongian (10M steps, proptio)
TRPO Lagrangian (10M steps, propeio)
Transpurent box
Opaque
Figure 9: Learning curves for the PointPushl task and ObservablePoindPushl task which par- tially transparent box. We also show the baseline algorithms performance with a "pseudo-LiDAR" observation.
the goal is visible through the box. We conclude that the PointPushl task makes an interesting test case for future research on partially observable environments.
Please https: / thub. con cor http:/yyaddas///// a-mbda for video illustration.
18

---

Published as a conference paper at ICLR 2022
E
UNSAFE LAMBDA
300
Averuge sorard
200
Coo regrel
PuintDetion
0.1
1.5
1.5
1.0
1.5
2.0
X109
X10
XIO
0.08
remard
40-
Cort reteet
0.06
DeggoGool
100
20-
50
0.0
1.0
1.5
2,0
0.0
1.0
1.5
2.0
1.0
1.5
2.0
X10
X10
X10
20
zerard
15
Cos re02e5
10
Average
50
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
MI1
X10
X10
2.5
Crt neteet
0.00
PuintPush?
-2.5
-5.0
Mm M Mmmn
0.02
0.00
0.5
1.0
1.5
2.0
0.00
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
X100
X10
XIC"
Antrage peward
15
Coe regee
0.100
CarGoal1
10
Average
0.075
0.025
01.01
0.2
11.4
00.05
11.88
1.0
11.0
0.22
01.4
10.6
11.88
1.11
0.00
0.2
0.4
0.6
0..8
1.0
X100
X10F
X10F
0.31
20
Ancing renard
15-
Cod regat
0.2
PointGooi2
10
0.1
0.2
0.4
1.0
0.2
0.4
0.8
1.0
0.2
0.4
0.8
Training
x10
Training
X10F
Training
X10F
LAMBDA
Unsade LAMBDA
Figure 10: Benchmark results of LAMBDA and its unsafe implementation. In the majority of the tasks, LAMBDA is able find policies that perform similarly to the unsafe version while satisfying the constraints. Interestingly, apart from the DoggoGoall and PointGoal2 tasks LAMBDA's 's policies able achieve similar returns while learning to satisfy the constraints.
19

---

Published as a conference paper at ICLR 2022
F COMPARISON WITH CEM-MPC
Average revard meturn
20
Average rest returs
15
100
0.08
CarGool1
Clol
6-
0.04
0.0
0.8
0.0
0.2
0.6
x100P
K100
X10P
Aviragy roand teturn
Average coet return
PointPushl
Ciol
0.04
0.02
x100
x100
x10
Average cranad rsturn
15
Averag not muum
150
Cuel regess
/M
DoggoGoolt
100
0.06
50
0.94
0.022
x100*
100
Aveage roward eeturm
Avernge coet neam
200
150-
Cool regatt
PoinsGral2
100
0.
300
Avenge nntm
7.5
PoinButtonl
Averagr
2.5
200
Cool
0.10
0.05
6.5
2,0
0.0
1.0
1.5
0.5
Averge proeed eetura
20
Averggg searr
200
0.08
15
150
PoinsGoal1
Coo
50
0.04
0.02
0.0
0.2
0.4
E.G
0.8
1.0
0.00
0.2
0.4
11.1
0.8
1.0
0.0
12.2
0.4
010
03.8
3.0
Training steps
Troining steps
100
Training stege
Figure 11: Comparison of LAMBDA when ablating the policy optimization and using CEM-MPC with rejection sampling as introduced in Liu et al. (2021). As shown, LAMBDA performs substan- tially better than CEM-MPC. We believe that when the goal is not visible to the agent, CEM-MPC's policy fails to locate it and drive the robot to it. On the contrary, in our experiments, we observed that LAMBDA typically rotates until the goal becomes visible, thus allowing the robot to gather significantly more goals.
20

---

Published as a conference paper at ICLR 2022
G
OPTIMISM AND PESSIMISM COMPARED TO GREEDY EXPLOITATION
Averay rovant return
Average coet return
10-
CerGrall
10
0.06
Cool
0-
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.4
0.8
0.0
0.2
0.4
0.6
0.8
1.0
X110
X100
x10
Aweagg muard witon
1
Averogh rest returs
75-
0.00
PolntPushi
Oool
0.5
0.0
1.5
x100P
K100
100
Average tewent petaru
201
Avonage cost return
150
Cbol reged
0.06
DegneGrod
10
100
0.04
50-
0.022
x100
x10P
x100
Average mosel estur
Averngg teturn
10
0.20
150
Col regget
PointGoo12
100
0.15
0.10
300
Avenge whmm
7.5
MM
PoinButtonl
5.0
Awtrage
2.5
200
Cool
0.10
0.05
0.5
2,0
0.0
1.0
1.5
2.0
0.5
2.0
100
x10F
Avemge noweed retur
20
Avernge codt peam
200
150
Cood regest
PointGoall
20
0.06
50-
0.04
0.2
0.48
0..
0.8
1.0
0..0
0.2
0.4
0..5
0.8
1.0
0.0
12.2
0.4
0.0
0..8
3.0
Training steps
x11
steps
x10P
Training steps
x10
LAMBDA
Greedy LAMBDA
Figure 12: Comparison of LAMBDA and greedy exploitation. LAMBDA generally solves that tasks with better performance and with improved sample efficiency. Notably, the greedy exploitation variant fails solve the the PointButton| task.
21

---

Published as a conference paper at ICLR 2022
H COMPARING ALGORITHMS IN THE SG6 BENCHMARK
To get a summary of how different algorithms behave across all of the SG6 tasks, we follow the proposed protocol in Ray al. (2019). That is, we obtain "characteristic metries" for each of the environments by taking the metrics recorded at the end of training of an unconstrained PPO agent (Schulman et al., 2017). We denote these metrics as jPPo, jPPP and pe.0 We then normalize the recorded metries for each environment as follows:
(
J(T) JPPO
Je(T) max(0, ]T( d)
(13)
10 max d)
Pe(
Pe(T)
pePo
By performing this normalization with respect to the performance of PPO, we can take an average of each metric across all of the SG6 environments.
To produce Figure 1, we scale the normalized metrics to [0, 1]. such that for each metric, the best performing algorithm attains score of 1.0 and the worst performing algorithm attains score of 0.
22

---

Published as a conference paper at ICLR 2022
I BACKPROPAGATING GRADIENTS THROUGH A SEQUENCE
Algorithm 3 Sampling from the predictive density pa (8 ar- -1
Require: कh. (a. [8)), (o)))pp((0 (8 1t 0z), 87-1
1: for - to + H do
2: a: ~ (=-m((ssop-grraiinn(ee)
# Stop gradient from s; when conditioning the policy on it.
3: 8:41 p(,
4: end for
5: return 8T.++
We use the reparametrization trick (Kingma & Welling, 2014) to compute gradients through sam- pling procedures as both тH and pa are modeled as normal distributions. Backpropagating gradients through the model can be easily implemented with modern automatic differentiation tools.
Importantly, we stop the gradient computation in Algorithm 3 when conditioning the policy on 8:-1- We do to s, such that eventually, backpropagation to actions occurs only from their dependant succeeding values. We further illustrate this in Figure 13.
avoid any recurrent connections between action a;-1 and the preceding states
a
a2
a:-1
a2
81-1
S++1
8
$++1
(a) Forward pass of trajectory sampling. We stop gra- dient flow on the magenta dashed arrows.
(b) Backward pass from values to their inducing tions.
Figure 13: Computational graphs for the backward and forward passes of Algorithm 3.
23

---

Published as a conference paper at ICLR 2022
J SCORES FOR SG6
The jsoon format files, summarizing the scores for the experiments of this work are available at https://gitte com/yardeaa///s/aaaa mbda.
Table 2: LAMBDA's unnormalized scores for the SG6 tasks.
(xx
Jx(
Po(TT)
PointGoall
18.822
11.200
0.034
PointGoal2
13.300
9.100
0.043
CarGoall
16.745
23.100
0.036
PointPush1
0.314
21.400
0.017
PointButtonI
5,372
21.700
0.038
DoggoGoall
5.867
11.400
0.046
Average
10.07
16.317
0.0360
Table 3: Experiment results for the SG6 benchmark. We present the results with the tuple (TT) the normalized metrics.
TRPO-Lagrangian
PPO-Lagrangian
CPO
LAMBDA
PointGoall
0.51, 0.004, 0.405
0.24, 0.0, 0.419
0.898, 0.302, 0.599
1.077, 0.0. 0.483
PointGoal2
0.119, 0.059, 0.304
0.09, 0.197, 0.349
0.306, 0.132, 0.377
0.902, 0.0, 0.229
CarGoall
0.501, 0.0, 0.522
0.255, 0.0, 0.474
1.579, 0.604, 0.924
1.284, 0.0, 0.704
PointPush1
0.714, 0.0, 0.315
0.185, 0.0, 0.249
1.606, 0,311, 0.687
0.203, 0.0, 0.309
PointButton1
0.077, 0.0, 0.223
0.058, 0.0, 0.242
0.516, 0.343, 0.495
0.287, 0.0, 0.302
DoggoGoall
-1.257, 0.227, 0.624
-0,891, 0:293, 0.707
-0.723, 0.643, 0.769
5.400, 0.0, 0.770
SG6 (average)
01111, 0.048, 0.399
-0.011, 0.082, 0.407 0.697, 0.389, 0.642
1.526, 0.0, 0.466
24


Reviews:
[{"review_id": "oBvSH-8L6ht", "reviewer": "Reviewer_Z22c", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The paper is very well written and easy to follow.\n\n- I believe that using model-based approaches to increase sample efficiency in safety-sensitive applications is very relevant. I also find interesting the use of a formulation for the transition model that is optimistic w.r.t. to the main objective while being pessimistic in terms of constraint satisfaction to be a very interesting idea.\n\n- The proposed algorithm and design choices are clearly presented and motivated.\n\n- The related work section is well organised and seemed sufficient to me.\n\n- The experiments are in my opinion sufficient in volume and in supporting the claims made by the authors.\n\n- The code for the experiments is publicly available online and many implementation details and hyperparameters are listed in the appendix which should allow for reproducibility of this work.\n\n## Questions and weaknesses\n\n1. The Augmented Lagrangian method presented in Section 4.1 is described in the Methods sections for the proposed algorithm (LAMDBA) rather than in the background section. Are the authors framing the use of Augmented Lagrangian for solving CMDPs (as opposed to the regular Lagrangian relaxation) as a contribution of this work? Was this version of Lagrangian method used in previous work on Constrained RL? If it has been used before, I believe citations to such previous work should be added. If this is presented as a contribution, I believe it would have been interesting to show comparisons between the augmented and regular lagrangian methods in the experiment section.\n\n2. I believe that it would be more clear and impactful to show the comparison of sample efficiency between LAMBDA and the model-free algorithms in a table rather than simply in text (section 5.1). It is in my opinion one of the most important claims and results of the presented work and thus these results should be very easily retrievable when going through the manuscript. \n", "summary_of_the_paper": "The authors present LAMBDA, a novel algorithm for learning in CMDPs. The approach seeks to improve on the sample efficiency of previous algorithms by learning a probabilistic model of the environment. This model is then used to train a policy to maximise the return and respect the constraints by backpropagating through imaginary trajectories. The uncertainty of the model can further be used to steer the agent towards for optimistic or pessimistic behavior. The method is benchmarked against one online planning method and previous model-free approaches on the SG6 set of tasks of the Safety Gym environment. The results show that the proposed method compares favourably to competing algorithms, particularly in terms of constraint satisfaction and sample efficiency.", "summary_of_the_review": "I believe that this is a good paper making interesting and important contributions to the area of constrained RL and that it is ready to be shared with the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dE5E1UbUI2m", "reviewer": "Reviewer_HHZf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I will summarize the Pros and Cons of the paper entitled with \"Constrained Policy Optimization via Bayesian World Models.\"\n\n### Pros\n- New and promising paradigm. Utilizing Bayesian world models for CMDPs is indeed powerful tool for RL community. Since humans would decide their behavior using such way of thinking in real life; thus, the idea for using Bayesian world models for CMDPs would be worthwhile to study.\n- Experiments. The effectiveness of the proposed method is clearly demonstrated in SG6 tasks in Safety-Gym compared with reasonable baselines.\n- Well-written texts. It is easy to understand the key ideas in this paper. Also, related work is fully surveyed and introduced.\n- Source-code is nice. If it is open-sourced, it would be quite helpful for the research community. (I just read through it. I did not run the code on my environment.)\n\n### Cons\n- Given there are many papers on Bayesian world models for \"unconstrained\" MDP, the ideas in this paper are little-bit straight forward. Incorporating optimism for reward and pessimism for safety is not a new idea, which is not an essential contribution.\n\n### Questions\nAlthough this paper is clearly presented and I don't have many questions, there are several unclear points.\n- Q1: The world model (i.e., RSSM) is trained from scratch in the authors' experiments? My understanding is that, in the experiments presented in the paper, the world model is trained from scratch. If a pre-trained model is obtained, the training of the agent will become much safer. Is my understanding correct?\n- Q2: Though it may be trivial, necessary assumptions should be clearly listed. For example, do we need slater condition or strong duality for ensuring LAMBDA's training is successful?\n\n### Typos\n- Page 3: CMPDs --> CMDPs", "summary_of_the_paper": "This paper proposed a method called LAMBDA for optimizing an agent's policy in constrained Markov decision processes. LAMBDA is a model-based approach leveraging Bayesian world models, which deals with reward optimistically and safety pessimistically. The authors demonstrate the effectiveness of SG6 tasks in Safety-Gym in terms of sample efficiency and satisfaction of safety constraint(s).", "summary_of_the_review": "I think this paper is well-written and the key ideas would be useful for the research community. Though I have a minor concern regarding how much contribution exists for extending unconstrained MDP to constrained MDP settings, the overall contribution of this paper is sufficient; hence, I vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QjEOZu4XoO", "reviewer": "Reviewer_E48X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Review of: CONSTRAINED POLICY OPTIMIZATION VIA BAYESIAN WORLD MODELS\n\nThe paper presents an argument that model-based RL should have special value in situations when safety and efficiency is a concern. The paper raises interesting questions, but ultimately cannot compellingly make the case for the proposed approach. Specifically, the suitability of the model would seem to be a critical issue. However, the paper does not systematically address where such a model would come from, or how errors in the supplied model might affect performance. This is a critical issue in differentiating between model-free and model-based RL. Also, the argument in the paper lacks structure in a way that renders the contribution unclear. What is the relationship between the contributions outlined at the beginning of the paper and the questions outlined in the experiment section? Which aspects of the proposed approach are novel? How does the novelty of the proposed approach contribute to performance? Ultimately the lack of a clear source for the model that explains why performance may be improved is a critical limitation for the current work. \n\nMinor comments:\n- Figure one is unhelpful. The vertical axis is not labeled and three different scale metrics are plotted side by side. Also, what is SG6 and why should we care about it? \n- It is unclear how one could experience unsafe events through model-generated trajectories, unless one could already solve the problem.\n- The contributions do not make sense in the context of the introduction. Why are these problems to be solved? \n- When you say \"harness\" what does that mean? \n- \"Curi et al. (2021) and Derman et al. (2019) also take\na Bayesian optimistic-pessimistic perspective to find robust policies. However, these approaches do not use CMDPs and generally do not explicitly address safety.\" Did you compare with them? \n- \"revolves around the repetition\"\n- The definition of model based reinforcement learning could be much more precise\n- Please introduce acronyms prior to use \n- \"MBRL achieves superior sample efficiency compared to its model-free counterparts\" This claim could be more precise. \n- \"We utilize\"\n- Please justify why we should consider the predictive distribution to be differentiable. \n- \"Thus, it is possible ...\" repetitive. \n- \"in the model, to identify\" --> \"in the model to identify\"\n- The usefulness of uncertainty quantification depends on the degree to which the model is accurate with respect to the world. \n- The logic of the experimental design and relation to the main questions was difficulty to understand given the high degree of overlap between the current method and previous work. ", "summary_of_the_paper": "The manuscript introduces LAMBDA a Bayesian model-based policy optimization algorithm that adheres to supplied safety constraints. The approach relies on the model to simulate trajectories and therefore improve efficiency of learning and effectiveness of safety. Experiments on SG6 compare the proposed algorithm to previous approaches and ablations of LAMDA. Based on experimental results, authors conclude that LAMDA is more efficient and effective for learning and safety. \n", "summary_of_the_review": "Interesting paper focusing on model-based RL that does not address where the model comes from or the consequences of misspecification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "66R00yN6NPH", "reviewer": "Reviewer_11DZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper this paper is building on a lot of different recent ideas. While I'm well versed in CMDPs and constrained optimization in general, I had to do some additional reading to understand the remaining pieces. This is not a bad thing, on its own and in fact your paper made me learn about a lot of new ideas. Thanks for that!\n\nA couple of points: your decision to use SWAG, is interesting, especially from a computational perspective. Now from what I undertand from the original paper, this may a give you a crude approximation to the true posterior. What do we know about SWAG in the RL context? Does it still give you the right approximation? I presume that because your applying SWAG at the level of the dynamics themselves, you are probably dealing with a supervised learning problem, in which case prior results still hold. Is that the case? Because otherwise, I may see some potential complications when using SWAG in a setting where you do derivative estimation (reparameterization or score function) due to the additional estimation noise. \n\nWhen I look over your experiment section, I would have like to seem some results on this component alone. I would want you to show me that your SWAG approximation does work. You must have gone through this exercise yourself when developing your approach and coding it up, so I'm sure you thought of a way to assess it. \n\nRegarding the optimization method itself, I'm quite happy with your decision to go for an augmented lagrangian approach. However, I would have like you again to weight the pros and cons of this method, and convince me empirically that it's the right choice. A naive baseline here would be to do gradient descent-ascent/first order Lagrangian method/Arrow-Hurwicz; a fancier one would be SQP/interior point method. I was a bit confused too when I first read your algorithm description because it doesn't look like a typical augmented Lagrangian method due to the fact that you use a quadratic penalty for the magnitude of the lagrange multipliers themselves and not for the constraints themselves. I had to open up Nocedal and search for the exact algorithm that your are describing (p. 523 \"unconstrained formulation\" in my copy). There I found a description that looks a lot like what you have, but with better explanation as to how you obtain this (non-standard) variant on the augmented lagrangian. Also interesting note in Nocedal: \"Unlike the bound-constrained and linearly constrained formulations, however, this unconstrained formulation is not the basis of any widely used software packages, so its practical properties have not been tested.\", which explains my confusion. \n\n", "summary_of_the_paper": "This paper is about solving CMDPs in an uncertainty-informed model-based fashion, without going through the usual LP route. The blueprints for this paper is the following: \n\n1. Take a standard CMDP formulation\n2. Express the objective and constraints in a UCRL (Auer, 2009) fashion (joint maximization over the set of possible dynamics),\n3. Represent and learn the dynamics as a  Recurrent State Space Model from Hafner et al. (2019)\n4. Estimate the objective and constraints themselves  (which involve a max operator) by sampling, and where the Bayesian posterior over the \"world model\" parameters is computed by SWAG (Maddox et al. 2019), where you average out the iterates generated over a trajectory of stochastic gradient descent \n5. Using 3 and 4, compute a step of Augmented Lagrangian (a variant of) to solve the CMDP in 2.\n\nThe authors provide motivation as to why the UCRL aspect is important, and how the underlying optimism principle gives rise to persimism in the constraints, which is important from a \"safety\" perspective. This gives rise to an interesting tension (perhaps more \"balance\") between optimism in maximizing the objective while remaining cautions in abiding to the constraints. \n", "summary_of_the_review": "I like the set of methods proposed by the authors, although motivation is lacking for why the particular sub-components have been chosen. This seems like a believable/feasible approach to Bayesian RL, and it inspires me to try things out along those lines. The criticism in my main review is about performance assessment of the SWAG component as well as the lack of motivation for the specific  (non-standard, but interesting!) optimization method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a new model-based RL technique for constrained MDPs based on Bayesian world models.  It improves sample efficiency and safety. The reviewers are unanimous in their recommendation for acceptance.  This represents an important advance in RL.  Great work!
Decision: Accept (Spotlight)