Exploring Under Constraints with Model-Based Actor-Critic and Safety Filters

Authors: ahmed  agha
baris  kayalibay
atanas  mirchev
patrick  van der smagt
justin  bayer
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:32:35

Abstract: Applying reinforcement learning (RL) to learn effective policies on physical robots without supervision remains challenging when it comes to tasks where safe exploration is critical. Constrained model-based RL (CMBRL) presents a promising approach to this problem. These methods are designed to learn constraint-adhering policies through constrained optimization approaches. Yet, such policies often fail to meet stringent safety requirements during learning and exploration. Our solution ``CASE'' aims to reduce the instances where constraints are breached during the learning phase. Specifically, CASE integrates techniques for optimizing constrained policies and employs planning-based safety filters as backup policies, effectively lowering constraint violations during learning and making it a more reliable option than other recent constrained model-based policy optimization methods.

Main paper: 
 introduction: 
 Many real-world robotic systems can be effectively modeled as constrained Markov decision processes (CMDPs) [1] , particularly in contexts where safety is paramount, and robots must adhere to specific conditions while learning to accomplish tasks. In addition, CMDPs offer a structured framework for injecting inductive biases into the policy optimization process, thereby reducing the number of interactions required to learn effective policies. Thus, developing deep RL algorithms for solving CMDPs has the promise of unlocking RL's potential across various real-world robotic applications where safety is an issue. However, applying RL to systems that must operate under constraints all the time, even as they explore and learn, remains an open challenge.Model-free constrained policy optimization methods have primarily adapted actor-critic techniques to the constrained setting, employing Lagrangian relaxation, such as the augmented Lagrangian method, to train constrained policies. While these algorithms are appealing for their relative simplicity, they face significant challenges, particularly in terms of sensitivity to Lagrangian multipliers [2] . Moreover, model-free RL approaches for CMDPs often suffer from high sample complexity, making them ill-suited for constrained environments where a high number of interactions during Our Contributions In this work, we combine constrained model-based policy optimization with a planning-based safety filter that acts as a backup policy to minimize constraint violations during exploration. In addition, we introduce modifications to the constrained model-based policy optimization training to ensure stable training. We also modify the safety filter's objective to consider the behavior of the constrained base policy during planning. We evaluate our method on constrained tasks from the Omnisafe [9] , RWRL [10] , and the safe-control-gym [11] benchmarks and show that our combination of constrained policy optimization and planning can lead to significantly reducing constraint violations during training in comparison with other model-free and model-based methods. 
problem setting: 
 We consider a CMDP, which is defined by the tuple (S , A , p , ρ 0 , r , γ , c , γ safe ). S and A are the state and action spaces respectively. The discount factor and the safety discount factor are represented by γ and γ safe . The dynamics of the system are represented by p(s t+1 | s t , a t ), and the initial state distribution is represented by ρ 0 . The reward function is represented by r(s t , a t ), and the constraint cost function is represented by c(s t ). In this work, we consider constraint costs represented with the indicator function I(s t ) where the unsafe state is represented as S unsafe = {s t | I(s t ) = 1} and the safe states are represented as S safe = {s t | I(s t ) = 0}.The challenge in CMDPs is maximizing the performance of the agent while satisfying constraintsEQUATION)where l is a problem-specific threshold.In constrained model-based RL, we use transition tuples in the form of {s t , a t , r t , c t , s t+1 } to learn a transition model p. The transition model can then be used in online planning as done in [12, 13] , or by amortizing decision-making by offline training of a parametric policy using an actor-critic approach as in [14, 15] .Recent methods [3, 4] have shown the effectiveness of MBRL in solving CMDPs as formulated in Eq. ( 1). Both methods adapt previous model-based RL methods to the constrained setting; LAMBDA [3] is based on Dreamer [14] , where the RSSM model from PLANET [13] is used as a differentiable simulator for on-policy actor-critic training to learn a policy π θ and a critic v π ;Figure 1 : Overview of CASE. A dynamics model is used to train a constrained policy, a critic, and a constraint critic. The task policy is used alongside a safety filter to ensure low constraint violation rates during exploration.safeSLAC [4] is based on SLAC [16] , where the model is used to fill a replay buffer used in an off-policy algorithm. Both methods extend their base MBRL algorithm by adding a constraint critic v π,safe and using a Lagrangian relaxation to train a constrained policy by optimizingEQUATIONWe follow a similar constrained model-based actor-critic (CMBAC) scheme to LAMBDA. However, we do not follow a similarly complex approach to the multiplier updates; rather, we add some minor modifications to the calculation of the value functions in constrained policy optimization to ensure the stability of the multiplier updates using gradient descent. Additionally, we avoid learning the policy π θ using a pessimistic constraint cost, which leads to learning an overtly conservative policy [4] . P ψ , π θ , V ϕ , V safe ξ end forIn CASE, we combine CMBAC with a planning-based safety filter to lower constraint violation rates while exploring CMDPs. Our motivation is to leverage the advantages of look-ahead control and the computational efficiency of actor-critic methods to explore the CMDP while keeping constraint violations to a minimum. This combination of learning and planning has been studied in recent work [6, 17, 18] , leveraging the ability of actor-critic methods to learn parametric policies and critics efficiently and compensating for the bias and limited expressiveness of an amortized parametric policy with look-ahead control using the model. We learn a probabilistic ensemble transition model P ψ similar to that introduced in PETS [12] , which we discuss further in appendix B. We use the learned dynamics model in a model-based actor-critic algorithm as a learned simulator for constrained on-policy actor-critic training. Online exploration is performed primarily using the constrained parametric policy; in addition, we implement a pessimistic planning-based safety filter to avoid violating constraints. An overview of our methods can be seen in figure 1 and algorithm 1. 
constrained model-based actor-critic: 
 Our approach to policy optimization is most similar to [3, 4] , where CMBAC methods were shown to result in data-efficient learning of constrained policies. For the policy optimization, we learn a parameterized policy π θ . In addition, we follow [19] and learn an ensemble value function V ϕ = {v ϕ1 , . . . , v ϕ B }, where v ϕi are individual ensemble members, similarly, we learn an ensemble safety critic V safe ξ = {v safe ξ1 , . . . , v safe ξ B }.Learning Dynamics Model We adapt the probabilistic ensembles from PETS [12] for solving CMDPs. We add predictions heads for rewards p(r t |s t ), constraint cost p(I t |s t ), and termination flags p(d t |s t ) for environments with early termination conditions. We model the reward distribution p(r t |s t ) as a Gaussian distribution, while the binary constraint cost I t and termination flag d t are modeled as Bernoulli distributions.Learning Critics For learning of value function ensemble members, we use imagined rollouts τ i,st using respective transition ensemble members p ψi and branching off real states s t . This approach leads to the disagreement of critic ensemble members capturing the epistemic uncertainty in the transition function ensemble P ψ similar to the approach followed in [20] . We use TD(λ) to calculate the targets for the value function similar to Dreamer [14] and train each member in the value function ensemble on its own independent targets as done in [21] min ϕi E s t ′ ∼p ψ,i ,a t ′ ∼π θ t+Hv t ′ =t 1 2 v ϕ,i (s t ′ ) -R i (s t ′ ) 2 ,whereR i (s t ) = r t + γ(1 -d t ) (1 -λ reward )v ϕ,i (s t ) + λ reward R i (s t+1 ) , R i t+Hv = v ϕ,i (s t+Hv ).Similarly we learn the safety critic ensemble V safe ξ using TD(λ) targets We note that the termination flag d t is only included in calculating the task critic targets. We do not include it in the constraint critic's training, which would lead the constraint critic to underestimate the cost for states near termination states. We discuss this design choice further in the ablation studies in appendix A.Constrained policy optimization For solving the constrained problem in equation (1), we resort to Lagrangian relaxation by including the constraint cost term in the policy objective weighted by the Lagrangian multiplier λ, thus turning the problem into an unconstrained problem as in equation (2) , where we solve a min-max optimization over the policy π θ and the Lagrangian multiplier λEQUATIONwhere R and C are the means over the TD(λ) returns from the different ensemble members R i and C i for the rewards and the constraint costs respectively. Designing stable update rules for Lagrangian multipliers is a challenging task in the model-based setting, as using biased model rollouts for updating the multipliers can lead to a rapid increase in their magnitudes, thus derailing training. Furthermore, model rollouts in MBRL normally use observations from the replay buffer as starting states, which can lead to the multipliers being updated to reflect the behavior of the policy used to collect the data rather than the optimized policy π θ . In [3] , a complicated optimization scheme is used to decelerate the updates of the multipliers. In [4] , the multipliers are updated solely using real online rollouts, presumably to avoid inaccuracies in the model from causing erroneous multiplier updates. We follow a more straightforward scheme and update the multipliers using stochastic gradient descent with no additional heuristics.Our changes center around the calculation of the policy's objective and are highlighted in objective (3) in cyan. We only use constraint-satisfying states S safe as initial states for our rollouts and only include the tail of the rollouts in the calculation of the terms in the objective, thus avoiding situations where the agent is already doomed but starts in a safe state. Thus giving the policy enough time to steer the system away from constraint-violating regions. Over time, these changes decelerate the increase in the multipliers and prevent them from exploding in value. In addition, we normalize the whole objective by a factor of 1 + λ, which helps keep the absolute value of the loss in the same scale as the multiplier λ increases in value similar to [2] .Our aim is to enable safer online exploration. Thus, we do not use conservatism to prevent the policy from exploring online as in [22, 23, 3] , which would lead the policy to learn an overtly conservative behavior. In contrast, we rely on a conservative safety filter MPC as a backup policy that intervenes to prevent constraint violation, guided by the critic.Trigger To trigger the filter, we rely on the ensemble transition model and use it to roll out the learned policy π θ for horizon H filter starting from the current state s t , generating separate imagined trajectories {(s i t ′ , a i t ′ )} t+H filter t using each member p θ,i . We use the worst-case value of the ensemble safety critic V safe max = maxs t ′ ∈P ψ V safe (s t ′ )to define a pessimistic recovery setEQUATIONwhere V safe max is the maximum prediction of the ensemble critic across the look-ahead trajectories. Our pessimistic formulation of the objective and the trigger of the safety filter consider the epistemic uncertainty inherent in the exploration task, where the filter is more likely to be triggered in situations with high epistemic uncertainty due to the effect of considering the worst case prediction of the ensemble safety critic.We roll out separate trajectories starting from current state s t with each separate transition ensemble member p ψ,i and evaluate the states s i t ′ in each trajectory with its respective safety critic v ξ,i . The state s t is considered part of S recovery in case the worst case prediction of V safe of the imagined trajectories starting from s t exceeds the threshold ϵ safe .Optimization The safety filter used in this paper solves the optimization problem min ut...ut+m E st+1..t+m∼P ψ (.|s t ′ ,u t ′ )EQUATIONThis MPC-filtering approach is similar to the recovery RL method [7] , which minimizes the safety critic along the look-ahead horizon, thus enabling a longer look-ahead at a reduced computational cost. In addition, we use pessimism in triggering the filtering and optimizing its objective. Thus taking epistemic uncertainty into consideration. Furthermore, our objective includes rollouts from the base policy π θ in the objective, thus encouraging the MPC to drive the systems to regions of the state space where the base policy is predicted to keep the system safe. We optimize the objective in (5) using gradient descent where we optimize the whole term for the actions {u t , . . . , u t+m }. 
experimental setup: 
 We compare our method with model-free baselines on constrained locomotion tasks from the Omnisafe [9] benchmark and model-based baselines on the RWRL [10] and safe-control-gym [11] benchmarks. We run each method on four seeds and show the mean and the 95% confidence interval performance in figure 2 . We explain our experimental setup and hyperparameters in more detail in appendix D.For benchmarking against model-free baselines, we use the constrained velocity control tasks from the Omnisafe benchmark, where the goal is to solve locomotion tasks while maintaining the system below a maximum velocity. We choose different constrained RL baselines based on TD3 [24] and SAC [25] . We first compare the performance of CASE with the model-free baselines SAC-Lagrangian and TD3-Lagrangian, which extend SAC and TD3 with a constrained policy optimization scheme using Lagrangian relaxation. In addition, we compare CASE to SAC-PID and TD3-PID, which use the Lagrange multipliers updating scheme presented in [2] . CASE generally reaches the same task performance as the model-free baselines while maintaining comparatively low constraint violation rates throughout training. However, we still see that even with the addition of a planner as a safety policy, we are still not able to completely avoid constraint violations.The baselines eventually learn to solve the task while having low constraint violation rates. However, as constrained policy optimization tends to have instabilities during training, they have high rates of constraint violation during exploration. This is compounded by the implementation in Omnisafe, which only starts updating the Lagrange multipliers after 100 warm-up epochs, corresponding to 200k environment steps. This initial warm-up phase ensures the off-policy methods have enough variety in the replay buffer. In cases where the robot needs to learn in the wild, such exploration behavior might not be acceptable.The results from our experiments put into question the suitability of the model-free method for safe reinforcement tasks, where the agent needs to learn online. Although model-free methods might have advantages in their asymptotic performance compared to model-based methods [12] , they lack the ability of model-based methods to do look-ahead controlling and deciding online to avoid actions where the robot might be uncertain or that might be deemed to be possibly dangerous.Making model-based methods more suitable for learning on physical robots, especially when the robot needs to explore under certain restrictions.We compare our method to two similar model-based approaches: LOOP [6] and the model-based version of Recovery RL [7] .LOOP integrates learned value functions with planning for decision-making. Unlike our method, LOOP uses look-ahead control to maximize rewards and minimize constraint violations at each step, does not account for epistemic uncertainty in planning, and assumes access to the ground-truth constraint cost function. Despite this, CASE outperforms or matches LOOP in constraint adherence without access to the real constraint cost function. Additionally, CASE demonstrates superior task performance, except in the cartpole environment, where LOOP has a higher constraint violation rate.As LOOP requires optimising a look-ahead control problem in every environment step, we find that it is much less compute efficient than CASE, we discuss the differences in frequency between CASE and similar methods in appendix C.In contrast, Recovery RL uses a model-free approach to learn an unconstrained task-solving policy and a safety critic. The model-based version introduces a forward model for triggering interventions and to design a look-ahead recovery controller. However, Recovery RL suffers from conflicts between the task-solving policy and the recovery policy, leading to worse performance compared to CASE, as seen in figure 3 . 
related work: 
 Constrained model-based reinforcement learning Different RL methods have been introduced targeting CMDPs [26, 27, 28] . The most common variants are papers leveraging the augmented Lagrangian method for learning constrained policies. CPO [29] extended TRPO to CMDP, and building on that work [30] combined the Lagrangian method with PPO to learn constrained policy. Using an on-policy method rendered the method unsuitable for learning in the real world, where sample efficiency is essential. In [31] , the authors implemented a similar approach but replaced the on-policy algorithm with an off-policy approach to reduce sample complexity. Saute RL [32] provides an alternative to using Lagrangian relaxation for solving CMDPs by using state space augmentation. Different papers proposed MBRL methods combining learned models with constrained policy optimization. In [3] , the authors use a Bayesian approximation of an RSSM [13] as a model. The posterior of the transition is used to learn an optimistic objective regarding the rewards but pessimistic regarding the constraint cost. A similar approach was introduced in [4] where the authors extend SLAC [16] to the constrained setting. Concurrent with our work, safeDreamer proposes combining planning with CMBAC. However, SafeDreamer differs in not leveraging the planner as a backup policy, as is the case with CASE. Further, safeDreamer does not consider the epistemic uncertainty into decision-making, as proposed in our method.Safety filters Learning a policy to satisfy constraints and, at the same time, maximize the agent's utility can be a challenging task. As a result, multiple methods have attempted to circumvent this problem by introducing backup policies π safe (s t ) that minimize constraint costs in addition to a taskreward maximizing base policy π(s t ). Model-free methods that use a backup mechanism [7, 33] generally use an off-policy critic to trigger the intervention mechanism. Critics tend to be sensitive to out-of-distribution data and, in general, hard to learn. The two-policy setup is also investigated in [34] , where the authors improve on the safety layers approach [35] by replacing the layers with a separate parametric policy that allows them to handle more complicated constraints.The predictive safety filter paper [8] introduces an MPC wrapper to a base policy to prevent constraint violations during learning, relying on assumptions about the model and the system dynamics and the availability of a controller that keeps the system in a predetermined safe terminal state to have guarantees on constraint satisfaction. In [36] , a control barrier function (CBF) is used in learning a safe policy, but their method is limited to control-affine systems and assumes prior knowledge of a simplified model and access to a forward invariant safe set. In general, our method differs from other safety filter methods based on CBFs and Hamilton-Jacobi reachability in that we do not make any assumptions on the underlying CMDP and do not assume prior knowledge of the nominal dynamics of the systems. This contrasts to CBF and HJ safety filters where it is common to make assumptions on the dynamics [37, 38, 39, 40, 36, 41] , Lipschitz continuity [37, 38, 39, 40, 36] or existence of prior knowledge of prior safe sets [36] . SAILR [42] provides an advantage-based intervention mechanism and derives performance bounds under the assumption of having an MDP with an absorbing state. However, their model-based experiments seem to be based on engineered models.Online planning using offline learned functions Combining online planning with offline learned functions has proven effective in several works [18, 6, 17] . Online planning suffers less from bias compared to parameterized policies, and offline-learned functions can enhance the planner's performance [43] . The idea of planning using offline learned functions was shown to improve on parametric policies in [18] , and performance bounds were derived that show the benefits of using a planner in combination with a value function. However, the methods and the bounds introduced all used an unbiased dynamics model. In [6] , these insights were extended to learned models. 
conclusions and limitations: 
 We present a method that combines constrained model-based policy optimization with a pessimistic planning-based safety filter for exploration in CMDPs with the aim of facilitating the learning of effective policies entirely on physical robots. We find that leveraging the ability of model-based methods to function as a simulator for actor-critic methods and being used in look-ahead-control schemes, thus combining the advantages of model-based actor-critic with low-bias properties of planning, can lead to a significant reduction in constraint violation in comparison with model-free and model-based baselines. We evaluated our method on different constrained RL benchmarks, which show the versatility of our method in adapting to different task dimensionalities and its ability in robotic tasks such as drone trajectory tracking and stabilization.Our method differs from previous safety filtering methods [8, 7] , where the base policy π θ is unaware of constraints and the agent's safety is left to the backup mechanism, often causing oscillatory behavior [44] . We assume no prior knowledge of the CMDP's dynamics and make no assumptions regarding its properties as done in previous safety filtering methods [36, 8, 39] and aim for a versatile method applicable to any CMDP, including those with contact dynamics discontinuities and high dimensional observations such as images. We believe combining our approach with other model-based actor-critic methods [45, 46] can enable efficient robot learning from high dimensional observations with minimal supervision.Limitations The main limitation of our method lies in the inability to guarantee constraint satisfaction. Such guarantees would involve making assumptions on the target system, such as the smoothness of the dynamics and the availability of a safe terminal set as in methods leveraging HJ reachability and CBF [8, 36, 40, 37, 38] . Such assumptions might not be satisfied in many target systems of interest. Although our method leads to less constraint violation than the baselines during exploration, online planning can be computationally expensive. Furthermore, our design for the learned transition model limits our method to fully observable systems, which we aim to rectify in future work by leveraging models that can learn the dynamics of partially observable systems. : For ablating design choices in the safety filter. We evaluate the average rewards per episode in the last 10 evaluations (left) and the average percentage of steps with constraint violations in an episode during exploration (right). We first evaluate the role of the safety filter in CASE. Exploring with only π θ leads to a significant increase in constraint violations. Next, we evaluate the effect of the base policy rollouts. We see a decrease in the average rewards reached after exploration, especially in the ant environment. Finally, to evaluate the effect of pessimism in the filter, we optimize the filter by using the mean over the ensemble predictions instead of considering the worst-case ensemble prediction. Without pessimism, the filter tends to have higher constraint violation rates. 
 
 Appendix: Effect of safety filter To assess the effect of the safety filter in our method, we explore CASE without the safety filter. In figure 4 , we see that exploring solely using the constrained policy π θ leads to higher constraint violation rates. This indicates the necessity of the filtering mechanism in our method. By looking at the maximum rewards reached after exploration in figure 4 , we see that the increase in rewards after removing the safety filter is insignificant except in the halfcheetah environment, where the constraint violation rates more than double after removing the filter.Effect of base policy term in the filter objective The filter objective in (5) includes rolling out the base policy π θ in the tail of the look-ahead trajectory of the controller. We ablate this term by replacing it at the tail of the look-ahead trajectory, thus maintaining a fixed total horizon length. This equates to the following objectivemin ut...u t+H E s t+1..t+H ∼P ψ (.|s t ′ ,u t ′ ) t+H t ′ =t V safe max (s t ′ ) .The purpose of the base-policy rollout is to inform the look-ahead controller of the base policy's behavior and thus steer the system towards regions of the state space where the base policy can perform without violating constraints. Removing the base policy rollouts leads to slightly worse task performance, especially in the ant environment. However, the difference in task performance is less significant than expected. In future work, exploring other possibilities for combining the intervention controller with a constrained base policy, such as penalizing the difference between the intervention mechanism's actions and those of the base policy as done in [8] , might be interesting.Effect of pessimism in the safety filter In our safety filter design, we leverage a pessimistic loss to consider the effect of epistemic uncertainty during exploration. To study the effect of pessimism on the filter, we use the mean over the ensemble prediction constraint critic predictions V safe mean , instead of using the maximum aggregation over the ensemble of the constraint critic predictions V safe max , introduced in section 3.2. We found that in our setup, removing the pessimism leads to higher constraint violation rates, especially in the more complex halfcheetah and ant environments.Effect of Including Termination flag in Constraint Critic's Targets The constraint critic target described in 3.1 does not include the termination flag d t . The termination flag is usually needed in environments with termination conditions, which assumes that terminating states are absorbing 1 : Comparing FPS for LOOP and LOOP Slow. We see that reducing the controller budget massively increases the FPS while having no effect on the performance, as seen in figure 7 .states. Including the termination flag in the task critic V leads the critic to predict low values for such states, incentivizing the policy to avoid termination. This effect is not desired in the constraint critic V safe as it would lead the constraint critic to assign erroneous cost-to-gos to states near the termination states, which would have an adverse effect on the safety filter and the constrained policy optimization. We compare the predictions of two constraint critics in figure 5 . The first in figure 5a is trained using targets including the termination flag d tC i (s t ) = I t + γ safe (1 -d t ) (1 -λ safe )v safe ξ,i (s t ) + λ safe C i (s t+1 )where,C i (s t+Hv ) = v safe ξ,i (s t+Hv ).The second in figure 5b is trained with the targets described in (5) . We find that including d t in TD (λ) returns has a negative effect on the constraint critic predictions where the critic assigns much lower values to states near the termination state, despite constraint violations.Due to our focus on exploration, we need a model that can provide well-calibrated epistemic uncertainties. Deep ensemble models present a straightforward approach to provide representations of the epistemic uncertainty due to their ability to provide good approximations of the Bayesian posterior predictive distribution of the neural network [47] . Thus, we rely on an ensemble of dynamic models P ψ = {p ψ,1 , . . . , p ψ,B }, where each ensemble member p ψ,i is a neural network that predicts the transition as a Gaussian distribution with a diagonal covariance p ψ,i (s t+1 | s t , a t ) = N (s t+1 | µ ψ,i (s t , a t ), Σ ψ,i ). We optimize each ensemble member by minimizing the negative log-likelihoodL p ψ,i = - T t=1 log p ψ,i (s t+1 | s t , a t ).The use of ensemble-based transition models has already shown very good performance in modelbased RL papers such as [48] and [12] . In deep ensemble transition models, each ensemble member p ψ,i captures the aleatoric uncertainty of the ground-truth MDP. In contrast, the disagreement between the ensemble members captures the epistemic uncertainty on the learned transition function. We find that despite the online computations, we maintain a reasonable FPS that makes CASE feasible to use in physical systems.Although using a look-ahead planner as a safety filter helps reduce the constraint violation rates of the reactive parametric base policy, this comes at the cost of computational efficiency. The safety filter involves an online optimizer that solves the optimization problem in objective (5) . We compare the frequency of the filter across the different environments in figure 6 , and we find that the filter performs around 50 Hz in all environments. Looking at other methods combining actor-critic methods and planning, we find that TD-MPC [17] , which does not consider the constrained setting, performs similarly to our method in runtime with about 50 Hz for the default setting. LOOP [6] , which also explores CMDPs among other settings, reports a lower frequency 14.3 Hz. Generally, the low frequency of planning-based methods represents the biggest disadvantage compared to methods leveraging only parametric policies for decision-making.We use the constrained velocity tasks from safety gymnasium [49] in our experiments. Specifically, we use the half cheetah, hopper, walker, and ant tasks. These tasks are attractive as they pose conflicting objectives, where the task reward incentivizes the agent to move with high velocity with the correct pose, and the constraint cost punishes the agent for moving above a velocity limit.In training LOOP [6] for the RWRL benchmark tasks, we used the hyperparameters (HPs) without tuning (we also do not tune our method). We noticed that the runtime for LOOP is extremely long and much worse than the latency mentioned in the paper when evaluated in high dimensional cartpole walker quadruped CASE (Ours) 70.5 ± 3.3 69.03 ± 1.83 79.7 ± 7.8 LOOP 7.9 ± 0.13 1.12 ± 0.03 0.88 ± 0.016 Table 2 : FPS for CASE in comparison with LOOP [6] . We find that despite the online computations, we maintain a reasonable FPS that makes CASE feasible to use in physical systems. environments, as seen in table 2. We found that this is mostly due to the loops in the samplingbased look-ahead controller in LOOP. To avoid the long LOOP training times in our experiments in safe-control-gym, we tuned the look-ahead controller aiming to decrease LOOP's latency while maintaining the same performance. We compare the performance of the new LOOP HPs with the original LOOP HPs on two of the safe-control-gym tasks in figure 7 , and we compare their runtimes in table 1.Our method involves hyperparameters for the model training, the CMBAC, and the safety filtering. We list our hyperparameters in table 3  


Decision: Reviewers agree that the problem solved is important and relevant for the robotics community. Unfortunately, the paper has three main weak points that should be handled in the rebuttal:
- The method seems to be not completely novel, and rather incremental. The authors should clarify the novelty of the approach.
- There seem to be some technical issues and unclear points in the proposed theory. Please refer to the reviewers' comments for more details. I expect a detailed answer to all their comments (and follow-up responses)
- There are no real robot experiments or interesting robotics tasks beyond simple Mujoco benchmarks. Most reviewers agree that this is a major drawback, and I'll not vote for accepting this paper if a more convincing benchmark is presented, to match the scope of the CoRL conference.

===

During the rebuttal, the authors managed to convince some of the reviewers about the technical soundness of their approach. Also, the authors added a reasonable robotics experiment (the 3d quadrotor stabilization).
While some reviewers argue that the novelty of the paper is a bit limited, it is overall an ok submission.
An important point to fix is to improve the description of the optimization (after equation 8)  and to add a safety critic baseline together with lagrangian approaches.
Also, I would require the reviewers to reference in the paper some of the works mentioned by reviewer 1NEA, specifically:

[1] Hsu, Kai-Chieh, Duy Phuong Nguyen, and Jaime Fernandez Fisac. "Isaacs: Iterative soft adversarial actor-critic for safety." Learning for Dynamics and Control Conference. PMLR, 2023.

[2] He, Tairan, et al. "Agile but safe: Learning collision-free high-speed legged locomotion." arXiv preprint arXiv:2401.17583 (2024).

[3] Hsu, Kai-Chieh, et al. "Safety and liveness guarantees through reach-avoid reinforcement learning." arXiv preprint arXiv:2112.12288 (2021).


I would mark this paper for acceptance, but I would not have any problem if the program chairs decide to reject this paper.

Reviews: review: title: title: Official Review of Paper631
paper_summary: paper_summary: The paper provides a novel approach for constrained model-based RL for reducing constraint violation during exploration by utilizing planning based safety filter as a backup policy. In addition, they utilize deep ensemble models to capture epistemic uncertainty in learned model thereby improving the ability to satisfy constraints. For instance, the filter is more often triggered in areas with high epistemic uncertainty leading to lesser safety violations during exploration.
main_review: main_review: Quality: Overall, the presentation of the paper is very clear and the paper is structured well.

Originality: The paper focuses on exploring using the model information alongside training the constrained policy which is a novel contribution. Their method also incorporates capturing epistemic uncertainty in the learned model.

Significance: The paper proposes a constrained model based reinforcement learning pipeline for reduced safety violations during exploration. However, they only compare the results with model-free baselines which I believe is not an apple-to-apple comparison and somehow doesn't highlight how their method is better than other model-based methods. In terms of runtime, the paper argues that average runtime of the proposed filter is around 50 Hz and therefore it can be tested on physical systems. However, the authors don't perform any hardware experiments. If the authors would have added hardware experiments, it could have shown proof of concept of their method in real world.
questions: questions: The paper claims that their proposed method helps in reducing constraint violations during training as compared to other constraint model-based optimization methods but in the results section they only compare their method with model-free baselines which is not what they claim in the abstract. How their methods compares with the Recovery RL paper for example should be shown. Also in "our contribution" section where discussion about Figure 1 is shown, there the authors claim comparison to CMBRL methods which is not the case.

The paper also uses only simple safety constraints such as constrained velocity rather than considering complex scenarios such as obstacle avoidance or multiple safety constraints. Please add few experiment with some additional safety constraints.
limitations: limitations: 3
review_summary: review_summary: Please read the review above.
title: title: Review
paper_summary: paper_summary: This paper presents a model-based safe reinforcement learning (RL) method that incorporates a learned Markov Decision Process (MDP) and Model Predictive Control (MPC) to ensure safe exploration. The proposed method uses ensembles for each component to determine the worst-case scenario, which then triggers a safe MPC controller to ensure safety during exploration. The method is validated in the Safe-Mujoco tasks, showing fewer violations during the training process.
main_review: main_review: 1. The authors use a Gaussian network to approximate the dynamics model. However, it is unclear how the variances are utilized in the rollout. How is this uncertainty information leveraged during training?

2. The paper uses two $\lambda$ parameters to represent different quantities. It is advisable to use different notations to avoid confusion.

3. The authors only include the tail of the rollout in the optimization. How does this ensure that the midpoint along the rollout is safe? Is this choice simply to reduce the value of the multiplier, and would a smaller learning rate work just as well? Furthermore, is a threshold value $l$ missing in equation (6)? Since the constraint used here is an indicator function, the safety critic should be greater than 0. With the current formulation, i.e., $l=0$, the explosion of the Lagrangian multiplier is foreseeable.

4. From my understanding, the authors learn an ensemble of the value function, which is not parameterized as Gaussian. The model itself cannot distinguish between aleatoric and epistemic uncertainty. Instead, the safety filter uses the worst-case prediction instead of high epistemic uncertainty scenarios.

5. Since $V_{\text{max}}^{\text{safe}}(s_{t’})$ already captures the worst-case violations, why is a sum still needed in equation (8)?

6. Why does CASE have small violations at the very beginning? If no prior knowledge is given for the training, the dynamics and the value function will have poor estimations, and the policy should also contain certain violations until sufficient data is collected.

7. Omnisafe provides benchmarking results with 1e6 steps when the policy converges. However, the authors only show results with 1e5 steps, which is insufficient. The experiment shown in the paper is not convincing.

8. The experiments are evaluated exclusively on the Safe-Mujoco tasks, with no inclusion of robotic-related tasks.

[1] https://github.com/PKU-Alignment/omnisafe/tree/main/benchmarks
questions: questions: Please find them in the comments.
limitations: limitations: 2
review_summary: review_summary: The paper presents a model-based safe RL approach. Integrating rollout, ensembles, and MPC for safe-critical problems are rational choices. However, the futher details of the experiments are needed to justify the claim.
title: title: A model-based reinforcement learning approach that uses safety filters to enhance safety at training time.
paper_summary: paper_summary: This paper proposes a model-based reinforcement learning (MBRL) approach that leverages safety filters to improve safety at training time. This is in contrast to existing MBRL that does not explicitly account for safety during exploration. The proposed approach uses planning-based backup policies to lower constraint violations during training. The method is tested with simulated examples.
main_review: main_review: This paper attempts to improve safety for model-based RL approaches, a topic that is relevant to robot learning. The paper is well-structured, with a clear indication of contributions and limitations. Unfortunately, the paper, in its current form, does not meet the standard of CoRL due to a significant lack of novelty, technical quality, and practicality to real-world robotics tasks. Detailed comments are attached below.

First, the mathematical definition of the unsafe state set (also the safe state set) $\mathcal{S}_{\text {unsafe }}= \\{s_t \mid \mathcal{I}\left(s_t, a_t\right)=1\\}$ is problematic. The action $a_t$ is used to define the indicator function. However, the author's definition implies that the classification of states as safe or unsafe is dependent solely on the state $s_t$​, without regard to the action $a_t$​.

Generally speaking, the proposed approach is a straightforward combination of existing works in MBRL (e.g., [12], [19]) and safety filters (e.g., Hsu et al., The safety filter: A unified view of safety-critical control in autonomous systems, which is not even cited). The few modifications to existing approaches are also minimal. For example, in (6) the authors propose to use safe samples when evaluating the objective, which is already a common practice in safe RL. 

The title "Exploring Under Constraints" and subsection title "Constrained policy optimization" are misleading, since the safety objectives are incorporated using Lagrangian relaxation with a heuristic weight $\lambda$. Such formulation enjoys no guarantees as in constrained optimization and learning, and is ultimately unconstrained optimization, as in conventional RL.

The safety filter formulation encoded in (7) is very handwavy. The state-of-the-art safety filter approaches (e.g., HJ and CBF; again, see Hsu et al.) usually involve a backup policy that renders a safe set forward invariant. Why do the authors abandon those principled approaches for guaranteeing closed-loop safety, while resorting to some heuristic methods?

Regarding the experiments, the method is only evaluated with (simple) simulated benchmark examples - this is not acceptable for CoRL, especially given that the technical contributions of the paper are already on the weaker side. The author needs to provide either hardware demonstrations with real robots or a challenging example with simulations (e.g., humanoid locomotion, large-scale autonomous driving, or collaborative manipulation tasks).

Finally, the literature review is rather incomplete. The paper is closely related to safe RL with reach-avoid objectives, shielding (and its integration with safety-critical planning and/or learning), *learning* controlled forward invariance (e.g., Hamilton-Jacobi Reachability, control barrier functions), and safe learning of residual models. However, none of these is mentioned in the Related Work.

Minor issues:
* The full name of the acronym "CASE" was never mentioned.
* The fonts in Figure 3 and 4 are too small.
questions: questions: * What is the correct mathematical definition of safe/unsafe state sets? Why did the authors use indicator functions to define the safety objective? It has been shown to underperform the reach-avoid objectives that render dense rewards and much better safety performance (see, e.g., He et al., Agile but Safe, RSS 2024).
* What is the novelty of the paper aside from assembling existing approaches of MBRL and safety filters?
* What is a proper characterization of the proposed optimization formalism? Is it proper to call it "constrained optimization" while the actual problem being solved is unconstrained?
* Is the safety filter defined by (7) a principled approach to improve robustness in RL? Why did the authors choose not to use state-of-the-art safety filter methods such as HJ Reachability or CBFs?
* How does the proposed approach perform for realistic robotics tasks?
limitations: limitations: 3
review_summary: review_summary: My recommendation is based on these weaknesses of the paper: lack of novelty, several technical issues, and dubious relevance to robotics. See my review for detailed justifications.