Exploring Under Constraints with Model-Based Actor-Critic and Safety Filters
Authors: Ahmed Agha
Baris Kayalibay
Atanas Mirchev
Patrick van der Smagt
Justin Bayer
Venue: CoRL 2024 (2024)
Keywords: Model-based RL
Safe RL
Safety Filter
Exploration
TL;DR: We combine constrained model-based policy optimization with planning-based safety filters as a backup policy to reduce constraint violation rates during exploration.

Abstract: Applying reinforcement learning (RL) to learn effective policies on physical robots without supervision remains challenging when it comes to tasks where safe exploration is critical. Constrained model-based RL (CMBRL) presents a promising approach to this problem. These methods are designed to learn constraint-adhering policies through constrained optimization approaches. Yet, such policies often fail to meet stringent safety requirements during learning and exploration. Our solution ``CASE'' aims to reduce the instances where constraints are breached during the learning phase. Specifically, CASE integrates techniques for optimizing constrained policies and employs planning-based safety filters as backup policies, effectively lowering constraint violations during learning and making it a more reliable option than other recent constrained model-based policy optimization methods.

Exploring Under Constraints with Model-Based Actor-Critic and Safety Filters
Ahmed Agha Volkswagen Group
Baris Kayalibay
Volkswagen Group
Germany ahmed agaaealoogghlllllsaaeeenn de
Germany
baris kayalibss@volkkswggaa
Atamas Mirchev
Patrick van der Smagt
Volkswagen Group
Volkswagen Group
Germany
Germany
atanas mi rcchevvvokswweeen de
Justin Bayer
Volkswagen Group
Germany
justin..byyerrvolkswagg..
Abstract: Applying reinforcement learning (RL) to learn effective policies on physical robots without supervision remains challenging when comes to tasks where safe exploration is critical. Constrained model-based RL (CMBRL) presents promising approach this problem. These methods are designed learn constraint-adhering policies through constrained policy optimization ap- proaches. Yet, such policies often fail meet stringent safety requirements dur- ing learning and exploration. Our solution "CASE" aims to reduce the instances where constraints are breached during the learning phase. Specifically, CASE integrates techniques for optimizing constrained policies and employs planning- based safety filters as backup policies, effectively lowering constraint violations during learning and making more reliable option than other recent constrained model-based policy optimization methods.
Keywords: Model-based RL, Safe RL, Safety Filter, Exploration
Introduction
Many real-world robotic systems can be effectively modeled as constrained Markov decision pro- cesses (CMDPs) [1]. particularly in contexts where safety paramount, and robots must adhere to specific conditions while learning to accomplish tasks. In addition, CMDPs offer a structured framework for injecting inductive biases into the policy optimization process, thereby reducing the number of interactions required to learn effective policies. Thus, developing deep RL algorithms for solving CMDPs has the promise of unlocking RL's potential across various real-world robotic applications where safety is an issue. However, applying RL to systems that must operate under constraints all the time, even as they explore and learn, remains an open challenge.
Model-free constrained policy optimization methods have primarily adapted actor-critic techniques the constrained setting, employing Lagrangian relaxation, such the augmented Lagrangian method, to train constrained policies. While these algorithms
appealing for their relative sim- plicity, they face significant challenges, particularly in terms of sensitivity to Lagrangian multipliers [2]. Moreover, model-free RL approaches for CMDPs often suffer from high sample complexity, making them ill-suited for constrained environments where a high number interactions during
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

learning constrained policies leads to a high number of constraint violations as well as increased wear and tear, reducing their suitability for learning without supervision on physical robots.
Various authors have proposed applying model-based RL methods to constrained settings tackle the challenge posed by the high sample complexity of model-free approaches. These methods fall into two main categories. The first category includes methods that improve learning efficiency and reduce constraint violations during exploration by using differentiable models [3, 4]. These models act as simulators for both learning the policy and the value function. However, despite their advan- tages, they often face difficulties with the instability that comes with constrained policy optimization. The second category involves methods that incorporate planning [5, 6. 7, 8] possibly alongside the primary, unconstrained policies. This approach helps in preventing constraint violations throughout as simulators for
the learning phase. We find that combining the ability of learned models constrained policy optimization with the possibility of using learned models in look-ahead control schemes remains a largely unexplored direction. Furthermore, we find that most CMBRL methods do not consider epistemic uncertainty, which is essential in cases where the policy is exploring online and needs to reason about the effect of epistemic uncertainty on its ability to satisfy constraints.
Our Contributions In this work, we combine constrained model-based policy optimization with a planning- -based safety filter that acts as a backup policy to minimize constraint violations during exploration. In addition, we introduce modifications to the constrained model-based policy opti- mization training to ensure stable training. We also modify the safety filter's objective to consider the behavior of the constrained base policy during planning. We evaluate our method on constrained tasks from the Omnisafe [9], RWRL [10], and the safe-control-gym [1 [] benchmarks and show that our combination of constrained policy optimization and planning can lead to significantly reducing constraint violations during training in comparison with other model-free and model-base methods.
2 Preliminaries
2.1 Problem Setting
We consider CMDP, which is defined by the tuple (S A, P, Po C, S and A are the state and action spaces respectively. The discount factor and the safety discount factor are represented by and "safe- The dynamics of the system are represented by p(8++1 8t, az), and the initial state distribution is represented by po- The reward function is represented by r(5;, at), and the constraint cost function represented by c(s;). In this work, we consider constraint costs represented with the indicator function I(s;) where the unsafe state is represented Sunsafe {st I(s:)= 1} and the safe states are represented Saaee {st [(s:) -).
The challenge in CMDPs is maximizing the performance of the agent while satisfying constraints
J(TT)
E
rr(ss ax)
such that J((T)=
E ap
Σ t=
I(st)
$ (((1)
where is cabblem-specifie threshold.
2.2 Constrained Model-based Reinforcement Learning
In constrained model-based RL, transition tuples in the form of (st, oe, rt, Ct; 81+1) } to learn transition model p. The transition model can then be used in online planning done in [12, 13], or by amortizing decision decision-making by offline training of a parametric policy using an actor-critic approach as in [14, 15].
Recent methods [3, 4] have shown the effectiveness of MBRL in solving CMDPs as formulated in Eq. (1). Both methods adapt previous model-based RL methods to the constrained setting:
LAMBDA [3] is based Dreamer [14], where the RSSM model from PLANET [13] is used as differentiable simulator for on-policy actor-critic training to learn policy тo and critic v*:

---

Cra:ic
Model
Pollcy
Yes
Safety Filtee
Filter
Safety
No
V
Mi
Depiedt
D
Figure I: Overview of CASE. dynamics model is used train constrained policy, critic, and a constraint critic. The task policy is used alongside safety filter to ensure low constraint violation rates during exploration.
safeSLAC [4] is based on SLAC [16], where the model is used to fill a replay buffer used in an off-policy algorithm. Both methods extend their base MBRL algorithm by adding a constraint critic u ssale and using a Lagrangian relaxation to train a constrained policy by optimizing
min max E.o~po [J[s)] + (o X20
(2)
We
We follow similar constrained model-bart based actor-critic (CMBAC) scheme to LAMBDA. However, we do not follow similarly complex approach to the multiplier updates; rather, we add some minor modifications to the calculation of the value functions in constrained policy optimization to ensure the stability of the multiplier updates using gradient descent. Additionally, we avoid learning the policy ne using a pessimistic constraint cost, which leads to learning an overtly conservative policy [4].
Approach
In CASE, combine CMBAC with planning-based safety filter to lower constraint violation rates while exploring CMDPs. Our motivation to leverage the advantages of look-ahead control and the computational efficiency of actor-critic methods to explore the CMDP while keep- ing constraint violations to minimum. This combination of learning and planning has been studied in recent work [6, 17, 18]. leveraging the ability of actor-critic meth- ods to learn parametric policies and crities efficiently and compensating for the bias and limited expressiveness of an amor- tized parametric policy with look-ahead control using the model. We learn a proba- bilistic ensemble transition model Ps: sim- ilar to that introduced in PETS [12], which we discuss further in appendix B. We use the learned dynamies model in model-based actor-critic algorithm as a learned simulator for con- strained on-policy actor-critic training. Online exploration is performed primarily using the con- strained parametric policy; in addition, implement pessimistic planning-bast based safety filter to avoid violating constraints. An overview of our methods can be seen in figure I and algorithm I.
Algorithm 1 Pseudocode of CASE
Initialize parameters 0, é, &, + for N episodes do for timesteps do
sample a, from policy (a(s))
if s; E Srewwery according to (4) then
Trigger filter and optimize filter objective in (5) resulting in plan [u.,... ((+H))
Apply 14
else Appy az end if
end for
Add Depisode to D update Pv. ne. Vs. Vsale
end for

---

3.1 Constrained Model-based Actor-Critic
Our approach to policy optimization is most similar to [3, 4], where CMBAC methods were shown to result in data-efficient learning of constrained policies. For the policy optimization, we learn a parameterized policy Ty. In addition, we follow [19] and learn an ensemble value function V4 (VBL)... VVo where va, individual ensemble members, similarly, we learn ensemble safety ssafe]
critic Vaale (ue P..
Learning Dynamics Model We adapt the probabilistic ensembles from PETS [12] for solving CMDPs. We add predictions heads for rewards p(rz)se). constraint cost p{Za(st), and termination flags [[))]]s) for environments with early termination conditions. We model the reward distribution p(re(st) as a Gaussian distribution, while the binary constraint cost Z, and termination flag de are modeled as Bernoulli distributions.
Learning Crities For learning of value function ensemble members, we imagined rollouts Ta, using respective transition ensemble members Pv: and branching off real states 8.- This ap- proach leads to the disagreement of critic ensemble members capturing the epistemic uncertainty in the transition function ensemble Pv similar to the approach followed in [20]. We use TD(A) to calculate the targets for the value function similar to Dreamer [14] and train each member in the value function ensemble on its own independent targets as done [21]
+ H Σ
min Esg Po. aagen
2 [s.) - R (sx))]
where
r. d.) ((1 No. + reward R² (s(+1)) Ri+ (8.4(
Similarly we learn the safety critic ensemble Vsafe using TD(A) targets We note that the termination flag de is only included in calculating the task critic targets. We do not include it in the constraint critic's training, which would lead the constraint critic to underestimate the cost for states near termination states. We discuss this design choice further in the ablation studies in appendix A.
Constrained policy optimization For solving the constrained problem in equation (I), we resort Lagrangian relaxation by including the constraint cost term in the policy objective weighted by the Lagrangian multiplier \, thus turning the problem into unconstrained problem in equation (2), where we solve min-max optimization over the policy ma and the Lagrangian multiplier À
1
min max A20 m
1+> Ba
(((++)) + AC(se+k) st
(3)
where R and C are the means over the TD(A) returns from the different ensemble members R and CH for the rewards and the constraint costs respectively. Designing stable update rules for La- grangian multipliers is challenging task in the model-based setting, using biased model rollouts for updating the multipliers can lead to rapid increase in their magnitudes, thus derailing training. Furthermore, model rollouts in MBRL normally use observations from the replay buffer as starting states, which can lead to the multipliers being updated to reflect the behavior of the policy used to collect the data rather than the optimized policy πy. In [3], a complicated optimization scheme is used to decelerate the updates of the multipliers. In [4], the multipliers are updated solely using real online rollouts, presumably to avoid inaccuracies in the model from causing erroneous multi- plier updates. We follow more straightforward scheme and update the multipliers using stochastic gradient descent with additional heuristics.
Our changes center around the calculation of the policy's objective and are highlighted in objective (3) in cyan. We only use constraint constraint-satisfiing states State as initial states for our rollouts and only include the tail the rollouts in the calculation of the terms in the objective, thus avoiding situations where the agent is already doomed but starts in safe state. Thus giving the policy enough time to steer the system away from constraint-violating regions. Over time, these changes decelerate the increase in the multipliers and prevent them from exploding in value. In addition, we normalize the

---

whole objective by a factor of + A, which helps keep the absolute value of the loss in the same scale as the multiplier increases in value similar to [2].
Exploration with Safety Filter
Our aim enable safer online exploration. Thus, do not use conservatism prevent the policy from exploring online in [22, 23, 3], which would lead the policy learn overtly conservative behavior. In contrast, rely conservative safety filter MPC backup policy that intervenes to prevent constraint violation, guided by the critic.
Trigger To trigger the filter, we rely on the ensemble transition model and use it to roll out the learned policy Ty for horizon Hilller starting from the current state 8t, generating separate imagined trajectories {( ) 1 using each member Po,i- We use the worst-case value of the ensemble safety critic Vislee max (sale(sr) ) to define a pessimistic recovery set E
max
Sreooerry {(s)) S:
Dssafe max
(s/:) Z Esafe
(4)
where Vasale is the maximum prediction of the ensemble critic across the look-ahead trajectories.
Our pessimistic formulation of the objective and the trigger of the safety filter consider the epistemic uncertainty inherent in the exploration task, where the filter is more likely to triggered in situations with high epistemic uncertainty due the effect of considering the worst case prediction of the ensemble safety critic.
We roll out separate trajectories starting from current state 8; with each separate transition ensemble member Paú and evaluate the states si. in each trajectory with its respective safety critic tgg The state 8, considered part Speooeryy in case the worst case prediction Dafe of the imagined trajectories starting from 8 exceeds the threshold safe-
Optimization
The safety filter used in this paper solves the optimization problem
T Haier
min Mg... M...-.
8111.. mn P (ollemme))
Dsade (str)
(5)
This MPC-filtering approach is similar to the recovery RL method [7], which minimizes the safety critic along the look-ahead horizon, thus enabling a longer look-ahead at a reduced computational cost. addition, we use pessimism triggering the filtering and optimizing its objective. Thus taking epistemic uncertainty into consideration. Furthermore, our objective includes rollouts from the base policy Te in the objective, thus encouraging the MPC drive the systems regions of the state space where the base policy is predicted to keep the system safe. We optimize the objective in (5) using gradient descent where optimize the whole term for the actions { (u....., Ut+m }
4 Results
4.1
Experimental Setup
We compare our method with model-free baselines on constrained locomotion tasks from the Om- nisafe [9] benchmark and model-based baselines the RWRL [10] and safe-control gym [11] benchmarks. We run each method four seeds and show the mean and the 95% confidence in- terval performance in figure 2. We explain our experimental setup and hyperparameters in more detail in appendix D.
4.2 Comparison to Model-free Baselines
For benchmarking against model-free baselines, use the constrained velocity control tasks from the Omnisafe benchmark, where the goal is to solve locomotion tasks while maintaining the sys- tem below maximum velocity. We choose different constrained RL basclines based on TD3 [24]
5

---

CA5 O01
SAC
Tepper
6990
2000
2003
3243
1003
-100
03
23
emalton 75811 staps
enditermett steps
envitormett Steps
Figure 2: Performance of CASE compared to other constrained RL methods constrained envi- ronments from the Omnisafe benchmark. We find that constraint violation rates during exploration reduced significantly while maintaining competitive task performance in comparison with other constrained RL methods..
and SAC [25]. We first compare the performance of CASE with the model-free baselines SAC- Lagrangian and TD3-I Lagranggnnn which extend SAC and TD3 with a constrained policy optimiza- tion scheme using Lagrangian relaxation. In addition, we compare CASE to SAC-PID and TD3-PID, which use the Lagrange multipliers updating scheme presented in [2]. CASE generally reaches the same task performance as the model-free baselines while maintaining comparatively low constraint violation rates throughout training. However, we still see that even with the addition of planner as safety policy, are still not able to completely avoid constraint violations.
The baselines eventually learn to solve the task while having low constraint violation rates. However, as constrained policy optimization tends to have instabilities during training, they have high rates of constraint violation during exploration. This is compounded by the implementation in Omnisafe, which only starts updating the Lagrange multipliers after 100 warm-up epochs, corresponding to 200k environment steps. This initial warm-up phase ensures the off-policy methods have enough variety in the replay buffer. In cases where the robot needs to learn in the wild, such exploration behavior might not be acceptable.
The results from our experiments put into question the suitability of the model-free method for safe reinforcement tasks, where the agent needs to learn online. Although model-free methods might have advantages in their asymptotic performance compared to model-based methods [12], they lack the ability of model-based methods to do look-ahead controlling and deciding online to avoid actions where the robot might be uncertain that might be deemed to be possibly dangerous. Making model-based methods more suitable for learning on physical robots, especially when the robot needs to explore under certain restrictions.
4.3 Comparison to Model-based Baselines
We compare our method to two similar model-based approaches: LOOP [6] and the model-based version of Recovery RL. [7].
LOOP integrates learned value functions with planning for decision-making Unlike our method, LOOP uses look-ahead control to maximize rewards and minimize constraint violations at each step, does not account for epistemic uncertainty in planning, and assumes access to the ground-truth constraint cost function. Despite this, CASE outperforms or matches LOOP in constraint adherence without access the real constraint cost function. Additionally, CASE demonstrates superior task performance, except in the cartpole environment, where LOOP has a higher constraint violation rate. As LOOP requires optimising look-ahead control problem in every environment step, we find that

---

CASE PAPS
Fecseryll
110
400
330
140
12
82
28
02 2/2 5105
28 5.5
20
2.6
340
Figure 3: CASE outperforms both LOOP and Recovery RL in terms of constraint violations across constrained environments in the RWRL and Safe- Control-Gym benchmarks. Although CASE shows slightly higher violations in the walker environment compared to LOOP (still under 5%). has the added challenge of predicting constraint costs using the learned model. In contrast, LOOP uses the environment's 's ground-trume constraint function to calculate costs, giving it an advantage in constraint adherence that CASE achieves without such prior knowledge.
it is much less compute efficient than CASE, we discuss the differences in frequency between CASE and similar methods in appendix C.
In contrast, Recovery RL uses a model-free approach learn an unconstrained tosk-solving policy and safety critic. The model-based version introduces forward model for triggering interventions and to design look-ahead recovery controller. However, Recovery RL suffers from conflicts be- tween the task-solving policy and the recovery policy, leading to worse performance compared to CASE, as seen in figure 3.
5 Related Work
Constrained model-based reinforcement learning Different RL methods have been introduced largeting CMDPs [26, 27, 28]. The most common variants are papers leveraging the augmented Lagrangian method for learning constrained policies. CPO [29] extended TRPO to CMDP, and building on that work [30] combined the Lagrangian method with PPO learn constrained policy. Using an on-policy method rendered the method unsuitable for learning in the real world, where sample efficiency essential. In [31], the authors implemented similar approach but replaced the on-policy algorithm with off-policy approach to reduce sample complexity. Saute RL [32] provides an alternative to using Lagrangian relaxation for solving CMDPs by using state space aug- mentation. Different papers proposed MBRL methods combining learned models with constrained policy optimization. In [3], the authors use a Bayesian approximation of an RSSM [13] as a model. The posterior of the transition used to learn an optimistic objective regarding the rewards but pes- simistic regarding the constraint cost. similar approach was introduced in [4] where the authors extend SLAAC [16] to the constrained setting. Concurrent with our work, safeDreamer proposes combining planning with CMBAC. However, SafeDreamer differs in not leveraging the planner as backup policy, is the case with CASE. Further, safeDreamer does not consider the epistemic uncertainty into maassion-makingg as proposed in our method.
Safety filters Learning a policy to satisfy constraints and, at the same time, maximize the agent's s utility can be a challenging task. As a result, multiple methods have attempted to circumvent this problem by introducing backup policies π ssfff(s;) that minimize constraint costs addition to a task- reward maximizing base policy (sx). Model-free methods that use backup mechanism [7, 33] generally an off-policy critic to trigger the intervention mechanism. Critics tend to be sensitive to cat-of-distribution data and, in general, hard to learn. The two-policy setup is also investigated
7

---

in [34], where the authors improve on the safety layers approach [35] by replacing the layers with a separate parametric policy that allows them to handle more complicated constraints.
The predictive safety filter paper [8] introduces an MPC wrapper to a base policy to prevent con- straint violations during learning, relying on assumptions about the model and the system dynamics and the availability of a controller that keeps the system in predetermined safe terminal state to have guarantees on constraint satisfaction. In [36], a control barrier function (CBF) is used in learn- ing a safe policy, but their method is limited to control-affine systems and assumes prior knowledge of a simplified model and access to a forward invariant safe set. In general, our method differs from other safety filter methods based on CBFs and Hamilton-Jacoke reachability in that we do not make any assumptions on the underlying CMDP and do not assume prior knowledge of the nom- inal dynamies of the systems. This contrasts to CBF and HJ safety filters where it is common to make assumptions the dynamics [37, 38, 39, 40, 36, 41J, Lipschitz continuity [37, 39, 40, 36] or existence of prior knowledge of prior safe sets [36]. SAILR [42] provides advantage-based intervention mechanism and derives performance bounds under the assumption of having an MDP with an absorbing state. However, their model-based experiments seem to be based on engineered
models.
Online planning using offiine learned functions Combining online planning with offline learned functions has proven effective in several works [18, 6, 17]. Online planning suffers less from bias compared parameterized policies, and offline-leareed functions enhance the planner's per- formance [43]. The idea of planning using offline learned functions was shown to improve on parametric policies in [18], and performance bounds were derived that show the benefits of using planner in combination with a value function. However, the methods and the bounds introduced all used an unbiased dynamics model. In [6], these insights were extended to learned models.
Conclusions and Limitations
We present a method that combines constrained model-based policy optimization with a pessimistic planning- based safety filter for exploration in CMDPs with the aim of facilitating the learning of effective policies entirely on physical robots. We find that leveraging the ability of model-based methods to function as simulator for actor-critic methods and being used in look-ahead-control schemes, thus combining the advantages of model-based actor-critic with low-bias properties of planning, can lead to significant reduction in constraint violation in comparison with model-free and model-based baselines. We evaluated our method on different constrained RI. benchmarks, which show the versatility of our method in adapting to different task dimensionalities and its ability in robotic tasks such as drone trajectory tracking and stabilization.
Our method differs from previous safety filtering methods [8. 71, where the base policy me is un- aware of constraints and the agent's safety is left to the backup mechanism, often causing oscillatory behavior [44]. We assume no prior knowledge of the CMDP's dynamics and make no assump- tions regarding its properties as done in previous safety filtering methods [36, 8, 393 and aim for a versatile method applicable to any CMDP, including those with contact dynamics discontinuities and high dimensional observations such as images. We believe combining our approach with other masel-based actor-critic methods [45, 46] can enable efficient robot learning from high dimensional observations with minimal supervision.
Limitations The main limitation of our method lies in the inability to guarantee constraint sat- isfaction. Such guarantees would involve making assumptions on the target system, such as the smoothness of the dynamics and the availability of a safe terminal set as methods leveraging HJ reachability and CBF [8, 36, 40, 37, 38]. Such assumptions might not be satisfied in many target systems of interest. Although our method leads to less constraint violation than the baselines dur- ing exploration, online planning can be computationally expensive. Furthermore, our design for the learned transition model limits method to fully observable systems, which aim to rectify in future work by leveraging models that can learn the dynamics of partially observable systems.
8

---

References
[1] E. Altman. Constrained Markov Decision Processes, volume 7. CRC Press, 1999.
[2] A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. International Conference on Machine Learning, pages 9133-9143.
PMIR, 2020.
[3] Y. As, I. Usmanova, S. Curi, and A. Krause. Constrained policy optimization via bayesian world models. In International Conference Learning Representations, 2021.
[4] Y. Hogewind, T. D. Simão, T. Kachman, and N. Jansen. Safe reinforcement learning from pixels using a stochastic latent representation. In The Eleventh International Conference on Learning Representations, 2022.
[5] M. Wen and U. Topeu. Constrained cross-entropy method for safe reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
[6] H. Sikchi, W. Zhou, and D. Held. Learning off-policy with online planning. Conference on Robot Learning, pages 1622-1633. PMLR, 2022.
[7] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, E. Gonzalez, Ibarz, C. Finn, and K. Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, pages 4915-4922, 2021.
[8] K. P. Wabersich and M. N. Zeilinger. A predictive safety filter for learning-bast control of constrained nonlinear dynamical systems. Automatica, 129:109597, 2021.
[9] Ji, J. Zhou, B. Zhang, Dai, X. Pan, R. Sun, W. Huang, Y. Geng, M. Liu, and Y. Yang. Om- nisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304, 2023.
[10] G. Dulac-Amold, N. Levine, J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges real-world reinforcement learning. 2020.
[11] Z. Yuan, A. W. Hall, S. Zhou, Brunke, M. Greeff, J. Panerati, and A. Schoellig. Safe- control-gym: A unified benchmark suite for safe Cearning-basee control and reinforcement learning in roboties. IEEE Robotics and Automation Letters, 7(4): 2022.
[12] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018
[13] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning, pages 2565-2565. PMLR, 2019.
[14] D. Hafner, T. Lillicrap. Ba, and M. Norouzi. Dream to control: imagination. In International Conference on Learning Repressetaatinns, 2019.
Learning behaviors by latent
[15] P. Becker-Ehmek, M. Karl, J. Peters, and P. van der Smagt. Learning to fly via deep model- based reinforcement learning, arXiv preprint arXiv:2003.08874 2020.
[16] A. X. Lee, A. Nagahandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein-
forcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
[17] N. A. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In International Conference Machine Learning, pages 8387-8406. PMLR, 2022.

---

[18] K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, 2018.
[19] X. Chen, C. Wang, Z. Zhou, and K. W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2020.
[20] C. E. Luis, A. G. Bottero, J. Vinogradska, F. Berkenkamp, and J. Peters. Model-based uncer- tainty in value functions. In International Conference on Artificial Intelligence and Statistics, pages 8029-8052. PMLR, 2023.
[21] K. Ghasemipour, S. S. Gu, and O. Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267-18281, 2022.
[22] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14122-1442,, 2020.
[23] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline re- inforcement learning. Advances in neural information processing systems, 33:21810 13:21810-21825, 2020.
[24] Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[25] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[26] K.-C. Hsu, D. P. Nguyen, and Fisac. Isaacs: Iterative soft adversarial actor-critic for safety. Learning for Dynamics and Control Conference, pages 90-103, PMLR, 2023.
[27] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint arXiv:2401. / /7583, 2024.
[28] K.-C. Hsu, V. Rubiess Royo, C. J. Tomlin, and J. F. Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. arXiv preprint arXiv:21 2021.
[29] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[30] A. Ray, Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01700 7(1):2, 2019.
[31] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. arXiv preprint arXiv:2002.08550, 2020.
[32] A. Sootla, A. I. Cowen- T. Jafferjee, Z. Wang, D. H. Mguni, J. Wang, and H. Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pages 20423-20443. PMLR, 2022.
[33] H. Bharadhwaj. Kumar, N. Rhinehart, S. Levine, Shkurti, and Garg. Conservative safety critics for exploration. In International Conference on Learning Representations, 2020.
[34] H. Yu, W. Xu, and H. Zhang. Towards safe reinforcement learning with safety editor policy. Advances in Neural Information Processing Systems, 35:2608-260 2621, 2022.
10

---

[35] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa. Safe exploration in continuous action spaces. arXiv preprint /01.00800008777 2018.
[36] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387-3395, 2019.
[37] K.-C Hsu, H. Hu, and J. F. Fisac. The safety filter: A unified view of safety-critical control in autonomous systems. Annual Review Control, Robotics, and Autonomous Systems, 7, 2023.
[38] L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig. Safe learn- ing roboties: From learning- based control safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):411-44)) 2022.
[39] J. Choi, F. Castaneda, C. J. Tomlin, and K. Sreenath. Reinforcement learning for safety- critical control under model uncertainty, using control Iyapunov functions and control barrier functions. arXiv preprint arXiv:2004.07556 2020.
[40] C. Dawson, B. Lowenkamp, D. Goff, and C. Fan. Learning safe, generalizable perception- based hybrid control with certificates. IEEE Robotics and Automation Letters, 1(2):1904-1911, 2022.
[41] N.-M. Kokolakis, K. G. Vamvoudakis, and W. Haddad. Reachability analysis-based safety- critical control using online fixed-time reinforcement learning. In Learning for Dynamics and Control Conference, pages 1257-1270. PMLR, 2023.
[42] N. C. Wagener, B. Boots, and C.-A. Cheng. Safe reinforcement learning using advantage-basel intervention. In International Conference Machine Learning, pages 10630-10640, PMLR, 2021.
[43] A. Mesbah, K. Wabersich, Schoellig, M. Zeilinger, Lucia, T. A. Badgwell, and J. A. Paulson. Fusion of machine learning and mpe under uncertainty: What advances are on the horizon? In 2022 American Control Conference (ACC), pages 342-357. IEEE, 2022.
[44] F. P. Bejarano, L. Brunke, and A. P. Schoellig. Multi-step model predictive safety filters: Reducing chattering by increasing the prediction horizon. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 4723-4730 IEEE, 2023.
[45] P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning, pages 2226-2240 PMLR, 2023.
[46] P. Lancaster, N. Hansen, A. Rajeswaran, and V. Kumar. Modem-v2: Visuo-motor world models for real-world robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7530-7537. IEEE, 2024.
[47] A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of gen- eralization. Advances in neural information processing systems, 33:4697-4708, 2020.
[48] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[49] J. Ji, B. Zhang. J. Zhou, X. Pan, W. Huang. R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
11

---

A Ablation Studies
3893
3330
Figure 4: For ablating design choices the safety filter. We evaluate the average rewards per episode in the last 10 evaluations (left) and the average percentage of steps with constraint violations in an episode during exploration (right). We first evaluate the role of the safety filter CASE. Exploring with only To leads to a significant increase in constraint violations. Next, we evaluate the effect of the base policy rollouts. We see decrease the average rewards reached after exploration, especially in the ant environment. Finally, to evaluate the effect of pessimism in the filter, we optimize the filter by using the mean over the ensemble predictions instead of considering the worst-case ensemble prediction. Without pessimism, the filter tends to have higher constraint violation rates.
Effect of safety filter To assess the effect of the safety filter in our method, we explore CASE without the safety filter. In figure 4, we see that exploring solely using the constrained policy leads to higher constraint violation rates. This indicates the necessity of the filtering mechanism in our method. By looking at the maximum rewards reached after exploration in figure 4, we see that the increase rewards after removing the safety filter insignificant except the halfcheetah environment, where the constraint violation rates more than double after removing the filter.
Effect of base policy term in the filter objective The filter objective in (5) includes rolling out the base policy Te in the tail of the look-ahead trajectory of the controller. We ablate this term by replacing it at the tail of the look-ahead trajectory, thus maintaining fixed total horizon length. This equates to the following objective
t+H Σ
min EE Se H (lleeu
mux (ser)
The purpose of the base-policy rollout is inform the look-ahead controller of the base policy's behavior and thus steer the system towards regions of the state space where the base policy can perform without violating constraints. Removing the base policy rollouts leads to slightly worse task performance, especially in the ant environment. However, the difference in task performance less significant than expected. In future work, exploring other possibilities for combining the intervention controller with constrained base policy, such as penalizing the difference between the intervention mechanism's 's actions and those of the base policy done in [8], might be interesting.
Effeet of pessimism in the safety filter In our safety filter design, leverage pessimistic loss to consider the effect of epistemic uncertainty during exploration. To study the effect of pessimism on the filter, we use the mean over the ensemble prediction constraint critic predictions mean instead of using the maximum aggregation over the ensemble of the constraint critic predictions ssafe duced in section 3.2. We found that in our setup, removing the pessimism leads to higher constraint violation rates, especially in the more complex halfcheetah and ant environments.
max intro-
Effeet of Including Termination flag in Constraint Critic's Targets The constraint critic target described in 3.1 does not include the termination flag d, The termination flag is usually needed in environments with termination conditions, which assumes that terminating states are absorbing
12

---

(a)
(b)
Figure 5: We compare the predictions of the constraint critic when including the terminal flag in its target calculation in 5a and excluding the terminal flag in 5b for two different episodes with early termination. The top row shows the ground truth constraint costs, and the bottom row shows the constraint critic predictions for each state the episodes. In 5a, we see the effect of adding the termination flags in the targets, where the critic predicts a lower constraint cost near the terminal state. Excluding the terminal state flag leads the constraint critic in 5b avoid this effect.
quad2D stab 12.01 + 0.86
quad2D track
LOOP
11.68 + 0.75
LOOP Slow
0.94 0.01
0.55 0.01
Table 1: Comparing FPS for LOOP and LOOP Slow. We see that reducing the controller budget
massively increases the FPS while having no effect on the performance, as seen in figure
states. Including the termination flag in the task critic V leads the critic to predict low values for such states, incentivizing the policy to avoid termination. This effect is not desired in the constraint critic Daae as it would lead the constraint critic to assign erroneous cost-to-gos to states near the termination states, which would have an adverse effect on the safety filter and the constrained policy optimization. We compare the predictions of two constraint crities in figure The first in figure 5a trained using targets including the termination flag de
(s:) = Iz + snfe (1 de) ((1 ssafe )u (8t) + s Cr where,
C' 11.) ve,1 (..)
The second in figure 5b is trained with the targets described in (5). We find that including de in TD (A) returns has a negative effect on the constraint critic predictions where the critic assigns much lower values states near the termination state, despite constraint violations.
B Deep Ensemble Transition Models
Due focus exploration, need model that provide well-calibrated epistemic uncer- tainties. Deep ensemble models present straightforward approach to provide representations of the epistemic uncertainty due to their ability to provide good approximations of the Bayesian posterior predictive distribution of the neural network [47]. Thus, we rely on an ensemble of dynamic models Pe {Pp.1..... Pv.n). where each ensemble member Pv.i is a neural network that predicts the transition as a Gaussian distribution with diagonal covariance (# St, ar) = A5(++11 (St ot). E. We optimize each ensemble member by minimizing the negative loglliiid
Cpo.c
log Pe.((S441 $t, st)a.).
The use of ensemble-hased transition models has already shown very good performance in model- based RL papers such [48] and [12]. In deep ensemble transition models, each ensemble member Pv.s captures the aleatoric uncertainty of the ground-truth MDP. In contrast, the disagreement be- tween the ensemble members captures the epistemic uncertainty the learned transition function.
13

---

Quadrecor 20 s18
quadrotor track
230
1.0
30
50
0.0
5.5
1.0
1.3
2.5
25
30
35
0.2
0.5
1.0
1.5
20
2.3
50
35
CondErain wirlatonn rate
15
30
60
20
Invilumment
1ed
1et
Figure 7: To reduce the runtime LOOP seen table 2, we tune the budget the sampling- -based look-ahead control used in LOOP. LOOP Slow is LOOP with the original hyperparameters, LOOP the variant with the updated HPs. We see that both HP sets have similar results, and we see in table the massive increase in FPS with the new HPs.
C Runtime
Although using a look-ahead planner as a safety filter helps reduce the constraint violation rates of the reactive parametric base policy, this comes at the cost of computational efficiency. The safety filter involves an online optimizer that solves the optimization problem in objective (5). We compare the fre- quency of the filter across the different environments in figure 6, and we find that the filter performs around 50 Hz in all en- vironments. Looking at other methods combining actor-critic methods and planning, we find that TD-MPC [17], which does consider the constrained setting, performs similarly to our method in runtime with about 50 Hz for the default setting. LOOP [6], which also explores CMDPs among other settings, reports lower frequency 14.3 Hz. Generally, the low fre- quency of planning-based methods represents the biggest dis- advantage compared to methods leveraging only parametric policies for decision-making.
50
waller
Figure 6: Average runtime for CASE. We find that despite the on- line computations, we maintain reasonable FPS that makes CASE feasible to use in physical systems.
Experimental Setup Details
D.1 Benchmarks
We use the constrained velocity tasks from safety gymnasium [49] in our experiments. Specifically, we use the half cheetah, hopper, walker, and ant tasks. These tasks are attractive as they pose conflicting objectives, where the task reward incentivizes the agent to move with high velocity with correct pose, and the constraint cost punishes the agent for moving above a velocity limit.
D.2 Tuning LOOP for Safe Control Gym Tasks
In training LOOP [6] for the RWRL benchmark tasks, used the hyperparameters (HPs) without tuning (we also do not tune method). We noticed that the runtime for LOOP is extremely long and much worse than the latency mentioned in the paper when evaluated in high dimensional
14

---

cartpole
walker
quadruped
CASE (Ours)
70.5 + 3.3
69.03 = 1.83
79.7 7.8
LOOP
7.9 0.13
1.12 0.03
0.88 0.016
Table 2: FPS for CASE in comparison with LOOP [6]. We find that despite the online computations,
we maintain a reasonable FPS that makes CASE feasible to use in physical systems.
environments, as seen table We found that this mostly due to the loops in sampling- based look-ahead controller in LOOP. To avoid the long LOOP training times in our experiments in safe-control-gymn
tuned the look-ahead controller aiming decrease LOOP's latency while maintaining the same performance. We compare the performance of the new LOOP HPs with the original LOOP HPs on two of the safe-control-ge gym tasks in figure 7, and we compare their runtimes in table 1.
D.3 Hyperparameters
Our method involves hyperparameters for the model training, the CMBAC, and the safety filtering- We list our hyperparameters in table 3.
HalfCheetah
Hopper
Ant
Walker
Model Ps
number of bootstraps B
learning activation
1e-3
softsign
number of hidden layers
number of hidden units
200
Critic V& and safety critic Vaate
Horizon
12
Activation
softsign
TD roward
0.9
discount
0.99
safety critic discount rafe
0.9
safety fo
0.75
number of hidden layers
2
number of hidden units
256
Policy
Horizon
Activation
ReLU
learning rate
5e
number of hidden layers
2
number of hidden units
256
Polyak factor
0,995
Learning step lagrangian multiplicrs Filter
3e-4
Filter Horizon
number optimization steps
50
learning rate
1e-1
sade
Table 3: Hyperparameters for CASE
15


Reviews:
[{"review_id": "9hZL2y08Qw", "reviewer": "Reviewer_Frme", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper provides a novel approach for constrained model-based RL for reducing constraint violation during exploration by utilizing planning based safety filter as a backup policy. In addition, they utilize deep ensemble models to capture epistemic uncertainty in learned model thereby improving the ability to satisfy constraints. For instance, the filter is more often triggered in areas with high epistemic uncertainty leading to lesser safety violations during exploration.", "summary_of_recommendation": "Please read the review above.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "The paper claims that their proposed method helps in reducing constraint violations during training as compared to other constraint model-based optimization methods but in the results section they only compare their method with model-free baselines which is not what they claim in the abstract. How their methods compares with the Recovery RL paper for example should be shown. Also in \"our contribution\" section where discussion about Figure 1 is shown, there the authors claim comparison to CMBRL methods which is not the case.\n\nThe paper also uses only simple safety constraints such as constrained velocity rather than considering complex scenarios such as obstacle avoidance or multiple safety constraints. Please add few experiment with some additional safety constraints.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WpVwgoUfS1", "reviewer": "Reviewer_QmYe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a model-based safe reinforcement learning (RL) method that incorporates a learned Markov Decision Process (MDP) and Model Predictive Control (MPC) to ensure safe exploration. The proposed method uses ensembles for each component to determine the worst-case scenario, which then triggers a safe MPC controller to ensure safety during exploration. The method is validated in the Safe-Mujoco tasks, showing fewer violations during the training process.", "summary_of_recommendation": "The paper presents a model-based safe RL approach. Integrating rollout, ensembles, and MPC for safe-critical problems are rational choices. However, the futher details of the experiments are needed to justify the claim.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "Please find them in the comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLhA8Z87HA", "reviewer": "Reviewer_1NEA", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a model-based reinforcement learning (MBRL) approach that leverages safety filters to improve safety at training time. This is in contrast to existing MBRL that does not explicitly account for safety during exploration. The proposed approach uses planning-based backup policies to lower constraint violations during training. The method is tested with simulated examples.", "summary_of_recommendation": "My recommendation is based on these weaknesses of the paper: lack of novelty, several technical issues, and dubious relevance to robotics. See my review for detailed justifications.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 1, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "* What is the correct mathematical definition of safe/unsafe state sets? Why did the authors use indicator functions to define the safety objective? It has been shown to underperform the reach-avoid objectives that render dense rewards and much better safety performance (see, e.g., He et al., Agile but Safe, RSS 2024).\n* What is the novelty of the paper aside from assembling existing approaches of MBRL and safety filters?\n* What is a proper characterization of the proposed optimization formalism? Is it proper to call it \"constrained optimization\" while the actual problem being solved is unconstrained?\n* Is the safety filter defined by (7) a principled approach to improve robustness in RL? Why did the authors choose not to use state-of-the-art safety filter methods such as HJ Reachability or CBFs?\n* How does the proposed approach perform for realistic robotics tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the problem solved is important and relevant for the robotics community. Unfortunately, the paper has three main weak points that should be handled in the rebuttal:
- The method seems to be not completely novel, and rather incremental. The authors should clarify the novelty of the approach.
- There seem to be some technical issues and unclear points in the proposed theory. Please refer to the reviewers' comments for more details. I expect a detailed answer to all their comments (and follow-up responses)
- There are no real robot experiments or interesting robotics tasks beyond simple Mujoco benchmarks. Most reviewers agree that this is a major drawback, and I'll not vote for accepting this paper if a more convincing benchmark is presented, to match the scope of the CoRL conference.

===

During the rebuttal, the authors managed to convince some of the reviewers about the technical soundness of their approach. Also, the authors added a reasonable robotics experiment (the 3d quadrotor stabilization).
While some reviewers argue that the novelty of the paper is a bit limited, it is overall an ok submission.
An important point to fix is to improve the description of the optimization (after equation 8)  and to add a safety critic baseline together with lagrangian approaches.
Also, I would require the reviewers to reference in the paper some of the works mentioned by reviewer 1NEA, specifically:

[1] Hsu, Kai-Chieh, Duy Phuong Nguyen, and Jaime Fernandez Fisac. "Isaacs: Iterative soft adversarial actor-critic for safety." Learning for Dynamics and Control Conference. PMLR, 2023.

[2] He, Tairan, et al. "Agile but safe: Learning collision-free high-speed legged locomotion." arXiv preprint arXiv:2401.17583 (2024).

[3] Hsu, Kai-Chieh, et al. "Safety and liveness guarantees through reach-avoid reinforcement learning." arXiv preprint arXiv:2112.12288 (2021).


I would mark this paper for acceptance, but I would not have any problem if the program chairs decide to reject this paper.
Decision: Accept