Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems
Authors: Haozhe Tian
Homayoun Hamedmoghadam
Robert Noel Shorten
Pietro Ferraro
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Safe Crticial System Control
Policy Regularization
Primary area: reinforcement_learning

Abstract: Reinforcement Learning (RL) is a powerful method for controlling dynamic systems, but its learning mechanism can lead to unpredictable actions that undermine the safety of critical systems. Here, we propose RL with Adaptive Regularization (RL-AR), an algorithm that enables safe RL exploration by combining the RL policy with a policy regularizer that hard-codes the safety constraints. RL-AR performs policy combination via a "focus module," which determines the appropriate combination depending on the state—relying more on the safe policy regularizer for less-exploited states while allowing unbiased convergence for well-exploited states. In a series of critical control applications, we demonstrate that RL-AR not only ensures safety during training but also achieves a return competitive with the standards of model-free RL that disregards safety.

Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems
Haozhe Tian" Homayoun Hamedmoghadam
Robert Shorten
Pietro Ferraro
Dyson School of Design Engineering
Imperial College London SW7 2AZ, London, UK
(haozhe. tian21, h. hamed, r. shorten, p.ferraro)@immeriall .ac. acc uk
Abstract
Reinforcement Learning (RL) is powerful method for controlling dynamic sys- tems, but its learning mechanism can lead to unpredictable actions that undermine the safety critical systems. Here, we propose RL with Adaptive Regularization (RL-AR), an algorithm that enables safe RL exploration by combining the RL pol- icy with a policy regularizer that hard-codes the safety constraints. RL-AR per- forms policy combination via a "focus module," which determines the appropriate combination depending on the stat-rrclying more on the safe policy regularizer for cess-exploited states while allowing unbiased convergence for well-exploited states. In a series of critical control applications, we demonstrate that RL-AR not only ensures safety during training but also achieves return competitive with the standards of model-free RL that disregards safety.
Introduction
A wide array of control applications, ranging from medical to engineering, fundamentally deals with critical systems, i.e., systems of vital importance where the control actions have to guarantee no harm to the system functionality. Examples include managing nuclear fusion [Degrave et al., 2022], performing robotic surgeries [Datta et al., 2021], and devising patient treatment strategies (Komorowski al., 2018]. Due to the critical nature of these systems, the optimal control policy must be explored while ensuring the safety and reliability of the control algorithm.
Reinforcement Learning (RL) aims to identify the optimal policy by learning from agent's 's in- teractions with the controlled environment. RL has been widely used to control complex systems [Silver et al., 2016, Ouyang et al., 2022]; however, the learning of an RL agent involves trial and error, which can violate safety constraints in critical system applications [Henderson et al., 2018, Recht, 2019, Cheng al., 2019b]. To date, developing reliable and efficient RL-based algorithms for real-world "single-life" applications, where the control must avoid unsafety from the first trial [Chen al., 2022), remains challenge. The existing safe RL algorithms either fail ensure safety during the training phase [Achiam et al., 2017, Yu et al., 2022a] or require significant computational overhead for action verification [Cheng al., 2019a, Anderson et al., 2020]. As result, classic control methods are often favored critical applications, even though their performance heavily relies on the existence of an accurate model of the environment.
Here, we address the safety issue of RL in scenarios where "estimated" environment models are available (or can be built) derive sub-optimal control policy priors. These scenarios representa- tive of many real-world critical applications [Hovorka et al., 2002, Liepe et al., 2014, Hippisley-Cox et al., 2017, 7, Rathi et al., 2021]. Consider the example of devising control policy that prescribes the
"Correspondngg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

optimal drug dosages for regulating patient's health status. This is a single-life setting where no harm to the patient is tolerated during policy exploration. From available records of other patients, an estimated patient model can be built to predict the response to different drug dosages and ensure adherence to the safety bounds (set based on clinical knowledge). However, a new patient's 'ss response can deviate from the estimated model, which poses significant challenge in control adaptability and patient treatment performance.
We propose a method, RL with Adaptive Regularization (RL-AR), that simultaneously shows the safety and adaptability properties required for critical single-life applications. The method interacts with the actual environment using two parallel agents. The first (safety regularizer) agent avoids unsafe states by leveraging the forecasting ability of the estimated model. The second (adaptive) agent is a model-free RL agent that promotes adaptability by learning from actual environment in- teractions. Our method introduces a "focus module" that performs state-dependent combinations of the two agents' policies. This approach allows immediate safe deployment in the environment by initially prioritizing the safety regularizer across all states. The focus module gradually learns to ap- ply appropriate policy combinations depending the state-relying more on the safety regularizer for less-exploited states while allowing unbiased convergence for well-exploited states.
We analytically demonstrate that: i) RL-AR regulates the harmful effects of overestimated RL poli- cies, and ii) the learning of the state- detendennnd focus module does not prevent convergence to the optimal RL policy. We simulate series of safety-crite crittcal environments with practically obtainable sub-optimal estimated models (e.g., from real-life sampled measurements). Our empirical results show that even with more than 60% parameter mismatches between the actual environment model and the estimated model, RL-AR ensures safety during training while converging to the control performance standard of model-free RL approaches that prioritize return over safety.
2 Preliminaries
Through environment interactions, an RL agent learns a policy that maximizes the expected cumula- tive future reward, i.e. the expected return. We formalize the environment as a Markov Decision Pro- cess (MDP) M (S, 7), where S is finite set of states, A = {a R2 a convex action-space, P x > P(S) ( is the state transition function, r S x A - 1 Rmaxx Rmax] the reward function, and y (0, 1) is discount factor. Let π denote stochastic policy + P(A), the value function V and the action-value function Q" are:
7) is
(*t(s) = E Ear,
Σ (41))
(" = E
Σ (rr(set (1)
where at ~ (st). Sc+1 ~ P(s,, a.) for t 2 0. The optimal policy π = argmax, (s) maxi- mizes the expected return for any state Both V and Q" satisfy the Bellman equation [Bellman, 1966]:
(*(() Eu,s' (r(s,a) + ++)) (=((, = Ey, [r(s,a) + +)Boraa(d 1Q (s') (2)
For practical applications with complex S and A, Q and N are approximated with neural networks Qo and To with learnable parameters ó and 0. To stabilize the training of Q and ny. they are updated using samples B from a Replay Buffer D [Mnih et al., 2013], which stores each previous (s, ,T, d), where d equals I for terminal states and 0 otherwise.
environment transitions
In this work, we are interested in acting on safe regularized RL policy that can differ from the raw RL policy. RL approaches that allow learning from different acting policy are referred to as "off-policy" RL. The RL agent in our proposed algorithm follows the state-of-the-an off policy RL algorithm: Soft Actor-Critic (SAC) [Haarnoja et al., 2018], which uses a multivariate Gaussian policy to explore environmental uncertainties and prevent getting stuck in sub-optimal policies. For Q-network updates, SAC mitigates the overestimation bias by using the clipped double Q-learning which updates the two -entworkks Q i = 1,2 using gradient descent with the gradient:
Σ (8,0,8' Fdd)EE
(s, - = 1,2,
(3)
&rong (s',a)) - a log Pao (d)lss)) a'~ mo(s), 1=122
y = r 711 -
min Q

---

where the entropy regularization tern log Pxo (a' s') encourages exploration, thus avoiding local optima. Target Q-networks Otarg.i are used reduce drastic changes in value estimates and stabilize training. The target Q-network parameters are initialized with (Paag.,i (4., i = 1, Each time Φ1 and Φ2 are updated, 0targ,1, Ptarg.2 slowly track the update using 7 E (0, 1):
otarg, = Totargi + (1 - T)d,, == 1,2.
(4)
For policy updates, the policy network To is updated using gradient ascent with the gradient:
min Qo (s)) a log Pro (a l))) ao(s) ~ (s))..
V
(5)
B
1=1,2
SEB
Methodology
Safety regularizer reg
Areg
Focus module B
Off-policy RL Fl
(s)
ap (s) ((s))reg()) - (1-B(s))andC
[(s,a,s',r,d)) update
update B(s) a (s, cs()))
[(s, (re.)
Replay buffer D = {e}
RL-AR
(s,a,s', areg)
a
Environment
Schematic overview of the proposed RL-AR algorithm. RL-AR integrates the policies of the RL agent and the safety regularizer agent using state-dependent focus module, which is updated to maximize the expected return of the combined policy.
Figure 1:
Here, we propose RL-AR, an algorithm for the safe training and deployment of RL in safety-critical applications. A schematic view of the RL-AR procedures is shown in Fig. 1. RL-AR comprises two parallel agents and focus module: (i) The safety regularizer agent follows a deterministic policy proposed by a constrained model predictive controller (MPC); (ii) The off-policy RL. agent is an adaptive agent with S + P(A) that can learn from an acting policy that is different from Ty₁ (iii) The focus module learns state-dependent weight 3 S > [0, 1] for combining the deterministic (s) (s) and the stochastic as)s) (s)).. Among the components, the safety regularizer has built-in estimated environment model f : x A - S that is different from the actual environment model, while the off-policy RL agent and focus module are dynamically updated using observed interactions in the actual environment.
Trgg
The RL-AR workflow is as follows: (i) (reg(s) generates Oreg(s), which hard-codes safety con- straints in the optimization problem over a period forecasted by f. The forecasting ability antic- ipates and prevents running into unsafe states for the critical system; (ii) (a(s)) generates an to allow stochastic exploration and adaptation to the actual environment; (iii) 3(s) is initialized to B(s) 2 1 c, Vs E S, hence prioritizing the safe Treg before ny learns viable policy. As more interactions are observed for state and the expected return of xa((s) improves, B(s) gradually shifts the focus from the initially suboptimal (T)g(s) to (8).
3.1 The safety regularizer
The safety regularizer of RL-AR is a constrained MPC, which, at any state 8t, optimizes the N- step system behavior forecasted using the estimated environment model f by solving the following constrained optimization problem:
HN-1
min
Σ
Ju ax) + (N(
(6)
s.t. Sk+1 f(sk, ax), (5x) 0, E A.
where Ja(S,, ag) and NN) are the stage and terminal cost functions and g(x)) is the
safety constraint. By hard-coding the safety constraints in the optimization (via ( 0) over the
3

---

Algorithm RL-AR
1: Initialization: empty replay buffer D; MPC controller with estimated environment model f:
policy network rT: Q-networks Qs and target Q-networks Qrearg. with Starg, Φi i == 1, pretrained focus module By with (s) 2 1 e, V's E S: time step t O.
2: repeat
3: Observe state 8
4: Take action a = (x) (5) + (1
B, (x) (s, a, 8' ,,, d, (reg) in 'D
areg (8) = Trog (0), ar((s) TO(
5: Store transition
6: if > time to update then
7:
Randomly sample batch B of transitions from D Update Qo 2, by gradient descent with Eq. (3) Update Te by gradient ascent with Eq. (5)
8:
9:
Update 3g by gradient ascent with:
V
min Qs; (s, Bo (s) + (1 (8)))a((()), aa(s) ~ To())
11: Update Q Deerggat 1, using Eq. (4)
12: end if
13:
t == +++1
14: until convergence true
prediction horizon, MPC prevents failure events that are not tolerated in critical applications. MPC iteratively solves for the N-step optimal actions in each time step and steers the environment to the desired state. At any time step f, solving the optimization problem in Eq. (6) yields a sequence of N actions N1.+ with only the first action a, in the sequence adopted for the current time step, i.e., (reg((5)) = Treg = dg. The system transitions from 8; 81411 by taking the action Greg (sx). and the optimization problem solved again over {++ 1: applications with continuous state space, the optimization problem in Eq. (6) is efficiently solved using the Interior Point Optimizer [Andersson et al., 2019]. Since MPC solves similar problems with slight variations at each time step, the computational complexity is further reduced by using the solution from the previous step as the initial guess.
+ N} to obtain (re4( For practical
3.2 Policy regularization
The focus module in RL-AR combines the actions proposed by the safety regularizer agent and the RL agent using weighted sum. At state s, the combined policy πy takes the following action as(s):
(s(s) = (())a((((())))) + (1- (s)))ans)), (seg(() = Treg (8), an(s) ~ TT(())
(7)
Lemma 1. (Policy Regularization) In any state E S. for multivariate Gaussian RL policy mi
with mean ( and covariance matrix tion of the combined action as(s) derived from Eq. (7) is the solution to the following regularized optimization with regularization parameter X ((s) 3s))):
diag(o (s), (s)...... ..... (s)) Bkxk the expecta-
3(s)
E [as(s)) = argmin ||a [n(s)]s> + 1 [+)) ||a (reg())]ls-
(8)
a
We provide the proof of Lemma in Appendix A.I, which is a state-dependent extension of the proof in [Cheng et al., 2019b]. Lemma shows that the state-dependent ((s) offers a safety mech- anism on top of the safety regularizer. Since 3(s) is initialized close to for Vs E S, strong regularization (A + oo) from the safety regularizer policy is applied at the early stages of training. As learning progresses, the stochastic combined policy inevitably encounters rarely visited states, where Te. is poor due to the overestimated Q. However, the regularization parameter À remains large for these states, thus preventing the combined policy from safety violations by regularizing its deviation from the regularizer's 's safe policy. This deviation is quantified in the following theorem.
Theorem 1. Assume the reward R and the transition probability P of the MDP M are Lipshitz con- tinuous over A with Lipschitz constants Lg and Lp. For any state E S, the difference in expected

---

return between following the combined policy T and following the safety regularizer policy Treg i.e., |V* (s) V (s)1, , has the upper-bounds
Va (+)1 (5)5= (1->)]S)L L
[V (8)
(1 -1- 3(5))
(1 (1-7)2
where [S] is the cardinality of S, Aa = an(s)) arog (ren(s)] is the bounded action difference at 8.
We provide the proof of Theorem 1 in Appendix A.2. Theorem 1 shows that when state has not been sufficiently exploited and its corresponding (s)) updates have been limited accordingly, the sub-optimality of the RL policy πe has limited impact on the expected return of the combined policy тH. which is the actual acting policy. This is because 1 - 3(s) remains close to zero at this stage, leading only minor expected return deviations from the safety regularizer's policy Trog-
3.3 Updating the focus module
The focus module derives the policy combination (from the policies of safety regularizer and off- policy RL agent) that maximizes the expected return. For any state s, the state-dependent focus weight 3(s) is learned through updates according to the following objective:
( = argmax E [QT (s, (Brez(s) + (1 ())a(())]] arog (8) = reg an(s) ~ Ty(ss) (10) |E(U,1]
Equation (10) is similar to the actor loss in actor-critic methods, however, instead of optimizing the policy network, Eq. (10) optimizes B(s) for policy combination.
Compared to scalar combination weight that applies the same policy combination across all states (e.g., as in [Cheng et al., 2019b]), the updated state-dependent weight (s(s) in Eq. (10) guarantees monotonic performance improvement at least in the tabular cases, i.c., the update 3' (s,) at state 8 results in T (s) 2 ***(s) (s) for all states & E S, where a is the combined policy proposed by B' This can be proved by observing that the update in Eq. (10) results in non-negative advantage for all states s, i.e., Q TT') 2 Q (s, TT), Vs E S, where (with slight abuse of notation) we use (s,, T) denote (s),a) with a T(s). See Theorem in Appendix A.3 for the detailed proof.
Lemma 2. (Combination Weight Convergence) For any state s, assume the RL policy TTl converges to the optimum policy x* that satisfies Q(s, T*) > Q(s, 7), VT # ग", then 3 (s) = will be the solution to Eq. (10) that achieves the optimal policy combination.
Lemma 2 follows as Treg # x* due to the sub-optimal model used to derive Treg. Let (s) ~ T (s) denote the optimum action at state s. If (()) A O, then ((s)areg(3) + (1 B(s))an 3(s)aray (8) + (1 B(s))a² )aa (s) o a" (s). Therefore, the solution to Eq. (10), i.e., the updated focus weight 3 (s). can only be O.
Theorem 3. (Policy Combination Bias) For any state s, the distance between the combined action ag (s) and the optimal action a" (s) has the following lower-bound:
las(s) a" (s) 2 (areg (s) - a" (s)) - (1 B(s) areg (s) ar1(s)
(1I)
If a Gaussian RL policy मe converges the optimum policy π² (s) with Q(s, TT) Q(s x), VT + xt, then the combined policy Ts(s) can have unbiased convergence to the optimum Gaussian policy x² with total variance distance Drv (T))), n" (s)) =0. O.
The proof of Theorem 3 is given in Appendix A.5. Theorem 3 shows that by adaptively updating 3(s), the unbiased convergence of the combined policy can be achieved assuming i) a unique opti- mum solution and ii) the convergence of the RL agent, where the former follows naturally for most real-life control applications and the latter is well-cstablished in the RL literature (the convergence of the specific RL agent used in RL-AR was proved in [Haarnoja et al., 2018J).
Algorithm shows the pseudo-code of RL-AR, where Q, Tel. and are approximated with neural networks for practical applications with large or continuous state space. Note that the policy reg- ularization (Lemma 1) and the convergence of RL-AR to the optimum RL policy (Lemma 2 and Theorem 3) still hold when using function approximation. For the RI. agent, we take the standard approach of approximating Q" and æ with neural networks Qo and ma. is approximated with neural network 3 with outputs scaled to the range (0, 1). Before learning
The focus module (())
5

---

begins, Sy is pretrained to output values close to 1 (e.g., (y () 2 c) for all states to prioritize the safe Freg. While interacting with the environment, each transition in replay buffer DD Since Frg is deterministic and not subject to updates, by storing the action term Greg in D. the optimization problem in Eq. (6) needs to be solved only once for any state s, significantly lowering the computational cost.
(s, a, s', ,,, d, (reg) stored
In Algorithm 1, details of Qo. and hhe updates (lines 8-9) are omitted as they follow the standard SAC [Haarnoja et al., 2018) paradigm elaborated in Section 2. After updating Qo, and TT the focus module Be is updated using samples (s, Greg) from replay buffer D. As shown in line 10, B is updated using gradient ascent with the gradient:
Σ (8
min Qo (s, By(()areg + (1 (s)))))))))), ar((s) ~ mo(s). 111
(12)
Note that the updated Qd., 1, and Te are used in Eq. (12) allow quick response to new information. The clipped double-Q learning (taking the minimum Q i 1) mitigates the overestimation error. Although exploitation level not explicitly considered Eq. (10), the use of replay buffer and the gradient-based updates in Eq. (12) mean more frequently-visiter states with well-estimated Q values will affect Bi (8) more, whereas rarely-visited states with overestimated Q values affect (x.(s) less.
4 Numerical Experiments
Here, RL-AR is validated in critical settings described in Section 1, where the actual environment model P is unknown, but an estimated environment model ff is available (c.g.. from previous obser- vations in the system or similar system).. Four safety-critical environments are implemented:
Glucose is the critical medical control problem of regulating blood glucose level against meal- induced disturbances [Batmani, 2017]. The observations are denoted as (G, G. t), where G the blood glucose level, G = G. G and & the time passed after meal ingestion. The action is insulin injection, denoted as ar. Crossing certain safe boundaries of G can lead to catastrophic health consequences (hyperglycemia or hypoglycemia).
BiGlucose similar to the Glucose environment but capturing more complicated blood glucose dynamics, with 12 internal states (11 unobservable), 2 actions with large delays, and nondifferen- tiable piecewise dynamics. [Kalisvaart et al., 2023]. The observations are the same as Glucose. The actions are insulin and glucagon injections, denoted as (ar,ax).
CSTR is a continuous stirred tank reactor for regulating the concentration of a chemical On [Fiedler et al., 2023]. The observations are (CA, Cn, TR, Tk), where CA and Cn are the con- centrations of two chemicals; TR and TK are the temperatures of the reactor and the cooler, respectively. The actions are the feed and the heat flow, denoted as (af, aQ). Crossing safe boundaries of C.. CB, and TR can lead to tank failure even explosions.
. Cart Pole is classic control problem of balancing an inverted pole on cart by applying horizon- tal force to the cart. The environment is adapted from the gymnasium environment [Towers et al., 2023] with continuous action space. The observations are (x, i, 0, 0), where æ is the position of the cart, f is the angle of the pole, J = x2 - It-1, and = 02 - 0 The action is the horizontal force, denoted a-- The control fails if the cart reaches the end of its rail or the pole falls over.
All environments simulated following widely accepted models and parameters [Sherr et al., 2022, Yang and Zhou, 2023], which are assumed to be unknown to the control algorithm. The esti- mated models and the actual environments are set to have different model parameters. For Glucose and BiGlucose, the estimated model parameters are derived from real patient measurements [Hov- orka et al., 2004, Zahedifar and Keymasi Khalaji, 2022]. The environment models, parameters, and reward functions are detailed in Appendix B.
The bascline methods used in the experiments are: i) MPC [Fiedler et al., 2023], the primary method for control applications with safety constraints [Hewing et al., 2020]; ii) SAC [Haarnoja et al., 2018],
'ooee available https: //gtt///b.tttttt//// www./ooooheeiiaan//.e...oooo

---

model-free RL that disregards safety during training, but achieves state-of-the-ar normalized re- turns: iii) Residual Policy Learning (RPL) [Silver et al., 2018], an RL method that improves a sub-optimal MPC policy by directly applying a residual policy action; iv) Constrained Policy Opti- mization (CPO) [Achiam al., 2017], widely-used risk-aware safe RI benchmark based on the trust region method; and v) SEditor [Yu et al., 2022b], more recent, state-of-the-an safe RL method that learns a safety editor for transforming potentially unsafe actions.
The proposed method, RL-AR, uses MPC as the safety regularizer agent and SAC as the off-policy RL agent. The two agents in RL-AR each follow their respective baseline implementations. The focus module in RL-AR has [128, 32] hidden layer size with ReLU activation, and & outputs scaled (0, 1) by a shifted tanh. Additional detail and hyperparameters of the implementations are provided in Appendix C. Our RL-AR implementation has an average decision and update time of 0.037 seconds per step on a laptop with a single GPU, meeting real-time control requirements across all environments. In Appendix D we present ablation studies on the benefit of state-dependent focus weight and the choice of SAC as the RL. agent.
4.1 Safety of training
We begin by evaluating training safety in the actual environment by count- ing the number of failed episodes out of the first 100 training episodes. An episode is considered a failure and ter- minated immediately a visited state exceeds predefined safety bound. As shown in Table 1, only RL-AR com- pletely avoids failure during training in the actual environment. Although MPC does not fail, it does not adapt or up- date its policy in the actual environ- ment. RPL relatively safe by relying on a safe initial policy, but its un-regularized residual policy action results in less stable combined action, leading to failures. Due to their model-free nature, CPO and SEditor must observe failures in the actual environment before learning a safe policy, thus failing many times during training. Note that SAC averages the largest number of failures over all environments.
Table 1: The mean (+: standard deviation) number of fail-
ures out of the first 100 training episodes, obtained over 5 runs with different random seeds.
Method
Glue.
BiGl.
CSTR
Carl.
RL-AR
0.0
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)
MPC
0.0
0.0 (00)
0.0 (0.0)
0.0 (0.0)
SAC
19.0
59.4
99.2 (0.)
93.6 (7.3)
RPI.
7.8 (6.4)
5.6 (3:9
3..5
(2.2)
CPO
8.0 (2.1)
72.4 (6.7)
100.0 (0.0)
21.8 (3.7)
SEditor
(17)
74.6
97.2 (5.6)
17.4 (10.6)
Next, since the estimated environment model f is integrated into RL-AR, MPC, and RPL., for fair comparison we pretrain the model-free SAC, CPO, and SEditor using f as an environment simulator; this allows all methods to access the estimated model before training on the actual environment. Figure 2 compares the normalized episodic return and the number of failures for different methods over training episodes; the proposed method is compared with SAC and RPL in Fig. 2A, and with MPC. CPO, and SEditor sufett--wware methods) in Fig. 2B. The mean (solid lines) and standard deviation (shaded area) in Fig. 2 are obtained from independent runs using different random seeds. Episodes are terminated on failure, resulting in varying episode lengths, thus, the episodic returns are normalized by episode lengths.
Two important insights can be drawn from Fig. 2. First, the normalized return curves show that RL- AR consistently achieves higher returns faster than other methods across all environments. RL-AR begins with a reliable initial policy derived from the safety regularizer and incrementally integrates a learned policy, resulting stable return improvements (as suggested by Lemma and Theorem I). RL-AR shows steady return improvement, except for some fluctuations in the CSTR environment which the method quickly recovers from. In contrast, the baseline methods SAC and RPL, which apply drastic actions based on overestimated returns, or CPO and SEditor, which impose constraints using biased cost estimates derived from simulations using f e-exiibit significant return degradation and even failures. Second, RL-AR effectively avoids failure during training (see the bottom rows in Fig. 2A&B). Note that pertaining on f leads to fewer failures the actual environment for SAC, CPO, and SEditor (compare with the results in Table 1). However, SAC, CPO, and SEditor continue to fail despite the pretraining (the only exception is SEditor in the Glucose environment), indicating that pretraining on estimated model is not an effective approach to achieve safety.
7

---

Giacose
B:Glucore
CSTR
10"
Cart Pole
normalizei return
0.25
-0.9
-0.4
6.50
0.1
6.75
1.5
0.5
1.00
1.1
RL-AR SAC-pt RPL
12
1.5
mumberr of fallures
1.2
50
100
150
250
300
25
50
100
125
1.0
50
75
100
150
30
64
120
150
<pt-ondo
epiendo
sptand
opionds
B
100
Glucose
10"
BiGhucose
100
CSTR
10"
Cart Pole
nomalized retum
0.5
3.0
RL-AR MPC CPO-pt
number of faillres
SEditor-pt
50
100
150
200
250
300
25
50
75
125
150
50
75
100
150
20
60
120
150
<pisode
episods
Figure 2: The normalized return curves and the number of failures during training (standard devi- ations are shown in the shaded areas). SAC, CPO, and SEditor are pretrained using the estimated model f as simulator (as indicated by "-pt") to ensure fair comparison, given that RL-AR, MPC, and RPI. inherently incorporate the estimated model. This pretraining allows SAC, CPO, and SEdi- tor to leverage the estimated model, resulting in more competitive performance in the comparison.
In CSTR and Cart Pole environments, and only in limited number of episodes during the early stages of training, the proposed RI-AR policy's normalized return falls below that the static MPC policy. This values). Nevertheless, since (ss) is close to 1 for these insufficiently learned states, the dominance of the safety regularizer agent allows RL-AR converge to high returns without compromising the safety (as shown in Theorem l).
due to RL-AR reaching insufficiently learned states (with overestimated Q
4.2 Achieved return after convergence
Besides ensuring safer training. RL-AR theoretically enables unbiased convergence to the optimal RL policy (as shown in Theorem 2). We validate this by testing whether RL-AR matches the return of SAC. SAC is shown to consistently converge to well well-performing control policies, competitive not better than other state-of-thed -ote---rttth-ttttt RL algorithms [Raffin al., 2021, Huang et al., 2022]. In Fig. 3, we compare the control trajectories of RL-AR, SAC, and MPC and the returns of their con- verged policies after training; we run the converged policies without stochastic exploration. RI-AR significantly outperforms MPC, achieving faster regulation, reduced oscillation, and smaller steady-

---

Glucose
RiGlucase
state G1 (mg/dl.)
400
normalized
(mgid)))
400
normalized
aoturn
100
-1.37
33.60
200
-1.51
200
-7.43
desired state
4.55
destred edl state FARIE
-7.60
100
300
1000
(taas) CSTR
Unee (muas)
Cart Pole
Co (maltt))
0.05
normalizzed
normalizee
0.60
-0.10
desired stare nange
0.55
-1.74
-0.24
-8.78
desirad range
-1.54
0.50
(e) SAC (with failures during training)
RL-AR (without failure during training)
MPC
Figure 3: Comparison of the converged trajectories and their corresponding normalized return. In the upper row, the agents try to retain the desired state under vare-na disturbances; in the lower row, the agents try to steer the system to a desired state. Although SAC fails before converging, here compare with the converged SAC results to show that RL-AR can achieve the performance standard model-free RL that prioritizes return and disregards safety.
state error. Furthermore, in terms of normalized return, RL-AR is competitive with SAC in the Cart Pole environment and outperforms SAC in the other three environments. The results demon- strate that RL-AR not only effectively ensures safety during training, but also finds control policies competitive with the state-of-the-an model-free RL.
4.3 Sensitivity to parameter discrepancies
Inherently, RL-AR's training safety relies on the effectiveness of the safety regularizer, which depends on the quality of the estimated model f. Thus, large discrepancy between and the actual environment might compromise the training safety of RL-AR. We empirically quantify this effect by deploying RL-AR in dis- crepant Glucose environments created by vary- ing the environment model parameters and
n/n
1/16
2/16
3/16
4/16
5/16
6/16
64/16
P2/52
0
0
3/8
100
0
0
0
0
0
0
2/8
1000
100
100
100
0
P2 (values chosen based ñ and P2 in f' to mimic deviating characteristics new patients, and counting the number of failed episodes out of the first 100 episodes in Fig. 4. Lower P2/ P2 and n/ / makes the environment more suscep- tible to failure; see Appendix B. The results show that RL-AR can withstand reasonable dis- crepancies between f and the actual environment. Failures only occur when the actual environment deviates significantly from f with P2 < 3e and n < 6 regularizer due its misleading estimated model with largely discrepant parameters. When RI. adapts (by updating No and (x) sufficiently to correct the misleading regularizer action, the com- bined agents effectively recover from failure. Here in our tests the Glucose environment, even with large model discrepancies, RL-AR is shown to be as safe the classic MPC. Appendix Fig. provides insights into the adaptation of the focus module by showing the progression of Bp in the learning process.
1/8
Figure 4: Number of failed training episodes out of the first 100 in Glucose environment with dif- ferent degrees of parameter discrepancy.
T6 n. All failures are caused by the safety
5 Related Works
The existing safe RL works can be roughly divided into two categories [Garcıa and Fernández, 2015). The first category does not require knowledge of the system dynamics. These methods often rely probabilistic approaches [Geibel and Wysotzki, 2005, Liu et al., 2022] constrained

---

MDP [Achiam et al., 2017, Yang et al., 2020]. More recent methods use learned models to filter out unsafe actions [Bharadhwaj et al., 2020]. However, these methods need to observe failures to estimate the safety cost, thus do not ensure safety during training. This category of methods does not apply to the single-life setting in this work, i.e., no failure is tolerated in the actual environment. Nevertheless, in Fig. 2 we evaluate pertaining CPO [Achiam et al., 2017] and SEditor [Yu et al., 2022b] using simulation with the estimated model to obtain risk estimation before training in the actual environment.
The second category relies on an estimated model of the system dynamics. Some methods enforce safety constraints using Control Barrier Function (CBF) [Cheng et al., 2019a]. However, CBF min- imizes the control effort without directly optimizing the system performance. In contrast, the MPC regularizer used in RL-AR enforces safety constraints while optimizing the predicted performance, resulting in high performance during training. Some methods compute a model-based projection to verify the safety of actions [Bastani, 2021, Kochdumper et al., 2023, Fulton and Platzer, 2018]. However, the scalability of verification- based methods for complex control applications is an issue. Anderson et al. [2020] propose using neurosymbolic representations to reduce verification complex- ity, but the computational cost remains to be high. On the other hand, the average time for RL-AR to take step (including the environment interaction and network updates) in the four environments in Section 4 is 0.037 s, which is practical for real-time control.
Gros and Zanon [2019] and Zanon et al. [2020] use MPC as the policy generator and use RL to dynamically tune the MPC parameters in the cost functions and the estimated environment model. Assuming discrepancies between the estimated model parameters and the actual environment param- eters, the tuning increases the MPC's performance. However, this a strong assumption since there are other discrepancies, such as neglected dynamics and discretization errors. However, the RL-AR proposed in this work can theoretically converge to the optimal policy by utilizing the model-free RL agent.
It important to note that although the MPC regularizer accelerates the learning of the RL agent, RL-AR is not a special case of transferring learned policy. The MPC regularizer used in our proposed algorithm forecasts the system behavior and hard-codes safety constraints in the optimiza- tion. The main role of the MPC is to keep the RL-AR actions safe in the actual environment not transferring knowledge. Transfer learning in RI. studies the effective reuse of knowledge, especially across different tasks ['Taylor and Stone, 2009, Glatt et al., 2020]. By reusing prior knowledge, trans- ferred RL. agents skip the initial random trial-and-errov and drastically increase sampling efficiency [Karimpanal et al., 2020, Da Silva and Costa, 2019]. However, transferred RL agents are not inher- ently risk-aware, and thus can still steer the actual environment into unsafe states. For this reason, transferred RL is not generally considered effective for ensuring safety.
6 Conclusion and Future Works
Controlling critical systems, where unsafe control actions can have catastrophic consequences, has significant applications in various disciplines from engineering to medicine. Here, we demon- strate that the appropriate combination of control regularizer can facilitate safe RL. The proposed method, RL-AR, learns focus module that relies on the safe control regularizer for less-exploited states and simultaneously allows unbiased convergence for well-exploited states. Numerical experi- ments in critical applications revealed that RI.-AR is safe during training, given control regularizer with reasonable safety performance. Furthermore, RL-AR effectively learns from interactions and converges to the performance standard of model-free RL that disregards safety.
One limitation of our setting is the assumption that the estimated model has reasonable accuracy for deriving a viable control regularizer. Although this assumption is common in the control and safe RL literature, one possible direction for future work to design more robust algorithms against inaccurate estimated models of the actual environment. A potential approach is to update the esti- mated model using observed transitions in the actual environment. However, the practical challenge is to adequately adjust all model parameters even with small number of transitions observed in the actual environment. In addition, for such an approach, managing controllability, convergence, and safety requires careful design and tuning.
10

---

Acknowledgments and Disclosure of Funding
This project partly supported by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 101084642).
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. Advances in Neural Information Processing Systems, 33:6172-6183, 2020.
Joel A E Andersson, Joris Gillis, Greg Horn, James Rawlings, and Moritz Diehl. CasADi software framework for nonlinear optimization and optimal control. Mathematical Programming Computation, 11 (1):1 36, 2019.
Osbert Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. 2021 American Control Conference, pages 3488-3494, IEEE, 2021.
Yazdan Batmani. Blood glucose concentration control for type diabetic patients: a non-linear suboptimal approach, IET Systems Biology, 1194)1119125,, 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34 -3,, 1966.
Homanga Bharadhwajj. Aviral Kumar. Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010. 14497, 2020.
Annie Chen, Archit Sharma, Sergey Levine, and Chelsea Finn. You only live once: Single life reinforcement learning. Advances in Neural Information Processing Systems, 35:14784 14799414797, 2022.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395 2019a.
Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control reg- ularization for reduced variance reinforcement learning. International Conference on Machine Learning. pages 1 1141-1150, PMLR, 2019b.
Felipe Leno Silva Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
Shounak Datta, Yanjun Li, Matthew Ruppert, Yuanfang Ren, Benjamin Shickel, Tezcan Orrazgat-Baslane Parisa Rashidi, and Azra Bihorac. Reinforcement learning in surgery. Surgery, 170(1):329-332, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414 419, 2022.
Felix Fiedler, Benjamin Karg, Lukas Lüken, Dean Brandner, Moritz Heinlein, Felix Brabender, and Sergio Lucia. do-mpe: Towards fair nonlinear and robust model predictive control. Control Engineering Practice, 140:105676, 2023.
Ian Fox, Joyce Lee, Rodica Pop-Busui. and Jenna Wiens. Deep reinforcement learning for closed-loop blood glucose control. In Machine Learning for Healthcare Conference, pages 508 536. PMLR, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation emror in actor-critic meth- ods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and leaming. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Javier Garcia and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480 2015.
Peter Geibel and Fritz Wysotzki. sensivvee reinforcement learning applied to control under constraints. Journal of Artificial Intelligence Research, 24:81-108. 2005.

---

Ruben Glatt, Felipe Leno Da Silva, Reinaldo Augusto da Costa Bianchi, and Anna Helena Reali Costa. Decaf: deep case-based policy inference for knowledge transfer in reinforcement learning. Expert Systems with Applications, 156:1 13420, 2020.
Sébastien Gros and Mario Zanon. Data-driven economic nmpe using reinforcement learning. IEEE Transac- tions on Automatic Control, 65(2))6366644 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum en- tropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870 PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep rein- forcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Pau Herrero, Pantelis Georgiou, Nick Oliver, Monika Reddy, Desmond Johnston, and Christofer Toumazou. A composite model of glucagon-glucose dynamics for in silico testing of bihormonal glucose controllers. Journal of Diabetes Science and Technology, ((4):911955) 2013.
Lukas Hewing, Kim Wabersich, Marcel Menner, and Melanie N Zeilinger. Learning- based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, (1):269-296, 2020.
Julia Hippisley-Con Cox, Carol Coupland, and Peter Brindle, Development and validation of qrisk3 risk predic- tion algorithms to estimate future risk of cardiovascular disease: prospective cohort study. British Medical Journal, 357, 2017.
Roman Hovorka. Fariba
Paul V Carroll, Ludovic Chassin, lan J Gowrie, Nicola C Jackson, Romulus S Tudor, Margot Umplchy, and Richard H Jones. Partitioning glucose distributionnraanpoor,, disposal, and endogenous production during ivgtt. American Journal of Physiology-Fndoe Endorrinoogy and Metabolism, 282(5):9992-E1007 2002.
Roman Hovorka, Valentina Canonico, Ludovic Chassin, Ulrich Haueter, Massimo Massi Bensi-Benedetti. Marco Orsini Federici, Thomas R Pieber, Helga C. Schaller, Lukas Schaupp. Thomas Vering, et al. Non- linear model predictive control of glucose concentration in subjects with type diabetes. Physiological Measurement, 25(4):905, 2004.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18. 2022.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference Machine Learning, pages 267-274, 2002.
Dylan Kalisvaart, Jorge Bonekamp. and Sergio Grammatico. Bi-hormonal lincar varying model predictive control for blood glucose regulation type diabetes patients. 2023 IEEE Conference on Control Technology and Applications, pages 552-558. IEEE, 2023.
Thommen George Karimpanal, Santu Rana, Sunil Gupta, Truyen Tran, and Svetha Venkatesh. Learning trans- ferable domain priors for safe exploration in reinforcement learning. In 2020 International Joint Conference on Neural Networks, pages 1-10. IEEE, 2020.
Niklas Kochdumper, Hanna Krasowski, Xiao Wang, Stanley Bak. and Matthias Althoff. Provably safe rein- forcement learning via action projection using reachability analysis and polynomial zomotopes. IEEE Open Journul of Control Syxtems, 2:79-92, 2023.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24 (11):1716-172) 2018.
Juliane Liepe, Paul Kirk, Sarah Filippi, Tina Toni, Chris P Barnes, and Michael PHI Stumpf. A framework for parameter estimation and model selection from experimental data systems biology using approximate bayesian computation. Nature Protocols, 9(2):439 456, 2014.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained varia- tional policy optimization for safe reinforcement learning. In International Conference on Machine Learn ing, pages 136444-13663 PMLR, 2022.
12

---

Volodymyr Mnih, Koray Kavukcuogiu, David Silver, Alex Graves, Ioannis Antonoglou. Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-2777 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emestus, and Noah Dormann. : Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8 8, 2021.
Meghana Rathi, Pietro Ferraro, and Giovanni Russo. Driving reinforcement le with models. Intelligent Systems and Applications: Proceedings of the 2020 Intelligent Syxtems Conference Volume 1, pages 70-85. Springer, 2021.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv: 1910.19900007088 7(1):2, 2019.
Benjamin Recht. tour of reinforcement learning: The view from continuous control. Annual Review of
Control, Robotics, and Autonomous Systems, 1-53-279, 2019.
Jennifer L Sherr, Melissa Schoelwer, Tiago Jeronimo Dos Santos, Leenatha Reddy, Torben Biester, Alfonso Galderisi. Jacobus Cornelius van Dyk. Marisa E Hilliard. Cari Berget, and Linda A DiMeglio, Ispad clinical practice consensus guidelines 2022: diabetes technologies: insulin delivery. Pediatric Diabetes, 23(8): 1406-1431, 2022.
David Silver, Guy Lever. Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395 Pmir. 2014.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwwiseer, loannis Antonoglou, Veda Pannecrshelvam, Marc Lanctot, al. Mastering the game go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Tom Silver, Kelsey Allen, Josh Tenenbaum, Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv: 1812.0628,, 2018.
Matthew Taylor Peter Stone. Transfer learning reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John Balis, Gianluca Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https: //zenodo brg/record//1170000
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.
Xiong Yang and Yingjiang Zhou. Optimal tracking contoll of continuous stirred tank reactor systems: A dynamic event-driven approach. IEEE Transactions on Artificial Intelligence, 2023.
Dongjic Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning, pages 25636-25655 PMLR, 2022a.
Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with safety editor policy. Advances in Neural Information Processing Systems, 35:2608-2622 2022b.
Rasoul Zahedifar and Ali Keymasi Khalaji. Control of blood glucose induced by meals for type-1 diabetics using an adaptive backstepping algorithm. Scientific Reports, 12(1):12228, 2022.
Mario Zanon. Vyacheslav Kungurtsev, and Sébastien Gros. Reinforcement learning based on real-time iteration ampe. IFAC-PPpersOOnLine, (3(2):5213-5211 2020.
13

---

Appendix
Table of Content
Theoretical analysis
14
B Environments for validations
17
C Implementation details
D Additional experiment results
22
A Theoretical analysis
A.1 Policy combination as regularization
Lemma (Policy Regalarization) any state S, for a multivariate Gaussian RL policy Ta R the expecta- tion of the combined action as(s) derived from Eq. (7) is the solution to the following regularized optimization with regularization parameter À = B ()(s) / B())):
with mean (s(s) and covariance matrix Σ = diag(o) (s),
B(s)
E [aas(s)] argmin ||a ml(s))|> 1 B(s) ||a- (s)g(s)
(13)
Proof. For state E S, the focus module outputs fixed / = 3(s). For the fixed 8, the proof of Lemma is similar the proof by Cheng et al. [2019b]. Since the RL policy is Gaussian distributed policy (s), Σ) with the mean action (s) and an exploration noise with covariance diag(o?, o2....) the combined action a also follows Gaussian distribution:
(s(s) ~ N (Bareg (s) + +1 (1- (s), (1 -3)=).
(14)
Let f(u, Σ) be the probability density function (PDF) of (M, Σ). The product of two multivariate Gaussian PDFs is proportional to another multivariate Gaussian PDF with the following mean and covariance:
f (Mi, f (12, E2) of ((2T Eー') #n + E2 p2), (2) +
(15)
The mean of (s(s), B0reg (s) + (1 (n), be expressed in the following form:
Barar (s) B) (s) = -BI- (Erexx)) + (1 13) (En())
((²z)
-1
areg (s) + 11 (1 -1
(16)
TT((s)
The covariance matrix E be expanded into the following form:
1 (1-10)
=
(17)
Using Eq. (15), the PDF of as)s) can be expressed as the multiplication of two PDFs, as shown below:
f(Boreg (s) + (1 (s), (1 -1- 3) = cof (orec(s) f (n((s), 1 --) (18)
With the definition || = -----= the PDF of as(s) can be written as:
B lla (=)s))
f(ao(s)) =cy0 exp
- lla Areg(s))) exp
2
(19)
1-B
8
=cexp
(lla [n(())|E
1- lla arog (8)]lx)
2
14

---

where the constant c is as follows:
(2m) (2m)
(20)
Since 5 is in the range (0, f(as(s)) monotonically decreases as ||a - # + I-8 la
dreg (5) I increases. Therefore, the probability of (s) is maximized when the term is minimized,
leading to the following optimization problem:
[(s)
E [{as())] = argmin ||a Fn(())l + 1 -+((s) ||a - (s)g())]l>=
(21)
Assuming the RL policy converges to argmax, (()), Eq. (21) can be written as:
3(x) ||a arog (8)lls-
E [aa(s)) = argmin
E argmax ((8,
+ 1- B(s)
(22)
A.2 Deviation of combined policy from safety regularizer
Theorem 1. Assume the reward R and the transition probability P of the MDP M are Lipshitz con- tinuous over with Lipschitz constants La and Lp. For any state s E S, the difference in expected return between following the combined policy Tg and following the safety regularizer policy Teg. i.e. [V* (s) V Teok (s)l, () has the upper-bound:
[V* - ( Sror (x)) (1 (1-)]|5)L LR + Lp (1 ((s)
(23)
(1 71-
where |S] is the cardinality S, S, and Aa (an(s))n)))))))))))))){))) - Oreg( (8) is the bounded action difference at 8.
Proof. Let us define value function vectors ve and Vreg in R| for which the s-th entries [v = 4 (s) and [Vreg], = V (s). Also, let us by rs and Freg denote reward vectors in R| with the s-th entries of the reward vectors being [rel, = r(s, az(s)) and [Freg], = rss, areg (s)). We define state-transition matrices P3 and P reg in IS x|8), with (s, s')-h entries as [P = P(s, as(s)) and (Pregle.e' = ((s, Oreg(s)). According to the vectorized Bellman equation, the difference between V, and Vrrg satisfies the following relationship:
Vreg
rg + P. Freg - P reg.
Pree Voo
rg Freg + pv. PPP + P.oVee - PPrgg Vieg
(24)
rg Frog + pV - Vrog) +7(P, - P Prgg) Frog
(I PP - (rg Freg + 7(P - 1Pp-Pres -Pegg) Vreg)
Let dh be the s-th row of (I PP and d be the s-th row of (I - (Ieg Since (I -P3) can be expanded as Neumann series yPs + 7² P3 ..., the upper-bound for the elements dg,, and drog,s is 1/(1 - 7), i.e., ||d and ||dreg ooo are less than or equal to 1/(1 >) From Eq. (24), the value functions (**(s) and Vamas (s) for specific state E S satisfies:
(s) VTTo < Id (ra Treg)l + +y d reg. (P&
P Prog/ Prog)/regl Vregle
(25)
(a)
First, we consider part (a) of Eq. (25). Lipschitz constant LR, have:
Assuming R(s, is Lipschitz continuous over A with
||r Treglli 11
S)||R(s, (s(()) R(s, Greg (xt))
(Lipschitz)
[S]LR(1 - 3(s))(an() areg (8)
(26)
s [l] ()))))(())))))))))))))))))) Ao
15

---

Using the Holder's inequality, part (a) of Eq. (25) has the following upper- upper-bound: bound:
| L
d (T& Freg)) S dds.llooll 1| - Freglli 1 S 1-7 (1 3(s))
(27)
Second, we consider part (b) of Eq. (25), For each state E S, define the s-th rows of Ps and Preg as P/.s and Pieg.s The vectors P3,s and Preg,s each represents a discrete probability distribution. Because we assume P(s, a) Lipschitz continuous over A with Lipschitz constant Lp, the upper- bound on each item in vector (Pa Prog rog Vrog is derived below:
KPs,. Preg,s)
" Vrogl I
|ps, Preg,s //vellllll
Rmax
P(s, as()))
P(s, Greg (8))]l
L Rmmax (1 ((8))](a((s) areg (s)
(28)
(Tipschitz)
7
< Lp Rmax (1 B(s))
Therefore, part (b) of Eq. (25) has the following upper-bound:
y/d (P reg Vregl S 1lo - -Pree)Vreglll S 2/5|Lp (1 B(s)) (29)
(1 7)²
The proof is complete by substituting Eq. (27) and Eq. (29), respectively, into parts (a) and (b) of Eq. (25).
A.3 Performance improvement of focus module update
Theorem 2. (Focus Module Performance Improvement) The focus weight (s() updated by Eq. (10) satisfies t (8) 2 V*0 (s), Vs E S, i.e., the expected return monotonically improves.
Proof. According to the performance difference lemma in [Kakade and Langford, 2002], all states E S, the expected return difference between the two policies satisfies:
1 E
(s) (8)
(AT T')), VT,
(30)
where (s, -) (T) is the advantage function, d=' is the normalized dis- counted occupancy induced by policy x'' from the starting state s. The update in Eq. (10) results in (s, ) 2 1 (s. R)), Vs E S. Thus (s(s) satisfies:
A (s, ) = T (s, ) ( TT) 20. 0.
(31)
Using the above in Eq. (30), and combined with the fact that the discount factor is in range (0, 1), we can rewrite Eq. (30) as V (s) 2 V (s), Vs S.
A.4 Convergence of combination weight
Lemma 2. (Combination Weight Convergence) For any state s, assume the RL policy ni converges the optimum policy =² that satisfies ((s,T*) ((s, VT # n², then (3((s) 0 will be the solution to Eg. (10) that achieves the optimal policy combination.
Proof. Since a sub-optimal model used derive Trog Treg * n² Let Let (s) ~ T (s)) denote the optimum action at state If ((s) o 0, then (())ureg (8) + (1 {s))an((s) = B(s)areg (s) +
6(s))a((e (s) * a" (s). Therefore, the solution to Eq. (10), i.e., the updated focus weight 8' (s)) can only be 0.
16

---

dt min A.5 Convergence to the RL policy
Theorem 3. (Policy Combination Bias) For any state s, the distance between the combined action a (s) and the optimal action a" (s) has the following lower-bound:
lao(s) a" (s) 2 [s) - a" (s) (1-((s) azeg (s) ar((s)).
(32)
Ifa a Gaussian RI. policy 3nt converges the optimum policy (s) with Q(s, x²) ) (8, VT +
n, then the combined policy Ts(s) can have unbiased convergence to the optimum Gaussian policy
π² with total variance distance DTv(Tal) π- (s)) O.
Proof. First, we prove the lower-bound for (s)(8) a² (s)). The proof of lower-bound is similar to [Cheng et al., 2019b]. We expand the distance between the safety regularizer action areg and the combined action ag follows:
[aveg (s) as(s) | = [aro (8) ((s))rou(s) (1- (()))nn(s))
(33)
(1 3(s)) a,((s),
Following triangle inequality, the distance between the combined action and the optimum action is as follows:
la aa)s a 2 areg (s) - a" (s) - |areg as(s)] arog (8) - a (s) - (1 - B()) arag (8) an)())
(34)
To prove the convergence of Ty, first examine the convergence of ((s). According Lemma 2, ((s) converges to under the assumption that the RL policy тH converges to π r*(s) with ((s, x²) > (s,, T), Vn * x*. Because (ss) 0, the combined policy has variance equal to ma. According Lemma 1, the following holds for any state 8:
B(s)
E[as(s)] = argmin |a argmax Q an())ll 1 1 B(s) |la (recg()))ll
e,(s))
(35)
argmin la- argmax Q an (s))lxs = (s).
a
ea(())
Thus the the converged combined policy xp(s) na(s) (s) with total variance distance DTv(Ta(s), x (s)) = O.
Environments for validations
In this section, introduce the four environments (Glucose, BiGlucose, CSTR, and Cart Pole) used in Section 4, with detailed environment models, parameters, and reward functions. MPC minimizes the stage cost Jj. and terminal cost Jx. while RL. maximizes rewards r, for consistency, set the MPC costs to JR = JN -r when validating the MPC the environments.
B.1 Glucose
Table Glucose parameters for the estimated model and the actual environment
Parameters
Unit
Estimated Model
Actual Environment
G
mg/dL 138
138
/0/mL
7
n
min
0.2814
0.2
Pi
min
min
0.0142
0.005
P3
min
15c-6
5c-6
Do
10
10
The Glucose environment simulates blood glucose level, denoted as G, against meal-induced disturbances which elevate G' and cause hyperglycemia. The observations are (G, G)t), where
17

---

G G and t is the total time passed after meal ingestion. The action is the injection of insulin o,, which gradually lowers G but with large delay, Excessive injection of insulin can cause life-threatening hypoglycemia. Safety constraints are imposed on the blood glucose level G, as both the elevated and reduced result in severe health risks (hyperglycemia and hypoglycemia, respectively). The blood glucose model contains state variables regulated by the ordinary differ- ential equations (ODEs) given below:
G PPi Gb) GX + Dr
X P2 Ps It)
(36)
i n(I I,) aj.
among which only G can be observed. Because not all states are observed, the closed loop is maintained only for the observed G for MPC and RL-AR. The term D, represents the time-varying disturbance caused by the meal disturbance:
D. Do exp(-0.011).
(37)
The model parameters adopted by the estimated model and the actual environment are given by Table The initial states are determined by setting the left-hand side Eq. (36) to zeros and solving the steady-state equations. The reward function for Glucose is the Magni risk function [Fox al., 2020], which gives stronger penalties for low blood glucose levels prevent hypoglycemia:
(3.35506 x ((In G) 3.7932)) 10 < G < 1000
r
(38)
le5,
otherwise
B.2 BiGlucose
Table 3: BiGlucose parameters for the estimated model and the actual environment
Parameter
Unit
Estimated Model
Actual Environment
DG
kg
0.08
0.08
Va
L/kg
0.14
0.18
min
0.0968
0.0343
Foi
mmol/(kg min)
0.0199
0.0121
EGP.
mmol/(kg min)
0.0213
0.0148
Ag
0.8
0.8
tmaa, G
min
40
40
tmea.
55
V1
L kg
0.12
0.12
ke
min
0.138
0.138
kad
min
0.0088
0.0031
kua
min
0.0302
0.0752
kaa
min -1
0.0118
0.0472
L/(min? mU)
7.58e-5
9.1 1c-6
kaa
L/(mnn mU)
1.42e-5
6.77c-6
kas
L/min mU)
8.5e-4
1.89e-3
Iman,N
min
20.59
32.46
EN
min
0.735
0.620
Vs
mL kg
23,46
16.06
p
min
0.074
0.016
SN 10
mL/pg min
1.98
1.96
M,
g/mol
180.16
180.16
BW
kg
68.5
68.5
No
pg/mL
48.13
48.13
dt
min
10
10
The BiGlucose environment simulates blood glucose level G against meal-induced disturbances. which clevate G and cause hyperglycemia. The observations are (G, G,t), where G G. G
18

---

and t is the total time passed after meal ingestion. The actions are insulin and glucagon injections (ay, aN). Insulin injection a, lowers but causes hypoglycemia when overdosed. Glucagon jection ox elevates G and thus can be used to mitigate the hypoglycemia caused by as- Similar to Glucose, safety constraints are imposed on G. The blood glucose model contains 12 internal states (11 of them unobservable) and actions with large delays, regulated by the ODEs given below:
Q = -i 21 + 42 - Fn
+ (1 x3) EGPo Cconv YQ1
Q2 = I1 Q1 (k12 + 2/2 T2)
#1 ka1 + Kbll
b = ka2 + kazl
43 Kaz + Kosl
S1
$1 = 1max.1
Si
S2
$2
tmaz. S mmaz
(39)
S2
i
I
Vitmaxx,1
Z1
71 aN Imax.N
Z1
Ze
i2 Imaz,N N
imaax N
Zz
Ñ = -NN(N - No) +
VxlmannNN
Y -py + +PSN(N No),
where the intermediate variables Fúi and FR are piecewise functions of the measurable blood glu- mass 18 Q1 Va. shown below:
FFor
G 81mg/dL
F6i
(40)
Fo1G/81,
otherwise
[0.003 - V).
G 2 152mg/dL
FR
(41)
0.
otherwise
Since only G can be observed among the 12 states, the closed loop is maintained only for MPC and RL-AR. The term UG represents the time-varying disturbance caused by the meal disturbance:
DcAG . -t/tmoo
UG
(42)
22 t
The model parameters adopted by the estimated model and the actual environment are given by Table This extended model proposed by Herrero et al. [2013] and Kalisvaart et al. [2023] captures more complicated blood glucose dynamies, allowing the of both insulin injection and glucagon injection as actions. This leads to better regulation performance, but also drastically increases the complexity of the problem due to its large number of unobservable states, delayed action responses, and nondifferentiable piecewise dynamics [Kalisvaart et al., 2023].
The initial states are determined by setting the left-hand side of Eq. (39) to zeros and solving the steady-state equations. The reward function for BiGlucose is the Magni risk function [Fox et al., 2020], which gives stronger penalties for low blood glucose levels to prevent hypoglycemia:
10 x (3.35506 x ((In U..533 - -3.7932)
10 < G 1000
(43)
1e5,
otherwise
19

---

Table 4: CSTR actual environment model parameters
Parameter
Unit
Actual Environment
Parameter
Unit
Actual Environment
kb,ab
h-
1.287e12
kg/L kl/kg.K
0,9342
bo
h-
1.287e12
Gp
3.01
Ro,ad
L/molh
9.043e9
Cp.k
kJ/kg.K m²
2.0
Rgan
kJ/molK
8.3144621e-3
AR
0,215
EA,ab
kJ/mol
9758.3
VR
L
10.01
EA
kJ/mol
9758.3
kg
5.0
Ex,ad
kJ/mol
8560.0
Tin
"C
130.0
HR
kJ/mol
4.2
Kw
kJ/h m² K
4032.0
HR
kJ/mol
-11.0
CA.0
mol/I.
5.1
H HR
kJ/mol
-41.85
dt h
0.05
Table 5: CSTR different parameters for the estimated model and the actual environment
Parameter
Estimated model
Actual Environment
1.05
3
1.1
B.3 CSTR
The CSTR environment simulates the concentration of target chemicals in a continuous stirred tank reactor. The observations are (Ca, CB. Tn, Tk), where CA and CB are the concentrations of two chemicals, TR is the temperature of the reactor, and Tk is the temperatures of the cooling jacket. The actions are the feed and the heat flow (ar, aq). Safety constraints are imposed on chemical concentrations and reactor temperature, crossing the safe boundaries for any of them can lead to tank failure or even explosions. This model contains four state variables regulated by the ODEs given below:
ĆA = (C -CA) [0J-KiCA -
ČB -aFCB + KICA- K2CB
Tk KiCAHRoo + KgCDHRRDe KgC2 HR K ART - -RR
aT -R) (44)
pCip
pCpVR
Tx uQ + AR Tx)
m& C&Cp.k where the intermediate variables are:
EA,ab
K1 = BBkk, ub exp(
Tr + 273.15
FAA.bC
K₂ ko exp(
(45)
TR 273.15
K3 Ro exp(
Tn + 273.15
The model parameters adopted by the estimated model and the actual environment are given by Table 4 and Table 5. The initial concentration of the target chemical CB,p is set to 0.5. The reward function for CSTR is follows:
(100 x (CB 0.6)
0.15CA52, 0.15Cp52, 505TA5200, 505
(46)
(100 x (Cg 0.6)² 1e4, otherwise
B.4
Cart Pole
The Cart Pole environment simulates an inverted pole on cart. The environment the continu- action adaptation of the gymnasium environment [Towers et al., 2023]. The observations
20

---

Table 6: Cart Pole parameters for the estimated model and the actual environment
PARAMETER
UNIT
ESTIMATED MODEL
ACTUAL ENVIRONMENT
m -
9.8
9.8
kg
1.0
0.8
kg
0.1
0.3
I
0.5
0.6
dt
0.02
0.02
Table 7: RL.-AR hyperparameters. The baseline methods utilized the same network structures and training hyperparameters.
Parameter
Value
Learning rate for Q network
1 10
Learning rate for policy network
10
Batch size [B] for updating
256
Start learning
256
Target network update factor
0.005
Forgetting factor
0.99
Frequency for updating policy network
Frequency for updating target network
1
Learning rate for the focus module focus module pretraining threshold 1 - Minimum log policy variance Maximum log policy variance
10-6
0.999
-5
2
Policy network hidden layers Q Network hidden layers
[256, 256]
[26,, 256
focus module hidden layers
[128, 32]
Glucose MPC horizon
100
Other envs MPC horizon
20
(1r, i., 0, 0), where is the position of the cart and the angle of the pole. The action the hori- zontal force ay- Safety constraints are imposed and 0 as the control fails if the cart reaches the end of its rail or the pole falls over. This model contains four state variables regulated by the ODEs given below:
g sin do A
Ø
1(4/3 mp cos? 0/(mp + mc))
(47)
# d mp10 cos
mp + m.
The intermediate variable d is:
d x a + m mp sin
(48)
mp
The model parameters adopted by the estimated model and the actual environment are given by Table 6. The initial tilt of the pole is 6 degrees. The reward function for Cart Pole is as follows:
10000 max (0, |x| 0.25),
-244. -22//390501122///0
(49)
10000 max (0, |a| 0.25) le4, otherwise
Implementation details
Experiments are conducted using Python 3.12.5 Ubuntu 22.04 machine with 13th Gen Intel Core i7- 1-58HX CPU, Nvidia RTX 3500 Ada GPU, and 32GB RAM. For RL-AR, the average time for taking step (interaction and network updates) is 0.0235 for Glucose, 0.0667 for BiGlucose,
21

---

0.0378 for CSTR, and 0.0206 for Cart Pole. The above decision times are practical for real-tine time control these environments.
The MPC and the safety regularizer in RL-AR are implemented using [Fiedler et al., 2023]- The implementations for SAC and the RL agent in RL-AR based [Huang et al., 2022]. RPL., CPO, and SEditor follow the implementations in [Silver et al., 2014], [Ray et al., 2019], and [Yu al., 2022b], respectively. The same hyperparameters in Section 4, which are listed in Table 7. The baseline methods' network structure and training parameters are set be the same as RL-AR.
used by RL-AR for all experiments
Additional experiment results
D.1 State-dependent vs. scalar policy combination
By(s) scalar B
policy combination weight
nomallzed
04
0.6
0.8
250
10000
20200
cpisode
step
Figure 5: Comparing the detendee focus module (s)s) with the scalar 3 by plotting the normalized return curves (left) and focus weight curves (right) in the Glucose environment. Shaded areas indicate standard deviations.
As discussed in Section and Theorem 2, using state-dependent focus module ((s) for policy combination (compared to using a scalar weight 3) offers the advantage of achieving monotonic performance improvements at least in the tabular setting. Here, we empirically verify the advantage of using the state-dependent (s) versus a scalar weight in Fig. by analyzing the normalized returns (left panel) and the focus weights (right panel) during training in the Glucose environment (Fig. 5). The mean (solid lines) and standard deviation (shaded area) in Fig. 5 are obtained from independent runs using different random seeds.
Practically, RL-AR with dete-dependent (s) does not show a strictly monotonic policy improve- ment, which can be attributed to the neural network approximation. However, improvements in the normalized return are significantly more steady when using (s) rather than a scalar 3, as shown by the blue and red curves in the left panel of Fig. 5. The right panel of Fig. 5 shows the evolution of the focus weights (used by RL-AR) versus the training steps. Although both (x) and / con- verge to zero after approximately the same number of steps, (s)(s) applies different focus weights depending on specific states encountered as seen by the fluctuations in the blue curve in the right panel of Fig. 5.
D.2 Entropy Regularization
We conduct an ablation study to compare using SAC as the RL agent in RL-AR with using an- other sfate-of-the- RL algorithm, TD3 [Fujimoto et al., 2018]. The key difference between SAC and TD3 is that SAC incorporates the entropy regularization term, a log P.,. (a]s). ). in Eq. (3) and Eq. (5). The normalized return curves, shown in Fig. 6, demonstrate that RL-AR with SAC as the RL agent achieves higher normalized return at faster rate. While RI-AR promotes safety and stabil- ity at the cost of reducing the exploration intensity the combined policy (as proved in Lemma I), which could potentially slow down the discovery of the optimal policy, SAC's entropy regularization
22

---

-10"
0.2
normalizee senaaizzed return
0.3
0.4
SAC the RL module
TD3 as the RL module
episode
Figure 6: Comparison of normalized return between using the SAC and using TD3 [Fujimoto et al., 2018] as the RL agent the Glucose environment (standard deviations shown in the shaded area). The main difference between SAC and TD3 that SAC has the entropy regularization terms in its objectives, which are intended encourage diverse policies and stabilize training.
counteracts this by promoting the use of more diverse policies. closer examination reveals that the SAC curve locally exhibits more minor fluctuations than the TD)3 curve, illustrating SAC's ability to use more diverse policies.
D.3 focus weight curves during training
policy combination weight B (s))
0:95
0.85
n 3/16 Å, P2 = 1/4 P2
3/16 Å, P2 1/2 P2
0.75
3/16 Å, P2 = P2
MPC fails from the beginning
n 3/16 Å, p2 p2=2 P2
0.30
0.62
2000
4000
6000
8000
10000
step
Figure 7: The focus weights when training with varying levels of discrepancies between the esti- mated Glucose model (with parameters f2, ñ) and the actual Glucose environment (with parameters P2, n1)
In Fig. 7, show several (st) curves from training RL-AR in various Glucose environments, created by varying the environment model parameters n and p2. Let ñ and 2. be the parameters of the estimated model f. the actual environment models have = 3/ /16 and P2 = P2 P2/ 1p2, 2p mimic deviating characteristics of new patients. When there are large discrepancies between the environment and f (e.g.
3n/ 16 and P2 42)4), the safety regularizer fails initially, but the focus weight Sy (s:) decreases rapidly. enabling RL-AR to recover from initial failures by rapidly shifting from the sub-optimal safety regularizer policy to the stronger learned RL policy. Conversely, (st) converges more slowly to zero when the safety regularizer performs well in the actual en- vironment, as it becomes more challenging to find RI. policy that significantly outperforms the safety regularizer policy in such cases.
23

---

normalized return
number of failures
-155
20
RL-AR
MPC SAC
10
(pretrained)
2.5
1000
2000
300
400
500
.00
7000
100
100
200
320
4103
100
500
700
800
episode
episode
Figure 8: Normalized return (left) and the number of failures (right) during training in the Acrobot environment (standard deviations are shown in the shaded area).
D.4 The Acrobot environment
In Section 4, we evaluate RL-AR in several widely recognized challenging safety-critical environ- ments. For example, the BiGlucose environment (see Appendix B.2) involves 11 unobservable states, two actions with significant delays, and complex, nondifferentiable piecewise dynamics. Here, we provide further evidence of the effectiveness of RL-AR the Acrobot environment, an adaptation of the Gymnasium environment [TTwerr et al., 2023] with a continuous action space (Fig. 8). The Acrobot environment simulates two links connected by a joint, with one end of the connected links fixed. The links start f'cingg downward. The objective is swing the free end above a given target height as quickly as possible by applying torque to the joint. Failure is defined as the tip not reaching the target height in 400 time steps. For validation, we set the actual Acrobot environment parameters
1.1/g, m2 m2, where l2 1.0 is the lower link length parameter of the estimated model f. and #t 1.0 is the lower link weight parameter of the estimated model f.
The Acrobot environment is particularly challenging for RL-AR's policy regularizer due to: i) its highly nonlinear, under-actuated dynamics, and ii) its definition of failure as not achieving the target within given time limit, which cannot be easily formalized as a constraint. AR can swing up the tip in the first episode by initially relying the viable safety regularizer policy. Throughout training, RL-AR ensures safety while converging to a similar normalized return as SAC, which focuses only return and fails many times during training. This illustrates RL-AR's robustness in challenging tasks and its potential applications where failures are defined in terms of time limit and are hard to formalize as constraints.
Fig. 8 shows, RL-
E Impact statement
This paper presents work that aims advance the field of Machine Learning. There are many poten- tial societal consequences of our work, none of which, based on our judgment, must be specifically highlighted here.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims described in the abstract and introduction are: 1) safe during training and converging the performance standard of model-free RI., given reasonable model environment discrepancies.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section identifies the requirement of reasonable model environment dis- crepancies, which are further quantified using experiments in Fig. 4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[Yes]
Justification: The proofs for theoretical results (Lemma 1, Theorem 1, Theorem 2, Lemma 2, and Theorem 3) are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 3, Section 4, and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code available httip:://gichubb.com//oo/zz//
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips https/ccccpullc/giieeeeeeeee / commiissiiio onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: Described in Section 4, Appendix C, and the code at https:///ithubbb. com/ HaozheTian/RL.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Mean and std. included for the main results (Table and Fig. 2), Fig. 5, Fig. 6, and Fig. 8.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httopp:///neue http://eeerpp/eeeeeeeeeee htcppublic/Ethicsuuuueuuu
Answer: [Yes]
Justification:
The paper conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Assets properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi teitchhodee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
Provided at http://hwh..cccloccccooaaaaaaaaa
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "WoL8SqvGYJ", "reviewer": "Reviewer_ChN3", "summary": "The paper proposed RL-ACR algorithm to solve safe RL exploration problem where the RL policy must be safe during and after training. It achieves this by learning a \"focus network\" which mixes the action from MPC-based safety regularizer and conventional RL module. The algorithm is claimed to address \"single-life\" safe RL applications where safety is critical even during training.", "questions": "1. (Related to Weakness 3) Do you think safe editor policy in [1] can be a good a baseline for comparison?\n2. (Related to Weakness 3) Do you have results which evaluate your approach in longer horizon (and higher state-action dimension) setting where it is more challenging to predict future trajectory reliably?\n3. In your experiments, how do SAC-pt and CPO-pt make use of the model $\\tilde{f}$? To my understanding, they are both model-free algorithms and do not exploit any dynamic model.", "limitations": "1. The paper discusses one limitation which is having reasonably accurate model for control regularizer. It is pointed out in [4] that even for a model which can predict the next step reliably, it is very difficult to predict future N steps accurately since the error cascade over the prediction timesteps. The paper could possibly discuss this in detail.\n2. Another point about learning the model is that it should be able to predict transition to unsafe states in order to support the safety regularizer. This inevitably means that there needs to be unsafe transitions being collected to train the model. How does this model collection fit into the entire safe exploration framework is an interesting area to discuss.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Quality\n* The approach itself is sound and the paper provided proof that the combined policy (which mixes actions from two modules) is unbiased if the RL module converges to optimal policy.\n2. Clarity\n* The paper is written in easy-to-understand manner and clearly illustrates how different components work together. Figures, equations and algorithm are all well-designed and well-written to support the entire paper.", "weaknesses": "1. Originality\n* Most of the components proposed in RL-ACR have been explored previously: (i) combining two-prong policy [1], (ii) MPC with learned model [2, 3], (iii) safe RL with model rollout [4, 5, 6].\n* The proposed focus network learns to outputs a scalar which is a mixing coefficient between the two policies. The focus network is similar to the learnable Safe Editor component in [1]. However, Safe Editor seems more sophisticated as it learns to edit the action itself. I wonder if there could be comparison between the two for evaluation.\n2. Significance\n* RL-ACR relies heavily on the the estimated model $\\tilde{f}$ for safety. Consequently, the reliability (and accuracy) of the model is an important factor to guarantee safety. As pointed out in [4], even for a highly accurate model, it becomes increasingly challenging to predict future trajectory with reasonably good accuracy when the model unrolling horizon becomes larger because the error cascades over many timesteps. In Eq7 of the paper, RL-ACR is required to perform N step model unrolling. I have some doubts on how to obtain a reliable model to support RL-ACR.\n* Related to previous point, in the conducted experiments, the model $\\tilde{f}$ used in the paper is synthetically generated. It is not clear whether a model, which can predict lengthy future trajectory reliably, can be trained from ground-up. The tasks experimented (Glucouse, BiGlucouse, CSTR, Cart Pole) also do not seem to have long horizon or high dimension. Thus, I have reservation whether RL-ACR can be used to reliably solve safe exploration problems in more sophisticated tasks.\n3. Baselines used in Experiments\n* The safe RL baselines used in experiments: SAC & CPO are neither model-based nor intended for safe exploration. SAC only learns to maximize reward and disregards safety. It is also unclear how SAC makes use of the model in SAC-pt. For CPO, although it is a safe RL algorithm, it is not intended to solve safe exploration problem. It is also not clear to me how CPO-pt exploits the model $\\tilde{f}$ for this purpose.\n* Perhaps one possible baseline could be the safe editor in [1], which also claims to solve safe exploration problem?\n\nReferences  \n[1] Yu, H., Xu, W. and Zhang, H., 2022. Towards safe reinforcement learning with a safety editor policy. Advances in Neural Information Processing Systems, 35, pp.2608-2621.  \n[2] Nagabandi, A., Kahn, G., Fearing, R.S. and Levine, S., 2018, May. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 7559-7566). IEEE.  \n[3] Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems. 2018.  \n[4] Janner, M., Fu, J., Zhang, M. and Levine, S., 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.  \n[5] Clavera, I., Fu, Y. and Abbeel, P., Model-Augmented Actor-Critic: Backpropagating through Paths. In International Conference on Learning Representations.  \n[6] Thomas, G., Luo, Y. and Ma, T., 2021. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34, pp.13859-13869.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**: The reviewer stated that some components of the proposed method have been previously explored. \n\n**Response**: While our work contains components of the suggested references, the resulting algorithm has a completely new approach to safety-critical control; in general, we believe that the combination of previously explored components does not take away from the originality of our work. Nevertheless, there are several key differences between our work and the mentioned references: i) Ref. [1] combines two learned policies, but RL-ACR regulates one learned policy with an established safe policy to ensure safety from the beginning of training. ii) Refs. [2, 3] use the MPC mainly to improve the “sample efficiency” rather than achieving “safe control” which is our aim; iii) In [4, 5, 6] the rollouts based on learned models only avoid unsafe actions after convergence, unlike RL-ACR that is designed to avoid safety violations from the beginning of training. \n\n**[W2]**: The reviewer raised that generating an estimated model for the safe MPC module is challenging. The reviewer was also concerned about whether the problems used in our experiments were complex enough. \n\n**Response**: For an environment with critical safety requirements, an estimated model (e.g. derived from limited observed data points) is sufficient for the safe MPC module of RL-ACR to perform very well. (One can think of the estimated model as a fit to the observed data points). This is shown in our Glucose and BiGlucose environments, where the model parameters are derived using real data from ~10 patients [7, 8]; this is practical in most real applications with limited data.  \n\nWe chose the environments to showcase the effectiveness of the method in “safety-critical” applications rather than focusing on environment complexity. Nevertheless, the BiGlucose model is generally regarded as a hard regulation problem, with 12 internal states (11 of them unobservable), 2 actions with large delays, and nondifferentiable piecewise dynamics. We have also provided further evidence of the effectiveness of RL-ACR in Acrobot, which is highly nonlinear and underactuated (see Figure R.II in the global rebuttal PDF). \n\n**[W3]**: The reviewer raised questions on the rationale for the selection of baselines SAC and CPO, and how SAC-pt and CPO-pt make use of the estimated model. \n\n**Response**: While it is true that SAC and CPO do not explicitly use models, the rationale behind our choice is threefold: i) SAC was chosen to show the optimal performance without considering safety. RL-ACR achieving stronger performance than SAC in Fig. 3 indicates that RL-ACR is not trading performance for safety. ii) CPO is a commonly chosen safe RL baseline that explicitly handles safety constraints during exploration (through CMDP formulation and recovery updates). iii) SAC-pt and CPO-pt are pretrained on the estimated model $\\tilde{f}$ (which is used as a simulator of the actual environment); note that this is elaborated in the manuscript, see Line 217. This setting benefits SAC-pt and CPO-pt by allowing them to exploit the information in the available estimated model. \n\n**[Question 1]**: The reviewer asked if SEditor [1] can be used as a baseline. \n\n**Answer**: We thank the reviewer for pointing out this more up-to-date baseline and we have included it in our comparisons (see Figure R.I in the global rebuttal PDF). Results show SEditor from [1] does not guarantee safety during training because safety violations need to be observed before learning the cost return. SEditor also performs worse during training in 3 out of the 4 environments, potentially restricted by the constraint calculated from suboptimum cost estimation. \n\n**[Q2]**: The reviewer asked for results in environments where predicting the future is more challenging. \n\n**Answer**: We tested RL-ACR in Acrobot, where predicting the future is more challenging because of underactuation and high nonlinearity. Figure R.II in the PDF attached to the global rebuttal shows that RL-ACR maintains safety as only the first action in MPC rollout is adopted. Also, as mentioned in the response to **W2**, The Glucose and BiGlucose models are generally regarded as a hard regulation problem. The results demonstrate that RL-ACR is robust even against >60% model parameter mismatch (Fig. 4). \n\n \n\n**[Q3]**: The reviewer asked how SAC-pt and CPO-pt make use of the estimated model. \n\n**Answer**: SAC and CPO are model-free, thus not safe during training. We additionally generate SAC-pt and CPO-pt. which are pretrained using the estimated model $\\tilde{f}$ as the simulator. This makes them more competitive by skipping the initial random trial-and-error. This also equips CPO-pt with safety knowledge through the pretrained cost value function, thus delivering more competitive safety performance upon deployment. \n\n$~$ \n\n[7] Zahedifar, Rasoul, and Ali Keymasi Khalaji. \"Control of blood glucose induced by meals for type-1 diabetics using an adaptive backstepping algorithm.\" Scientific Reports 12.1 (2022): 12228. \n\n[8] Hovorka, Roman, et al. \"Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes.\" Physiological measurement 25.4 (2004): 905.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Emuksqq98", "reviewer": "Reviewer_ta4e", "summary": "In this article, the authors address the problem of safe reinforcement learning in single life setting. The agent must learn an optimal policy in one single episode without being unsafe throughout the learning. The proposed method relies on mixing a prior safe policy (the safety regularizer) with a reinforcement learning policy. The mixing is done through a linear combination of the actions proposed by the safety regularizer and the reinforcement learning agent. The weight put on either action is determined by a state dependent learned neural network. \n\nThe safety regularizer is given by a model predictive control algorithm which assumes an estimated model of the environment is available. The RL agent follows the SAC algorithm.\n\nThe authors show that the beta parameter is equivalent to a regularization weight that forces the learned policy to be closed to the regularizer policy.\n\nThey evaluate their algorithm empirically on several simulated environments that have mathematical models. The proposed method is able to outperform the baselines in terms of both safety and return.\n\nI acknowledge reading the authors rebuttal.", "questions": "-\tIsn’t equation (10) equivalent to an actor loss in actor critic method? It could be useful to discuss a parallel here. \n-\tWhen updating $\\pi_\\theta$, what would be the effect of using gradient ascent on equation (10) instead?\n-\tWhy is the performance of SAC worse than ACR in terms of normalized return?", "limitations": "The method requires the existence of a safety regularizer. In my opinion it is a fairly reasonable limitations and the method can still be used in many problems.\nThe method does not provide any safety guarantee.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is fairly easy to implement and is practical (under the condition that a safety regularizer exists). The overall idea is sound. The idea of mixing policies has been explored but they propose a principled way to learn the weights to give to both the safe policy and the learning policy which I think is useful. The idea of using MPC as a safe policy is original and interesting.\n\nThe author provide lemma and theorem that help with the interpretation of the method. They show that the mixing parameter they introduce can be interpreted as a regularization weight. They also show that the policy should converge under the optimal solution as beta goes to 1.\n\nThe method clearly outperforms the baselines in terms of safety level and provides better or equivalent performance in terms of return.\n\nThe article is overall well written and easy to follow.", "weaknesses": "- It would have been useful to quantify the deviation of the current policy from the safety regularizer as a function of $\\beta$. Since the method does not provide any guarantee on the safety level, it would be useful to quantify the amount of risk being taken by not following strictly the safe policy.\n\n- In the evaluation of the method, it would have been useful to add another baseline that takes advantage of the existence of a safe policy E.g. SPIBB (https://arxiv.org/abs/1712.06924), or residual policy learning (https://arxiv.org/abs/1812.06298). This last one would be a great baseline since they have a similar policy mixing concept but they don’t use any weight beta. \nI acknowledge that the author have pre-trained SAC and CPO on the estimated model which is a good baseline as well.\n\n- The framework is targeted towards “single life” settings, yet the training is episodic (figure 2). It makes it hard to conclude that the experiment are really evaluating the adaptability of the algorithm. \n\n- Some of the results lack discussion. In particular, the authors claim that RL-ACR has less variance in the returns, it is not very obvious from figure 2 in my opinion. The figure clearly show that SAC is less stable but it is not really discussed why. Since RL-ACR is using SAC at its core, I would expect SAC without the regularizer to perform at least as well in terms of returns (with the caveat of more failures). \n\n- The results shown in figure 3 are hard to evaluate as their lack some information about how the experiment is carried. Is it the performance in a single episode? Is it the same episode used for training? \n\n- In figure 5, the authors claim “$\\beta_{\\psi}(s)$ can apply different policy combination weights depending on how well the current state is exploited”. I agree that it is state dependent but there is no explicit term that takes into account the “exploitation” level of a state. The experiment is not showing that either I believe. $\\beta$ is learned by maximizing the $Q$ function estimate. If a state has not been visited often its Q values might still be overestimated and lead to high $\\beta$.\n\nPresentation:\n\n-\tThe title and name of the algorithm is a bit misleading, I thought that the method would use techniques from the field of adaptive control https://en.wikipedia.org/wiki/Adaptive_control,  there is no actual analogy in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**: The reviewer pointed out that it is useful to quantify the deviation of the combined policy from the regularizer policy.  \n\n**Response**: We thank the reviewer for the interesting suggestion. We analytically quantified that for state $s$, the upper bound on the return deviation of the combined policy from the regularizer policy is proportional to $(1-\\beta)\\Delta a$, assuming the environment is Lipschitz continuous. The theorem and the proof sketch are given at the end of the rebuttal, and we will add the full proof to the paper.  \n\n**[W2]**: The reviewer suggested adding another safe RL baseline.  \n\n**Response**: We thank the reviewer for pointing out RPL as a baseline. RPL combines a safe policy with a learned residual policy and prioritizes the safe policy by initializing the residual policy as 0. However, the magnitude of the residual policy action can increase drastically. We have added RPL as an additional baseline, and our results show that RL-ACR is safer and more stable than RPL (see Figure R.I in the global rebuttal PDF).  \n\n**[W3]**: The reviewer was concerned that the episodic training undermines the evaluation of the algorithms for the “single life” setting.  \n\n**Response**: “Single life” describes the setting where no failure is tolerated during learning. Episodic training can be roughly regarded as introducing a disturbance every M step. Taking the Glucose problem, every episode is analogous to prescribing insulin dosages to a patient with off-balanced Glucose levels, and the single-life setting requires the algorithm not to fail in any (training or deployment) episode. Considering that RL-ACR instantly achieves the tasks, introducing further disturbances through episodic training adds difficulty to better evaluate its performance (as well as the baselines).  \n\n**[W4]**: The reviewer raised that the results in Fig. 2 require more discussion as to why SAC shows larger return variances compared to RL-ACR.  \n\n**Response**: We now add the discussion, which is covered in detail in the answer to **Q3**. \n\n**[W5]**: The reviewer pointed out that the experimental setting in Fig. 3 requires further explanation.   \n\n**Response**: Fig. 3 shows the best performance after sufficient training, with methods deployed without exploration. We aim to show In Fig. 3: 1) how methods compare at their best, 2) RL-ACR improves the initial MPC policy, and 3) RL-ACR converges to a similar or stronger performance than standard SAC that disregards safety during training. We appreciate this comment and will clarify the above in the manuscript.  \n\n**[W6]**: The reviewer questioned the effect of exploitation level on $\\beta(s)$.  \n\n**Response**: The $\\beta(s)$ is updated iteratively using gradient ascent, thus the change in regularization level depends on the number of visits for state $s$ (besides the gradient). For any state $s$, the number of visits, which is the number of gradient ascent-based updates, affects the level to which the regularization level can be changed and thus reflects the exploitation level of state $s$.  \n\n**[W7]**: The reviewer raised that the name of the algorithm might be confusing.   \n\n**Response**: In the term “adaptive control regularization”, the “adaptive” describes “control-regularization”. However, we see the reviewer’s point that readers may think of “adaptive control”. We will change the term to “adaptive regularization”, to ensure clarity.   \n\n**[Question 1]**: Discuss the similarity of Eq. (10) to the actor loss.  \n\n**Answer**: We thank the reviewer for the comment and will add the following discussion: \"Eq. (10) is similar to the actor loss in actor-critic methods. However, instead of updating the policy network, Eq. (10) updates $\\beta(s)$ to find the optimal combination between the safe policy regularizer and the learned policy.\"  \n\n**[Q2]**: What is the effect of using gradient ascent on Eq. (10) when updating $\\pi_\\theta$?  \n\n**Answer**: Compared to Eq. (10), the RL-ACR loss has an additional entropy regularization term, which encourages exploring diverse actions and avoids sticking to the sub-optimum. Empirically, we tested the effect of using Eq. (10) to update $\\pi_\\theta$ (Figure R.III in the global rebuttal PDF), which showed reduced training stability.  \n\n**[Q3]**: Explain the worse performance of SAC in Fig. 2.  \n\n**Answer**: The difference between RL-ACR and SAC’s return variance comes from the regularizer in RL-ACR and its ability to prevent failure. Failure corresponds to a penalty affecting the return. SAC fails in some episodes (see the bottom panel of Fig. 2) and gets penalized, thus showing unstable returns compared to RL-ACR. As Fig. 3 shows, RL-ACR and SAC eventually converge to a similar return after sufficient training (as the reviewer identified). We will clarify the relationship between failure and observed return in Fig. 2.  \n\n$~$\n\n**Theorem**: (Deviation from the Regularizer) Assume $R(s, a)$ and $P(s'|s, a)$ are $L_R$- and $L_P$-Lipshitz continuous. For $\\forall s \\in \\mathcal{S}$, the following holds for the regularizer policy $\\pi_r(s)$ and the combined policy $\\pi_c(s)= \\beta \\pi_r(s) + (1-\\beta(s)) \\pi_\\theta(s)$:  \n\n$|V^{\\pi_r}(s) - V^{\\pi_c}(s)| \\leq \\frac{(1-\\gamma)|\\mathcal{S}|L_R + \\gamma |\\mathcal{S}| L_P R_\\text{max}}{(1-\\gamma)^2} (1-\\beta(s)) \\Delta a$  \n\n**Proof Sketch**: Lipshitz continuous means:  \n\\begin{align}  \n            &|R(s, \\pi_r(s)) - R(s,\\pi_c(s_t))| \\leq L_R (1-\\beta(s)) \\Delta a \\\\\\\\  \n            &\\|P(\\cdot|s, \\pi_r(s)) - P(\\cdot|s, \\pi_c(s))\\|_r \\leq L_P (1-\\beta(s)) \\Delta a  \n\\end{align} \n\nAccording to the vectorized Bellman equation:  \n\n$v_r-v_c = (I - \\gamma P_r)^{-1}(r_r - r_c + \\gamma (P_r - P_c) v_c)$  \n\nLet $d_{., s}^T$ be the $s$-th row of $(I-\\gamma P_.)^{-1}$  with entries $\\leq 1/(1-\\gamma)$. For $\\forall s$:  \n\n$|V_r(s) - V_c(s)| \\leq |d_{r, s}^T (r_r - r_c)|+ \\gamma| d_{c, s}^T (P_r -P_c)v_c|$.  \n\nThe proof follows by applying Holder's inequality to the two parts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edaxXCOgbJ", "reviewer": "Reviewer_QQKo", "summary": "The paper proposed a safe RL algorithm using adaptive regularization from model predictive controller. The regularization is implemented via weighted sum of a MPC controller and model-free RL policies. The weights is adaptive and learnable through a “focus network”. The focus network is updated through optimizing the expected Q function of weighted sum of actions. Experimental results shows that the proposed RL algorithm can learn a safe policy with zero constraint violations and outperforms MPC on the performance.", "questions": "1. Can you clarify what types of references model $\\tilde f$ you assumed for the regularizer policy $\\pi_r$? See weakness 1 for the reason.\n2. Why not compared to the Cheng et, al. as you already mentioned it? Now you only has one safe RL baseline CPO (and its variants). CPO is too old. \n3. If you already has the safe model, why not just do CBF-QP on your learned policy? What is the potential advantages of the proposed algorithm \n4. Please clarify the MPC solver and computational resources used for solving MPC.\n5. For the focus network update, it seems like the focus network will finally choose $\\beta(s) = 0$, which is the optimal policy. But the optimal policy might not be safe. What guarantees the safe policy is learned as you shown in Table 1?\n\n[1] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387–3395, 2019a. \n[2] Ames A D, Coogan S, Egerstedt M, et al. Control barrier functions: Theory and applications[C]//2019 18th European control conference (ECC). IEEE, 2019: 3420-3431.", "limitations": "The authors addressed the limitation adequately.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 4, "contribution": 1, "strengths": "The idea is interesting to combine MPC as regularizers. The writing is clear and easy to understand. The experimental results looks promising under strict assumptions of known models.", "weaknesses": "1. There is a key assumption that, the MPC policy $\\pi_r$ via estimated model is safe, is too strict, not realistic and not mentioned explicitly. The authors claimed that the model is sub-optimal in line 31, which means the model is not accurate. The model mismatches usually cause constraint violations, but the experimental results in Table 1 showed no constraint violations for MPC policies. \n2. The computational cost is too high, which makes the proposed algorithm not generalizable to nonlinear/high-dimensional inputs/ unknown models. Solving a MPC is very computationally heavy. The authors didn’t mention what optimization package, how many CPUs they used for solving MPC problem (7). For nonlinear system, MPC are not guaranteed to be solved in polynomial time and converged to global optimal solutions. For system with visual inputs and unknown models, the methods are not applicable.\n3. The focus network update might lead to unsafe policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**: The reviewer stated that assuming the existence of a safe MPC policy is too strict, not realistic, and not mentioned in the manuscript. \n\n**Response**: we believe that the reviewer may have missed a few points. First, this assumption is explicitly mentioned and discussed in the paper at **L335**. Second, the existence of an estimated model for MPC to generate a safe policy is realistic and reasonable, supported by the fact that MPC is arguably the most widespread method used for constrained control applications. Furthermore, to showcase the amount of uncertainty that our approach can withstand before failing, we test the sensitivity of RL-ACR with respect to the model accuracy. This is shown in **Fig. 4**, which shows that RL-ACR maintains safety even when model parameters have > 60% deviations. \n\n**[W2]**: The reviewer stated that we have not mentioned the optimization package, number of CPUs, etc. used for our experiments and that the high computational cost of MPC is a weakness of our work. Moreover, they state that the proposed approach is not applicable to systems with unknown models and visual inputs. \n\n**Response**: In **L560**, we reported that the average decision+learning time for RL-ACR is 0.037s on a single CPU, which is reasonable for most real-time control applications. It is explicitly mentioned in  **L122** that we use do-mpc with the IPOPT solver package. The solver reduces the computational complexity by using the solution from the previous time step as the initial guess, leveraging the fact that MPC solves similar problems with slight variations at each time step. Moreover, while we agree with the reviewer that RL-ACR is not readily applicable to systems with unknown models and visual inputs, we want to stress two points: \n\n* Having an approximate model of the system at hand is the starting point of our work. This is a realistic assumption for many applications in real-world scenarios, where approximated models can be obtained by data-driven methods. \n\n* A very large class of problems does not rely on visual inputs. While RL-ACR could not be directly applied to systems that only rely on visual inputs (e.g., the ATARI suite), it can be applied to numerous other domains (e.g., medical applications, robotics, chemical processes, industrial settings).  \n\n**[W3]**: The reviewer stated that the focus network update might lead to unsafe policy. \n\n**Response**: The focus network is iteratively updated using gradient ascent, which prevents $\\beta(s)$ (the state-dependent focus weight) from a sudden decrease. This ensures regularization even if the learned RL policy is affected by overestimation error. The regularization from the safe policy is reduced to a small degree only after state $s$ is visited multiple times. \n\n**[Question 1]** The reviewer asked for clarification of reference(estimated) model used. \n\n**Answer**: The MPC reference models are widely adopted for safety-critical control applications [1]. In our work, the model parameters used to obtain the safe policy regularizer for Glucose and BiGlucose are estimated with real patient data, just as it can be practically done in real-world scenarios [2,3]. The environment models with their actual and estimated parameters are detailed in **Appendix B**. \n\n**[Q2]**: The reviewer suggested that we should have compared with an additional method (Cheng et al) and that our baseline (CPO) is too old. \n\n**Answer**: CPO is a well-established baseline used by most of the similar works in the recent literature (to give an example see the very recent paper [4]). Nevertheless, we compared our method to two additional more up-to-date baselines (Silver et al., 2018) and (Yu et al., 2022) in Figure R.I in the PDF attached with the global rebuttal. The results show that RL-ACR outperforms the new baselines, providing more evidence of the effectiveness of the proposed algorithm for safe control. \n\n**[Q3]**: The reviewer asked for the advantages of our method over CBF-QP. \n\n**Answer**: CBF-QP minimizes the control effort, without considering performance. The advantage of MPC is that it optimizes the predicted performance while maintaining safety constraints, resulting in a high performance of RL-ACR during the learning phase. Also, note that the theoretical safety guarantee of CBF-QP relies on the existence of a “perfect” model.\n\n**[Q4]**: The reviewer asked for clarification on the MPC solver and computation resources used. \n\n**Answer**: The original manuscript provides this information. MPC was solved using do-mpc, with its implementation based on CasADi and IPOPT (**L122** and **L197**). The average decision+learning time for RL-ACR is 0.037s on a single Intel Core i7-13850HX CPU (**L-560**). \n\n**[Q5]**: The reviewer raised a concern that after \\beta(s) reaches 0 the proposed method might not be safe. \n\n**Answer**: In the problems, the policy that optimizes return is within the safe region by design. The state-dependent $\\beta(s)$ will converge to 0 only for states where a better policy than the safe regularizer policy has been learned. If the exploration continues even after converging to the optimum policy and the system runs into less-exploited states,  $\\beta(s)$ will be non-zero and RL-ACR will apply safe regularization.  \n\n$~$\n\n[1] Sherr, Jennifer L., et al. \"ISPAD clinical practice consensus guidelines 2022: diabetes technologies: insulin delivery.\" Pediatric diabetes 23.8, 2022. \n\n[2] Zahedifar, Rasoul, and Ali Keymasi Khalaji. \"Control of blood glucose induced by meals for type-1 diabetics using an adaptive backstepping algorithm.\" Scientific Reports, 2022. \n\n[3] Hovorka, Roman, et al. \"Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes.\" Physiological measurement, 2004. \n\n[4] Anderson G, Chaudhuri S, Dillig I. Guiding Safe Exploration with Weakest Preconditions. In The Eleventh International Conference on Learning Representations, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rzr7Ycmx9f", "reviewer": "Reviewer_zn6h", "summary": "The authors develop RL with Adaptive Control Regularization (RL-ACR) to enable safe RL exploration. This solution uses two agents: a safety regularizer to enforce safety constraints, and an adaptive agent to perform exploration. They demonstrate their method on four critical applications against leading benchmarks.", "questions": "1. If an optimality result is available, does it require that the estimated system model $\\tilde{f}$ be exact to the actual physical process? I am thinking about this in light of your assumptions for Theorem 1. Presumably an imperfect model would void the convergence to the optimal policy and thereby the sufficient conditions required for Theorem 1. I do appreciate the empirical attention paid to model uncertainty in Section 4.2.\n\n2. How do the compute/data requirements of RL-ACR compare to the baselines?", "limitations": "The authors adequately address limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The use of two parallel agents (safety regularizer and adaptive) is novel.\n\n2. The work is clearly presented with good explanations, algorithm procedures, and visuals.\n\n3. The work studies real-world application problems.\n\n4. The evaluations test the proposed RL-ACR method against a large number of baselines.\n\n5. The evaluations substantiate the authors' claims of 1) improved performance, and 2) reduced failure rate.\n\n6. The evaluations are accompanied by thorough analysis and discussion of the results by the authors.", "weaknesses": "1. Theorem 1 requires the assumption that the RL policy converges to the optimal policy. However, I do not see a result guaranteeing that the RL-ACR policy converges to the optimal. Instead, I see a monotonic performance improvement result in Theorem 2, but this is not an optimality result.\n\n2. The work is intended to address safety, yet the authors do not provide a theoretical guarantee of closed-loop stability. Such guarantees are important for control of dynamical systems (e.g., airplanes, cars).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Weakness 1]**: The reviewer asked about the convergence result of RL-ACR.  \n\n**Response**: For state $s$, the regulation on RL policy converges to 0 ($\\beta(s)$ converges to 0) with the assumption that RL learns a better policy than the regularizer. This is a reasonable assumption since the regularizer was derived from a suboptimal model. When the regularization converges to 0, the problem is equivalent to a standard RL problem, where the assumption of RL convergence is reasonable and widely accepted in research. We mentioned this in the proof in L487-492, but we thank the reviewer for the review and will clarify the assumption for Theorem 1. \n\n**[W2]**: The reviewer asked about the closed-loop stability of RL-ACR.  \n\n**Response**: We fully understand the reviewer’s point but would like to add a couple of remarks that might help their assessment. We considered not knowing the exact model (a characteristic phenomenon in real-world application) and thus did not focus on finding a theoretical guarantee of closed-loop stability, which would be model-dependent, and the analysis would be application-specific. However, we believe that the proof on guaranteed policy return improvement (although addressing the tabular case) is a valuable theoretical result for RL-based methods such as ours. \n\n**[Question 1]**: The reviewer asked if an imperfect model prevents the convergence to the optimality result.  \n\n**Answer**: The optimality proof does not depend on the perfect model. In fact, we assume that the model used for MPC is suboptimum. If the model is imperfect, stronger policies than MPC exist and can be explored by the RL policy. This causes $\\beta(s)$ to converge to 0. If $\\beta(s)$ converges to 0, there is no regularization term for the policy optimization, thus reducing the problem to a standard RL problem with proven convergence to the optimum policy. \n\n**[Q2]**: The reviewer asked about the compute/data requirements of RL-ACR compared to the baselines. \n\n**Answer**: the average per step solving time for RL-ACR is 0.0375 on one CPU, which is similar to MPC (widely adopted in applications). Model-free methods such as SAC and CPO are faster than RL-ACR, but they cause constraint violations during training. Considering the RL-ACR solve time is sufficient for real-time control, the computation requirement is reasonable to guarantee safety during training. In terms of data, RL-ACR and the baselines collect the same amount of environment samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Rebuttal

We thank the reviewers for reviewing the paper and providing valuable feedback. We have provided point-by-point responses to the weaknesses and questions listed by the reviewers. Here, we list a few important notes on some of the comments (some leading to additional data/experiment) that may be of interest to others. 

1. In the original paper, we deliberately included methods that are recent but “well-established” as baselines in our experiments. As suggested by the reviewers, we have added two additional, more up-to-date, baselines: (silver et al., 2018) and (Yu et al., 2022), compared to which RL-ACR shows stronger performance in both safety and performance (see Figure R.I in the attached PDF). 

2. For experimental analyses, we chose safety-critical environments to showcase the important applications of our algorithm. This includes some challenging environments with nondifferentiable environment dynamics and large delays. We have now provided some additional evidence on the robustness of RL-ACR in a highly nonlinear and underactuated environment, the Acrobot (Figure R.II in the PDF). 

3. As suggested by reviewer ta4e, we performed an additional ablation study on the effect of entropy regularization in the actor loss (Figure R.III in the PDF), which shows that entropy regularization encourages diverse policies, and avoids sticking to a suboptimum policy. 

4. Although some of the technical components presented in our work are explored in the existing literature, our proposed algorithm is unique in its ability to avoid control failure from the first training episode while simultaneously converging to the performance standards of RL methods that disregard safety; this is crucial for many real-world applications and yet very rare in the literature.

Meta-review: Recently, safe reinforcement learning has been intensively studied for its application in many domains. This paper develops a new approach for safe RL exploration by combining the RL policy with an adaptive control regularizer. 
The proposed algorithm, RL-ACR, ensures safety throughout the training process by incorporating a safety regularizer that helps avoid unsafe states. RL-ACR has analytical guarantees on unbiased convergence to the optimal policy in well-explored states while prioritizing safety in less-explored states.
The combination of adaptive control and RL is an important way to achieve safe RL. 

As suggested by reviewers, more representative baselines could be added to make it a solid paper and justify the value of the theoretical analysis.The AC reads the paper and discussions and agrees with the reviewers that originality and baselines could be improved. After the rebuttals, the major concerns raised by the reviewers were addressed in the authors' responses.
Decision: Accept (poster)