Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems

Authors: haozhe  tian
homayoun  hamedmoghadam
robert  shorten
pietro  ferraro
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:25:10

Abstract: Reinforcement Learning (RL) is a powerful method for controlling dynamic systems, but its learning mechanism can lead to unpredictable actions that undermine the safety of critical systems. Here, we propose RL with Adaptive Regularization (RL-AR), an algorithm that enables safe RL exploration by combining the RL policy with a policy regularizer that hard-codes the safety constraints. RL-AR performs policy combination via a "focus module," which determines the appropriate combination depending on the state—relying more on the safe policy regularizer for less-exploited states while allowing unbiased convergence for well-exploited states. In a series of critical control applications, we demonstrate that RL-AR not only ensures safety during training but also achieves a return competitive with the standards of model-free RL that disregards safety.

Main paper: 
 introduction: 
 A wide array of control applications, ranging from medical to engineering, fundamentally deals with critical systems, i.e., systems of vital importance where the control actions have to guarantee no harm to the system functionality. Examples include managing nuclear fusion [Degrave et al., 2022] , performing robotic surgeries [Datta et al., 2021] , and devising patient treatment strategies [Komorowski et al., 2018] . Due to the critical nature of these systems, the optimal control policy must be explored while ensuring the safety and reliability of the control algorithm.Reinforcement Learning (RL) aims to identify the optimal policy by learning from an agent's interactions with the controlled environment. RL has been widely used to control complex systems [Silver et al., 2016 , Ouyang et al., 2022] ; however, the learning of an RL agent involves trial and error, which can violate safety constraints in critical system applications [Henderson et al., 2018 , Recht, 2019 , Cheng et al., 2019b] . To date, developing reliable and efficient RL-based algorithms for real-world "single-life" applications, where the control must avoid unsafety from the first trial [Chen et al., 2022] , remains a challenge. The existing safe RL algorithms either fail to ensure safety during the training phase [Achiam et al., 2017 , Yu et al., 2022a] or require significant computational overhead for action verification [Cheng et al., 2019a , Anderson et al., 2020] . As a result, classic control methods are often favored in critical applications, even though their performance heavily relies on the existence of an accurate model of the environment.Here, we address the safety issue of RL in scenarios where "estimated" environment models are available (or can be built) to derive sub-optimal control policy priors. These scenarios are representative of many real-world critical applications [Hovorka et al., 2002 , Liepe et al., 2014 , Hippisley-Cox et al., 2017 , Rathi et al., 2021] . Consider the example of devising a control policy that prescribes the optimal drug dosages for regulating a patient's health status. This is a single-life setting where no harm to the patient is tolerated during policy exploration. From available records of other patients, an estimated patient model can be built to predict the response to different drug dosages and ensure adherence to the safety bounds (set based on clinical knowledge). However, a new patient's response can deviate from the estimated model, which poses a significant challenge in control adaptability and patient treatment performance.We propose a method, RL with Adaptive Regularization (RL-AR), that simultaneously shows the safety and adaptability properties required for critical single-life applications. The method interacts with the actual environment using two parallel agents. The first (safety regularizer) agent avoids unsafe states by leveraging the forecasting ability of the estimated model. The second (adaptive) agent is a model-free RL agent that promotes adaptability by learning from actual environment interactions. Our method introduces a "focus module" that performs state-dependent combinations of the two agents' policies. This approach allows immediate safe deployment in the environment by initially prioritizing the safety regularizer across all states. The focus module gradually learns to apply appropriate policy combinations depending on the state-relying more on the safety regularizer for less-exploited states while allowing unbiased convergence for well-exploited states.We analytically demonstrate that: i) RL-AR regulates the harmful effects of overestimated RL policies, and ii) the learning of the state-dependent focus module does not prevent convergence to the optimal RL policy. We simulate a series of safety-critical environments with practically obtainable sub-optimal estimated models (e.g., from real-life sampled measurements). Our empirical results show that even with more than 60% parameter mismatches between the actual environment model and the estimated model, RL-AR ensures safety during training while converging to the control performance standard of model-free RL approaches that prioritize return over safety. 
preliminaries: 
 Through environment interactions, an RL agent learns a policy that maximizes the expected cumulative future reward, i.e. the expected return. We formalize the environment as a Markov Decision Process (MDP) M = (S, A, P, r, γ), where S is a finite set of states, A = {a ∈ R k : a ≤ a ≤ a} is a convex action-space, P : S ×A → P(S) is the state transition function, r : S ×A → [-R max , R max ] is the reward function, and γ ∈ (0, 1) is a discount factor. Let π denote a stochastic policy π : S → P(A), the value function V π and the action-value function Q π are: V π (s t ) = E at,st+1,... ∞ i=0 γ i r(s t+i , a t+i ) , Q π (s t , a t ) = E st+1,... ∞ i=0 γ i r(s t+i , a t+i ) , (1) where a t ∼ π(s t ), s t+1 ∼ P (s t , a t ) for t ≥ 0. The optimal policy π ⋆ = argmax π V π (s) maximizes the expected return for any state s. Both V π and Q π satisfy the Bellman equation [Bellman, 1966] :EQUATIONFor practical applications with complex S and A, Q π and π are approximated with neural networks Q ϕ and π θ with learnable parameters ϕ and θ. To stabilize the training of Q ϕ and π θ , they are updated using samples B from a Replay Buffer D [Mnih et al., 2013] , which stores each previous environment transitions e = (s, a, s ′ , r, d), where d equals 1 for terminal states and 0 otherwise.In this work, we are interested in acting on a safe regularized RL policy that can differ from the raw RL policy. RL approaches that allow learning from a different acting policy are referred to as "Off-policy" RL. The RL agent in our proposed algorithm follows the state-of-the-art off-policy RL algorithm: Soft Actor-Critic (SAC) [Haarnoja et al., 2018] , which uses a multivariate Gaussian policy to explore environmental uncertainties and prevent getting stuck in sub-optimal policies. For Q-network updates, SAC mitigates the overestimation bias by using the clipped double Q-learning, which updates the two Q-networks Q ϕi , i = 1, 2 using gradient descent with the gradient:where the entropy regularization term log P π θ (a ′ | s ′ ) encourages exploration, thus avoiding local optima. Target Q-networks ϕ targ,i are used to reduce drastic changes in value estimates and stabilize training. The target Q-network parameters are initialized with ϕ targ,i = ϕ i , i = 1, 2. Each time ϕ 1 and ϕ 2 are updated, ϕ targ,1 , ϕ targ,2 slowly track the update using τ ∈ (0, 1):EQUATIONFor policy updates, the policy network π θ is updated using gradient ascent with the gradient:EQUATION)3 Methodology (𝑠) Figure 1 : Schematic overview of the proposed RL-AR algorithm. RL-AR integrates the policies of the RL agent and the safety regularizer agent using a state-dependent focus module, which is updated to maximize the expected return of the combined policy.Here, we propose RL-AR, an algorithm for the safe training and deployment of RL in safety-critical applications. A schematic view of the RL-AR procedures is shown in Fig. 1 . RL-AR comprises two parallel agents and a focus module: (i) The safety regularizer agent follows a deterministic policy π reg : S → A proposed by a constrained model predictive controller (MPC); (ii) The Off-policy RL agent is an adaptive agent with π rl : S → P(A) that can learn from an acting policy that is different from π rl ; (iii) The focus module learns a state-dependent weight β : S → [0, 1] for combining the deterministic a reg (s) = π reg (s) and the stochastic a rl (s) ∼ π rl (s). Among the components, the safety regularizer has a built-in estimated environment model f : S × A → S that is different from the actual environment model, while the off-policy RL agent and focus module are dynamically updated using observed interactions in the actual environment.The RL-AR workflow is as follows: (i) π reg (s) generates a reg (s), which hard-codes safety constraints in the optimization problem over a period forecasted by f . The forecasting ability anticipates and prevents running into unsafe states for the critical system; (ii) π rl (s) generates a rl (s) to allow stochastic exploration and adaptation to the actual environment; (iii) β(s) is initialized to β(s) ≥ 1 -ϵ, ∀s ∈ S, hence prioritizing the safe π reg before π rl learns a viable policy. As more interactions are observed for a state s and the expected return of π rl (s) improves, β(s) gradually shifts the focus from the initially suboptimal π reg (s) to π rl (s). 
the safety regularizer: 
 The safety regularizer of RL-AR is a constrained MPC, which, at any state s t , optimizes the Nstep system behavior forecasted using the estimated environment model f by solving the following constrained optimization problem:EQUATIONwhere J k (s k , a k ) and J N (s t+N ) are the stage and terminal cost functions and g(s k ) ≥ 0 is the safety constraint. By hard-coding the safety constraints in the optimization (via g(s k ) ≥ 0) over the Take action a = β ψ (s)a reg (s) + (1 -β ψ (s))a rl (s), a reg (s) = π reg (s), a rl (s) ∼ π θ (s)Store transition e = (s, a, s ′ , r, d, a reg ) in D Update π θ by gradient ascent with Eq. ( 5) 10:Update β ψ by gradient ascent with:∇ ψ 1 |B| (s,areg)∈B min i=1,2 Q ϕ i (s, β ψ (s)areg + (1 -β ψ (s))a rl (s)) , a rl (s) ∼ π θ (s)11:Update Q ϕtarg,i , i = 1, 2, using Eq. ( 4)12:end if 13: t = t + 1 14: until convergence is true prediction horizon, MPC prevents failure events that are not tolerated in critical applications. MPC iteratively solves for the N-step optimal actions in each time step and steers the environment to the desired state. At any time step t, solving the optimization problem in Eq. ( 6) yields a sequence of N actions a t:t+N -1 , with only the first action a t in the sequence adopted for the current time step, i.e., a reg (s t ) = π reg (s t ) = a t . The system transitions from s t to s t+1 by taking the action a reg (s t ), and the optimization problem is solved again over {t + 1 : t + 1 + N } to obtain a reg (s t+1 ). For practical applications with continuous state space, the optimization problem in Eq. ( 6) is efficiently solved using the Interior Point Optimizer [Andersson et al., 2019] . Since MPC solves similar problems with slight variations at each time step, the computational complexity is further reduced by using the solution from the previous step as the initial guess.The focus module in RL-AR combines the actions proposed by the safety regularizer agent and the RL agent using a weighted sum. At state s, the combined policy π β takes the following action a β (s):EQUATIONLemma 1. (Policy Regularization) In any state s ∈ S, for a multivariate Gaussian RL policy π rl with mean πrl (s) and covariance matrix Σ = diag(σ 2 1 (s), σ 2 2 (s), . . . , σ 2 k (s)) ∈ R k×k , the expectation of the combined action a β (s) derived from Eq. ( 7) is the solution to the following regularized optimization with regularization parameter λ = β(s)/(1 -β(s)):EQUATIONWe provide the proof of Lemma 1 in Appendix A.1, which is a state-dependent extension of the proof in [Cheng et al., 2019b] . Lemma 1 shows that the state-dependent β(s) offers a safety mechanism on top of the safety regularizer. Since β(s) is initialized close to 1 for ∀s ∈ S, a strong regularization (λ → ∞) from the safety regularizer policy is applied at the early stages of training.As learning progresses, the stochastic combined policy inevitably encounters rarely visited states, where π rl is poor due to the overestimated Q. However, the regularization parameter λ remains large for these states, thus preventing the combined policy from safety violations by regularizing its deviation from the regularizer's safe policy. This deviation is quantified in the following theorem. Theorem 1. Assume the reward R and the transition probability P of the MDP M are Lipshitz continuous over A with Lipschitz constants L R and L P . For any state s ∈ S, the difference in expected return between following the combined policy π β and following the safety regularizer policy π reg , i.e., |V π β (s) -V πreg (s)|, has the upper-bound:EQUATION)where |S| is the cardinality of S, ∆a = |a rl (s) -a reg (s)| is the bounded action difference at s.We provide the proof of Theorem 1 in Appendix A.2. Theorem 1 shows that when a state s has not been sufficiently exploited and its corresponding β(s) updates have been limited accordingly, the sub-optimality of the RL policy π rl has limited impact on the expected return of the combined policy π β , which is the actual acting policy. This is because 1 -β(s) remains close to zero at this stage, leading to only minor expected return deviations from the safety regularizer's policy π reg .The focus module derives the policy combination (from the policies of safety regularizer and offpolicy RL agent) that maximizes the expected return. For any state s, the state-dependent focus weight β(s) is learned through updates according to the following objective:β ′ (s) = argmax β∈[0,1] E [Q π β (s, βa reg (s) + (1 -β)a rl (s))] , a reg (s) = π reg (s), a rl (s) ∼ π rl (s). (10)Equation ( 10) is similar to the actor loss in actor-critic methods, however, instead of optimizing the policy network, Eq. ( 10) optimizes β(s) for policy combination.Compared to a scalar combination weight that applies the same policy combination across all states (e.g., as in [Cheng et al., 2019b] ), the updated state-dependent weight β ′ (s) in Eq. ( 10) guarantees monotonic performance improvement at least in the tabular cases, i.e., the update β ′ (s t ) at a state s t results in V π β ′ (s) ≥ V π β (s) for all states s ∈ S, where π β ′ is the combined policy proposed by β ′ . This can be proved by observing that the update in Eq. ( 10) results in a non-negative advantage for all states s, i.e., Q π β (s, π β ′ ) ≥ Q π β (s, π β ), ∀s ∈ S, where (with slight abuse of notation) we use Q(s, π) to denote Q(s, a) with a ∼ π(s). See Theorem 2 in Appendix A.3 for the detailed proof. Lemma 2. (Combination Weight Convergence) For any state s, assume the RL policy π rl converges to the optimum policy π ⋆ that satisfies Q(s, π ⋆ ) > Q(s, π), ∀π ̸ = π ⋆ , then β ′ (s) = 0 will be the solution to Eq. (10) that achieves the optimal policy combination.Lemma 2 follows as π reg ̸ = π ⋆ due to the sub-optimal model used to derive π reg . Let a ⋆ (s) ∼ π ⋆ (s) denote the optimum action at state s. If β(s) ̸ = 0, then β(s)a reg (s) + (1 -β(s))a rl (s) = β(s)a reg (s) + (1 -β(s))a ⋆ (s) ̸ = a ⋆ (s). Therefore, the solution to Eq. (10), i.e., the updated focus weight β ′ (s), can only be 0. Theorem 3. (Policy Combination Bias) For any state s, the distance between the combined action a β (s) and the optimal action a ⋆ (s) has the following lower-bound:EQUATION)If a Gaussian RL policy π rl converges to the optimum policy π ⋆ (s) with Q(s, π ⋆ ) > Q(s, π), ∀π ̸ = π ⋆ , then the combined policy π β (s) can have unbiased convergence to the optimum Gaussian policy π ⋆ with total variance distance D TV (π β (s), π ⋆ (s)) = 0.The proof of Theorem 3 is given in Appendix A.5. Theorem 3 shows that by adaptively updating β(s), the unbiased convergence of the combined policy can be achieved assuming i) a unique optimum solution and ii) the convergence of the RL agent, where the former follows naturally for most real-life control applications and the latter is well-established in the RL literature (the convergence of the specific RL agent used in RL-AR was proved in [Haarnoja et al., 2018] ).Algorithm 1 shows the pseudo-code of RL-AR, where Q, π rl , and β are approximated with neural networks for practical applications with large or continuous state space. Note that the policy regularization (Lemma 1) and the convergence of RL-AR to the optimum RL policy (Lemma 2 and Theorem 3) still hold when using function approximation. For the RL agent, we take the standard approach of approximating Q π and π with neural networks Q ϕ and π θ . The focus module β(s) is approximated with a neural network β ψ with outputs scaled to the range (0, 1). Before learning begins, β ψ is pretrained to output values close to 1 (e.g., β ψ (s) ≥ 1 -ϵ) for all states to prioritize the safe π reg . While interacting with the environment, each transition e = (s, a, s ′ , r, d, a reg ) is stored in a replay buffer D. Since π reg is deterministic and not subject to updates, by storing the action term a reg in D, the optimization problem in Eq. ( 6) needs to be solved only once for any state s, significantly lowering the computational cost.In Algorithm 1, details of Q ϕi and π θ updates (lines 8-9) are omitted as they follow the standard SAC [Haarnoja et al., 2018] paradigm elaborated in Section 2. After updating Q ϕi and π θ , the focus module β ψ is updated using samples (s, a reg ) from replay buffer D. As shown in line 10, β ψ is updated using gradient ascent with the gradient:EQUATIONNote that the updated Q ϕi , i = 1, 2, and π θ are used in Eq. ( 12) to allow quick response to new information. The clipped double-Q learning (taking the minimum Q ϕi , i = 1, 2) mitigates the overestimation error. Although exploitation level is not explicitly considered in Eq. ( 10), the use of replay buffer and the gradient-based updates in Eq. ( 12) mean more frequently-visited states with well-estimated Q values will affect β ψ (s) more, whereas rarely-visited states with overestimated Q values affect β ψ (s) less. 
numerical experiments: 
 Here, RL-AR is validated in critical settings described in Section 1, where the actual environment model P is unknown, but an estimated environment model f is available (e.g., from previous observations in the system or a similar system)1 . Four safety-critical environments are implemented:• Glucose is the critical medical control problem of regulating blood glucose level against mealinduced disturbances [Batmani, 2017] . The observations are denoted as (G, Ġ, t), where G is the blood glucose level, Ġ = G t -G t-1 , and t is the time passed after meal ingestion. The action is insulin injection, denoted as a I . Crossing certain safe boundaries of G can lead to catastrophic health consequences (hyperglycemia or hypoglycemia).• BiGlucose is similar to the Glucose environment but capturing more complicated blood glucose dynamics, with 12 internal states (11 unobservable), 2 actions with large delays, and nondifferentiable piecewise dynamics. [Kalisvaart et al., 2023] . The observations are the same as Glucose.The actions are insulin and glucagon injections, denoted as (a I , a N ).• CSTR is a continuous stirred tank reactor for regulating the concentration of a chemical C B [Fiedler et al., 2023] . The observations are (C A , C B , T R , T K ), where C A and C B are the concentrations of two chemicals; T R and T K are the temperatures of the reactor and the cooler, respectively. The actions are the feed and the heat flow, denoted as (a F , a Q ). Crossing safe boundaries of C A , C B , and T R can lead to tank failure or even explosions.• Cart Pole is a classic control problem of balancing an inverted pole on a cart by applying horizontal force to the cart. The environment is adapted from the gymnasium environment [Towers et al., 2023] with continuous action space. The observations are (x, ẋ, θ, θ), where x is the position of the cart, θ is the angle of the pole, ẋ = x t -x t-1 , and θ = θ t -θ t-1 . The action is the horizontal force, denoted as a f . The control fails if the cart reaches the end of its rail or the pole falls over.All environments are simulated following widely accepted models and parameters [Sherr et al., 2022, Yang and Zhou, 2023] , which are assumed to be unknown to the control algorithm. The estimated models and the actual environments are set to have different model parameters. For Glucose and BiGlucose, the estimated model parameters are derived from real patient measurements [Hovorka et al., 2004, Zahedifar and Keymasi Khalaji, 2022] . The environment models, parameters, and reward functions are detailed in Appendix B.The baseline methods used in the experiments are: i) MPC [Fiedler et al., 2023] , the primary method for control applications with safety constraints [Hewing et al., 2020] ; ii) SAC [Haarnoja et al., 2018] ,a model-free RL that disregards safety during training, but achieves state-of-the-art normalized returns; iii) Residual Policy Learning (RPL) [Silver et al., 2018] , an RL method that improves a sub-optimal MPC policy by directly applying a residual policy action; iv) Constrained Policy Optimization (CPO) [Achiam et al., 2017] , a widely-used risk-aware safe RL benchmark based on the trust region method; and v) SEditor [Yu et al., 2022b] , a more recent, state-of-the-art safe RL method that learns a safety editor for transforming potentially unsafe actions.The proposed method, RL-AR, uses MPC as the safety regularizer agent and SAC as the off-policy RL agent. The two agents in RL-AR each follow their respective baseline implementations. The focus module in RL-AR has a [128, 32] hidden layer size with ReLU activation, and k outputs scaled to (0, 1) by a shifted tanh. Additional detail and hyperparameters of the implementations are provided in Appendix C. Our RL-AR implementation has an average decision and update time of 0.037 seconds per step on a laptop with a single GPU, meeting real-time control requirements across all environments. In Appendix D we present ablation studies on the benefit of state-dependent focus weight and the choice of SAC as the RL agent. Next, since the estimated environment model f is integrated into RL-AR, MPC, and RPL, for a fair comparison we pretrain the model-free SAC, CPO, and SEditor using f as an environment simulator; this allows all methods to access the estimated model before training on the actual environment.Figure 2 compares the normalized episodic return and the number of failures for different methods over training episodes; the proposed method is compared with SAC and RPL in Fig. 2A , and with MPC, CPO, and SEditor (safety-aware methods) in Fig. 2B . The mean (solid lines) and standard deviation (shaded area) in Fig. 2 are obtained from 5 independent runs using different random seeds.Episodes are terminated on failure, resulting in varying episode lengths, thus, the episodic returns are normalized by episode lengths.Two important insights can be drawn from Fig. 2 . First, the normalized return curves show that RL-AR consistently achieves higher returns faster than other methods across all environments. RL-AR begins with a reliable initial policy derived from the safety regularizer and incrementally integrates a learned policy, resulting in stable return improvements (as suggested by Lemma 1 and Theorem 1). RL-AR shows a steady return improvement, except for some fluctuations in the CSTR environment which the method quickly recovers from. In contrast, the baseline methods-SAC and RPL, which apply drastic actions based on overestimated returns, or CPO and SEditor, which impose constraints using biased cost estimates derived from simulations using f -exhibit significant return degradation and even failures. Second, RL-AR effectively avoids failure during training (see the bottom rows in Fig. 2A&B ). Note that pertaining on f leads to fewer failures in the actual environment for SAC, CPO, and SEditor (compare with the results in Table 1 ). However, SAC, CPO, and SEditor continue to fail despite the pretraining (the only exception is SEditor in the Glucose environment), indicating that pretraining on estimated model is not an effective approach to achieve safety. In CSTR and Cart Pole environments, and only in a limited number of episodes during the early stages of training, the proposed RL-AR policy's normalized return falls below that of the static MPC policy. This can occur due to RL-AR reaching insufficiently learned states (with overestimated Q values). Nevertheless, since β(s) is close to 1 for these insufficiently learned states, the dominance of the safety regularizer agent allows RL-AR to converge to high returns without compromising the safety (as shown in Theorem 1).Besides ensuring safer training, RL-AR theoretically enables unbiased convergence to the optimal RL policy (as shown in Theorem 2). We validate this by testing whether RL-AR matches the return of SAC. SAC is shown to consistently converge to well-performing control policies, competitive if not better than other state-of-the-art RL algorithms [Raffin et al., 2021 , Huang et al., 2022] . In Fig. 3 , we compare the control trajectories of RL-AR, SAC, and MPC and the returns of their converged policies after training; we run the converged policies without stochastic exploration. RL-AR significantly outperforms MPC, achieving faster regulation, reduced oscillation, and smaller steady- MDP [Achiam et al., 2017 , Yang et al., 2020] . More recent methods use learned models to filter out unsafe actions [Bharadhwaj et al., 2020] . However, these methods need to observe failures to estimate the safety cost, thus do not ensure safety during training. This category of methods does not apply to the single-life setting in this work, i.e., no failure is tolerated in the actual environment. Nevertheless, in Fig. 2 we evaluate pertaining CPO [Achiam et al., 2017] and SEditor [Yu et al., 2022b] using simulation with the estimated model to obtain risk estimation before training in the actual environment.The second category relies on an estimated model of the system dynamics. Some methods enforce safety constraints using Control Barrier Function (CBF) [Cheng et al., 2019a] . However, CBF minimizes the control effort without directly optimizing the system performance. In contrast, the MPC regularizer used in RL-AR enforces safety constraints while optimizing the predicted performance, resulting in high performance during training. Some methods compute a model-based projection to verify the safety of actions [Bastani, 2021 , Kochdumper et al., 2023 , Fulton and Platzer, 2018] . However, the scalability of verification-based methods for complex control applications is an issue. Anderson et al. [2020] propose using neurosymbolic representations to reduce verification complexity, but the computational cost remains to be high. On the other hand, the average time for RL-AR to take a step (including the environment interaction and network updates) in the four environments in Section 4 is 0.037 s, which is practical for real-time control. Gros and Zanon [2019] and Zanon et al. [2020] use MPC as the policy generator and use RL to dynamically tune the MPC parameters in the cost functions and the estimated environment model. Assuming discrepancies between the estimated model parameters and the actual environment parameters, the tuning increases the MPC's performance. However, this is a strong assumption since there are other discrepancies, such as neglected dynamics and discretization errors. However, the RL-AR proposed in this work can theoretically converge to the optimal policy by utilizing the model-free RL agent.It is important to note that although the MPC regularizer accelerates the learning of the RL agent, RL-AR is not a special case of transferring a learned policy. The MPC regularizer used in our proposed algorithm forecasts the system behavior and hard-codes safety constraints in the optimization. The main role of the MPC is to keep the RL-AR actions safe in the actual environment-not transferring knowledge. Transfer learning in RL studies the effective reuse of knowledge, especially across different tasks [Taylor and Stone, 2009, Glatt et al., 2020] . By reusing prior knowledge, transferred RL agents skip the initial random trial-and-error and drastically increase sampling efficiency [Karimpanal et al., 2020, Da Silva and Costa, 2019] . However, transferred RL agents are not inherently risk-aware, and thus can still steer the actual environment into unsafe states. For this reason, transferred RL is not generally considered effective for ensuring safety.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See Section 3, Section 4, and Appendix C. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code available at https://github.com/HaozheTian/RL-AR. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion and future works: 
 Controlling critical systems, where unsafe control actions can have catastrophic consequences, has significant applications in various disciplines from engineering to medicine. Here, we demonstrate that the appropriate combination of a control regularizer can facilitate safe RL. The proposed method, RL-AR, learns a focus module that relies on the safe control regularizer for less-exploited states and simultaneously allows unbiased convergence for well-exploited states. Numerical experiments in critical applications revealed that RL-AR is safe during training, given a control regularizer with reasonable safety performance. Furthermore, RL-AR effectively learns from interactions and converges to the performance standard of model-free RL that disregards safety.One limitation of our setting is the assumption that the estimated model has reasonable accuracy for deriving a viable control regularizer. Although this assumption is common in the control and safe RL literature, one possible direction for future work is to design more robust algorithms against inaccurate estimated models of the actual environment. A potential approach is to update the estimated model using observed transitions in the actual environment. However, the practical challenge is to adequately adjust all model parameters even with a small number of transitions observed in the actual environment. In addition, for such an approach, managing controllability, convergence, and safety requires careful design and tuning. 1 (s), σ 2 2 (s), . . . , σ 2 k (s)) ∈ R k×k , the expectation of the combined action a β (s) derived from Eq. ( 7) is the solution to the following regularized optimization with regularization parameter λ = β(s)/(1 -β(s)): 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Assets properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: EQUATIONProof. For state s ∈ S, the focus module outputs a fixed β = β(s). For the fixed β, the proof of Lemma 1 is similar to the proof by Cheng et al. [2019b] . Since the RL policy is a Gaussian distributed policy N (π rl (s), Σ) with the mean action πrl (s) and an exploration noise with covariance Σ = diag(σ 2 1 , σ 2 2 , . . . ), the combined action a β (s) also follows a Gaussian distribution:EQUATION)Let f (µ, Σ) be the probability density function (PDF) of N (µ, Σ). The product of two multivariate Gaussian PDFs is proportional to another multivariate Gaussian PDF with the following mean and covariance:EQUATION)The mean of a β (s), βa reg (s) + (1 -β)π rl (s), can be expressed in the following form:EQUATIONThe covariance matrix Σ can be expanded into the following form:EQUATION)Using Eq. ( 15), the PDF of a β (s) can be expressed as the multiplication of two PDFs, as shown below:EQUATION)With the definition ∥x∥ Σ = x T Σ -1 x, the PDF of a β (s) can be written as:EQUATIONwhere the constant c is as follows:EQUATION)Since β is in the range (0, 1), f (a β (s)) monotonically decreases as ∥a -πrl (s)∥ Σ + β 1-β ∥aa reg (s)∥ Σ increases. Therefore, the probability of π(s) is maximized when the term is minimized, leading to the following optimization problem:EQUATIONAssuming the RL policy π rl converges to argmax π Q(s, π(s)), Eq. ( 21) can be written as:EQUATIONA.2 Deviation of combined policy from safety regularizer Theorem 1. Assume the reward R and the transition probability P of the MDP M are Lipshitz continuous over A with Lipschitz constants L R and L P . For any state s ∈ S, the difference in expected return between following the combined policy π β and following the safety regularizer policy π reg , i.e., |V π β (s) -V πreg (s)|, has the upper-bound:EQUATION)where |S| is the cardinality of S, and ∆a = |a rl (s) -a reg (s)| is the bounded action difference at s.Proof. Let us define value function vectors v β and v reg in R |S|×1 , for which the s-th entries are[v β ] s = V π β (s) and [v reg ] s = V πreg (s). Also, let us by r β and r reg denote reward vectors in R |S|×1 , with the s-th entries of the reward vectors being [r β ] s = r(s, a β (s)) and [r reg ] s = r(s, a reg (s)). We define state-transition matrices P β and P reg in R |S|×|S| , with (s, s ′ )-th entries as [P β ] s,s ′ = P (s, a β (s)) and [P reg ] s,s ′ = P (s, a reg (s)). According to the vectorized Bellman equation, the difference between v β and v reg satisfies the following relationship:EQUATIONLet d T β,s be the s-th row of (I -γP β ) -1 and d T reg,s be the s-th row of (I -γP reg ) -1 . Since (I -γP β ) -1 can be expanded as a Neumann series I + γP β + γ 2 P 2 β • • • , the upper-bound for the elements of d β,s and d reg,s is 1/(1 -γ), i.e., ∥d β,s ∥ ∞ and ∥d reg,s ∥ ∞ are less than or equal to 1/(1 -γ). From Eq. ( 24), the value functions V π β (s) and V πreg (s) for a specific state s ∈ S satisfies:EQUATIONFirst, we consider part (a) of Eq. ( 25). Assuming R(s, a) is Lipschitz continuous over A with Lipschitz constant L R , we have:EQUATIONUsing the Holder's inequality, part (a) of Eq. ( 25) has the following upper-bound:EQUATIONSecond, we consider part (b) of Eq. ( 25). For each state s ∈ S, define the s-th rows of P β and P reg as p T β,s and p T reg,s . The vectors p β,s and p reg,s each represents a discrete probability distribution. Because we assume P (s, a) is Lipschitz continuous over A with Lipschitz constant L P , the upperbound on each item in vector (P β -P reg )v reg is derived below:EQUATIONTherefore, part (b) of Eq. ( 25) has the following upper-bound:γ|d T reg,s (P β -P reg )v reg | ≤ ∥d reg,s ∥ ∞ ∥(P β -P reg )v reg ∥ 1 ≤ γ|S|L P R max (1 -γ) 2 (1 -β(s))∆a (29)The proof is complete by substituting Eq. ( 27) and Eq. ( 29), respectively, into parts (a) and (b) of Eq. (25).Theorem 2. (Focus Module Performance Improvement) The focus weight β ′ (s) updated by Eq. (10) satisfies V π β ′ (s) ≥ V π β (s), ∀s ∈ S, i.e., the expected return monotonically improves.Proof. According to the performance difference lemma in [Kakade and Langford, 2002] , in all states s ∈ S, the expected return difference between the two policies satisfies:EQUATIONwhereA π (s, π ′ ) = Q π (s, π ′ ) -Q π (s, π) is the advantage function, d π ′ ,sis the normalized discounted occupancy induced by policy π ′ from the starting state s. The update in Eq. ( 10) results inEQUATION)Using the above in Eq. ( 30), and combined with the fact that the discount factor γ is in range (0, 1), we can rewrite Eq. ( 30) asV π β ′ (s) ≥ V π β (s), ∀s ∈ S.Lemma 2. (Combination Weight Convergence) For any state s, assume the RL policy π rl converges to the optimum policy π ⋆ that satisfies Q(s, π ⋆ ) > Q(s, π), ∀π ̸ = π ⋆ , then β ′ (s) = 0 will be the solution to Eq. (10) that achieves the optimal policy combination.Proof. Since a sub-optimal model is used to derive π reg , π reg ̸ = π ⋆ . Let a ⋆ (s) ∼ π ⋆ (s) denote the optimum action at state s.If β(s) ̸ = 0, then β(s)a reg (s) + (1 -β(s))a rl (s) = β(s)a reg (s) + (1 - β(s))a ⋆ (s) ̸ = a ⋆ (s). Therefore, the solution to Eq. ( 10), i.e., the updated focus weight β ′ (s), can only be 0.A.5 Convergence to the RL policy Theorem 3. (Policy Combination Bias) For any state s, the distance between the combined action a β (s) and the optimal action a ⋆ (s) has the following lower-bound:EQUATION)If a Gaussian RL policy π rl converges to the optimum policy π ⋆ (s) with Q(s, π ⋆ ) > Q(s, π), ∀π ̸ = π ⋆ , then the combined policy π β (s) can have unbiased convergence to the optimum Gaussian policy π ⋆ with total variance distance D TV (π β (s), π ⋆ (s)) = 0.Proof. First, we prove the lower-bound for |a β (s) -a ⋆ (s)|. The proof of lower-bound is similar to [Cheng et al., 2019b] . We expand the distance between the safety regularizer action a reg and the combined action a β as follows:EQUATION)Following triangle inequality, the distance between the combined action and the optimum action is as follows:EQUATION)To prove the convergence of π β , we first examine the convergence of β(s). According to Lemma 2, β(s) converges to 0 under the assumption that the RL policy π rl converges to π ⋆ (s) with Q(s, π ⋆ ) > Q(s, π), ∀π ̸ = π ⋆ . Because β(s) = 0, the combined policy has variance equal to π θ . According to Lemma 1, the following holds for any state s:EQUATIONThus the the converged combined policy π β (s) = π rl (s) = π ⋆ (s) with total variance distance D TV (π β (s), π ⋆ (s)) = 0.In this section, we introduce the four environments (Glucose, BiGlucose, CSTR, and Cart Pole) used in Section 4, with detailed environment models, parameters, and reward functions. MPC minimizes the stage cost J k and terminal cost J N , while RL maximizes rewards r, for consistency, we set the MPC costs to J k = J N = -r when validating the MPC in the environments. The Glucose environment simulates blood glucose level, denoted as G, against meal-induced disturbances which elevate G and cause hyperglycemia. The observations are (G, Ġ, t), where t is the time passed after meal ingestion. The action is the injection of insulin a I , which gradually lowers G but with a large delay. Excessive injection of insulin can cause life-threatening hypoglycemia. Safety constraints are imposed on the blood glucose level G, as both the elevated and reduced G result in severe health risks (hyperglycemia and hypoglycemia, respectively). The blood glucose model contains 3 state variables regulated by the ordinary differential equations (ODEs) given below:EQUATIONamong which only G can be observed. Because not all states are observed, the closed loop is maintained only for the observed G for MPC and RL-AR. The term D t represents the time-varying disturbance caused by the meal disturbance:EQUATION)The model parameters adopted by the estimated model and the actual environment are given by Table 2 . The initial states are determined by setting the left-hand side of Eq. ( 36) to zeros and solving the steady-state equations. The reward function for Glucose is the Magni risk function [Fox et al., 2020] , which gives stronger penalties for low blood glucose levels to prevent hypoglycemia:EQUATIONB.2 BiGlucose EQUATIONwhere the intermediate variables F c 01 and F R are piecewise functions of the measurable blood glucose mass G = 18 × Q 1 /V G , as shown below:EQUATIONEQUATIONSince only G can be observed among the 12 states, the closed loop is maintained only for G in MPC and RL-AR. The term U G represents the time-varying disturbance caused by the meal disturbance:EQUATIONThe model parameters adopted by the estimated model and the actual environment are given by Table 3 . This extended model proposed by Herrero et al. [2013] and Kalisvaart et al. [2023] captures more complicated blood glucose dynamics, allowing the use of both insulin injection and glucagon injection as actions. This leads to better regulation performance, but also drastically increases the complexity of the problem due to its large number of unobservable states, delayed action responses, and nondifferentiable piecewise dynamics [Kalisvaart et al., 2023] .The initial states are determined by setting the left-hand side of Eq. ( 39) to zeros and solving the steady-state equations. The reward function for BiGlucose is the Magni risk function [Fox et al., 2020] , which gives stronger penalties for low blood glucose levels to prevent hypoglycemia:EQUATIONThe CSTR environment simulates the concentration of a target chemicals in a continuous stirred tank reactor. The observations are (C A , C B , T R , T K ), where C A and C B are the concentrations of two chemicals, T R is the temperature of the reactor, and T K is the temperatures of the cooling jacket. The actions are the feed and the heat flow (a F , a Q ). Safety constraints are imposed on the chemical concentrations and reactor temperature, as crossing the safe boundaries for any of them can lead to tank failure or even explosions. This model contains four state variables regulated by the ODEs given below:EQUATIONwhere the intermediate variables are:K 1 = = βk 0,ab exp( -E A,ab T R + 273.15 ) K 2 = = k 0,bc exp( -E A,bc T R + 273.15)K 3 = = k 0,ad exp( -αE A,bc T R + 273.15).(EQUATIONThe model parameters adopted by the estimated model and the actual environment are given by Table 4 and Table 5 . The initial concentration of the target chemical C B,0 is set to 0.5. The reward function for CSTR is as follows:EQUATIONThe Cart Pole environment simulates an inverted pole on a cart. The environment is the continuous action adaptation of the gymnasium environment [Towers et al., 2023] . The observations are (x, ẋ, θ, θ), where x is the position of the cart and θ is the angle of the pole. The action is the horizontal force a f . Safety constraints are imposed on x and θ as the control fails if the cart reaches theThe MPC and the safety regularizer in RL-AR are implemented using [Fiedler et al., 2023] . The implementations for SAC and the RL agent in RL-AR are based on [Huang et al., 2022] . RPL, CPO, and SEditor follow the implementations in [Silver et al., 2014] , [Ray et al., 2019] , and [Yu et al., 2022b] , respectively. The same hyperparameters are used by RL-AR for all experiments in Section 4, which are listed in Table 7 . The baseline methods' network structure and training parameters are set to be the same as RL-AR. As discussed in Section 3 and Theorem 2, using a state-dependent focus module β(s) for policy combination (compared to using a scalar weight β) offers the advantage of achieving monotonic performance improvements at least in the tabular setting. Here, we empirically verify the advantage of using the state-dependent β ψ (s) versus a scalar weight in Fig. 5 by analyzing the normalized returns (left panel) and the focus weights (right panel) during training in the Glucose environment (Fig. 5 ). The mean (solid lines) and standard deviation (shaded area) in Fig. 5 are obtained from 5 independent runs using different random seeds.Practically, RL-AR with state-dependent β ψ (s) does not show a strictly monotonic policy improvement, which can be attributed to the neural network approximation. However, improvements in the normalized return are significantly more steady when using β ψ (s) rather than a scalar β, as shown by the blue and red curves in the left panel of Fig. 5 . The right panel of Fig. 5 shows the evolution of the focus weights (used by RL-AR) versus the training steps. Although both β ψ (s) and β converge to zero after approximately the same number of steps, β ψ (s) applies different focus weights depending on specific states encountered as seen by the fluctuations in the blue curve in the right panel of Fig. 5 .We conduct an ablation study to compare using SAC as the RL agent in RL-AR with using another state-of-the-art RL algorithm, TD3 [Fujimoto et al., 2018] . The key difference between SAC and TD3 is that SAC incorporates the entropy regularization term, -α log P π θ (a|s), in Eq. (3) and Eq. ( 5). The normalized return curves, shown in Fig. 6 , demonstrate that RL-AR with SAC as the RL agent achieves a higher normalized return at a faster rate. While RL-AR promotes safety and stability at the cost of reducing the exploration intensity of the combined policy (as proved in Lemma 1), which could potentially slow down the discovery of the optimal policy, SAC's entropy regularization counteracts this by promoting the use of more diverse policies. A closer examination reveals that the SAC curve locally exhibits more minor fluctuations than the TD3 curve, illustrating SAC's ability to use more diverse policies. SAC as the RL module TD3 as the RL moduleFigure 6 : Comparison of normalized return between using the SAC and using TD3 [Fujimoto et al., 2018] as the RL agent in the Glucose environment (standard deviations are shown in the shaded area). The main difference between SAC and TD3 is that SAC has the entropy regularization terms in its objectives, which are intended to encourage diverse policies and stabilize training. In Fig. 7 , we show several β ψ (s t ) curves from training RL-AR in various Glucose environments, created by varying the environment model parameters n and p 2 . Let ñ and p2 be the parameters of the estimated model f , the actual environment models have n = 3ñ/16 and p 2 = p2 /4, p2 /2, 1p 2 , 2p 2 to mimic deviating characteristics of new patients. When there are large discrepancies between the environment and f (e.g., n = 3ñ/16 and p 2 = p2 /4), the safety regularize fails initially, but the focus weight β ψ (s t ) decreases rapidly, enabling RL-AR to recover from initial failures by rapidly shifting from the sub-optimal safety regularizer policy to the stronger learned RL policy. Conversely, β ψ (s t ) converges more slowly to zero when the safety regularizer performs well in the actual environment, as it becomes more challenging to find an RL policy that significantly outperforms the safety regularizer policy in such cases.In Section 4, we evaluate RL-AR in several widely recognized challenging safety-critical environments. For example, the BiGlucose environment (see Appendix B.2) involves 11 unobservable states, two actions with significant delays, and complex, nondifferentiable piecewise dynamics.Here, we provide further evidence of the effectiveness of RL-AR in the Acrobot environment, an adaptation of the Gymnasium environment [Towers et al., 2023] with a continuous action space (Fig. 8 ). The Acrobot environment simulates two links connected by a joint, with one end of the connected links fixed. The links start facing downward. The objective is to swing the free end above a given target height as quickly as possible by applying torque to the joint. Failure is defined as the tip not reaching the target height in 400 time steps. For validation, we set the actual Acrobot environment parameters l 2 = 1.1 l2 , m 2 = m2 , where l2 = 1.0 is the lower link length parameter of the estimated model f , and m2 = 1.0 is the lower link weight parameter of the estimated model f .The Acrobot environment is particularly challenging for RL-AR's policy regularizer due to: i) its highly nonlinear, under-actuated dynamics, and ii) its definition of failure as not achieving the target within a given time limit, which cannot be easily formalized as a constraint. As Fig. 8 shows, RL-AR can swing up the tip in the first episode by initially relying on the viable safety regularizer policy. Throughout training, RL-AR ensures safety while converging to a similar normalized return as SAC, which focuses only on return and fails many times during training. This illustrates RL-AR's robustness in challenging tasks and its potential in applications where failures are defined in terms of time limit and are hard to formalize as constraints.This paper presents work that aims to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which, based on our judgment, must be specifically highlighted here.Answer: [Yes] Justification: The proofs for all theoretical results (Lemma 1, Theorem 1, Theorem 2, Lemma 2, and Theorem 3) are provided in Appendix A. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Code available at https://github.com/HaozheTian/RL-AR.Related WorksThe existing safe RL works can be roughly divided into two categories[Garcıa and Fernández, 2015]. The first category does not require knowledge of the system dynamics. These methods often rely on probabilistic approaches[Geibel and Wysotzki, 2005, Liu et al., 2022] or constrained 


Decision: Recently, safe reinforcement learning has been intensively studied for its application in many domains. This paper develops a new approach for safe RL exploration by combining the RL policy with an adaptive control regularizer. 
The proposed algorithm, RL-ACR, ensures safety throughout the training process by incorporating a safety regularizer that helps avoid unsafe states. RL-ACR has analytical guarantees on unbiased convergence to the optimal policy in well-explored states while prioritizing safety in less-explored states.
The combination of adaptive control and RL is an important way to achieve safe RL. 

As suggested by reviewers, more representative baselines could be added to make it a solid paper and justify the value of the theoretical analysis.The AC reads the paper and discussions and agrees with the reviewers that originality and baselines could be improved. After the rebuttals, the major concerns raised by the reviewers were addressed in the authors' responses.

Reviews: review: paper_summary: paper_summary: The paper proposed RL-ACR algorithm to solve safe RL exploration problem where the RL policy must be safe during and after training. It achieves this by learning a "focus network" which mixes the action from MPC-based safety regularizer and conventional RL module. The algorithm is claimed to address "single-life" safe RL applications where safety is critical even during training.
strength_weakness: strength_weakness: 1. Originality
* Most of the components proposed in RL-ACR have been explored previously: (i) combining two-prong policy [1], (ii) MPC with learned model [2, 3], (iii) safe RL with model rollout [4, 5, 6].
* The proposed focus network learns to outputs a scalar which is a mixing coefficient between the two policies. The focus network is similar to the learnable Safe Editor component in [1]. However, Safe Editor seems more sophisticated as it learns to edit the action itself. I wonder if there could be comparison between the two for evaluation.
2. Significance
* RL-ACR relies heavily on the the estimated model $\tilde{f}$ for safety. Consequently, the reliability (and accuracy) of the model is an important factor to guarantee safety. As pointed out in [4], even for a highly accurate model, it becomes increasingly challenging to predict future trajectory with reasonably good accuracy when the model unrolling horizon becomes larger because the error cascades over many timesteps. In Eq7 of the paper, RL-ACR is required to perform N step model unrolling. I have some doubts on how to obtain a reliable model to support RL-ACR.
* Related to previous point, in the conducted experiments, the model $\tilde{f}$ used in the paper is synthetically generated. It is not clear whether a model, which can predict lengthy future trajectory reliably, can be trained from ground-up. The tasks experimented (Glucouse, BiGlucouse, CSTR, Cart Pole) also do not seem to have long horizon or high dimension. Thus, I have reservation whether RL-ACR can be used to reliably solve safe exploration problems in more sophisticated tasks.
3. Baselines used in Experiments
* The safe RL baselines used in experiments: SAC & CPO are neither model-based nor intended for safe exploration. SAC only learns to maximize reward and disregards safety. It is also unclear how SAC makes use of the model in SAC-pt. For CPO, although it is a safe RL algorithm, it is not intended to solve safe exploration problem. It is also not clear to me how CPO-pt exploits the model $\tilde{f}$ for this purpose.
* Perhaps one possible baseline could be the safe editor in [1], which also claims to solve safe exploration problem?

References  
[1] Yu, H., Xu, W. and Zhang, H., 2022. Towards safe reinforcement learning with a safety editor policy. Advances in Neural Information Processing Systems, 35, pp.2608-2621.  
[2] Nagabandi, A., Kahn, G., Fearing, R.S. and Levine, S., 2018, May. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 7559-7566). IEEE.  
[3] Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems. 2018.  
[4] Janner, M., Fu, J., Zhang, M. and Levine, S., 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.  
[5] Clavera, I., Fu, Y. and Abbeel, P., Model-Augmented Actor-Critic: Backpropagating through Paths. In International Conference on Learning Representations.  
[6] Thomas, G., Luo, Y. and Ma, T., 2021. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34, pp.13859-13869.
questions: questions: 1. (Related to Weakness 3) Do you think safe editor policy in [1] can be a good a baseline for comparison?
2. (Related to Weakness 3) Do you have results which evaluate your approach in longer horizon (and higher state-action dimension) setting where it is more challenging to predict future trajectory reliably?
3. In your experiments, how do SAC-pt and CPO-pt make use of the model $\tilde{f}$? To my understanding, they are both model-free algorithms and do not exploit any dynamic model.
limitations: limitations: 1. The paper discusses one limitation which is having reasonably accurate model for control regularizer. It is pointed out in [4] that even for a model which can predict the next step reliably, it is very difficult to predict future N steps accurately since the error cascade over the prediction timesteps. The paper could possibly discuss this in detail.
2. Another point about learning the model is that it should be able to predict transition to unsafe states in order to support the safety regularizer. This inevitably means that there needs to be unsafe transitions being collected to train the model. How does this model collection fit into the entire safe exploration framework is an interesting area to discuss.
paper_summary: paper_summary: In this article, the authors address the problem of safe reinforcement learning in single life setting. The agent must learn an optimal policy in one single episode without being unsafe throughout the learning. The proposed method relies on mixing a prior safe policy (the safety regularizer) with a reinforcement learning policy. The mixing is done through a linear combination of the actions proposed by the safety regularizer and the reinforcement learning agent. The weight put on either action is determined by a state dependent learned neural network. 

The safety regularizer is given by a model predictive control algorithm which assumes an estimated model of the environment is available. The RL agent follows the SAC algorithm.

The authors show that the beta parameter is equivalent to a regularization weight that forces the learned policy to be closed to the regularizer policy.

They evaluate their algorithm empirically on several simulated environments that have mathematical models. The proposed method is able to outperform the baselines in terms of both safety and return.

I acknowledge reading the authors rebuttal.
strength_weakness: strength_weakness: - It would have been useful to quantify the deviation of the current policy from the safety regularizer as a function of $\beta$. Since the method does not provide any guarantee on the safety level, it would be useful to quantify the amount of risk being taken by not following strictly the safe policy.

- In the evaluation of the method, it would have been useful to add another baseline that takes advantage of the existence of a safe policy E.g. SPIBB (https://arxiv.org/abs/1712.06924), or residual policy learning (https://arxiv.org/abs/1812.06298). This last one would be a great baseline since they have a similar policy mixing concept but they don’t use any weight beta. 
I acknowledge that the author have pre-trained SAC and CPO on the estimated model which is a good baseline as well.

- The framework is targeted towards “single life” settings, yet the training is episodic (figure 2). It makes it hard to conclude that the experiment are really evaluating the adaptability of the algorithm. 

- Some of the results lack discussion. In particular, the authors claim that RL-ACR has less variance in the returns, it is not very obvious from figure 2 in my opinion. The figure clearly show that SAC is less stable but it is not really discussed why. Since RL-ACR is using SAC at its core, I would expect SAC without the regularizer to perform at least as well in terms of returns (with the caveat of more failures). 

- The results shown in figure 3 are hard to evaluate as their lack some information about how the experiment is carried. Is it the performance in a single episode? Is it the same episode used for training? 

- In figure 5, the authors claim “$\beta_{\psi}(s)$ can apply different policy combination weights depending on how well the current state is exploited”. I agree that it is state dependent but there is no explicit term that takes into account the “exploitation” level of a state. The experiment is not showing that either I believe. $\beta$ is learned by maximizing the $Q$ function estimate. If a state has not been visited often its Q values might still be overestimated and lead to high $\beta$.

Presentation:

-	The title and name of the algorithm is a bit misleading, I thought that the method would use techniques from the field of adaptive control https://en.wikipedia.org/wiki/Adaptive_control,  there is no actual analogy in the paper.
questions: questions: -	Isn’t equation (10) equivalent to an actor loss in actor critic method? It could be useful to discuss a parallel here. 
-	When updating $\pi_\theta$, what would be the effect of using gradient ascent on equation (10) instead?
-	Why is the performance of SAC worse than ACR in terms of normalized return?
limitations: limitations: The method requires the existence of a safety regularizer. In my opinion it is a fairly reasonable limitations and the method can still be used in many problems.
The method does not provide any safety guarantee.
paper_summary: paper_summary: The paper proposed a safe RL algorithm using adaptive regularization from model predictive controller. The regularization is implemented via weighted sum of a MPC controller and model-free RL policies. The weights is adaptive and learnable through a “focus network”. The focus network is updated through optimizing the expected Q function of weighted sum of actions. Experimental results shows that the proposed RL algorithm can learn a safe policy with zero constraint violations and outperforms MPC on the performance.
strength_weakness: strength_weakness: 1. There is a key assumption that, the MPC policy $\pi_r$ via estimated model is safe, is too strict, not realistic and not mentioned explicitly. The authors claimed that the model is sub-optimal in line 31, which means the model is not accurate. The model mismatches usually cause constraint violations, but the experimental results in Table 1 showed no constraint violations for MPC policies. 
2. The computational cost is too high, which makes the proposed algorithm not generalizable to nonlinear/high-dimensional inputs/ unknown models. Solving a MPC is very computationally heavy. The authors didn’t mention what optimization package, how many CPUs they used for solving MPC problem (7). For nonlinear system, MPC are not guaranteed to be solved in polynomial time and converged to global optimal solutions. For system with visual inputs and unknown models, the methods are not applicable.
3. The focus network update might lead to unsafe policy.
questions: questions: 1. Can you clarify what types of references model $\tilde f$ you assumed for the regularizer policy $\pi_r$? See weakness 1 for the reason.
2. Why not compared to the Cheng et, al. as you already mentioned it? Now you only has one safe RL baseline CPO (and its variants). CPO is too old. 
3. If you already has the safe model, why not just do CBF-QP on your learned policy? What is the potential advantages of the proposed algorithm 
4. Please clarify the MPC solver and computational resources used for solving MPC.
5. For the focus network update, it seems like the focus network will finally choose $\beta(s) = 0$, which is the optimal policy. But the optimal policy might not be safe. What guarantees the safe policy is learned as you shown in Table 1?

[1] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387–3395, 2019a. 
[2] Ames A D, Coogan S, Egerstedt M, et al. Control barrier functions: Theory and applications[C]//2019 18th European control conference (ECC). IEEE, 2019: 3420-3431.
limitations: limitations: The authors addressed the limitation adequately.
paper_summary: paper_summary: The authors develop RL with Adaptive Control Regularization (RL-ACR) to enable safe RL exploration. This solution uses two agents: a safety regularizer to enforce safety constraints, and an adaptive agent to perform exploration. They demonstrate their method on four critical applications against leading benchmarks.
strength_weakness: strength_weakness: 1. Theorem 1 requires the assumption that the RL policy converges to the optimal policy. However, I do not see a result guaranteeing that the RL-ACR policy converges to the optimal. Instead, I see a monotonic performance improvement result in Theorem 2, but this is not an optimality result.

2. The work is intended to address safety, yet the authors do not provide a theoretical guarantee of closed-loop stability. Such guarantees are important for control of dynamical systems (e.g., airplanes, cars).
questions: questions: 1. If an optimality result is available, does it require that the estimated system model $\tilde{f}$ be exact to the actual physical process? I am thinking about this in light of your assumptions for Theorem 1. Presumably an imperfect model would void the convergence to the optimal policy and thereby the sufficient conditions required for Theorem 1. I do appreciate the empirical attention paid to model uncertainty in Section 4.2.

2. How do the compute/data requirements of RL-ACR compare to the baselines?
limitations: limitations: The authors adequately address limitations.

Comments: title: Thanks to the authors
One last clarifying remark in this discussion.
Thanks to the authors for clarifying these points. Retaining my scores.
Our responses and actions regarding the weaknesses pointed out by the reviewer (as well as the reviewer's concerns)
My thanks to the authors. Overall, I remain a positive review of this work. However, I am troubled by assertions made in the rebuttal, which does not convincingly address the raised weeknesses. Decrementing my score.
Thank you for the replies and for adding the RPL evaluation
comment: I thank the authors for their succinct and to-the-point rebuttal.  

The comparison with SEditor (Yu et al., 2022) is helpful as it demonstrated that RL-ACR is more suitable for environment with critical safety requirement where an established safe policy is helpful to ensure that. Due to the new results, I'm willing to increase my final rating.  

UPDATE: Final rating updated.
My thanks to the authors for this discussion and for these concluding points of clarification.
We sincerely thank the reviewer for the feedback and respect the decision; we have revised the manuscript to reflect the feedback as detailed in our previous response. Here, we add a few minor clarifications in response to the most recent comments for your consideration. For clarity, first, we would like to state that this manuscript deals with two performance aspects: i) the expected cumulative reward (i.e. the expected return) measured by Eq. (1) in the manuscript, ii) safety measured by the number of failures (defined as breaking the safety bounds). 

> Reviewer (Re W1): …your final converged policy can perform no better than SAC (a standard method). Why not bypass the extra complexity of your regularizer and choose to implement SAC directly then, if SAC already comes with a stability guarantee? 

**Response**: That is theoretically correct only in terms of “the cumulative reward”. However, our policy remains significantly safer (in terms of the number of failures) all the way from initiation of training to convergence when compared to the standard SAC. Bypassing the regularizer would harm the method’s performance substantially in terms of safety. 

> Reviewer (Re W1): [Authors] have simply offloaded the vagueness from "better" to "higher". This is still not quantitative/systematic. 

**Response**: “Higher expected return” (i.e., the expected cumulative reward) is quantified by Eq. (1) of the paper. This is what we referred to in our previous response and we hope it is clarified further now.  

> Reviewer (Re W2): I see the point made. But the final conclusion which must be made is that the method achieves: Performance no worse than the regularizer (MPC). Performance no better than the RL module (SAC). 

**Response**: The reviewer is correct with this conclusion; note that this is a theoretical result and for performance in terms of the “expected cumulative reward”. Practically, the proposed RL-ACR trains with safe actions (due to the effect of the regularizer) unlike the standard SAC, so the conclusion (more optimistically if you will) would be that the proposed method performs up to SAC in terms of “cumulative reward” and performs significantly better in terms of safety (measured by the number of failures). (Note that SAC achieves the state-of-the-art cumulative reward.)
We thank the reviewer for the response.

### Re Re to response to W1: 

We agree that the existence of a safe MPC is an important consideration for the application and we will highlight this in section 3.1 the Safety Regularizer (where we introduce the estimated mode). Although parameter uncertainty in our experiments reflects one of many possible reasons for the model uncertainty, it is commonly used in the literature to simulate model uncertainty that is encountered in applications; one example is [1]. 

### Re Re to response to W3: 

The reviewer is correct, and we acknowledge that the safety of the intermediate action is not formally guaranteed. However, we made an addition to the manuscript following the feedback from you and another reviewer (t4ae), which may be of interest for your assessment. We have analytically quantified that the upper bound on the return deviation of the combined policy from the regularizer (safe) policy is proportional to $(1-\beta)\Delta a$, assuming the environment is Lipschitz continuous. This provides some theoretical insight on the safety of the combined policy. 

### Re Re to response to Q1 and Q2, Q4: 

We are glad that these comments are addressed and appreciate the reviewer’s acknowledgement. 

### Re Re to response Q3: 

As the reviewer identified, theoretically, both MPC and CBF-QP rely on the existence of a “safe” model. However, the fact that RL-ACR improves performance through the MPC objective justifies the use of proposed method over CBF-QP. We hope that this clarifies our original response. 

### Re Re to response to Q5: 

We thank the reviewer for the valuable comment, and we will highlight this point in the paper. Also, we would like to point out, even reward designs where higher rewards correspond to preferable actions (in terms of safety) is common in the safe-critical applications (e.g., [2]). (In regulation tasks, for example, states closer to the target correspond to higher rewards and simultaneously states further away from the target are less safe states; thus increasing rewards corresponds to improving safety.) 

### Re “after reading reviewer zn6h’s concern about convergence…”: 

We would like to mention that our initial response to zn6h’s included a vague term and raised a concern about convergence, but we have clarified this soon after in a follow-up response and made minor revisions to the manuscript to prevent any confusion. 

$~$

[1] Cheng R, Orosz G, Murray RM, Burdick JW. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. InProceedings of the AAAI conference on artificial intelligence 2019 Jul 17 (Vol. 33, No. 01, pp. 3387-3395). 

[2] Laroche R, Trichelair P, Des Combes RT. Safe policy improvement with baseline bootstrapping. InInternational conference on machine learning 2019 May 24 (pp. 3652-3661). PMLR.
And I want to make it clear here: I recognize the novelty of **the idea of weighting two independent policies** as providing novel elements to this work (see my original review), rather than strictly basing my score on **the standard policies which have been combined** 

If you will, my score lies in a convex combination of these two points -- my own focus network of sorts. I remain at a weak accept and leave further deliberation to the ACs.
**Weakness 1.**

* Overall, I appreciate the clarification provided. Please include the appropriate citation.

> More specifically, our method uses the SAC algorithm in its RL module

* I am trying to follow the logic here:

1. The authors provide a proof that the regularizer $\beta$ chooses the RL Module policy if it exhibits improved performance.

2. The RL module is SAC.

So the end result here is that your converged policy is simply SAC? 

* We therefore can't avoid the following conclusion: **your final converged policy can perform no better than SAC (a standard method).**

* Why not bypass the extra complexity of your regularizer and choose to implement SAC directly then, if SAC already comes with a stability guarantee?

> We would like to add that by “a better policy” in our response, we meant a policy with a higher expected return [...]

* I understood the authors's intent, but I want to point out that they have simply offloaded the vagueness from **"better"** to **"higher"**. This is still not quantitative/systematic.

* With this said, I now indirectly have somewhat of a foothold to establish some quantitative guarantees from the method, which it inherits from SAC. But this runs into the issues I delineate in my above points.

**Weakness 2.**

> This indicates that the proposed model is guaranteed (proved in Theorem 2 for the tabular case) to perform at least as well as (again in terms of the expected return) the initial regularizer module.

* I see the point made. But the final conclusion which must be made is that the method achieves:

1. Performance no worse than the regularizer (MPC).

2. Performance no better than the RL module (SAC).

So the results deliverable sit between standard methods already available.
We thank the reviewer for the feedback. Our response was intended to render our agreement with the Weakness points identified by the reviewer whilst raising additional points that may aid the assessment of our work; we sincerely apologize if the response seemed dismissive in any form. Below, we provide clarifications for our responses (taking the reviewer’s concerns into account) as well as our action on the manuscript to address the received feedback. 

> **[Weakness 1]:** 

The reviewer correctly identified that: Theorem 1 requires the assumption that the RL policy converges to the optimal policy, and that, this policy converging to the optimal is not established anywhere in the manuscript. This assumption, however, is justified by an existing proof in [Haarnoja et al., 2018] and we will revise the manuscript appropriately to address this comment. 

More specifically, our method uses the SAC algorithm in its RL module, which has been proven to “converge to the optimum policy” before in [Haarnoja et al., 2018]. Our Theorem 1 takes this existing result (RL module convergence) as an assumption and provides more insight into the behavior of the combined policy. We will revise Theorem 1 in the manuscript, and in particular, we will add the following to the paper, directly referring to the existing proof of RL module convergence that we take as an assumption for further analysis. 

“[…] Haarnoja et al. [2018] prove (see Theorem 1 in the reference) that the learned RL policy with the Soft Actor-Crtic losses formulated in Eqs. (3) and (6), after repeated updates converges to a policy $\pi^\star$, such that no policy with a larger expected return than that of $\pi^\star$ can be found, i.e., $Q^{\pi^\star}(s_t, a_t) \geq Q^{\pi }(s_t, a_t)$ for all $\pi$ and $(s_t, a_t)$.” 

We would like to add that by “a better policy” in our response, we meant a policy with a higher expected return, and we extend our apology for using a vague/inaccurate term only to compress our response. We hope that the reviewer finds the updated response/action appropriate for the weakness pointed out. 

> **[Weakness 2]:** 

We agree with the reviewer’s both initial and the new comment on [Weakness 2]. Although we completely agree with its significance here, a theoretical closed-loop stability guarantee is very difficult to achieve where RL (since the agent is blind to the model) is involved. Therefore, we took the approach of demonstrating that the policy update will only improve or retain the expected return of the combined policy. This indicates that the proposed model is guaranteed (proved in Theorem 2 for the tabular case) to perform at least as well as (again in terms of the expected return) the initial regularizer module. We will clarify the above further in Theorem 2 and will state the weakness pointed out by the reviewer in the Conclusion section as a limitation of the approach. 

We have made an unintentional error in the rebuttal, which we understand sent an unclear message. We agree with the reviewer that model uncertainty and stability guarantees are not mutually exclusive, and our performance guarantee is simply approached from a policy improvement perspective rather than a closed-loop stability perspective.
We thank the reviewer for the constructive feedback and for acknowledging our addressal of the comments. Regarding the new question, the reviewer is right, and we clarify this in the discussion of Fig. 5 that “exploitation is not explicitly taken into account by $\beta_\psi(s)$ but more frequent states will have more effect on the value of beta, although the exact value of beta cannot be predicted”.
Thanks for the detailed rebuttal. 

### Re to response to W1: 

First, L335 is the limitation section, you mentioned here does not make the assumption more reasonable. For the results in figure 4, parameter uncertainty is only one of many possible reasons for the model uncertainty. A safe MPC controller for uncertain dynamics is generally very difficult.

### Re to response to W2. 

Thanks for the clarification on the calculation time. I won’t call it fast (you might need to consider the computational limit on the real hardware, not all real-time control application has a CPU as powerful as an intel i7.) but it’s okay.

### Re to response to W3: 

What I mean is that you have a linear combination of MPC actions and RL actions. These two actions might be all safe, but there is no guarantee that the intermediate actions are safe, which is the key weakness of this paper.

### Re to response to Q1 and Q2, Q4: 

my concerns are addressed. thanks.

### Re to response Q3: 

I agree with the performance improvement. However, you said,

> note that the theoretical safety guarantee of CBF-QP relies on the existence of a “perfect” model.

The proposed method also requires a perfectly safe MPC controller, which is no less strict than the CBF QP. Moreover, even with such a controller, you cannot guarantee intermediate safety theoretically.

### Re to response to Q5: 

>  In the problems, the policy that optimizes return is within the safe region by design. 

This is key information and should be highlighted in the manuscript. Then the proposed methods require a special reward design that the optimal policy is always safe, which is another limitation.

Overall, and after reading reviewer zn6h’s concern about convergence, I will not change my score. I also encourage the authors to try to implement it on a real system if you do believe this is a reasonable way to do real-world safe exploration. The results will be much more powerful.
My thanks to the authors for their responses to my posed weaknesses and questions.

The positive aspects of this paper I identified in the review remain unchanged.

However, the authors failed to adequately address my riased weaknesses. In fact, their response, which borders on cavalier at times, raises concerns I did not have previoulsy.

In light of these points, I am decrementing my score to a weak accept.

Below can be found my responses to your specific points. I have focused on the remarks which troubled me: 

**[Weakness 1]:**

> For state $s$ the regulation on RL policy converges to 0 ($\beta(s)$ converges to 0) with the assumption that RL learns a better policy than the regularizer. 

* I am confused by the point the authors are trying to make here. $\beta(s)$ converging to zero implies that the focus is choosing the RL module agent you are presenting to it -- it says nothing about the convergence of the RL module policy or its quality.

* **"Better"** here is vaguely defined and does not imply **convergence** at all.

* My weakness centers on the failure of the convergence of the **sequence of policies** (i.e., convergence with respect to the algorithm iteration of the training scheme) to the optimal policy, not on the focus network simply selecting between the safety regularizer and the RL module.

> [...] with the assumption that RL learns a better policy than the regularizer.

* This is not rigorous or substantial by any means. It's expected that your focus network should pick the RL module policy **if it's assumed to be a better policy.** I'm asking for **justification via provable results that the RL module converges and delivers improved performance.**

> [...] the assumption of RL convergence is reasonable and widely accepted in research

* This comment is very troubling. In other words, you mean to assert that every RL algorithm developed is universally accepted to converge?

* So then, **simply calling your algorithm an RL algorithm automatically gives it established convergence performance?**

* I don't see any other way to interpret this statement.

**[Weakness 2]:**

> a theoretical guarantee of closed-loop stability [...] would be model-dependent, and the analysis would be application-specific. 

* This is simply not true in control settings -- in fact, the vast bulk of standard stability results in the control literature are formulated for general systems rather than application-specific examples.

> We considered not knowing the exact model (a characteristic phenomenon in real-world application) and thus did not focus on finding a theoretical guarantee of closed-loop stability, 

* This seems to suggest that model uncertainty and stability guarantees are mutually exclusive, which is also not the case in the literature at all.

* In fact, there is a really important subtlety here: One of the core reasons for opting for RL frameworks is to address uncertainty in the environment. The authors seem to suggest that performance guarantees cannot be derived in the presence of model uncertainty -- exactly the core use-case of RL.
The replies to the questions were very clear. The experiment with RPL was useful to convince me of the advantage of the method. 

Regarding weakness 6, I understand that the more frequent states will have more effect on the value of beta as you will do more gradient updates that will have these states in the batch, but I don't think we can really get insights on what beta will be for those states. Did I miss something?