Real-DRL: Teach and Learn at Runtime
Authors: Yanbing Mao
Yihao Cai
Lui Sha
Venue: NeurIPS 2025 poster ()
Keywords: Safety
Dual Self-Learning and Teaching-to-Learn
Safety-Status-Dependent Batch Sampling
Autonomous Systems
Primary area: reinforcement_learning

Abstract: This paper introduces the Real-DRL framework for safety-critical autonomous systems, enabling runtime learning of a deep reinforcement learning (DRL) agent to develop safe and high-performance action policies in real plants while prioritizing safety. The Real-DRL consists of three interactive components: a DRL-Student, a PHY-Teacher, and a Trigger. The DRL-Student is a DRL agent that innovates in the dual self-learning and teaching-to-learn paradigm and the safety-status-dependent batch sampling. On the other hand, PHY-Teacher is a physics-model-based design of action policies that focuses solely on safety-critical functions. PHY-Teacher is novel in its real-time patch for two key missions: i) fostering the teaching-to-learn paradigm for DRL-Student and ii) backing up the safety of real plants. The Trigger manages the interaction between the DRL-Student and the PHY-Teacher. Powered by the three interactive components, the Real-DRL can effectively address safety challenges that arise from the unknown unknowns and the Sim2Real gap. Additionally, Real-DRL notably features i) assured safety, ii) automatic hierarchy learning (i.e., safety-first learning and then high-performance learning), and iii) safety-informed batch sampling to address the experience imbalance caused by corner cases. Experiments with a real quadruped robot, a quadruped robot in Nvidia Isaac Gym, and a cart-pole system, along with comparisons and ablation studies, demonstrate the Real-DRL's effectiveness and unique features.

Real-DRL: Teach and Learn at Runtime
Yanbing Mao' Engineering Technology Division
Yihao Cai
Department of Electrical and Computer Engineering
Wayne State University
Wayne State University
Detroit, MI 48202
Detroit, MI 48202 yihao. cai cei@@wayneecc. edu
hm90622wayne..em edu
Lui Sha
Siebel School of Computing and Data Science University of Illinois Urbana- Chammppaan
Urbana, II 61801 ldd
Abstract
This paper introduces the Real-DRL framework for safety-critical autonomous systems, enabling runtime learning of deep reinforcement learning (DRL) agent to develop safe and high- high-performance action policies in real plants (i.e., real physical systems to be controlled), while prioritizing safety! The Real-DRL consists of three interactive components: DRL-Student, a PHY-Teacher and a Trigger. The DRL-Student is DRL agent that innovates in the dual self-learning and teaching-to-leary paradigm and the real-time safety-informed batch sampling. On the other hand, PHY-Teacher is physics-model-basee design action policies that focuses solely on safety-critical functions. PHY-Teacher is novel in its real- time patch for two key missions: i) fostering the seccinng-toleaan paradigm for DRL-Student and ii) backing up the safety of real plants. The Trigger manages the interaction between the DRL-Student and the PHY-Teacher. Powered by the three interactive components, the Real-DRL that arise from the unknown unknowns and the Sim2Real gap. Additionally, Real-DRL notably features i) assured safety, ii) automatic hierarchy learning (i.e., safety-first learning and then high-performance learning), and iii) safety-informed batch sampling address the learning experience imbalance caused by corner cases. Experiments with a real quadruped robot, quadruped robot in NVIDIA Isaac Gym, and cart-pole system, along with comparisons and ablation studies, demonstrate the Real-DRL's effectiveness and unique features.
effectively address safety challenges
Introduction
Deep reinforcement learning (DRL) has been incorporated into many autonomous systems and has shown significant advancements in making sequential and complex decisions in various fields, such as autonomous driving [32 33 and robot locomotion [28 34
However, the incident database has revealed that machine learning (ML) techniques, including DRI., can achieve remarkable performance
but without safety guarantee [60]. Therefore, ensuring high-performance DRL. with verifiable safety more crucial than ever, aligning with the growing market demand for safe ML techniques.
"Equal contribution.
AAIID: https///incieenttaaabaeead
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.1 Safety Challenges
Two significant challenges emerge when developing segh-performncce DRL with verifiable safety.
Unknown Unknowns. The dynamics of learning-enablle autonomous systems governed by conjunctive known known (e.g., Newtonian physics), known unknowns (e.g.. Gaussian noise without identified mean and variance), and unknown unknowns. One significant example of an unknown unknown arises from DNN's colossal parameter space, intractable activation, and random factors. Moreover, many autonomous systems, such as quadruped robots [9] and autonomous vehicles [44), have -ystem-eniironmeetss interaction dynamics. Unpredictable and complex environments, such as freezing rain [51], can introduce a new type of unknown unknowns. Safety assurance also requires the resilience unknown unknowns since they are hard-to-simulate and hard-to- to-peedict, and often lack sufficient historical data for scientific modeling and analysis.
Challenge 1:
Challenge 2: Sim2Real Gap. The prevalent DRL strategies involve training policy within
simulator and deploying it to a physical platform. However, the difference between the simulated environment and the real world creates the Sim2Real gap. 'his gap causes significant drop in performance when using pre-trained DRL in a novel real environment. Numerous approaches have
been developed to address the Sim2Real gap 40] 38 52
20 54 56). These methods
aim to improve the realism of the simulator and can mitigate the Sim2Real gap to varying degrees. However, undisclosed gaps continue hinder the safety assurance of real plants.
1.2 Related Work
Significant efforts have been devoted to addressing safety concerns related to DRL, detailed below.
Safe DRL. Since the reward guides DRL's learning of action policies, one research focus of safe DRL the safety-embedded reward to develop a high- performnrroem action policy with verifiable safety. The control Lyapunov function (CLF) proposed in 42 5 15 62 is candidate. Meanwhile, the seminal work in [55] revealed that a CLF-like reward could enable DRL with verifiable stability. Enabling verifiable safety is achievable by extending CLF-like rewards with given safety conditions. However, the systematic guidance for constructing such CLF-like rewards remains open. The residual action policy is another shift in safe DRL, which integrates data-driven action policy and physics-model- based action policy. The existing residual diagrams focus on stability guarantee 45
[8] 30) with the exception being (17] on safety guarantee. However, the physics models considered are nonlinear and intractable, which thwarts delivering a verifiable safety guarantce, if not impossible. The recent Phy-DRL (physicsreguulated DRL) framework [13] [2] can satisfactorily address the open problems. exhibits verifiable safety, but it is only theoretically possible due to the underlying assumption of the manageable Sim2Real gap and unknown unknowns.
35
Fault-Toleran DRL. It is another direction for enhancing safety assurance of DRL in real plants. Recent approaches include neural Simplex [43] runtime assurance 8 (6]16]]]1 and runtime learning machine 100 I1 for deterministic safety definition, and model predictive shielding 4 for proba- bilistic safety definition. They treat the DRL. agent as nigh-performance but black-box module that operates in parallel with high-assurance module. The high-assurance module is a physics-model- based controller, assumed to have assured safety to tolerate the DRL faults. However, this assumption becomes problematic in situations where there are significant model mismatches, particularly due to unknown unknowns such as dynamic and unpredictable operating environments.
1.3 Contribution: Real-DRL Framework: From Theory to Implementation
To address Challenges and 2, we propose the Real-DRL framework, which enables the runtime learning of DRL agent to develop safe and high-performance action policies for safety-critical autonomous systems in real physical environments, while prioritizing safety. As illustrated in Figure Real-DRL, building on Simplex principles 49 has three interactive components:
DRI--Student is a DRL agent that can learn from scratch. It is innovative in the dual self-learning and teaching- teaching-to-lears to-lra paradigm and the safety-informed batch sampling to efficiently develop safe and high-performance action policy in real plants.
design with assured safety, focusing on safety- critical functions only. It aims fostering the teaching-to- -toleernttt mechanism for DRL-Student and backing up the safety of real plants, in the face of Challenges and 2.
PHY-Teacher is a physics-model-base

---

Safedy
Rapley
DRI Stndent
Treching
Algorithm
SQLL Tesching tb lo-kaam space, marg ofgh
PHY-Tracher Theorem 5.2
Citie
- acreie
L: Self loaming spaco
Prescher
Randen
Mision 2: Bock Up Sukty
Practting
Rapley Buller SefiLeariig
Swiety
Trierse
(L Marghalksstee boundaried
Eq.
Teaching Safeey
sasic sik)
karainal action at&)
Hates(k)
Safety Critical Enviroament: Real
Exumples:
Figure 1: Real-DRL framework, which is also formally described in Algorithm of Appendix B
Trigger is responsible for monitoring the real-time safety status of the real plant, facilitating interactions between DRL-Student and PHY-Teacher.
Powered by the three interactive components, the Real-DRL notably features assured safety in the face of Challenges I and 2, ii) automatic hierarchy learning (i.e., safety-first learning and then high- performance learning), and iii) safety-informe batch sampling to address the experience imbalance caused by corner cases.
We present the preliminaries Section 2 and elaborate on the Real-DRL design in Sections The experimental results and implementation details of Real-DRL are provided in Sections 6 and
Preliminaries
2.1 Notation
In this paper, we use P > 0 to represent matrix P is positive definite. R" is the set of n-dimennional real vectors, while N the set of natural numbers. The superscript 'T' indicates matrix transposition. We define On, and 1m as the m--dimensional vectors with entries of all zeros and ones, respectively, and Im as the m-dimensional identity matrix. The e O, means all entries of c R" are strictly positive. Lastly, the diag(c) c} is diagonal matrix whose diagonal entries are vector c's elements.
2.2 Definitions
The dynamics of safety-critcall autonomous systems can be described by the following equation:
(1) where s(t) I=" represents the real-time state of the system sampled by sensors at time t, while a(t) E R" signifies the real-time action command at time t. The equilibrium point of the system O,, For a safety-critical system ( there exist the sets that system states and actions must always adhere to:
s(t 1) f(s(t), te N
Safety set: A {s R - < C C < with c > Opp,
Admissible action set: A E RM
< a-ddd, with d > 0,)
(3)
where C and are provided in advance to describe p E N safety conditions, while D and d are used describe q E N conditions on admissible action set. The inequalities presented Equation ( are sufficiently generic to describe many safety conditions. Examples include yaw control for quadruped robots to prevent collisions (22. and lane keeping and slip regulation in autonomous vehicles to follow traffic regulation and avoid slipping and skidding [44 37].

---

In our Real-DRL, the DRL-Student will engage in dual self-learning and teaching- to-learn paradigm to develop safe and high-performance action policy. To better describe this aspect, we introduce:
(4) Remark 2.1 (Buffer Zone). When system states arrive at a safety boundary of safety set S, the action policy of PHY-Teacher usually cannot immediately drive them back to the safety set S without delay. These delays arise from multiple factors, such as computation time, communication latency, sampling periods, and actuator response times. This consideration motivates the self-learning space L. (4). Specifically, the condition 0 < 1 < 1 in conjunction with Equation (2) and Equation (4) indicates that IL C S, which creates a non-emply buffer zone, i.e., S \ II.. The Trigger will reference boundary OL (not US) to trigger PHY-Teacher for assuring safety, providing adequate margins for response time and decision latency.
Self-learnngg space: IL - {s E R- l1 < C. <1 C, with < <1 <イ11) 1).
Building on the foundation we introduce the concept of the safe action policy of DRL-Student: Definition 2.2 (Safe Action Policy of DRL-Student). Consider the self-learning space LL 44, the admissible action & (3). and the real plant ( The action policy of the DRL-Student, denoted as Tstudena (.) R + R",, is considered safe if. under its control, the system states satisfy the condition: Given (11 LL. the s(t) E L and the astuuent (t) sudeet (s(t) E & hold for all time t N.
In our Real-DRL framework, PHY-Teacher's action policy is designed have assured safety only. We introduce the concept of "assured safety" below, which will guide the design of PHY-Teacher.
Definition 2.3 (Safe Action Policy of PHY-Teacher). Consider the safety set S the action set A. 3 the self-learning space TL A the real plant [ and denote:
k Tk], where Th E N. The action policy of PHY-Teacher, denoted as (. IR" + RR,, is said to have assured safety, if s(k) S LL, then i) the s(t) E S holds for all time t E Tg, ii) the Aseacher (1) Twrcher ((()) E A holds for all time / E T,, and iii) the s(k + Tx) E LL holds.
Activation Period of PHY-Teacher: Tn A (k+1, k + 2,
(5)
By Definition 2.3] the PHY-Teacher is deemed trustworthy for guiding the DRL-Student in safety learning and backing up the safety of real plant, only if its action policy simultaneously satisfies the corresponding conditions: i) it must ensure the real plant's ' real-time states consistently adhere to the safety conditions within S; ii) must keep its real-time actions within admissible action set A; and iii) it must ensure the real plant's 's states return to the self-learming space L as activation period ends.
With the definitions established, we next detail the design of Real-DRL's three interactive components respectively.
in Sections 3 to 5
3 Real-DRL: Trigger Component
Real-DRL's Trigger facilitates interactions between the DRL-Student and PHY-Teacher by monitoring the triggering condition outlined below.
Triggering condition of PHY-Teacher: s(k: - 11) E L and s(k) E Öl,
(6)
where ØL denotes the boundaries of the self- space LL 4 The condition (6). in conjunction
with the Tx defined in (5). can describe the conditional activation periods of PHY- TeaThee and DRL-Student as follows:
PHYYTeacher is active for time E Tk, if condition
6 holds at time k,
(7)
DRL-Student is active for time & Tk, otherwise.
Furthermore, the logic for managing actions applied to the real plant is follows:
Ateacher (l), if condition ( holds at time & and E Ta.
a(t (.studen(()) (t), otherwise,
(8)
where a(t) represents the terminal real-time action applied to the real plant, illustrated in Figure
The interactions between the DRL-Student and PeY-Teacher, as facilitated by the Trigger, can be summarized from Equations (6) to 8 as follows: When the states of the plant controlled by the DRL-Student approach marginal-safety boundary (i.e., they are leaving the safe self-learning space L), the Trigger prompts the PHY-Teacher to instruct the DRL-Student on safety learning and back up the safety of real plant. Once system states return to the (safe) self-learning space L., the DRL-Student regains the control of real plant, Next, we detail the design of DRL-Student and PHY Teache..

---

4 Real-DRL: DRL-Student Component
As shown in Figure | DRL-Student adopts an actor-critic architecture [36 27 with dual replay buffers. DRL-Student operates within a dual self-learning and teaching-to-learm paradigm to develop safe and nigh-performance action policies in real plants. Specifically, it learns safety from the PHY-Teacher's s experience of backing up safety acquired in the teaching-to-lcarr space, denoted as I. At the same time, learns achieve high task mosk-pefoomance from its own (purely data-driven) experience in the seaf-neg space, represented as L. Meanwhile, DRL-Student employs our proposed defett-inffrmed batch sampling effectively 1) address the experience imbalance (caused by corner cases) throughout learning, and 2) promote the dual self-learning and teaching-to-learn paradigm. Next, we present the novel design of DRI-SStudent.
4.1 Self-Learning Mechanism
The self-learning objective is to optimize an action policy Astudent (k) Tstudent (s(k)) which maxi- mizes the expected return from the initial state distribution:
Q Fstssent (s(k) Astudent (1:) = (X)
Σ
1 R (s(t), agudem (t), s(t))(()))))))))))))))))))))))
(9)
where is the self-learning space defined in Equation (4). R(,, -) is the reward function that maps the state-action-next-ttale triples to real-value rewards, and E [0, 1] is the discount factor. The expected return (9) and action policy (-tuden(-)) are parameterized by the critic and actor networks, respectively. The reward function is designed to enhance task- -riented performance through runtime learning. One such example minimizing travel time and power consumption in the robot navigation.
Referring to Figure we can summarize that during the activation period of DRL-Student, its interactions with real plants generate the purely data-driven experience data as
Self-learnigg experience:
s(t)
astuden (t) s(t+1) | R(s(t), astudem (t), s(t+1]]
(10)
which is stored in the self-learning replay buffer, and then batch sampled to update the critic and actor networks using the off- off- policy algorithm, such deep deterministic policy gradient (DDPG) (36).
twin delayed DDPG [23] or soft actor-critic (27). We thus conclude that the self-learning mechanism
is the learning from data-driven experience through the trial-and-error in self-learning space L. (4)
This mechanism is also formally described by Lines 25 to 29 in Algorithm Tf Appendix
4.2 Teaching-to-LLearn Mechanism
The designed PHY-Teacher has successful Sahysics--odel-bassee experience - not grounded in backing up the safety of real plants. As shown in Figure T this data-driven trial-and-error teaching-to-lear (or marginally-safe space $ LT, where the DRL- experience is acquired in the Student is deactivated due to safety violations that occur as a result of its actions' failure in assuring safety. If PHY-Teacher shares its successful experiences in backing up the safety of real plants, then DRL-Student can learn about safety from these experiences. This process is known as the teaching- le-learn mechanism. Referring Figure the Raching-to-lean mechanism can be summarized as that during the activation period PHY Teacher, it gains its experience data in backing up safety as
Teaching-to-lest le-lann experience: st))
Aqeacher (t) s(t+1) | R(s(t), (geacher(t) s(t+1)) (11)
where aqacher (t) is computed according to Pea-heesher's action policy (18) and its design is detailed
in Theorem 5.2] The experience data (11) is stored in the reaching-to-Lest replay buffer, and is then
batch sampled to update the critic and actor networks using the off-policy algorithm. This teaching-
to-learn mechanism is also formally described by Lines T2 to 16 in Algorithm Tf Appendix B
4.3 Safety-Informed Batch Sampling
This section proposes a batch sampling approach to facilitate the dual self-learning and teaching-to. learn paradigm. The benefits of the proposed approach go beyond achieving this paradigm. Before proceeding, we recall a real problem that the incidents of learning-enablle
autonomous systems, such as self-driving cars, often occur in corner cases [3 6) As illustrated in Figure [ the marginally- safe (or coohiiiggt--leamm space can represent the space of corner cases, where a slight disturbance
5

---

or fault can take system out of control. Such marginal samples are safety-critical and are not often and hard visited normally, creating an experience imbalance for learning. Intuitively, focusing the learning on such comer--cae experience will enable a more robust action policy. However, the current 47 which cannot guarantee
DRL frameworks usually adopt random sampling 45 sufficient sampling of such samples, and thus cannot overcome the experience imbalance problem. To address the problem, we propose the safety-informed batch sampling for DRL-Student's learning, which also facilitates the dual self-learning and saching-to-lean paradigm. To explain the sampling mechanism, we first introduce:
35 18
Safety-status indicator: V(s) S S.
(12)
where P is computed according to the following:
P = arg min (log det (P)), subject to (C .0.diag" (c]) p-1. dia-dag-- > 0, (13)
P=0
with C and с given in Equation 2 for defining the safety set S. We can derive the following property
related to safety- status indicator, whose proof is given in Appendix C
Theorem 4.1. Consider the safety set S and the function V(s) (22 with its matrix P computed
in Equation 13 The ellipsoid set {s E R V(s) < 1] S S and it has the maximum volume.
The proposed mechanism of safety-informed informed batch sampling is based on Theorem and the safety- status indicator in Equation [2).. As shown in Figure [ the sampling mechanism is to generate the batch samples to update critic and actor networks, which, thus, can be summarized
min {T., [T. (p1 Vss(t) + p2)l} + L min {T:, [T. (p1 V(s(t) + p2)1 (14)
Batch size
Number of samples from teaching. to learn reply huffer
Number of samples from self learving reply huffer
where P1
O and P2 are given hyperparameters.
An illustration example of the proposed batch sampling is provided in Figure 2 Along with Theorem 4.1 and Equation (14 reveals two key insights of DRL-Studentss s learning:
Samples
Teechingytn-een ecperience somple (Comer Casel
Safety-Status Indicator
(15) 1.0
M5) 0.8
The batch comprises experience sam- ples from the soaching-to-lean and self-learning replay buffers, with their contribution ratios depending on real- time safety status due V(s(()). Consequently, the DRL-Student par- ticipates in the dual self-learning and teaching-to Baaching-to-leary le-rn paradigm, discussed in Sections 4.1 and 4.2
(15) 0.5
Safety Set S
Settiaming exporiance sample
Figure 2: An illustration example of safety- informed batch sampling, given batch size L = 5, P1 1, and p2 O.
The contribution ratio of teaching-to-
learn (also, corner-case) experience samples is increasing with respect to the real-time safety status ((s(t),, thereby addressing the experience imbalance. For example, by Theorem 4.1 and illustrated in Figure an extreme value of V(s(t)) suggests that the real-time states are approaching safety boundaries. Consequently, according to Equation (14), all batch samples are from the marginally-sate space. This enables the DRL-Student concentrate on learning from corner-case experience samples (stored in teaching-to-learn replay buffer) about safety, when the real plant only marginally safe.
5 Real-DRL: PHY-Teacher Component
One aim of PHY-Teacher is tolerate real-time unknown unknowns (which result Sim2Real gap) to assure safety. We note many autonomous systems, such as quadruped robots [19 and autonomous vehicles have system-environments interaction dynamics. So, the real-time unknown unknowns (e.g., unpredictable weather conditions and complex wild terrains) can have significant influence on system dynamics. To tolerate such unknowns, we develop the real-time patch as for Teache..her. Its design is detailed below.
PHY-Teacher is a prssics-model--aee design to have assured safety by Definition 2.3

---

The proposed real-time time patch first defines the state space in which PHY-Teacher's action policy has assured safety according to Definition 2.3
Real-time patch: P& _ {s R2 - C.c.c.ccc sf.) < c, with < 0 < 1). (15)
where st is the patch center, and also PHY-Teacher's real-time control goal, which is defined below.
s x x-sk)) sk)), where s(k) satisfies the triggering condition o and < X 1.
(16)
With the real-time control goal at hand, we introduce the following dynamics model of tracking errors, which is transformed from the dynamics model described in Equation (
e(1 + 1) (s())) e(() + B(s(k)) aleacher + h(e(()), for / E Tx:
(17)
where T& (as defined in Equation 5 indicates the PHY-Teacher's activation period, e(t) A s(t) - st. and (()( represents an unknown model mismatch, arising from unknown model uncertainties, parameter uncertainties, unmodeled dynamics, and external disturbances that are not captured by the nominal model knowledge (s))),,, (s))).). The (A(s(k)), (ss())) will be utilized for
designing PHY-Teacher's action policy:
ageacher (t) Fx e(t) = F2 (s(()) si)), for t E Tk.
(18)
We can summarize the working mechanism of PHY- Teacher from Equations 15 to (8)) When the PHY-Teacher is triggered by the condition (6) at time k. it uses the most recent sensor data s(k) to
update the poysics-mode knowledge {A (s))) ) (ss)))) This update is then utilized to compute the real-time patch and the associated action policy. Meanwhile, the patch center (T6) lies within DRL-Student's ' self-learning space L (since of 0 < < 1), and also represents the real-time control goal. This arrangement defines the teaching task for the PHY-Teacher: to instruct the DRL-Student in learning a safe action policy that can return the system to space L. Before presenting the overall design, outline practical assumption regarding the model mismatch.
Assumption 5.1. The model mismatch h(e) in Equation 17) is bounded in the patch P& (1) In other words, there exists K > 0 such that h (e) Pk h(e) S k holds for Ps > 0 and any e E PR.
We present the design of PHY-Teachar in the theorem below; its proof can be found in Appendix D
Theorem 5.2 (Real-Time Patch). Recall the sets defined in Equations (2) to (4). and consider the Peachee'her's action policy in Equation (8). whose F& is computed according to
F5 R QF
and
P& =
(19)
with R and Q2 sattssfyin the following linear matrix inequalities (LMIs):
I Q C 0, and I, - -D. D > 0, and Qs n dddaga (s(k)) > 0, and (20)
a Q&
Q Qc-AT(s(k))) (s())) R -B (s({))
Q2 R R
A (s(k)) Qa B(skk))) R
Q I+
~0, and
Te
> 0, (21)
where C 2 C- diag -1 10 c). D diag (d), 0 < a < 1, and & > O. Meanwhile, the given
parameters 1 in Equation 4.. 0 in Equation (15). X in Equation (66.. a and ø in Equation 21 and
in Assumption 5.1 satisfy
< x<0+x-1-- and (0+1) (1- (1-x)2.1
(22)
(1-a) -a)a..
0
Then, according to Definition 2.3 the PHY-Teacher's 's action policy real-time putch PR..
18 has assured safety in the
Remark 5.3 (Policy Computation). By using the CVX (26) or LMI 24 toolbox, the matrices
Qj and R2 can be computed from Equations (20) and (1) for the matrix Fx. The pseudocode for computing F, & using CVX and LMI, with a comment, is presented Appendix
Remark 5.4 (Adaptive Mechanism for Assuring Validity of Assumption 5.1). PHY- CaY-Teacher is designed to have verifiable expertise in safety protocols, as formally stated in Theorem 5.2) but this relies on Assumption 5.7 In real-world applications, particularly in hard-to-predic and hard-to- simulate environments (e.g.. natural disasters and snow squalls), we cannot always guarantee that will Assumption will hold. To address this concern, we introduce an adaptive mechanism, presented in Appendix F that utilizes real-time state to estimate real-time model mismatch, thereby enabling monitoring of Assumption [. validity. When the assumption is violated, the mechanism computes compensatory actions real-fime to mitigate the effects of model mismatch and restore the validity of Assumption 5.1 timely.
7

---

Implementation Best Practices: Minimizing Runtime Overhead
The PHY-Teacher and DRL-Sttdent are designed operate in parallel. The DRL-Student uti- lizes GPU acceleration and can be deployed on embedded hardware such as the NVIDIA Jetson series, while the PHY-Teacher efficiently handles time safety-critical tasks on the CPU. To minimize runtime overhead, the LMI solver responsible for automatic and rapid computation of the PHY-Teacher's real-time action policies - should be implemented improve both memory efficiency and computational speed. To facilitate this implementation, we provide ECVXCONE (embedded CVX for cone programming), an online solver for embedded conic optimizztion capable of solving LMIs in real-time using C. The code is available at GitHub: https://github.com wwtp:/ittuuucoooocoooooooooo httpllooooooo/ccceceeeeee Charlescail http://cccai////ooooeoooooo 233ecvxcone.
to significantly
Table 2 in Appendix G presents an experimental evaluation of ECVXCONE's computational resource usage across four platforms with varying CPU frequencies, which demonstrates that ECVXCONE makes the Real-DRL well-suited for edge-Al devices with limited computational resources.
7
Experiment
The section presents experiments on a real quadruped robot, quadruped robot in the NVIDIA Isaac Gym, and cart-pole system, along with comprehensive comparisons and ablation studies. The complete code and details can be found on GitHub: https//ggihuub.oo//haaaaeee 223/Real
7.1 Real Quadruped Robot: Indoor Environment
The detailed design of this experiment is presented in Appendix During pre-training in the simulator, we set Tu= = 0.6 m/s and T5 = 0.24 m. To better demonstrate Real-DRL. the real robot's velocity command is Tu_ 0.35 m/s, which is very different from the one in the simulator. For the runtime learning in the real robot, one episode is defined as "rumning robol for 15 seconds."
We compare our Real-DRL. with the state-of-the- art safe DRL, called Phy-DRL, which claims theoretically and experimentally fast training to-
0.8
Phy-DRL
Continual Phy-DRL
wards safety guarantee 13 12). Mcanwhile,
Our Real-DRL
Phy-DRL integrates the concurrent delay ran- domization [29] and domain randomization [46]
0.6
CoM V-elocity
(randomizing friction force) for addressing the Sim2Real gap into its training. Here, we have 'Continual
0.4
three models for comparison: Phy-DRL representing a well-trained Phy-DRL. in the simulator, which performed continual learning in the real robot to fine-tune the ac- tion policy: 2) 'Phy-DRL' representing the well- trained Phy-DRL policy directly deployed to the real robot, and 3) our Real-DRL. The robot's CoM height and CoM-x velocity are shown in Figure and a comparison video is available at http::/www.youtulee www.youuuee.ooo/wwatet wwwp//tcc/////////eooooooooo SaaecchvvnSSPPKKEEAXe
0.2
0.10
0.20
0.25 CoM Height
0.30
0.35
0.40
0.45
Figure 3: Phase plots.
From the comparison video and Figure 3 we concluded that Phy-DRL well-trained in simulator cannot guarantee safety of the real robot due to the Sim2Real gap and unknown unknowns that the delay randomization and force randomization failed to capture. In contrast, our Real-DRL can always guarantee the safety of the real robot. Addi- tionally, recall that PHY-Tcacher fosters DRL-Student's 's tcaching-to-lle -o-ar about safety. To verify this, we deactivate the PHY-Teacher in episodes and 20, and compare system behavior. The demon- stration video is available at https://wwww, wwutube.uubbbcoooo http///oooro/w/Mrrrrt
which illustrates that DRL-Student quickly learned from PeY-Teacher to be safe, within 20 episodes, i.e., 300 seconds.
Finally, we showcase the Real-DRL's ability to tolerate various unknown unknowns. In addition to inherent ones in noisy samplings and complex DRL agents, we add five unknown unknowns that have never occurred in DRL-Studentt historical learning. They are 1) Beta: Disturbances injected into DRI--Stdent's actions, generated by a randomized Beta distribution (see Appendix [H] for an explanation of its representation of unknown unknowns); 2) PD: Random and sudden payload

---

(around 4 lbs) drops on the robot's back; 3) Kick: Random kick by human; 4) DoS: A real denial- of-service fault of the platform, which can be caused by task scheduling latency, communication delay, communication block, etc., but is unknown to us, and 5) SP: A sudden side push. We consider three combinations of them: 'Beta PD, ii) 'Beta DoS + Kick,' and iii) 'Beta SP. The demonstration video is available at https://www: www.yyu.uuuuoooooo wwt..coo///ssssseeeee wwmm//hotts/Tiii 22jtts////jjEJMMe demonstrates that our Real-DRL successfully assures the safety of the real robot by tolerating such complex combined unknowns.
which
7.2 Quadruped Robot in NVIDIA Isaac Gym: Wild Environment (see Figure 8
The operating environment plays a crucial role in introducing real-time unknown unknowns and Sim2Real gap. So, this benchmark focuses on comprehensive comparisons in complex environments. We utilized NVIDIA Isaac Gym to create a wild environment, assuming a real one. This environment transitions from flat terrain to unstructured and uneven ground, further complicated by unex- pected ice that results in low friction. We here can conclude that the robot's 's operating environ- ment are non statsotti and unforeseen. Be- sides, our Real-DRL. allows for learning from scratch, which is performed here. Meanwhile, we integrate all comparison models with nav-
15000
Episade Retum
5000
igation module, which enable the robot to au- tonomously traverse the uneven and unstruc- tured terrains, and dynamic obstacles in the wild. The architecture of Real-DRL with inte- gration of navigation shown in Figure 9 The experiment design appears in Appendix!!
Real-DRL
50000
Neural Simplex
CLF-DRL
Assurance
2000 2500
3500 4000
Episode Numbers
Figure 4: Trajectories of episode return.
Table 1: Safety metrics are highlighted in red. If the robot falls, other task-related metrics are labeled
as N/A. If collision is not avoided, travel time and energy consumption are represented as oc.
Navigation Performance
Energy Efficiency
Model ID
Success
Fall
Collision
Num (wp)
Travel Time (s)
Avg Power (W)
Total Energy (J)
CLF-DRL
No
Yes
No
0
NIA
NIA
N/A
Phy-DRL
No
No
Yes
1
0c
507.9441
0c
Runtime Assurance
No
Yes
No
2
N/A
N/A
Neural Simplex
No
No
Yes
2
487.9316
PHY Teacher
Yes
No
No
4
55.5327
482.8468
26817.68
Our Real-DRL
Yes
No
No
4
45.3383
479.4638
21742.42
The experiment compares our Real-DRL with state-of-the-ard frameworks: i) safe DRLs: CLF-DRL and Phy-DRL [13 12:: ii) fault-tolerant DRLs: neural Simplex [3] and runtime assurance
55
iii our sole PHY-Teacher. The testing models are chosen after 4500 episodes. Their performance comparisons summarized in Table [ Figure pressents the episode returns, and the http:/oouutube.coo/sootttttto SHjhqqvvg w/ PHY-Teacher during early stage
g
50 [6]
corresponding comparison video available at Additionally, comparison video of Real-DRL w/o wtt.s/ootoooccccceeeeeee wwwpsshott//xx//ww/EOOOO0 These of runtime learning is available at https:// http:://wwwwwwtt www.yyyuuuubcc... woputube.com/hoooeee comprehensive comparisons showcase Real-DRL's notable features: its high-performance learning and assured safety.
http:///wwwyouubb....mm
7.3 Cart-Pole System: Ablation Studies
teaching-to-learn mechanism focused safety, ii) safety-informed batch sampling to overcome the experience imbalance, and iii) an automatic hierarchy learning, i.e., safety-first learning and then higf-performance learning. In this benchmark, we primarily conduct ablation studies to demonstrate the impact of the three novel designs. The
Our Real-DRL has three learning innovations:
system's ss state consists of pendulum angle 0, cart position &, velocities = Ú, and v ã. The detailed experiment design can be found in Appendix K

---

Inferty- Batch Sampling: To demon-
1.5
strate the effect of this batch sampling scheme
1.0
on addressing experience imbalance, we set up an imbalanced learning environment, shown in
0.5
Figure [O We evaluate two DRL-Students in this
0.0
environment: one learned in Real-DRL with our
0.5
proposed batch sampling scheme (denoted as 'w/
-1.0
5BS-4
SBS"), and the other one learned in Real-DRL us-
-1.5
58S-2
w/ SBS-1
w/ SBS-5
ing a single replay buffer without this scheme (de-
-2.0
w/o SBS
w/ SBS
Initial
noted as 'w/o SBS'). The phase plots in Figure 5 and the rewards in Figure II iillustrate that, after reaching convergence, the DRL-Student learned in Real-DRL. with our batch sampling scheme has significantly enhanced safety assurance corner- case environment, compared to the learned from Real-DRL without this scheme.
2.5
w/o SBS-4
w/ SBS-3
Terminal
2
1
Figure 5: Phase plots, given five random initial states within self-learning space.
Teaching-to-Lest Mechanism: To evaluate the effect this learning mechanism on policy learning, we compare two models: Real-DRL w/ and w/o the teaching-to-learn. The episode- average rewards (defined in Equation (84) across five random initial states are shown in Figure 6 (a), demonstrating that the teaching-tes to-gg-ttolearnn mechanism significantly improves sample efficiency, enabling faster convergence, more stable learning. and higher reward performance.
Finally, we showcase the automatic hierarchy learning. To do so, we disable PHY-Teacher in episodes and 50 and observe system behavior under the control of sole DRL-Student. Intuitively, DRL-Student shall automatically become independent of PHY-Teacher and self-learn for high-performance action policy. This can be seen from PHY-Teacher's s activation ratio is rarely activated. Meanwhile, Figure 12 of Appendix K illustrrass the high mission performance of action policy in episode 50: much more clustering and closer proximity to the mission goal.
Automatic Hierarchy Learning:
(defined in Equation (85)) )) in F where after episode 50,
Activation Ratio of Pea-Teacher
0.8
Without Teaching-to-Lean With Teaching Sotthinngtt--Lannme
Episode-Awes Average Reward
0.7
0.6
0.4
-5
0.3
0.2
-10
0.1
-15
0.0
20
40
60
80
100
40
60
80
100
Episode Numbers
Episode Steps
(a) Trajectories episode-average reward (95% CT).
(b) PHY-Teacher's activation ratio over 100 episodes.
Figure 6: Ablation studies. (a): Teaching-to-learn mechanism, (b): Automatic hierarchy learning.
Conclusion and Discussion
This paper introduces Real-DRL, a framework specifically designed for safety-critical autonomous systems. The Real-DRL consists of three main components: a DRL-Student, a PHY TeaTTee,her, and a Trigger. The DRL-Student engages in a dual self-learning and teaching-tor -ooiingtt--eaamn paradigm. The PHY- Teacher serves two primary functions: fostering the teaching-to-Icet to-latro-arrn mechanism for DRL-Student and backing up safety. Real-DRL notably features i) assured safety in the face of unknown unknowns and Sim2Real gap, ii) teaching-to-lext for safety and self-learning for high performance, and iii) safety-informe batch sampling address the experience imbalance. Experiments conducted three benchmarks have successfully shown the effectiveness and unique features of Real-DRL.
In current Real-DRL. the Trigger continuously monitors the system safety status to manage DRL- Student and Tey-Teacher. To alleviate this monitoring burden, Trigger shall have an early warning function of future safety status. Utilizing reachability techniques [2] could provide viable solution.
10

---

Acknowledgments
This research was partially funded by the National Science Foundation through Grants CPS-2311084 and CPS-231 UPS-2311085, as well as by the NVIDIA Academic Grant Program.
We thank Hongpeng Cao at the Technical University of Munich, Germany, for his assistance with the indoor experiment on the real quadruped robot.
References
[1] Martin S Andersen, Joachim Dahl, Lieven Vandenberghe, et al. CVXОРT: A Python package for convex optimization. https://ewxopt.org.
[2] Stanley Bak, Taylor T Johnson, Marco Caccamo, and Lui Sha. Real-time reachability for verified Simplex design. In 2014 IEEE Real-Time Systems Symposium, pages 138-148, 2014.
[3] Arko Banerjee, Kia Rahmani, Joydeep Biswas, and Isil Dillig. Dynamic model predictive shielding for provably safe reinforcement learning. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[4] Osbert Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. 2021 American control conference, pages 3488-3494, 2021.
[5] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in Neural Information Processing Systems, 30, 2017.
[6] Daniel Bogdoll, Jasmin Breitenstein, Florian Heidecker, Maarten Bieshaar, Bernhard Sick, Tim Fingscheidt, and Marius Zöllner. Description of corner cases in automated driving: Goals and challenges. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1028-1028, 2021.
[7] Joshua Bote. Waymo driverless car brings san francisco traffic to a halt during rush hour. http://wwwww.otooooootee//e//o/a 2023.
[8] Guillaume Brat and Ganeshmadhav Pai. Runtime assurance of aeronautical products: prelimi- nary recommendations. NTRS NASA Technical Reports Server, 2023.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang. and Wojciech Zaremba. OpenAI Gym. arXiv:1606.0111 01116006600110 https://axxiv.rrg/dd///c http::oorgg////1060011044440 2016.
[10] Yihao Cai, Yanbing Mao, Hongpeng Cao, Lui Sha, and Marco Caccamo. Runtime learning machine. To appear in ACM Transactions Cyber Physical Systems (Special Issue Embodied Artificial Intelligence in Cyber-Physical Systems: Algorithms, Computing Systems, 2025.
Applications, and Trustworthiness),
[11) Yihao Cai, Yanbing Mao, Lui Sha, Hongpeng Cao, and Marco Caccamo, Runtime learning of quadruped robots in wild environments. arXiv:2503.04794 https:/aarrivoorraaa//233344.444 2025.
[12] Hongpeng Cao, Yanbing Mao, Lui Sha, and Marco Caccamo. Physics-moodel-eguulee deep reinforcement learning towards safety & stability guarantees. In 62nd IEEE Conference Decision and Control, pages 8300-8305, 2023.
[13] Hongpeng Cao, Yanbing Mao, Lui Sha, and Marco Caccamo. Physics-regulatee deep reinforce- ment learning: Invariant embeddings. The Twelfth International Conference on Learning Representations, 2024.
[14] Hongpeng Cao, Mirco Theile, Federico G. Wyrwal, and Marco Caccamo. Cloud-edge training architecture for sim-to-real deep reinforcement learning. 2022 IEFE/RSJ International Conference Intelligent Robots and Systems, pages 9363-9370, 2022.

---

[15] Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost Lyapunov critics. In 2021 IEEE International Conference on Robotics and Automation, pages 1803-1809, 2021.
[16] Shengduo Chen, Yaowei Sun, Dachuan Li, Qiang Wang, Qi Hao, and Joseph Sifakis. Run- time safety assurance for learning-enabled control of autonomous driving vehicles. In 2022 International Conference on Robotics and Automation, pages 8978-8984, 2022.
[17] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks, In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387-3395, 2019.
[18] Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. International Conference on Machine Learning, pages 141-11550 2019.
[19] Jared Di Carlo, Patrick M Wensing, Benjamin Katz, Gerardo Bledt, and Sangbae Kim. Dynamic locomotion in the MIT Cheetah through convex model-predictin control. In 2018 IEEE/RSI international conference on intelligent robots and systems, pages 1-9, 2018.
[20] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In 202/ IEEE International Conference on Robotics and Automation, pages 1290-1296, 2021.
[21] Razvan V Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies,
2007.
[22] Zipeng Fu, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Jitendra Malik, and Deepak Pathak. Coupling vision and proprioception for navigation of legged robots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17273-17283, 2022.
[23] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596, 2018.
[24] Pascal Gahinet, Arkadii Nemirovskii, Alan J Laub, and Mahmoud Chilali. The LMI control toolbox. In Proceedings of 1994 33rd IEEE conference on decision and control, volume 3, pages 2038-1 1994.
[25] Yuanzhe Geng, Erwu Liu, Rui Wang, Yiming Liu, Weixiong Rao, Shaojun Feng, Zhao Dong, Zhiren Fu, and Yanfen Chen. Deep reinforcement learning based dynamic route planning for minimizing travel time. In 2021 IECE International Conference on Communications Workshops, pages 1-6, 2021.
(26) Michael Grant, Stephen Boyd, and Yinyu Ye.
CVX
users'
guide.
https//cwxx..om//xx//c////C
2020.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870, 2018
[28] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, (0:45))6698-1 2021.
[29] Chicko Sarah Imai, Minghao Zhang, Yuchen Zhang, Marcin Kierebiriski. Ruihan Yang, Yuzhe Qin, and Xiaolong Wang. Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. In 2022 IEEE/RS. International Conference Intelligent Robots and Systems, pages 5556-5563, 2022.
[30] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 20/9 International Conference on Robotics and Automation, pages 6023-6029, 2019.
12

---

[31] Norman L Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Continuous univariate distributions, volume volume 289. John Wiley & Sons, 1995.
[32] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in day. In 2019 International Conference on Robotics and Automation, pages 8248-82545 2019.
[33] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-499) 2021.
[34] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
[35] Tongxin Li, Ruixiao Yang. Guannan Qu, Yiheng Lin, Steven Low, and Adam Wierman. Equip- ping black-box policies with model-based advice for stable nonlinear control. arXiv:2206.01341,
https//arxiv.or///d//226... 3.ff..
[36] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, 2016.
[37] Yanbing Mao, Yuliang Gu, Naira Hovakimyan, Lui Sha, and Petros Voulgaris. Sl1-simplex: safe velocity regulation of self-driving vehicles in dynamic and unforeseen environments. ACM Transactions on Cyber-Physicee Systems, 7(1):1-24, 2023.
[38] Anusha Nagabandi. Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference on Learning Representations, 2019.
[39] NVIDIA. Isaac Gym - preview release, 2023. https//deeelooerraawiia.ccaaaaaa
[40] Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, May 2018.
[41] Robert Penicka, Yunlong Song, Elia Kaufmann, and Davide Scaramuzza. Learning minimum- time flight in cluttered environments. IEEE Roboties and Automation Letters, 7(3):7209-7211 2022.
[42] Theodore Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 3(De)):803 832, 2002.
[43] Dung T Phan, Radu Grosu, Nils Jansen, Nicola Paoletti, Scott A Smolka, and Scott D Stoller. Neural Simplex architecture. In NASA Formal Methods Symposium, pages 97-114. Springer, 2020.
[44] Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
[45] Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, and Niko Sünderhauf. Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for roboties. The International Journal of Robotics Research, 42(3):123-146, 2023.
[46] Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real single-image flight without a single real image. arXiv: 1.04201, https://arxxx.org//dtt// 1,04011,,
[47] Remo Sasso, Michelangelo Conserva, and Paulo Rauber. Posterior sampling for deep rein- forcement learning. The Fortieth International Conference on Machine Learning, pages 1-19, 2023.
[48] Danbing Seto and Lui Sha. An engineering method for safety region development. CMU Technical Report, 1999.
[49] Lui Sha. Using simplicity to control complexity. JEEE Software, 18(4):20-28) 2001.
13

---

[50] Joseph Sifakis and David Harel. Trustworthy autonomous system development. ACM Transac- tions on Embedded Computing Systems, 22(3):122, 2023.
[51] Cassandra Smith. State police urge caution as freezing rain leads to several crashes. WCIA, crashes/, 2020.
[52] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. Robotics: Science and Systems, 2018.
[53] Stefan Van Aclst and Peter Rousseeuw. Minimum volume ellipsoid. Wiley Interdisciptinary Reviews: Computational Statistics, 1(1):71-82, 2009.
[54] Quan Vuong, Sharad Vikram, Hao Su, Sieun Gao, and Henrik I. Christensen. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv: 1774 https://arxivv.org/dd////////// 11774, 2019.
[55] Tyler Westenbrock, Fernando Castaneda, Ayush Agrawal, Shankar Sastry, and Koushil Sreenath. Lyapunov design for robust and efficient robotic reinforcement learning. https:/aaxiv.or//p///220..... 06721, 2022.
[56] Tsung- Yen Yang. Tingnan Zhang, Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu. Safe reinforcement learning for legged locomotion. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2454-2461, 2022.
[57] Yuxiang
Yang.
GitHub:
Quadruped
robot
simulator.
httss//gittbb.ommyyyyyy/oooooooo sirirommmtiiossmmutttttooooo simulation.
[58] Yuxiang Yang, Tingnan Zhang, Erwin Coumans, Jie Tan, and Byron Boots. Fast and efficient locomotion via learned gait transitions. In Conference Robot Learning, pages 773-783, 2022.
[59] Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. Robotics: Science and Systems, 2017.
[60] Arnold Zachary and Toner Helen. Al Accidents: An emerging threat. Center for Security and Emerging Technology,
2021.
[61] Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006.
[62] Liqun Zhao, Konstantinos Gatsis, and Antonis Papachristodoulou. Stable and safe reinforcement learning via Barrier-Lyapunov actor-critic approach. In 62nd IEEE Conference Decision and Control, pages 1320-1325, 2023.
[63] Chris Ziegler. A Google self-driving car caused crash for the first time. 2016.
14

---

Appendices
Auxiliary Lemmas
16
Pscudocode: Real-DRL Framework
17
Proof of Theorem 4.1
18
D
Proof of Theorem 5.2
19
Pseudocode: Computation of PHY-Teacher's Action Policy
24
F
Compensation for Large Model Mismatch
25
Computation of Compensation Action
25
F2
Conditional Usage of Compensation Action
25
Experimental Evaluation: Python CVXPY v.s. CECVXCONE
26
Unknown Unknown: Randomized Beta Distribution
27
Experiment Design: Real Quadruped Robot: Indoor Environment
28
Learning Configurations and Computation Resources
28
Safety Conditions
28
L.3
PHY-Teacher Design
28
1.4
DRL-Student Design
29
J
Experiment Design: Quadruped Robot: Wild Environment
31
Learning Contigurations and Computation Resources
32
J.2
Safety Conditions)
................................
32
J.3
PHY-Teacher Design
32
J.4
DRL-Student Design
32
J.5
Ablation Experiment
33
Experiment Design: Cart-Pole System
34
K.1
Learning Configurations and Computation Resources
................................
34
K.2
Safety Conditions]
K.3
PHY-Teacher Design
34
K.4
DRL-Student Design
35
K.S
Auxiliary Experiment
35
15

---

A Auxiliary Lemmas
In this section, we present auxiliary lemmas that will be used in the proofs of the main results of this article.
A
B
Lemma A.1 (Schur Complement [1]) For any symmetric matrix M holds if and only if the > 0 and B.C. C-1 B > hold.
the M
B
C
Lemma A.2. [48] We define two state sets: S = {sE R" C C. 1m] and 5 -
{s E" T
s 1, Q }. We have the Q S. if the I. C. C 0 holds.
Lemma A.3. 133 We define two action sets: A A {aER D a < 1,) and E -
{a ("IRaaa a T-l.a a 1, T> 0}. We have the = S A. the Ip D. 0 holds.
Lemma A.4. For two vectors X E R", E IR" and matrix P > 0, we have
1 y P.y, y, with 2 x.x P.y P p
Proof. The proof straightforwardly follows inequality of:
(Vo x Vo Y) y) P. P.PVV Jo v) Φ TT.
1
P p x+ ..y.-2xx..P.x22
.P.
which is due to P > 0 and 0.
Lemma A.S. Given any X 0 and matrix X E R mxn, the matrix X X X Im invertibl.
x. For any non-zero vector x, we have x [X X + + x = ||yl? ||x|| > O, where Il-Il, denotes the Ly-norm of a vector. So,
Proof. The proof is straightforward by letting y
X X + A Im is a positive definite matrix, thus always invertible.
16

---

B Pseudocode: Real-DRL Framework
Algorithm 1 Real-DRL Algorithm
1: Real plant operates from the safe self-learning space IL initially, i.e., s(t) E Il at initial time = 1;
2: DRU--ttnent's real-time actor-network outputs the action a (t) given the input s(t);
DRL-Student applies aunden (t) to real plan to have the state s(t 1);
4: Update time index: lc t- ++
5: if s( + 1) IL then
6: PHY-Teacher uses the real-time state s(k) update physics-model knowledge (A(s())), B(s(k));
PHY-Teacher
real-time state s(k) updatc control goal si in Equation (6):
8:
PHY-Teacher computes R3 and Q2 from incqualities in Equations (20) and (21):
9:
PHY-Teacher computes action policy, denoted by Fs. according to Equation (99
10:
PHY--Tacceer computes action Bucnceer (f 1) according Equation (18), given real-time state s(t 1);
11:
PHY-Teacher applies Blacher (t 1) to real plan back up safety and have s(t 2);
12:
PHY-Teacher computes (ss(t + 1), aleacher(
1),s(( 2));
13:
Teacher compiles the teaching to leare experience data as described in Equation |
14:
Teacher stores the teaching to-learn experience data [1] into teaching- toagg-to-low to-lennning replay buffer;
15:
Generate batch samples according the safety informedd batch sampling mechanism (14):
16:
Apply off-policy algorithms update critic and networks, using the generated batch samples;
17:
if stt + 2) ş T. then
18:
Update time index: + 1 + +
19:
Go to Line 10
20:
else
21:
Update time index: 1 <- L 2; Go to Line 2
22:
23:
end if
24:
else
25:
DRL-Student computes the reward R(s(t), (f), ((), 1)):
26:
DRL-Student compiles self-learning experience data as described in Equation (T)b:
27:
DRL-Student stores the self-learning experience data (TO) into self-lcarning replay buffer;
28:
Generate batch samples according to the safety-in informedd batch sampling mechanism (14):
29:
Apply off pollcy algorithm update critic and actor networks, using the generated batch samples;
30:
Update time index:
31:
Go Line 2
32:
end if
The working mechanism of our proposed Real-DRL. is explained in Algorithm T Below, we provide additional clarifications. The actions of the activated DRL-Student and PHY-Teacher are highlighted the yellow and green blocks, respectively. As indicated Line [and Line when the triggering condition described by (6) is met, the PHY-Teacher is activated to back up the safety of the real plant (as detailed in Lines to [T] and to foster the teaching-to-lent
paradigm (as described in Lines [2 to [6). Furthermore, Lines 17 to 23 outlines the process in which, once the PHY-Teaches has guided the real plant back to the self-learnng space, control returned to the DRL-Student. At that point, the DRL-Student is activated while the PHY-Teacher is deactivated.
The Lines 15 and T6 in the yellow block (i.e., the activation period of the PHY-Teacher) and Lines 28 and 29 in the green block (i.e., the activation period of the DRL-Student) indicate that during the activation period of either the DRI--Studett or the PHY-Teacher, the critic and actor networks continue to update through off-policy algorithms. The generated batch samples are based -afety-informed batch sampling, which includes samples drawn from both the teach-to-learn and self-learning replay buffers. Their contribution ratios depend on the real-time safety status, represented as V (s(k).. Therefore, we can conclude that Real-DRL operates within a dual paradigm of self-learning and teaching-to- learn during its runtime operation.
17

---

C Proof of Theorem 4.1
The overall proof consists of the following steps.
Step 1: Conclude: Set {s TR V(s) 1) C.
The safety set defined in Equation
2 can be transformed into an equivalent representation as follows:
S = fs RR" | (C diag -1 *(c]) s 1p).
(23)
Meanwhile, letting Q = p- the conditions (i.e., - (c-(CC-daa *(c}) (C--(C- ((c))
>
and P 0) included in Equation 13) imply that
I (C .diaaaa
[c}) p-1. -C-dia (c)) and 0.
(24)
Then, by applying Lemma Equation we conclude from Equations
A.2
from Appendix A and taking into account the V(s) defined in (23 and that:
{s E R
Q 1 = {SE
P
S
= {s {SER" V(s) $1)5 CS (25)
Step 2: Proof of maximum volume.
We recall that the volume of ellipsoid, defined by the set Es P
{s E R V(s) I 1}, is proportional to /det(PP-)) 48 53). Therefore, maximizing the vol-
ume is equivalent to minimizing the determinant: det(P ((()P))). '). This leads to the optimal problem
outlined in Equation [13 As a result, the ellipsoid set R" V(s) I 1] achieves the maximum volume. Thus, the proof completed.
18

---

D Proof of Theorem 5.2
has assured safety, we shall prove the PHY-Teacher's real-time actions and the resulted system states simultancously satisfy three requirements. They are when s(k) ØL, then i) the s(t) E holds for any time Tk. ii) Ateacher (t) (saache(((())) (s(t)) E A holds for any time E Tg, and iii) there exists E N, such that s(k + Tk) E L. The overall proof will separated into seven steps conclude the three conditions i) i)-i hold in Steps 4, 5, and 7, respectively. These steps are detailed below.
According Definition 2.3 prove the PHY-Teacher designed by Theorem 5.2
Step 1: Derivations from the first inequality in Equation (21)
Recalling the Schur Complement in Lemma A.1 of Appendix A and considering that Q2 > conclude that the first inequality in Equation 21 is equivalent to
a Q: (1+ (Qx-A (s(()) R B (s(k)) Q (A(s())). Qs + ((s(k))) > 0,
multiplying both the left-hand side and the right-hand side of which by Q yields:
Q (1+ (A (s(+)) Q
R B (s(ss))))
Q (A(s(A)) + B(s(R)) R Q
O,
substituting the definitions in Equation 19 into which, obtain:
Pk + +) (A (s())) F. B (s(+))) Pa (A(s(H)) + B(s(k)) F&) 0, (26)
which is equivalent to
a P (1+ Φ) A'AA (s())) PE A(s(k)) > O,
(27)
where we define:
A(s(K)) A A(s(k)) + B(s(k) Fk.
(28)
Step 2: Derivations from the first inequality in Equation (20).
We define the set:
2 {sl (s 5tt] Q (s - <1, Q2 > 0}.
(29)
Considering the definition e()) _ s(l) - s and the transformation C ف C diag (0- c). the patch
defined in Equation (15) can be equivalently expressed as PA- { E. C (s- st) < 1}. Mcan- while, by considering the first inequality in Equation (20p and applying Lemma A.2 of Appendix A
we conclude:
24 S PR because I, - C. Q C 0.
(30)
We note that the patch in Equation
(15) can also be rewritten as follows:
Fx {s
E
E
8 c.c C C- s
(31)
From Equations
6 and 16.
we obtain that C s = X C . s(k)
Substituting this into Equation (31) yields:
Px (SE R"I -0-
<
cj
(s R'I - (0 + x c< < c) (32)
The first item in Equation 222 shows that 0 0 . < 1, which is equivalent to -0-X-
Since both 0 and 1 > can also deduce that 0 - X 17 and -0 + x- バ -1. Therefore,
conclude from Equations and (2 that PA C S. This, in combination with Equation 330 leads to the following result:
S C PR S.
(33)
19

---

Step 3: Derivations from the third inequality in Equation (20).
The vector skk) can have zero entries, for which we define:
[5()] + 5, if [f))] 0
B( 8)
i= i=,,,.....
[s(k)]
otherwise
where 8 + and [s(]]] represents the i-th entry of the vector skk). Thus, the vector s(k) contains no zero entries. We therefore conclude:
lim s(k, 6) = s(k), 8-0
(34)
in light of which and the third inequality in Equation 20 we can conclude the following:
lim { Qx n diag"((((e) 6))} Q+ n diag"(s(k)) > 0.
(35)
8-0
The limp-o Qk n diag" 8))} > 0 is equivalent to Qs lims {n diag" 6))} > O. Given that (kk, 5) does not have zero entries and the Qs 0, can further conclude that lims-o(n diag (5(k, 6))} 0 > 0, which means
L, lim diag (5kk 8)) Q diag(E(k, 8))3 > 0,
from which, the (lag(5())) 1, {kk) and the Q PA. have:
lim [1, diaag(((k,))) 6)) Q diag (sk, 6)) 1, s (k: 8) P s(k, 6)) n- 1.. 1 1, 8-40
In conjunction with Equation this leads to the following conclusion:
s (k) Pk s(k) 5-+0
(s (kt, Ps 5(k, 8)} < 1.
(36)
Step 4: Conclude: s(t) S, Tg.
Considering the dynamics outlined in Equation 17 the action policy described in Equation (18)
and the definition provided in Equation (28). we obtain:
T (t + 1) Pa e - e (t) ee
=c (t) A (s(k)) Pk A(s(k)) - a: PP)) e (t) + Pa ((e(t(((t)
- (t) A (s(k)) PA h(e(t)), te T (37)
In light of Lemmal A.4 of Appendix we have:
2 e T (t) A (s(k) P. h (e S A (s(k) Pk ((-A(s(k)) e
+ hT (e(1)) PE (e())) (38)
We note that Assumption 5.1 means:
h (e)) Py h(e(())) < <k,
Ve(() Pt
(39)
Substituting inequalities in Equations 388 and (39h into Equation 37 yields:
T (t + 1) P e (t (t+11 a. ee (t) P e
(6 + (6+1).K
s T (t) ((+ A (s(k)) Pk ((s(k) PPs) e(t) +
te TT- (40)
We now are able to conclude from Equations and 40 that
(0 + 1)
11+ Pt c( <a (t) P c c(t)
te TT.
(41)
20

---

The left-hand inequality in the first item of Equation 22 is equivalent to (1 -1-X0 x) 6. Given that
0 < X < 1, this leads to the conclusion that (1-x)²
Considering this, we can derive from the 1, which can be rewritten as ( 1. From
02
second item in Equation (22) that [++1))
(I-a)-6
this inequality and Equation (6).. we obtain:
(t + 1) Px et < 1, for any e (t) Pk (t) <1
(42)
Additionally, recalling Equation (6). we have: e(k) = s(k) s = s(k) -x.s(()) (1 x) s(k).
where < X < This expression, along with Equation (36), leads to result: e² (k).Px e e(k) <1
Combining this with Equation (41) and considering the process of iteration, we can draw the following conclusion:
(t) P ( S 1, Vt E TT.
(43)
Furthermore, referring to e(t) A s(l) si and Equation (9) with Q
PA. the result in
Equation (43 means that s (t) E Qg VI E Tk- Finally, considering Equation conclusion:
we reach the final
(f) Qg CP CS, VE
(44)
Step 5: Conclude: ateacher (t) E A.
According to Lemma A. in Appendix A. the second inequality in 21 implies the following:
R
T-1 R. >
(45)
By substituting F3- Q Ry into Equation 45 we obtain:
Q (Fx 2x)
T (F, Q+) > 0.
(46)
multiplying both left-hand and right-hand sides which by Q yields:
Q₂ Q
F
T
Fx > 0,
from which and Equation (8) we thus have
(t) Qi e(t) - e T (t) T F& c(t) = c T(t) Q c(t)
a (t) T
aseacher (t) > 0. (47)
We observe that the patch set in Equation (29) can be expressed in an equivalent form as follows:
2 cT Q- cl.cc <eckl where Q₂ > and c _ s }
(48)
from which, we conclude that e (1) E 24 means e Equation 47 that
T (t) Q e(t) < 1. So, we can obtain from
areacher
Areacher (1) < e (t) Q e(l) < for (1) E Ωr-
(49)
Additionally, based Equation 48). the conclusion in Equation
44 indicates that e (1) E 2g Vi
Tk. This, in light of Equation (49), results in
a qeaccer (t) Te
ascacher (t) < 1, Vt TT.
(50)
We define a set: E, { E R a TE a<l,, Tt > 0}. light of Equation (50). we obtain:
Agecher Ex, Vt E T
(51)
Noting D D diag [d), the definition of action set A in Equation can be equivalently expressed as A 1 D 5 1}. Besides, recalling the second inequality (i.e., L, D. Tx-
D
0) from Equation 20p and applying Lemma A.3 with consideration of Equation 51). we can
conclude that:
Asencher (a) ( C A, W Te.
(52)
21

---

Step 6: Further derivations from the first inequality in Equation (20).
Referring to Equation
15 we define an auxiliary real-time set as follows:
G2 s
E
R
c c C (s si. c. with (1 x) -1 0).
(53)
where X is given in Equation (6). 1j is given in Equation (4. and is given in Equation (13 The
Gg in Equation (33 be equivalently expressed
C G = E C. - 7 CS s +
with (1 - x)= ->
(54)
We recall from Equation (6) that s(k) denotes a system state that approaches the boundary of safe self- space IL defined in Equation (4) Hereto, by considering si = x(k() and Equation 15
we have X 7) C. s x- 7) c, substituting which into (54) yields:
G { R I (x 17 7)
.. (x 7 c, with 7 (1 - x) x7 <)
0).
(55)
We note that the relationship 7 (1 -x x) 17 is equivalent to the equation X + - This implies
that X 11 - > -1, considering that X 1 0. In summary, we establish both x 7 = 1/ and
1 m.. Considering them, we can derive result from Equations 4 and (55 as follows:
G3 C L.
(56)
Considering that Pk- = Q and the patch defined in Equation 15 we can derive from Equa-
tions 29
and 30 that:
{s
E
R"
(s
s/
PA (s sf < 1) S -
0.c5 C (s st)
c), cc.
c > 0 (57)
where & C. diag 1(0.c) c}. Letting Č - C diag {r c} and considering > 0, we can obtain
from Equation (57) that
0 {s e R" (s si)
( Pt) (s -s)) < 1) C {se -5 C. (s -s)) S c.c.
if - è. 2 P Č > 0. (58)
Moreover, it is straightforward to verify from و C diad (0 c} and Ò a C.
diag ( c} that
Ć C-222 - Č C C So, with the consideration of Equation 53). we can further
conclude from Equations (57) and 58 that:
E (s (t) Pk
si)T S 82
C Gr, if I, - C. C 0.
(59)
Furthermore, recalling the first inequality (i.e., I, -C Qu C from Equation 20),
x² arrive at the conclusion {s E IR (s st)T Pk P s I &z
C G.. This, combined with
Equation (56). leads to the following result:
72
E R. (s - st) .k (s - st)
c
f
substituting 7 = (1- - x) 17 into which yields:
{s E R (s - st) Pk (s as-stt si) 5 (1 x)2 .7 n7²
82 C IL.
(60)
22

---

Step 7: Conclude: s(k: + Tk) E L for a E N.
We can derive from that
(k +) Pa- sa (k) Px e(k) e ( +0+1) 1) e
1 1 a 7-1
Φ
--0 0
where 0 a < This leads to the following result:
(0c) Pk (0o) v (6 10+1)
(1 a1) Ø"
which with the second item in Equation 22 leads to
(0o) Pk e (00) - (0+1) K
(1 (1- x1-x)2-11
(61)
(1 a)
1
Given that e(t) _ s(t) i we can equivalently express the result in Equation 61
as:
PA (0o) s - (Ø + 1). (1
x)²
n7²
(s (oo) $i))
(62)
(1 a)
@2
Finally, we can conclude from Equations (60) and 66 that s (oo) E L. This indicates that there exists Tk N such that s (k: + Tk) IL. Thus, the proof is completed.
23

---

E Pseudocode: Computation of PHY-Teacher's Action Policy
Referring to Equation (8)) the computation of the PHY-Teacher's action policy involves calculating
F& based on the inequalities presented in Equations (20) and (2T). To compute F5. we can utilize
the Python CVXPY toolbox [26) the Matlab LMI toolbox 24 described in Algorithm 2 and
Algorithm 3 respectively. In both Algorithm 2 and Algorithm 3 the A, B, S, C, D, and correspond
to (ss(k), (s)))),, skk), C. D, and Fg, respectively.
Finally, we recall that the ECVXCONE toolbox described in Section 6 can solve the LMIs using C
Algorithm 2 Python CVXPY Toolbox for Computing F3 from Equations (20) and (21
1: Input: A, B, s. C, D, m, p. 9. and 0 < a < 1 and / > 0 satisfying conditions in Equation 22
2: Q = cp. Variable((n, n), PSD=True);
3: T cp. Variable((m, m), PSD=True);
4: R = cp. Variable((m, n));
5: constraints [cp.bmat([[a Q.Q A.T B.T, Q R. ( x)]l) 0;
6: Q n * npp.dia((s) @ np.diag(s) > 0;
7: np.identity(p)- C @ @ C. 0:
8: np.identity(q) -
@ Q D.T > 0:
10: pobhemssovvesovveepppccvC. constrra)))
9: problem www..ooollmccp.iiiiiiieeeee
11: optimal Q.vaue;
12: optimal_ R = vavale;
P= ap..inalg.invooppimall
_Q):
14: F == optimal R @ P.
Algorithm 3 Matlab LMI Toolbox for Computing Fx from Equations (20) and (21
1: Input: A. B, s. C, D, m. p. 9, and 0 < < and d >> 0 satisfying conditions in Equation 22
2: setlmis(()):
3: Q = Imivar(1.[n I]);
4: T Imivar(1.[m I]):
5: R 1mivar(2.[m n]);
6: Imiterm([-11 1 I Q].1,a);
7: Imiterm([-122 2 1 Q].A,1);
8: Imiterm([-122 1 RJ,B,1);
9: Imiterm([-1 2 Q).1,1/(1 + +));
10: Imiterm( (1-2 1 Q],1,1);
Imiterm([-22) R],1,1);
12: Imiterm (-22211)(2222T)111111
13: I Q),1,1);
14: mmierm((33110)---'ppppp.
Imiterm([-411 Q],D.."))
16: Imiterm([-4 O].eye(p));
Imiterm([-5 T),-C,C));
18: Imiterm([-5] O).eye(g));
mylmi = getlmis;
20: [tmin, psol] = feasp(mylmi);
21: Q = dec2mat(myyli,, psol, Q);
22: R dec2mat(mylmi, psol, R);
inv(Q)
24: F == R * P.
Remark E.1. We observed that the computation time for the methods described in Algorithm and Algorithm 3 is quite short, ranging from 0.01 0.04 seconds (see evaluation in Table therefore, its impact can be considered negligible. However, we found that the default CVX solver, SCS, demonstrated instability and inconsistent accuracy across different computing platforms. As a result, we opted to use the CVXOPT solver Ml in our experiments to achieve more stable and accurate results. This issue, however, does not arise when using the Matlab LMI toolbox.
24

---

Compensation for Large Model Mismatch
When the actual dynamics described by Equation (7 experiences significant model mismatch, the physics-model knowledge (A (s(k)) (k)(k)) may be inadequate. such situations, the PHY- Teacher should implement strategy address this considerable model mismatch. This section outlines the technique used to compensate for the discrepancies in the model.
F.1 Computation of Compensation Action
Given the state sample at the current time step t (denoted as e(t)), we can obtain the current control command ateacher (4), calculated according to the method described in Equation (18). In addition
to this information, we also have the previous state e(t 1) and the moysics-model knowledge represented by (A(s()))B(((()) We can use the dynamics outlined in Equation (17) to predict the
current state follows:
( = (s())) e(t - 1) + B(s(k)) Aseacher (t),
(63)
where é() denotes the predicted state at time
In the context of compensating for model mismatch, the dynamics at the current time, as described in be expressed in equivalent manner:
c(t) = (s())) c(t 1) + B(s(k)) (aueachertt) (t) + acompensation (t)) + h(c(t)),
(64)
where the term @compensation (t) signifies the real-time actions taken to address the model mismatch h(e(t)).
We can further derive from Equations
163 and
64 that
(s())) ccompensation
h(e(t)) (tt) a(t),
(65)
where e(t) and @(t) are known. From Equation 65 find that if the compensation action meets the condition outlined Equation 66). the model mismatch h(e(()) Equation (64) can effectively
compensated by acompensation (t) (i.e., h(((()) = O,,).
(ss()) ccompensation (4) = e(t) @().
(66)
We next explain how to calculate the compensation action, denoted compe (4), based on the conditions outlined in Equation (66).
By multiplying the left-hand side of Equation (66 by B (s(k)), we obtain
B (s(k)) B(s(k)) @ccompesation (t) B (s(k)) (e(t) ē()))
(67)
By incorporating a sufficiently small value of À > in Equation 677) we achieve the following outcome:
(B (s(k)) B(s(k)) + + Acumpensation (t) ~ B (s(k) (e(t) o()))
which, in light of Lemma A.5 in Appendix the solution for acompensation (1) is yielded as follows:
Acompensation (t) (B (s(k)) (ss())) + X Im -.. (s(()))) (e(t) - @()))
(68)
F.2 Conditional Usage of Compensation Action
At the current time I, possess the following information: the current state e(l), the previ- ous state e(l 1), the previous action a(l -), and the physics model knowledge denoted by (A (s(s))), B(s(k))) With this data, we estimate the real-time value of model mismatch at time / using the dynamics described in Equation (17). The estimation of model mismatch can be expressed as follows:
h(e(()) 2 h(e(t 1)) = e(l) - (s(k)) e(l 1) + B(s(k)) Atencher 1),
where h(e())) 2 hee( 1)) holds true due to the sufficiently small sampling period of sensors. Given the estimated model mismatch, we can carry a real-time evaluation of Assumption 5.1 5.1
decide whether to implement the compensation action. The process is outlined as follows:
Atencher (!) + Acompensation (t); if h (e) Pk h(e) >
ageacher (t) t
Deeacher (t),
if (c) Pk h(c) S
25

---

G
Experimental Evaluation: Python CVXPY V.S. CECVXCONE
Table 2: Python CVXPY v... CECVXCONE
CPU Configurations
Runtime Memory Usage
LMIs Solve Time
Hardware Platforms
Arch
Core
Frequency
CVXPY ECVXCONE
CVXPY
ECVXCONE
Dell XPS 8960 Desklop
x86/64
32
5.4 GHz
485 MB
9.87 MB
49.15 ms
13.81 ms
Intel GBRKOM 13 Mini
GHx
MB
61.76
33.26
NVIDIA Jetson AGX Orin
ARM64
12
2.2 GHz
423 MB
8.16 MB
137.54 35.73 ms
Raspberry Pi Model B
ARM64
4 1.5 GHz
436 MB
8.21 MB
509.41 ms
149.87 ms
We summarize the experimental evaluation comparing Python CVXPY and C ECVXCONE regarding computational resource usage four platforms with varying CPU frequencies in Table 2 Both CVXPY and ECVXCONE solve the same LMIs for the experiment 'Quadruped Robot in NVIDIA Isaac Gym: Wild Environment' in Section 7.2
26

---

H Unknown Unknown: Randomized Beta Distribution
In the real world, plants can encounter multitude of unknown variables, each with unique char- acteristics. To tackle this challenge, we propose utilizing a variant of the Beta distribution [31 effectively model one type of these unknowns. This approach holds promise in mathematically defining and addressing these uncertainties.
Definition H.1 (Randomized Beta Distribution). The disturbance, noise, or fault, denoted by d(k),
considered be a bounded unknown (i) d(k) ~ (eta(a(k), B(k), a), and (ii) a(k) and 3(k)
are random parameters. In other words, the disturbance d(k) is within the range of Ia, c), and its probability density function (pdf) is given by
(d(k); ((k), ((k), a, c) (d(k) (c-d))) - (a(k) +B()))((())
(69)
(c- a) ((())+((k)))---))))) (a(k)) (B(K))
where I' (((()) Jo (a(k)-Le- Re Re(k)))) 0, a(k) and (kk) are randomly given every k.
The randomized Beta distribution, defined in Definition H.T is essential for describing particular type of unknown unknown. This is important for two main reasons.
First, the nature of unknown unknowns is characterized by lack of historical data, making them highly unpredictable in terms of time and distribution. As a result, existing models for scientific discoveries and understanding are often insufficient.
Second, as the examples shown in Figure the parameters a and B3 directly influence the probability density function (pdf) of the distribution, and consequently, the mean and variance. Suppose and 3 are randomized (expressed as a<k) and 3())). In that case, the distribution of d(() can take the form of uniform distribution, exponential distribution, truncated Gaussian distribution, combination of these. However, the specific distribution is unknown. Therefore, the randomized (Ak) and (kk). which result in randomized Beta distribution, effectively capture the characteristics of "unavailable model" and "unforeseen" traits associated with unknown unknowns in both time and distribution. Furthermore, the randomized Beta distributions are bounded, with the bounds denoted as a and c. This is motivated by the fact that, in general, there are no probabilistic solutions for handling unbounded unknowns, such earthquakes and volcanic eruptions.
C = 0, 11
a(k) A() 0.15
a(k) ,3(k)
a(k) = -8(k) 11
f(d(k);a(k)) ((k)c.,a)
3
a(k) =5, 3(K) = 1
a(k) =1, 3(K) 3
a(k) 2. 3()
0
0.2
0.4
0.6
0.8
1
d(k)
Figure 7: akk and ((k) control the probability density function of the distribution.
27

---

I Experiment Design: Real Quadruped Robot: Indoor Environment
In our real quadruped robot experiment, we used Python-based framework developed for the Unitree A1 robot, which was released on GitHub by [57]. This framework includes PyBullet-based simulator, an interface for direct simulation-to-ree transfer, and an implementation of the Convex Model Predictive Controller for essential motion control.
1.1 Learning Configurations and Computation Resources
The runtime learning machine and Phy-DRL are designed accomplish the safe mission detailed in Section 7.2 The policy observation consists of a 10-dimensional tracking error vector that represents the difference between the robot's state vector and the mission vector. Both systems utilize off-policy algorithm DDPG.
The actor and critic networks are implemented MLPs with four fully connected layers. The output dimensions for the critic network are 256, 128, 64, and I, while those for the actor network are 256, 128, 64, and 6. The input the critic network comprises the tracking error vector and the action vector, while the input to the actor network is solely the tracking error vector. The activation functions for the first three layers of the neural networks are ReL.U, while the output layer of the actor network uses the Tanh function and the critic network uses a Linear function. Additionally, the discount factor y is set 0.9, and the learning rates for both the critic and actor networks are set 0.0003, Finally, the batch size is configured to 512.
For computation resources, utilized desktop rumning Ubuntu 22.04, equipped with 12th Gen Intel(R) CTr)(TM) i9- 1290K 16-core processor, 64 GB of RAM, and an NVIDIA GeForce GTX 3090 GPU. The algorithm implemented in Python, utilizing the TensorFlow framework alongside the Python CVXPY toolbox for solving real-time patches.
I,2 Safety Conditions
The robot's learning process involves controlling its center of mass (CoM) height, CoM velocity, and other states in order to track the commands Tv.* The and zeros, respectively, while maintaining system states within safety set, denoted as S. This safety set is defined as:
S {s [CoM x-velocity Tu, I 0.6 m/s, [CoM z-height ral I 0.12 m). Additionally, PHY- Teacher's action space is defined as: = { a E R |a S [10, 10, 10, 20, 20, 20) T This ensures that the robot operates within specified limits for safety and performance.
1.3 PHY-Teacher Design
The robot's physics-modee model knowledge used in the design relies on its dynamics, which focuses on single rigid body subject to forces at the contact points [19] Our model of the robot's dynamics includes several key parameters: the CoM height (h), the CoM velocity (v), represented as 3D) vector [CoM x-velocity, CoM y-velocity, CoM 7-velocity] the Fuler angles (e), described as a 3D) vector [roll, pitch, yaw] T, and the angular velocity in world coordinates. According to 19. this model below effeetively characterizes the dynamics of quadruped robots.
O1xi
O1x5
O R( 0, v)
d df
O 3x3
3x3
O 3x3
0 3x1
+Br
O3x3
O3x3
O3x3
Oax3
O3x1
3x3
3x3
3x3
O3x3
g
è A(6.8.0)
+ f()), (70)
where g 0,,0, -g] R3, with g being the gravitational acceleration. f($) denotes unknown
model mismatch, B O8x4, Ie] T and R(Ø, 0, +) (2)()) R,, (d)
3 is the
rotation matrix, with
cos
sin 0
cos vb
sin Lb
0
R. (0)
0 cos Φ
sin
R, (0)
0
1
0
R (1b)
sin vb
cos
0
sin Ø
cos
sin 0
cos
0 1
28

---

133.1 Physics-Model Knowledge
Given the equilibrium point (control goal) s", we define s s". From this, we can obtain the following state-space model derived from the dynamies model in Equation (70).
h
O 1x1
1x3
1
Oxx R ( 0 +))
h e
d
3x1
Oax3
3x3
di
3x1
Oxx3
3x3
O 3x3
W
3x1
O3x3
3xx
3xx3
(s)
O 1x3
1x3
O xx
O1xa
O Oxx
Oxx
O3
O3xa
+
O 3x3
3x3
I3
Oxx3
-a g(s). (71)
O 3x3
Oxx3
O3
I3
((s)
where g(s) represents the model mismatch in the updated model. The sampling technique changes the continuous-tiin time dynamics model in Equation (71) into discrete- model:
s(A: + 1) = (I + 7 A()) s(k) + T B(s) a(k) + 7 +
from which we obtain the knowledge (ssk)) and (ss()) in Equation 17 as follows:
Ass(A)) I10 +T (ssk)) and B(s(k)) = T B(s(k)),
(72)
where T
second, i.e., the sampling frequency is Hz.
30
1.3.2
Parameters for Real-Time Patch Computing
Given that 7 30 seconds, to satisfy Assumption 5.1 we let 6 = 0.01. For other parameters, we let X = 0.3, 0.9, 0.6, 0 0.3, and & 0.15. This ensures that the inequalities presented in Equation (22 are satisfied.
1.4 DRL-Student Design
In this experiment, our DRL-Students build upon two state-of-the-an safe DRLs: CLF-DRL, as proposed in (55), and Phy-DRL outlined in [2]. critical difference CLF-DRL has purely data-driven action policy, while Phy-DRL has a residual action policy:
astudent (k)
add (A) + apty (k) (= F s(k))
data-driven
model-based
where
0
23.65
0
0
0
0
0
20
0
0
0
63.11
0
0
0
20
F
(73)
32.51
0
0
0
0
0
-
21.88
0
0
32.51
0
21.88
0
0
30.95
0
0
0
0
222.88
comparison of these models will be conducted, with all other configurations remaining the same.
In all models, the parameter for defining DRL-Student's self-learning space in Equation (4) is the same as 17 = 0.3. Both CLF-DRL and Phy-DRL adopt the safety-embedded reward:
R. (s(t), Astudent (t)) = T(( P s(t) - (t+ 1) s(t + 1),
(74)
29

---

where
122.1647861
0
0
2.487166
0
1.5e
0
0
1.5e
0
0
480.6210753
2.487166
0
0
3.2176033
0
P
0
1.3c 6
0
0
0
1.2c
9e
5
O
0
9e
5
O
0
0
155.2954557
0
9e
5
0
9c
0
155.2954559
0
(75)
0
0
0
7e
5
0
7e
5
0
156.3068079
30

---

Experiment Design: Quadruped Robot: Wild Environment
We utilize NVIDIA Isaac Gym 33 and the Unitree Go2 robot to evaluate our Real- DRL approach. Isaac Gym, we create dynamic wild environments that include various natural elements, such as unstruc- tured terrain, movable stones, and obsta- cles like trees and large rocks. These en- vironments are further complicated by un- expected patches of ice that result in low friction. We can conclude that the Go2 robot operates non non-stationary and unpre- dictable conditions.
Additionally, we arrange multiple way- points at reasonable intervals across the ter- rain to simulate real- world tasks for the robot, such outdoor exploration and search-and-resee operations. The goal is guide the robot to its destination by quentially following these waypoints while minimizing traversal costs and adhering safety constraints. The established wild environments are detailed in Figure
Figure 8: quadruped robot navigates to its destination by follow- ing waypoints with minimal costs through runtime learn- ing. The next waypoint is highlighted in blue, while the remaining waypoints are marked in red.
overview of wild environments. The
Our Real-DRL must integrate perception and planning to complete tasks safely in real-time wild environments. The integrated Real-DRL. framework illustrated Figure where the perception and planning components form the navigation module.
Real-DRL. for Locomotion Learning and Control
Navigation
Planned Motions
DRL-Student
Roplay Butks
Cost Map
le l10
Mission Back Up Safety
Artoe
"studem! (4)
Trigger: Eq. (6)
Mission 1: Foster Seaching-to-Lente
Occupancy Map
(eachee
PHY-Teacher: Theorem 5.2
Sensing
Proprioception
Locomotion cotrol commands
BEV Map
Exteroception
Figure Integration of Real-DRL with Navigation: A seamless integration of perception, planning, learning, and control is essential for robotic operation in complex environments. The sensing module collects both proprioceptive and exteroceptive data from the robot navigates challenging surroundings. The navigation module constructs Bird's Eye View (BEV) map by filtering Regions of Interest, generates an occupancy map using height thresholding. and computes cost map with the Fast Marching method. Using this cost map, planner generates planned motions for the locomotion controller. the locomotion learning and control module, the trigger facilitates interaction between the PHY-Teacher and the DRL-Student to ensure the robot's 's safety. This setup allows the DRL- Student learn from the PHY-Teacher, while also engaging self-learning to develop a safe and peg--performance locomotion policy.
31

---

J.1 Learning Configurations and Computation Resources
The observation space is defined | where z represents the exteroceptive observation and s indicates the proprioceptive tracking-erne error vector. A waypoint is considered successfully reached when the Euclidean distance between the robot's center of mass and the target waypoint is less than 0.3 meters.
The actor and critic networks implemented as Multi-Layer Perceptrons with four fully connected layers. The output dimensions are structured as follows: the critic network has output sizes of 256, 128, 64, and I, while the actor network has output sizes of 256, 128, 64, and The input for the critic network consists of both the observation vector and the action vector, while the actor network takes only the observation vector input. The activation functions used in the first three layers of both networks are ReLU, while the output layer of the actor network employs the Tanh function, and the critic network uses a linear activation function. Additionally, the discount factor y set to 0.9, and both the critic and actor networks have learning rates of 0.0003. Finally, the batch size is configured to be 512.
For the computational resources, we utilized a desktop running Ubuntu 22.04, equipped with a 13th Gen Intel@ CM 19-13900K 16-core processor, 64 GB of RAM, and an NVIDIA GeForce GTX 4090 GPU. The algorithm was implemented in Python using the Py PyTorch framework, and the MATLAB LMI toolbox was employed for solving real-time patches.
J.2 Safety Conditions
The Real-DRL is designed to track the planned movements generated by the navigation mod- ule while ensuring that the system's 's states remain within predefined safety
This is cru-
cial for avoiding falls and collisions with obstacles. The safety set is defined as follows: {s R100 |CoM x-velocity Tv. S 1.5 m/s, |CoM z-height r=l 0.6 Jraw Trw 0.8 rad, |pitch- - Tithhh I 1 rad]. The action space is defined as: A = {a E R a
[25, 25, 25, 50, 50, 50)50]] 50] These formulations ensure that the system operates within safe parameters
while effectively following the intended navigation paths.
J.3 PHY-Teacher Design
The physics model knowledge (A(s( B(s())) used for the design of
is described
in Equation (2).. Given sampling period T 30 second, set 0.0008 satisfy the conditions outlined in Assumption 5.1 Additionally, the following parameters are defined: x = 0.15, a = 0.9, 17 0.6, 8 = 0.3, and
0.2, ensuring that the inequalities in Equation (2) are upheld.
J.4 DRL-Student Design
In this experiment, our DRL-Students build upon two state-of-the-arr safe DRLs: CLF-DRL, as proposed in (55), and Phy-DRL, outlined in [13 [2]. A critical difference is CLF-DRL has purely data-driven action policy, while Phy-DRL has a residual action policy:
Astudent (k)
Adr (k)
Aphy (k: (= F s(A)),
data-driven
model-based
where F is given in Equation 73 comparison of these models will be conducted, with all other configurations remaining the same, as detailed below.
J.4.1 Reward
The reward for the DRL-Student primarily focuses on three key aspects: robot safety, travel time, and energy efficiency.
Safety. We consider Control--yapunovv reward in [55] which incorporates both safety and stability considerations:
Reaeety = =e T() e(t)) - e (t 1) P. + 1),
(76)
where P is given in Equation 75
32

---

Travel Cost. The Euclidean distance between the robot and the waypoint is defined as:
d(t) = [(x((t) Pupll2
(77)
where Puop is the location of next waypoint, and &5 the position the robot base the world frame. Inspired by 225 and [1 the navigation reward is defined as:
Reavigation C1 (di + fup + fobs
(78)
where Pais (t) d(t) d is the reward for forwarding the waypoint. fup e X-
- A-Treuch rewards the robot as reaches the waypoint. À E (0, 1) is a time decay factor and Treach denotes the time step when the waypoint is reached. The Fobs serves as penalty when the quadruped collides with the obstacles. C1 and 02 are used hyperparamters. This reward structure incentivizes the robot to achieve an efficient and collision-free navigation through the waypoints.
Energy Consumption. Power efficiency remains major challenge for robots outdoor settings. We model the motor as non-regenerative braking system (58
Pmotor
max
Wm
Loopper
0}
(79)
output power
heat dissipation
where Tm and Wm are motor's torque and angular velocity respectively. Leopper is copper loss coefficient. Taking Cm as hyperparameter, we define the energy consumption reward:
Renergy = -Cn Pmotor
(80)
The ultimate reward function, designed to guide the DRL-Student in learning safe and high- performance policy as defined in Equation (9). is given by integrating the components from above Equations 76). 78) and 180)..... (s(t), astuden(()) = Ratety + Ruavigation + Resergy + ē Ruxillirr. where auxillary denotes the auxiliary reward for tracking velocity and orientation, with small coefficient promote smooth locomotion. In experiment, we choose e1 5, c2 10, C. 0.02 and 2 = 0.5.
J.4.2 Self-Learning Space
The parameter used define the self-learning space for DRL-Student, as mentioned in Equation is set at 17 = 0.6.
J.4.3
Safety- Batch Sampling
For the safety-informen batch sampling in Equation we let P1 0.01 and P2 0.05
J.5
Ablation Experiment
We recall that the model knowledge of PHY-Teacher, along with its resulting action policy and control goal, is dynamic and responsive complex and dynamic operating environments in real-time. In contrast, the physics or dynamics models used in fault-tolerant DRLs, such as neural Simplex 43], runtime assurance [8 50 [6], and model predictive shielding 3. static. This difference suggests that current fault-tolerant DRLs struggle to guarantee safety in complex and highly dynamic environments. To illustrate this point, we compare the PeY-Teachers with dynamic (real-time) and static models in an unstructured terrain filled with random stones. The comparison video can be found at https://www., youtubeu com/shorts/Ct xXlohSiM. This video demonstrates that when using the static model, the PHY-Teacher cannot guarantee safety in the environment. In contrast, with the real-time dynamic model, our PHY-Teacher can not only guarantee safety but also withstand unexpected sudden kicks.
33

---

K
Experiment Design: Cart-Pole System
We use the cart-pole simulator available in OpenAl Gym [ to emulate the real system, allowing the deployed Real-DRL lear from scratch. The unknown unknowns introduced in this scenario consist disturbances that affect the DRL-Student' 's action commands and the friction force. These disturbances are generated using randomized Beta distribution. In Appendix H explain why the randomized Beta distribution serves as a form of unknown unknown.
K.1 Learning Configurations and Computation Resources
The actor and critic networks are designed as Multi-Layer Perceptrons consisting of four fully connected layers. The output dimensions for the critic and actor networks are 256, 128. 64, and respectively. The activation functions for the first three layers are RcLU, while the final layer of the actor network uses the Tanh function and the critic network employs linear activation function. The input to the critic network is [s; al. which combines the state and action, while the actor network takes only the state s as its input. We have set the discount factor y to 0.9, and both the critic and actor networks have the same learning rate of 0.0003. The batch size is = 512, and each episode consists of 1,000 steps, with sampling frequency of 50 Hz.
For the computational resources, we utilized a desktop equipped with Ubuntu 22.04, 12th Gen Intel@ TM 19-1 1990K 16-core processor, 64 GB of RAM, and an NVIDIA GeForce GTX 3090 GPU. The algorithm was implemented in Python using the TensorFlow framework, and we employed the Matlab LMI toolbox for solving real-time patches.
K.2 Safety Conditions
The objective of Real-DRL is to stabilize the system at the equilibrium point s* 00,0,00 0, 0] while
ensuring that the system states remain within the safety set S { RR |2| S |0] 1.0]. The action space is defined as A E R| < 50).
K.3 PHY-Teacher Design
The physics- model knowledge about the dynamics of cart-pole pole systems for the design of PHY-Teacher from the following dynamics model presented in (21]:
g sin 0 + cos 0 F= mmmm...0 16 sin
m
(81a)
( mp cos
m.+mpp
F + mnd (02 sin - cos
(81b)
m. + mp
whose physical representations the model parameters can be found in following Table
Table 3: Notation table
Notation
Value
Unit
me
mass of cart
0.94
kg
mp
mass of pole
0.23
kg
gravitational acceleration half length of pole
9.8
s
0.32
m
F
actuator input position cart velocity of cart angle of pole
[-50, 50]
N
x
[-1, 1]
m
[-3,3]
m 8-18
[]1,
rad
angular velocity of pole
[-4.5,
rad s
34

---

K.3.1
Physics-Model Knowledge
The PHY-Teacher converts the dynamics model
81 into state-space model as follows:
x
sin O cos
$ m, 06 (m + m c08
d
Ac
e F (m. + c06
dt
0
gsin mm m
my sin 0 A0
10 (m, m1 p cos
(m.++mp))).. m m
((s)
0
(m, tmg
m muuss
(82)
0
[ (M +m m cos
((s)
where Ass) and (s)) are known the PHY-Teacher. The sampling technique transforms the continuous-time dynamics model (2 to the discrete-time one:
s(k 1) (4 + As)) s(k) +T B(s) a(k),
from which we obtain the model knowledge (s(k)) and (ss())) in Equation 17 as
(s())) I. + (s(k)), B(s(k)) = T B(s(k)),
(83)
where 7 second, i.e., the sampling frequency is 50 Hz.
50
K.3.2 Parameters for Real-Time Patch Computing
Given that T second, to satisfy Assumption 5.1 let 0.0008. Additionally, the parameters are defined as follows: = 0.3, a = 0.9, 0.7, = 0.5, and Φ = 0.01. These values chosen to ensure that the inequalities stated in Equation (22) hold true.
K.4 DRL-Student Design
In the ablation studies, the DRI--Sudent builds on the CLF-DRL proposed in 55 CLF reward in this experiment is
T (sk),, Astudent (I:) = (s (k) s-k( s (k+ + 1) s + 1)) n - 2 (kc)
722,
where we let ?1 1, 722 0.015, and
54.113417860698 96.26005996337755
61.7975412804211
12.995941 82581 26
26. 26005923377755
34.6710819094179
7.273215838 58381861
P
61. 277541
34.6710819094177
88. 73944866456266
15.0856894519164
12. 22.9559
7.27321583818861
18.0856894519164
3.83961074325448
The parameter for defining DRL-Student's self-learning space in Equation is 1 - 0.7.
The ablation studies involve experimental comparisons of our Real-DRL models with and without the teaching-to- -o-larr mechanism. When the teaching-to to-lammtt- mechanism is used, the DRL-Student utilizes both self-learning replay buffer and teaching- to-learo replay buffer, with each buffer con- figured to hold maximum of 100,000 instances. Conversely, when the seaching-o--lcae mechanism not employed, the DRL-Student relies single replay buffer to store self-learning experience data. To ensure fair comparison, the size of this single buffer is set to match the combined capacity of the two buffers, totaling 200,000.
K.S Auxiliary Experiment
This section provides additional experimental results that complement the main ablation study findings.
35

---

Testing Environment
Runtime Learning Environment
Friction cuefficient of cart-roud: 40
Friction coelficient of curt-rond: 10
Figure 10: Learning and testing environments. The learning environment has corner case (pink area), which is also the testing environments.
K.S.1 Safety-Informed Batch Sampling
For the safety-informed batch sampling in Equation (14), we let p1 and P2 0.1. demon- strate the proposed create a learning environment characterized by this imbalance, as shown Figure [O] Specifically, the brown area closely resembles the pre-training environment of the DRL-Student, representing majority the continual learning environment for Real-DRL, accounting In contrast, the gray area indicates a corner case that significantly differs from the DRL-Student's pre-training conditions, comprising only 10% of the total runtime learning cases. This corner-case environment also serves as the testing environment for the DRL-Student after it has completed continual learning.
batch sampling method for addressing experience imbalance, we
90% of the cases.
0
-100
-200
Return
-300
440
-500
w/o SBS
w/ SBS
20
40
60
80
100
120
140
Episode Number
Figure 11: Return trajectories of DRL-Student.
Figure shows the return trajectories of the DRL-Student, which were learned using the Real-DRL framework. It compares the results obtained with our proposed infery-i batch sampling method (referred to as "w/ SBS') to those obtained without this method (denoted as "w/o SBS'), which utilizes only single united reply buffer. Figure [ shows that the tested DRL-Students, which were obtained by disabling PHY-Teacher, are selected from both Real-DRI. models once their learning processes have reached convergence. This can ensure fair comparisons.
36

---

K.5.2 Teaching-to-Learn Paradigm
To evaluate the effect of the Bacching-to-leane paradigm policy learning, we introduce the metric of DRL-Student's episode-average reward, defined below.
Return (i.e., cumulative reward) in one episode
Episode-Average Reward
(84)
DRL-Student's total activation time in one episode
K.5.3 Automatic Hierarchy Learning
To further demonstrate PHY-Teacher's contributions to the claimed automatic hierarchy learning mechanism, we define the metric of PHY-Teacher's activation ratio as follows.
PHY-Teacher's activation ratio PHY-Teacher's total activation times in one episode
(85)
one episode length
which has value range of [0. 1]. A ratio of means PHY-Teacher never activated throughout the entire episode of learning, while ratio of means PHY-Teacher completely dominates DRL-Student for the entire episode.
Given five random initial conditions, the phase plots, each running for 1000 steps, are displayed in Figure Figure [2] Observing Figure
conclude that the DRI-Student has successfully learned from the DRL-Teacher how to ensure safety in episode 5: its action policy effectively confines the system states the safety set (indicated by the red area). In episode 50, the PHY-Teacher is rarely activated by the condition in Equation (6), as illustrated in Figure 6 Meanwhile, the action policy of the DRL-Student demonstrates improved mission performance, achieving faster clustering and much closer proximity to the mission goal, which is located at the center of the safety set.
1.0
0.5
0.0
-0.5
x
-1.0
Initial State
-1.5
Terminal State: Episode
Terminal State: Episode 50 Mission Goal
-2.0
1. 1.00-0.75-0.50-001 0.00 0.25 0.50 0.75 1.00
0
Figure 12: Demonstrating automatic hierarchy learning by phase plots, running over five random initial states within the safety set.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims accurately reflect the contributions and scope of the paper, as demonstrated in the Experiment Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Remarks 5.3 and E. and Section
g
Mean-
while, the proposed and potential solutions are provided therein.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer: [Yes]
Justification: Detailed assumptions and proofs are provided; please refer to Assumption 5.1 and Appendices C
Cand D for more information.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendices [ to K present detailed experimental designs that can be used to replicate the results of the experiments. Appendices pesudo-codes for implementation.
and includes the detailed
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
39

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The source code has been submitted as supplemental material and will be disclosed to the public via GitHub. Besides, Appendices KJoooK/ppeeenn detailed experimental designs that can be used to replicate the experimental results, and Appendices B and include the detailed pesudo-codes for implementation.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines http:://nips.ccppull//eee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (htips:
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendices [
K include essential and detailed information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Figures 4 and and Table 1 for the statistical result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
40

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendices L.1 and K. for the detailed information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpss//eeeriis..cccoaii/fhhhhee
Answer: [Yes]
Justification: Completely conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no such issue, as does not involve human subjects, animals, privacy, social security.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
41

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper presents no such risks, as its application domain learning- enabled autonomous systems, and the "models" used are well-validated physies models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The developed Real-DRL system is based on three frameworks: DDPG, Phy-DRL, and CLF-DRL. For the quadruped robot experiments, we utilize Python-based framework with the real Unitree Al robot, as well as NVIDIA Isaac Gym for the Unitree Go2 robot. Additionally, experiments on the cart-pole system are conducted using OpenAl Gym. All sources are explicitly and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
42

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assett are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcode con.ddd.o//dttsetttt. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All new assets are well documented and submitted as supplemental material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve human subjects and animals. The applications are the starning-enabled autonomous systems.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects and animals.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
43

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not involve any use of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ttpss//eeuris..cccooocceeeeeess for what should or should not be described.
44


Reviews:
[{"review_id": "TyLrTxi0t3", "reviewer": "Reviewer_Pf5q", "summary": "The methodpresented in this paper, Real-DRL, is focused on don safety-critical learning for robotics applicaitons. Themethod is hierarchical with a \"teacher\" that guides a \"student\" policy to achieve safety, and a \"student\" focused on the control task.\nA Safety-Status-Dependent Batch Sampling technique  oversamples safety-critical situations during training, such that safety constraints are prioritized before optimizing for performance (by keeping the teacher in a larger set than the student. The method is validated experimentally on quadruped robots, and cart-pole systems with comparison to existing safe RL approaches and shows a superior performance.", "questions": "Currently, safety is maintained by addressing nearly unsafe, or safety-critical situation during learning. What would happen when the system encounters a disturbance not included as safety-critical (i.e., unforeseen disturbance)? Can it be detected as potntially unsafe - in this case, can the policy adapt or restrict itself of acting in the presence of that disturbance?\n\nThe physics-guided teacher relies on a model of the real system. What happens in the presence of uncertainties, mismatch, or drifts in this model with respect to the real system? How important is the model's fidelity? A study on parameter uncertainties could be useful to clarify this, in particular with respect to safety constraint violations.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The proposed framework is supported by solid experimental validation on simulated and real platforms, also in the presence of disturbances, and achieves superior performance in the demonstrated cases.\nThe paper is clearly written.\nThe method provides runtime assurance for safety.\n\nWeaknesses: The implementation complexity as compared to other methods is higher, due to the hierarchical nature of the method. Computational costs during training are not analysed. \nThe design an appropriate PHY-Teacher for each new task or robot requires domain expertise. \nThe approach lacks formal mathematical safety guarantees or proofs. This means that for safety critical systems, there is still a probability of violating safety.", "was_revised": true, "final_justification": "The real-time adaptive mechanism in Appendix F addresses the question about unforeseen deisturbances. I have previously misunderstood that there is no safety guarantee, therefore I am increasing the rating to 5.", "rebuttal": "---\n**Weaknesses: The implementation complexity as compared to other methods is higher, due to the hierarchical nature of the method. Computational costs during training are not analysed. The design an appropriate PHY-Teacher for each new task or robot requires domain expertise. The approach lacks formal mathematical safety guarantees or proofs. This means that for safety critical systems, there is still a probability of violating safety.**\n\n**Response:** We thank the reviewer for this thoughtful and multifaceted critique. We address each point below. \n* **Implementation Complexity:** The proposed framework introduces a hierarchical architecture that increases implementation complexity compared to end-to-end learning pipelines. This hierarchical design is intentional and offers several benefits. The hierarchical learning architecture enables clear separation between safety-critical control (managed by the PHY-Teacher) and task-specific learning (handled by the DRL-Student), providing enhanced modularity, flexibility, and safety supervision during real-world deployment. In practice, this separation significantly simplifies debugging processes and improves system interpretability—both critical requirements for safety-critical robotic systems, such as legged robots, UAVs, autonomous vehicles, etc. \n\n* **Computational Cost:** The PHY-Teacher and DRL-Student are designed to operate in parallel. The DRL-Student utilizes GPU acceleration and can be deployed on embedded hardware such as the NVIDIA Jetson series, while the PHY-Teacher handles real-time safety-critical tasks efficiently on the CPU. To minimize runtime overhead, we implement the `LMI-based solver (for automatic and rapid computation of PHY-Teacher's real-time action policies)` in C, which significantly improves both memory efficiency and computational speed. This implementation strategy makes the framework well-suited for deployment on edge devices with limited computational resources. \nWe provide a summary of computational resource usage across two platforms with varying CPU frequencies in the following table. Empirically, the C-wrapped PHY-Teacher achieves control update rates exceeding 50 Hz on desktop-class CPUs and over 20 Hz on embedded platforms with constrained computational resources. These frequencies are sufficient for high-level policy updates in a time-critical operating environment. Furthermore, the ability to run the PHY-Teacher efficiently across diverse CPU architectures demonstrates the portability and scalability of our design. \n| Platform | CPU Architecture | CPU Core | Memory Size | Runtime Memory Usage: Python | Runtime Memory Usage: C | LMI Solver: Python | LMI Solver: C |\n|---|---|---|---|---|---|---|---|\n| Dell XPS 8960  Desktop  | x86/64 | 32 | 64 GB |  52 MB |  `10 MB` |  44.6 ms | `18.2 ms` | \n| NVIDIA Jetson AGX Orin           | ARM    | 12 | 64 GB | 113 MB | `8.3 MB` |  116.7 ms | `42.4 ms` | \n\n\n* **Domain Expertise and Formal Mathematical Poof:**  The design of PHY-Teacher relies on the \"physics model,\" which refers to a general mathematical model derived from fundamental physics laws (e.g., Newton's laws and conservation of energy) that describe system dynamics. Building on the physics model, the PHY-Teacher is designed to foster the teaching-to-learn mechanism for the DRL-Student and provide safety backup when the DRL-Student behaves unsafely or unexpectedly. To achieve these objectives, we establish three fundamental requirements for the PHY-Teacher: (i) assured safety with a `rigorous theoretical guarantee (mathematical poof is in Appendix D)`, (ii) real-time action policies that can adapt to complex, dynamic, and unpredictable operating environments, and (iii) rapid and automatic computation of action policies in real time. These requirements can only be met when the PHY-Teacher possesses domain expertise in the physics model of system dynamics. According to our knowledge, the physics models of physical engineering systems are well-studied and are readily available. Examples include quadruped robots, drones, on-road and off-road vehicles, UAVs, humanoid robots, power systems, and aircraft. \n\n---\n\n**Question 1: Currently, safety is maintained by addressing nearly unsafe, or safety-critical situation during learning. What would happen when the system encounters a disturbance not included as safety-critical (i.e., unforeseen disturbance)? Can it be detected as potntially unsafe - in this case, can the policy adapt or restrict itself of acting in the presence of that disturbance?**\n\n**Response:** Our studies have addressed unforeseen disturbances, which belong to the category of unknown unknowns. \nPHY-Teacher is a physics-model-based design with verifiable safety. Its theoretical safety guarantee holds under Assumption 5.1, which establishes an upper bound on model mismatch. The model mismatch arises from unknown uncertainties, unmodeled dynamics, software and hardware faults, unforeseen external disturbances, and other factors that are not accounted for in the nominal model used by PHY-Teacher. This means that our PHY-Teacher can deliver rigorous safety guarantee as long as Assumption 5.1 holds. \n\nIn real-world applications, particularly in hard-to-predict and hard-to-simulate environments (e.g., natural disasters and snow squalls), we cannot always guarantee that Assumption 5.1 will hold. To address this challenge, we propose a real-time adaptive mechanism in Appendix F that employs real-time state samplings to estimate the real-time model mismatch, thereby monitoring the validity status of Assumption 5.1. If the assumption is violated, compensatory actions are computed in real-time to counteract the model mismatch and restore the validity of Assumption 5.1. \n\nWithin our framework, any disturbances, faults, or unknowns that lead to the detection of approaching the marginal-safety boundaries are deemed a safety violation. This safety violation triggers the PHY-Teacher to take over DRL-Student to steer the system back toward the safe self-learning space.\n\nOur paper validated the robustness of PHY-Teacher through two comprehensive experiments:\n\n* **Simulator:** In IsaacGym, we evaluate the quadruped robot under external disturbances such as random pushes while traversing uneven, unstructured terrain populated with movable stones (see Section I.5).\n\n* **Real World:** In real-world experiments on the real quadruped robot, we introduce various common unforeseen disturbances, including lateral pushes, kicks, random payload drops, and DDoS faults to emulate system or communication delays  (see Section 6.1).\n\nIn the experiments, PHY-Teacher successfully maintains system safety, demonstrating its ability to tolerate the environmental uncertainties, model imperfections, unforeseen disturbances, and platform faults. \n\n---\n\n**Question 2: The physics-guided teacher relies on a model of the real system. What happens in the presence of uncertainties, mismatches, or drifts in this model with respect to the real system? How important is the model's fidelity? A study on parameter uncertainties could be useful to clarify this, in particular with respect to safety constraint violations.**\n\n**Response:** The design of the physics-guided teacher utilizes the physics model:\n$\\mathbf{s}(t+1) = \\mathbf{A}(\\mathbf{s}(t)) \\cdot \\mathbf{e}(t) + \\mathbf{B}(\\mathbf{s}(t)) \\cdot \\mathbf{a}(t) + \\mathbf{h}(\\mathbf{s}(t))$, where $\\{\\mathbf{A}(\\mathbf{s}), \\mathbf{B}(\\mathbf{s})\\}$ represents the available knowledge from the physics model used by the teacher. Additionally, $\\mathbf{h}(\\mathbf{s})$ denotes unknown model uncertainties, parameter uncertainties, unmodeled dynamics, and external disturbances that are not captured by the nominal model knowledge. Consequently, the design and its theoretical foundation explicitly account for uncertainties, mismatches, and drifts in the physics model that relate to the real system, summarized below. \n\n* **Theoretical Safety Guarantee:** The teacher's theoretical safety guarantee is valid under Assumption 5.1, which establishes an upper bound on model mismatch: $\\mathbf{h}^\\top \\cdot \\mathbf{P} \\cdot \\mathbf{h} \\le \\kappa$. This indicates that the physics-guided teacher can deliver rigorous safety assurance as long as Assumption 5.1 is satisfied. However, if Assumption 5.1 does not hold, the teacher can still ensure safety, though it cannot provide a theoretical proof for this. Therefore, a high-fidelity model is always preferred, as it guarantees the validity of Assumption 5.1 and, consequently, the theoretical safety guarantee outlined in Theorem 5.2. \n\n* **Practical Challenge and Approach:**  In reality, the dynamics of learning-enabled autonomous systems are governed by three categories of knowledge: known knowns (e.g., Newtonian physics), known unknowns (e.g., Gaussian noise without identified mean and variance), and unknown unknowns. One significant example of an unknown unknown arises from DNN's colossal parameter space, intractable activation functions, and random factors. Moreover, many autonomous systems, such as quadruped robots and autonomous vehicles, exhibit complex dynamics-environment interactions. Unpredictable and complex environments, such as freezing rain conditions, can introduce new types of unknown unknowns. Given these challenges, it is not practical to expect that a high-fidelity model exists that can guarantee Assumption 5.1 always holds in real-world scenarios. To address this challenge, the paper introduces a real-time adaptive mechanism that utilizes real-time state sampling to estimate model mismatches in real-time. This allows for monitoring the satisfaction of Assumption 5.1. When this assumption is violated, the mechanism calculates compensatory actions in real-time to counteract the model mismatch and restore the validity of Assumption 5.1. Due to the nine-page limit of the submitted paper, the details of this proposed mechanism are provided in Appendix F.\n\n---"}, {"review_id": "c0R1vJYnFT", "reviewer": "Reviewer_gyUQ", "summary": "This paper proposes Real-DRL, a framework enabling runtime learning of a deep reinforcement learning (DRL) agent with safety assurance and high-performance learning, which improves on existing DRL frameworks' abilities to provide safety guarantees especially in the face of corner cases and the sim2real gap. The framework has 3 components: DRL-Student, PHY-Teacher, and Trigger. DRL-Student is the agent being trained and only acts within a predefined subset of the \"safe\" state space. PHY-Teacher is responsible for taking control of the actions when DRL-Student navigates too close to the \"unsafe\" space and provides samples of its own experience returning the plant back to the \"safe\" region of the state space to DRL-Student to learn from. Trigger is the monitoring component that determines when to switch between DRL-Student and PHY-Teacher.", "questions": "Why does the framework perform so poorly without the safety-status-dependent batch sampling method? I would expect safety performance to be worse without it, but every trajectory in the phase plot appears to enter the unsafe region. Are none of these samples from PHY-teacher being used at all when evaluating without the safety-status-dependent batch sampling method?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The general problem studied is significant and interesting, with overall a rigorous development of the results presented for the developed framework\n- Most writing of the paper is clear and they provide ample intuition for how their framework works (albeit there are some caveats noted below).\n- Experiments are interesting and broad, and the results appear to beat SOTA in some cases.\n \nWeaknesses:\n- Some equations are missing references to the variables in them and therefore are not clear or intuitive at all (e.g., the Real-Time Patch in equation 15 or equation 21 and 22).\n\n- The main motivation for combining learning and control is to avoid the strong assumption of fully known Markovian dynamics. In general, I am not fully convinced learning-based control techniques while still assuming the dynamics are known is reasonable. Assuming a known model significantly limits the novelty and practical impact of the contribution. \n\n- Unlike standard reinforcement learning algorithms that do not assume model access, the authors rely on having access to the model to design PHY-Teacher. This makes comparisons with other techniques challenging.\n\n- Regarding the definition of the trigger’s role: How can you guarantee that the student policy will be able to bring the system back into the learning-enabled region when its actions are restricted to the set A? This is a critical challenge that must be addressed. Encountering unseen data may lead to large errors that push the system far outside the learning-enabled region, making recovery infeasible or requiring prohibitively large control inputs. This issue is especially likely if the system dynamics are highly nonlinear.\n\n- The tracking error dynamics in Equation 17 are assumed to be linear, which is a very strong and potentially unrealistic assumption.\n\n- While the authors present strong and compelling plans for addressing unknown unknowns, they do not provide equally convincing arguments that their controllers—PHY-Teacher and DRL-Student—will reliably behave as intended in practice. Although the experimental results are promising, they depend on having access to the model during training.\n\n- \"reply\" should be \"replay\" on line 213 (pg. 6).", "was_revised": true, "final_justification": "Based on the response and discussion, assuming the authors clarify what safety means in this context and given how the discussion has focused much on safety/verification, I am in favor of acceptance, along with the condition the authors release their code that will help further clarify what safety means in this context.", "rebuttal": "---\n**Weakness 1: Some equations are missing references to the variables in them and therefore are not clear or intuitive at all (e.g., the Real-Time Patch in equation 15 or equation 21 and 22).**\n\n**Response:** We thank the reviewer for bringing this issue to our attention. The references to the variables in Equations (15), (19), and (20) are summarized in Lines 254-256 and Remark 5.3 of the paper. Due to the 9-page limit, more detailed descriptions of these variables, along with their physical representations, are provided in Appendix E. Since the accepted paper allows for one additional content page in the camera-ready version, we will place the variables' physical interpretations directly after Equations (15), (19), and (20), if our paper is accepted.\n\n---\n\n**Weakness 2: The main motivation for combining learning and control is to avoid the strong assumption of fully known Markovian dynamics. In general, I am not fully convinced learning-based control techniques while still assuming the dynamics are known is reasonable. Assuming a known model significantly limits the novelty and practical impact of the contribution.**\n\n**Response:**  The aims of this paper extend beyond merely eliminating the assumption of fully known Markovian dynamics. \n* **Motivation:** In today's world, there are two approaches to achieve the same control tasks: a high-dimensional ML approach (e.g., DRL), and a low-dimensional, physics-model-based approach (e.g., adaptive control).  The ML approach often delivers superior task performance but lacks analyzable and verifiable behavior. In contrast, the physics-model-based approach enables analyzable and verifiable stability and safety, but has limited task performance due to model mismatch.  This dichotomy motivates our Real-DRL: integrating these two approaches to overcome their respective limitations while preserving their advantages.\n\n* **Novelty and Practicality:** The physics model in our paper refers to a general mathematical model derived from physics laws (e.g., Newton's laws and the conservation of energy) for describing the dynamics of a system. Using this physics model does not detract from the novelty of the proposed Real-DRL; rather, it enhances Real-DRL by featuring: i) verifiable safety, ii) automatic hierarchy learning, and iii) safety-status-dependent batch sampling to address the experience imbalance. Moreover, this requirement does not limit the practical impact of Real-DRL, because the physics models for safety-critical autonomous systems are readily accessible. \n\n---\n\n**Weakness 3: Unlike standard reinforcement learning algorithms that do not assume model access, the authors rely on having access to the model to design PHY-Teacher. This makes comparisons with other techniques challenging.**\n\n**Response:** At a high level, our proposed Real-DRL belongs to the emerging topic of Physical AI, infusing domain knowledge in physics into data-driven ML. As a result, our RL is inherently different from standard RL frameworks. Comparing our Real-DRL with other techniques is straightforward (not challenging). This is because recent efforts have been devoted to physical RL, e.g.,  Phy-DRL, neural Simplex, and runtime assurance. This paper includes such comparisons. \n\n---\n\n**Weakness 4: Regarding the definition of the trigger’s role: How can you guarantee that the student policy will be able to bring the system back into the learning-enabled region when its actions are restricted to the set A? This is a critical challenge that must be addressed. Encountering unseen data may lead to large errors that push the system far outside the learning-enabled region, making recovery infeasible or requiring prohibitively large control inputs. This issue is especially likely if the system dynamics are highly nonlinear.**\n\n**Response:** Our design and theoretical foundation have taken the issue of large unforeseen errors or faults into account. \n\n* **Theoretical Guarantee and Adaptive Mechanism:** PHY-Teacher is designed to have verifiable expertise in safety protocol, as formally stated in **Theorem 5.2**. When the system controlled by the DRL-Student approaches a marginal-safety boundary $\\partial\\mathbb{L}$, the Trigger activates the PHY-Teacher to take over the DRL-Student to steer the system back to $\\mathbb{L}$.  This safety protocol has a theoretical safety guarantee and is valid under Assumption 5.1, which establishes an upper bound on model mismatch. The model mismatch therein can arise from unknown uncertainties, errors, unmodeled dynamics, and disturbances not captured in the nominal physics model. In practice, especially when confronted with unpredictable significant errors, faults, or disturbances, it is unrealistic to expect Assumption 5.1 always to hold. To address this challenge, the paper introduces an adaptive mechanism (detailed in Appendix F) that utilizes real-time state samplings to estimate the real-time model mismatches, enabling the monitoring of the validity of Assumption 5.1. When the assumption is violated, the mechanism computes compensatory actions in real-time to mitigate the effects of model mismatch and restore the validity of the assumption.\n\n* **Buffer Zone:** In our Real-DRL, the Trigger monitors the system's real-time safety status to manage interactions between DRL-Student and PHY-Teacher. However, the safety set $\\mathbb{S}$ cannot be directly utilized to indicate the safety status. This is because when the system approaches its boundaries, the PHY-Teacher cannot instantaneously steer the system back into the safety set due to inherent system delays, especially in the face of large errors, faults, or disturbances.  These delays are caused by factors such as communication latency, sampling periods, actuator response times, etc. To address this issue, we introduce the self-learning space $\\mathbb{L}$ that ensures $\\mathbb{L} \\subset \\mathbb{S}$, creating a buffer zone: $\\mathbb{S} \\setminus \\mathbb{L}$. Meanwhile, the Trigger refers to the boundary $\\partial\\mathbb{L}$ to proactively manage interactions between the DRL-Student and PHY-Teacher. The configuration provides adequate margins for system response time and decision-making latency. \n\n\n\n---\n\n**Weakness 5: The tracking error dynamics in Equation 17 are assumed to be linear, which is a very strong and potentially unrealistic assumption.**\n\n**Response:** Perhaps the formula format in Eq. (19) impressed the reviewer that it is linear, which is not true. Demonstration examples of highly nonlinear dynamics are presented in Appendix H.3 and Appendix J.3 of the paper. \n\n---\n\n**Weakness 6: While the authors present strong and compelling plans for addressing unknown unknowns, they do not provide equally convincing arguments that their controllers—PHY-Teacher and DRL-Student—will reliably behave as intended in practice. Although the experimental results are promising, they depend on having access to the model during training.**\n\n**Response:**  Our `Response to Weakness 2` explains that the dependence on physics models enhances rather than limits its novelty and practical impact. Our `Response to Weakness 4` explains that our Real-DRL, integrating PHY-Teacher and DRL-Student, explicitly addresses the safety challenge due to unknown unknowns, from theory to real-world implementation. \n\n---\n\n**Weakness 7: \"reply\" should be \"replay\" on line 213 (pg. 6).**\n\n**Response:**  We have corrected it. \n\n---\n\n**Questions: \"Why does the framework perform so poorly without the safety-status-dependent batch sampling method? I would expect safety performance to be worse without it, but every trajectory in the phase plot appears to enter the unsafe region. Are none of these samples from PHY-teacher being used at all when evaluating without the safety-status-dependent batch sampling method?**\n\n**Response:**  In the paper,  Figure 6 includes 10 trajectories: 5 for learning with the safety-status-dependent batch sampling scheme (denoted as `w/ SBS` ) and 5 for learning without this scheme (denoted as `w/o SBS`). Perhaps due to the color scheme, the phase plots impressed the reviewer with a contradictory conclusion. Actually, the results meet our expectations entirely: the trajectories denoted by `w/o SBS-1 - w/o SBS-5` all enter the unsafe region. In contrast, the remaining `w/ SBS-1 - w/ SBS-5` represent ours, and none of them enter the unsafe area.\n\nThe integrated architecture `Dual Replay Buffers + Safety-Status-Dependent Batch Sampling` achieves enhanced learning performance—its absence results in significantly degraded performance. In addition to facilitating a dual teaching-to-learn and self-learning paradigm, the architecture also significantly addresses uneven experience distribution (due to corner but safety-critical cases) in learning. \n\n* **Dual Replay Buffers:** `Figure 1` in the paper shows that teaching-to-learn experience data is acquired in the marginally safe space, capturing corner-case experiences. However, existing DRLs typically employ a single replay buffer and random batch sampling. As a result, corner-case data is often omitted entirely, due to the lower probability of selection. Our dual replay buffers address this limitation by storing the corner-case and normal data separately and specifically. This design ensures that batch samples consistently incorporate corner-case data, thereby maintaining their representation during learning.\n\n* **Safety-Status-Dependent Batch Sampling:**  In the paper, `Figure 2`, `Theorem 4.1`, and `Eq. (14)` reveal that the contribution of corner-case experience data increases proportionally with the real-time safety status. For instance, the $V(s(t)) = 1$ indicates that the real-time states are approaching safety boundaries. Consequently, all batch samples are drawn from the marginally safe space (of the corner cases) stored in the teaching-to-learn replay buffer. This focuses the learning on corner-case experiences, precisely when the real system is marginally safe.\n\n---"}, {"review_id": "TEl5jP3NtF", "reviewer": "Reviewer_BbAN", "summary": "This work introduces Real-DRL, a hybrid runtime learning architecture for safety-critical systems composed of a DRL-Student (learning high-performance policies), a PHY-Teacher (guaranteeing safety via physics-based backup control), and a Trigger that switches control based on safety conditions. The goal is to deal with the unknown unknowns and the sim2real gap.\n\nThe manuscript seems to be a mix of describing the proposal in the context of a controlling a plant, and some examples in other domains, which is confusing. If the focus is on safe controlling of plants, examples in this domain should be provided. Otherwise, remove the excessive \"plant\" indication.", "questions": "- Why does the DRL-Student adopt an actor-critic architecture? Are other architectures possible?\n- Why mentioning so many times \"plants\" but provide no example of real application to a plant?\n- Cannot the proposal be applied in general and remove the mention to plants in places where it is not necessary?\n- How robust is the PHY-Teacher under model mismatch or poorly characterized environments? This is address in Annex F, but would be good to mention it in the main text.\n- Could the PHY-Teacher be replaced by a simpler rule-system? Or is the \"physic-model\" a generalization or any system that can provide control actions?\n- Is the PHY-Teacher ever trained? If so, how? If not, does it need to be formally described for each problem?", "limitations": "The manuscirpt does not highlight any limitation of the proposal, but I have listed several ones (or at least, questions for which I cannot find a proper answer) in my previous comments.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "STRENGHTS:\n- Verifiable safety of DRL systems is a must these days, and this is a work in that direction.\n- The proposal is formally described. \n- Experiments are solid and they are very well described in the Annex.\n\n\nWEAKNESSES:\n- The manuscripts mentions, since the very beginning, \"plants\" (which?). But given examples have nothing to do with them. Why such a focus on plants? Cannot the proposal be described in general without that focus?\n- The proposal has a heavy reliance on PHY-Teacher performing ok at all times. This might not always be possible to guarantee in real scenarios.\n- The novelty, besides the concrete implementation details, is limited compared to existing Fault-tolerant DRL that uses HPM and HAM.", "was_revised": true, "final_justification": "It’s clear that this is a solid piece of work, and I appreciate the effort the authors have put into identifying and addressing potential drawbacks in the annexes and in the rebuttals.\n\nThe only drawback that I can currently see in the proposal is that lack of generality as the PHY-Teacher needs to be defined/tweaked for each domain (cannot be done automatically). Said that, if you are focused in just one domain, you define it once and forget about it. \n\nIn any case, I am increasing my score due to the thorough responses to me and my fellow reviewers.", "rebuttal": "---\n\n**Weakness 1: The manuscripts mentions, since the very beginning, \"plants\" (which?). But given examples have nothing to do with them. Why such a focus on plants? Cannot the proposal be described in general without that focus?**\n\n**Response:** In the communities of control, robotics, and cyber-physical systems, the term \"real plant\" (often referred to as \"plant\") refers to the actual physical system, process, or device that is being controlled or manipulated. The \"real plant\" can be anything from a DC motor, robotic arm, or chemical process to a vehicl. It is a generic term that signifies \"the thing to be controlled.\" So, in our real-world experiments, the real quadruped robot serves as the \"real plant.\" In our mimicked real-world experiments, the quadruped robot from Nvidia Isaac Gym and the cart-pole system from OpenAI Gym represent the \"real plants.\" \n\nTo address the reviewer's concern and avoid potential confusion, we have replaced the \"real plant\" with \"real system\" in the revised paper.\n\n---\n\n**Weakness 2: The proposal has a heavy reliance on PHY-Teacher performing ok at all times. This might not always be possible to guarantee in real scenarios.**\n\n**Response:** Our Real-DRL framework does not always have heavy reliance on PHY-Teacher. We provide the following summaries of the learning journal and teaching-to-learn process of the DRL-Student to clarify this point.\n\n* **Learning journal of DRL-Student.**  PHY-Teacher is activated only when DRL-Student behaves unsafely, leading to a safety violation of the system. When the system returns to normal, PHY-Teacher returns the control role to DRL-Student. The Trigger in our Real-DRL framework orchestrates their interactions, assuring system safety. Under the management of Trigger, the DRL-Student's learning journey involves acquiring knowledge from the PHY-Teacher (i.e., teaching-to-learn), who possesses verifiable domain expertise in safety protocols. As learning deepens, the DRL-Student gradually becomes `independent` of the PHY-Teacher, ultimately achieving `self-mastery` upon acquiring safe and high-performance action policies.\n\n* **Continuous teaching-to-learn, independent of PHY-Teacher activation!** The DRL-Student continuously acquires safety knowledge from PHY-Teacher through our proposed `dual teaching-to-learn and self-learning replay buffers + safety-status-dependent batch sampling`. This design ensures  DRL-Student has continuous access to PHY-Teacher's safety experiences stored in the teaching-to-learn replay buffer, allowing the DRL-Student to refine its policy while maintaining safety guarantees. The teaching-to-learn mechanism thus operates independently of PHY-Teacher activation status, providing persistent safety learning.\n\n---\n\n**Weakness 3: The novelty, besides the concrete implementation details, is limited compared to existing Fault-tolerant DRL that uses HPM and HAM.**\n\n**Response:** Our research aims to advance verifiably safe Deep Reinforcement Learning (DRL) for `safety-critical` autonomous systems, including legged robots, drones, UAVs, autonomous vehicles, and aircraft. This topic encompasses two primary directions: 1) safe DRL and 2) fault-tolerant DRL. Our paper provides a comprehensive comparison of these approaches, examining their motivations, underlying assumptions, design principles, and experimental implementations.\nSection 1.2 summarizes the fundamental differences in motivations and assumptions between these approaches. For implementation and experimentation, Sections 6.1 and 6.2 present comparisons with recently published state-of-the-art safe DRL frameworks, including Phy-DRL (which claims both theoretical and experimental fast training towards safety guarantee) and CLF-DRL (a seminal CLF-like reward function towards verifiable stability and safety). Additionally, Section 6.2 compares the notable fault-tolerant DRL frameworks, including neural Simplex and runtime assurance frameworks.\n\n---\n\n**Question 1: Why does the DRL-Student adopt an actor-critic architecture? Are other architectures possible?**\n\n**Response:** There are several key motivations behind our adoption of the actor-critic architecture.\n\nFirst, the actor-critic architecture provides natural support for continuous action spaces, which is essential for the safety-critical autonomous systems we consider, such as legged robots, drones, UAVs, autonomous vehicles, and many others. This capability is crucial given the continuous control requirements inherent in these applications.\n\nSecond, one of our core contributions—the conjunctive safety-status-dependent batch sampling mechanism and dual replay buffers—integrates seamlessly with the actor-critic framework's modular structure. This integration significantly enhances learning efficiency in task performance and safety assurance.\n\nThird, the actor-critic architecture represents one of the most widely adopted paradigms in deep reinforcement learning. Its balanced approach to policy optimization and value estimation forms the foundation for numerous state-of-the-art algorithms, including both on-policy methods (e.g., PPO, TRPO) and off-policy methods (e.g., SAC, DDPG, TD3).\n\nWhile we focus on actor-critic architectures in the paper, we note that the proposed mechanism is not architecturally constrained and could, in principle, be adapted to other deep reinforcement learning frameworks that employ similar modularity and replay-based training paradigms.\n\n ---\n\n**Question 2: Why mentioning so many times \"plants\" but provide no example of real application to a plant?**\n\n**Response:** In the communities of control, robotics, cyber-physical systems, and IoT, the \"real plant\" is a standard terminology signifying \"the system to be controlled.\" In our real-world experiments, the real quadruped robot serves as the real plant. In our simulated experiments, the quadruped robot from Nvidia Isaac Gym and the cart-pole system from OpenAI Gym represent the real plants within their respective environments. \n\n---\n\n**Question 3: Cannot the proposal be applied in general and remove the mention to plants in places where it is not necessary?**\n\n**Response:** We appreciate the reviewer's suggestion. To address the reviewer's concern and avoid potential confusion, we have replaced the terminology \"real plant\" with \"real system\" in the revised paper.\n\n---\n\n**Question 4: How robust is the PHY-Teacher under model mismatch or poorly characterized environments? This is address in Annex F, but would be good to mention it in the main text.**\n\n**Response:** PHY-Teacher is a physics-model-based design that focuses exclusively on safety-critical functions. Its theoretical safety guarantee is valid under Assumption 5.1, which provides an upper bound on model mismatch. The model mismatch arises from unknown uncertainties, unmodeled dynamics, and external disturbances that are not accounted for in the nominal model used during the design process.  In real-world environments, particularly in complex, dynamic, and unpredictable situations (such as wildfires, freezing rain, and snow squalls), we cannot always guarantee that Assumption 5.1 will hold. To address this challenge, we propose a real-time adaptive mechanism that employs real-time state sampling to estimate the current model mismatch, thereby monitoring whether Assumption 5.1 is satisfied. If the assumption is violated, compensatory actions are computed in real-time to counteract the model mismatch and restore the validity of Assumption 5.1. \n\nLogically and at a high level, the goal of the proposed adaptive mechanism is only to ensure that Assumption 5.1 remains valid by compensating for model mismatches in real-time, thereby ensuring that Theorem 5.2 holds.  Also, due to the nine-page limit of the submitted paper, we have moved the details of this proposed mechanism to Appendix F. The accepted paper is permitted to include one additional content page for the camera-ready version. If our paper is accepted, we will transfer this content from Appendix F to the main text.\n\n---\n\n**Question 5: Could the PHY-Teacher be replaced by a simpler rule-system? Or is the \"physic-model\" a generalization or any system that can provide control actions?**\n\n**Response:**  Safety-critical autonomous systems typically operate within continuous action and state spaces. Consequently, the actions of our PHY-Teacher are continuous. If a rule-based system can deliver verifiable safety, it could potentially serve as a replacement for the PHY-Teacher. To our knowledge, no research efforts have explored this direction.\n\nThe \"physics model\" in our paper refers to general mathematical models derived from fundamental physics laws (e.g., Newton's laws and conservation of energy) that describe system dynamics. Therefore, our PHY-Teacher design framework can be generalized to all physical engineering systems that have available physics models describing their dynamics.  This generalizability is demonstrated in our experiments, where we evaluated our framework on two distinctly different systems: a quadruped robot and a cart-pole system. \n\n---\n\n**Question 6: Is the PHY-Teacher ever trained? If so, how? If not, does it need to be formally described for each problem?**\n\n**Response:** PHY-Teacher is a physics-model-based design with assured safety, focusing exclusively on safety-critical functions. Unlike data-driven learning approaches, it does not require training. PHY-Teacher satisfies our three key requirements: (i) assured safety with formal theoretical guarantee, (ii) real-time action policies that can respond to complex, dynamic, and unpredictable operating environments, and (iii) rapid and automatic computation of action policies in real time. All of these expected capabilities are formally described in Section 5. Additionally, Appendix E provides pseudocode for rapid and automatic computation of PHY-Teacher's real-time action policies via different toolboxes: Python CVXPY and Matlab LMI.\n\n---"}, {"review_id": "h1UAe9YOlA", "reviewer": "Reviewer_zVT6", "summary": "The Real-DRL framework enables safe deep reinforcement learning in physical autonomous systems by integrating three components: a DRL-Student that learns action policies through self-learning (for performance) and teaching-to-learn (safety lessons from interventions); a PHY-Teacher using physics models to provide real-time safety patches for unknown unknowns and Sim2Real gaps; and a Trigger that monitors state boundaries to switch control. It features automatic safety-first hierarchy learning and safety-status-dependent batch sampling to address corner-case imbalances. Validated on a real quadruped robot, NVIDIA Isaac Gym wild environments, and cart-pole, Real-DRL guarantees runtime safety while outperforming baselines.", "questions": "1. How long does it take to collect sufficient data for training in the real world? In Section 6.1, fixed vel commands are used, which is a simpler setup compared to the quadruped robot algorithms widely adopted nowadays. If real-world training is conducted using continuous commands, similar to state-of-the-art quadruped robot algorithms, would the time cost incurred be acceptable? This is the point I am most concerned.\n2. How are the marginal safety boundaries determined? How does the setting of marginal safety boundaries affect the results of the algorithm? \n3. Can these marginal safety boundaries be gradually widened as the capability of the DRL-Student improves?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Formal Safety Guarantees: Addresses unknown unknowns and Sim2Real gaps through PHY-Teacher's physics-based interventions, providing verifiable safety.\n2. Runtime Adaptability: Teaching-to-learn enables the DRL-Student to acquire safety knowledge from real-time failures. Safety-status-dependent sampling mitigates data imbalance in corner cases.\n\nWeekness\n1. Even model-based approaches cannot fully guarantee safety, especially when encountering unknown unknowns. \n2. In the field of quadruped robots, existing sim2real training frameworks can already achieve highly robust real-world performance. Yet the authors did not adopt state-of-the-art quadruped robot training strategies.\n3. Training a sufficiently good RL strategy requires a huge amount of data, and real-world data collection is much more time-consuming compared to data collection in simulations, even with the addition of safety guarantees.", "was_revised": true, "final_justification": "The algorithm proposed in this paper can serve as an \"Evolving Brain\" for real-world control policy. The experiments in the paper are solid, and the innovations are novel. \n\nThe only drawback, in my opinion, is that the experiments on real-world deployment are rather simplistic. Therefore, I maintain my original score of 4.", "rebuttal": "---\n\n**Weakness 1: Even model-based approaches cannot fully guarantee safety, especially when encountering unknown unknowns.**\n\n**Response:** No technique, including ours, can guarantee safety against `arbitrarily large` unknown unknowns, such as unpredictable engine failures. This acknowledgment is formally reflected in  Assumption 5.1 in the paper, which imposes an upper bound on model mismatch.  The model mismatch can arise from various sources, including unknown uncertainties, unpredictable platform faults, unmodeled dynamics, and external disturbances not captured in the nominal model. PHY-Teacher's theoretical safety guarantee (formally stated in Theorem 5.2) through tolerating unknown unknowns is valid under Assumption 5.1. When this assumption is violated, PHY-Teacher can still maintain safety, but cannot provide a theoretical guarantee (i.e., Theorem 5.2 does not hold).\n\nIn reality, particularly when facing large, unpredictable, real-time unknown unknowns, it is unrealistic to expect Assumption 5.1 to hold consistently. To address this challenge, we introduce a real-time adaptive mechanism (detailed in Appendix F) that employs the real-time state sampling to estimate real-time model mismatches to enable continuous monitoring of Assumption 5.1's validity. When violations are detected, the mechanism computes compensatory actions in real time to mitigate model mismatch effects and restore the assumption's validity. \n\n--- \n\n**Weakness 2: In the field of quadruped robots, existing sim2real training frameworks can already achieve highly robust real-world performance. Yet the authors did not adopt state-of-the-art quadruped robot training strategies.**\n\n**Response:** Many state-of-the-art frameworks leverage GPU-accelerated simulators (e.g., Nvidia Isaac Gym) to train robust locomotion policies through large-scale parallel sampling. They often require millions to billions of training steps per instance to enable the deployment of sufficiently robust policies directly onto real hardware. However, such policies are typically static once deployed, lacking the flexibility to adapt or continue learning in real-world environments. This limitation reduces their biological plausibility and adaptability, which are critical for achieving general-purpose intelligence in the physical world. \n\nMoreover, we provide the motivation and vision of our research to address the reviewers' comment.\n\n* **Motivation: Dynamics and Environments: Complexity and Unpredictability.** Deep reinforcement learning (DRL) possesses vast parameters, intractable activation functions, and stochastic learning dynamics. Meanwhile, many safety-critical autonomous systems (e.g., legged robots, drones, autonomous vehicles, and UAVs) have complex dynamics-environment interactions. Their operating environments (e.g., wildfires, freezing rain, snow squalls) are often unpredictable and hard to simulate accurately. Consequently, the dynamics of DRL-enabled autonomous systems exhibit inherent complexity and unpredictability, creating substantial Sim2Real and domain gaps that threaten system safety. While numerous approaches have been developed to address these gaps, these methods can only mitigate the gaps to varying degrees. Undisclosed gaps continue to hinder the safety assurance of real systems, arising from the fundamentally complex and unpredictable nature of real-world dynamics and operating environments.\n\n* **Vision: \"Evolving Brain\" Architecture.**  To thoroughly address Sim2Real and domain gaps caused by complex and unpredictable dynamics and operating environments, learning-enabled autonomous systems should possess an \"evolving brain\"—similar to human cognitive adaptation—that enables continuous learning in the real physical world, using real-time sensor data generated from real-time physical operating environments. This approach allows action policies to evolve dynamically in real-time, achieving adaptive responses to complex and unpredictable operating conditions. Our proposed Real-DRL framework represents an essential milestone toward delivering this ``evolving brain\" capability, enabling autonomous systems to maintain safety while continuously learning to adapt to novel environmental challenges.\n\n--- \n\n**Weakness 3: Training a sufficiently good RL strategy requires a huge amount of data, and real-world data collection is much more time-consuming compared to data collection in simulations, even with the addition of safety guarantees.**\n\n**Response:**  We would like to clarify that our primary goal in conducting the experiments and designing our Real-DRL is not to replace a high-fidelity simulator or to advocate for starting real-world learning from scratch. Instead, our Real-DRL aims to enable the system to continue learning after deployment, while assuring safety through addressing Sim2Real gaps and unknown unknowns that inevitably arise in the complex real world. \n\nOur framework positions the simulator as a crucial component for pre-training suboptimal policies, providing reasonable starting points. However, a simulator cannot capture all real-world complexities—including unmodeled dynamics, sensor noise, and unexpected disturbances. Our Real-DRL addresses this gap by enabling safe and real-time learning in real physical environments.\n\nThe significance of hardware implementation demonstrates a key advantage over conventional Sim2Real pipelines: while traditional approaches halt learning after deployment, our framework maintains the robot's plasticity, allowing continuous improvement through real-world experience within certified safety bounds. This capability is essential for long-term autonomy and adaptability in open-ended and non-stationary environments.\n\n\n\n--- \n\n**Question 1: How long does it take to collect sufficient data for training in the real world? In Section 6.1, fixed vel commands are used, which is a simpler setup compared to the quadruped robot algorithms widely adopted nowadays. If real-world training is conducted using continuous commands, similar to state-of-the-art quadruped robot algorithms, would the time cost incurred be acceptable? This is the point I am most concerned.**\n\n**Response:** The duration and cost of real-world training are influenced by the Sim2Real gap and the specific task configuration. \n\n* **Learning in the Real World: Safety First:** The time cost incurred during real-world learning is not our primary concern, as our application systems are `safety-critical`, where learning must adhere to the safety-first principle. Our Real-DRL prioritizes and assures safety, meaning that even without learning, Real-DRL can still succesfully perform the safety-critical functions, but with `low` task performance. The learning objective is to achieve high task performance, such as minimizing travel time, average power consumption, and total energy expenditure. While high task performance is desired, it is secondary to assuring safety. \n\n* **Static Command:**  In Section 6.1 of the paper, we deliberately employ a fixed velocity command as a controlled and simplified test case to validate our framework's real-world learning capability. The policy is pre-trained in a simulator, then deployed and continuously refined in the real world on the real robot at a different speed. Notably, the robot successfully learned a safe locomotion policy `< 20 episodes`, with each episode lasting approximately 15 seconds, resulting in `< 5 minutes` of total physical interaction time. \n\n* **Dynamic Commands:** Section 6.2 utilizes NVIDIA Isaac Gym to create a wild environment, `assumed` to be the real one. The architecture of Real-DRL, integrated with a navigation module, is illustrated in `Figure 10` of the paper. The navigation module outputs dynamic planned motions for our Real-DRL. We compared Real-DRL with safe DRLs such as CLF-DRL and Phy-DRL, and fault-tolerant DRLs like neural Simplex and runtime assurance. The trajectories of episode returns in `Figure 4`  demonstrate that Real-DRL exhibits significantly faster and more efficient learning in identical environments. \n\n--- \n\n**Question 2: How are the marginal safety boundaries determined? How does the setting of marginal safety boundaries affect the results of the algorithm?**\n\n**Response:**  The marginal-safety boundaries are the ones of the self-learning space $\\mathbb{L}$. So, the reviewer's first question transforms to: how is the $\\mathbb{L}$ determined? The $\\mathbb{L}$ is to intentionally create a buffer zone $\\mathbb{S} \\setminus \\mathbb{L}$. This configuration provides adequate margins for system response time and decision-making latency. Therefore, the $\\mathbb{L}$ in our work is determined offline through analyzing a worst-case dynamics model, obtained by considering the worst-case model mismatch, CPU computation time, sensor sampling period, and other relevant system constraints.  \n\nFor the second question, the extensive safety boundaries would result in a minimal buffer zone, which would reduce decision-making time and space, making it difficult for the PHY-Teacher to implement a feasible backup action policy for safety. Conversely, smaller safety boundaries do not pose safety concerns because they yield a larger buffer zone, but they result in a reduced self-learning space. This limitation makes it slow for DRL-Student to develop a robust and high-performance action policy.\n\n--- \n\n**Question 3: Can these marginal safety boundaries be gradually widened as the capability of the DRL-Student improves?**\n\n\n**Response:** Yes. DRL-Student engages in a dual teaching-to-learn and self-learning paradigm. So, its safe learning space is $\\mathbb{L}  \\bigcup \\mathbb{P}_{k}, k \\in \\mathbb{N}$, where $\\mathbb{L}$ represents the self-learning space, the other is the teaching-to-learn space, and $k$ represents the activation time of the teacher. As the learning deepens, the total activation number of the teacher increases; consequently, the safe learning space expands. \n\n---"}]

Author rebuttal:

Meta-review: This paper introduces Real-DRL, a safe reinforcement learning framework composed of three components: a DRL-Student that learns task policies, a PHY-Teacher that intervenes in safety-critical states, and a Trigger that switches control. Safety-status-dependent batch sampling is used to emphasize critical scenarios. The framework is evaluated in cart-pole, quadruped robots, and both in simulation and on real platforms.

**Strengths:** The approach is well-motivated and addresses an important challenge in safe RL. The teacher-student-trigger design and safety-status sampling are novel and effective. Experiments are thorough, covering both simulation and real robots, and demonstrate improvements over baselines. The rebuttal clarified important points regarding safety, including robustness to unforeseen disturbances and the scope of the formal guarantees, which apply to certain systems.

**Weaknesses:** The method relies on designing a task-specific PHY-Teacher, which reduces generality and requires domain expertise. While some formal guarantees are provided, they depend on the fidelity of the PHY-Teacher’s model relative to the real system. Since the PHY-Teacher itself does not adapt or improve with data, this reliance represents an important limitation. Real-world validation, though present, remains limited compared to state-of-the-art quadruped RL work. Some clarity issues (notation, terminology) were also noted.

**Final Recommendation:** In line with reviewers' consensus, I recommend acceptance (as a poster). The contribution is solid and relevant for the safe RL community.
Decision: Accept (poster)