Runtime Learning Machine

Authors: Hongpeng Cao, Yanbing Mao, Yihao Cai, Lui Sha, Marco Caccamo
Primary area: reinforcement learning
Keywords: Runtime Learning, Deep Reinforcement Learning, Safety, Unknown Unknown, Autonomous Systems

Abstract:
This paper proposes the **Runtime Learning Machine** for safety-critical autonomous systems. The learning machine has three interactive components: a high-performance (HP)-Student, a high-assurance (HA)-Teacher, and a Coordinator. The HP-Student is a high-performance but not fully verified Phy-DRL (physics-regulated deep reinforcement learning) agent that performs safe runtime learning in **real** plants, using **real**-time sensor data from **real**-time physical environments. On the other hand, HA-Teacher is a verified but simplified design, focusing on safety-critical functions. As a complementary, HA-Teacher's novelty lies in real-time patch for two missions: i) correcting unsafe learning of HP-Student, and ii) backing up safety. The Coordinator manages the interaction between HP-Student and HA-Teacher. Powered by the three interactive components, the runtime learning machine notably features i) assuring lifetime safety (i.e., safety guarantee in any runtime learning stage), ii) tolerating unknown unknowns, iii) addressing Sim2Real gap, and iv) automatic hierarchy learning (i.e., safety-first learning, and then high-performance learning). Experiments involving a cart-pole system, two quadruped robots, and a 2D quadrotor, as well as comparisons with state-of-the-art safe DRL, fault-tolerant DRL, and approaches for addressing Sim2Real gap, demonstrate the machine's effectiveness and unique features.

Decision: reject
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730345475729

thread_type:
official_review

<thread_object>
creator: reviewer_xkge

<content>
summary:
This paper introduces a runtime learning machine for safety-critical autonomous systems, featuring an interactive HP-Student, HA-Teacher, and Coordinator. It ensures lifetime safety by addressing unknowns and the Sim2Real gap, enabling real-time learning for safe, high-performance policies. Experimental results validate its effectiveness on cart-pole and quadruped robot systems.

strengths:
- The proposed method combines a High Assurance Teacher with a Performance optimizing student to yield strong real word robustness results on a challenging quadruped robot.
- The experiments show scalable computation that can work fast enough for real-time learning using matrix toolboxes and related libraries.

weaknesses:
- The setting considered assumes knowledge of the environment dynamics by the HA-Teacher. System identification is proposed but does not appear to have been experimented with.
- The presentation is cluttered at times and causes some difficulty in parsing. For example, some of the numerous assumptions, remarks, characteristics, and definitions should be consolidated if possible. Section 1.3 is almost entirely italicized.

questions:
1. How is this approach placed in the context of other shielding approaches [1,2]?
2. The triggering condition in Eq 7 is entirely causal (i.e., depends only on the current time and prior to the current time). Why is this better than using the environment model to predict for a future failure from the current state and input?
3. How does the approach generalize to different safety sets (e.g., velocity < 1.5 vs velocity < 3)? Will the entire training process be needed for each change? How long would the training time be for each minor change?
4. Are there any experiments or examples with System Identification as mentioned in Remark 6.5 or Appendix C?

### References:
[1] Safe Reinforcement Learning with Nonlinear Dynamics via Model Predictive Shielding, Bastani, ACC 2020

[2] Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning, Banerjee et al, arXiv 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498144187

thread_type:
official_review

<thread_object>
creator: reviewer_z8i5

<content>
summary:
The paper presents a novel approach to runtime learning for agents, guided by a verified teacher during runtime to ensure safe operation. This teacher-student framework leverages physics-regulated deep reinforcement learning (PhyRL) and aims to address safety concerns arising from unknown variables and Sim2Real gaps. The validation of the approach is conducted on two benchmarks, including a real quadruped robot, which adds to the practical value of this research.

strengths:
1. The paper addresses a important safety issue in learning-based systems, particularly focusing on challenges like unknown unknowns and Sim2Real gap.
2. The paper presents solid theorem.
3. The evalution is performed on a real robot, which exhibits the practicality of this work.

weaknesses:
Though the results look impressive, I found some parts of the paper needs further clarification and a few more discussion about related works are expected.
1. The paper lacks details on the verifiability of the PhyRL structure, especially regarding how the teacher's safety is guaranteed in unknown environments. In Equation 2, the system safety is only related to its state. Does this mean that this paper assumes such simple constraint and not consider how the environment change impacts on the states?
2. The paper should discuss whether switching back to the teacher affects operational performance, as such switches may potentially degrade it. Additionally, the evaluation would benefit from presenting metrics on how frequently the teacher’s guidance is triggered.
3. The work currently assumes a single-step safety model, where each action ensures safety only for the immediate next state. However, a broader perspective on end-to-end safety—considering the effects over several steps or even entire trajectories—is the cases that usually happen in reality.
4. There are lots of verified learning works that are not cited/discussed by this paper.  For example, Neurosymbolic reinforcement learning with formally verified exploration. Neurips 2020.
5. There are only two benchmarks presented. Though one of them is the real robot, it would be necessary to evaluate on a more diverse set of benchmarks.

questions:
1. How often does the teacher’s intervention occur during runtime, and is there an impact on the agent's performance with frequent switches?
2. If the teacher is adjusted or modified during runtime, how is its performance or reliability guaranteed in changing conditions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730907304852

thread_type:
official_review

<thread_object>
creator: reviewer_mpnq

<content>
summary:
This paper introduces a "Runtime Learning Machine" designed for autonomous systems operating in safety-critical environments. The machine combines three main components: an HP-Student (high-performance learner), an HA-Teacher (high-assurance safety monitor), and a Coordinator. The HP-Student learns in real-time from physical environments, with the HA-Teacher correcting unsafe actions and enforcing safety. This design aims to address critical challenges in deep reinforcement learning, including tolerating unknown unknowns and bridging the Sim2Real gap.

strengths:
The paper presents an innovative hierarchical safety mechanism where the HA-Teacher manages safety while the HP-Student prioritizes performance optimization within safety constraints. This interactive approach is practical and well-suited to the paper’s focus on real-world, safety-critical applications. The experimental setup, involving both a cart-pole system and a quadruped robot, is comprehensive and shows the method’s robustness across different unknowns. The results suggest that the system can maintain stability through real-time patching by the HA-Teacher, showing practical value in environments where safety and performance trade-offs are essential. Additionally, the framework addresses the Sim2Real gap, which remains a significant challenge in deploying DRL systems in physical settings.

weaknesses:
The design details for the teacher, a core element of this framework, could be expanded, particularly in scenarios where unknowns extend beyond simple disturbances to include hardware or sensor faults. Additionally, the theoretical justification for the teacher’s resilience against unknown unknowns could be strengthened, as the current analysis lacks formal rigor or quantitative measures. While the selected experimental domains are relevant, further discussion on generalizability to applications like autonomous driving or complex industrial robots would broaden the framework’s impact. Although comparisons with other DRL methods are made, the analysis could be enhanced by discussing specific scenarios where other methods may outperform this framework or identifying potential limitations.

questions:
Could you provide more insights into the teacher's handling of complex unknowns, such as hardware failures or significant sensor errors?

What theoretical guarantees exist for the teacher's performance in completely unanticipated scenarios?

How might this system perform in applications outside of the tested environments, such as in higher dimensional or more complex environments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731963629577

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Part II

comment:
$\textbf{II. Automatic Triggering Frequency of HA-Teacher (Reviewer Z8i5)}$. The original paper impressed the **Reviewer Z8i5** that the paper lacks the studies of HA-Teacher's triggering frequency. The paper has such studies, which are partially presented in the appendix. We explain them in detail below. 

In our runtime learning machine, when the real-time safety status of the real plant being controlled by HP-Student approaches the safety boundary, the Coordinator **automatically** prompts the HA-Teacher to intervene, ensure the safety of the real plant, and correct HP-Student's unsafe learning. When the real-time states return to a safe region, the Coordinator **automatically** triggers the switch back to HP-Student and terminates the learning correction. We can further conclude that the frequency of triggering HA-Teacher depends on HP-Student's capability of assuring safety in the face of the Sim2Real gap and unknown unknowns. This can be viewed from the phase plot available at [HA-Teacher v.s. HP-Student (anonymous link)](https://www.dropbox.com/scl/fi/j70n27wtyufmjgrzi4tn8/2DHAHP.pdf?rlkey=si9ifb43ac6d6z3s8whyjq45g&st=bdjmfqgu&dl=0) (also, Figure 21 in the paper). Specifically, the HA-Teacher was frequently activated because the HP-Student had not yet learned how to ensure safety. However, as the HP-Student spent more time learning, his ability to maintain safety improved. This progress is particularly evident in episodes 1 to 5, during which the frequency of HA-Teacher activation decreased. By episode 6, the HP-Student had mastered the capability to ensure safety, and thereafter, his continued runtime learning no longer required assistance from the HA-Teacher. This advancement enabled him to autonomously develop a high-performance action policy, resulting in an end state close to the goal (i.e., the center of the ellipse safety envelope). Our paper includes more such studies with evaluation metrics; see Appendix G 5.3 of the revised paper. 

----- 

$\textbf{III. Tolerating Much More Complex Unknown Unknowns (Reviewer xKGE)}$. The dynamics of real safety-critical autonomous systems are very complex so that they can face very different types of unknown unknowns. 

In the original paper, the unknowns in the experiment of cart-pole systems include the unknown-unknown friction forces and noisy samplings of action/control commands.  In the experiment with the real A1 quadruped robot, the unknowns include 1) **Beta**: Disturbances injected into HP-Student's action/control commands, 2) **PD**: Random and sudden payload (around 4 lbs) drops on the robot's back, 3) **Kick**: Random and sudden kick by a human, 4) **DoS**: A real denial-of-service fault of the platform, which can be caused by task scheduling latency, communication delay, communication block, etc., is unknown to us, 5) **SP**:  A sudden side push. In our implementation, we consider three complex combinations of them: i) **Beta + PD**, ii) **Beta + DoS + Kick**, and iii) **Beta + SP**. The demonstration video is available at [A1 Robot: unknown unknowns (anonymous link)](https://www.dropbox.com/scl/fi/jjxlym8xtdxqgr7d5hclp/unk.mp4?rlkey=kh76ojwhspqgslzolmowuuu0l&st=igbjnji3&dl=0). 

Many safety-critical autonomous systems, especially those operating outdoors, have dynamics-environment interactions. So, the operating environment could also induce unknown unknowns. According to comments from Reviewer xKGE, the experiment of the new benchmark Go2 robot includes such demonstrations about environments, whose demo videos are available on [Go2 Forward (anonymous link)](https://www.dropbox.com/scl/fi/b1l7uskmx5wg2uw6n8tzz/forward.mp4?rlkey=6k4mfs1f8symff3ilquajsd3o&st=a17uvfo9&dl=0),  [Go2 Backward (anonymous link)](https://www.dropbox.com/scl/fi/fclb0npdyatq93qzq2i72/backward.mp4?rlkey=7u9bcmw3sz30zk6njqu79vtel&st=yje3jfwt&dl=0), and [Go2: Kick--Sensor (anonymous link)](https://www.dropbox.com/scl/fi/glqv1o9viltxvp083vmdt/sensorkick.mp4?rlkey=cj06mdbrdfnmcdqr688jjn38k&st=u0cs6312&dl=0). The videos demonstratee our runtime learning machine's significantly enhanced safety assurance in **non-stationary, unstructured, uneven, and unforeseen operating environments (low friction due to unexpected freezing rain)**, along with very different mission commands/goals and unknown-unknown noisy samplings of state sensors.

-----

$\textbf{IV. Two New Benchmarks}$. Initially, we were able to provide more benchmarks; however, we need to demonstrate that we have achieved the proposed prospect along with all four claimed features, which requires more pages. Due to the page limit, the benchmarks in our original paper were reduced to a cart-pole system and the real A1 quadruped robot. Having only two benchmarks leads to questions about generalizability, according to **Reviewers Z8i5 and mpnQ**. We added two new benchmarks to the revised paper to address the concern. They are the 2D quadrotor and the Go2 quadruped robot. 

----


**-END-**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732206402991

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Part I

comment:
We appreciate our reviewers' supportive feedback, thorough evaluations, and insightful remarks. The paper has been updated as needed, and the updated Supplementary Material includes all the benchmark codes and **six demonstration videos**. Although our response to reviewers is lengthy, it addresses all the reviewers' comments in detail. We'd like to summarize the four significant clarifications in response to the reviewers' critical comments as a global response. 

------

$\textbf{I. Lifetime Safety, not Step-wise Safety (Reviewer Z8i5)}$. In the post-rebuttal, **Reviewer Z8i5** argues that our theoretical safety guarantee or definition is step-wise, implying it only ensures safety for one step. **Definition 2.1** (page 4) and **Theorem 6.4** (page 7) of the paper clearly and formally state this is untrue! To demonstrate this, we recall them from the paper.  

**Definition 2.1 (Lifetime Safety).** Consider the safety set $\mathbb{X}$. The real plant is said to have lifetime safety, if given any $\mathbf{s}(1) \in \mathbb{X}$, the $\mathbf{s}(k) \in  \mathbb{X}$ holds at any time $k \in \mathbb{N}$, regardless of HP-Student's failure. 

**Statement 1 in Theorem 6.4.** The real-time patch ${\Psi_{\sigma(k)}} \subseteq {\mathbb{X}}$ holds for any time $k$. 

Definition 2.1 specifies one of our design aims: rendering the given safety set $\mathbb{X}$ invariant by our learning machine, that is, under the control of our runtime learning machine, operating from the safety set, system states will never leave the safety set at any current or future time. This design aim is mathematically described by 'given any $\mathbf{s}(1) \in \mathbb{X}$, the $\mathbf{s}(k) \in  \mathbb{X}$ holds at any time $k \in \mathbb{N}$' in Definition 2.1. Lifetime safety is achieved, as indicated by Statement 1 in Theorem 6.4. We note that 'holds at any time $k \in \mathbb{N}$' in Definition 2.1 and Statement 1 in Theorem 6.4 explicitly state that our safety definition is lifetime, not step-wise. 

During real operation, we must constantly monitor the real-time system states to decide if to trigger HA-Teacher for backup safety. The triggering condition is $\mathbf{s}^\top(k-1) \cdot \mathbf{P} \cdot \mathbf{s}(k-1) \leq 1$ and $\mathbf{s}^\top(k) \cdot \mathbf{P} \cdot \mathbf{s}(k) > 1$. The condition is only used to trigger the switching.  If we only consider safety assurance, i.e., not caring about mission performance, we can always use HA-Teacher. The triggering condition can be removed because its design ensures safety for a **lifetime**, **not step-wise**, as indicated by Theorem 6.4 of the revised paper.

In conclusion, our safety definition and design are **lifetime** and **deterministic**, which has been a standard design aim for many other safety- and time-critical autonomous systems, such as stated by Definition 2.1 in [R1], Definitions 2.3 and 2.4 in [R2], Section 2 in [R3], Definition 1 in [R4].  

Importantly, lifetime safety is our claimed notable feature of runtime learning machine, which has been well demonstrated by experimental results in **Figures 3 (a) and (b)**, **Figures 9 to 12**, and **Figure 21** (or the plot available at [HA-Teacher v.s. HP-Student (anonymous link)](https://www.dropbox.com/scl/fi/j70n27wtyufmjgrzi4tn8/2DHAHP.pdf?rlkey=si9ifb43ac6d6z3s8whyjq45g&st=bdjmfqgu&dl=0)) of the updated paper. 



**Lastly, we believe the reviewer’s concern can be further addressed by the (safety) demonstration videos available at [A1 Robot: unknown unknowns (anonymous link)](https://www.dropbox.com/scl/fi/jjxlym8xtdxqgr7d5hclp/unk.mp4?rlkey=kh76ojwhspqgslzolmowuuu0l&st=igbjnji3&dl=0), [A1 Robot: Sim2Real gap (anonymous link)](https://www.dropbox.com/scl/fi/mjwi9u6ng72oghbm4wkx7/sim2real.mp4?rlkey=2lg4fayh2mrpj2m6nqvx02tox&st=ono7z5na&dl=0), [Go2 Forward (anonymous link)](https://www.dropbox.com/scl/fi/b1l7uskmx5wg2uw6n8tzz/forward.mp4?rlkey=6k4mfs1f8symff3ilquajsd3o&st=a17uvfo9&dl=0),  [Go2 Backward (anonymous link)](https://www.dropbox.com/scl/fi/fclb0npdyatq93qzq2i72/backward.mp4?rlkey=7u9bcmw3sz30zk6njqu79vtel&st=yje3jfwt&dl=0), and [Go2: Kick--Sensor (anonymous link)](https://www.dropbox.com/scl/fi/glqv1o9viltxvp083vmdt/sensorkick.mp4?rlkey=cj06mdbrdfnmcdqr688jjn38k&st=u0cs6312&dl=0).**

**References**

[R1] Cao et al. Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings. In The Twelfth International Conference on Learning Representations, 2024.

[R2] Seto, D., \& Sha, L. An engineering method for safety region development. Carnegie Mellon University, Software Engineering Institute.

[R3] Bak et al. Real-time reachability for verified simplex design. In 2014 IEEE Real-Time Systems Symposium. IEEE.

[R4] Tabas, D., \& Zhang, B. Computationally efficient safe reinforcement learning for power systems. In 2022 American Control Conference.

-----
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a runtime monitoring and correction system for safe deep RL that remains performant. The primary methodology introduces a student whose goal is to maximize performance, a teacher whose goal is to ensure safety and a coordinator. These act together to ensure system safety by projecting back into a safety envelope

Strength: 
Important premise for any deployed system to be safe and performant
Seems like a principled and well motivated approach
Results on a real robot system

Weaknesses
The paper is written in a way that is deeply inaccessible to most DRL researchers. In it's current form, the paper will be completely unreadable by it's target audience
Secondarily, there are lots of unusual terms, acronyms and arcane terminologies used throughout the paper that make it hard to parse.
Connections to other constrained deep RL methods should also be made more clear

Overall this paper is quite borderline,  I will rate it towards a reject because despite an interesting premise, it's got too much going on to actually communicate to the target audience. The paper would be much stronger with a simplification of the writing and a focus of the system so that it is far more accessible to the broad DRL and safety audience.