Title: Real-DRL: Teach and Learn at Runtime
Venue: NeurIPS 2025
Accepted: yes
Keywords: Safety

Dual Self-Learning and Teaching-to-Learn

Safety-Status-Dependent Batch Sampling

Autonomous Systems
Abstract: This paper introduces the Real-DRL framework for safety-critical autonomous systems, enabling runtime learning of a deep reinforcement learning (DRL) agent to develop safe and high-performance action policies in real plants while prioritizing safety. The Real-DRL consists of three interactive components: a DRL-Student, a PHY-Teacher, and a Trigger. The DRL-Student is a DRL agent that innovates in the dual self-learning and teaching-to-learn paradigm and the safety-status-dependent batch sampling. On the other hand, PHY-Teacher is a physics-model-based design of action policies that focuses solely on safety-critical functions. PHY-Teacher is novel in its real-time patch for two key missions: i) fostering the teaching-to-learn paradigm for DRL-Student and ii) backing up the safety of real plants. The Trigger manages the interaction between the DRL-Student and the PHY-Teacher. Powered by the three interactive components, the Real-DRL can effectively address safety challenges that arise from the unknown unknowns and the Sim2Real gap. Additionally, Real-DRL notably features i) assured safety, ii) automatic hierarchy learning (i.e., safety-first learning and then high-performance learning), and iii) safety-informed batch sampling to address the experience imbalance caused by corner cases. Experiments with a real quadruped robot, a quadruped robot in Nvidia Isaac Gym, and a cart-pole system, along with comparisons and ablation studies, demonstrate the Real-DRL's effectiveness and unique features.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Real-DRL framework enables safe deep reinforcement learning in physical autonomous systems by integrating three components: a DRL-Student that learns action policies through self-learning (for performance) and teaching-to-learn (safety lessons from interventions); a PHY-Teacher using physics models to provide real-time safety patches for unknown unknowns and Sim2Real gaps; and a Trigger that monitors state boundaries to switch control. It features automatic safety-first hierarchy learning and safety-status-dependent batch sampling to address corner-case imbalances. Validated on a real quadruped robot, NVIDIA Isaac Gym wild environments, and cart-pole, Real-DRL guarantees runtime safety while outperforming baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Formal Safety Guarantees: Addresses unknown unknowns and Sim2Real gaps through PHY-Teacher's physics-based interventions, providing verifiable safety.\n2. Runtime Adaptability: Teaching-to-learn enables the DRL-Student to acquire safety knowledge from real-time failures. Safety-status-dependent sampling mitigates data imbalance in corner cases.\n\nWeekness\n1. Even model-based approaches cannot fully guarantee safety, especially when encountering unknown unknowns. \n2. In the field of quadruped robots, existing sim2real training frameworks can already achieve highly robust real-world performance. Yet the authors did not adopt state-of-the-art quadruped robot training strategies.\n3. Training a sufficiently good RL strategy requires a huge amount of data, and real-world data collection is much more time-consuming compared to data collection in simulations, even with the addition of safety guarantees.\n\nQUESTIONS:\n1. How long does it take to collect sufficient data for training in the real world? In Section 6.1, fixed vel commands are used, which is a simpler setup compared to the quadruped robot algorithms widely adopted nowadays. If real-world training is conducted using continuous commands, similar to state-of-the-art quadruped robot algorithms, would the time cost incurred be acceptable? This is the point I am most concerned.\n2. How are the marginal safety boundaries determined? How does the setting of marginal safety boundaries affect the results of the algorithm? \n3. Can these marginal safety boundaries be gradually widened as the capability of the DRL-Student improves?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Real-DRL, a hybrid runtime learning architecture for safety-critical systems composed of a DRL-Student (learning high-performance policies), a PHY-Teacher (guaranteeing safety via physics-based backup control), and a Trigger that switches control based on safety conditions. The goal is to deal with the unknown unknowns and the sim2real gap.\n\nThe manuscript seems to be a mix of describing the proposal in the context of a controlling a plant, and some examples in other domains, which is confusing. If the focus is on safe controlling of plants, examples in this domain should be provided. Otherwise, remove the excessive \"plant\" indication.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGHTS:\n- Verifiable safety of DRL systems is a must these days, and this is a work in that direction.\n- The proposal is formally described. \n- Experiments are solid and they are very well described in the Annex.\n\n\nWEAKNESSES:\n- The manuscripts mentions, since the very beginning, \"plants\" (which?). But given examples have nothing to do with them. Why such a focus on plants? Cannot the proposal be described in general without that focus?\n- The proposal has a heavy reliance on PHY-Teacher performing ok at all times. This might not always be possible to guarantee in real scenarios.\n- The novelty, besides the concrete implementation details, is limited compared to existing Fault-tolerant DRL that uses HPM and HAM.\n\nQUESTIONS:\n- Why does the DRL-Student adopt an actor-critic architecture? Are other architectures possible?\n- Why mentioning so many times \"plants\" but provide no example of real application to a plant?\n- Cannot the proposal be applied in general and remove the mention to plants in places where it is not necessary?\n- How robust is the PHY-Teacher under model mismatch or poorly characterized environments? This is address in Annex F, but would be good to mention it in the main text.\n- Could the PHY-Teacher be replaced by a simpler rule-system? Or is the \"physic-model\" a generalization or any system that can provide control actions?\n- Is the PHY-Teacher ever trained? If so, how? If not, does it need to be formally described for each problem?\n\nLIMITATIONS:\nThe manuscirpt does not highlight any limitation of the proposal, but I have listed several ones (or at least, questions for which I cannot find a proper answer) in my previous comments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Real-DRL, a framework enabling runtime learning of a deep reinforcement learning (DRL) agent with safety assurance and high-performance learning, which improves on existing DRL frameworks' abilities to provide safety guarantees especially in the face of corner cases and the sim2real gap. The framework has 3 components: DRL-Student, PHY-Teacher, and Trigger. DRL-Student is the agent being trained and only acts within a predefined subset of the \"safe\" state space. PHY-Teacher is responsible for taking control of the actions when DRL-Student navigates too close to the \"unsafe\" space and provides samples of its own experience returning the plant back to the \"safe\" region of the state space to DRL-Student to learn from. Trigger is the monitoring component that determines when to switch between DRL-Student and PHY-Teacher.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The general problem studied is significant and interesting, with overall a rigorous development of the results presented for the developed framework\n- Most writing of the paper is clear and they provide ample intuition for how their framework works (albeit there are some caveats noted below).\n- Experiments are interesting and broad, and the results appear to beat SOTA in some cases.\n \nWeaknesses:\n- Some equations are missing references to the variables in them and therefore are not clear or intuitive at all (e.g., the Real-Time Patch in equation 15 or equation 21 and 22).\n\n- The main motivation for combining learning and control is to avoid the strong assumption of fully known Markovian dynamics. In general, I am not fully convinced learning-based control techniques while still assuming the dynamics are known is reasonable. Assuming a known model significantly limits the novelty and practical impact of the contribution. \n\n- Unlike standard reinforcement learning algorithms that do not assume model access, the authors rely on having access to the model to design PHY-Teacher. This makes comparisons with other techniques challenging.\n\n- Regarding the definition of the trigger’s role: How can you guarantee that the student policy will be able to bring the system back into the learning-enabled region when its actions are restricted to the set A? This is a critical challenge that must be addressed. Encountering unseen data may lead to large errors that push the system far outside the learning-enabled region, making recovery infeasible or requiring prohibitively large control inputs. This issue is especially likely if the system dynamics are highly nonlinear.\n\n- The tracking error dynamics in Equation 17 are assumed to be linear, which is a very strong and potentially unrealistic assumption.\n\n- While the authors present strong and compelling plans for addressing unknown unknowns, they do not provide equally convincing arguments that their controllers—PHY-Teacher and DRL-Student—will reliably behave as intended in practice. Although the experimental results are promising, they depend on having access to the model during training.\n\n- \"reply\" should be \"replay\" on line 213 (pg. 6).\n\nQUESTIONS:\nWhy does the framework perform so poorly without the safety-status-dependent batch sampling method? I would expect safety performance to be worse without it, but every trajectory in the phase plot appears to enter the unsafe region. Are none of these samples from PHY-teacher being used at all when evaluating without the safety-status-dependent batch sampling method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe methodpresented in this paper, Real-DRL, is focused on don safety-critical learning for robotics applicaitons. Themethod is hierarchical with a \"teacher\" that guides a \"student\" policy to achieve safety, and a \"student\" focused on the control task.\nA Safety-Status-Dependent Batch Sampling technique  oversamples safety-critical situations during training, such that safety constraints are prioritized before optimizing for performance (by keeping the teacher in a larger set than the student. The method is validated experimentally on quadruped robots, and cart-pole systems with comparison to existing safe RL approaches and shows a superior performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The proposed framework is supported by solid experimental validation on simulated and real platforms, also in the presence of disturbances, and achieves superior performance in the demonstrated cases.\nThe paper is clearly written.\nThe method provides runtime assurance for safety.\n\nWeaknesses: The implementation complexity as compared to other methods is higher, due to the hierarchical nature of the method. Computational costs during training are not analysed. \nThe design an appropriate PHY-Teacher for each new task or robot requires domain expertise. \nThe approach lacks formal mathematical safety guarantees or proofs. This means that for safety critical systems, there is still a probability of violating safety.\n\nQUESTIONS:\nCurrently, safety is maintained by addressing nearly unsafe, or safety-critical situation during learning. What would happen when the system encounters a disturbance not included as safety-critical (i.e., unforeseen disturbance)? Can it be detected as potntially unsafe - in this case, can the policy adapt or restrict itself of acting in the presence of that disturbance?\n\nThe physics-guided teacher relies on a model of the real system. What happens in the presence of uncertainties, mismatch, or drifts in this model with respect to the real system? How important is the model's fidelity? A study on parameter uncertainties could be useful to clarify this, in particular with respect to safety constraint violations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}