Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination
Authors: Duy Phuong Nguyen
Kai-Chieh Hsu
Wenhao Yu
Jie Tan
Jaime Fernández Fisac
Venue: CoRL 2024 (2024)
Keywords: Robust Safety
Adversarial Reinforcement Learning
Game Theory

Abstract: Despite the impressive recent advances in learning-based robot control, ensuring robustness to out-of-distribution conditions remains an open challenge. Safety filters can, in principle, keep arbitrary control policies from incurring catastrophic failures by overriding unsafe actions, but existing solutions for complex (e.g., legged) robot dynamics do not span the full motion envelope and instead rely on local, reduced-order models. These filters tend to overly restrict agility and can still fail when perturbed away from nominal conditions. This paper presents the gameplay filter, a new class of predictive safety filter that continually plays out hypothetical matches between its simulation-trained safety strategy and a virtual adversary co-trained to invoke worst-case events and sim-to-real error, and precludes actions that would cause failures down the line. We demonstrate the scalability and robustness of the approach with a first-of-its-kind full-order safety filter for (36-D) quadrupedal dynamics. Physical experiments on two different quadruped platforms demonstrate the superior zero-shot effectiveness of the gameplay filter under large perturbations such as tugging and unmodeled terrain. Experiment videos and open-source software are available online: https://saferobotics.org/research/gameplay-filter

Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination
Duy P. Nguyen
Kai-Chich Hsu Wenhao Yu²
Jie Tan? Jaime Fernández Fisac
Princeton University, United States
Google DeepMind, United States
jf @princeton.... edu,
(magicmelon, jietan @google. jiitettttgooolleeccomm com
Simulation
Tugging force
Irregular terrain
Gameplay
Filter
Unfiltered
Policy
Go2 robat
S40 robot
We demonstrate game-theoretic predictive safety filter approach two different quadruped robots equipped with safety-agnostic walking policies. The gameplay filter continually monitors the robot's safety by rapidly simulating worst-case futures invoked by learned virtual adversary, and it preemptively disallows task-driven actions that would cause the robot to lose to it (violate safety). The gameplay-fitered robots exhibit rich and highly robust behaviors such as counterbalancing to fight persistent pulls and springing into wide stance to break imminent falls.
Figure 1:
Abstract: Despite the impressive recent advances in learning-basel robot control, ensuring robustness to out-of-distribution conditions remains an open challenge. Safety filters can, in principle, keep arbitrary control policies from incurring catastrophic failures by overriding unsafe actions, but existing solutions for complex (e.g., legged) robot dynamics do not span the full motion envelope and instead rely on local, reduced-order models. These filters tend to overly restrict agility and can still fail when perturbed away from nominal conditions. This paper presents the gameplay filter, a new class of predictive safety filter that continually plays out hypothetical matches between its simulation-trained safety strategy and virtual adversary co-trained to invoke worst-case events and sim-to-real error, and precludes actions that would cause failures down the line. We demonstrate the scalability and robustness of the approach with first-ofiitskkidd full-order safety filter for (36-D) quadrupedal dynamics. Physical experiments on two different quadruped platforms demonstrate the superior zero-shot effectiveness of the gameplay filter under large perturbations such as tugging and unmodeled terrain. Experiment videos and open-source software are available online: http://ssferbbboii..oog/ssaaaagp
Keywords: Robust Safety, Adversarial Reinforcement Learning, Game Theory
Introduction
Autonomous robots are increasingly required to operate reliably in uncertain conditions and quickly adapt to carry out broad range of jobs on the fly [1-5]. Rather than synthesize an intrinsically safe control policy for every new assigned task, it is efficient to endow each robot with a safety filter that automatically precludes unsafe actions, relieving task policies of the burden of safety altogether.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Unfortunately, today's 's safety filter methods fall short of this promise for most modern- -ay robots. To cover a diverse range of tasks and environments, a safety filter needs to give the robot significant freedom to execute varied motions across its state space while robustly protecting it from catas- trophic failures throughout this large envelope. To date, such minimally restrictive safety filters are only systematically computable for systems with 5-6 state variables [6-8], woefully short of the 12 needed to accurately model drone flight and the 30-50 needed for legged locomotion. Existing safety filters for high-order robot dynamics rely on reduced-order models [9-12]. These filters re- strict the rohot's motion to local envelope, such as the vicinity of a stable walking gait, and become ineffective whenever the robot is perturbed away from it by external forces or unmodeled environ- ment features (Fig. 1). How can we tractably and systematically compute safety filters that cover broad regions of robots' high-dimensional state spaces and wide variety of deployment conditions?
Contribution. This paper introduces the gameplay filter, novel type of predictive safety filter that can scale to full-order robot dynamics and enforce safety across broad motion envelope and a designer-specifeee range of possible conditions (operational design domain). The filter is first synthesized by simulated self-play between a safety-seeking robot control policy and a virtual ad- versary that invokes worst-case realizations of uncertainty and modeling error (or sim-to-real gap). At runtime, the deployed filter continually rolls out hypothetical games between the two learned agents, overriding candidate actions that would result in the robot losing a future safety game. This methodology- based case opponent must also win against all fihers-unlocks real-time filtering in the robot's full state space by only requiring a single, highly informative trajectory rollout. We demonstrate the effective- ness of our approach experimentally on two quadruped robots that differ in physical parameters and sensing capabilities (Fig. 1). Each gameplay filter is synthesized and deployed using an off-the-shelf physics engine to simulate a manufacturer-proont providded robot model with a 36-D state space and 12-D control space. We observe highly robust zero-shot peeee--pees behavior without incurring the conservativeness typical of robust predictive filters. To the best of our knowledge, this constitutes the first successful demonstration of full-order safety filter on legged robot platforms.
based on the core game-theoretii principle that strategy that wins against the worst-
Related Work. The last decade has seen important advances in robot safety filters. We briefly discuss the techniques most relevant to our work and direct interested readers to recent survey ef- forts [13-15] that shed light on safety filters' common structure and relative strengths.
Value-based filters. Hamilton-Jacobi (HJ) reachability methods use finite-iiffrrnne dynamic pro- gramming to compute the best available safety fallback policy and the worst possible uncertainty realization from each state on finite grid [6, 16, 17], which enables minimally restrictive safety filters. Although highly general, HJ computational tools suffer exponential blowup and do not scale beyond 5-6 state dimensions [18, 19]. Control barrier function (CBF) filters keep the system in- side smaller safe set while discouraging excessive control overrides [20]. CBFs lack a general constructive procedure and instead rely on manual design [21], of-suurrss synthesis [22], or learning from demonstrations [23]. Robust formulations are comparatively less mature [24-27]. Self-supervised and reinforcement learning techniques
synthesize safety-oriented control poli- cies and value functions ("safety critics") for systems beyond the reach of classical methods, but they are inherently approximate and offer no formal assurances [28-32]. 323.. Statistical generalization theory may be used to bound the probability of failure under the assumption that the robot can be tested on a statistically representative sample of environments and conditions before deployment [3].
Roslout-based filters. Predictive safety filters perform model-based runtime assurance by continually simulating and in some cases optimizing the robot's future safety efforts for a short lookahead time horizon [33-38]. Recent advances in fast forward-reachable set over-approximation [39-41] make it possible to check safety against all possible uncertainty realizations, although this runtime robustness comes at the cost of significant added conservativeness: for example, Hsu et al. [38] ob- serve safety overrides 5 times as frequent as those of restricvst HJ filter. Bastani and Li [35] instead propose sampling multiple possible trajectories, assuming a well-characterized disturbance distribution, to maintain a statistical guarantee. Our approach mimies Hsu al. [38] in co-training safety controller and worst-case disturbance through simulated self-play but it eschews over- conservative reachable sets by instead simulating single closed-loop match between the two.

---

Legged robot safety filters. Legged robots have attracted increasing interest from researchers due to their versatility and increasing availability, as well as their challenging high-order and contact- rich dynamics [42]. Recent simulation-train trained controllers leveraging domain randomization are showing promising agility and adaptability [1, 2. 43, 44]; however, robustness to out-of-distribution conditions cannot be easily quantified and remains an open issue. Unfortunately, all safety filters demonstrated legged robots date based on simplified reduced-order dynamical models [3, 10-12], sometimes combined with local analysis around nominal walking gaits [9, 45, 463. The dynamic envelope protected by these safety filters is limited to local state space regions where the simplified models apply, and their robustness to disturbances and modeling errors is contingent on the effectiveness of low-level tracking controllers. Our demonstration of the gameplay filter uses a full-order dynamical model of the robot, both synthesis and at deployment, which enables it to enforce safety across a broad range of motions and operating conditions.
2 Preliminaries: Robust Robot Safety in an Operational Design Domain
We wish to ensure the safe operation of a robot with potentially high-order nonlinear dynamics under wide range of environments and task specifications, which may be unknown at design time. Formally, we consider robotic system with uncertain discrete-time dynamics
Tk++1 [( Uk, dk);
where, at each time step l E N, I&k E X S IR the state of the system, uk E C R is the bounded control input (typically from a control policy x" II; + U), and dy D C R is a disturbance input, unknown a priori but bounded by compact D. While the control bound U en- codes actuator limits, the disturbance bound D is key part of the operational design domain (ODD).
Operational Design Domain. The ODD can be viewed as social contract between the system operator and the public, delineating the set of conditions under which the robotic system is required function correctly and safely [47]. In this paper, we are interested robust safery, where the disturbance (or "domain") bound 'D may encode range of potential perturbations like wind or contact forces, environmental parameters like terrain friction, manufacturing tolerances, variations in actuator performance and state estimation accuracy, and other factors contributing to designer uncertainty about future deployment conditions and modeling error. The ODD further specifies deployment set Xg C X' of allowable initial states (for example, the robot is always turned on while static on flat ground) and, crucially, a failure set F C X, which characterizes all configurations that the system state must never reach, such as falls or collisions. The required safety property can then be succinetly expressed as:
Va E Xo Vk 2 0, Vdo..... de ED, IK F,
(2)
that is, once deployed in an admissible initial state, the robot must stay clear of the failure set for any realization of the domain uncertainty.
Safety Filter. Explicitly ensuring the safety property in the synthesis of every robot task policy T can be impractically cumbersome, especially for increasingly general-purpose robotic systems with broad ODDs. Instead, we aim to relieve task policies of the burden of safety by augmenting them with safety filter & that depends on the robot's ODD but not on the task specification. Rather than directly applying the proposed task action u, = n7 (xx) from each state Ik. the robot executes'
Uk Φ [z π
The safety filter's role is to prevent the execution of any candidate actions that would jeopardize future safety, while also avoiding spurious interventions that unnecessarily disrupt task progress. In
For the scope of this paper, assume that the robot maintains appropriately accurate estimate of its dynamical state through onboard perception. We make two observations: First, moderate state estimation errors typical in many robotic systems can be absorbed by inflating the failure set J and dynamical uncertainty D. Second, more substantial state uncertainty, e.g., induced by sensor faults, occluding objects, or multiagent interaction, may be handled with information space safety filters, subject of ongoing research [48-50].
3

---

fact, for any well-defined ODD there exists perfect safery filter that allows every safe candidate action and overrides every unsafe one, robustly enforcing (2) with no overstepping [13, Prop. 1]. Formally, a perfect safety filter only disallows actions that may cause the state to exit the maximal safe set Q2* C X, the set of all states from which there exists control policy that can enforce (2). While computing such perfect filter is known to be intractable for most practical systems [7], we aim to synthesize effective safety filters that allow robots significant freedom perform a wide range of tasks (including online learning and exploration) while maintaining safety across their ODD. Intuitively, we would like to obtain a safety filter that robustly keeps the robot inside a conservative safe set Ω C Q as close as possible to the theoretical f2* Our proposed method uses game- theoretic reinforcement learning and faster-than-reall reae-time gameplay simulation to approximate a perfect safety filter for any given robot ODD, targeting the robot's full envelope in contrast with existing reduced-order filters, which aim to enforce safety within a significantly smaller set Ω.
envelope,
Reach-Avoid Safety Game. Whether it is possible for the robot to robustly maintain safety, as in (2), can be seen as the categorical (true/false) outcome of a game of kind between the robot's controller and an adversarial disturbance that aims to drive it into the failure set. In turn, this result can be encoded implicitly through a game of degree with a continuous outcome (for example, the closest distance that will separate the robot and any obstacle). In particular, for the purposes of pre- dictive safety filtering, we consider a sufficient finite-time time condition for all-time safety: it is enough for the robot to reach known controlled-invarian set T C Je (for example, coming to a stable stance) in H steps without previously entering the failure set F. Once there, the robot can switch to a policy T that keeps it in T indefinitely. This induces a reach-avoid game [17, 32] with outcome
{ e (I+),, min 9 (1..) (1 =E(k,T]
J
(x)
max min re(k.M]
where g and are the (Lipschitz) failure and target margins, satisfying g(x) < 0 + x E F, ((x) 2 0 + x E TT The outcome summarizes the aforementioned condition for all-time safety: For any given T E [0, H), if we previously enter the failure set F, g(0,) < 0, then for k E [0. r].
Je (x) < O, denoting that past failure overrides future successes. The value function of this game satisfies the reach-avoid Isaacs equation
(z) max min min (g(z), max {((x), Vi+z(f(e,
(5a)
"
VH (x:) = min (E(x), g(z)} and the robot's controller is guaranteed a winning strategy from any state where Vo(x) 2 0}.
(5b)
Predictive Gameplay Safety Filters
3.1 Offline Gameplay Learning
We extend the Iterative Soft Adversarial Actor-Critic for Safety (ISAACS) scheme [38] to reach- avoid games (4), approximately solving the infinite-horizon counterpart of the Isaacs equation (5).
Simulated Adversarial Safety Games. At every time step of gameplay, we record the transition (It, d, x² f. g') in the replay buffer B, with x' := fa, u, d), ll : ((x') and g' == ==x²).
Policy and Critic Networks Update The core of the proposed offline gameplay learning is to find an approximate solution to the time-discounted infinite-horizon version of (5). We employ the Soft Actor-Critic (SAC) [51] framework to update the critic and actor networks with the following loss functions. We update the critic to reduce the deviation from the Isaacs target?
L(w)= (a. de. )') (Qu d. y)²
y ? min g', max {E, (2 vi dd) + (1 (1-7) min (L'g'))
(6a)
DDeep reinforcement learning typically involves training an auxiliary target critic QLa with parameters w' critic parameters w. This process aims to stabilize the regression
that undergo slow adjustments to align with the by maintaining fixed target within relatively short timeframe.

---

In practice the computation of a full gameplay rollout may span multiple time steps (i.e., multiple multi-step variant Gameplay rollout
Real time
-E
E rolitut computitio beercy
Imagination time
Figure 2: The L-step gameplay filter evaluates T for L steps. The rollout started at k k= 0 returns win result by k= so the safety filter allows Th to proceed for k= k= 2L - 1. Meanwhile, the next rollout, initiated
L, predicts a defeat, so the filter selects T from 2L 3L
with u' T- x²), d' T. x'). We update control and disturbance policies following the policy gradient induced by the critic with entropy regularization:
I(Ø) = E
Qu (z, ú, d) + a log to x))
(6b)
(x,d)~
L( = E
Qw u, d) + a log & mt).
(6c)
(x,0)~B
where TT(- | x). d Tp(- 2)) and a",,"" at hyperparameterss incentivizing exploration (entropy in the stochastic policies), which decay gradually in magnitude through the training.
Following the ISAACS scheme, we jointly train the safety critic, controller actor and disturbance actor through (6). For better learning stability, the controller actor can be updated at slower rate (only every r 2 disturbance updates), consistent with the asymmetric information structure of the game, and leaderboard of best-performing controllers and disturbances can be maintained to mitigate mutual overfitting to the latest adversary iteration [38].
3.2 Online Gameplay Filter
This section demonstrates how the synthesized reach-avoid control actor za and disturbance actor Ty. can systematically used at runtime to construct highly effective safety filters for general non- linear, high-dimensional dynamic systems. The gameplay rollout considers applying the candidate task policy 인 followed by the learned fallback policy x with the whole rollout under attack by the learned disturbance policy T to check if accepting the candidate action from task policy nu will lead to an inevitable failure even if we then apply our best-effort attempt to maintain safety. The gameplay outcome follows the reach-avoid outcome defined in (4) A runtime gameplay filter can then be defined with the simple switching rule:
A
B [I, B =
(z.
1
Br E (1.....,HI.
H). 2, ETA
π
A
ss (I...., T), f. $ EF)
(7a)
with to x, fr--1 = f(i,, ú, To T (fr)), 0. and
n (ir), T= 0, (w,), (1..,,,,
To(x),
I ETT, I FE T.
ú
(x) -
(7b)
TTx),
That is, if the gameplay monitor predicts win (the simulated trajectory safely reaches the target set), the filter selects the task policy 8 otherwise, the filter selects the fallback safety policy π
control policy executions). In that case, the filter in (7) can be extended in which decisions made by the filter every L steps, appropriately accounting for the rollout computation latency. Fig. 2 illustrates the gameplay safety filter logic with T-sttep latency.
5

---

4 Experimental Evaluation
We run hardware experiments and an extensive simulation study, focusing on quadruped robots as an informative platform but stressing that our proposed methodology is general and can be applied to other types of robots. We aim to evaluate the extent to which the synthesized gameplay filters can maintain safety within the ODD specified at training, generalize beyond the ODD, and avoid unnecessarily impeding task execution. Specifically, we use 50 N force applied to the robot's torso in training, with failure defined as any time non-foot part contacts the ground. We test the robots under two conditions: tugging forces on flat terrain (similar to ODD) and irregular terrain (out-of-ODD). The task of moving from start location to goal across a terrain is evaluated on the Ghost Robotics Spirit S40 and Unitree Go2 (Fig. 1). We also conduct ablation studies to investigate the importance of reach-avoid reinforcement learning (RARL) and adversarial self-play in the filter synthesis and of the gameplay rollout in the filter's runtime monitoring by considering three prior reinforcement learning algorithms: (1) standard SAC [51] with (sparse) reward defined as +1 inside T. -1 inside F. and 0 everywhere else; (2) single-agent RARL [32] with and without domain randomization (DR); and (3) adversarial SAC with the above sparse reward. We also compare to critic (value-based) filter, which intervenes when (1, ))) < with the threshold determined by running a parameter sweep and used in all experiments. Implementation details are in Appendix B.
4.1 Physical Results
Safe walking within and beyond the ODD. We evaluate the effectiveness of our proposed game- play filter in terms of both safety and disruption of task performance. We run similar experiments with baseline methods for rough comparison purposes but caution that, due to the impossibility of reproducing identical conditions, these results should not be taken as a fine-graine tne-ggained quantitative comparison between methods. Such a comparison is conducted at scale, albeit in simulation, in Section 4.2. Table shows the results for the S40 robot, subject to tugging forces and irregular ter- rain (not considered in the ODD), and Table 2 shows the results for the Go2 robot under larger range of tugging forces (up to 4x the ODD bound). Our proposed gameplay safety filter is remark-
Table I: We evaluate S40 walking under tugging forces (modeled) and irregular terrain (unmodeled). We record the fraction of safe (fall-free) runs, the frequency of filter interventions (as a fraction of all time steps) and the average time taken to reach the goal. In the tug test, we also report the fraction of withstood attacks (tugs separated by at least 1.0 s), both overall and restricted to those roughly (within 10% error) inside the ODD bound of 50 N. and the peak force statistics in successful/failed runs. The gameplay filter completes all terrain runs safely and withstands almost all tugs (even out- of-ODD) without impractically hindering task progress.
Tugping Forces
Irregular Terrain
Successfall Runs
Faiked Runs
Successful Runs
Policy
Safe/All Runs
All (wishin 100% ODD)
Withstood Attacks
Filter Frrg
Tgeal
praak
Safe/All Runs
Filter Freq.
Tpel
7/10
5355 (33/35)
0.17
26.3
67.5N
70.5K
59.8
52.7N
10110
0.19
41.2
22/28 (10/15)
0.10
26.8
73.7N
80.9N
53,6N
40.0N
0.22
33.5
0/10
(1/5
56.5N
41.4N
5/10
16.4
Table 2: We evaluate Go2 under tugging forces, comparing our gameplay filter against the robot's production-gaane grade walking policy. The gameplay filter withstood stronger tugs and had fewer failures. remained safe for all 5 runs in which tugs remained within the ODD bound (50 N) and only failed under forces of at least 2 the ODD bound (while resisting some forces of over 4 x the ODD bound)
Togying Forces Succesoful
Policy
Failed Runs
Safe/Toial Runs
Jepak
geleak mnn
All Runs (Runs within ODD)
8/10 (515)
42.4N
215N
105 7N
104,4N
P
0/10 (0/10)
52.7N
15.3 N
balk-in
7/10 (5/5)
23.0N
134.7N
IN
94,5N

---

Table 3: Maximum force magnitude withstood by the S40 with various policies at different tugging angles, Our learned fallback π outperforms the task policy and other safety fallback basclines and has comparable robustness to the policy used in the target set.
Maximum Force
Safety pulicies from reward based RL and ISAACS with the avoid-only objective fail immediately before applying force. The policy ahle withstand this mugnitude of forve. Be- cause the policy made the quadruped move in the tugging direc- tion, were not able to apply larger force 10 pull attempts.
Algorithm
Low
High
Low
High
IN
61,IN*
99, 99.3N'
59.IN"
100.5N 5N"
150.3N
121,6"
121,9"'
KARL + DR
46-4N
45N
57.2N
72.IN"
83.2N
96.9N
82.8K*
59N
151.9N
173.7N
140.3N
142.6N
Safe Rate
Filter Frequency
Distance (m)
0.3
reachavaid --- task only
02
0
100 300
300
10
100
300
Gameplay Horizon
Figure 3: Sensitivity analysis of gameplay horizon and monitoring criterion. The proposed gameplay safety filter, utilizing the reach-avoid criterion, maintains 100% safe rate (fraction of failure-free runs) for all gameplay horizons. As the horizon shortens, the gameplay filter with reach-avoid crite- rion only experiences decrease in filter efficiency without compromising safety. In contrast, the fil- ter with avoid-only criterion shows more safety violations than the task policy under short horizons. Overall, our proposed gameplay filter shows both higher safe rates and distance traveled (measured horizontally between the initial and final position) than the critic safety filter and task policy.
ably robust across robot platforms and test conditions; while not unbeatable outside of the specified ODD, it still withstands large tugging forces before violating the safety constraints. Importantly, the gameplay filter does not disproportionately interfere with the task-oriented actions: it maintains comparable filter frequency and task performance as the critic filter while drastically reducing safety failures. Fig. shows the gameplay filter in action on the S40, dynamically counterbalancing tugs or springing into a wide stance. Time plots of tugging forces in all $40 runs are given in Appendix D.
External forces. We measure the maximum tugging force withstood by various safety policies and filters, reported in Table 3. We pull the quadruped from different directions, with "low" indicating angles in the range 0-0.1, 0.4] rad, and "high" in [0.5, 1.0) rad. The employed my can withstand 150 N from all directions, but the toneeece- counterpart (RARL+DR) is vulnerable to the tugging from the left and can only withstand 43 N. This suggests that DR struggles to capture the worst-case realization of disturbances in a bounded class. This arises from its inherent nature: as the dimension of the disturbance input increases, the likelihood of the random policy simulating the worst-case disturbance decreases exponentially. Further, we notice the reward-based RL baselines and ISAACS with avoid-only objective fail almost immediately by overreacting and flipping over. Reach-avois policies behave more robustly by bringing the robot to stable stance. We also include tests for task policy ब and the fixed-pos pose policy xJ (used when the state is in the target set). We observe that ISAACS control actor is strictly better than ق and is comparable to TT.
4.2 Simulated Results
Bespoke ultimate stress test (BUST). To test each policy's 's robustness when taken to the limit, we RL-train a specialized adversarial disturbance o. to exploit its safety vulnerabilities (Table 4).
For each ribtt-disturbance policy pair, we play 1,000 finite horizon games and record the safe rate- overall fraction of failure-free runs. All pairs use the same set of 1,000 initial states. We observe that x is vulnerable to all xi. while the proposed gameplay filter is only exploited by its associated
7

---

Table 4: We perform bespoke ultimate stress test (BUST) of each control scheme in simulation by explicitly RL-training a specialized adversarial disturbance to find and exploit its vulnerabilities. We also sample random disturbances uniformly from D (I"" from its extreme points (x ) and observe that the domain randomization test is far less challenging (all safety methods excel).
Si (#o)
< (1
1
0.44
0.0
0.0
0.0
0.0
0.03
0.03
0.35
0.45
0.89
0.37
0.34
0.10
0.44
0.86
0,86
BUST disturbance TH (chsamc). Further, the robustness of dgeae pushes T. (disamc) to learn effective attacks that also exploit other policies (the third column has the lowest safe rates compared to other columns across the board). The last two columns show the safe rate under random disturbances, All safety filters and safety policies remain at remarkably high safe rates, suggesting that our ad adversarial BUST evaluation method establishes more demanding safety benchmark for policies than DR.
Sensitivity analysis: reach-avoid criteria vs. avoid-only. We evaluate the significance of using reach-avoid criteria in the gameplay filter by performing a sensitivity analysis of the horizon in the imagined gameplay. Fig. shows that the gameplay filter with reach-avoid criteria still remains 100% safe rate even when the gameplay horizon short (H 10). In contrast, an "avoid-only" gameplay filter that only requires not reaching for H steps incurs more safety violations as the horizon decreases. The difference is due to shorter imagined gameplay resulting in more frequent filter intervention for reach-avoid criteria but overly optimistic monitoring for avoid-only criteria (oblivious to imminent failures beyond H). Further, as the gameplay horizon increases, the reach- avoid gameplay filter's intervention frequency decreases.
5 Conclusion
This work presents a game-theoretic learning approach to synthesize safety filters for high-order, nonlinear dynamics. The proposed gameplay safety filter monitors system safety through imagined games between its best-effort safety fallback policy and learned virtual adversary, aiming to realize the worst-case uncertainty in the system. We validate our approach on two different quadruped robots under strong tugging forces and unmodeled irregular terrain while maintaining zero-shot safety. An exhaustive simulation study is also performed to rigorously stress-test the approach and quantify its reliability and conservativeness.
Limitations. Despite the strong empirical robustness in both simulated and physical experiments, we do not have strong theoretical guarantees on convergence of offline gameplay learning, and therefore learned disturbance policy can in general be expected behave suboptimally in least certain regions of the state space. Naturally, the potential implications suboptimal -roottrlly-oost-asss) disturbance model may lead the gameplay rollout to erroneously conclude that a proposed course of action is safe, only to then be met by an ODD realization that unexpectedly drives the robot into a catastrophic failure state. Without strong theoretical assurances that for now remain elusive, this is not a method that should be placed in sole charge of truly safety-critical system where an eventual catastrophic failure can carry inadmissible cost.
quite serious, since
The remarkably high effectiveness demonstrated by the gameplay filter across various within-ODD experiments and even under out-of-ODD conditions could indicate that this new type of filter does in fact enjoy desirable properties yet to be established. This calls for future theoretical work at the intersection of game-theooe theerettccettc reinforcement learning and nonlinear systems theory. In parallel, we research leverage the computational scalability and de facto robustness of gameplay filters tackle ongoing challenges in robot learning, for example for safe acquisition of novel skills well rapid detection of shifts in operating conditions enabling safe runtime adaptation of ODD assumptions.
see an opportunity for application-drine
8

---

Acknowledgments
This work has been supported in part by the Google Research Scholar Award and the DARPA LINC program. The authors thank Zixu Zhang for his help in preparing the Go2 robot for experiments.
References
[1] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Proc. Robotics:
Science and Systems, 7 2021. 000:10.15600/RS3.222...2
[2] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger C. Finn, and H. Zhao. Robot parkour learning. In Conf. Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 73-92, 11 2023. URL https://proceediig
mlr. prwsspesssss...... /t2////22/////// /tt///uaaanaaa.../a// html.
[3] K.-C. Hsu, A. Z. Ren, D. Nguyen, A. Majumdar, and Fisac. Sim-to-lab-to-reall Safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence, 3 314:103811, 2023. ISSN 0004-3702.
URL
[4] G. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. In Proc. Robolics: Science and Systems, New York City, NY, USA, 6 2022.
[5] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Var- ley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners. URL https://openreet wowperrrviee.eee//ormll net/forum? id=4ZK80DNyFXx.
[6] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-Jacobi reachability: A brief overview and recent advances. In Proc. IEEE Conf. Decision and Control, pages 2242-2253, 2017. doi: 109/CDCC.011.8266333
[7] M. Bui, M. Lu, R. Hojabr, M. Chen, and A. Shriraman. Real-time Hamilton-Jacobi reachability analysis of autonomous system with an FPGA. In IEEE/RSJ Int. Conf. Intelligent Robots & Systems, pages 1666-1673, 2021. doi: 10. 1090000OS0 262.2021 19396410
[8] R. Mattila, Y. Mo, and R. M. Murray, An iterative abstraction algorithm for reactive correct- by-construction controller synthesis. In Proc. IEEE Conf. Decision and Control, pages 6147- 6152, 2015. doi: 10. 10.110990
[9] Q. Nguyen and K. Sreenath. Robust safety-critical control for dynamic robotics. 6773):1073- 1088. ISSN 1558-2523. doi: 10. 10A11099...2222233595155
[10] T. G. Molnar, R. K. Cosner, A. W. Singletary, W. Ubellacker, and A. D. Ames. Model-free safety-critical control for robotic systems. 7(2):444-99) 951. ISSN 2377-3766. doi:10. 109/LRA. 2021.3135569.
[11] T.-Y. Yang, T. Zhang, L. Luu, S. Ha. J. Tan, and W. Yu. Safe reinforcement learning for legged locomotion. In JEEE/RSJ Int. Conf. Intelligent Robots & Systems, pages 2454-2461 2022. 1oi: 010:0001000000044422222299999999
[12] T. He. C. Zhang. W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision-fii high-speed legged locomotion. In Proc. Robotics: Science and Systems, 2024.
[13] K.-C. Hsu, H. Hu, and J. F. Fisac. The safety filter: A unified view of safety-critich control in autonomous systems, 2023, URL https /ttp/aaavvvv///a///av//aa.//..aa org/ abs/2309. 05837.

---

[14] K. P. Wabersich, A. J. Taylor, J. J. Choi, K. Sreenath, C. J. Tomlin, A. D. Ames, and M. N. Zeilinger. Data-driven safety filters: Hamilton-Jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137- 177, 2023. doi: (009/MCCS.222332999111
[15] K. L. Hobbs, M. L. Mote, M. C. Abate, S. D. Coogan, and E. M. Feron. Runtime Assurance for Safety-Critical Systems: An Introduction to Safety Filtering Approaches for Complex Control Systems. (3(2):28-65. 65. ISSN 1941-000X. doi: 10. 109/MC9/MMSS. 1000//MC..222333333333333 URL https: // //ieeexplore. ieee. wttpeeeeooooccuuttt0o00111..
[16] I. Mitchell, A. Bayen, and Tomlin. time-dependent Hamilton-Jacob formulation reach- able sets for continuous dynamic games. IEEE Transactions Automatic Control, 50(7):947- 957, 2005. ISSN 1558-2523. 1oi:10.1109/TAA. 109/TAC..00558881444
[17] J. F. Fisac, M. Chen, C. J. Tomlin, and S. S. Sastry. Reach-avoid problems with time-varying dynamics, targets and constraints. In //yrrdd Systems: Computation and Control, pages 11-20, Seattle, WA, USA, 2015. 1oi: (400011144/7228662786666
[18] I. M. Mitchell. The flexible, extensible and efficient toolbox of level set methods. Journal Scientific Computing, 35(2):300-329) 2008. doi: 100::000000//10015000---111111
[19] M. Bui, G. Giovanis, M. Chen, and A. Shriraman. OptimizedDP: An efficient, user-friendly library for optimal control and dynamic programming, 2022. URL attps://arxiv.orr abs/2204 05520.
[20] A. D. Ames, Coogan, Egerstedt, G. Notomista, Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In European Control Conference, pages 3420- 3431, 2019. 09011002991//EE022211999969600
[21] X. Xu, J. W. Grizzle, P. Tabuada, and A. D. Ames. Correctness guarantees for the composi- tion of lane keeping and adaptive cruise control. IEEE Transactions Automation Science and Engineering, 15(3):1216-122 1229, 2017.
[22] L. Wang, Han, and M. Egerstedt. Permissive barrier certificates for safe stabilization using sum-of-squares. In Proc. American Control Conference, pages 585-590, 2018. doi: 10.2390939 ACC.2018.8431617
[23] L. Lindemann, H. Hu, A. Robey, H. Zhang, D. Dimarogonas, S. Tu, and N. Matni. Learning hybrid control barrier functions from data. In Conf. Robot Learning, pages 1351-1370, 2021.
[24] X. Xu, P. Tabuada, W. Grizzle, and A. Ames. Robustness control barrier functions for safety critical control. 48(27):54-61, 2015. ISSN 24058963, do:: 10t.:/...111...... 200 URL https: //t:::///iiiiinn/eee weaebbeeellllii elsevier http:///wc..aaaoooo//////////aaa
[25] A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni. Learn- ing control barrier functions from expert demonstrations. In Proc. IEEE Conf. Decision and Control, pages 3717-3724, 2020. doi:10. 11010001009CD0022222000000000000
[26] Choi, D. Lee, Sreenath, J. J. Tomlin, and S. L. Herbert. Robust control barrier-value functions for safety-critical control. In Proc. IEEE Conf. Decision and Control, pages 6814- 6821, 2021. doi: 10. 10111//0DD44444.20222 9683085.
[27] A. Robey, I.. Lindemann, S. Tu, and N. Matni. Learning robust hybrid control barrier functions for uncertain systems. IFAC-PapersOnLLne, 54(5):1-6, 2021.
[28] S. Bansal and C. J. Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In Proc. IEEE Conf. Robotics and Automation, pages 1817-1824, 2021. doi: 00.10090CRAA4899022220..6.66
10

---

[29] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin. Bridging hamilton- jacobi safety analysis and reinforcement learning. In Proc. IEEE Conf. Robotics and Automa- fion, pages 8550-8556, 2019. 10:1001109//CRAA211.817788
[30] H. Bharadhwaj. A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg. Conservative safety critics for exploration. In Int. Conf. Learning Representations, 2021. URL https: /ppeneeview. www/teeunu/iieeee net//oorum?ideia00000tt.
[31] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, J. E. Gonzalez, Ibarz, C. Finn, and K. Goldberg. Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones. IEEE Robotics and Automation Letters, 6(3):4915 (29)4295--492,, 2021. doi:10. 109/ L01.2021.0700252.
[32] K.-C. Hsu, Rubies-Royo, C. J. Tomlin, and J. Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. In Proc. Robotics: Science and Systems, 2021. 1of:10..5007//S..2022.... 2021XXX1077.
[33] K. Wabersich and M. N. Zeilinger. Linear model predictive safety certification for learning- based control. In Proc. IEEE Conf. Decision and Control, pages 7130-7135, 2018. doi: 10.1 0..110//CDC.211.888111181
[34] K. P. Wabersich and M. N. Zeilinger. predictive safety filter for learning- based con- trol of constrained nonlinear dynamical systems. Automatica, 129:109597, 2021. ISSN 0005-1098. oothtt///oabooy////ll/f//ummuaip
URL https://wwe
[35] O. Bastani and S. Li. Safe reinforcement learning via statistical model predictive shielding. In 20000//RSS.22..XIIIII
Proc. Robotics: Science and Systems, 7 2021.
[36] A. Leeman, J. Köhler, S. Bennani, and M. Zeilinger. Predictive safety filter using system level synthesis. In Learning for Dynamics & Control, volume 211 of Proceedings of Machine Learning Research, pages 1180-1192, 6 2023. URL attps://proceeddie mlr. prrsss/ v211 v211/leeman23a. httml
[37] A. Ramesh Kumar, K.-C. Hsu, P. Ramadge, and F. Fisac. Fast, smooth, and safe: Implicit control barrier functions through reach-avoid differential dynamic programming. JEEE Control Systems Letters, 7:2994-2999, 2023. doi:
[38] K.-C. Hsu, D. P. Nguyen, and J. F. Fisac. ISAACS: Iterative soft adversarial actor-critic for safety. In Learning for Dynamics & Control, volume 211 of Proceedings of Machine Learning Research, 2023. URL https: /ppps////occcccesss....e mmr. press/ petess/v211/ee w2tssssu22//ss html.
[39] H. Hu, M. Fazlyab, M. Morari, and G. J. Pappas. Reach-sdp: Reachability analysis of closed- loop systems with neural network controllers via semidefinite programming. In Proc. IEEE Conf. Decision and Control, pages 5929-5934, 2020. doi: 110:00.109/CDC422422200000 930429...
[40] E. Luo, N. Kochdumper, and S. Bak. Reachability Analysis for Linear Systems with Uncer- tain Parameters using Polynomial Zonotopes. Proceedings of the 26th ACM International Conference on Hybrid Systems: Computation and Control, HSCC '23, pages 1-12. Associa- tion for Computing Machinery. ISBN 9798400000330.. doi: 112 URL https://dl. acm. www./oogg/ooooooooo wtt.e111433577//0
[41] J. Bird, H. C. Pangborn, N. Jain, and Koeln. Hybrid zonotopes: A new set representation for reachability analysis of mixed logical dynamical systems. Automatica, 154:1 107, 2023. ISSN 0005-1098. http:///pp////..oo////oooo//o.d.
URL https: //www.

---

[42] J. Reher and A. D. Ames. Dynamic walking: Toward agile and efficient bipedal robots. Annual Review of Control, Robotics, and Autonomous Systems, 4(1):535-572, 2021. doi:10. 146/ amnure-contto--0710000--00
[43] H. Lai, W. Zhang, X. He, C. Yu, Z. Tian, Y. Yu, and J. Wang. Sim-to-real transfer for quadrupedal locomotion via terrain transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5141-5147. IEEE. ISBN 9798350323658. doi: 10. N0.1109/CCA44881... 010891.202311000 10160497 URL 10160497/.
ieee. org/document.
[44] L. Campanaro, S. Gangapurwala, W. Merkt, and I. Havoutis. Learning and deploying ro- bust locomotion policies with minimal dynamics randomization. In Learning for Dynamics & Control, volume 242 of Proceedings of Machine Learning Research, pages 578-590. PMLR, 15-17 Jul 2024. URL https:
wtt.ttppe/pprrreeiiiggggg http://poooeeeei...iae mlr.....r..... wtw.s/v2e22e campanaro24a. 2aap:///......ae/a////////a/aaaa html.
[45] Q. Nguyen and K. Sreenath. Optimal Robust Time-Varying Safety-Critical Control With Ap- plication to Dynamic Walking on Moving Stepping Stones. In Proc. ASME Dynamic Sys- tems and Control Conf. volume Volume 2: Mechatronics; Mechatronics and Controls in Ad- vanced Manufacturing: Modeling and Control of Automotive Systems and Combustion En- gines; Modeling and Validation; Motion and Vibration Control Applications; Multi-Agent and Networked Systems; Path Planning and Motion Control; Robot Manipulators; Sensors and Ac- tuators; Tracking Control Systems; Uncertain Systems and Robustness: Unmanned, Ground and Surface Robotics; Vehicle Dynamic Controls; Vehicle Dynamics and Traffic Control, page V002T28A005, 10 2016. doi:10. 10. 1/D.C//0//SC
[46] Z. Gu, Y. Zhao, Y. Chen, R. Guo, J. K. Leestma, G. S. Sawicki, and Y. Zhao. Robust- coommotion-byloggee Perturbation-resilees bipedal locomotion via stl-based model predictive control. arXiv preprint arXiv:2403. 2024. URL https: //arxi///////////aa arg/abs/2403 15993.
[47] O.-R. A. D. O. committee. Taxonomy and definitions for terms related driving automa- tion systems for on-road motor vehicles. URL https: //t::////wwwwww/ org/ j3t/ctttonttt 202104.
[48] Laine, C.-Y. Chiu, and C. Tomlin. Eyes-Closed Safety Kernels: Safety of Autonomous Systems Under Loss Observability. In Proc. Robotics: Science and Systems, Corvalis, Oregon, USA, 2020. 4bl::00156077R35.2200.....
[49] Z. Zhang and F. Fisac. Safe Occlusion-Aware Autonomous Driving via Thee--heeoeetic Active Perception. In Proc. Robotics: Science and Systems, 2021. doi: 10:10..560//RS.200. XVII.066.
[50] H. Hu, Z. Zhang, K. Nakamura, A. Bajesy, and J. F. Fisac. Deception game: Closing the safety- learning loop in interactive rohot autonomy. URL. 0o2Jgy1zMUc.
[51] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Int. Conf. Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870, 7 2018. URL /t
proceedings mlr. www.:mm.w........11. press//80/haarroj] html..
[52] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybull/ org, 2016-2021.
12

---

Appendices
A Frequently Asked Questions
We discuss some design choices and implications of our method using an informal FAQ format.
Why choose worst-case safety and not probabilistic analysis? Although not as established in the robot learning community, robust/worst-case formulations are widely used across engineering. Their key advantage is that they can enforce systematic handling of all scenarios in a well-defined class, even if some of them are highly unlikely-e.g, e., the robot must withstand all (rather than most) external forces of up to 50 N. even the unlucky push that happens to maximally disturb its stance. This is consistent with much of the safety analysis found in bridges, elevators, automobiles, aircraft, and other safety-critical engineering systems, in great part because it facilitates clear-cut social contract between their designers and the broader public. For example, we do not certify elevators for 95% of loads up to 300 kg or bridges for 99% of earthquakes up to magnitude 8, but rather all such loads and earthquakes, and we treat any loss of safety within the specified bounds as a serious failure to comply with the promise made to society. As robots and autonomous systems become more widely deployed, we argue that their safety should be certified and held to similar standards, at least truly safety-critical settings where people could otherwise get hurt.
Isn't worst-case safety too conservative to be useful? Actually, this is a common misconception. Robust/worst-car assessments are not intrinsically more or less conservative than probabilistic ones: this depends entirely on what set and distribution we choose to run these assessments against. The term "wworstcasse" doesn't mean a system must preserve safety in the worst conceivable scenario (whatever that means), but rather under all conditions-iicluding the worst one-in specified set. Worst-case safety lets designers and regulators draw this line (the ODD boundary), and it ensures that the system then maintains safety across all certified (in-ODD) conditions. If your robot's 's behav- ior is "too conservative" this means it's guarding against eventualities you don't 't really care about: just exclude them from your ODD. But, if you do want safety under these conditions, then your robot is not actually too conservative: it's doing what it should. With the gameplay filter, you are never left wondering: each time it overrides the task policy, it logs the specific future it's preempting. Then, only one question remains: did you or did you not want your robot to avoid that hypothetical crash? Worst-case safety is extremely powerful, and it lets you control exactly what situations your robot is required to handle. You just need to be ready to answer to some hard what-if questions.
What does it mean for the proposed gameplay filter to approximate a perfect filter? If we had the exact solution to the Isaacs reach-avois equation (5), our gameplay rollouts would be necessary and sufficient for safely reaching T H (or fewer) steps. Since is typically chosen to be a broad, naturally reachable class of robot states (e.g., coming to a stable stance for walking robot or pulling over for an autonomous vehicle), safely reaching T within long enough horizon H is possible as long as remaining safe is possible in the first place. In other words, the sufficient reach- avoid condition becomes a tight approximation of the all-time safety condition. We can observe this phenomenon in Fig. 3, where the reach-avoid filter's overstepping vanishes with long H.
Why is computing gameplay rollout better than just querying the learned reach-avoid critic? In theory, the critic should make fairly accurate predictions of game outcomes after training. In practice, we have found that it's often unreliable and/or overly conservative. A key advantage of the gameplay rollout is that the uncertainty linked to the learning-based based safety analysis is much more structured: the robot's 's future safety fallback is perfectly predicted (since it will be implemented as- is), and the dynamics can be reliably simulated given players' actions, so all uncertainty falls on the learned disturbance. One very useful implication of this structure is that, even if the disturbance is suboptimally adversarial, predicted gameplay rollout ending in a safety failure constitutes valid certificate (i.e., a proof) that there exists an ODD realization in which the robot will violate safety if the filter does not intervene immediately. That is, we know the gameplay safety monitor isn't 'tt falsely crying wolf-we can't 't prove anything like that about the black-box neural safety critic's 's predictions.
13

---

Why is reach-avoid preferable if it's more conservative than avoid-only? This is an important aspect of predictive safety filtering and relates to a deeper tenet in safety engineering philosophy: whatever the safety boundary is (i.e., a strategy that is "just safe enough"), it preferable to approach from the safe side than from the unsafe side. practice, we don't know a priori how many prediction steps H we need avoid being blindsided by future failures just beyond the lookahead horizon. When in doubt, it's preferable to risk being overly conservative than to risk losing safety.
Having a terminal state constraint is common in MPC, how is reach-avoid different? The use of a terminal controlled-invariant set in MPC is well established and ensures recursive feasibility. Our choice of reach-avoid over an avoid-only safety condition is an instance of the same principle. An important difference is that the (also well-established) reach-avoid condition gives our filter extra flexibility by allowing the gameplay trajectory to reach the forever-safe set 'I any time within the horizon. This reduces conservativeness and often lets to terminate the gameplay rollout early.
How do you determine T? In practice, a suitable T is obtained from domain knowledge, offline computation, pre-deployment learning, or some combination, often in the form of a stability basin (region of attraction) around a desirable class of equilibrium points sufficiently away from failure. For example, most robots can be robustly stabilized around static or steady cruising configurations by comparatively simple linear feedback controllers (e.g., most modern walking robots ship with built-in controllers that can stabilize them around default stance). Larger all-time safe regions may be found by (robust) Lyapunov analysis or even optimized through control Lyapunov functions.
What are the implications of the choice of T? Broadly speaking, the larger the T we can char- acterize offline, the easier the job of the gameplay filter at runtime, and, potentially, the fewer steps we'll need to reach it from more dynamic configurations. In fact, in the extreme case, we could be remarkably lucky and find T 52" in which case, the gameplay filter's job is made much easier, since all candidate actions that are safe will keep the state in T, immediately terminating the rollout check. Conversely, all actions that leave T are unsafe and the gameplay rollout will not be able to return to T. In order to avoid initializing the gameplay filter from a no-win scenario, designers should ensure that T contains the range of expected robot deployment conditions (A)) in the ODD.
Why aren't you using onboard cameras or lidar? Our empirical focus in this paper is on demon- strating automatically synthesized safety filters that account for the full-order (36-D) walking dy- namics of quadruped robots. We think that the simplest and clearest demonstration of this concept is by having the filter only consider the robot's own state (proprioception) without accounting for the environment, obstacles, etc. (exoception). That said, incorporating information about the robot's surroundings can be extremely valuable- and often critical to safety. We are very excited by the scalability and generality that new safety approaches like the one we present this paper seem to enjoy, and we expect they will soon unlock full-order safety filters that incorporate rich exoceptive information in real time, whether straight from raw sensor data through intermediate representa- tions provided by the perception and localization stack.
14

---

B Implementation Details
Robot hardware. Both the Ghost Robotics Spirit S40 and Unitree Go2 have built-in IMUs to obtain body angular velocities and linear acceleration, and internal motor encoders to measure joint positions and velocities. The S40 has no foot contact sensing; the Go2 receives Boolean contact signal for each foot. Neither robot's safety filter is given access to visual perception.
Gameplay filter runtime implementation. To easily deploy our gameplay safety filter across two different robots for the physical experiments, encapsulate its computation inside a ROS service, which we run an offboard computer. Each robot's onboard process calls this service wirelessly (approximately 3.5 times per second) passing its current state estimate and proposed course of action /newcandidate task policy; the offboard server then simulates a H Hsttep gameplay for a fixed horizon (for us, = 300 or 3 s) and returns a Boolean indicating which policy to use for the next L time steps; our choice of I, = 10 accounts for the wireless round trip, which makes up a significant fraction (approximately 70%) of the total latency.
We note that the computational resources used for the offboard computation are comparable to those available on current mobile robot platforms. In particular, the entire simulation and filter logic was run on one single core of an Intel 17-1185G7 processor at 3GHz. For comparison, the Go2 is equipped with a second computer (not used our experiments) with 6-core 8GB NVIDIA Jetson Orin Nano processor 1.5 GHz. We estimate that the total latency of the gameplay filter run fully onboard the Go2 with the same simulator would be roughly comparable, possibly lower given the absence of wireless roundtrip.
Operational design domain. The safety filter is computed for a fairly simple ODD, defined by the nominal robot simulator perturbed by forces of up 50 N applied anywhere the robot's torso; the disturbance adversary acts by vector d E D Ie6 encoding what force to apply and where. We intentionally limit the ODD to only consider flat ground. The failure set F is defined as all fall states, in which any non-foot robot part makes contact with the ground. The deployment set and controlled-invariant set Xn = T are chosen empirically to contain all four-legged stances with a lowered torso, around which the robot is robustly stable with a simple leg position controller T!.
Test conditions. The irregular terrain is 2 m X 4 m area with a 15-degree incline along one edge, and two memory foam mounds, 5 cm and 15 cm high, positioned 1.8 m from each other. Tugging forces are applied manually through a rope, attached to the robot's torso and to a motion-tracked dynamometer with rated capacity of 50ON, 0.1 N resolution and sampling rate of 1000 Hz and set provide audiovisual alerts at 80% and 100% of the ODD limit
Policies. The learned control and disturbance actors, as well as the safety crities, are independent of the robot's absolute position Py and heading angle 0g: of these, only distance to the ground has effect the dynamics, but since it is hard observe without vision, we do not make it available. In the case of the Go2 quadruped (but not the S40), the policies additionally depend on the discrete contact state, encoded as Boolean (true/false) indicator for each foot. In simulation, each neural network policy receives as input the ground-truth state of the robot in the simulator; in hardware experiments, they instead receive a state estimate computed by the robot's on-board perception stack. Each policy is implemented by a fully-connected feedforward neural network with hidden layers of 256 neurons, and crities have hidden layers with 128 neurons. We handeraft task policy using inverse kinematics gait planner for forward/sideways walking. We use low-level PD position controller that outputs torques T¹ = Kg(00)) Ka wi to the robot motor controller with K,, Ka the proportional and derivative gains.
State and action spaces. For the scope of this paper, aim construct proprioceptive safety filter that relics onboard estimation of the robot's kinematic state but exoceptive information
15

---

(from camera, lidar, etc.) about the surrounding environment." state and action vectors as follows:
We encode the quadrupedal robots'
x: = [z: Pus Uz Uy Uz Oz By Oz. Wr. wy, Wz, (0]) [wi}]
u [[50]]]
with Py. Ps the position of the body frame with respect to a fixed reference ("world") frame; Ux. uy Vz the velocity of the robot's torso expressed in (forward-left-up) body coordinates; On, Oy Og the roll, pitch, and yaw angles of the robot's body frame with respect to the world frame;" We: Wl, W/ the body frame's axial rotational rates; and wj. 883 the angle, angular velocity, and commanded angular increment of the robot's 7th joint.
The above constitutes full-order state representation of the robot's 's idealized Lagrangian mechanics. A total of 18 generalized coordinates encode the 6 degrees of freedom of the torso's 's rigid-body pose in addition to the configuration of 3 rotational joints (hip abduction, hip flexion. and knee flexion) for each of the 4 legs; the robot's rate of motion is expressed through 18 corresponding generalized velocities, for total 36 continuous state variables. We discuss discrete contact variables below.
The robot's control authority is achieved by independently modulating the torque applied on each of its 12 rotational joints by electric motor; in modern legged platforms, these motors typically have dedicated low-level controllers, so our control policy sends a tracking reference to each motor controller rather than directly commanding a torque.
Finally, the disturbance is modeled as an external force that can act on any point of the robot's torso and in any direction of Euclidean space, with bounded modulus. The specified range of admissible disturbance forces is discussed below.
Black-box simulator(s). The dynamical model is implemented by the off- off-the-shelf PyBullet physics engine [52] using the standardized robot description files made available by the manufacturers of each platform. Our method treats the simulator as black-box environment for both training and runtime safety filtering, allowing the engine and/or robot model be easily swapped out. The generality and modularity of our approach is perhaps best illustrated by the fact that synthesized and deployed the safety filter for the Go2 robot using identical hyperparameter values as for the S40 robot. Our only modification, other than replacing the robot model in the physics engine, was to append 4 state components to each neural network's input space to account for foot contact information; we note that even this straightforward addition is entirely optional, since we could have alternatively constructed safety filter that simply disregarded the extra sensor data.
Safety specification. We are interested in preventing falls, understood as any part of the robot other than its feet making contact with the ground. To encode the failure set of all such falls with simple margin function, we define a small number of critical points Pe, including the corners of a (tight) 3-D bounding box around the robot's torso as well as its four knee joints. The failure margin is
g(x) min { minn (min(=comer) Zcomer,gs min (=xnse)
Sknee
with =corer the vertical distance to the ground of the hh robot body corner point and = the vertical distance to the ground of the 7th robot knee point. The target (all-time safe set) is defined as a narrow neighborhood of a static stance with all four feet on the ground and a sufficiently lowered torso,
'Ranged perception can improve the robustness of walking controllers by sensing terrain geometry and texture, and it is strictly needed for ODDs including unmapped or moving obstacles. Full-order legged robot safety filters combining proprioception and exoception have significant potential and ripe for investigation. *For the purposes of this demonstration, we find that an Euler angle representation of body attitude performs adequately and makes the failure set straightforward to encode. In general, a quatemnon based representation may be preferable, avoiding the risk of computational issues in the neighborhood of singularities (at 0, = =
16

---

chosen so that the robot is robustly stable with simple stance controller. The target margin is
((x)) = min
@ Wal. B - -kyl a w
8 Uiel Ū - 10y1, T - 10al=
max = #comee)) Zfoot max { (=foot) }
corner,
with 2oor the vertical elevation the th robot foot relative the ground. The threshold values we used for our failure and target specification are as follows.
Zeomer. 0.1 m
knee 0.05 m
zurner, f 0.4
Eoot 0.05
10"/s
6 0.2 m/s
Uncertainty specification. To account for uncertainty in the deployment conditions as well as gen- eral modeling error (or sim-to-real gap), our operational design domain (ODD) includes an external force that may push or pull any point on the robot's torso any direction with maximum magni- tude of 50 N:
= Fy,
where F F= [Fa, Fy. Fa] represents the force vector applied at position defined by P Py .P in the F E 0-1.1 0.1), p J0, 0.05 m. The red arrows in the imagined gameplay
body coordinates p Py of Fig. show examples of learned adversarial disturbance.
17

---

C Extended Evaluation
To further demonstrate the strengths of our approach and shed light on its superior scalability to complex robot dynamics, we compare the gameplay performance of the self-play-trained controller and disturbance policies as training proceeds. The results in Figures and suggest that the dense temporal difference signal in reach-avoid games plays determining role in enabling data-efficient learning, while previously proposed safety methods that use reward-basee RL with (sparse) failure indicator consistently require more training episodes before starting to learn meaningfully robust safe control strategies.
0.8
0.6
0.4
0.2
Step
500k
1M
1.5M
2M
2.51
Figure 4: Safe rate achieved by the robot controller against the co-trained adversarial disturbance as the adversarial RL synthesis proceeds under reach-avoid objective (blue) and reward-based objective (red).
4e+5
3e45
2e45
1e45
Step
IM
1.5M
2.51
Figure 5: Cumulative safety violation count as the adversarial RL synthesis proceeds under reach- avoid objective (blue) and reward-based objective (red).
D Detailed Tugging Force Plots
We provide time plots for all runs of the tug test experiment on the S40 robot (summarized in Ta- ble 3), displaying the magnitude of the tugging force over the course of each trial. We present all 10 runs for each of the three evaluated control schemes: gameplay filter giame, critic (value-based) filter geritic and unfiltered task policy 8 Each is annotated to show individual attacks, defined sequences of significant tug forces (2 10N) that are applied continually close together in time (less than s interruption within an attack). Conversely, distinct attacks are at least s away from each other, to ensure that the effects of the previous attack have died off before the next one begins.
Looking individual attacks within each run provides more fine- -naiggained insight on the perfor- mance of each control scheme under various disturbances (both within-ODD and out-of-ODD). Importantly. it allows attribute safety failure to the attack that immediately preceded it in given run, but mark all earlier attacks in the same as safely handled.
18

---

20
M
Time (5)
M M MM
Gameplay filter Tugging force experiment
50
50
50
50
50
Dlonk
M
Time (s)
Dookk Doviry Bock
POLRS
M
M
556 SOUNd
Foree [N]
(N) ForCa
(N)) 8O0o
(N) a3u2
(N) FOUo
Figure 6: Time plots of the force applied in all 10 runs of the Spirit S40 physical tugging test using the gameplay filter. Across the 10 runs, there were 56 attacks, 35 of which were roughly (up 10% error) within the ODD bound of 50 N. Plots in red indicate a failed run (ending in a fall), plots in blue indicate safe (fall-free) run.
19

---

Time (s)
M
M
2
Safely critic filter Tugging force experiment
M
M
50
50
50
WMM Mo
M
30
Time (s)
M MWW 14)
10 M
COR SOURS
ForCe [N]
(N)) Fonce
(N) FOno
(N) 8312
(N) FOUo
Figure 7: Time plots of the force applied in all 10 runs of the Spirit S40 physical tugging test using the critic -value-based) filter. Across the 10 runs, there were 28 attacks, 15 of which were roughly (up to 10% error) within the ODD bound of 50 N. Plots in red indicate a failed run (ending in a fall), plots in blue indicate a safe (fall-free) run.
20

---

Time (s)
Task policy Tugging force experiment
50
50
50
Time (s)
Foree [N]
(N) ForCa
(N) FO0o
(N) e3u2
(N) PoUo
Figure 8: Time plots of the force applied in all 10 runs of the Spirit S40 physical tugging test using the gameplay filter. Across the 10 runs, there were attacks, 5 of which were roughly (up 10% error) within the ODD bound of 50 N. Plots in red indicate a failed run (ending in a fall), plots in blue indicate safe (fall-free) run.
21


Reviews:
[{"review_id": "jn0zCC4V6M", "reviewer": "Reviewer_ScUX", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a new type of safety filter designed for high-dimensional robotic systems, such as quadrupedal robots. The proposed method, known as the \"gameplay filter,\" integrates scalable learned representations with model-based safety analysis to provide a lightweight safety assurance module for various robotic tasks. It operates by simulating \"worst-case\" scenarios in real-time, ensuring that the robot avoids actions leading to safety violations. The paper claims that the gameplay filter enhances safety without significantly limiting the robot's agility, and demonstrates its effectiveness through experiments on quadrupedal robots under various perturbations.", "summary_of_recommendation": "I recommend a major revision.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Disturbance Modeling: What specific types of disturbances are modeled in the experiments? The paper needs to clarify the experimental setup, including the nature and range of disturbances considered.\n\n2. Game Theory Clarification: The connection to game theory needs to be better defined and clarified. While the paper frequently mentions game theory, it lacks a clear explanation of how game-theoretic principles are applied in the proposed method.\n\n3. Implementation of the Simulator: Where is the simulator running during the experiments? Is it on the robot itself, or is it running on an external system? This information is crucial for understanding the feasibility and real-time performance of the proposed safety filter.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kQpbrL94GC", "reviewer": "Reviewer_ZswE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a real-time safety filter for legged robot locomotion via simulated adversarial self-play. A disturbance actor that simulates the sim-to-real gap is trained alongside and against a control actor using SAC-like scheme. Online, gameplay filters forward simulates the robot against learned disturbance actors, and uses the learned control policy as the fallback policy. Then a simple switching rule is used to select control actions.", "summary_of_recommendation": "Weak Accept, organization and clarity of the experiment section can be significantly improved, limitation section is missing and needs to be added. After rebuttal, clarity is improved and the limitation section addresses my biggest concern, and ratings are adjusted accordingly.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How are tugging forces decided in each experiment? Are they the same/similar across different baseline comparisons? If not, are there any metrics that can show that the experiments are still valid?\n2. Could you provide some quantification on how \"unmodeled\" the terrains are in the experiments?\n3. How do you deal with the cases when the stochastic policies produce \"false positive\" winning cases against the disturbance policy during forward simulation?\n4. What does \"specialized\" mean in the BUST experiment?\n5. What is the \"Distance\" in the third figure of Figure 3?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vs2usQZARy", "reviewer": "Reviewer_tETa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a “gameplay filter” that consists of simulating a potential worst-case disturbance online and choosing actions from either a task-oriented policy or a fallback policy at each timestep depending on whether the simulated gameplay results in success or failure. The gameplay filter is trained using a game-theoretic adversarial RL approach that trains a reach-avoid safety policy and disturbance policy via self-play. The authors instantiate this problem for a quadruped robot trying to stay upright. When compared to other safety filters, their approach is more robust to disturbances both in simulation and in real-world experiments. This paper is ultimately one of the first approaches to successfully roll out a safety filter for a physical quadruped on a full-fidelity (36D) model of the robot, instead of relying on reduced-order dynamics models, which most prior work has done.", "summary_of_recommendation": "This paper constructs a novel safety filtering method based on HJ reachability that simulates potential worst-case disturbances online to stay robust to them. Their approach is shown to work well for a quadruped robot both in simulation and on two different hardware platforms.   Based on the revised manuscript and authors' comments in the rebuttals, my main concerns about the value-based filter and limitations section have been addressed, and my review has been adjusted accordingly. I continue to recommend accepting this paper.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. What is $\\phi^{critic}$? This is not explicitly written out in the text, and it should be made clear if this is a value-based filter that uses the trained critic from the proposed adversarial RL framework to do the switching control. If it is such a filter, *why* does it perform worse than the proposed gameplay filter? If it is not such a filter, then the authors should do a comparison to a value-based filter using the trained value function.\n2. How can you generally define a margin function $l(x)$ for a control-invariant set, which may not just be a single static pose? (e.g. walking or running gait for a quadruped or trim controls for an aircraft)\n3. What is the ODD used for experiments, and can you split the results in table 1 for tugs in and out of the ODD?\n4. There is no limitations section, please add one.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The approach is interesting, but the reviewers raise several issues that should be addressed:

- Clarification is needed on the experiment setup, including the type of disturbances used
- The connection to game theory is quite unclear
- The authors should consider the effect of out-of-distribution states/actions on the method
- The novelty of the method is perhaps overstated, since much of this fits into the standard MPC framework

Please see reviewer comments for more details.

-----Post-rebuttal comments----

The rebuttal addressed several important issues brought up by the reviewers. While there is not a clear consensus among the reviewers, the overall sentiment about the paper was positive and I believe this work is above the bar for acceptance.
Decision: Accept