Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy
Authors: Haimin Hu
Zixu Zhang
Kensuke Nakamura
Andrea Bajcsy
Jaime Fernández Fisac
Venue: CoRL 2023 Poster (2023)
Keywords: Learning-Aware Safety Analysis
Active Information Gathering
Adversarial Reinforcement Learning

Abstract: An outstanding challenge for the widespread deployment of robotic systems like autonomous vehicles is ensuring safe interaction with humans without sacrificing performance. Existing safety methods often neglect the robot’s ability to learn and adapt at runtime, leading to overly conservative behavior. This paper proposes a new closed-loop paradigm for synthesizing safe control policies that explicitly account for the robot’s evolving uncertainty and its ability to quickly respond to future scenarios as they arise, by jointly considering the physical dynamics and the robot’s learning algorithm. We leverage adversarial reinforcement learning for tractable safety analysis under high-dimensional learning dynamics and demonstrate our framework’s ability to work with both Bayesian belief propagation and implicit learning through large pre-trained neural trajectory predictors.

Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy
Haimin Hu
Zixu Zhang*)
Kensuke Nakamura²
Andrea Bajesy?
Jaime Fernández Fisac!
Princeton University
Carnegie Mellon University
(haiminh,
edu, (kensuken, aba abajcsy
Sandrew | cmu, cmu.edu
Deception Game
Conservative
Optimistic
b(ped) 0.7
b(seg) = 0.3
8 0:35
(1-) 0.15
MB u.
(a) Road crossing scenario (running example).
(b) Deep trajectory prediction.
Figure 1: Our proposed Deception Game framework improves safety and efficiency of uncertain, multi-modal interactions with other agents by closing the safety decision loop around robot's runtime learning algorithm. (a) An autonomous vehicle is uncertain of the type of human crossing the road. Our method computes a safe, yet non-conservative control policy by explicitly accounting for how interaction uncertainty will evolve. Color- shaded areas denote the complement of the safe projected onto the 2D plane. (b) Our framework can scale systems using modern data-driven trajectory predictors such as the Motion Transformer [1].
Abstract: An outstanding challenge for the widespread deployment of robotic systems like autonomous vehicles is ensuring safe interaction with humans with- out sacrificing performance. Existing safety methods often neglect the robot's s ability to learn and adapt at runtime, leading overly conservative behavior. This paper proposes a new closed-loop paradigm for synthesizing safe control policies that explicitly account for the robot's evolving uncertainty and its ability to quickly respond to future scenarios they arise, by jointly considering the physical dy- namics and the robot's ' learning algorithm. We leverage adversarial reinforcement learning for tractable safety analysis under high-dimensional learning dynamics and demonstrate our framework's 's ability to work with both Bayesian belief propa- gation and implicit learning through large pre-trained neural trajectory predictors.
Keywords: Learning-Awwn Aware Safety Analysis, Active Information Gathering, Ad- versarial Reinforcement Learning.
Introduction
With the rise of autonomous vehicles on the road, robots are operating at new scales and around real people. A daily challenge for these systems is maintaining human safety in close-proximity interac- tions without impeding performance. This balance made difficult by inherent interaction uncer- tainry: uncertainty induced by others' intents (e.g., does human driver want to merge, cut behind, or stay in the lane?), responses (e.g., if the robot accelerates, how will the human react?), and seman- tic class (e.g., is this person pedestrian eyclist?). These aspects all unknown a priori, the robot must learn about them runtime. As the robot and the human take action over time, the robot gathers more information about the human's 's internal state and can make more effective decisions.
H. Hu Zhang contributed equally.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Unfortunately, existing safety methods predominantly ignore the robot's ability to learn about the human during runtime interaction, instead making static modeling assumptions [2, 3, 4, 5, 6]. For example, if the robot currently believes that the human driver intends to stay in their lane, then will compute a safety controller assuming that the human always intends to stay in their lane during the safety analysis horizon. Put more formally, these safety methods only account for the evolution of the physical system state while assuming that the information available the robot remains constant throughout the safety analysis horizon. This simplification can lead to overly conservative robot decisions and in extreme cases catastrophic safety failures.
Contributions. We propose a novel Deception Game safety analysis framework that closes the loop between the robot's interactive decisions and the runtime learning process by which it updates its prediction of other agents. Safety is assessed in an augmented state space that jointly describes agents' physical motion and the robot's internal belief updates, explicitly accounting for how interac- tion uncertainty will evolve as a function of agents' upcoming behavior and, crucially, how rapidly the robot can detect and respond to unfety-threateninn tail events (Figure 1). We demonstrate the effectiveness of our approach on 200-dimensional state space problems involving black-box learn- ing dynamics by leveraging a recently introduced approximate solution method [7] for zero-sum dynamic safety games via model-free adversarial reinforcement learning (RL).
Related work. A common approach safe multi-agent interaction compute robust predictions of other agents' behavior. Forward reachability methods use (possibly learned) predictive agent models [8, 9, 10] alongside exact Hamilton-Jacohi (HJ) reachability analysis or approximations such as zonotopes [5, 11, 12, 13, 14, 15] to compute possible future states that are used as constraints by downstream motion planners [16, 17]. While safe, these methods tend to be overly conservative since they do not account for the robot's 's ability to take evasive maneuvers in response to the human.
Another approach is to directly synthesize robust robot control policies. One common tool control barrier functions (CBFs) [18], which encode safety as an algebraic constraint in runtime optimization. However, CBFs lack general constructive methods, often requiring manual per-system design and leading to restrictive underapproximations of the safe set [19]. On the other hand, game-theeretic HJ backward reachability analysis computes the optimal safety value function and the corresponding safety controller for the robot [20, 21J. We ground our work in this literature, and introduce novel formulation of thee-the HJ reachability in belief space.
To date, all provably safe human-robot interaction schemes treat the human model as static during the entire safety analysis horizon. In other words, they assume that the robot will never gain new in- formation about the human's behavior; we refer to this as open-loop safety in the information space. In reality, however, robots can gain information about other agents intent', semantic class, etc., dur- ing interaction. By reasoning about the future information they might receive, robots can afford to be less conservative: for example, in occluded environments, safety can be maintained less conser- vatively by leveraging the future observability of hypothetical yet-undetected objects [22, 23). Our approach accounts for runtime inference, yielding novel dlseed-loppinnfon
safety analysis.
Finally, any approach that considers multiple agents and/or hypotheses will face challenges with the curse of dimensionality, Although HJ reachability has traditionally been intractable for systems with more than five continuous state variables, neural approximations have shown promise in scaling to high-dimensional systems [24, 25. 7J. In this work, we leverage recent advancements in adversarial reinforcement learning approximations [7] to scale our novel safety problem.
2 Preliminaries and Problem Formulation
Let the discrete-time nonlinear dynamics of an "ego" autonomous robot (e) interacting with an
"opponent" agent (o) be Jt-1 f(r,, ut, ut, wt) where x, = (of, x?) E X S R is the joint state vector. The control vectors of the robot and the opponent are uf Ut C R and uf u2 C m. Finally, w, E W Rm= is bounded process noise (external disturbance), and
f : X x u xU x W + X describes the physical system dynamics.

---

Hamilton-Jacobi reachability. Game-theoretic HJ reachability analysis [20, 26] uses robust optimal control theory to compute provable policies to reach and/or avoid re- gions of the state space. Define the target and failure sets as T= fx {(x) 2 0} S R" and F: { g(x) < 0] R", encoded by Lipschitz-continuous margin functions E(.) and g()). analysis captures the opponent's worst-case behavior by formulating an infinite-horizon zero-sum dynamic game. In discrete time, the game's solution can be obtained via the fixed-point Isaacs equation [27] (the game-theoretic counterpart to the Bellman equation):
Background:
((x) = min g(x), max (((z). max min u"Eu Eo WEW
(I)
The value function V-)) encodes the reach-avoid set RA(TT.)) == { VVx) 2 0}. from which the ego agent is guaranteed a policy to safely reach the larget without entering the failure set. Given value function V(-), its zero superlevel set contains the ego's recursively safe states, and the optimal ego and opponent policies are obtained by taking argmax/argmin controls in (1) [20].
Opponent uncertainty. We model opponent agents as taking actions from an uncertain control set affected by their unknown individual characteristics I E O. We refer 0 as the opponent's type, following game theory literature [28]. This type may encode aspects about the agent, including its semantic class (e.g., a pedestrian
cyclist) and its internal state (e.g., intended destination, attention state). We assume that the hypothesis space θ is discrete, and that each hypothesized type 0 is associated with a known (possibly statc-dependent) control set Ug (Iz) of admissible actions for an opponent agent of this type.
Reducing uncertainty via learning. The robot represents its uncertainty over the agent type 0 E θ through a probabilistic belief b(0) E A. Since @ is finite set, b is categorical distribution and the belief space A is (19) - s-smmpee The belief evolves over time as the ego agent collects new observations (e.g., observing the opponent nudging into its lane). Our approach is agnostic to the specific runtime learning algorithm used for propagating the belief state, which includes, among others, Bayesian inference and transformer neural networks [1]. Thus, we refer to the abstracted evolution of the belief state as the learning dynamics of the form
be+1 = [b, yt), y 1( ne,
where y/ y C R"v is the robot's indirect observation of the opponent at time t from state Zt. 2e E V C R is a bounded measurement noise, and h X' x x - y is the observation model. Defining the joint state vector = == (xx b), we can combine the physical dynamics function / and the learning dynamics fL from (2) into a joint dynamical system:
2++1 = (2e, uf, ug, we; vt) = [ fIT,, ut, ug, w(,w))
L (be, Max, ug, vx))
which compounds the physical evolution of the system with the robot's belief update. We note that the learning dynamics need not impose (or be aware of) the type-specific control bounds Ug (are) used for safety analysis purposes.
Running example: In Figure 1, the autonomous vehicle (ego) is uncertain about whether the crossing human (opponent)
pedestrian or a Segway rider. We model the opponent's unknown type as E = {ped, seg), with associated control sets 4 = 0.75, 0.75] x [-2, 2] m/s and U2g = [-0. (-0.75,00 0.75) x [-8, 8, 0) m/s. The joint state space is defined as = = (x _ b(O = ped)) E R with x e g and x" E R².. The robot uses Bayesian inference as the lcarning dynamics fL.
Closing the Safety Loop Around Runtime Learning
Our approach reduces conservativeness by explicitly modeling the dynamic coupling between the opponent's control bound and the robot's internal runtime learning. For example, imagine that the robot begins with strong prior that humans do cross highways: by the time the human does begin to cross the highway, the robot must have observed enough evidence that suggests that
3

---

Deception Game
Deployment
World
plysical dyn.
leanning dyn.
g E of (ai)
safe rohot policy
Gume- Game--Thorren Dhooretic Safety Analyais
? 15.
learning
priceptio
Figure 2: Deployment of the Deception Game policy trained with adversarial deep RI. in closed-loop with pretrained deep neural network learning dynamics model. The robot policy takes as inputs both the physical state æ4 and belief state bt and operates under the inference hypothesis that the opponent behaves according to the inferred control bound (zx) imposed by learning dynamies.
the human's true intent might be to cross. Note that this results in belief-based restriction of the opponent's s behavior that is not causal, but inferential: it is based on the premise that an agent cannot take actions only explained by f without the robot's 's belief already placing nonnegligible probability on this type. We formalize this below as the inference hypothesis.
Assumption 1 (Inference Hypothesis). At each time t, the robot's 's belief by must have assigned no less than ca probability to at least one type 8 consistent with the opponent's 's next action ug. That is, the opponent's 's action uf, from joint state 24 must belong to the inferred control bound given by
((x+)) = Udg(z), Üg(z)) = [Ug(x)) if 2 (f) <
0.
where 6o 2 0 is a designer-specifted threshold capturing the reliability of the runtime inference.
Assumption 1 implics that when (0) falls below Eg the corresponding control set Ud](z) (zt) can be temporarily omitted, reducing the opponent action uncertainty to smaller inferred bound (zx)- The belief threshold Eş encodes the designers' confidence in the reliability of the robot's runtime inference. A suitable value can be determined empirically by evaluating the inference mechanism on a large dataset, or may be provided by an upstream component the autonomy stack as part of a "contract" [29]. As eg approaches zero (no confidence in the robot's inference), we recover the purely robust formulation, where the robot safeguards against all hypotheses regardless of its belief.
Remark 1. Unlike scenario praning approaches [30, 31, 32], within our framework no hypothesis aside until and unless its probability increases. Specifically, previously small b(0) may regain probability and surpass threshold eg, at which point the corresponding control set Ug (xt) is immediately added back to the overall inferred control bound (za). This possibility is factored into our safety analysis via the learning dynamics.
ever permanently ruled out, but only provisionally
Belief-space HJ reachability: formalizing learning-aware safety. The coupling between the robot's learning dynamics and the opponent's action uncertainty induces new safety problem: a belief-space reach-avoid game in which the ego agent attempts to robustly ensure safety against the worst-case behavior of an opponent whose actions must be plausible in light of past behav- ior, yet may be strategically deceptive. To model this game, we extend the game-theoretic reachability formulation (Section to the joint physical beelif state 2. Although the target set T: (= f(x) 0} C X A and failure set J {= g(x) < 0} X' A are still defined in terms of only the physical state, the reach-avoid game entangles physical and belief states through the joint dynamics (3). This interdependence is characterized by new belief-space Isaacs equation:
}
V(2) = min g(2), max {(2), max
min u EMe ar WEW
V (F(ez, u², u. w. (.,,))) v))
u.
(5)
The key differences with respect to the standard Isaacs equation (1) are highlighted in red. In par- ticular, the state space now is defined over (the joint state that additionally includes the robot's belief bi) and the inferred control bound (z) becomes a function of the joint state instead of fixed

---

priori. We also are robust to observation noise as modelled by c. Note that the ego and opponent controls given by solving (5) are optimal in the sense of a zero-sum partially observable stochastic game [33] (or dual control [34, 35]): they not only aim to influence the physical state x, but also the ego's uncertainty about the opponent, encoded by the belief state The ego's control u strategi- cally forces the opponent reveal information about its type 9 whenever reducing the uncertainty is critical to safely reaching the target. The opponent's s control u² is also dual: to sabotage the ego's reach-avoid task completion, it may behave deceptively to maintain strategic ambiguity that it can exploit later, for example, by forcing a collision (safety failure). This is formalized in the result below, which is a direct consequence of HJ reachability theory [21].
Proposition 1. If the opponent satisfies the inferred control bound (4), ie us (t) Vt 0, then the superlevel set {2 V(z) 2 0} is a robust reach- avoid set from which the robot has a guaranteed control strategy-the maximizer of (5)-10 drive the joint system (3) into T while avoiding F.
The above proposition guarantees that, long as the inference hypothesis encoded through (4) holds, the reach-avoid policy is recursively feasible when joint state z is initialized within the reach-avoid set {2 V(z) 2 0}. This naturally accounts for future belief updates, which may discard low- uw-probabiiiiyy hypotheses or add one back as it regains probability (see Remark 1).
Remark 2.
Running example: The ego vehicle's s target set is defined as T = {z p 2 85 m}, where pt denotes the ego vehicle's longitudinal position. Failure set F captures the ego colliding with the opponent or driving out of the road boundary. The complement of the belief-space reach-avoid set RAA(T, computed numerically (with simplified motion models) on grid is visualized in blue in Figure 1.
Synthesizing safe robot policies. Given a pre-computed value function V(z), and any joint physical-belief state 2, the optimal agent policies are obtained by argmin/argmax over (5). The resultant value functions and safety policies can seamlessly integrate with existing safety-aware control frameworks (c.g.. HJ value functions are optimal Control Barrier Functions (CBFs) [18, 36), and the HJ safety policy can be used fallback controller [37, 38, 391).
Scaling-up: embracing model-free deep RL. Since the Isaaes equation (5) is generally intractable to solve exactly, and an explicit Markovian model of the belief dynamics may not always be avail- able, we introduce an adversarial RL approach to approximate time-discount variant of (5) based on the recently developed Iterative Soft Adversarial Actor-Crite for Safety (ISAACS) framework [7]. During training, runtime learning algorithm is queried each time step with the current obser- vation (y or, depending on the algorithm, history of recent observations) produce updated belief state bc+1, without the explicit requirement to be Markovian in (2). The physics simulator enforces the inference hypothesis during training by projecting the opponent's ' action T T'(2) onto the inferred control bound E/0 (z) defined in (4). At runtime, the ego agent receives the current state timate and an observation y of the opponent, queries the same learning algorithm used in training for a belief state b, and executes control action u x (z). Figure 2 presents a schematic of our proposed safe control framework.
4
Case Studies
In this section, we illustrate the applicability and scalability of our approach in three simulated driv- ing examples that differ in problem scale, computation approach, and runtime learning mechanism. Our main hypothesis is that the proposed Deception Game policy yields a planning performance close to an optimistic baseline while achieving a much lower failure rate.
4.1 Low-dim (5D): Exact solution with Bayesian learning dynamics
We first consider a simplified variant of the running example. The ego vehicle's motion is de- scribed by 3D point mass model and the human restricted travel along the P6 90 m line, with possible agent classes 0 E (ped, seg). The robot leams via updating Bayesian belief: (0) x P(ug Iz; (O). The likelihood function is Gaussian P(u Iz; = N N M)
5

---

MerricMethod
Opponent policy
MAP (optimistic)
Contingency
Robust (w/o learning)
Deception Game (ours)
Failure
11 %
3.1 %
0%
Completion time (s)
Modeled (ey 0.2)
4.49
0.6
5.2 1.12
6.27
0.86
4,73 + 0.97
Failure rate
16.4
3.7
3.3 %
Completion time (s)
Unmodeled (6. 0.06)
4.54 + 0.6
5.13 = 1.14
6.26 + 0.85
4.81 + 0.85
Table 1: Case study: Low-dim (5D). Failure rate and mean completion time in 1000 randomized trials with
policies synthesized with numerical dynamic programming [40]. The Deception Game is computed with éo = 0.2. We also tested with an unmodeled human that violated the inference hypothesis by taking actions from types as unlikely safety in all trials while closely matching the performance of an optimistic, but unsafe MAP policy. the unmodeled case, the Deception Game policy incurs a non-zero failure rate, although lower than the MAP and contingency baselines.
ég 0.05. In the modeled human case, the proposed Deception Game policy ensures
where po is the average control of the bound Lg (zze). The joint system (3) is 5-dimensional, which lends itself to numerical grid-based dynamic programming [40] for solving Isaacs equation (5) to high accuracy. We compare our Deception Game policy to three baselines (all synthesized with the same grid resolution as the proposed method); (1) maximum a posteriori (MAP) policy [41], which optimistically uses standard (belief-less) reach-avoid policy based on the MAP estimate O, arg maXage (0) during each planning cycle, (2) contingency policy [30, 31J, which dis- cards currently unlikely hypotheses for which b(0) < Eg while safeguarding against all remaining hypotheses, and (3) a robust policy that conservatively safeguards against all hypotheses 0 E θ at all times. Note that the Robust baseline is equivalent to the optimal control barrier function [18]. We consider two metrics: the failure rate, defined as Naan/Niaa X 100%, where Niiall is the number of trials and Nam is the number of failed trials in which 21 E F for some t, and the completion time averaged across all trials, which measures policy's 's ability to make progress. A trial is considered complete if pt 100 m. For each policy, we simulated under the same random seed Nirial 1000 randomized scenarios, each with different initial state, prior belief, and the (hidden) opponent type. Table displays results obtained from all 1000 trials. The Deception Game policy maintained a zero failure rate when the inference hypothesis held, which empirically validated Proposition 1, and achieved an average completion time close to that of the optimistic but unsafe MAP baseline.
4.2 High-dim (18D): Approximate solution with Bayesian learning dynamics
Now, we turn to the full setup of the running example with the 7D physical system (the ego and opponent are modeled as a 5D kinematic bicycle model and 2D particle with velocity control) and two agent class hypotheses f E {ped, seg), and additionally model the human's intent with hidden goal position ggoal E Qoal goal == {
Ngee - respectively, where E I22 denote each of Nggoll = 9 goal hypotheses scattered along the road boundary. Since the resulting 18-dimensional state space is intractable for exact dynamic programming, we apply the deep RL framework in Sec. 3. We simu- lated 1000 randomized scenarios with different initial states, prior belief, opponent types, and latent goals for each policy under the same random seed. The simulation was repeated for four different (1). Belief-space ad versarial (modeled): The human uses adversarial policy (2) trained with the belief-space deep RL that approximates (5) and was projected to the inferred con- trol bound (z) at simulation time, (2). Belief-space adversarial (unmodeled): Same as (1) but without the projection step, thereby allowing violations of the inference hypothesis, (3). Non-belief adversarial: The opponent uses adversarial policy (z) trained with the non-belicf deep RL that approximates (1) while the ego safeguards against all hypotheses, and (4). Noisily-rational: The human uses a perturbed LQR policy to reach a goal position (randomized across trials). The failure rate and completion time results are shown in Table and Figure 3, respectively. The proposed Deception Game policy outperforms both the MAP and contingency baselines in all scenarios in terms of the failure rate. Remarkably, the completion time statistics of the Deception Game policy closely resemble that of the optimistic MAP policy in all four scenarios. The robust policy, albeit safe, was, however, overly conservative and unable to complete the task within 10 seconds in most trials. These results empirically support our main hypothesis.
human policies:
In Figure 4, examine one representative simulation trial of the running example initialized with uniform prior belief. As the ego vehicle approached the human during & E [0, 4.4) s, the human

---

Metric/Method
Opponent policy
(optimistic)
Contingency
(wio learning)
Deception Game (ours)
Belief Adv. (Modsted)
40.7 % (40.7 %)
31.2 % (39.5 %)
22.3 % (84.6
5.7 % (5.9 %)
Failure
Adv. (Unmodeled)
(62.5 %))
51.3 (53.6
(84.4 96)
(36.4 %)
(Incomplettion rale)
Non belief Adv.
5.2.% (5.2 %6
1.5 (2.3%)
7.4 7.4% (81.2%)
1 % (1.5 %)
Notsily-rational
%))
6.4.5
5.2%% (54.5
0.5 %
Table 2: Case study: Running example (18D). Failure rate and task incompletion rate (counting both failures
time-outs) in 1000 randomized trials with deep RL-trained policies. The Deception Game policy achieves consistently lower failure rate than the MAP and contingency baselines and leads completion across the board.
Opponent
Belief Adv. (Modeled)
Belief Adv, (Unmodeled)
Noe-bellef Adv.
Notsily-rational
0.9
0.9
0.9
0.9
Connrganc
Pregeuncy
10
6
10
2
10
10
Completion Time (s)
Completion Time
Completion
Completion Time
Figure 3: Case study: Running example (18D). Completion time over 1000 randomized trials. Approximations
via deep adversarial RI.. Deception Game policy performance closely matches the optimistic MAP baseline.
The robust baseline overly conservative and often times out ( 10
3
6.5
Case study: Running example (18D). Simulation snapshots and b(type) over time. Both agents use the optimal Deception Game policies synthesized with adversarial RL. Red circles denote human's 's goal hypotheses and color intensity represents their belief probability. A line between the human and a goal gi means that b(g.)) 2 The pedestrian and Segway icons indicate the ground-truth human type. The orange question mark implies that neither hypothesis has a probability below the threshold. Despite the human's 's adversarial deceptive motion, ego vehiele managed to safely complete the task.
Figure 4:
deceptively alternated between applying pedestrian and Segway actions in order to hide its type. The ego vehicle then deliberately turned left, causing the adversarial human to move upwards and be exposed as pedestrian. Due to the pedestrian's 's limited ability to move downwards compared to the Segway type, the ego vehicle could safely turn right and pass in front of the human.
4.3 High-dim (200D): Implicit learning dynamics from neural predictor
Finally, demonstrate that method can scale to data-driven, non-Markovian learning process. Consider a scenario from the Waymo Open Motion Dataset (WOMD) [42], where an autonomous car aims to cross the intersection without colliding with the oncoming human-driven car or violating the road restrictions (left column, Fig. 5). The robot's learning process is governed by pretrained Motion Transformer (MTR) [1] trajectory predictor, which at each timestep produces a Gaussian mixture model of 64 state trajectories given the scene's history. Using a proportional controller to invert the dynamics, the robot infers the human's action (acceleration, steering angle) from these state predictions. We use the inferred actions as the mean of each mixture component (mode 0). Following (4), add small (0) around each predicted action when constructing the predictive control bound Lgg Since the robot will repeatedly invoke the MTR to forecast the human's 's behav- ior, and these predictions will inevitably change over time during the closed-loop interaction, this defines the robot's implicit learning dynamics f-().. Ultimately, this problem formulation has 200 200-dimensional joint state space: an 8D physical state and belief vector comprised of 64 mixture components, each with 2D mean and a scalar weight, totaling 64 x 3 = 192 belief dimensions.
Both the robot and the human's control policies were trained using the deep RL approach in sec- tion taking as inputs the physical states, predicted nominal actions, and the corresponding proba- bilities. We implement two baselines: (1) a robust policy that safeguards against the entire human action space 4t, and (2) an iterative linear quadratic regulator (ILQR) policy that re- re-plans using
7

---

Deception Game (Ours)
1.000
1-03
[-03s
Robust
ILOR
Figure Case study: neural predictor (200D). The adversary suddenly turned left intersection. De- ception Game (top) and Robust policy (middle) safely reached the target T. while ILQR (bottom) leads to collision. Compared to the robust policy, our proposed approach is also smoother.
MTR predictions at each timestep. By closing the safety and learning loop, the Deception Game pol- icy produces proactive ego behaviors, preparing the robot for potential shifts helief. For example, in Figure 5, at 0.3 s, the robot predicted the human's ' left turn and maintained lower velocity a larger safety margin. This allowed the robot to safely pass the human later / = 1.5 when slowed down, attempting to block the robot. In contrast, the robust baseline produced more radical evasive actions protect against the adversary regardless of belief. In Appendix C, we include additional preliminary results to look closer our Deception Game policy against two baselines under different initial states. Moreover, also demonstrate its generalizability to non-adversarial human behaviors by replaying ground-truth trajectories from WOMD, as well as its scalability to multiple-agent scenarios under the commonly used pairwise decomposition assumption [43, 44].
5 Limitations and Future Work
While our empirical results indicate that our Deception Game improves robot efficiency and main- tains a low failure rate compared to baselines, we cannot provide theoretical safety guarantees when the solution is approximated via RL. addition, our approach relies on a designer-specinfe old Ep for the control bound, which requires careful construction. Moreover, our MTR case study only evaluates one intersection scenario with hand-crafted target and failure sets. One direction for future work is to leverage the learned latent information (about the scene or interactions) from the scene-centric trajectory prediction models [45, 46, 47, 48] to improve generalization. We are also interested in incorporating our approach with full autonomy stack, accounting for raw sensor data (e.g. camera images) and imperfect state observations (e.g. occlusions [22]). Finally, we see an open opportunity to extend our methodology for other pressing issues beyond robot safety, e.g., detecting and preventing manipulative behaviors in generative AI (such as large language models).
thresh-
6 Conclusion
In this paper, we present the Deception Game, novel safety framework that closes the loop between the robot's 's planiniionn-lannngg. control pipeline and its runtime learning process by jointly reasoning over agents' physical states and the robot's belief states. Our approach builds upon a game theoretic formulation and scale up high-dimensional problems and handle implicit learning dynamics via model-free adversarial reinforcement learning. Our experiment results indicate that the proposed method works with different types of interaction uncertainty, can be well combined with off-the- shelf prediction models, and plan efficient and safe trajectories.

---

Acknowledgments
The authors thank Hongkun Liu from the University of Science of Technology Beijing for providing insightful suggestions on trajectory prediction and behavior forecasting.
References
[1] S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 2022.
[2] R. Tian, L. Sun, A. Bajesy, M. Tomizuka, and A. D. Dragan. Safety assurances for human- robot interaction via confidence-aware game-theoretic human models. CoRR, abs/2109.14700, 2021. URL https::/aaraee
hwtrg/abs/210099 14700.
[3] Borquez, K. Nakamura, and Bansal. Parameter-conditione reachable sets for updating safety assurances online. arXiv preprint arXiv:2209. 14976, 2022.
[4] Shalev-Shwartz, S. Shammah, and A. Shashua. On formal model safe and scalable self-driving cars. arXiv preprint arXiv:1708.063774 2017. 2017.
[5] M. Althoff and M. Dolan. Online verification of automated road vehicles using reachability analysis. IEEE Transactions on Robotics, 30(4):903-918, 2014.
[6] Y. Chen, H. Peng, and J. Grizzle. Obstacle avoidance for low-speed autonomous vehicles with barrier function. IEEE Transactions on Control Systems Technology, 26(1):194- 19: 2017.
[7] K.-C. Hsu, D. P. Nguyen, and J. F. Fisac. Isaacs: Iterative soft adversarial actor-critic for safety. In Proceedings of the Sth Conference on Learning for Dynamics and Control, 2023.
[8] M. Kwon, E. Biyik, A. Talati, K. Bhasin, D. P. Losey, and D. Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In Proceedings of the 2020 ACM/IEEE International Conference on Saaman-Robo Interaction, HRI '20, page 43-52, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367462. doi: 10. 10.1145/3319502.337443
URL https //tp::/r/ooo......///.o 11t.///..111///111022.1.o
[9] H. Nishimura, J. Mercat, B. Wulfe, R. T. McAllister, and A. Gaidon. Rap: Risk-aware predic- tion for robust planning. In Conference on Robot Learning, pages 381-392. PMIR, 2023.
[10] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasibee trajectory forecasting with heterogeneous data. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[11] K. Nakamura and S. Bansal. Online update of safety assurances using confidence- based pre- dictions. arXiv preprint arXiv:2210.01199, 2022.
[12] A. Bajesy, S. Bansal, E. Ratner, C. J. Tomlin, and A. D. Dragan. A robust control framework for human motion prediction. IEEE Robotics and Automation Letters, 6(1):24-31, 2020.
[13] A. Muthali, H. Shen, S. Deglurkar, M. H. Lim, R. Roelofs, A. Faust, and C. Tomlin. Multi- agent reachability calibration with conformal prediction. arXiv preprint arXiv:2304.00432, 2023.
[14] Liu, Shao, L. Lymburner, H. Qin, V. Kaushik, Trang, R. Wang, V. Ivanovic, H. E. Tseng, and R. Vasudevan. Refine: Reachability- based trajectory design using robust feedback linearization and zonotopes. arXiv preprini 1. 11997, 2022.
[15] S. Kousik, S. Vaskov, F. Bu, M. Johnson-Roberson, and R. Vasudevan. Bridging the gap between safety and real-time performance in receding-horizon trajectory design for mobile robots. The International Journal of Robotics Research, 19(12):1419-1469 2020.

---

[16] L. Lindemann, M. Cleaveland, G. Shim, and G. J. Pappas. Safe planning in dynamic environ- ments using conformal prediction. IEEE Robotics and Automation Letters, 166:5311-5 2023. doi: 10. 101.1199..222.32222222.
[17] T. Brüdigam, M. Olbrich, D. Wollherr, and M. Leibold. Stochastic model predictive control with safety guarantee for automated driving. IEEE Transactions on Intelligent Vehicles, 2021.
[18] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pages 1420-3431. IEEE, 2019.
[19] L. Wang, D. Han, and M. Egerstedt. Permissive barrier certificates for safe stabilization using sum-of-squares. In 2018 Annual American Control Conference (ACC), pages 585-590. IEEE, 2018.
[20] I. M. Mitchell, A. M. Bayen, and C. J. Tomlin. A time-dependent hamilton-jacohi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control, 50 (7):947-957, 2005.
[21] J. F. Fisac, M. Chen, J. Tomlin, and S. S. Sastry. Reach-avoid problems with time-varying dynamics, targets and constraints. In Proceedings of the 18th international conference on hybrid systems: computation and control, pages 11 11-20, 2015.
[22] Z. Zhang and J. F. Fisac. Safe Occlusion-Aware Autonomous Driving via Game-Theoreit Active Perception. In Robotics: Science and Systems XVII. Robotics: Science and Systems Foundation. ISBN 978-0-99233747-7-8 Moi:100.1507/RRS5 http://SS...222.XX...0. URL http:// www www. www.roboticcprooedddngg. org/rss17/p066. pdf.
[23] C. Packer, N. Rhinehart, R. T. McAllister, M. A. Wright, X. Wang, J. He, S. Levine, and J. E. Gonzalez. Is anyone there? learning a planner contingent on perceptual uncertainty. In Conference on Robot Learning, pages 1607-1617. PMLR, 2023.
[24] S. Bansal and C. J. Tomlin. Deepreach: A deep learning approach to figh-iieessiona reach- ability. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1817-1824. IEEE, 2021.
[25] K.-C. Hsu, V. Rubies-Royye Royo. C. J. Tomlin, and J. F. Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. In Proceedings of Robotics: Science and Systems, Held Virtually, July 2021. doi: 10. 15011567//SS.200 220....L0001 XVIIL077.
[26] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-jacon reachability: A brief overview and recent advances. In 2017 IEEE 56th Anmual Conference on Decision and Control (CDC), pages 2242-2253 IEEE, 2017.
[27] R. Isaacs. Differential games i: Introduction. Technical report, RAND CORP SANTA MON- ICA CA SANTA MONICA, 1954.
[28) A. Kajii and S. Morris. The robustness of equilibria to incomplete information. Econometrica: Journal of the Econometric Society, pages 1283-1309, 1997.
[29] I. Incer, A. Badithela, J. Graehener, P. Mallozzi, A. Pandey, S.-J. Yu, A. Benveniste, B. Cail- laud, R. M. Murray, A. Sangiovanni- vinovannnn-iineetllli and S. A. Seshia. Pacti: Scaling assume- guarantee reasoning for system analysis and design, 2023.
[30] A. Bajesy, A. Siththaranjan, C. J. Tomlin, and A. D. Dragan. Analyzing human models that adapt online. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 2754-2760. IEEE, 2021.
[31] J. Hardy and M. Campbell. Contingency planning over probabilistic obstacle predictions for autonomous road vehicles. IEEE Transactions Roboties, 29(4)9113992,, 2013,
10

---

[32] L. Peters, A. Bajesy, C. Chiu, D. Fridovich-Keil, F. Laine, L. Ferranti, and J. Alonso-Mora, Contingency games for multi-agent interaction. arXiv preprint (rXiv:2334405583, 2023.
[33] E. A. Hansen, D. S. Bernstein, and S. Zilberstein. Dynamic programming for partially observ- able stochastic games. In AAAI, volume 4, pages 709-715, 2004.
[34] A. A. Feldbaum. Dual control theory. i. Avtomatika i 'Telemehhanika, pages 1240-1249, 1960.
http::/mii
[35] A. Mesbah. Stochastic model predictive control with active uncertainty learning: A survey dual control. Annual Reviews in Control, 45:107-117, 2018. 160:100.006)) 001.
[36] Choi, D. Lee, Sreenath, C. Tomlin, and L. Herbert. Robust control barrier-value functions for safety-critical control. In 202/ 60th IEEE Conference Decision and Control (CDC), pages 6814-6821, IEEE, 2021.
[37] K. Wabersich and M. N. Zeilinger. Linear model predictive safety certification for learning- based control. In 2018 IEEE Conference on Decision and Control (CDC), pages 7130-7135.
IEEE, 2018.
[38] O. Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shield- ing. In 202/ American control conference (ACC), pages 3488-3494 IEEE, 2021.
[39] K.-C. Hsu, H. Hu, and J. F. Fisac. Safety filter: A unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems, 2023, to appear.
[40] M. Bui, G. Giovanis, M. Chen, and A. Shriraman. Optimizeddp: An efficient, user-friendly library for optimal control and dynamic programming, arXiv preprint arXiv:2204.05520, 2022.
[41] L. Peters, D. Fridovich- Keil, C. J. Tomlin, and Z. N. Sunberg. Inference- based strategy align- ment for general-sum differential games. In 19th International Conference on Autonomous Agents and Multi Agent Systems, pages 1037-1045, 2020. URL https: /tstps://dl... http://lll..cc acm. org/doi/ 3398882.
10. 5555/3398761. 113398886...9888...
[42] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. R. Qi. Y. Zhou, Z. Yang. A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9710-9719, October 2021.
[43] S. Shalev-Shwartz, S. Shammah, and A. Shashua. On formal model of safe and scalable self-driving cars. arXiv preprint
2017.
[44] Y. Chen, A. Singletary, and A. D. Ames. Guaranteed obstacle avoidance for multi-robot op- erations with limited actuation: A control barrier function approach. IEEE Control Systems Letters, 5(1):177-332. 2021. 1. 1t0.00..1101LSSSY220000..000
[45] J. Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, et Scene transformer: A unified architecture for predicting multiple agent trajectories. arXiv preprint arXiv:2 2021.
[46] Y. Chen, B. Ivanovic, and M. Pavone. Scept: Scene-consistent, policy-based trajectory pre- dictions for planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17102-17112, 2022.
[47] Shi, L. Jiang, D. Dai, and B. Schiele. Mir++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. arXiv preprint Sttv:220061777000 2023.

---

[48] Z. Huang. H. Liu, and C. Lv. Gameformer: Game-theoretic modeling and learning of transformer-hasee interactive prediction and planning for autonomous driving. arXiv preprint arXiv:2303.05760, 2023.
[49] H. Hu, D. Iscle, S. Bac, and J. F. Fisac. Active uncertainty reduction for safe and efficient interaction planning: A shielding-awar dual control approach. The International Journal of Robotics Research, 2023, to appear.
[50] Traautman and A. Krause. Unfreezing the robot:
Navigation in dense, interacting crowds. In 2010 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 797-803. IEEE, 2010.
[51] Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[52] D. González, J. Pérez, V. Milanés, and Nashashibi. A review motion planning techniques for automated vehicles, IEEE Transactions on intelligent transportation systems, 17(4):1133 1155, 2015.
12

---

A Comparison of Learning-Aware Safety Methods
This section provides a table comparing our proposed Deception Game framework with recent learning-aware safe planning methods. To the best of our knowledge, our method is the first safety analysis framework that jointly reasons the agent's physical states and robot's belief states in closed-loop fashion and can scale up to high-dimensional systems with implicit learning dynamics.
Table 3: Comparison of learning- aware safe planning methods.
Feature/Method
Bujcsy al. [30]
Zhang
Packer al. [23]
al. [32]
al. [2]
al. [49]
al. [22]
Recursive safety guarantees
N
Y
Y
N
Active information gathering
N
Y
Y
Y
Uncertainty dependent safery analysis
N/A
N
N/A
Y
NIA
Belief refinement based on observations >
Y
Y
Y
N/A
NIA
Scaling high
N
Y
Allowing implicit learning dynamics
N
N
N
Y
N
Y
Y
Allowing continuous hypotheses space
Y
N
N/A
Fully online policy computation
Y
N
Y
Y
N
B When the Prior is Biased and the Inference Hypothesis is Violated
We provide another example of the high-dim (18D) scenario, where the robot's prior belief initially biased towards believing that the human will not cross the road, and that the human allowed to take actions that violate the inference hypothesis (Assumption 1). To this end, give the robot biased prior belief on the human's s intent, in which the goal encoding the human not crossing the road (gs) receives most of the probability mass (0.92), and the prior probabilities of the remaining goals are set to 0.01, which is below threshold Ep = 0.05. The simulation snapshots are plotted in Figure 6. We observe that as the robot receives new observations actions, runtime learning algorithm gradually adjusts the belief to better explain the observations.
the human's states and
As result, goal 93 on the other side of the road starts gaining probability, and its associated control bound is subsequently added the overall inferred control bound. At 6 s, all hypothesized goals in the upper part of the road temporarily thresholded out the human moves down. Then, the human abruptly begins to move upwards toward g7. violating the inference hypothesis (indicated by the yellow exclamation mark). In light of this highly unexpected behavior, the robot's runtime inference quickly catches on, which reactivates hypotheses 97 and gs in the robot's belief. The Deception Game policy then generates cautious robot motion based on the updated belief. In spite human behavior,
of the unfavorable conditions of biased prior and abrupt, assumption-breakie the robot avoids colliding with the human and safely reaches its target. This case study indicates practical robustness inaccurate prior beliefs well
that the Deception Game policy offers abrupt changes in agent behavior inconsistent with the inference hypothesis.
We also simulate trajectories under the same initial condition and human policy when the ego vehicle uses the three baseline policies, Figure displays the trajectories with the MAP policy. Due to overly optimistic assumption that only the most likely hypothesis considered during decision- making, the ego vehicle does take precautions until the last instant, when the runtime learning indicates that the human is moving downwards, by which time is already too late avoid colliding with the adversarial human. Simulation snapshots when the ego uses the contingency baseline policy are shown in Figure 8, which also constitutes failure case. The main reason for the loss of safety, in this case, is due to the biased prior- the algorithm is initialized with 9s being the only active goal hypothesis, with all other goal hypotheses ruled out, this case permanently. Finally, we investigate the trajectories with the ego using the robust baseline Figure 9. As the human strategically moves the center of the road maximize the chance of collision, the planning algorithm fails to find any safe passing "corridor" before times out, but controls the ego vehicle to stop. resulting in "freezing robot" situation [50].
13

---

Deception Game
0.92
t=
93
0.77
t E [0, 3]
0.14
92
93
98
t E [3, 6)
97
0.08
0.84
0.1
t E [6, 8.2] S
0.17
0.7
Figure 6: Simulation snapshots for Appendix B. Both the robot and the human use the Deception Game policy synthesized with adversarial RL. The human is allowed to take action that violates the inference hypothesis (Assumption 1). Ycllow exclamation marks indicate when the inference hypothesis violated as the human takes an action corresponding to a discarded hypothesis. Red circles denote the human's goal hypotheses and their color intensity represents their belief probability. A line drawn between the human and goal hypothesis g means that b(g:) Segway icons indicate the ground-truth human type. The orange question mark implies that neither hypothesis probability below threshold. ego vehicle managed to safely complete task even if the opponent human was not conformant to the inference hypothesis and the prior belief was biased towards the human not crossing the road.
= 0.05. The numbers denote the probabilities of hypothesized goals. The pedestrian and
C Deception Game with Motion Transformer
C.1 Problem Setup
Consider the traffic scenario when the robot aims to traverse the intersection without violating the road bound or causing a collision with the adversary. We model both vehicles using the kinematic bicycle dynamics with longitudinal acceleration and steering angle controls. In addition, limit their state and action space by bounding velocity [0, 10] m/s, acceleration [-5, 5] m/s² and
steering angle å [-0.5, 0.5] rad.
To infer the adversary's 's future actions, the robot uses a state-of-the-aar trajectory prediction model, Motion Transformer (MTR) [1]. which outputs Gaussian Mixture Model of trajectories over the next seconds from the 1.1 seconds of scene history. To construct the inferred control bound, we utilize a proportional controller track the mean trajectory of each mixture component (mode as the adversary's s nominal policy (Ix 8). In addition, set (0) by assuming it can deviate from the nominal policy up to +2 m/s2 in acceleration and +0.1 rad in steering angle. Since MTR outputs
14

---

Maximum a Posteriori Baseline
0.92
t -=
93
0.78
E [0, 3) 8
0.14
92
93
t E [3, 6.2]
0.91
0.08
Figure 7: Simulation snapshots for Appendix with the ego vehicle applying the MAP baseline policy. The human uses the Deception Game policy synthesized with adversarial RL, and allowed take an action that violates the inference hypothesis (Assumption 1). The ego vehicle collided with the human at t 6.2 s.
Contingency Baseline
0.92
t t=0
93
0.77
E
[0, 3] s
0.14
92
93
t E (3, 6.7
0.93
0.06
Figure 8: Simulation snapshots for Appendix B with the ego vehicle applying the contingency baseline policy. The human the Deception Game policy synthesized with adversarial RL, and allowed take action that violates the inference hypothesis (Assumption The ego vehicle collided with the human at 6.7
64 modes using a prior motion query, aggregate overlapping trajectories using non-maximum suppression and mask out modes with (0) < 0.05.
15

---

Robust Baseline
t t=0
93
t E [0, 3] 8
97
t E [3, 6] s
92
t E (6, 10] s
92
Figure Simulation snapshots for Appendix B with the ego vehicle applying the robust baseline policy. The human uses Deception Game policy synthesized adversarial RL, allowed to an action that violates the inference hypothesis (Assumption 1). The ego vehicle failed to pass the road section in 10 s.
C.2 Network Architecture and Training Stratergy
The MTR model is first trained with the entire Waymo Open Motion Dataset [42] for 30 epochs and achieves claimed results in their paper. Then, pre-trained MTR model in the loop generate predictions and inferred control bounds for the ad- versaries. We represent the state with the absolute pose of the robot w.r.t the map, the adversary's relative pose w.r.t the robot, the nominal actions for each valid prediction mode, and their probabili- ties. The Deception Game is trained using the Iterative Soft Adversarial Soft Actor-Crith (ISAACS) framework, where four neural networks are trained asynchronously.
up simulation environment with the
The ego actor is the policy of the robot. first encodes the states and each prediction mode independently by multi-layer perceptions (MLP). Then the state feature is concatenated to each prediction feature and passed through another MLP. We conduct max -oxxpopolig across all prediction modes generate the aggregated feature, which is processed by the final MLP and becomes the mean and standard deviation of the robot's action. Finally, sample the action using the squashed Gaussian distribution described in Soft Actor-Critic [51].
The adversarial actor the policy of the adversary. first encodes the states, the action of the robot, and each prediction mode independently by MLPs. Then we concatenate the state, action, and each prediction features, which later aggregated by another MLP. The final MLP processes the aggregated feature and outputs each prediction mode's 's mean,
16

---

standard deviation, and probability.. We sample the adversary's 's action using a mixture of the squashed Gaussian distribution to enforce the inferred control bound.
The static critic a simple MLP return the Q value of the robot only considering the road boundary and target set.
The interaction critic returns the residual Q value of the interaction between the robot and the adversary. It first encodes the states, the action of both actors, and each prediction mode independently by MLPs. Then we concatenate the state, action, and each prediction features, and generate the aggregated feature of each mode through an MLP. The final MLP processes this feature and outputs Q values for each prediction mode.
Unlike the standard ISAACS procedure, the ego actor and static critic are first trained by ignor- ing the collision with the adversary. Then we train ego actor, static critic, and adversarial actor jointly through domain randomization by randomly sampling the initial states of both actors and the adversary's 's action from its inferred control bound. In this process, We take the largest Q value from adversarial actor among all modes, activate it through the SoftPlus function, and add the output from the static critic the final value for the robot. In this way, the resulting Q value is strictly equal or larger than that from the static critic the robot will lose the game regardless of the adversary's 's state when violating the road constraint. Through our experiment, we found that pre-training the ego actor and static critic is necessary to stabilize learning process.
C.3 Additional Results
close-up in Figure 10 confirms that the adversary using the Deception Game policy indeed took adversarial actions actively shift the trajectory predictions to intercept the robot. This enables adversary to pursue the robot within the predictive action bound and indicates that our framework capture the implicit learning dynamics deep neural network represents. Moreover, even with one of the ttaeeoffthhe-art trajectory predictors, small deviations from the nominal action can dra- matically shift the robot's 's belief in the adversary over short period. Therefore, classical methods (including the baseline ILQR policy) that rely on "accurate" trajectory prediction in an open-loop fashion [52] and frequently replan are insufficient for safe interactions. The intelligent robot needs to close the loop of learning by accounting for future ability to learn and adapt.
Due to road boundaries and maneuverability limitations, the robust policy could not find feasible actions, shown in Figures ll and 12. The ILQR baseline failed all tests as it cannot account for the adversary's ' ability to change the robot's prediction in the future. Since our Deception Game policy is trained for intersections with randomized initial states of both vehicles, is generalized to safely navigate different scenarios.
t=000
t=0de
t=08 d
(=
Deceptoon Gumee (Ours)
Vetorry Along [m]]]
Rolox
Predicio
Figure 10: Case study: neural predictor (200D). Our framework can model an adversary whose policy adver- sarially influences the robot's implicit learning dynamics.
17

---

Deception Game (Our))
0-00
t-038
(-a5c
1-00
1-03+
1=23c
Robust
(-0.0
1-033
1-1.84
1-20A
1LQR
Valocry Along [msss
Advensary
Prediction
Figure 11: The adversary made an unprotected left turn when the oncoming robot entered the intersection. Robots using Deception Game (top) safely reached the target T by taking proactive action even when the adversary predicted yield. The Robust Policy (middle) overreacted the adversary's action, violated the road boundary constraints, and entered failure set IIQQ policy (bottom) was overly optimistic about the prediction and caused collision
Deception (Ours)
1= 0.0
1.0
1.8
2m38
0
1=100
t=18
1=26
Rebust
1=000
1.0
t=188
1=214
11.0R
Velocity Aloug Puth [mu)]
Advenary
Prediction
Figure 12: The robot interacted with the merging adversary. Robots using Deception Game (top) safely reached
the target by yielding to the adversary. The Robust Policy (middle) overreacted to the adversary's 's action, violated the road boundary constraints, and entered failure set F. The IIQQR policy (bottom) was overly optimistic about the prediction and caused collision
18

---

In Figure 13, we compared the robot's behaviors under different policies when interacting with non-adversarial vehicle replaying the trajectory from Waymo Open Motion Dataset. We observed all three policies successfully traversed through the intersection. Both the proposed Deception Game and the robust policy showed similar evasive behaviors, where the robot using the robust policy accelerated much faster. The ILQR demonstrated closer maneuvers to the robot's ground truth trajectory from the dataset.
Deception Game (Ours)
0.0
T- 3.
4.0
1- 4.0
1-184
I- 30
1-37
1.0.
1=18.
I=36.
1=48.
ILOR
0.00
1-18A
L-85
1-44s
WOMID GF
Valoeity Along Path [uu]]
Robat
GT
Prodiction
Figure 13: Robot's behaviors under different policies when interacting with non advervvrraaill vehicle replaying the trajectory from Waymo Open Motion Dataset.
19

---

Under commonly used pairwise decomposition assumption [43, 44], Figure 14-17 demonstrate our proposed Deception Game's scalability to multiple agents scenarios. In these examples, We treated the closest vehicle to the ego as the adversary. The ego and the adversarial agent used policies from Deception Game, while third nonplaying vehicle applied the MTR nominal action.
t=00
0
1=
15
2-200
1-20
1-4.0
[mbs]
10
Adversar
Robot
Nominal
Prediction
Figure 14: The robot interacted with the merging adversary while nonplaying vehicle (NPC) attempted to turn left from the opposite lane. The nouplaying vehicle applies the most likely control from MTR at each timestep.
<-00
(-15#
2-200
Adversary
Velocity Along Puth [mis]
The robot interacted with consecutive left-turning adversaries. Under the wise assumption, we treated the closest vehicle to robot as adversary at each timestep. The further vehicle a nonplaying character at each timestep and applics the most likely control from MTR.
Figure 15:
20

---

1=33
1=36
1=38
1=10x
Velncity Along Puth [mb]]
Adversary
Robot
Nominal
Prediction
Figure 16: Under the pair-wise assumption, the robot interacted with the left-turiing adversary by evading behind it. Then, robot interacted with another merging adversary reached the target without collision.
2=
1=25.
1=30#
t=
Velocity Along Path [m/)]]
Adverary
Robat
Nominal
Prodiction
Figure 17: Under the pair-wise assumption, the robot interacted with the left-turning adversary by evading in front of it. Then, robot interacted another merging adversary reached the target without collision.
21


Reviews:
[{"review_id": "qe1xENuY78", "reviewer": "Reviewer_BY31", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach to promote safe interaction between the robot and other agents/humans in the environment.\nThe idea is following a safe control policy that considers the scenario's uncertainty in the belief space. \nTo this end, the paper proposes to use belief games that optimize a safe policy given the current belief of how the scenario evolves. ", "summary_of_recommendation": "The paper addresses the important problem of safe planning in uncertain interactive scenarios. \nThe underlying idea of using belief spaces is interesting.\nYet, the experimental validation is too simple (few hand-picked scenarios, one other agent, simple models) and does not show how the proposed approach performs in general scenarios.\nThe comparison with baselines also requires more attention. \nThe paper does not discuss the current safety literature and how the approach compares to those works.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* What is the computational complexity of the approach?\n* How can one ensure persistent feasibility? \n* How robust is the approach to hypotheses that suddenly became likely and are not safe with respect to the current dynamics of the robot?\n* How does the approach compare to other safety approaches in the literature?\n* How does the approach perform in randomized scenarios?\n* How does the approach perform with multiple other agents?\n* Why does the vehicle steer into the adjacent lane even though it intends to cross the intersection?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The approach targets the important problem of promoting safe behaviors of robots in environments with uncertain motions of other agents/humans. \nThe paper is generally well written, and the visualizations help readers to follow the approach.\nThe proposed approach is an interesting way to increase the safety by considering the current belief of how the environment evolves. \nIt is great to see that the authors formulate a hypothesis for their experiments and motivate why the design of the experiments helps to investigate the hypothesis. \nThe metrics are also explained well to readers. \nThe experiments are carried out in simulation in three different scenarios. \nThe attached video helps to understand the experiments and compare the approach with the chosen baselines. \n\n\nThe proposed approach has an interesting and relevant focus, but is not yet convincing. \nThe experimental validation needs to be vastly improved and the comparison to the existing safety literature needs more attention.\nThe paper misses to relate the approach to many existing papers on safe planning.\nIn fact, this paper focuses primarily on citing papers originating from Berkeley and with a similar touch to the proposed approach.\nThere are no mentions of other relevant safety works, e.g., done at Michigan and TUM (reachability), ETH and Caltech (safe control), or Stanford (risk-based approaches). \nThe paper mentions in L23 that existing safety approaches make static modeling assumptions. \nHowever, many recent papers investigate this interactive component and dynamically ensure safety. \nFor instance, Brüdigam et al. (T-IV, 2023), Kwon et al. (HRI, 2020), Nishimura et al. (CoRL, 2022) or Lindemann et al. (RA-L, 2023), or reactive approaches such as REFINE from Liu et al. (arXiv, 2022).\nThe paper needs to better discuss the existing safety landscape.\n\nOne of the main motivations for the proposed approach is the focus on the current belief of how the situation will evolve.\nThe most likely predictions above the user-defined threshold are considered for planning.\nThe belief is modeled through a discrete set of hypotheses.\nFrom the paper, it is not yet clear how such discrete sets can look like in typical robot applications.\nThe type of an agent or its goal are interesting, but it is not clear how one would choose and model this set.\nFor instance, how does one know which goals are important?\nThe use of the prediction in the AV examples also only cover a set of possible predictions. \nWhat if those are incorrect at some point?\nThe approach adapts by recomputing the belief and updating the hypothesis.\nBut since less likely corner events are first discarded and not considered in the planner, the plan might navigate the robot into a dead-end in which it cannot react when such less likely behavior suddenly becomes likely.\nThis situation is not discussed in the paper.\nFor the central Proposition 1, there is no proof given in the paper or supplementary. \nThe computational complexity seems to scale with the number of hypotheses and the considered dynamical model.\nWhat is the computational complexity of the approach?\n\nThe experiments somehow indicate that the approach is computationally heavy. \nThe first experiment uses simplified models and small set of hypotheses. \nThe scenario is chosen so that the agent crosses the road and interferes with the autonomous car. \nSuch a scenario often causes longer completion times for baselines that consider all hypotheses to ensure safety.\nIt would be great if the experiments were conducted in a Monte Carlo fashion on randomized scenarios. \nWith this simple scenario, it is hard to validate that the proposed approach is generally better.\nIn the high-dimensional experiment, it is weird that the autonomous vehicle first steers to the left into the way of the other agent even though the autonomous vehicle intends to cross the intersection. \nWhy is this the case?\nMoreover, the paper claims that the trajectories of the robust approach are less smooth.\nAlthough, the robust baseline generates an evasive maneuver, the proposed approach produces trajectories that constantly alternate in control inputs, always causing shaky trajectories. \nThis result needs further investigation. \nThis experiment also requires more scenarios (including more agents) to validate the approach. \nLastly, another safety baseline, such as a reactive control barrier function approach, would make the experiments stronger. \n\nThe limitations section is good and reveals shortcomings of the approach. \nYet, it should be improved by mentioning the persistent feasibility problems of the discrete set of hypotheses. \nAlso, it is not clear whether the paper focus on \"safety analysis\", \"control synthesis\", or \"game theoretic planning\". These terms are used interchangeably in the paper.\n\nOther minor things are:\n- In Fig. 1, it is not yet clear what the illustrated polygons show? \n- Fig. 4 is not self-explanatory for readers, e.g., the changes, current belief, or planned trajectory. \n- What are \"road constraints\" in the second experiment?\n- The running example in L163 is not helpful for readers to understand the previous paragraph\n- The demonstrations with Fig. 6 are not well described in the paper.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "Z4sFgy0VFg", "reviewer": "Reviewer_6ETr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach to considering safety in the presence of uncertainty over the dynamics of other agents in the scene. Rather than take a fully robust approach, the authors propose a method to account for the ego's ability to infer the latent parameters from future observations. Specifically, they propose a HJI reachability based approach which operates in the belief state dynamics. Their approach, Belief Game, formulates a reach avoid game in belief-state space, where the ego chooses actions to reach a goal while avoiding the other agents, while the other agents choose actions adversarially, constrained to a belief-state dependent control set. To address the high dimensional state space induced by accounting for the belief dynamics, they propose using adversarial RL to approximately solve the HJI equation. They demonstrate that this approach is less conservative than a fully robust approach while still ensuring safety, unlike MAP estimate-based planning approaches.", "summary_of_recommendation": "Overall, I think this paper presents an interesting approach to learning-aware safe planning. While the approach is presented clearly, and the experiments demonstrate that the learned policies can solve relatively simple interaction scenarios better than baselines, it's not clear to me that the resulting policies lead to realistic behavior, and I am still left with questions on how the results are meant to be used to support safety analysis.\n\nThe discussion period has been fruitful, and I appreciate the improvements the authors have made to their paper. I continue to support the acceptance of this work, and keep my score as is.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "It would be interesting to see to what degree the resulting policies match real-world driving. Perhaps you could include results showing ego behavior in the traffic intersection scenario simulated with the other agent following its actual, real-world trajectory, and comparing the observed behavior of each of the policies against the real-world behavior of the ego?\n\nHow does performance depend on the accuracy of the belief updating model?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper tackles the challenging problem of robust, learning-aware planning under uncertainty: aiming to simultaneously be robust to uncertainty over latent parameters, while also accounting for how uncertainty may be reduced during interactions. The proposed approach is interesting, and well motivated and clearly explained, though fairly similar to the framework proposed by [10]. Empirically, the experiments demonstrate that the approach indeed reduces over-conservatism of a fully robust approach, and matches the performance of a MAP estimate based approach while simultaneously assuring safety. I also appreciated the discussion of how this approach could be scaled by using deep adversarial RL, and the demonstration that the approach did not need to rely explicitly on exact Bayesian inference, but could also work with the implicit approximate inference performed by state-of-the-art trajectory predictors which consume past agent trajectories.\n\nOne concern with this work is the scalability of the approach. Even with the proposed approximation techniques, the examples considered only considered a single non-ego agent with a limited set of latent parameter values. Furthermore, the approach yields a policy designed for a particular reach-avoid problem tailored to a particular scenario, and not a general policy. I would imagine that solving the adversarial RL problem is too computationally expensive to run online as a planner, and I'm not sure what value running this optimization on a single scenario provides in terms of \"safety analysis.\" The paper would be strengthened with a discussion on how the results from the proposed approach could be used, similar to Bajcsy et al. [10].\nDespite this, however, I think the problem formulation is interesting, and would be of interest to the robot learning community even if the algorithm cannot scale as is.\n\nHowever, my main concern with this work is in the interaction between the adversarial assumption on other agents and the assumed belief updating dynamics.\n- First, the authors show that the resulting ego policy learns to actively reduce uncertainty. However, the behavior this results in seems it may be overfit to adversarial agents. For example, to resolve ambiguity between a pedestrian and segway in Fig the agent chose to turn left, forcing an adversarial agent to move upwards toward the vehicle, something that only a pedestrian agent could do. However, in the real world, a pedestrian is unlikely to take such behavior -- in this way, it appears that the learned policy is taking information-gathering actions that work only for adversarial agents. I'm not sure if this behavior is desirable.\n- Next, I'm curious about the implications of the belief threshold epsilon on the predictive control bounds. It would seem that choosing epsilon to be too large could break guarantees, as a true adversarial agent could take actions that the ego's Belief Game didn't account for. Are there any theoretical guarantees on safety that can be provided for the safety of the optimal policy obtained from the Belief Game? What are the implications of epsilon on these guarantees?\n\n\n\nMinor comments:\n- In line 144, it states that the reach-avoid set is visualized in Fig 1a. In line 81, it states the reach-avoid set is the set from which the ego agent is guaranteed a safe control strategy. However, in Fig 1a, the visualized sets are largest for the conservative approach, and smallest for the optimistic. but this feels backwards -- are the sets that are visualized actually the complements of the reach-avoid sets? i.e. the sets where the the assumptions predict a possible collision?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "SYAw-NgwSQ-", "reviewer": "Reviewer_dWyx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the problem of developing a\nsafe controller in the presence of unpredictable\nagents. In particular, the authors note that\nstatic approaches such as Hamilton Jacobi\nReachability tend to be too conservative since\nthey do not factor in information available\nat run-time. Thus, the authors propose a\nbelief-based safety analysis framework that\ncouples the system's current belief about\nsurrounding agents with model-free reinforcement\nlearning (RL). Evaluation is provided on 3 driving\nbenchmarks.", "summary_of_recommendation": "I think the paper has some interesting ideas, particularly the idea of using RL to overcome some of the scalability challenges of standard Hamilton Jacobi methods. I am only recommending a weak accept (as opposed to a strong accept) because the evaluation doesn't show a massive improvement in scalability (at significant costs in safety), so it is unclear whether the issue is just figuring out the correct RL setup or if there is a fundamental limitation to the proposed method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. It seems that the RL framework may result in quite unsafe controllers, as compared to the more computationally heavy dynamic programming one. The authors are encouraged to comment on why and when the RL framework is not so useful in terms of safety. \n\n2. Also, how different is the computation time between the DP and the RL methods? Judging from Table 1 and Figure 3, the difference \ndoesn't seem to be that big. Perhaps a more involved scenario would be more convincing, where the DP method fails to terminate within some desired time (e.g., for run-time operation).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the paper is well written. The problem\nof safe control is important and relevant. I like\nthe systematic evaluation and explanation of each\nbenchmark.\n\nThe first comment I have is that no evaluation is\nprovided in a high-dimensional observation setting\nwhere the proposed approach is presumably envisioned\nto run. In particular, how are the evaluation\nscenarios modeled exactly? Is there a simulator?\nWhat is the object recognition accuracy? Also,\nhow does the number of objects affect the\nalgorithm's performance?\n\nA second comment I have is that the RL framework\nmay result in quite unsafe controllers, as compared\nto the more computationally heavy dynamic programming\none. The authors are encouraged to comment on\nwhy and when the RL framework is not so useful in\nterms of safety. Also, how different is the\ncomputation time between the DP and the RL methods?\nJudging from Table 1 and Figure 3, the difference \ndoesn't seem to be that big. Perhaps a more\ninvolved scenario would be more convincing, where\nthe DP method fails to terminate within some\ndesired time (e.g., for run-time operation).\n\nUPDATE AFTER REBUTTAL: The authors have addressed some of my concerns. Of course, the question of the ensuring the safety of RL controllers remains open. At the same time, absolute safety guarantees may require unnecessarily conservative policies. I think the paper has sufficient theoretical and experimental results to merit acceptance.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel way of synthesizing safe closed-loop control policies for robotic systems under uncertain future scenarios, using the framework of belief games. Reviewers agreed that the approach is interesting and novel, and valued the demonstrated results, especially after additional experiments and clarifications of the authors in their thorough rebuttal.
Decision: Accept (Poster)