Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures
Authors: Kensuke Nakamura
Thomas Tian
Andrea Bajcsy
Venue: CoRL 2024 (2024)
Keywords: Human-Robot Interaction
Trajectory Prediction
Failure Detection
TL;DR: We formalize system-level trajectory prediction failures via regret, and automatically mine for closed-loop human-robot interactions that state-of-the-art generative human predictors and robot planners struggle with.

Abstract: Robot decision-making increasingly relies on data-driven human prediction models when operating around people. While these models are known to mispredict in out-of-distribution interactions, only a subset of prediction errors impact downstream robot performance.  
We propose characterizing such ``system-level'' prediction failures via the mathematical notion of regret: high-regret interactions are precisely those in which mispredictions degraded closed-loop robot performance. 
We further introduce a probabilistic generalization of regret that calibrates failure detection across disparate deployment contexts and renders regret compatible with reward-based and reward-free (e.g., generative) planners.  
In simulated autonomous driving interactions, we showcase that our system-level failure metric can automatically mine for closed-loop human-robot interactions that state-of-the-art generative human predictors and robot planners struggle with. 
We further find that the very presence of high-regret data during human predictor fine-tuning is highly predictive of robot re-deployment performance improvements. 
Furthermore, fine-tuning with the informative but significantly smaller high-regret data (23% of deployment data) is competitive with fine-tuning on the full deployment dataset, indicating a promising avenue for efficiently mitigating system-level human-robot interaction failures.

Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures
Kensuke Nakamura' Ran Tian
Andrea Bajesy
Carnegie Mellon University 2 UC Berkeley
(kensuken, abajesy Bandrew emu.edu. rantian@berkeley.ce.
Robot decision-makes increasingly relies data-driven human pre- diction models when operating around people. While these models are known to mispredict in out-of-distribution interactions, only a subset of prediction er- rors impact downstream robot performance. We propose characterizing such "system-level" prediction failures via the mathematical notion of regret: high- regret interactions are precisely those in which mispredictions degraded closed- loop robot performance. We further introduce probabilistic generalization of regret that calibrates failure detection across disparate deployment contexts and renders regret compatible with reward-based and reward-free (e.g.. gener- ative) planners. In simulated autonomous driving interactions and social nav- igation interactions deployed on hardware, we showcase that our system-level failure metric can be used offline to automatically identify challenging closed- loop human-robot interactions that generative human predictors and robot plan- ners previously struggled with. We further find that the very presence of high- regret data during predictor fine-tuning is highly predictive of closed-loop robot performance improvement. Additionally, fine-tuning with the informative but sig- nificantly smaller high-regret data (23% of deployment data) is competitive with fine-tuning on the full deployment dataset, indicating a promising avenue for effi- ciently mitigating system-level human-robot interaction failures. Project website: https:/ / /////mmm cmmu como-intenttaan github. io/ /nott http//ottalllleeeeee -al--a--------- all-errors/
Abstract:
Keywords: Human-Robot Interaction, Trajectory Prediction, Failure Detection
Introduction
From autonomous in cities [1, 2, 3] to tabletop manipulators at home [4, 5, 6], robots operating around people rely on data-driven trajectory during planning to to anticipate how others will behave. Despite their widespread adoption in robot autonomy pipelines, current human prediction models are still imperfect: they frequently mis-predict when faced with out-of-distribution interactions [7, 8] and are prone to causal misidentification [9] wherein the model "lazily" converges to incorrect correlates in the data. These prediction errors can lead critical downstream robot system failures, impeding performance at best and compromising safety worst. Thus, analyzing when and how trajectory prediction errors lead to robot failures is critical for developing runtime monitors, creating safety-focused benchmarks, and incrementally improving the prediction models themselves.
While current methods detect interaction data that is dissimilar to the training distribution [10] or leads to highly uncertain or erroneous predictions [11, 12, 13], these are all fundamentally component-level approaches that detect only when the predictor faulty in isolation. However, prediction errors at the component-level do not always lead svstem-level robot decision-making errors downstream [14]. Consequently, relying component-level failures to evaluate, monitor, or improve the prediction model is inaccurate and inefficient. For example, consider Figure 1. Even though the robot's predictions were faulty in all three scenarios, it is only in the latter two where this
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Component-leved Failure
Level Failure: Collision
System-lrvel Failure: Inefficiency
C
Robot
Robot Action
Predictions
Figure I: All scenarios have component- prediction failures: mispredicting that parked cars will turn (left), marooned truck will to move (center), and nearby cars will lane change (right). But, only the center and right scenarios have system-level prediction failures which impact robot performance.
caused undesirable closed-loop robot behavior: collisions the center and inefficiency the right. The core challenge we address is the automatic detection of such system-level prediction failures.
Our key insight is that the mathematical notion of regret is rigorous way to identify system- level prediction failures. High-regret interactions are precisely those in which mispredictions caused the robot to make a suboptimal decision in hindsight. In Section 4 we formalize this idea and introduce probabilistic generalization of regret that no longer depends on explicit reward func- tions for regret computation, extending its applicability to reward-free planners such as generative models. In Section we extract system-level failures offline from dataset of collected deployment interactions and compare our approach alternative failure detection methods with a state-of-the- art autonomous driving trajectory predictor and planner, and demonstrate the compatibility of our approach with generative neural planners in a social navigation setting deployed on hardware with on-board sensing. Finally, in Section 7, we conduct a case study where we find that fine-tuning robot's human behavior predictor on the small amount of high-regret data (23% of the deployment data) is highly predictive of system-level performance improvements at re-deployment.
2 Related Work
Interaction-awarr Prediction. Interactive robots commonly conditional behavior predictions to model the influence between the robot's decisions and other agents' future behavior [15, 16, 4, 17]. More recently, there has also been shift towards using multi-agent prediction models or generative "foundation models" as planners [18, 19, 20, 21, 22, 23, 24]. However, these models are overwhelmingly evaluated on log-replay data, missing out on the closed-loop interaction effects between agents. This may lead to overly optimistic estimations of how the robot may influence the behavior of others [25] and poorly affect downstream performance. We introduce system- level metric for evaluating interaction-aware predictors that quantifies closed-loop performance of the trajectory predictor. We also show that fine-tuning such predictor identified system-level failures leads to significant downstream performance improvements.
Component-level Failure Detection. Many prior works have studied the detection of anomalous or out-of-distribttion data (see [26] and [27] for surveys) relative prediction model's training data: classifying poor predictions based average displacement error (ADE) [12, 28). Bayesian filtering [8, 29, 30], kernel density estimation [31]. uncertainty [11, 32), conformal prediction [33],
and evidential deep learning [10]. However, these approaches are fundamentally component-level, ignoring the "downstream" effects of errors in the predictor on robot decision-making.
System-level Prediction Metries. Our work contributes the small-yct-growing area of developing
system-level metrics for evaluating components of the autonomy stack [34, 14, 35, 34, 36, 37, 38]. Existing system-level trajectory prediction metrics compare the difference between predicted and incurred costs [37] or weight prediction errors by a task-dependent cost function [38, 39, 40J. In ad- dition to evaluating predictors, system-level metrics can also be useful in the offline setting, wherein datasets are mined for challenging examples. For instance, Stoler et al. [41] mine for safety-critical interactions via heuristic features and Hsu al. [42] mine via counterfactual responsibility metrics. Our approach is also instantiated in offline setting, but utilizes our novel regret metrie iden- tify sevel human-robot interaction failures. Furthermore. our method does not require cost function to detect system-leved failures, rendering method compatible with reward- fareefreeeeeeeeee planners.

---

3 Problem Formulation
Agent States, Actions, and Context. Let the robot state be denoted by S E R and the state of the .M human agents in the scene be denoted by IHt R i E (1....,M M). For example, for an autonomous driving application domain, each agent's state can be represented as position, heading, and velocity. All agents' states evolve according to their respective control actions: a E R for the robot (c.g., accelcration and turning rate) and a E R for the M humans, where nR and ma are the dimensions of the robot state and action and are similarly defined for each human. For shorthand, we denote the joint human-robot state at any time l as 8 = [o .66 H.. ,...,,((') HHM). Let
T-length future sequence of actions starting at time be a. [a ..... d+4rl any human and a. = [a. ...., a r] for the robot. Finally, let denote the scene or environment context (e.g,, semantic map the environment) used for both human prediction and robot decision-making.
Human Prediction & Robot Planning, We assume the robot has access to pre-trained human behavior prediction model Pg with parameters f capable of conditional generations [16, 18). Math ematically, this predictor is:
Po(a!l:HHa [ap S1, C).
The model can generate T-horizon predictions of all M humans in the scene, a H.:Harr (a." at HM) conditioned on robot action trajectory aR scene context C, and current! human- robot state st. We also use the robot's task-driven planner:
Rt (8,, Pa. C),
(2)
with parameters Φ. This planner generates best-effort robot plan at with respect to the current joint state 8t, scene context C and predictions, Po- Within our framework, the robot's planner could be optimization-based planner that uses reward function (where ó are feature weights of pre-specified reward function) or neural planner (where á are the pre-trained neural network weights). At deployment time, the robot's 's planning model can be re-queried given new observations and predictions in a receding horizon fashion. We assume that the planner is optimal under the given reward function or the available data for planner design.
Problem Statement. We deploy the robot for T time steps in N scenarios (e.g., different initial con- ditions, environments) and collect an aggregate deployment dataset D = {($, â " )n J consisting of observed state and action trajectories for the robot and M humans. Importantly, this observed data exhibits how real interactions between the humans and robot occur, and are a function of the robot's predictive human model used during planning. We seek to automatically identify a subset of deployment data, D)= DD that captures system-level prediction failures: deployment interactions wherein prediction failures significantly impacted closed-loop robot performance.
4 A Regret Metric for Detecting System-level Trajectory Prediction Failures
Canonically, regret measures the reward (RR) difference between the optimal action the robot could have taken in hindsight Ra and the executed action (aR) the robot took under uncertainty [43]: (eg,d)= = m R& R a, H &t, C*) (a a HH:Ha 8t, C),, d E D. If we were to rank all the interactions in DD by the robot's regret, we could identify those with the highest regret as failure dataset, Dy. While regret a principled measure of
a means of building our system- system-level failures, directly using rewards during regret computation may not always be desirable. First, not all robot planners use reward functions-fie for example, generative planners fit joint distribu- tions to multi-agent behavior [18] and choose trajectories via their likelihood rather than an explicit reward. This immediately makes canonical regret impossible to evaluate. Second, the canonical regret definition assumes that the same difference in reward values in different deployment contexts are fairly comparable. Unfortunately, in practice, when robots are deployed in wide variety of scenarios and contexts C, is difficult to have such perfectly calibrated reward. We illustrate this technical point with example from simulation experiments (details in Section 5).
'The predictor Po can also accept a state history, St-hit where is the number of previous timesteps.
3

---

Deplyyment Prodicion Plan
Revard
Illustrative Example. Consider Figure 2, where the same robot (blue car) is deployed in two different contexts: one where the robot drives behind a stopped truck (C1) and the other where it navigates an intersection (C2). The robot's reward is tuned linear combina- tion of control effort, collision penalty, road progress, and lane-keeping-
C2
Obsoree
In scenario C₁ the robot mispredicts that the stopped truck will accelerate; the robot initi- ales lane change and collides. In scenario Cz. the robot aggressively swerves to avoid nearby traffic. Because of the absolute reward difference between the best and the executed action (right column, Figure 2), canonical re- gret regards both scenarios as equally poor: C1 has a regret of 11.4 in C2 has 11.7. While the robot could have taken a more optimal action in both scenarios, canonical regret unable to identify that the robot's decision in C₁ more severely suboptimal than the decision in Cz.
Figure 2: Illustrative Example. Left column: robot's predictions and deployment-tine decision. Right column: the counterfactual analysis given observed human behavior. Each hindsight-optimal action and executed action's 's reward is on the right.
Generalized Regret Metric. To address this calibration issue we propose that instead of looking at the absolute reward of a decision, we consider the likelihood it. This probabilistic interpretation decision relative to the context-dependen behavior distribution and is a principled way place all decision comparisons on the same scale-a value between zero and one.
normalizes the quality
&, C) be a likelihood model that yields the counterfactual prob- ability of any candidate robot action sequence (a") given the observed actions of the other human agents (ā (a)) and the observed joint state trajectory (8) in that context C. Constructing this like- lihood model only requires access to the robot's planner; hence, it shares the same parameters as (o). For example, if the robot using soward-based planner,
Specifically, let (a
the robot used for decision-makes then this probability distribution shaped according the same reward weights and features that the robot used for decision decision-making (see example in Equation 4). Additionally, this generalization to probability-space allows to calculate regret for generative neural planners, where the likelihood model shares the same weights as the neural planner. For any human-row robot deployment interaction = (8, RR, H. Hxr) E DD we quantify the generalized regret of a robot's action at timestep / as:
Reg, (d) max Pa H.H..ar 8,, C) P& (á. | H.H.Har $1, C)
a.
Illustrative Example. Consider the reward-bused planning example from Figure 2. We model Po using the Luce-Shepard choice rule [44], which places exponentially more probability robot de- cisions that are high-reward. Our generalized regret metric results in inluitive failure separation where canonical regret failed: C1 has a higher regret of 0.56 compared to 0.34 in C2.
Simulation & Hardware Experimental Setup
We first instantiate our metric in the autonomous driving setting. Our prediction model Pa is an ego- conditioned Agentformer model [45] which takes as input a candidate robot (i.e., ego) trajectory a. a history of all the vehicles in the scene St-k:t and the map information C output prediction of the behavior of all human (i.e., non-ego) vehicles a, Hy:Har Our planner ाe is off-the-shelf -wserd-based MPC planner [46] who's parameters & are weights of hand-tuned reward function consisting of lane progress, lane-keeping, collision cost, and control cost (see Appendix 9.1). We pretrain Pe on the nuScenes training split [47] and we obtain our closed-loop deployment dataset D consisting of 96 simulated 20-second scenarios from the BITS simulator [48] where each scenario's map and initial agent history are initialized from the nuScenes validation split.
We further instantiate our approach for generative robot planners Interbotix LoCoBot [49], shown in Figure 3. The LoCoBot has an Intel D435 RGB-D camera and RPLIDAR A2 2-D Lidar

---

Correct Prediction
Irrelevant Mis-prediction
System-level Failure
Prediction
Buality
Predistion
Realit
Prodiction
Buality
Regret: 0.0777
Regret: 0.0529
Regret: 0.3069
Figure 3: Left: The robot correctly predicts the human will block its goal and proceeds straight. Middle: The robot incorrectly predicts that the human will walk straight ahead, however it is able to reach its goal despite the misprediction. Right: The robot mispredicts that the human will block its goal and proceeds straight, colliding with the human who also walked straight ahead. The robot's executed trajectory is unlikely conditioned on the human's 's true actions and is assigned high regret.
mounted on a Kobuki mobile base. We model the robot as a Dubins' car [50] which must reach goal position 9d unless it would cause a collision with a nearby freely moving pedestrian, in which case it navigates to a backup location (Figure 3). For constructing the robot's generative planner Tø. we model the joint human-robot state as the states of both agents along with low-dimensional behavior cue from the human 8 (e.g.. eye-gaze, head tilt) that leaks information about the human's 'ss future trajectory. We take 8 and fix all other initial conditions between deployments. The context of the scene is the intended goal of the robot C = = The generative planner T& is encoder- variational autoencoder (VQ-VAE). It is trained
decoder architecture based on the vector-quantiznn with a dataset of 10,000 simulated joint human-robot trajectories (a a generated via simple rules (described in Appendix). The encoder Pa(z |8 RR), with learnable parameters a, encodes the human observation 8 and the original goal 9 and produces a categorical distribution over latent embedding through vector quantization. The decoder (z a | =), with learnable parameters 3, takes the latent vector z P- 8th g and approximates the joint distribution over actions a a ! Implementation details on datasets and hyperparameters are included in Appendix 9.2.
Generalized Regret Computation: Reward- -base Planner. Our likelihood model follows from the Luce-Shepard choice rule [44] where the parameters e denote the weights of the reward function:
Po (a | HH::Has 8, C) = 17 eR (a HH:HHM 8, C,)))
(4)
where 77 :- 1/E= RR, HH:HH 8, C)}. For each timestep in deployment data d E D where the robot originally re-planned, we re-compute the reward of each candidate ego action with respect to the ground truth behavior of the other agents, instead of predictions. With this we compute the hindsight likelihood of each candidate ego action and obtain the scene's 's regret: Σ, Reg, (d).
Generalized Regret Computation: Generative Planner. our hardware experiments, the hu- man's positions were recorded as (z,y) y) planar positions at 12 Hz using the robot's LIDAR, and transformed to a global coordinate frame of a precomputed SLAM map. The behavioral eue 8 is obtained by tracking the position of the human and calculating the angle between timesteps, which is given to the planner to generate a 6-timestep action trajectory a which is executed at 2 Hz. After each action is executed, we recorded smoothed measurements of the human's 's trajectory by taking a simple moving average of the human position and measuring the angles between subsequent po- sitions to obtain " We construct the likelihood model using Bayes' theorem where we first form a kernel density estimate of (aR R H 8H g ) by sampling from the the planner 250 times and approximate the conditional probability of a atd by integrating the KDE around a small neighbor- hood of the ground-truth joint action trajectory. The conditional probability of à is approximated by marginalizing over the space of possible a Leveraging the approximate probabilities, we com-
5

---

pute an estimate of the counterfactual probability for any a²:
E. Pa a 2)Pa(2 | ,") Ln Pa(a", a², 2P ( (0))
(a a a", 1,"8".""))
This is a particular instantiation of Po where the planner and predictor both share parameters (i.e., = 0) given by a and 3. We repeat this process to evaluate the maximum probability action trajec- tory". giving us both terms necessary for our generalized regret computation (-) in Equation 3.
6 Experimental Results: Detecting System-Level Prediction Failures
6.1 How does Regret Compare to Other Prediction Failure Metrics?
We first quantitatively and qualitatively compare scenarios ex- tracted by our generalized regret metric (GRM) against three al- ternatives: average displacement error (ADE), a component-level metric that measures the L2 error between the true and predicted trajectory, the system-level metric of Farid et al. [37] (TRFD), and the canonical variant of our system-level regret metric (RM). We apply each metric to the initial closed-loop simulated driv- ing deployment data as described in Section 5. For GRM, ADE, and RM, we construct the failure dataset as the top 20-quantile scenes? (i.e., 20 scenarios with 20-second long interactions) as
scored by the metric, yielding DiRM, ,DADE and DJM. TRFD is unique in that it only assigns binary labels if the scenario is a system-level failure. The pairwise overlap between Dz from each metric is visualized in Figure 4.
C301
ADE is a component-level metric and has only 35% overlap with scenes identified by the two regret metries (GRM, RM). Qual- itatively, the scenarios with high displacement eиror tended to have many agents single agent that the baseline Agentformer model largely mispredicts. An example of a scenario with high ADE but not flagged by our two regret metrics is shown in Figure 4. The ADE in this scenario is dominated by the robot consistently mispredicting a faraway vehicle to change lanes. This indicates that component-lewe ADE may attend irrelevant prediction failures instead of system-level failures.
Figure 4: Qualitative compari- son between scenarios uniquely identified by each metric.
TRFD is a system-level metric that flags an interaction as a failure if the incurred robot reward is in the bottom p-quantile of the robot's 's reward distribution predicted during planning. Since this met- ric relies on a predicted reward distribution, is fundamentally linked to the quality of the robot's predictions of other agents' behavior. We find that TRFD assigns every scenario being anoma- lous (Figure 4). We hypothesize the distribution shift between the nuScenes data (that the baseline Agentformer model was trained on) and the simulated human behavior of the BITS agents causes the predicted reward distribution to be extremely misaligned with the realized reward, causing all interactions be flagged as failures. Instead of comparing the realized outcomes with the predicted reward distribution, GRM and RM compares against counterfactual ground-truth information, mak- ing them agnostic the base predictor quality.
RM is our absolute-reward variant of likelihood based regret metric, GRM. Because they are both grounded in the idea of regret, RM has 95% overlap with GRM. The scenario uniquely identified by each metric is shown in Figure 4. Notably, GRM identifies an additional scenario where the ego vehicle drives through stopped traffic (top. Figure 4), whereas RM identifies an instance of the robot driving near unmoving vehicles and showing slightly conservative behavior (middle, Figure 4). Although downstream rewards are common evaluation criteria for task-aware trajectory forecasting [37, 38), these results show that reward alone suffers from calibration challenges and may overlook pertinent seve failures.
6.2 Can We Detect System-Level Failures for Reward-Free Generative Planners?
We find this maximizing R by searching over a prespecified set action trajectories.
The choice of p is design decision. We chose p 20 due to the small size of the deployment dataset.

---

ADE/ UDDE/FDEE ()) High-Ragret
The qualitative results of our GRM approach in the real-world generative planning setting are shown in Figure 3. Importantly, no other system-level metrics (TRED, RM) generalize to generative planners because they require ex- plicit reward functions. We find that our ap- proach only rates interactions as high-regret (0.307 on a scale from 0 to I) when mispredic- tions lead unexpectedly close proximity in- teractions between the robot and human (right. Figure 3). In both nominal interactions (left, Figure 3) and interactions where mispredic- tions are irrelevant for robot performance (mid- dle, Figure 3), the deployment regret is close to zero (0.078 and 0.053). Overall, our new regret metric captures our motivating intuition on system-level prediction failures without the need for manually specifying an evaluative reward- function for either the high-level or low-level robot and human behaviors.
Fine-tuning Data
nuScenss
Low-Regret
Nome (Base model)
0.371 / 1.919
0.639 / 2.999
0.610/2.375
Low New-aegret-FF
0.353 / 1.554
0.386 2.333
0.386 1.783
Random-FT High-regret-F
0.315 / 1,435
0.306 2. 100
0,382 1,738
0.327 / 1,520
0.333 / 2,014
0.366 1.713
AII-FT
0.297 / 1.381 0.326 1,973
0.340 1.665
Table I: Fine-tuned Open-loop Prediction Perfor- mance. Average ADE / FDE of the base Po and the fine-tumed predictors on log-replay data. Columns are different log-replay validation data splits: orig- inal nuScenes data, closed-loop simulation data that does exhibit system-level failures (i.e. high-regret) and doesn't (i.e., low-regret). Lowest error denoted in italics-bold, and second lowest in hold. Each baseline was trained with three random seeds and the reported numbers are from best best-performing model.
7 Case Study: Mitigating Prediction Failures via High-Regre Fine-Tuning
We demonstrate one potential use for system-level failure data by fine-tuning the Agentformer model on high-regret interactions. We first hold out 20 (comprised of 17 low-regret and 3 high-regret scenarios) of the 96 deployment scenarios for evaluation and leave the rest (76) potential data for fine-tuning. We compare the closed-loop simulation (Table 2) and open-loop prediction (Table 1) performance of Agentformer fine-tuned on four different data subsets. Low-regret-FT uses the 20 scenarios with the lowest regret, Random-FT uses 20 random scenarios from the 76 scenarios available for fine-tuning, High-regret-FT uses the 20 scenarios with the highest regret, and All-FT is a privileged baseline with access to all 76 scenarios. During fine-tuning we unfreeze only the final 1.5M parameters of the 17.7M parameter Agentformer model, and we examine the results from fine-tuning the model on each dataset for 3 random seeds. Details and ablations in Appendix 10.
Takeaway 1: Hegh-Regret-FT improves log-replay prediction accuracy in high-regret scenarios, without degrading performance on low-regret scenarios and the original muScenes validation set.
We first evaluate the predictors "open-loop" on log-replay data from three different validation datasets: the original nuScenes validation data, the high-regret scenes in the deployment vali- dation dataset, and the 17 low-regret scenes in the deployment validation dataset. We report the Average Displacement Error (ADE) and Final Displacement Error (FDE) for each baseline's best performing seed in Table 1. The base predictor doubles its ADE and FDE on the simulated de- ployment data due to a distribution shift the driving behavior between nuScenes and simulation. Intuitively, High- Regret-Fe has the most ADE and FDE improvement data. We hypothesize that the improved open-loop prediction accuracy on low-regret scenes is be- cause the high-regret validation data leaks enough information about the general distribution shift between the nuScenes and the BITS simulation to enable improvements on nominal interaction data as well. While All-FT achieves the best prediction accuracy across the board, it is important to note that this is an information-advantagee advantaged baseline: it has the ability to learn from both sestem-level and non-system-level failures the deployment data. However, Righe-eeget--T is much more data efficient to train, rivaling the performance of All-FT despite using 77% less data. As the deployment dataset D grows and training time and costs increase along with it, this can be a promising avenue to alleviate these challenges while still achieving high predictive performance. Finally, the least improvement over the base model is exhibited by New-Regret-FT and followed by Random-FT which demonstrates the quantitative value of learning from high-regret data.
high-regret validation
Takeaway 2: Including high-regret data during fine-tuning improves closed-loop performance.
Our first evaluation metric for measuring closed-loop re-deployment performance differences is To- tal Collision Cost, which is the total cost the ego incurred from collisions. Second, to disentan- gle severe but infrequent collisions from frequent clipping grazing we measure Collision Sever- Total Collision Cost. Finally, we measure average Regret as defined in Section 4. We report average ity= #Frames in Collision
closed-loop performance for each method across three seeds of fine-tuning in Table 2.
7

---

RegheRt Scenarios
Low Now-Regret Scenarios
Fine-tuning Data
Col Cost (1)
Col Severity
Regret (1)
Col Cost (1)
Col Severity
Regret (L)
Base Model
10,829
0.848
0.034
0,411
0.051
0.006
Low-regret-FT
7.489 (#: 0.408)
0.794 (=: 0.059)
0.019 (= 0.002)
0.670 (# 0.206)
0.354 (#: 0.493)
0.009 (#: 0.000)
Random FT
7.209 (1: 0.883)
0.743 (2: 0.064)
0.016 (2: 0,003)
0.495 (4 0.213)
0.076 (:: 0.054)
0.006 (1: 0.000)
High-regret-FT
3.763 (+ 0.965)
0.588 (+ 0.101)
0.016 (+ 0.003)
0.397 (+ 0,814)
0.062 (+- 0.031)
0.006 (+ 0.001)
AIl-FT
(1: 0.775)
0.377 (1: 0.067)
(2 0.002) )
0.486 (#: 0.131)
0.060 (:: 0.030)
0.007 (4: 0.001)
Table 2: Closed-Loop Re-Deployment Performance. Average robot performance metrics for each fine-tuned predictor (standard deviation over three seeds of fine-tuning in parentheses). Two main columns are evaluations on high-regret and low-regret held-out validation scenes. stalics-bld indi- cates best, and bold indicates second best. Across all metrics and settings, High-regret-FT consis- tently improves robot performance, and is competitive with the privileged All-FT approach.
Model
We find that High-Regret-FTT decreases collision cost, sever- ity, and average regret compared to the base model across both high- and low-regret scenes. As hypothesized, most closed- loop performance gains are observed the high-regret scenes, reducing robot regret by -53% and collision cost by -65% (shown in Table 2). By being able see all the deployment data, AIl-FT observes the highest performance gains in high- regret scenes but shows slight (but not statistically significant) performance degradation in low-regret scenes. For each base- line, the degradations from the base model are within stan- dard deviation (except for Low-Regret-FT) In contrast, the improvement of High-Regret-FTT is more than one standard deviation better than the performance of Low-Regret-FTT and Random-FT with respect to the collision oollision-based metrics. The closed-loop performance gains resulting from fine-tuning on high-regret data is shown qualitatively in Figure 5.
Low-
Now-Roggoot
Rendom
Hligh-Regre Regret FT
Takeaway 3:
Closed-looy and open-loop performance are only correlated on high-regret scenes.
We find that in the high-regret scenes, the open-loop perfor- mance (Table is positively correlated with the closed-loop planner performance (Table 2). We hypothesize that this is be- cause these scenes are much more challenging (e.g.. highly interactive). encode more nuanced human behavior, and af- feet the planner the most; thus. the prediction improvements in such scenarios indicate that Po has learned to predict chal- lenging interactions better, naturally improving the closed-loop planning performance.
humans
Figure 5: As Po is fine-tuned on more high-regret interac- tions, closed-loop performance improves.
8 Limitations & Future Work
One limitation of our system- failure detection method is the need for a designer set p when detecting scenarios with the top p-quantile regret. Poorly tuning this could result in classifying benign scenarios as anomalous or overlooking true system-level failures. setting, extremely out-of citto--distribution interaction data (e.g., not in the support of the training dis- tribution) could lead to poor regret estimates. To remedy this, we hypothesize that our approach can be complemented by an anomaly detector that detect highly out-of-distributis deployment. Furthermore, our hardware experiments underscored how the logged data collected by the robot during deployment (e.g., states of agents) must be sufficiently high quality to compute accurate regret. Failures in detection/tracking from perception modules in real-world deployments could corrupt the data used to calculate regret and misidentify system-leve failures.
the generative planner
data during
While we instantiated our approach in simulated and hardware robot navigation interactions, collab- orative manipulation [4]) is an exciting future direction. Finally, future work should also investigate alternative uses for the datasets constructed with our regret metric, e.g., to inform runtime monitors that can anticipate seseem-level prediction failures online.

---

References
[1] E. Schmerling, K. Leung, W. Vollprecht, and M. Pavone. Multimodal probabilistic model- based planning for human-robot interaction. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3399-3406. IEEE, 2018.
[2] N. Rhinehart, R. McAllister, K. Kitani, and Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821-2830, 2019.
[3] S. Shi, L. Jiang, D. Dai, and B. Schicle. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 2022.
[4] K. Kedia, A. Bhardwaj. Dan, and Choudhury. Interact: Transformer models for human intent prediction conditioned on robot actions. International Conference Robotics and Au- tomation, 2024.
[5] K. Kedia, P. Dan, A. Bhardwaj, and S. Choudhury. Manicast: Collaborative manipulation with cost-aware human forecasting. Conference on Robot Learning, 2023.
[6] W. Abeyruwan, L. Graesser, D. B. D' A'mbooso,,, A. Singh, A. Shankar, A. Bewley, D. Jain, K. M. Choromanski, and P. R. Sanketi. i-sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops. In Conference on Robot Learning. PMLR, 2023.
[7] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's ' uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
[8] A. Filos, P. Tigkas, R. McAllister, N. Rhinehart, S. Levine, and Y. Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning, pages 3145-3153. PMLR, 2020.
[9] De Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. Advances in neural information processing systems, 32, 2019.
[10] M. Itkina and M. Kochenderfer. Interpretable self-aware neural networks for robust trajectory prediction. In Conference on Robot Learning, PMLR, 2023.
[11] B. Lakshminarayanan. A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[12] L. Sun, Jia, and Dragan. On complementing end-to-end human behavior predictors with planning. Robotics: Science and Systems, 2021.
[13] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fussion 76:243-297, 2021.
[14] R. Sinha, A. Sharma, S. Banerjee, T. Lew, R. Luo, S. M. Richards, Y. Sun, E. Schmerling, and M. Pavone. A system-level view out-of-distribution data in robotics. arXiv preprint arXiv:2212.1400 140200 2022.
[15] A. Cui, S. Casas. A. Sadat, R. Liao, and R. Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 16107-16116 16, 2021.
[16] E. Tolstaya, R. Mahjourian, C. Downey, B. Vadarajan, B. Sapp, and D. Anguelov. Identifying driver interactions via conditional behavior prediction. In 202 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[17] Y. Chen, S. Veer, Karkus, and M. Pavone. Interactive joint planning for autonomous vehicles. IEEE Robotics and Automation Letters, 2023.

---

[18] A. Seff. B. Cera, D. Chen, M. Ng, A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[19] X. Weng, B. Ivanovic, Y. Wang, Wang, and M. Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. 2024.
[20] Q. Sun, S. Zhang, D. Ma, J. Shi, D. Li, S. Luo, Y. Wang, Xu, Cao, and H. Zhao. Large tra- jectory models are scalable motion predictors and planners. arXiv preprint arXiv:2310. 19620, 2023.
[21] J. Mao, Y. Qian, H. Zhao, and Y. Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
[22] J. Mao, J. Ye, Y. Qian, M. Pavone, and Y. Wang. A language agent for autonomous driving. arXiv preprint arXiv:2311.0081, 2023.
[23] S. Wang, Z. Yu, X. Jiang, S. Lan, M. Shi, N. Chang, J. Kautz, Y. Li, and J. M. Alvarez. Om- nidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01555
2024.
[24] W. Wang. J. Xie, C. Hu, H. Zou, Fan, W. Tong, Y. Wen, S. Wu, H. Deng, Z. Li, et al. Drivemlm: Aligning multi-moda large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245,, 2023.
[25] C. Tang, W. Zhan, and M. Tomizuka. Interventional behavior prediction: Avoiding overly confident anticipation in interactive prediction. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.
[26] M. Basseville. Detecting changes in signals and systems-a a survey. Automatica, 24(3):309- 326, 1988.
[27] Q. M. Rahman, P. Corke, and F. Dayoub. Run-time monitoring of machine learning for robotic perception: A survey of emerging trends. IEEE Access, 9, 2021. ISSN 2169-3536.
[28] H. Liu, Y. Zhang, V. Betala, E. Zhang, J. Liu, C. Ding, and Y. Zhu. Multi-task interactive robot fleet learning with visual world models. In 8th Anmual Conference Robot Learning, 2024. URL https:tt////. /ttp::///ppeeeeeeeee net forum? DdR=IDIoSSSSSSTT
[29] D. Fridovich-Keil, A. Bajesy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and J. Tom- lin. Confidence-aware motion prediction for real-time collision avoidance. The International Journal of Robotics Research, 2020.
[30] K. Nakamura and S. Bansal. Online update of safety assurances using confidence-based pre- dictions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[31] J. Wiederer, A. Bouazizi, M. Troina, U. Kressel, and V. Belagiannis. Anomaly detection in multi-agent trajectories for automated driving. In Conference on Robot Learning, pages 1223 1233. PMLR, 2022.
[32] C.-M. Hung. L. Sun, Y. Wu, I. Havoutis, and Posner. Introspective visuomotor control: Exploiting uncertainty in deep visuomotor control for failure recovery. In 2021 IEEE In- ternational Conference on Roboties and Automation (ICRA), pages 6293-6299 2021. doi: N0.1109/CCA488506.20000 2044850662021..0 202..296111999999
[33] A. Dixit, L. Lindemann, X. Wei, M. Cleaveland, J. Pappas, and J. W. Burdick. Adaptive conformal prediction for motion planning among dynamic agents. In Learning for Dynamics and Control Conference, pages 300-314. PMLR, 2023.
[34] P. Antonante, S. Veer, K. Leung, X. Weng, L. Carlone, and M. Pavone. Task-aware risk es- timation of perception failures for autonomous vehicles. arXiv preprint vrXiv:2305.01870 2023.
10

---

[35] Philion, A. Kar, and S. Fidler. Learning to evaluate perception models using planner-centre metries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 14055-14064, 2020.
[36] K. Chakraborty and S. Bansal. Discovering closed-loop failures of vision-based controllers via reachability analysis. IEEE Robotics and Automation Letters, 2023.
[37] A. Farid, S. Veer, B. Ivanovic, K. Leung, and M. Pavone. Task-relevant failure detection for trajectory predictors in autonomous vehicles. In Conference Robot Learning. PMLR, 2023.
[38] B. Ivanovic and M. Pavone. Injecting planning-awarenee into prediction and detection evalu- ation. In 2022 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2022.
[39] X. Huang, G. Rosman, A. Jasour, S. G. McGill, J. J. Leonard, and B. C. Williams. Tip: Task-informed motion prediction for intelligent vehicles. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages I 11432-11439, IEEE, 2022.
[40] H. Nishimura, Mercat, B. Wulfe, R. McAllister, and A. Gaidon. Rap: Risk-aware predic- tion for robust planning. In Conference on Robot Learning, pages 381-392. PMLR, 2023.
[41] B. Stoler, I. Navarro, M. Jana, S. Hwang, J. Francis, and Oh. Safeshift: Safety-informed distribution shifts for robust trajectory prediction in autonomous driving. arXiv preprint M.Xiv:2309008889 2023.
[42] K.-C. Hsu, K. Leung, Y. Chen, J. F. Fisac, and M. Pavone. Interpretable trajectory prediction for autonomous vehicles viacounterfactual responsibility. In IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2023.
[43] M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), 2003.
[44] R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2005.
[45] Y. Yuan, X. Weng, Y. Ou, and K. Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[46] Y. Chen, P. Karkus, B. Ivanovic, X. Weng, and M. Pavone. Tree-structured policy planning with learned behavior models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[47] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, 2020.
[48] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.
[49] Specifications interbotix x-series locobot documentation. t coosenrobotics www.//nneeebbtiieaalllaaaooo htppr//ooocoooooooooooccccccccce speci catioon. html.
URL https: //cccs///ccc/////ooo
[50] L. E. Dubins. On curves of minimal length with a constraint average curvature, and with pre- scribed initial and terminal positions and tangents. American Journal of Mathematics, 79(3): 497-516, 1957. ISSN 00029327, 10806377. URL http:://ww. jstor. wotp://algg//s/beooooooooo..b..e 2372560.
[51] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), IEEE, 2017.
[52] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Con- ference on Learning Representations, ICLR 2014, 2014.
[53] B. Ivanovic and M. Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IFFF/CVF International Conference on Computer Vision, 2019.
11

---

9 Experiment Implementation Details
9.1 Reward-based Planner & Generative Predictor
Robot Policy: Tree Policy Planning (TPP), Our MPC planner T is tree-structured contingency planner developed by Chen et al. [46]. This reward-based planner is compatible with state-of-the- art conditional behavior prediction models to account for the influence of the robot's s actions when planning. The planner first generates a tree of randomly sampled multi-stage dynamically feasi- ble trajectories, after which each branch is fed into an ego-conditioned predictor Pe to generate scenario tree comprised of multimodal predictions. The optimal robot trajectory is obtained by max- imizing a reward function which in experiments consisted of linear combination of collision costs, lanekeeping costs, control costs, and distance-traveled reward (full details can be found in the original paper [46]). This reward function is fixed throughout the entire robot deployment and human predictor fine-tuning process.
The training loss for our ego-conditioned Agentformer model is a weighted sum of three terms: 1) prediction loss that penalizes incorrect predictions from the ground truth, 2) ego-conditioned collision (EC) loss that penalizes collisions between the predictions and condi- tioned ego-action, 3) other regularization losses such as diversity and KL divergence loss. Our base prediction model is trained on the Nuscenes dataset [47], consisting of 1,000 20-second driving logs collected in Boston and Singapore. During the initial training of the predictor used a learning rate of le-4.
Base Human Predictor.
Dataset Construction. Our deployment dataset D. consisted of N = 96 total closed-loop deploy- ment scenarios each of length 20 seconds. We ranked the ego agent's ss average regret and chose the top p 20-quantile of scenarios to be our set of scenes that suffered from system-level prediction failures, yielding total of 20 high-regret scenarios in Dz. The rest of the 76 scenarios were deemed sufficiently low-regret. We hold out 20% of both high-regret D and low-regret D \ Dr scenes (3 and 17 respectively). The remainder of the deployment data is used for fine-tuning the model.
9.2 Generative VQ-VAE Planner
Dataset Generation. We constructed a dataset of 'Dirain| 10, 000 human-robot trajectories based on a small set of simple rules in order to train our generative planner. The positions of the two goal locations and the robot and human's 's start locations were fixed in all training examples (shown in Figure 3), but the choice of goal g and initial human behavior cue ōh were uniformly distributed. The synthetic human trajectory generation abides by the following rule: a# PC(left) if бμ + < 1/3, (C(right) ) if 8h + > 2/3, PC PC(striight raight) otherwise. Here, PC(.) denotes a proportional controller noisily executing 6-timestep action trajectory that guides the human to the prespecified goal points to the left, right, or straight- -head from the human's ' initial position, and e is zero-mean noise.
The synthetic robot trajectory generation follows a similar process, but conditions on the desired goal IR and the synthetic human behavior a. The robot's action a is noisily executed 6-timestep action trajectory given by a proportional controller that guides the robot to gR. unless this brings the robot in close proximity to the human trajectory = in which case the robot's 's proportional controller guides it to the other goal with 80% probability. This dataset synthesis leads to six different high- level outcomes of the interaction depending on which directions the human and robot travel. Figure 3 shows three of these joint human-robot interaction outcomes.
Generative Planner Architecture. The robot's generative planner Th used an encoder-decoder architecture based on the VAF, which is a variant of the traditional VAE that uses vector quanti- zation to discretize the latent representation into K latent embedding vectors {zx ) The encoder (z 8H g) with learnable parameters a takes Oh and 9R as inputs to encode the input and pro- duce a categorical distribution over latent embedding through vector quantization. We further inject a small amount of zero-mean Gaussian noise into quantized embedding = E {2x { km to encour- age diverse outputs during the joint trajectory decoding phase. This decoder Pa(a R a. 2) with learnable parameters 33 takes sampled latent vector 2 as input (sampled from (a [z] S8. with
additional noise injection) to approximate the joint distribution over actions a R a. In our setting,
12

---

we set K = 6 and initialized the latent embeddings with K-means clustering to encourage each of the K embedding vectors to correspond to one of the six joint human-robot action trajectories.
9.3 Hardware Experiments
The hardware experiments were conducted on an Interbotix LoCoBot [49] equipped with an Intel RealSense D435 RGB-D camera and RPLIDAR A2 360 degree LIDAR. Offline, the robot generated a occupancy map of the environment with SLAM from RGB-D observations. This SLAM map was used to generate global coordinate frame for the human and robot. The human's 's position with 2-D LIDAR scan data and filtering out LIDAR measurements from static objects such as walls. The remaining LIDAR sensor measurements were then averaged to generate a noisy (I; v) points of the human's position relative to the robot that were measured at 12 Hz. These positions were then translated to the global coordinate frame obtained by transforming from the robot's local coordinate frame to the global coordinate frame.
We modeled the human as single integrator with constant velocity where the control action was heading direction. These were extracted by taking a simple moving average of the position measure- ments to obtain position measurements with reduced noise at rate of 2 Hz. The actions each time step were taken to be the angle between the current and previous human position measurement in the global coordinate frame.
10 Fine-tuning Details & Ablations
High-Regret Regret Scenarios Col Severity
Low-Regret Scenarios Col Severity
Fine nieettnnig Data
Col Cost
Regret
Col Cost
Regret
Base Model
10.829
0.848
0.034
0.411
0.051
0.006
AvgReg
4,286 (-60.4%)
0.690 (-18.79))
0.014 (-58.7%)
0.347 (-15,5%)
0.039 9%)
0.005 (-11.1%)
WstReg
4.713 (-56.88))
(-38.7%)
0.020 (-43.3%)
0.565 (+37.456)
(+80.9%)
0.006 (+2.02%)
AvgReg (scol, +div)
14.505 (+34.0 %)
0.960 (+13.2 8b)
0.048 (+399.4 %)
0.437 (+6.4 %3)
0.082 (+60.3 %)
0.006 (+9.1 %)
AvgReg (edix)
(-30.9 %b)
0.654 (-23.0
0.014 (-60.6 %)
0.366
0.0578 (+12.4
0.005 (.6.06 %)
Table 3: Finetuning when ranking with average regret (2nd row, also reported in Table as High-
regret-FT) vs worst single-timestep regret (3rd row). We also ablated the choice of loss function on closed-loop performance by using all of the original loss functions (4th row) and only removing the ego-conditioned collision loss (5th row).
Fine-tuning the Predictor. For fine-tuning, we unfreeze only the last 1.5M parameters of the 17.7M parameter Agentformer model. This corresponds to the last half of the future decoder module [45]. Across all our baselines, we fine-tune the predictor to see the data for 25 epochs at a reduced learning eliminate
rate of 5e-5 held constant throughout the 25 epochs. Furthermore, when fine-tuning the diversity loss and EC collision loss, focusing the model's 's efforts to only accurately predict the ground-truth deployment data. We ablate this choice in 3 on one seed. Due to the large size of the Nuscenes data, we trained on two Nvidia A6000 GPUS with a batchsize of 14 (effective batch size 28) and gradient accumulation of 5 steps.
Regret Formulation and Mitigation Methods. In our formalism from Section 4, we defined the regret on a per-timestep basis. For the experiments, we ranked the scenarios by the average regret in- curred over the 20-second interaction horizon T. However, inspired by the safety analysis literature that considers only the worst safety violation incurred over the time horizon [51], we also experi- mented with ranking scenarios by the worst single- tigle-timestep regret incurred over 7. The closed-loop simulation results are shown in the second row of Table 3.
Furthermore, during predictor fine-tuning described in Section 5, we chose to remove the ego- conditioned collision loss and diversity loss during the fine-tuning. We found that when fine-tuning with the ago-conditioned (EC) collision loss, the predictor was unable to learn from the high-regret data. This because much of the high-regret data was induced by ego collisions with other agents. Thus, the EC collision loss directly conflicts with learning from this new data. The results of fine- tuning with EC collision loss and diversity loss are shown the third row of Table We also observed that training without diversity loss leads to slightly better closed-loop performance. The baseline fine-tuning without EC collision loss but with diversity loss shown in the fourth row of Table 3.
13

---

11 Case Study: System-level Perception Failure Detection via Regret
We demonstrate toy setting where our regret metric is able to identify task-relevant detection failures when using a Conditional Variational Autoencoder (CVAE)-based robot navigation planner.
Problem Setup. A robot is deployed to reach goal point 91 (top flag in Figure 6). It has dynamics x = [ē, y. 0] = [cos 0, sin 0, u] where u is the control input. It is also equipped with sensor Peense(C) that returns if detects an obstacle O, and 0 otherwise. In the absence of obstacle, it should always try to reach However, if an obstacle is blocking the robot's path to it should instead navigate an alternate goal location 92 (right flag in Figure 6).
To train our robot planner, we generated a dataset of [Dtrain| = 10000 trajectories. For simplicity, the robot's s starting state and the locations of 91, 92. and O (if present) were the same across the entire training dataset. In each scene, the demonstrated trajectories were obtained by applying a small amount of Gaussian noise proportional controller guiding the robot to its goal location.
Correctly sees no obstacle Regret: 0.008
Correctly detects obstacle Regret 0,018
Fails detect obstacle Regret: 0,307
Hallucinates obstacle Regret 0.306
Figure 6: Generative Planner: Perception Failure. The robot's training data demonstrated that the robot should aim for the top flag in the absence of an obstacle, but move toward the rightmost flag if an obstacle is detected. The green flag denotes the robot's perception-conditioned planned goal. Orange obstacle denotes that there was actually an obstacle present, and grey obstacle denotes that the robot falsely detected obstacle. Our regret metric correctly flags detection failures and measured nominal scenarios as having low regret.
Inducing Generative Planner Failures. We wanted to identify which robot behaviors (if any) were assigned high regret by our regret metric. To do this, we deployed the robot four times. In two of the scenarios, there was the obstacle (top right, and bottom left of Figure 6). However, in one of those two scenarios, to be incorrect) that prevented the robot from detecting O. We similarly deployed the robot a setting with no obstacle two times but injected perception failure that falsely detected O in of the deployments (bottom right Figure 6).
injected perception failure (i.e.. manually setting the sensor observation
Trajectory Probabilities. To simulate the robot, we sampled 500 trajectories conditioning on Prense (C) and 500 on Psense 0. For each condition, we took the average of the 500 respective trajectories as exemplar for deployment. However, despite the ease of sampling from CVAE, finding the probability
particular sample from a generative model is known to be a difficult problem [52]. To approximate each exemplar trajectory's probability under the ground truth presence/absence of the obstacle, we formed a per-timestep kernel density estimate (KDE) of the conditional likelihood using the 1000 previously sampled trajectories (For more details, see Fig-
14

---

ure 5 of [53]). To compute the trajectory probability, we integrated the KDE for =ó around the most likely and executed action for 5 0.1 at each timestep. The trajectory's regret was computed via our metric as in Equation 3 by averaging over the per-timestep regret.
Results. As in Figure 6, the robot incurs very low regret whenever its sensor accurate. However, in both settings where the sensor failed, the robot incurs a much higher regret. We high- light that nowhere in this formulation (demonstrations
generative planner) was there reward function that determined the robot's 's behavior. Furthermore, this toy example demonstrates that our general calibrated regret metric may have beyond the human-robot interaction setting; for exam- ple, for detecting and mitigating failures in other autonomy modules beyond agent prediction (c.g., perception module) task-aware manner [34].
15


Reviews:
[{"review_id": "vw1W1XDtj6", "reviewer": "Reviewer_t3uw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper “Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures” presents a framework for detecting system-level failures resulting from trajectory prediction errors in human-robot interactive scenarios (e.g., autonomous driving). The method is applied in hindsight (i.e., offline) to identify system-level failures in a large corpus of driving data. The core contribution is a generalized regret metric (GRM) that approximates regret as the difference in likelihood scores between an optimal robot trajectory and the robot trajectory actually taken. The proposed GRM 1) circumvents the need for a reward model to compute a standard regret metric and 2) provides calibrated regret scores across a wide variety of scenarios / contexts likely to be present in self-driving datasets.", "summary_of_recommendation": "In summary, the paper was high quality, clearly written, presented an original idea, which yielded results of significance. I therefore recommend acceptance of this paper.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- Please address the weakness described above? In short, how can we trust the likelihood model to properly assess likelihoods of out-of-distribution scenarios that are likely to result in system-level failures?\n- I encourage the authors to make transparent earlier on that this paper is addressing an offline failure detection problem rather than an online one.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JOtR1CW12l", "reviewer": "Reviewer_HApx", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose characterising ‘system level’ prediction failures through formulating mathematically the idea of regret. This stems from the idea that high-regret interactions are precisely those in which mispredictions caused the robot to make a suboptimal decision in hindsight. Their main area of application is autonomous driving, although I do believe their work could be impactful more broadly. The authors propose their main contributions as the following: (1) Formalise system-level prediction failures via the mathematical notion of regret; (2) Propose a probabilistic generalization of regret that is calibrated and compatible with reward-free robot planners; (3) Demonstrate their approach in closed-loop autonomous driving interactions simulated via the BITS trafficc simulator; (4) Show that fine-tuning state-of-the-art human behavior predictors with high-regret data is a promising avenue for efficiently mitigating system-level prediction failures.", "summary_of_recommendation": "Given the simplistic experimental evaluation, I am leaning towards revise and resubmit due to the additional experiments/details that would be needed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "- The method proposed by the authors has some similarity to another method illustrated in the following paper: https://ieeexplore.ieee.org/abstract/document/9561749/. I believe this should be added to the related work and the authors should outline how their method differs to this one. It would be nice if they can have some form of comparison with this method.\n- In the contributions, I am not sure how you ‘show’ that fine-tuning state-of-the-art human behavior predictors with high-regret data is a promising avenue for efficiently mitigating system-level prediction failures. Could you rethink this contribution and give evaluation metrics how you illustrate this contribution through your experimental evaluation?\n- In the related work you seem to touch a bit on system-level prediction failures. However, an example you give is perception which naturally your method would suffer once deployed in the real world. Could you be a bit more specific and give examples on what system-level prediction failures you would be looking at and how your method addresses these in simulation and potentially in the real world?\n- I do not fully understand the motivation of using the Agentformer model. Is this the only model available or how did you conclude on this model? My understanding is that it is a very simplistic way of modelling human input. \n- While I do appreciate the challenges of real-world deployment of your method and favouring using the simulator for obvious reasons, I am not sure how good your method would overcome the sim2real gap, in addition to the uncertainty with humans and other environmental noise. Could you expand a bit in your limitations how you plan to overcome these challenges? In addition, you mentioned that this work could work for collaborative manipulation - do you have any simulation results for this or more concrete plans how you have/will carry this out?\n- Personally, I think the work would be hard to reproduce since you have not added any details on hyperparameters etc, although you mention section 9 and 10. My supplementary material seems to be only a video, hence I cannot assess well the reproducibility of your work. Could you update this part or link a more detailed appendix?\n\nMinor comments:\n- I found the caption for Figure 2 and the explanation for the illustrative example very repetitive. It might be worth rethinking how you present your work.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MtQPVlX76Z", "reviewer": "Reviewer_ehM7", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel metric based on the concept of regret to identify prediction errors that critically impact robot performance. The key contributions include formalizing system-level prediction failures, introducing a probabilistic generalization of regret, demonstrating the approach in autonomous driving simulations, and showing that fine-tuning with high-regret data can efficiently improve system performance. The results indicate that the method effectively identifies impactful failures and improves model performance with reduced data usage.", "summary_of_recommendation": "The paper presents a novel and effective approach to detecting system-level prediction failures using a regret-based metric. It demonstrates significant improvements in model performance with efficient data usage and offers a detailed and comprehensive evaluation. Despite some limitations regarding parameter sensitivity and real-world applicability, the contributions are substantial and valuable to the field. While simulations in CARLA or similar environments can be acceptable, the paper would be stronger if it demonstrated some level of transfer to real-world scenarios.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "• How sensitive is the performance of the proposed metric to the choice of the p-quantile for detecting high-regret scenarios?\n\n• Can you provide more details on how the simulation data might differ from real-world scenarios, and how this might affect the results?\n\n• Have you considered the computational complexity of the probabilistic generalization of regret, especially in real-time applications?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript received three quality reviews. The reviewers highlighted the following strengths and weaknesses, among many substantial pieces of feedback:
- (+) High-quality manuscript (well-organized text, good figures, good related work)
- (+) Well-motivated methods and good results
- (+) General approach
- (+) Good mathematical foundation
- (+) Extensive evaluation in simulation
- (-) Sensitivity to OOD cases
- (-) Make clearer that this is an offline approach
- (-) Missing hyperparameters and other implementation details, limiting reproducibility
- (-) Evaluation only on simplistic simulation
- (-) Lack of real-world experiments
- (-) Unsustained claim in contribution list
- (-) Unsustained use of a specific model (Agentformer)
- (-) Lack of a discussion on the generalization of the approach to other domains
- (-) Lack of a discussion on dependency on ground truth (simulation) data and sensitivity to the choice of the p-quantile
- (-) Lack of analysis of computation complexity and speed for real-time systems

After rebuttal and discussions among the reviewers, they agreed the quality of the paper improved and several of the concerns were covered.
Decision: Accept