Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures

Authors: kensuke  nakamura
ran  tian
andrea  bajcsy
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 06:53:26

Abstract: Robot decision-making increasingly relies on data-driven human prediction models when operating around people. While these models are known to mispredict in out-of-distribution interactions, only a subset of prediction errors impact downstream robot performance.  
We propose characterizing such ``system-level'' prediction failures via the mathematical notion of regret: high-regret interactions are precisely those in which mispredictions degraded closed-loop robot performance. 
We further introduce a probabilistic generalization of regret that calibrates failure detection across disparate deployment contexts and renders regret compatible with reward-based and reward-free (e.g., generative) planners.  
In simulated autonomous driving interactions, we showcase that our system-level failure metric can automatically mine for closed-loop human-robot interactions that state-of-the-art generative human predictors and robot planners struggle with. 
We further find that the very presence of high-regret data during human predictor fine-tuning is highly predictive of robot re-deployment performance improvements. 
Furthermore, fine-tuning with the informative but significantly smaller high-regret data (23% of deployment data) is competitive with fine-tuning on the full deployment dataset, indicating a promising avenue for efficiently mitigating system-level human-robot interaction failures.

Main paper: 
 introduction: 
 From autonomous cars in cities [1, 2, 3] to tabletop manipulators at home [4, 5, 6] , robots operating around people rely on data-driven trajectory during planning to to anticipate how others will behave. Despite their widespread adoption in robot autonomy pipelines, current human prediction models are still imperfect: they frequently mis-predict when faced with out-of-distribution interactions [7, 8] and are prone to causal misidentification [9] wherein the model "lazily" converges to incorrect correlates in the data. These prediction errors can lead to critical downstream robot system failures, impeding performance at best and compromising safety at worst. Thus, analyzing when and how trajectory prediction errors lead to robot failures is critical for developing runtime monitors, creating safety-focused benchmarks, and incrementally improving the prediction models themselves.While current methods detect interaction data that is dissimilar to the training distribution [10] or leads to highly uncertain or erroneous predictions [11, 12, 13] , these are all fundamentally component-level approaches that detect only when the predictor is faulty in isolation. However, prediction errors at the component-level do not always lead to system-level robot decision-making errors downstream [14] . Consequently, relying on component-level failures to evaluate, monitor, or improve the prediction model is inaccurate and inefficient. For example, consider Figure 1 . Even though the robot's predictions were faulty in all three scenarios, it is only in the latter two where this Figure 1 : All scenarios have component-level prediction failures: mispredicting that parked cars will turn (left), marooned truck will to move (center), and nearby cars will lane change (right). But, only the center and right scenarios have system-level prediction failures which impact robot performance. caused undesirable closed-loop robot behavior: collisions in the center and inefficiency in the right. The core challenge we address is the automatic detection of such system-level prediction failures.Our key insight is that the mathematical notion of regret is a rigorous way to identify systemlevel prediction failures. High-regret interactions are precisely those in which mispredictions caused the robot to make a suboptimal decision in hindsight. In Section 4 we formalize this idea and introduce a probabilistic generalization of regret that no longer depends on explicit reward functions for regret computation, extending its applicability to reward-free planners such as generative models. In Section 6, we extract system-level failures offline from a dataset of collected deployment interactions and compare our approach to alternative failure detection methods with a state-of-theart autonomous driving trajectory predictor and planner, and demonstrate the compatibility of our approach with generative neural planners in a social navigation setting deployed on hardware with on-board sensing. Finally, in Section 7, we conduct a case study where we find that fine-tuning a robot's human behavior predictor on the small amount of high-regret data (23% of the deployment data) is highly predictive of system-level performance improvements at re-deployment. 
related work: 
 Interaction-aware Prediction. Interactive robots commonly use conditional behavior predictions to model the influence between the robot's decisions and other agents' future behavior [15, 16, 4, 17] . More recently, there has also been a shift towards using multi-agent prediction models or generative "foundation models" as planners [18, 19, 20, 21, 22, 23, 24] . However, these models are overwhelmingly evaluated on log-replay data, missing out on the closed-loop interaction effects between agents. This may lead to overly optimistic estimations of how the robot may influence the behavior of others [25] and poorly affect downstream performance. We introduce a systemlevel metric for evaluating interaction-aware predictors that quantifies closed-loop performance of the trajectory predictor. We also show that fine-tuning such a predictor on identified system-level failures leads to significant downstream performance improvements.Component-level Failure Detection. Many prior works have studied the detection of anomalous or out-of-distribution data (see [26] and [27] for surveys) relative to a prediction model's training data: classifying poor predictions based on average displacement error (ADE) [12, 28] , Bayesian filtering [8, 29, 30] , kernel density estimation [31] , uncertainty [11, 32] , conformal prediction [33] , and evidential deep learning [10] . However, these approaches are fundamentally component-level, ignoring the "downstream" effects of errors in the predictor on robot decision-making.System-level Prediction Metrics. Our work contributes to the small-yet-growing area of developing system-level metrics for evaluating components of the autonomy stack [34, 14, 35, 34, 36, 37, 38] . Existing system-level trajectory prediction metrics compare the difference between predicted and incurred costs [37] or weight prediction errors by a task-dependent cost function [38, 39, 40] . In addition to evaluating predictors, system-level metrics can also be useful in the offline setting, wherein datasets are mined for challenging examples. For instance, Stoler et al. [41] mine for safety-critical interactions via heuristic features and Hsu et al. [42] mine via counterfactual responsibility metrics. Our approach is also instantiated in an offline setting, but utilizes our novel regret metric to identify system-level human-robot interaction failures. Furthermore, our method does not require a cost function to detect system-level failures, rendering our method compatible with reward-free planners. 
problem formulation: 
 Agent States, Actions, and Context. Let the robot state be denoted by s R ∈ R n R , and the state of the M human agents in the scene be denoted by s Hi ∈ R ni , i ∈ {1, . . . , M }. For example, for an autonomous driving application domain, each agent's state can be represented as position, heading, and velocity. All agents' states evolve according to their respective control actions: a R ∈ R m R for the robot (e.g., acceleration and turning rate) and a Hi ∈ R mi for the M humans, where n R and m R are the dimensions of the robot state and action and are similarly defined for each human. For shorthand, we denote the joint human-robot state at any time t as s t = [s R t , s H1 t , . . . , s Human Prediction & Robot Planning. We assume the robot has access to a pre-trained human behavior prediction model P θ with parameters θ capable of conditional generations [16, 18] . Mathematically, this predictor is:EQUATIONThe model can generate T -horizon predictions of all M humans in the scene, a H1:H M t := [a H1 t , ..., a H M t], conditioned on a robot action trajectory a R t , scene context C, and current1 humanrobot state s t . We also use the robot's task-driven planner:EQUATIONwith parameters ϕ. This planner generates a best-effort robot plan a R t with respect to the current joint state s t , scene context C and predictions, P θ . Within our framework, the robot's planner could be an optimization-based planner that uses a reward function (where ϕ are feature weights of a pre-specified reward function) or a neural planner (where ϕ are the pre-trained neural network weights). At deployment time, the robot's planning model can be re-queried given new observations and predictions in a receding horizon fashion. We assume that the planner is optimal under the given reward function or the available data for planner design.Problem Statement. We deploy the robot for T time steps in N scenarios (e.g., different initial conditions, environments) and collect an aggregate deployment dataset D = {(ŝ, âR , âH1:H M ) n } N n=1 consisting of observed state and action trajectories for the robot and M humans. Importantly, this observed data exhibits how real interactions between the humans and robot occur, and are a function of the robot's predictive human model used during planning. We seek to automatically identify a subset of deployment data, D F ⊂ D, that captures system-level prediction failures: deployment interactions wherein prediction failures significantly impacted closed-loop robot performance. 
a regret metric for detecting system-level trajectory prediction failures: 
 Canonically, regret measures the reward (R ϕ ) difference between the optimal action the robot could have taken in hindsight (a R ) and the executed action (â R ) the robot took under uncertainty [43] :Reg t (d) := max a R t R ϕ (a R t , âH1:H M t , ŝt , C) -R ϕ (â R t , âH1:H M t , ŝt , C), d ∈ D.If we were to rank all the interactions in D by the robot's regret, we could identify those with the highest regret as a means of building our system-level failure dataset, D F . While regret is a principled measure of system-level failures, directly using rewards during regret computation may not always be desirable. First, not all robot planners use reward functions-for example, generative planners fit joint distributions to multi-agent behavior [18] and choose trajectories via their likelihood rather than an explicit reward. This immediately makes canonical regret impossible to evaluate. Second, the canonical regret definition assumes that the same difference in reward values in different deployment contexts are fairly comparable. Unfortunately, in practice, when robots are deployed in a wide variety of scenarios and contexts C, it is difficult to have such a perfectly calibrated reward. We illustrate this technical point with an example from our simulation experiments (details in Section 5). Illustrative Example. Consider Figure 2 , where the same robot (blue car) is deployed in two different contexts: one where the robot drives behind a stopped truck (C 1 ) and the other where it navigates an intersection (C 2 ). The robot's reward is a tuned linear combination of control effort, collision penalty, road progress, and lane-keeping.In scenario C 1 , the robot mispredicts that the stopped truck will accelerate; the robot initiates a lane change and collides. In scenario C 2 , the robot aggressively swerves to avoid nearby traffic. Because of the absolute reward difference between the best and the executed action (right column, Figure 2 ), canonical regret regards both scenarios as equally poor: C 1 has a regret of 11.4 in C 2 has 11.7. While the robot could have taken a more optimal action in both scenarios, canonical regret is unable to identify that the robot's decision in C 1 is more severely suboptimal than the decision in C 2 .A Generalized Regret Metric. To address this calibration issue we propose that instead of looking at the absolute reward of a decision, we consider the likelihood of it. This probabilistic interpretation normalizes the quality of a decision relative to the context-dependent behavior distribution and is a principled way to place all decision comparisons on the same scale-a value between zero and one. Specifically, let P ϕ (a R | âH1:H M , ŝ, C) be a likelihood model that yields the counterfactual probability of any candidate robot action sequence (a R ) given the observed actions of the other human agents (â H1:H M ) and the observed joint state trajectory (ŝ) in that context C. Constructing this likelihood model only requires access to the robot's planner; hence, it shares the same parameters as the robot used for decision-making (ϕ). For example, if the robot is using a reward-based planner, then this probability distribution is shaped according to the same reward weights and features that the robot used for decision-making (see example in Equation 4). Additionally, this generalization to probability-space allows us to calculate regret for generative neural planners, where the likelihood model shares the same weights as the neural planner. For any human-robot deployment interaction d := (ŝ, âR , âH1:H M ) ∈ D, we quantify the generalized regret of a robot's action at timestep t as:EQUATIONIllustrative Example. Consider the reward-based planning example from Figure 2 . We model P ϕ using the Luce-Shepard choice rule [44] , which places exponentially more probability on robot decisions that are high-reward. Our generalized regret metric results in an intuitive failure separation where canonical regret failed: C 1 has a higher regret of 0.56 compared to 0.34 in C 2 . 
simulation & hardware experimental setup: 
 We first instantiate our metric in the autonomous driving setting. Our prediction model P θ is an egoconditioned Agentformer model [45] which takes as input a candidate robot (i.e., ego) trajectory a R t , a history of all the vehicles in the scene s t-h:t , and the map information C to output a prediction of the behavior of all human (i.e., non-ego) vehicles a H1:H M t . Our planner π ϕ is an off-the-shelf reward-based MPC planner [46] who's parameters ϕ are weights of a hand-tuned reward function consisting of lane progress, lane-keeping, collision cost, and control cost (see Appendix 9.1). We pretrain P θ on the nuScenes training split [47] and we obtain our closed-loop deployment dataset D consisting of 96 simulated 20-second scenarios from the BITS simulator [48] where each scenario's map and initial agent history are initialized from the nuScenes validation split.We further instantiate our approach for generative robot planners on an Interbotix LoCoBot [49] , shown in Figure 3 . The LoCoBot has an Intel D435 RGB-D camera and RPLIDAR A2 2-D Lidar Figure 3 : Left: The robot correctly predicts the human will block its goal and proceeds straight. Middle: The robot incorrectly predicts that the human will walk straight ahead, however it is able to reach its goal despite the misprediction. Right: The robot mispredicts that the human will block its goal and proceeds straight, colliding with the human who also walked straight ahead. The robot's executed trajectory is unlikely conditioned on the human's true actions and is assigned high regret. mounted on a Kobuki mobile base. We model the robot as a Dubins' car [50] which must reach a goal position g R unless it would cause a collision with a nearby freely moving pedestrian, in which case it navigates to a backup location (Figure 3 ). For constructing the robot's generative planner π ϕ , we model the joint human-robot state as the states of both agents along with a low-dimensional behavior cue from the human δ H (e.g., eye-gaze, head tilt) that leaks information about the human's future trajectory. We take s ≡ δ H and fix all other initial conditions between deployments. The context of the scene is the intended goal of the robot C ≡ g R . The generative planner π ϕ is an encoderdecoder architecture based on the vector-quantized variational autoencoder (VQ-VAE). It is trained with a dataset of 10,000 simulated joint human-robot trajectories (a R , a H ) generated via simple rules (described in Appendix). The encoder P α (z | δ H , g R ), with learnable parameters α, encodes the human observation δ H and the original goal g R and produces a categorical distribution over latent embedding through vector quantization. The decoder P β (a R , a H | z), with learnable parameters β, takes the latent vector z ∼ P α (• | δ H , g R ) and approximates the joint distribution over actions a R , a H . Implementation details on datasets and hyperparameters are included in Appendix 9.2.Generalized Regret Computation: Reward-based Planner. Our likelihood model follows from the Luce-Shepard choice rule [44] where the parameters ϕ denote the weights of the reward function:EQUATIONwhereη := 1/ āR exp{R ϕ (ā R , âH1:H M , ŝ, C)}. For each timestep in deployment data d ∈ Dwhere the robot originally re-planned, we re-compute the reward of each candidate ego action with respect to the ground truth behavior of the other agents, instead of predictions. With this we compute the hindsight likelihood of each candidate ego action and obtain the scene's regret: t Reg t (d).Generalized Regret Computation: Generative Planner. In our hardware experiments, the human's positions were recorded as (x, y) planar positions at 12 Hz using the robot's LIDAR, and transformed to a global coordinate frame of a precomputed SLAM map. The behavioral cue δ H is obtained by tracking the position of the human and calculating the angle between timesteps, which is given to the planner to generate a 6-timestep action trajectory âR which is executed at 2 Hz. After each action is executed, we recorded smoothed measurements of the human's trajectory by taking a simple moving average of the human position and measuring the angles between subsequent positions to obtain âH . We construct the likelihood model using Bayes' theorem where we first form a kernel density estimate of P ϕ (â R | âH , δ H , g R ) by sampling from the the planner 250 times and approximate the conditional probability of âR , âH by integrating the KDE around a small neighborhood of the ground-truth joint action trajectory. The conditional probability of âH is approximated by marginalizing over the space of possible a R . Leveraging the approximate probabilities, we com-pute an estimate of the counterfactual probability for any âR :P ϕ (â R | âH , δ H , g R ) = z P β (â R , âH | z)Pα(z | δ H , g R ) āR z P β (â H , āR , | z)Pα(z | δ H , g R ).This is a particular instantiation of P ϕ where the planner and predictor both share parameters (i.e., ϕ ≡ θ) given by α and β. We repeat this process to evaluate the maximum probability action trajectory2 , giving us both terms necessary for our generalized regret computation Reg t (•) in Equation 3.6 Experimental Results: Detecting System-Level Prediction Failures We first quantitatively and qualitatively compare scenarios extracted by our generalized regret metric (GRM) against three alternatives: average displacement error (ADE), a component-level metric that measures the L 2 error between the true and predicted trajectory, the system-level metric of Farid et al. [37] (TRFD), and the canonical variant of our system-level regret metric (RM).We apply each metric to the initial closed-loop simulated driving deployment data as described in Section 5. For GRM, ADE, and RM, we construct the failure dataset as the top 20-quantile scenes 3 (i.e., 20 scenarios with 20-second long interactions) as scored by the metric, yielding D GRM F , D ADE F and D RM F . TRFD is unique in that it only assigns binary labels if the scenario is a system-level failure. The pairwise overlap between D F from each metric is visualized in Figure 4 .ADE is a component-level metric and has only 35% overlap with scenes identified by the two regret metrics (GRM, RM). Qualitatively, the scenarios with high displacement error tended to have many agents or a single agent that the baseline Agentformer model largely mispredicts. An example of a scenario with high ADE but not flagged by our two regret metrics is shown in Figure 4 . The ADE in this scenario is dominated by the robot consistently mispredicting a faraway vehicle to change lanes. This indicates that component-level ADE may attend to irrelevant prediction failures instead of system-level failures. TRFD is a system-level metric that flags an interaction as a failure if the incurred robot reward is in the bottom p-quantile of the robot's reward distribution predicted during planning. Since this metric relies on a predicted reward distribution, it is fundamentally linked to the quality of the robot's predictions of other agents' behavior. We find that TRFD assigns every scenario as being anomalous (Figure 4 ). We hypothesize the distribution shift between the nuScenes data (that the baseline Agentformer model was trained on) and the simulated human behavior of the BITS agents causes the predicted reward distribution to be extremely misaligned with the realized reward, causing all interactions to be flagged as failures. Instead of comparing the realized outcomes with the predicted reward distribution, GRM and RM compares against counterfactual ground-truth information, making them agnostic to the base predictor quality.RM is our absolute-reward variant of likelihood-based regret metric, GRM. Because they are both grounded in the idea of regret, RM has 95% overlap with GRM. The scenario uniquely identified by each metric is shown in Figure 4 . Notably, GRM identifies an additional scenario where the ego vehicle drives through stopped traffic (top, Figure 4 ), whereas RM identifies an instance of the robot driving near unmoving vehicles and showing slightly conservative behavior (middle, Figure 4 ). Although downstream rewards are a common evaluation criteria for task-aware trajectory forecasting [37, 38] , these results show that reward alone suffers from calibration challenges and may overlook pertinent system-level failures. The qualitative results of our GRM approach in the real-world generative planning setting are shown in Figure 3 . Importantly, no other system-level metrics (TRFD, RM) generalize to generative planners because they require explicit reward functions. We find that our approach only rates interactions as high-regret (0.307 on a scale from 0 to 1) when mispredictions lead to unexpectedly close proximity interactions between the robot and human (right, Figure 3 ). In both nominal interactions (left, Figure 3 ) and interactions where mispredictions are irrelevant for robot performance (middle, Figure 3 ), the deployment regret is close to zero (0.078 and 0.053). Overall, our new regret metric captures our motivating intuition on system-level prediction failures without the need for manually specifying an evaluative rewardfunction for either the high-level or low-level robot and human behaviors. 
case study: mitigating prediction failures via high-regret fine-tuning: 
 We demonstrate one potential use for system-level failure data by fine-tuning the Agentformer model on high-regret interactions. We first hold out 20 (comprised of 17 low-regret and 3 high-regret scenarios) of the 96 deployment scenarios for evaluation and leave the rest (76) as potential data for fine-tuning. We compare the closed-loop simulation (Table 2 ) and open-loop prediction (Table 1 ) performance of Agentformer fine-tuned on four different data subsets. Low-regret-FT uses the 20 scenarios with the lowest regret, Random-FT uses 20 random scenarios from the 76 scenarios available for fine-tuning, High-regret-FT uses the 20 scenarios with the highest regret, and All-FT is a privileged baseline with access to all 76 scenarios. During fine-tuning we unfreeze only the final 1.5M parameters of the 17.7M parameter Agentformer model, and we examine the results from fine-tuning the model on each dataset for 3 random seeds. Details and ablations in Appendix 10.Takeaway 1: High-Regret-FT improves log-replay prediction accuracy in high-regret scenarios, without degrading performance on low-regret scenarios and the original nuScenes validation set.We first evaluate the predictors "open-loop" on log-replay data from three different validation datasets: the original nuScenes validation data, the 3 high-regret scenes in the deployment validation dataset, and the 17 low-regret scenes in the deployment validation dataset. We report the Average Displacement Error (ADE) and Final Displacement Error (FDE) for each baseline's best performing seed in Table 1 . The base predictor doubles its ADE and FDE on the simulated deployment data due to a distribution shift in the driving behavior between nuScenes and simulation. Intuitively, High-Regret-FT has the most ADE and FDE improvement on high-regret validation data. We hypothesize that the improved open-loop prediction accuracy on low-regret scenes is because the high-regret validation data leaks enough information about the general distribution shift between the nuScenes and the BITS simulation to enable improvements on nominal interaction data as well. While All-FT achieves the best prediction accuracy across the board, it is important to note that this is an information-advantaged baseline: it has the ability to learn from both system-level and non-system-level failures the deployment data. However, High-Regret-FT is much more data efficient to train, rivaling the performance of All-FT despite using 77% less data. As the deployment dataset D grows and training time and costs increase along with it, this can be a promising avenue to alleviate these challenges while still achieving high predictive performance. Finally, the least improvement over the base model is exhibited by Low-Regret-FT and followed by Random-FT which demonstrates the quantitative value of learning from high-regret data.Takeaway 2: Including high-regret data during fine-tuning improves closed-loop performance.Our first evaluation metric for measuring closed-loop re-deployment performance differences is Total Collision Cost, which is the total cost the ego incurred from collisions. Second, to disentangle severe but infrequent collisions from frequent clipping / grazing we measure Collision Sever-ity= Total Collision Cost #Frames in Collision . Finally, we measure average Regret as defined in Section 4. We report average closed-loop performance for each method across three seeds of fine-tuning in 7.489 (± 0.408) 0.794 (± 0.059) 0.019 (± 0.002) 0.670 (± 0.206) 0.354 (± 0.493) 0.009 (± 0.000) Random-FT 7.209 (± 0.883) 0.743 (± 0.064) 0.016 (± 0.003) 0.495 (± 0.213) 0.076 (± 0.054) 0.006 (± 0.000) High-regret-FT 3.763 (± 0.965) 0.588 (± 0.101) 0.016 (± 0.003) 0.397 (± 0.814) 0.062 (± 0.031) 0.006 (± 0.001) All-FT 1.176 (± 0.775) 0.377 (± 0.067) 0.013 (± 0.002 ) 0.486 (± 0.131) 0.060 (± 0.030) 0.007 (± 0.001) Table 2 : Closed-Loop Re-Deployment Performance. Average robot performance metrics for each fine-tuned predictor (standard deviation over three seeds of fine-tuning in parentheses). Two main columns are evaluations on high-regret and low-regret held-out validation scenes. Italics-bold indicates best, and bold indicates second best. Across all metrics and settings, High-regret-FT consistently improves robot performance, and is competitive with the privileged All-FT approach. We find that High-Regret-FT decreases collision cost, severity, and average regret compared to the base model across both high-and low-regret scenes. As hypothesized, most closedloop performance gains are observed in the high-regret scenes, reducing robot regret by -53% and collision cost by -65% (shown in Table 2 ). By being able to see all the deployment data, All-FT observes the highest performance gains in highregret scenes but shows slight (but not statistically significant) performance degradation in low-regret scenes. For each baseline, the degradations from the base model are within 1 standard deviation (except for Low-Regret-FT). In contrast, the improvement of High-Regret-FT is more than one standard deviation better than the performance of Low-Regret-FT and Random-FT with respect to the collision-based metrics. The closed-loop performance gains resulting from fine-tuning on high-regret data is shown qualitatively in Figure 5 . We find that in the high-regret scenes, the open-loop performance (Table 1 ) is positively correlated with the closed-loop planner performance (Table 2 ). We hypothesize that this is because these scenes are much more challenging (e.g., highly interactive), encode more nuanced human behavior, and affect the planner the most; thus, the prediction improvements in such scenarios indicate that P θ has learned to predict challenging interactions better, naturally improving the closed-loop planning performance. 
limitations & future work: 
 One limitation of our system-level failure detection method is the need for a designer to set p when detecting scenarios with the top p-quantile regret. Poorly tuning this p could result in classifying benign scenarios as anomalous or overlooking true system-level failures. In the generative planner setting, extremely out-of-distribution interaction data (e.g., not in the support of the training distribution) could lead to poor regret estimates. To remedy this, we hypothesize that our approach can be complemented by an anomaly detector that can detect highly out-of-distribution data during deployment. Furthermore, our hardware experiments underscored how the logged data collected by the robot during deployment (e.g., states of agents) must be sufficiently high quality to compute accurate regret. Failures in detection/tracking from perception modules in real-world deployments could corrupt the data used to calculate regret and misidentify system-level failures.While we instantiated our approach in simulated and hardware robot navigation interactions, collaborative manipulation [4] ) is an exciting future direction. Finally, future work should also investigate alternative uses for the datasets constructed with our regret metric, e.g., to inform runtime monitors that can anticipate system-level prediction failures online.9 Experiment Implementation Details 
reward-based planner & generative predictor: 
 Robot Policy: Tree Policy Planning (TPP). Our MPC planner π ϕ is a tree-structured contingency planner developed by Chen et al. [46] . This reward-based planner is compatible with state-of-theart conditional behavior prediction models to account for the influence of the robot's actions when planning. The planner first generates a tree of randomly sampled multi-stage dynamically feasible trajectories, after which each branch is fed into an ego-conditioned predictor P θ to generate a scenario tree comprised of multimodal predictions. The optimal robot trajectory is obtained by maximizing a reward function which in our experiments consisted of a linear combination of collision costs, lanekeeping costs, control costs, and distance-traveled reward (full details can be found in the original paper [46] ). This reward function is fixed throughout the entire robot deployment and human predictor fine-tuning process.Base Human Predictor. The training loss for our ego-conditioned Agentformer model is a weighted sum of three terms: 1) prediction loss that penalizes incorrect predictions from the ground truth, 2) ego-conditioned collision (EC) loss that penalizes collisions between the predictions and conditioned ego-action. 3) other regularization losses such as diversity and KL divergence loss. Our base prediction model is trained on the Nuscenes dataset [47] , consisting of 1,000 20-second driving logs collected in Boston and Singapore. During the initial training of the predictor we used a learning rate of 1e-4.Dataset Construction. Our deployment dataset D, consisted of N = 96 total closed-loop deployment scenarios each of length 20 seconds. We ranked the ego agent's average regret and chose the top p = 20-quantile of scenarios to be our set of scenes that suffered from system-level prediction failures, yielding a total of 20 high-regret scenarios in D F . The rest of the 76 scenarios were deemed sufficiently low-regret. We hold out 20% of both high-regret D F and low-regret D \ D F scenes (3 and 17 respectively). The remainder of the deployment data is used for fine-tuning the model.Dataset Generation. We constructed a dataset of |D train | = 10, 000 human-robot trajectories based on a small set of simple rules in order to train our generative planner. The positions of the two goal locations and the robot and human's start locations were fixed in all training examples (shown in Figure 3 ), but the choice of goal g and initial human behavior cue δ H were uniformly distributed. The synthetic human trajectory generation abides by the following rule:a H = P C(left) if δ H + ϵ < 1/3, P C(right) if δ H +ϵ > 2/3, P C(straight) otherwise.Here, P C(•) denotes a proportional controller noisily executing a 6-timestep action trajectory that guides the human to the prespecified goal points to the left, right, or straight-ahead from the human's initial position, and ϵ is zero-mean noise.The synthetic robot trajectory generation follows a similar process, but conditions on the desired goal g R and the synthetic human behavior a H . The robot's action a R is a noisily executed 6-timestep action trajectory given by a proportional controller that guides the robot to g R , unless this brings the robot in close proximity to the human trajectory a H in which case the robot's proportional controller guides it to the other goal with 80% probability. This dataset synthesis leads to six different highlevel outcomes of the interaction depending on which directions the human and robot travel. Figure 3 shows three of these joint human-robot interaction outcomes.Generative Planner Architecture. The robot's generative planner π ϕ used an encoder-decoder architecture based on the VQ-VAE, which is a variant of the traditional VAE that uses vector quantization to discretize the latent representation into K latent embedding vectors {z k } K k=1 . The encoder P α (z | δ H , g) with learnable parameters α takes δ H and g R as inputs to encode the input and produce a categorical distribution over latent embedding through vector quantization. We further inject a small amount of zero-mean Gaussian noise into quantized embedding z ∈ {z k } K k=1 to encourage diverse outputs during the joint trajectory decoding phase. This decoder P β (a R , a H | z) with learnable parameters β takes a sampled latent vector z as input (sampled from P α (z | δ H , g) with additional noise injection) to approximate the joint distribution over actions a R , a H . In our setting, 
case study: system-level perception failure detection via regret: 
 We demonstrate a toy setting where our regret metric is able to identify task-relevant detection failures when using a Conditional Variational Autoencoder (CVAE)-based robot navigation planner. Problem Setup. A robot is deployed to reach a goal point g 1 (top flag in Figure 6 ). It has dynamics ẋ = [ ẋ, ẏ, θ] ⊤ = [cos θ, sin θ, u] ⊤ , where u is the control input. It is also equipped with a sensor P sense (C) that returns 1 if it detects an obstacle O, and 0 otherwise. In the absence of an obstacle, it should always try to reach g 1 . However, if an obstacle is blocking the robot's path to g 1 , it should instead navigate to an alternate goal location g 2 (right flag in Figure 6 ).To train our robot planner, we generated a dataset of |D train | = 10000 trajectories. For simplicity, the robot's starting state and the locations of g 1 , g 2 , and O (if present) were the same across the entire training dataset. In each scene, the demonstrated trajectories were obtained by applying a small amount of Gaussian noise to a proportional controller guiding the robot to its goal location. Inducing Generative Planner Failures. We wanted to identify which robot behaviors (if any) were assigned high regret by our regret metric. To do this, we deployed the robot four times. In two of the scenarios, there was the obstacle O (top right, and bottom left of Figure 6 ). However, in one of those two scenarios, we injected a perception failure (i.e., manually setting the sensor observation to be incorrect) that prevented the robot from detecting O. We similarly deployed the robot in a setting with no obstacle two times but injected a perception failure that falsely detected O in one of the deployments (bottom right in Figure 6 ).Trajectory Probabilities. To simulate the robot, we sampled 500 trajectories conditioning on P sense (C) = 1 and 500 on P sense (C) = 0. For each condition, we took the average of the 500 respective trajectories as an exemplar for deployment. However, despite the ease of sampling from the CVAE, finding the probability of a particular sample from a generative model is known to be a difficult problem [52] . To approximate each exemplar trajectory's probability under the ground truth presence/absence of the obstacle, we formed a per-timestep kernel density estimate (KDE) of the conditional likelihood using the 1000 previously sampled trajectories (For more details, see Fig-ure 5 of [53] ). To compute the trajectory probability, we integrated the KDE for ±δ around the most likely and executed action for δ = 0.1 at each timestep. The trajectory's regret was computed via our metric as in Equation 3 by averaging over the per-timestep regret.Results. As seen in Figure 6 , the robot incurs very low regret whenever its sensor was accurate. However, in both settings where the sensor failed, the robot incurs a much higher regret. We highlight that nowhere in this formulation (demonstrations nor generative planner) was there a reward function that determined the robot's behavior. Furthermore, this toy example demonstrates that our general calibrated regret metric may have use beyond the human-robot interaction setting; for example, for detecting and mitigating failures in other autonomy modules beyond agent prediction (e.g., perception module) in a task-aware manner [34] .The predictor P θ can also accept a state history, s t-h:t where h is the number of previous timesteps.We find this maximizing a R by searching over a prespecified set of action trajectories.The choice of p is a design decision. We chose p = 20 due to the small size of the deployment dataset. 
 
 Appendix:  


Decision: The manuscript received three quality reviews. The reviewers highlighted the following strengths and weaknesses, among many substantial pieces of feedback:
- (+) High-quality manuscript (well-organized text, good figures, good related work)
- (+) Well-motivated methods and good results
- (+) General approach
- (+) Good mathematical foundation
- (+) Extensive evaluation in simulation
- (-) Sensitivity to OOD cases
- (-) Make clearer that this is an offline approach
- (-) Missing hyperparameters and other implementation details, limiting reproducibility
- (-) Evaluation only on simplistic simulation
- (-) Lack of real-world experiments
- (-) Unsustained claim in contribution list
- (-) Unsustained use of a specific model (Agentformer)
- (-) Lack of a discussion on the generalization of the approach to other domains
- (-) Lack of a discussion on dependency on ground truth (simulation) data and sensitivity to the choice of the p-quantile
- (-) Lack of analysis of computation complexity and speed for real-time systems

After rebuttal and discussions among the reviewers, they agreed the quality of the paper improved and several of the concerns were covered.

Reviews: review: title: title: Review 1
paper_summary: paper_summary: The paper “Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures” presents a framework for detecting system-level failures resulting from trajectory prediction errors in human-robot interactive scenarios (e.g., autonomous driving). The method is applied in hindsight (i.e., offline) to identify system-level failures in a large corpus of driving data. The core contribution is a generalized regret metric (GRM) that approximates regret as the difference in likelihood scores between an optimal robot trajectory and the robot trajectory actually taken. The proposed GRM 1) circumvents the need for a reward model to compute a standard regret metric and 2) provides calibrated regret scores across a wide variety of scenarios / contexts likely to be present in self-driving datasets.
main_review: main_review: Quality / Clarity: Overall, this is a high quality paper. It is well organized with figures that illustrate the core concepts of the paper. The introduction is compelling. The motivation and description of the method, the experiments, and the results are clear. Finally, the related work is concise but distinguishes the contribution of this paper from previous works on this topic.

Originality / Significance: The use of a probabilistic regret metric for identifying system-level failures is an original and useful idea. As such, the results are convincing and do hold significance toward increasing the robustness of mature autonomous systems in the real world.

Summary of strengths:
- The failure detection method (based on generalized regret) is general and as such is demonstrated to work for both reward-based planners and neural implicit robot planners. 
- Experiments validate the downstream effectiveness of the regret metric, demonstrating an increase in closed-loop re-deployment performance after fine-tuning on classified high-regret trajectories.
- The reviewer appreciates the effort put into the organization and presentation of this paper, which makes it easy to extract the core contributions and their significance.

Summary of weaknesses:
- There are cases in which system-level failures result from never-before-seen, out-of-distribution scenarios. In these cases, it would appear that the generalized regret metric is susceptible to erroneous regret estimates (when a reward model is not available), because likelihood scores can be unreliable w.r.t. OOD inputs. Thus, a preferable robot trajectory can be assigned a low likelihood and vice versa, potentially omitting high-regret, OOD scenarios.
questions: questions: - Please address the weakness described above? In short, how can we trust the likelihood model to properly assess likelihoods of out-of-distribution scenarios that are likely to result in system-level failures?
- I encourage the authors to make transparent earlier on that this paper is addressing an offline failure detection problem rather than an online one.
limitations: limitations: 3
review_summary: review_summary: In summary, the paper was high quality, clearly written, presented an original idea, which yielded results of significance. I therefore recommend acceptance of this paper.
title: title: Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures
paper_summary: paper_summary: The authors propose characterising ‘system level’ prediction failures through formulating mathematically the idea of regret. This stems from the idea that high-regret interactions are precisely those in which mispredictions caused the robot to make a suboptimal decision in hindsight. Their main area of application is autonomous driving, although I do believe their work could be impactful more broadly. The authors propose their main contributions as the following: (1) Formalise system-level prediction failures via the mathematical notion of regret; (2) Propose a probabilistic generalization of regret that is calibrated and compatible with reward-free robot planners; (3) Demonstrate their approach in closed-loop autonomous driving interactions simulated via the BITS trafficc simulator; (4) Show that fine-tuning state-of-the-art human behavior predictors with high-regret data is a promising avenue for efficiently mitigating system-level prediction failures.
main_review: main_review: The research presented a mathematical formulation of a regret metric, with a particular focus on autonomous driving. The presented work outlines a nice contribution towards robotic systems capable of improving failure recovery. The neat mathematical formulation of this method could potentially be strong enough to have application to other safety-critical domains. The quality of the proposed method is evidenced through the mathematical formulation and simplified simulation environments. There is evidence of a successful outcome for eg through the case study. Their clarity of the methods could be improved by adding more detail on the implementation and hyperparameters (missing appendix), including github repository/documentation for replicating the experiments once the work is accepted for submission. The main strength of this work is the formulation of the problem using a regret measure (although it would be nice to differentiate more explicitly to other probabilistic introspective methods) and its main weakness is the simplified simulated environments and lack of real-world experiments.
questions: questions: - The method proposed by the authors has some similarity to another method illustrated in the following paper: https://ieeexplore.ieee.org/abstract/document/9561749/. I believe this should be added to the related work and the authors should outline how their method differs to this one. It would be nice if they can have some form of comparison with this method.
- In the contributions, I am not sure how you ‘show’ that fine-tuning state-of-the-art human behavior predictors with high-regret data is a promising avenue for efficiently mitigating system-level prediction failures. Could you rethink this contribution and give evaluation metrics how you illustrate this contribution through your experimental evaluation?
- In the related work you seem to touch a bit on system-level prediction failures. However, an example you give is perception which naturally your method would suffer once deployed in the real world. Could you be a bit more specific and give examples on what system-level prediction failures you would be looking at and how your method addresses these in simulation and potentially in the real world?
- I do not fully understand the motivation of using the Agentformer model. Is this the only model available or how did you conclude on this model? My understanding is that it is a very simplistic way of modelling human input. 
- While I do appreciate the challenges of real-world deployment of your method and favouring using the simulator for obvious reasons, I am not sure how good your method would overcome the sim2real gap, in addition to the uncertainty with humans and other environmental noise. Could you expand a bit in your limitations how you plan to overcome these challenges? In addition, you mentioned that this work could work for collaborative manipulation - do you have any simulation results for this or more concrete plans how you have/will carry this out?
- Personally, I think the work would be hard to reproduce since you have not added any details on hyperparameters etc, although you mention section 9 and 10. My supplementary material seems to be only a video, hence I cannot assess well the reproducibility of your work. Could you update this part or link a more detailed appendix?

Minor comments:
- I found the caption for Figure 2 and the explanation for the illustrative example very repetitive. It might be worth rethinking how you present your work.
limitations: limitations: 1
review_summary: review_summary: Given the simplistic experimental evaluation, I am leaning towards revise and resubmit due to the additional experiments/details that would be needed.
title: title: Review of "Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures"
paper_summary: paper_summary: The paper proposes a novel metric based on the concept of regret to identify prediction errors that critically impact robot performance. The key contributions include formalizing system-level prediction failures, introducing a probabilistic generalization of regret, demonstrating the approach in autonomous driving simulations, and showing that fine-tuning with high-regret data can efficiently improve system performance. The results indicate that the method effectively identifies impactful failures and improves model performance with reduced data usage.
main_review: main_review: The paper introduces a unique approach to identifying impactful prediction failures using the concept of regret, marking a significant theoretical contribution. Demonstrating the method in the practical setting of autonomous driving showcases its real-world applicability. It highlights that fine-tuning models on high-regret data, which constitutes a smaller portion of the dataset, can achieve performance improvements comparable to using the entire dataset. The evaluation is comprehensive, with both qualitative and quantitative comparisons to other failure detection metrics, underscoring the superiority of the proposed approach. The detailed methodology, including the experimental setup, models used, and the process of calculating generalized regret, ensures reproducibility. However, the method's requirement for setting a p-quantile for detecting failures could lead to misclassification if not well-tuned. The reliance on accurate simulation data may not always reflect real-world scenarios due to potential perception module inaccuracies. While the approach is demonstrated in autonomous driving, its applicability to other domains is not explored in-depth, and the probabilistic generalization of regret may add computational complexity, potentially limiting its real-time application. Overall, the paper presents a significant advancement in failure detection for human-robot interactions, with promising results in improving model performance efficiently. Future work should address parameter sensitivity, explore real-world applications, and extend the approach to other domains.

Cons:

•  Lack of Physical Robot Testing: The paper does not demonstrate the proposed method on physical robots. While simulations are valuable, testing on physical robots is a key expectation for CoRL submissions.
questions: questions: • How sensitive is the performance of the proposed metric to the choice of the p-quantile for detecting high-regret scenarios?

• Can you provide more details on how the simulation data might differ from real-world scenarios, and how this might affect the results?

• Have you considered the computational complexity of the probabilistic generalization of regret, especially in real-time applications?
limitations: limitations: 2
review_summary: review_summary: The paper presents a novel and effective approach to detecting system-level prediction failures using a regret-based metric. It demonstrates significant improvements in model performance with efficient data usage and offers a detailed and comprehensive evaluation. Despite some limitations regarding parameter sensitivity and real-world applicability, the contributions are substantial and valuable to the field. While simulations in CARLA or similar environments can be acceptable, the paper would be stronger if it demonstrated some level of transfer to real-world scenarios.