Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress

Authors: k  wang
k  rao
k  pertsch
k  hausman
k  go
k  gopalakrishnan
k  goldberg
k  byrne
k  oslund
k  kawaharazuka
k  black
k  lin
k  zhang
k  ehsani
k  lekkala
k  ellis
k  rana
k  srinivasan
k  fang
k p singh
k.-h  zeng
k  hatch
k  hsu
l  itti
l y chen
l  pinto
l  fei-fei
l  tan
l j fan
l  ott
l  lee
l  weihs
m  chen
m  lepert
m  memmel
m  tomizuka
m  itkina
m g castro
m  spero
m  du
m  ahn
m c yip
m  zhang
m  ding
m  heo
m k srirama
m  sharma
m j kim
n  kanazawa
n  hansen
n  heess
n j joshi
n  suenderhauf
n  liu
n d palo
n m m shafiullah
o  mees
o  kroemer
o  bastani
p r sanketi
p t miller
p  yin
p  wohlhart
p  xu
p d fagan
p  mitrano
p  sermanet
p  abbeel
p  sundaresan
q  chen
q  vuong
r  rafailov
r  tian
r  doshi
r  mart'in-mart'in
r  baijal
r  scalise
r  hendrix
r  lin
r  qian
r  zhang
r  mendonca
r  shah
r  hoque
r  julian
s  bustamante
s  kirmani
s  levine
s  lin
s  moore
s  bahl
s  dass
s  sonawani
s  song
s  xu
s  haldar
s  karamcheti
s  adebola
s  guist
s  nasiriany
s  schaal
s  welker
s  tian
s  ramamoorthy
s  dasari
s  belkhale
s  park
s  nair
s  mirchandani
t  osa
t  gupta
t  harada
t  matsushima
t  xiao
t  kollar
t  yu
t  ding
t  davchev
t z zhao
t  arm- strong
t  darrell
t  chung
v  jain
v  vanhoucke
w  zhan
w  zhou
w  burgard
x  chen
x  wang
x  zhu
x  geng
x  liu
x  liangwei
x  li
y  pang
y  lu
y j ma
y  kim
y  chebotar
y  zhou
y  zhu
y.-h  wu
y  xu
y  wang
y  bisk
y  dou
y  cho
y  lee
y  cui
y  cao
y  tang
y  zhang
y  jiang
y  li
y  iwasawa
y  matsuo
z  ma
z  xu
z j cui
z  zhang
z  fu
z  lin
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:11:46

Abstract: Robot behavior policies trained via imitation learning are prone to failure under conditions that deviate from their training data. Thus, algorithms that monitor learned policies at test time and provide early warnings of failure are necessary to facilitate scalable deployment. We propose Sentinel, a runtime monitoring framework that splits the detection of failures into two complementary categories: 1) Erratic failures, which we detect using statistical measures of temporal action consistency, and 2) task progression failures, where we use Vision Language Models (VLMs) to detect when the policy confidently and consistently takes actions that do not solve the task. Our approach has two key strengths. First, because learned policies exhibit diverse failure modes, combining complementary detectors leads to significantly higher accuracy at failure detection. Second, using a statistical temporal action consistency measure ensures that we quickly detect when multimodal, generative policies exhibit erratic behavior at negligible computational cost. In contrast, we only use VLMs to detect modes that are less time-sensitive. We demonstrate our approach in the context of diffusion policies trained on robotic mobile manipulation domains in both simulation and the real world. By unifying temporal consistency detection and VLM runtime monitoring, Sentinel detects 18% more failures than using either of the two detectors alone and significantly outperforms baselines, thus highlighting the importance of assigning specialized detectors to complementary categories of failure. Qualitative results are made available at sites.google.com/stanford.edu/sentinel.

Main paper: 
 introduction: 
 Imitation learning represents one of the simplest yet most effective ways of learning robot control behaviors from data. Herein, generative modeling techniques have enabled robot policies to learn from highly heterogeneous, multimodal demonstration data collected by humans [1, 2, 3] , showing early signs of generalization to novel environments [4] and embodiments [5] . When deploying robots beyond controlled lab settings, however, even the most powerful generative policies will eventually encounter outof-distribution (OOD) test cases-scenarios that differ from the training data-on which their behavior becomes unpredictable [6] . In response, we will require methods that monitor the behavior of learned, generative polices at deployment time to detect whether they are failing as a result of distribution shift.Identifying when a learned model behaves unreliably is typically framed as an OOD detection problem, for which a taxonomy of methods exist [7, 8] . While these methods can signal distribution shift [9, 10] or quantify uncertainty [11, 12] w.r.t. individual input-output samples, they do not fully characterize closedloop policy failures that arise from multiple, time-correlated prediction errors along a trajectory rollout. Action multimodality further complicates the failure detection problem: that is, actions sampled from multimodal generative policies can vary greatly from one timestep to the next, leading to complex runtime behaviors and, by extension, diverse failure modes compared to previous model-free policies [13, 14] . Therefore, the special case of generative robot policies necessitates the design of new failure detectors suited to their multimodal characteristics and closed-loop operational nature in deployment.In this work, we present Sentinel, a runtime monitoring framework that splits the task of detecting generative policy failures into two complementary categories. The first is the detection of failures in which the policy exhibits erratic behavior as characterized by its temporal inconsistency. For example, the robot may collide with its surroundings if the policy's action distributions contain conflicting action modes across time. To detect erratic failures, we propose to evaluate how much a generative policy's action distributions are changing across time using Statistical measures of Temporal Action Consistency (STAC). The second category is the detection of failures in which the policy is temporally consistent but struggles to make progress on its task. For example, the robot can stall in place or drift astray if the policy produces constant outputs. We propose to detect task progression failures (undetectable by STAC) zero-shot with Vision Language Models (VLMs), which can distinguish off-nominal behavior when prompted to reason about the robot's progress in a video question answering setup. Notably, one would want to catch erratic failures (the first category) fast, whereas task progression failures (the second category) do not require immediate intervention.Our contributions are three-fold: 1) A formulation of failure detection for generative policies that splits failures into two complementary categories, thus admitting the use of specialized detectors toward system-level performance increases (i.e., a divide-and-conquer strategy); 2) We propose STAC, a novel temporal consistency detector that tracks the statistical consistency of a generative policy's action distributions to detect erratic failures; 3) We propose the use of VLMs to monitor the task progress of a policy over the duration of its rollout, and we offer practical insights for their use as failure detectors. Provided with only a set of successful policy rollouts and a natural language description of the task, Sentinel (which runs STAC and the VLM monitor in parallel) detects over 97% of unknown failures exhibited by diffusion policies [1] across simulated and real-world robotic mobile manipulation domains. 
related work: 
 Advances in robot imitation learning include new policy architectures [1, 15, 16, 17] , hardware innovations for data collection [18, 19, 20] , community-wide efforts to scale robot learning datasets [3, 5, 21] , and training high-capacity behavior policies on these datasets [2, 4, 22, 23] . Of recent interest is the use of generative models [24, 25, 26, 27] to effectively learn from heterogeneous and multimodal datasets of human demonstrations [1, 5, 2, 23] . Generative policies thereby learn to represent highly multimodal distributions from which diverse robot actions can be sampled. While state-of-the-art generative policies demonstrate remarkable performance, their inherent multimodality results in more stochastic runtime behavior than that of previous model-free policies [13, 14, 28, 29, 30, 31] . In this work, we focus on characterizing the behavior of generative robot policies for failure detection.Despite recent progress, it is well known that learned policies may fail beyond their training distribution [6, 7, 8] , in part due to compounding prediction errors on states induced by the policy [32, 33] . As such, a recent work proposes to bound the performance of imitation learned policies prior to deployment [34] . Other works propose to retrain the policy on OOD states using corrective supervision from humans [35, 36, 37, 38, 39] . Notably, these methods apply after failures have occurred, maintaining the need for runtime monitors that detect policy failures and prevent their downstream consequences. Thus, our focus can be viewed as complementary to methods that learn post hoc from corrective feedback.The existing literature on out-of-distribution detectors and runtime monitoring for learned models is highly diverse, spanning multiple categories of methods. Model-based methods (e.g., [40, 41] ) are not directly applicable to the model-free policies we consider. Some methods only pursue failure modes that are known a priori [42, 43, 44, 45] , whereas we seek to detect unknown failures at deployment time. Many OOD detection works detect dissimilarity from training data via reconstruction [9, 46] or embedding similarity [47, 10] , however, observational differences may not always result in policy failure. Other methods directly quantify epistemic uncertainty [11, 12, 48, 49] , but come with considerable computational expense or may not be suitable for autoregressive, generative policy architectures, e.g., diffusion policies [1] . Several works monitor symbolic states to detect manipulation failures [50, 51] , but assume access to multiple sensor modalities (e.g., vision, haptic, proprioception) for symbolic state estimation. Most related to our approach are algorithms that perform consistency checks across sensor modalities [52] and time [53] . Different from these, we directly monitor the consistency of a learned policy's action distributions and its task progress to detect closed-loop failure.There is a growing interest in the use of Foundation Models [54] toward increasing robustness in robotic systems. Large Language Models are used to detect anomalies [55, 56] and to replan under execution failures [56, 57, 58, 59] . Reward models in the form of visual representations [60, 61] or VLMs [62] could be repurposed for failure detection by thresholding predicted rewards. However, [62] shows that additional fine-tuning is required to obtain reliable reward estimates. Du et al. [63] fine-tunes a VLM for episode-level success classification using a human annotated dataset on the order of 10 5 trajectories. In contrast to this work, we 1) focus on zero-shot assessment with VLMs, 2) seek to detect policy failure amidst task execution, and 3) consider the system-level role of VLMs operating alongside policy-level failure detectors, and as such, assign each detector to a specified category of failure. 
problem setup: 
 Failure Detection The goal of this work is to detect when a generative robot policy π(a|s) fails to complete its task. The policy operates within a Markov Decision Process (MDP) with a finite horizon H, but it may terminate upon completion of the task at an earlier timestep. Given an initial state s 0 representative of a new test scenario, executing the policy for t timesteps produces a trajectory τ t = (s 0 ,a 0 ,...,s t ). We define failure detection as the task of detecting whether a trajectory rollout τ H constitutes a failure at the earliest possible timestep t. To do so, we aim to construct a failure detector f (τ t ) → {ok,failure} that, at each timestep t, can provide a classification as to whether the policy will fail if it continues executing for the remaining H -t timesteps of the MDP. Note that the failure detector makes its assessment solely based on the history of observed states and sampled actions up to the current timestep t.The failure detector may contain parameters that require calibration, such as a detection threshold (as in §4.1). Therefore, we assume a scenario in which the policy π is first trained, then validated on test cases where it is expected to perform reliably. This validation process yields a small dataset of M successful trajectories D τ = {τ i } M i=1 that can be used to calibrate the failure detector f (if it contains parameters). Intuitively, the dataset D τ characterizes nominal policy behavior within or near the distribution of states it has been trained on, which helps to ground the assessment of potentially OOD trajectories at test time.We measure failure detection performance in terms of true positive rate (TPR), true negative rate (TNR), and detection time (DT). We count a true positive if the failure detector raises a warning at any timestep in a trajectory where the policy fails, the earliest of which counts as the detection time. We count a true negative if the failure detector never raises a warning in a trajectory where the policy succeeds. We refer to §B.4 for supporting definitions of policy failure and key performance metrics. Policy Formulation We consider the setting where the policy π is stochastic and predicts a sequence of actions (also referred to as an action chunk [18] ) for the next h timesteps. That is, the action sequence sampled at the t-th timestep, a t ∼ π(•|s t ), consists of h actions a t := (a t|t ,a t+1|t ,...,a t+h-1|t ), where the notation a t+i|t denotes the action prediction for time t + i generated at timestep t (as in [64] ). The actions a •|t ∈ A may correspond to e.g., endeffector poses or velocities. To control the robot, we sample an action sequence and execute the first k < h actions, a t:t+k|t , after which we re-evaluate the policy at timestep t+k. We visualize this receding horizon rollout for k = 1 in Fig. 2 . Notably, a t and a t+k contain actions that temporally overlap for h-k timesteps (i.e., at a t+k:t+h-1|t and a t+k:t+h-1|t+k ).Several recently proposed policy architectures achieve state-of-the-art performance by sampling action sequences using generative models [1, 18, 2] , to which our approach is generically applicable. However, in this paper, we specify the failure detection problem for diffusion policies (DP) [1] , which a) are stable to train and b) address action multimodality by representing the policy distribution with a denoising diffusion probabilistic model (DDPM) [26] . We note that the computationally intensive, iterative nature of the denoising process makes it challenging to directly apply existing OOD detection methodologies (e.g., [65] ) to diffusion policies for failure detection, thus motivating several of our design decisions. Further details on the training and properties of these models are provided in §B.2. 
proposed approach: sentinel: 
 The failure behavior of a generative policy by OOD conditions can be highly diverse, and we therefore argue that the desiderata for a failure detector may vary between qualitative types of failures. In this work, we propose to split the failure detection task into two complementary failure categories.The first is the detection of failures resulting from erratic policy behavior, which may cause a robot to end up in states that are difficult or costly to reset from, knock over objects, or lead to safety hazards. Therefore, it is important to detect erratic behavior as quickly as possible ( §4.1). The second category is the detection of failures in which the policy struggles to make progress on its task (hereafter referred to as task progression failures) but does so in a temporally consistent manner. For example, the policy may confidently place an object in the wrong location. Here, we must observe the robot over a longer period of time to identify that the policy is not making progress towards task completion ( §4.2).The key insight of our approach is that by defining one failure category as the complement of the other, it becomes trivial to combine failure detectors to form an accurate overall detection pipeline whilst satisfying the requirements of each failure category. Our full pipeline, Sentinel, is shown in Fig. 3 .When a policy operates in nominal, in-distribution settings, it should complete its task in a temporally consistent manner. For example, a policy may plan to avoid an obstacle on the right or on the left, but not jitter between the two options. Moreover, as noted in [1] , training a diffusion policy that predicts action sequences rather than individual actions encourages temporal consistency between action predictions.Therefore, we propose to construct a quantifiable measure of temporal action consistency to detect whether the policy is behaving erratically, and hence, is likely to fail at the task. However, the multimodal distributional nature of DPs makes it difficult to directly compare two sampled actions a t ∼ π(a t |s t ) and a t+k ∼ π(a t+k |s t+k ), e.g., throughout execution. This is because the actions may differ substantially along their prediction horizon when the policy commits or switches between different action modes, or simply due to randomness in sampling. Instead, we quantify erratic policy behavior with statistical measures of temporal action consistency (STAC, which we term our approach).Let πt := π(a t+k:t+h-1|t |s t ) and πt+k := π(a t+k:t+h-1|t+k |s t+k ) be the marginal action distributions of the temporally overlapping actions between timesteps t and t + k. We compute the temporal consistency between two contiguous timesteps t and t+k as D(π t ,π t+k ) ≥ 0, where D denotes the chosen statistical distance function (e.g., maximum mean discrepancy, KL-divergence) 1 . In addition, we propose to take the cumulative sum of statistical distances along a trajectory as a measure of the overall temporal consistency in a policy rollout. At each policy-inference timestep t = jk with j ∈ {0,1,...}, we compute the temporal consistency score asEQUATIONComputing the consistency score in a cumulative manner has two advantages over thresholding the distance at each timestep individually. Firstly, it allows us to detect cases where the temporal consistency metric D is marginally larger than usual throughout the episode (e.g., jitter). Secondly, it allows us to detect instances where the policy is temporally inconsistent more often than in nominal scenarios. At runtime, we raise a failure warning at the moment that η t exceeds a failure detection threshold γ, which we calibrate offline using the validation dataset of successful trajectories D τ . Here, we first compute the cumulative temporal consistency scores throughout the entirety of the lengths H i ≤ H of trajectories in D τ , yielding {η i Hi } M i=1 . Then, we set the threshold γ to the 1-δ quantile of {η i Hi } M i=1 , where δ ∈ (0,1) is a hyperparameter. Intuitively, this ensures that the false positive rate (FPR)-the probability that we raise a false alarm and terminate on any trajectory that is i.i.d. with respect to D τ -remains low, such that any warnings are likely failures. We can formalize this intuition using recent results from conformal prediction [66, 67] : Proposition 1 (STAC has low FPR). Let P τ denote the distribution of success trajectories in the validation dataset D τ = {τ i } M i=1 iid ∼ P τ . Then, the FPR-the probability of raising a false alarm at any point during an i.i.d. test trajectory τ ∼ P τ of length H ′ ≤ H-is at most δ:FPR := P Pτ ∃ 0 ≤ t ≤ H ′ s.t. η t > γ ≤ δ.We refer to §A.1 for additional details on STAC and §D for a full statement and proof of Proposition 1.A policy operating in out-of-distribution settings may not always fail by exhibiting erratic behavior that we can detect with STAC ( §4.1). For example, suppose the policy confidently commits to the wrong plan or produces approximately constant outputs. Detecting such failures requires an understanding as to whether or not the policy is progressing on its task, which necessitates a more comprehensive analysis of the robot's behavior within the context of its task specification. Therefore, we propose to use VLMs to monitor the task progress of the policy by providing them with the robot's image observations up to the current timestep as a video. We do so because recent work has shown that high-capacity VLMs possess robotics relevant knowledge and contextual reasoning abilities [68, 69, 63, 70, 62] .We formulate the detection of task progression failures as a chain-of-thought (CoT) [71] , video question answering (QA) task [72, 73] , reflecting current best practices in prompting. To capture a notion of task progress, the VLM must reason across time and in the context of the policy's task. Thus, we construct a prompt that contains a description of the task and the VLM's role as a runtime monitor. We query the VLM online using the text prompt and the history of observed images (i.e., a video) I 0:t := (I 0 ,I νk ,I 2νk ,...,I t ) up to the current timestep t, where ν determines the frequency of the images relative to the execution horizon k of the DP ( §3). Differentiating between partial progress and task failure can be ambiguous for a slow moving robot, and thus, we also specify the current elapsed time t and the time limit for the task H. This enables the VLM to gauge whether the rate at which the robot is executing will result in a timely task completion. After performing a CoT analysis, the VLM concludes with a classification in {ok, failure}. For additional details on the VLM and prompt, please see §A.2.At the time of writing, cloud-querying a state-of-the-art VLM for video QA incurs significant latency (e.g., GPT-4o's mean response time was 14.0s). However, we emphasize that VLM inference latency is a lesser concern for detecting task progression failures because they are likely to occur at longer timescales and do not require urgent intervention. In contrast, we assign the rapid detection of erratic failures to STAC ( §4.1). Notably, the fast and slow detection requirements of our complementary failure categories mean that STAC and the VLM can operate at different timescales, offering potential benefits such as reduced costs and a lower likelihood of false positives when they run in parallel (Fig. 3 ). 
experiments: 
 We conduct a series of experiments to test our failure detection framework. These experiments take place in both simulation and the real world (Fig. 1 ), and host an extensive list of baselines. We refer to §B for a detailed description of our environments, hardware setup, baselines, and evaluation protocol.Environments. We include the PushT domain from [1] to evaluate the detection of failures under action multimodality. The Close Box and Cover Object domains involve two mobile manipulators and thus present the challenge of a high-dimensional, 14 degree-of-freedom action space. We additionally conduct hardware experiments with a mobile manipulator for a nonprehensile Push Chair task. This task presents greater dynamic complexity than the simulation domains [74] . At test time, we generate OOD scenarios by randomizing a) the scale and dimensions of objects in PushT and Close Box and b) the pose of the object in Cover Object and Push Chair beyond the policy's demonstration data.Baselines. We evaluate Sentinel (i.e., both STAC and the VLM) against baselines representative of multiple methodological categories in the OOD detection literature [7] . Intuitively, these categories represent different formulations of the failure detector's score function, responsible for computing the per-timestep scores that are then summed to compute the trajectory score as in Eq. 1. We consider score functions based on the embedding similarity of observed states w.r.t. D τ [47] , the reconstruction error of actions sampled from the DP [75] , and the output variance of the DP. To strengthen the comparison, we introduce a new baseline that uses the DDPM loss (Eq. 2) on re-noised actions sampled from the DP as the failure detector's score function. Where applicable, we implement temporal consistency variants of these baselines to ablate the design of STAC. Further details on these baselines are provided in §B.3.Evaluation Protocol. We train a DP for each environment and use standard settings for the DP's prediction and execution horizon [1] . We use the same calibration and evaluation protocol across all failure detection methods. That is, we calibrate detection thresholds to the 95-th quantile of scores in a dataset D τ = {τ i } M i=1 of M = 50 in-distribution rollouts for each simulated task and M = 10 in-distribution rollouts for the real-world task. Finally, we report standard detection metrics including TPR, TNR, Mean Detection Time, Accuracy, and Balanced Accuracy, following the definitions in §3. Our temporal consistency detector, STAC, accounts for when a policy fails (high true positive rate) and when it generalizes to out-of-distribution test cases (high true negative rate), in contrast to embedding-based methods that associate state atypicality with policy failure (low true negative rate). Select baselines that accurately detect erratic policy failures in this domain experience a decrease in performance under multimodal conditions (i.e., PushT, as shown in Fig. 5 ), whereas STAC continues to exhibit strong performance across multiple domains. VLMs must reason over video to attain high true negative rates, as is necessary to combine them with STAC (see Fig. 6 ). Sentinel, which runs STAC and the VLM monitor in parallel, detects 100% of erratic policy failures in this domain. Right: The best performance comes from measuring temporal consistency with statistical distance functions; augmenting baselines with temporal consistency does not always increase their performance.STAC detects diffusion policy failures in multimodal domains. Fig. 5 (Left) compares STAC against the best performing method of each baseline category in the PushT domain. Here, STAC is the only method to achieve a balanced accuracy of over 90%, indicating that temporal consistency (or lack thereof) is strongly correlated with success (or failure). Alternative output metrics, such as the DP's output variance, do not perform well because both successes and failures can exhibit high-variance outputs in multimodal domains. Interestingly, the embedding similarity approach performs strongly, which indicates that state dissimilarity w.r.t. the calibration dataset happens to be correlated with failure in this domain.Statistical measures of action similarity enable temporal consistency detection. Fig. 5 (Right) ablates the design decisions of STAC. First, we observe that augmenting baselines with temporal consistency will at most marginally increase their performance. Second, using a non-statistical distance function (e.g., min. distance) to measure temporal action consistency performs worse than the baselines because it omits action multimodality. Thus, it is the combination of statistical distance functions with temporal consistency that yields the best result. We refer to §C.1 for an extended ablation of STAC.STAC accounts for OOD failures and generalization. Results on the Close Box domains are shown in Table 1 . STAC attains the highest accuracy in aggregate (96%). However, two of our newly proposed baselines-using the DDPM loss (Eq. 2) and a temporal reconstruction variant of [75] -also perform well, perhaps due to a decrease in action multimodality relative to PushT. Notably, we find that embedding similarity methods conflate OOD states with policy failure, resulting in false positives when the policy succeeds OOD. In contrast, STAC effectively differentiates OOD successes from failures.VLMs must reason across time. In Table 1 , we find that a state-of-the-art VLM (GPT-4o) struggles to identify task success when given only a single image. Instead, it must observe the robot over the extent of a policy rollout to more accurately reason about task progression and changes in state (resulting in a 91% TNR). While erratic failures are time-sensitive, they are visually more subtle and thus difficult for the VLM to detect (77% TPR). The robot takes more obviously wrong actions (e.g., stalling, drifting astray) in task progression failures (Fig. 6 ). As expected, the VLM has a significantly slower detection time relative to STAC, further highlighting STAC's value at quickly detecting erratic behavior. Sentinel: Combining STAC and VLMs for system-level performance increases. We evaluate our failure detectors on distribution shifts that primarily lead to task progression failures in the Cover Object and Close Box domains. The result is shown in Fig. 6 . STAC achieves a low TPR (44%) when the policy fails in a temporally consistent manner, whereas the VLM (GPT-4o for Close Box, Claude for Cover Object) accurately detects task progression failures.As a result, their combination (Sentinel) achieves a 93% TPR whilst incurring only a 7% increase in FPR. The rise in detection time indicates that both STAC (fast) and the VLM (slow) are contributing to the detection of failures.Sentinel detects real-world, generative policy failures. We evaluate Sentinel on the Push Chair task across 10 successful and 10 failed policy rollouts. The results are shown in Table 2 . When calibrated on only 10 successful in-distribution rollouts, STAC detects 80% of policy failures and raises only one false alarm (90% TNR). The VLM exhibits stronger performance in the real world (90% TPR, 100% TNR) than it does in the simulation domains, perhaps because real-world images constitute a lesser domain gap for visual reasoning compared to images rendered in simulation. Overall, Sentinel achieves a 95% detection accuracy, highlighting its efficacy for detecting failures in real-world robotic settings. Discussion. Holistic analysis of Table 1 , Fig. 6 , and Table 2 shows that we can easily combine STAC and the VLM to yield a performant overall detector for both erratic and task progression failures, particularly because both detectors achieve a high overall TNR. Since all the baselines may 1) show low accuracy on either of the erratic failure domains (i.e., the multimodal PushT and Close Box domains) or 2) yield a low TNR, it is unclear how to combine them with other detectors in a way that outperforms Sentinel. 
conclusion: 
 In this work, we investigate the problem of failure detection for generative robot policies. We propose Sentinel, a runtime monitor that splits the failure detection task into two categories: 1) Erratic failures, which we detect by measuring the statistical change of a policy's action distributions over time; 2) task progression failures, where we use Vision Language Models to assess whether the policy is consistently taking actions that do not solve the task. Our results highlight the importance of targeting complementary failure categories with specialized detectors. Future work includes the use of Sentinel to monitor high-capacity policies [2, 23] , inform data collection, and accelerate policy learning.Limitations. While categorizing erratic and task progression failures leads to accurate detection of failures across the domains we consider, these two failure categories may not be exhaustive. In the future, introducing additional categories or further partitioning existing ones might provide a broader coverage of failures, allow for more efficient failure detection, and inform mitigation strategies. Furthermore, our approach does not provide formal guarantees on detecting failures. However, providing such guarantees would require data of both successful and unsuccessful policy rollouts to calibrate the detector [67] .Although our detectors attain low false positive rates in aggregate, taking the union of their predictions may, in the worst case, increase the risk of false alarms. Thus, exploring more sophisticated ways to combine complementary failure detectors is a possible point of extension. Finally, our approach is not targeted at predicting failures before they occur, but instead focuses on detecting failures as they occur. 
 
 Appendix: As shown in Fig. 3 , the Sentinel runtime monitoring framework consists of the parallel operation of two complementary failure detectors, each assigned to the detection of a particular failure category of generative policies. The first is a temporal consistency detector that monitors for erratic policy behavior via statistical temporal action consistency (STAC) measures. The second is a Vision Language Model (VLM) that monitors for failure of the policy to make progress on its task. In this section, we provide additional details w.r.t. the implementation of STAC ( §A.1) and the VLM runtime monitor ( §A.2).A. Hypothesis Our central hypothesis is that large statistical distances correlate with downstream policy failure. Intuitively, a predictive policy can be likened to possessing an internal world model that simulates how robot actions affect environment states. When the policy is in distribution, we expect this world model to be accurate, thus resulting in smaller statistical distances. More concretely, if the policy's internal model of state s t+k at timestep t coincides with the actual observed state s t+k at timestep t+k, the distribution of actions πt+k should be well-represented by the distribution πt . As a result, the distance D(π t ,π t+k ) will be small (for the right choice of statistical distance function D).Conversely, when the policy is out of distribution (OOD), its internal model of state s t+k at timestep t may be inaccurate, yielding a divergence between πt and πt+k and a larger statistical distance.Implementation Details As mentioned in §4.1, we propose to approximate D(π t ,π t+k ) with an empirical distance function D instead of computing it analytically, as doing so presents the challenge of marginalizing out both the non-overlapping actions (between timesteps t and t+k) and the intermediate steps of the diffusion process [76] . We found the following approximations to work well in practice:• Maximum Mean Discrepancy (MMD) with radial basis function (RBF) kernels. We computeD(π t ,π t+k ) = E at,a ′ t ∼πt [k(a t ,a ′ t )]+E a t+k ,a ′ t+k ∼π t+k k(a t+k ,a ′ t+k ) -2E at∼πt,a t+k ∼π t+k [k(a t ,a t+k )],where k(x,y;β 1 ) = exp - ||x-y|| 2 β 1 .Here, k : R (h-k)×|A| ×R (h-k)×|A| → R computes the similarity between two overlapping action sequences, and β 1 denotes the bandwidth of the RBF kernel. The expectations are taken over a batch of B action sequences sampled from the generative policy.• Forward KL-divergence via Kernel Density Estimation (KDE) of the policy distributions:D(π t ,π t+k ) = E a t+k ∼π t+k log p(a t+k ) q(a t+k ) ,where p and q are KDEs of πt+k and πt fit on a batch of B action sequences sampled from each policy distribution, respectively. As before, we use Gaussian RBF kernels of the form k(x,y; β 2 ), where β 2 denotes the bandwidth of the RBF kernels used for KDE.• Reverse KL-divergence via KDE of the policy distributions:D(π t ,π t+k ) = E at∼πt log p(a t ) q(a t ) ,where p and q are KDEs of πt and πt+k fit on a batch of B action sequences sampled from each distribution, respectively, and all other parameters follow the forward KL definitions.Hyperparameters The batch size B, MMD bandwidth β 1 , and KDE bandwidth β 2 are hyperparameters that we select for a given environment. As expected, we found that larger batch sizes are necessary for accurate mean embeddings and density estimates in domains with higher degrees of multimodality (e.g., B = 256 action sequences for PushT and Push Chair). We also found that using either default settings or dynamic calibration techniques are sufficient to obtain suitable MMD and KDE bandwidth parameters β 1 and β 2 , respectively. For example, setting β 2 in proportion to the maximum eigenvalue of the covariance of overlapping actions a t+k:t+h-1|• sampled from πt and πt+k worked well in multimodal domains. Further details on hyperparameters are provided in Table 3 . Additional Design Choices There are several additional settings that one could adjust to increase STAC's detection performance on their task. First, filtering components of the policy's action space that are either noisy or discrete can increase the quality of the statistical distance score function. For example, the policy's action space in our robotic manipulator domains include end-effector linear and angular velocities, as well as a binary gripper command. However, when computing statistical distances, we omit all binary gripper commands. Next, reducing the execution horizon k of the generative policy to compare action distributions that are closer in time can mitigate excessively large statistical distances in highly dynamic or stochastic environments. Likewise, comparing action distributions over a shorter prediction horizon h may be suitable if the tails of predicted action sequences e.g., exhibit high variance.As described in §4.2, we formulate the detection of task progression failures as a chain-of-thought (CoT) [71] , video question answering (Video QA) [73] task with VLMs. Below, we provide details on the implementation of our VLM runtime monitor and the prompt templates used in our experiments.Vision Language Models In extended experiments ( §C.2), we include variants of the VLM runtime monitor based on several models: OpenAI's GPT-4o, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5 Pro [79] . At the time of writing, these represent the current state-of-the-art VLMs for complex, multimodal reasoning tasks. We use consistent prompts across all models, however, we slightly vary the implementation of the monitor to reflect the suggested best practices of each VLM.We propose to query the VLM online with a parsed text prompt describing the runtime monitoring task and the history of images (i.e., a video) I 0:t := (I 0 ,I νk ,I 2νk ,...,I t ) captured by the robot's camera system up to the current timestep t. Here, the hyperparameter ν specifies the frequency of the images relative to the execution horizon k of the generative policy ( §3) for generality, as the video may be captured at a much higher frame rate than the policy's execution rate. In experiments, simply setting ν ∈ {1,2} provided sufficient granularity for the VLM to identify the motion of the robot.By making non-blocking API calls, the VLM runtime monitor can operate at relatively high frequencies.For example, the VLM can be queried at each policy-inference timestep t = jk for j ∈ {0,1,...} (i.e., at STAC's detection frequency) to provide a failure classification. However, depending on the task, doing so may neither be necessary nor desirable for two reasons. First, because task progression failures are likely to occur at longer timescales than erratic failures, querying the VLM at a reduced frequency might provide time for meaningful changes in state to occur. In turn, this would reduce redundancy in the VLM's predictions e.g., if no meaningful changes in state occurred since the last time it was queried. Second, while STAC-a statistical output monitor-can be evaluated at negligible cost (computationally and monetarily), querying state-of-the-art, closed-source VLMs may come with considerable expense. Since task progression failures are unlikely to require immediate intervention (in contrast to erratic querying the VLM less often than STAC could be preferable. In experiments, we queried the VLM to detect task progression failures twice per episode.The prompt template consists of three parts: 1) a brief description of the VLM's role as the runtime monitor of a robotic manipulator system, which it must execute by analyzing the attached video; 2) a description of the robot's task, the total amount of time that has elapsed2 , and time limit for the task (corresponding to the MDP horizon H in §3); 3) instructions to elicit a CoT response, ensuring that the VLM describes and analyzes the motion of the robot and all task-relevant objects and outputs a classification that can be easily parsed. To remove ambiguity over the expected behavior of the robot and what constitutes task completion, we make sure that the task description is sufficiently detailed:task_descriptions: cover: "hide the white box by covering it with the black blanket. The white box is located somewhere in front of the two robot arms and does not move. The black blanket starts directly in between the two robot arms" close: "close the white box by folding in the two smaller white side lids and the bigger white back lid. The white box is located in between the two robot arms and does not move. The robots should concurrently approach the side lids and push both side lids up, followed by approaching the back lid and folding up the back lid with both arms, without grasping the lids with the grippers" push_chair: "push the black chair into the circular table. The black chair starts directly in front of the robot. The robot should push black chair in a relatively straight line, without the chair rotating to the left or to the right, so that the seat of the chair is properly tucked under the circular table " We elicit a four-step CoT response from the VLM that 1) generates a set of task-relevant questions whose correct answers would fully characterize the motion of the robot and all task-relevant objects in the video, 2) answers the task-relevant questions while providing fine-grained visual details, 3) analyzes the questions, answers, and elapsed time to identify whether the robot is making progress towards task completion within the episode time limit, and 4) concludes with an overall classification in {ok, failure}. Interestingly, we found that prompting the VLM to generate its own questions instead of manually specifying them leads to more accurate descriptions of the video and ensuing predictions.Prompt Template (Video QA)You are the runtime monitor for an autonomous mobile manipulator robot capable of solving common household tasks. A camera system captures a series of image frames (i.e., a video) of the robot executing its current task online. The image frames are captured at approximately 1Hz. As a runtime monitor, your job is to analyze the video and identify whether the robot is a) in progress of executing the task or b) failing to execute the task, for example, by acting incorrectly or unsafely.The robot's current task is to {DESCRIPTION}. The robot may take up to { TIME_LIMIT} seconds to complete this task. The current elapsed time is {TIME } seconds.Format your output in the following form: [start of output] Questions: First, generate a set of task-relevant questions that will enable you to understand the full, detailed motion of the robot and all task-relevant objects from the beginning to the end of the accompanying video. Answers: Second, precisely answer the generated questions, providing finegrained visual details that will help you accurately assess the current state of progress on the task. Analysis: Assess whether the robot is clearly failing at the task. Since the video only represents the robot's progress up to the current timestep and the robot moves slowly, refrain from making a failure classification unless the robot is unlikely to complete the task in the allotted time. Explicitly note the amount of time that has passed in seconds and compare it with the time limit (e.g., x out of {TIME_LIMIT} seconds). Finally, based on the questions, answers, analysis, and elapsed time, decide whether the robot is in progress, or whether the robot will fail to complete its task in the remaining time (if any The Video QA failure detection task requires comprehensive and detailed reasoning of over potentially long sequences of images, which, at the time of writing, is a challenge for even the most capable VLMs.As such, we can expect the performance of the VLM runtime monitor to degrade when it is deployed in domains that are visually OOD w.r.t. the VLM's training data (e.g., images rendered in simulation or captured from unusual camera poses). In these settings, the VLM runtime monitor may provide a reasonable but imperfect signal on task progression failures, resulting in misclassifications.To strengthen the reliability of our VLM runtime monitor, we propose a simple prompt ensembling strategy [80] , whereby we construct multiple Video QA prompts, query the VLM with each prompt, and take the overall failure classification to be the majority vote of the predictions across all prompts. Intuitively, if the failure detectors associated with each individual prompt are fairly accurate to begin with, the resulting majority-vote detector will have an even higher probability of correctness.In experiments, we only found it necessary to use prompt ensembling in the Cover Object domain. We construct two variants of the Video QA prompt (3 prompts total), each of which follow a similar CoT structure while including additional information to diversify the VLM's reasoning. The first prompt variant, Video QA + Success Video, includes a video of a successful policy rollout for the current task. This allows the VLM to distinguish off-nominal policy behavior at test time from nominal policy behavior illustrated in the example video. The second prompt variant, Video QA + Goal Images, includes example images of the scene at the end of successful policy rollouts, which serve as a visual reference for task completion. In accordance with the assumptions of our framework, these prompt variants only require data associated with policy success to detect unknown failures at test time.Prompt Template (Video QA + Success Video)You are the runtime monitor... # Same as Video QATo inform your analysis, you will be provided with an example video that shows the full motion of the robot and all task-relevant objects when the task is successfully executed. For example, the last image frame in the example video will show what the scene should look like at the end of a successsfully executed task. By comparing the current video with the example video, you may be able to visually distinguish when the robot is failing at the task versus when it is making steady progress or has completed.The robot's current task is... # Same as Video QA Questions: First, generate a set of task-relevant questions that will enable you to understand the full, detailed motion of the robot and all task-relevant objects from the beginning to the end of the accompanying video. In addition, generate questions that will enable you to identify any key similarities or differences between the current video and the example success video. Answers: Second, precisely answer... # Same as Video QA Prompt Template (Video QA + Goal Images)You are the runtime monitor... # Same as Video QA To inform your analysis, you will be provided with several example images that show what the scene (i.e., the robot and all task-relevant objects) should look like at the end of a successfully executed task. By comparing the last few image frames of the current video with these example images, you may be able to visually distinguish when the robot is failing at the task versus when it is making steady progress or has completed.The robot's current task is... # Same as Video QA Questions: First, generate a set of task-relevant questions that will enable you to understand the full, detailed motion of the robot and all task-relevant objects from the beginning to the end of the accompanying video. In addition, generate questions that will enable you to identify any key similarities or differences between the current video and the example images of successfully completed tasks. Answers: Second, precisely answer... # Same as Video QAWe provide additional details on the environments used to evaluate Sentinel. These environments vary in terms of their data distribution (e.g., multimodal or high-dimensional actions) and support different types of distribution shift (e.g., object scale, pose, dynamics), under which the behavior of generative diffusion policies can be methodically studied. The environments are visualized in Fig. 7 . PushT Domain The policy is tasked with pushing a planar "T"-shaped object into a goal configuration.A trajectory is considered successful if the overlap between the "T"-shaped object and its goal exceeds 90% within 300 environment steps. The action space is the 2-DoF linear velocity of the end-effector.We generate OOD test scenarios by non-uniformly randomizing the scale and dimensions of the "T"-shaped object beyond the randomizations contained in the policy's demonstration data. The policy tends to fail by converging to a locally optimal configuration, where the "T" overlaps with its goal but in an incorrect orientation. Since the task can be solved in a number of ways, we include this domain to evaluate the performance of various score functions in the presence of action multimodality. We refer to [1] for the process of generating demonstration data in this domain.The policy is tasked with closing a box that has three lids. A trajectory is considered successful if all three lids are closed within 120 environment steps (24 seconds). The action space is the 14-DoF linear + angular velocities and gripper commands for the end-effectors of two mobile manipulators. Demonstration data is generated by an oracle policy that sets a series of waypoints for the end-effectors based on the initial state. We generate OOD test scenarios by non-uniformly randomizing the scale of the box beyond the randomizations contained in the policy's demonstration data. The policy tends to fail erratically when the robots e.g., collide with the box or its lids, however, task progression failures may also occur. This domain is primarily used to evaluate the detection of erratic policy failures on a bi-manual robotic system with a high-dimensional action space.Cover Object Domain The policy is tasked with covering a rigid object with a cloth. A trajectory is considered successful if over 75% of the object is covered by the cloth within 64 environment steps (13 seconds). The action space and process of generating demonstration data is identical to that of Close Box. We generate OOD test scenarios by non-uniformly randomizing the position of the object beyond the randomizations contained in the policy's demonstration data. The policy tends to fail by releasing the cover before reaching the object. Hence, this domain is used to evaluate the detection of task progression failures, where reasoning over longer durations is required to assess task progress.Mobile Robot Setup We use a holonomic mobile base equipped with a Kinova Gen3 7-DoF arm. A single ZED 2 camera is fixed in the workspace to capture visual observations for the generative policy. The ZED 2 camera first generates a partial-view point cloud of the environment, from which we segment task-relevant objects using the Grounded Segment Anything Model [81] based on a natural language description related to the task. The segmented point cloud serves as the visual input to the policy. Additionally, we use a motion capture system to track the pose of the mobile base. During evaluation, the policy processes the point clouds, predicts a sequence of 16 actions, of which the first 4 are executed on the robot. The mobile manipulator robot then maneuvers its arm according to these commands, adjusting the pose of the base if the end-effector moves outside a pre-defined workspace.Push Chair Domain The policy is tasked with tucking a chair into a table using a single-arm mobile manipulation platform. A trajectory is considered successful if the seat of the chair is properly tucked under the table by the end of the policy rollout. The action space is the 7-DoF linear + angular velocities and gripper command for the end-effector of the mobile manipulator robot. Demonstration data is extracted from human videos: we use an off-the-shelf hand detection model [82] , an object segmentation model [83, 81] , and a stereo-to-depth model to extract human hand poses and object point clouds from a subsampled set of frames in each of the 15 human demonstration videos. We generate OOD test scenarios by randomizing the initial pose of the chair beyond the randomizations contained in the demonstration data. The policy tends to fail erratically if the chair rotates away in either direction when pushed, but such failures are also visually apparent. Therefore, this task is used to test the efficacy of both STAC and the VLM runtime monitor in a dynamically complex [74] , real-world setting.We train a diffusion policy (DP) for each environment, using 200 demonstrations for the PushT domain, 50 demonstrations for each of the Close Box and Cover Object domains, and 15 demonstrations for the real-world Push Chair domain. In a DP, actions are generated by iteratively denoising an initially random action a N t ∼ N (0,1) over N steps as a N t ,...,a 0 t , where a i t with a superscript i denotes the generated action sequence at the i-th denoising iteration. In an imitation learning setting, the DP's noise prediction network ϵ θ is trained to predict the random noise ϵ i added to actions drawn from a dataset of expert demonstrations D train by minimizingEQUATION)where the constants ᾱi depend on the chosen noise schedule of the diffusion process.To increase the salience of distribution shift w.r.t. the position and scale of objects, we use point clouds as inputs to the policy instead of RGB images (i.e., a 5% increase in object scale may not be salient in an image). For simulation experiments, we use a diffusion policy architecture identical to the original paper [1] except for the visual encoder, where we substitute the ResNet-based encoder for a PointNet-based one: a 4-layer PointNet++ encoder [84] with hidden dimension 128. The output of this encoder is concatenated with the proprioceptive inputs and then fed to the noise prediction network. For real-world experiments, we use the recently proposed EquiBot diffusion policy architecture [85] , which additionally incorporates SIM(3) equivariance into the diffusion process. We use EquiBot to evaluate our failure detectors on a current state-of-the-art approach for learning generative policies in the real world. All diffusion policies produce an action over N = 100 denoising iterations. Unless otherwise specified, we use standard settings for the prediction h and execution horizon k of the diffusion policy.We outline the implementation details of our baselines as introduced in §5. First, with the exception of the VLM runtime monitors, all evaluated failure detection methods consist of computing a score S(•) at each policy-inference timestep in a rollout, taking the cumulative sum of scores up to the current timestep t, and then checking if the cumulative sum exceeds a calibrated threshold to detect policy failure. As such, the baselines differ in their score function, i.e., how they compute the pertimestep scores that are then summed and thresholded. Intuitively, a good score function should be well-correlated with policy failure, that is, it should output small values when the policy is succeeding and large ones when it is failing. For example, Fig. 4 demonstrates that STAC holds this property. We baseline against an extensive suite of score functions, some of which we newly introduce for the case of generative diffusion policies, and others that are common in the OOD detection literature [7] .• Policy Encoder Embedding quantifies the dissimilarity of the current point cloud observation o t w.r.t. to the point clouds in the calibration dataset of M successful policy rollouts D τ = {τ i } M i=1 (as described in §3) within the embedding space of the policy's encoder (here, o t denotes the point cloud input to the policy, which includes the point cloud at the current and previous timestep). More concretely, let E be the policy's encoder, z t = E(o t ) be the current point cloud embedding, and D z = E(D τ ) be the embeddings of all point clouds contained in the calibration dataset. We compute the per-timestep score as the Mahalanobis distanceEQUATIONwhere µ z is the mean and Σ zz is the covariance of the embeddings in D z . At test time, we raise a failure warning if the cumulative score η t exceeds a calibrated detection threshold γη t > γ, where η t = j-1 i=0 S(z i ; D z ), t = jk.Here, γ is set to the 1 -δ quantile of cumulative scores computed over the calibration trajectories{η i |τ i | } M i=1, where τ i ∈ D τ . Importantly, when computing the calibration scores η i , we do so in a leave-trajectory-out fashion: i.e., for a point cloud o t ∈ τ i where τ i ∈ D τ , we compute the per-timestep score as S(E(o t ); E(D τ \τ i )). This ensures that the dissimilarity of observation o t is computed w.r.t. trajectories other than its own, which a) aligns with how scores are computed at test time and b) ensures that calibration scores are not trivially low. We experimented with alternatives to the Mahalanobis distance in Eq. 3, substituting it with top-k scoring for k ∈ {1,5,10} based on cosine similarity and L2 distance metrics. However, we found the Mahalanobis distance to be the most stable. We also evaluated variants of this baseline that compute the dissimilarity of the full policy state s t (including both the point cloud embedding and the robots' end-effector poses), but found equivalent performance.• CLIP Pretrained Embedding quantifies the dissimilarity of the current image observation I t w.r.t. to the images in the calibration dataset D τ = {τ i } M i=1 within the embedding space of a pretrained CLIP encoder [86] . The score function (Eq. 3) and calibration process are identical to those of Policy Encoder Embedding. Importantly, the encoder used here is trained with a representation learning objective, which results in a structured embedding space and more interpretable embedding similarity scores. In our experiments, we use the open-source clip-vit-base-patch32 version of CLIP without any fine-tuning.• ResNet Pretrained Embedding is identical to CLIP Pretrained Embedding, except it quantifies image-space dissimilarity using embeddings from a ResNet18 pretrained model [87] .• Temporal Non-Distributional Minimum is similar to STAC ( §A.1) in that it seeks to compute a consistency score between overlapping actions a t+k:t+h-1|t and a t+k:t+h-1|t+k sampled from the generative policy at contiguous policy-inference timesteps t and t + k, respectively. However, it does so by using a non-statistical distance function. In particular, this baseline computes the per-timestep temporal consistency score at timestep t+k asS(s t+k ) = min b∈{1..B} a t+k:t+h-1|t -a b t+k:t+h-1|t+k ,where a b t+k:t+h-1|t+k ∼ πt+k (•|s t+k ). That is, we sample a batch of B action sequences at timestep t + k, compute their L2 distances w.r.t. the overlapping actions of the previously executed action sequence a t+k:t+h-1|t , and return the L2 distance associated with the most similar action sequence. Intuitively, this baseline attempts to find the closest action sequence at timestep t+k to the previously executed action sequence, while STAC attempts to quantify how well the action distribution πt+k at timestep t+k is represented in the distribution πt at timestep t. The values of B are provided in Table 3 . The calibration and runtime procedures of this baseline are identical to those of STAC ( §4.1).• Diffusion Reconstruction adapts the diffusion-based OOD detection approach of Graham et al. [75] for the case of diffusion policies. Specifically, this baseline computes the reconstruction error on re-noised action sequences sampled from the diffusion policy asEQUATIONwhere ϵ i:0 θ denotes the reverse diffusion process from the i-th denoising iteration to the 0-th iteration, resulting in the reconstructed action. We approximate the expectation in Eq. 4 over a batch of B = 256 action sequences sampled from the diffusion policy, each re-noised for i ∈ {5,10,25,50} forward diffusion steps (also referred to as reconstruction depths). We experimented with several sets of reconstruction depths and found comparable performance. We note that this baseline comes with significant computational expense as it needs to perform the denoising process multiple times: i.e., if we would like to compute R reconstructions, this baseline is approximately R times more expensive than a single reverse diffusion process. The calibration and runtime procedures of this baseline are identical to those of STAC ( §4.1).that also computes the reconstruction error on re-noised action sequences sampled from the diffusion policy, but reconstructs the action sequences conditioned on the previous state s t asS(s t ,s t+k ) = E a 0 t+k:t+h-1|t+k ∼π t+k ,ϵ i ,i â0 -ϵ i:0 θ ( √ ᾱi â0 + √ 1-ᾱi ϵ i ,s t ) 2 .Here, â0 denotes the action sequence over which reconstructions are computed, concatenating the first k (executed) actions sampled at timestep t with following h-k (predicted) actions sampled at timestep t+k: that is, â0 = a t:t+k|t ⊕a 0 t+k:t+h-1|t+k . This step is necessary to ensure that the denoising process conditioned on s t only considers actions within the policy's prediction horizon. This baseline represents an alternative form of temporal consistency. Intuitively, it asks whether action sequences sampled at timestep t+k would also be sampled at timestep t, to which the answer is likely yes if the policy is in distribution, and likely no if the policy is OOD-because the marginal distributions conditioned on s t versus s t+k may be different. The hyperparameters of this baseline follow those of Diffusion Reconstruction.• DDPM Loss computes the empirical DDPM loss on re-noised action sequences sampled from the diffusion policy asS(s t ) = E a 0 ∼π(•|st),ϵ i ,i ϵ i -ϵ θ ( √ ᾱi a 0 + √ 1-ᾱi ϵ i ,s t ,i) 2 .Here, the expectation is taken over a batch of B = 256 sampled action sequences and 10 sampled denoising iterations i ∼ U[0,N ), where N is the total number of denoising iterations ( §B.2). We can think of this baseline as a more efficient version of Diffusion Reconstruction, since it directly quantifies the diffusion policy's performance on its training task without the need to reconstruct actions over numerous denoising iterations. The calibration and runtime procedures of this baseline are identical to those of STAC ( §4.1).• Temporal DDPM Loss is a temporal variant of DDPM Loss that also computes the empirical DDPM loss on re-noised action sequences sampled from the diffusion policy, but does so conditioned on the previous state s t asS(s t ,s t+k ) = E a 0 t+k:t+h-1|t+k ∼π t+k ,ϵ i ,i ϵ i -ϵ θ ( √ ᾱi â0 + √ 1-ᾱi ϵ i ,s t ,i) 2 ,where â0 = a t:t+k|t ⊕a 0 t+k:t+h-1|t+k (as defined in Temporal Diffusion Reconstruction). The hyperparameters of this baseline follow those of DDPM Loss, over which it is expected to offer advantages via temporal consistency.• Diffusion Output Variance computes the variance over B action sequences sampled from the diffusion policy and thresholds it w.r.t. the 1-δ quantile of sample variances computed over the calibration dataset D τ . This baseline reflects an alternative output metric to temporal consistency that can be monitored to detect policy failure. While computing output variances might bear resemblance to ensemble methods [12] , we note that this approach does not quantify epistemic model uncertainty. Doing so would require training multiple diffusion policies and performing inference with each at test time, which we avoid due to computational expense. The hyperparameters of this baseline are identical to those of STAC (see Table 3 ).Discussion on Baselines First, we highlight that the embedding-based approaches predict failure solely based on the dissimilarity or atypicality of the current state. Hence, these baselines are not policy aware: they may raise failure warnings for states that are dissimilar from those contained in the calibration dataset D τ without understanding how the policy behaves in those states. In some cases, the policy may still succeed or generalize to minor distribution shifts in state, causing the detection performance of these baselines to significantly diminish. The reconstruction-based approaches may account for the generalization characteristics of the policy but come with computational expense, which may prohibit their use in real-time settings. The DDPM loss approaches present the next best alternative to STAC, as their score functions coincide with the diffusion policy's training task and can be computed at negligible computational cost. However, we note that the DDPM loss baseline is specific to diffusion policies, whereas STAC is agnostic to the generative policy formulation.As described in §4.2, we propose to monitor the task progress of a generative policy by zero-shot prompting a VLM to analyze a video of the robot's execution up to the current timestep. We contrast the performance of our Video QA approach with a variation, Image QA, that queries the VLM using only I t , the image recorded at the current timestep t, rather than the full video I 0:t This baseline is used to evaluate the importance of video-based reasoning compared to single images. We construct the Image QA prompt by minimally modifying the Video QA prompt ( §A.2) as shown below:Prompt Template (Image QA)You are the runtime monitor for an autonomous mobile manipulator robot capable of solving common household tasks. A camera system captures image frames (at approximately 1Hz) of the robot executing its current task online. As a runtime monitor, your job is to analyze the most recent image frame and identify whether the robot is a) in progress of executing the task or b) failing to execute the task, for example, by acting incorrectly or unsafely.The robot's current task is to {DESCRIPTION}. The robot may take up to { TIME_LIMIT} seconds to complete this task. The current elapsed time is {TIME } seconds.Format your output in the following form: [start of output] Questions: First, generate a set of task-relevant questions that will enable you to thoroughly analyze the image frame and identify what actions the robot has taken so far. Answers: Second, precisely answer the generated questions, providing finegrained visual details that will help you accurately assess the current state of progress on the task. Analysis: Assess whether the robot is clearly failing at the task. Since the image frame only represents the robot's progress up to the current timestep and the robot moves slowly, refrain from making a failure classification unless the robot takes unsafe actions or is unlikely to complete the task in the allotted time. Explicitly note the amount of time that has passed in seconds and compare it with the time limit (e.g., x out of {TIME_LIMIT} seconds). Finally, based on the questions, answers, analysis, and elapsed time, decide whether the robot is in progress, or whether the robot will fail to complete its task in the remaining time (if any τ t ) = t-1 t ′ =0 R(s t ′ ,a t ′ ,s t ′ +1 ).We define policy failure simply in terms of task completion. More formally, given a defined success threshold R τ , the policy fails if the return on its trajectory τ t does not exceed the success threshold within the MDP horizon: R(τ t ) < R τ where t ≥ H. In the simplest case, the success threshold R τ equals 1, and the reward model R(s,a,s ′ ) equals 1 iff the task is complete at state s ′ . For example, if the robot is tasked with picking up a cup and receives a reward of 1 only once the cup is firmly grasped. In experiments, we adhere to this definition of policy failure and threshold trajectory returns as R(τ H ) < R τ to compute ground-truth labels for whether or not a policy failed in a trajectory τ H .Relation to failure detection: In §3, we provide a definition of the failure detection task-to detect whether a trajectory τ H constitutes a policy failure at the earliest possible timestep t-that is different from detecting the specific timestep at which (or before) the policy "fails." Doing so removes the need to manually specify task-specific failure criteria required to e.g., label each timestep in a trajectory. While the goal of our failure detectors is thus to flag failure episodes, it is still beneficial to catch failures at the earliest possible timestep, which is why a) our above definition of policy failure is formulated in terms of a partial trajectory τ t up to the current timestep t ≤ H of the MDP, b) we propose an online detection scheme that monitors for failure at each timestep t based on the trajectory up to the current timestep (i.e., f (τ t ) → {ok,failure}), and c) we report the detection time as a performance metric.Calibrating STAC and its baselines requires a small dataset of successful policy rollouts D τ = {τ i } M i=1 , which provide grounding on the nominal, in-distribution behavior of the policy. This allows us to evaluate the test-time behavior of a potentially failing policy w.r.t. its known nominal behavior.Calibration Data Quality We found it important to ensure the quality of trajectories τ i ∈ D τ . Specifically, trajectories in which the policy succeeds but in an undesired or unacceptable manner should not be used for calibration. For example, the policy may solve the Close Box task (Fig. 7 ) but damage the lids of the box in the process. Including such a trajectory in the calibration dataset would define this behavior as nominal and degrade the sensitivity of the detectors at test time. Returning to our example, the detectors may not raise a failure warning if the policy damages a box at test time.In practice, such a calibration dataset could be collected during a policy validation phase prior to deployment. For instance, we collect M = 50 successful policy rollouts for each simulation domain, manually filtering episodes where the policy succeeded with unacceptable behavior (e.g., with jitter). We hypothesize that the performance of the detectors w.r.t. the number of rollouts M is task specific. For example, a smaller calibration dataset may be sufficient for tasks with low variability (i.e., in a single, structured environment), while a larger dataset may be necessary if the policy is to be deployed at scale. We note, however, that increasing the calibration dataset size may be desirable to achieve stronger conformal guarantees on the detector's FPR (as derived in §D).Calibrating on Demonstration Data Finally, in attempt to eliminate the need to collect an additional calibration dataset of successful policy rollouts, we experimented with variants of STAC that directly calibrated on trajectories contained in the policy's demonstration dataset. However, doing so led to a significant increase in the detector's FPR. We attribute this to the well-known covariate shift problem for imitation learned policies [32, 33] . That is, their prediction error increases quadratically on states induced under the policy, causing the detectors' to mistake successful test-time rollouts for failures.Instead of evaluating the failure detectors online (i.e., during policy rollouts), we collect several test datasets of policy rollouts, which consist of both successes and failures. Each trajectory is labeled either success or failure by thresholding the return at the final state of the episode (as detailed in §B.4.1). We then perform offline evaluation of the failure detectors by invoking them at each timestep of the trajectory, which allows us to identify the first timestep at which the detectors issue a warning.We expand on the metrics outlined in §3. We first define a positive as a trajectory where the policy fails and a negative as a trajectory where the policy succeeds. A true positive is counted if the failure detector raises a warning at any timestep in a trajectory where the policy fails. A true negative is counted if the failure detector never raises a warning in a trajectory where the policy succeeds. The definitions for false positive and false negative follow accordingly. Detection time is defined as the earliest timestep in which the failure detector raises a warning in a trajectory where the policy fails.In our experiments, we report true positive rate (TPR), true negative rate (TNR), false positive rate (FPR), detection time (DT), accuracy, and balanced accuracy. TPR measures the number of true positives (detected failures) over total number of positives (failures). TNR measures the number of true negatives (detected successes) over total number of negatives (successes). FPR measures the number of false positives (false alarms) over the total number of negatives (successes). Accuracy and balanced accuracy account for both the TPR and TNR of the detector. However, we report balanced accuracy when the test set contains a non-negligible imbalance of positive and negative trajectories.C.1 Ablation Experiments on STAC Does STAC's performance depend on the policy's prediction and execution horizon?We conduct an ablation study on the PushT domain to test how the performance of STAC varies with respect to the prediction horizon h and execution horizon k of the diffusion policy. Together, the prediction and execution horizons determine the number of temporally overlapping action components (i.e., between a t+k:t+h-1|t and a t+k:t+h-1|t+k ) that are statistically compared by STAC, while the execution horizon governs how far apart in time the action distributions πt and πt+k are generated. The result is shown in Fig. 8 . We find that STAC (MMD) performs comparatively across execution horizons of k = 4 and k = 8, but performs best with the standard diffusion policy settings of k = 8 and h = 16 (used for the main result in Fig. 5 ). The detector's performance drops when using the smallest execution horizon of k = 2. We attribute this to the relatively small amount of environment change that occurs within two execution steps, which causes πt and πt+k to be similarly distributed and leads to overly conservative statistical distances. This is reflected in our results, where the detectors attain > 95% TNRs across various execution horizons, but using k = 2 leads to a significant drop in TPR to 61%, while k = 4 and k = 8 attain TPRs of 78% and 95%, respectively. To further ablate the choice of policy prediction horizon, we conduct a similar study on the simulated Close Box and real-world Push Chair domains. The result is shown in Table 4 , where we find that STAC is quite robust to the choice of prediction horizon, while the best result is achieved by using the standard setting of h = 16.Overall, STAC's performance may vary with the policy's execution horizon, but is relatively stable across choices of the policy's prediction horizon. The fact that we calibrate STAC and deploy it with the same prediction horizon may normalize the influence of this parameter at test time.How does STAC's performance vary with the choice of statistical distance function? Fig. 9 ablates STAC's performance across various choices of statistical distance functions in the PushT domain. Here, we find that STAC performs comparably across common choices like maximum mean discrepancy (MMD) with RBF kernels and KL-divergence via kernel density estimation (details in §A.1). This finding is corroborated in Table 5 , where all variants of STAC attain a detection accuracy of over 90% in the Close Box domain. In contrast, we observe a large performance drop when using a non-statistical distance function ("Temporal Non-Distr.Min"; §B.3.1) to measure temporal action consistency. This performance drop can be attributed to stochastic multimodality of the generative policy, which makes it challenging to sample individual actions that are similar to those at preceding timesteps during policy rollout. Because the non-statistical distance function is more sensitive than statistical distance functions to stochasticity in action sampling, we see an increased occurrence of false alarms.To supplement the analysis provided in §6, we herein focus on the performance of our VLM runtime monitor and its complementary role to STAC for the detection of erratic and task progression failures.Erratic Failure Analysis Table 5 presents the extended results of our experiments on the Close Box domain, where we aim to detect erratic policy failures that result from OOD scaling of the box. We run several evaluations of our VLM runtime monitor, varying the choice of VLM (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro [79] ) and prompt template (Video QA, Image QA). Since erratic failures in this domain are assigned to STAC-which detects 99% of them-we would like the VLM to avoid raising false alarms so as to keep the overall FPR of Sentinel low when the two detectors are combined. detector, STAC, detects 99% of erratic failures exhibited by diffusion policies. VLMs raise many false alarms when prompted with just a single image (Image QA), whereas performing Video QA (Ours) leads to a stark increase in TNR across all models. Performance also varies across the choice of VLM; GPT-4o is the most reliable in this domain, in contrast to other VLMs that struggle to accurately characterize the robot's task progress in the video. Overall, Sentinel detects 100% of failures, while combining STAC and the VLM increases false alarms to 13%.We first discuss the poor performance of the Image QA baseline. When prompted with just a single image, we find that all VLMs struggle to distinguish policy successes from failures and thereby do not exceed a TNR of 7%. Without observing the initial state (i.e., the box with its lids open) and the actions of the robot, the VLM is unable to identify the lids of the box and whether they have been closed. Thus, once the task's time limit is exceeded, the VLM simply declares failure. We iterated on several prompts that included detailed questions in attempt to coerce the VLM to reason about the location of the box and its lids, but this yielded negligible changes in performance. As a result, all the outputs of the Image QA baseline resemble the following example of a false positive:Analysis: The current observation shows the manipulator's arms positioned near the white box, with the grippers open and not grasping the lids. The two smaller white side lids and the bigger white back lid of the box are not visible, suggesting they are not yet folded. The elapsed time is 30 out of 30 seconds, which means the robot has reached the time limit for completing the task. Given that the lids are not folded and the task is not completed, the robot is clearly failing the task.Overall assessment: failurePrompting the VLMs in a Video QA setup ( §A.2) results in a strict increase in TNR across all models. Although, depending on the domain, we clearly observe that some VLMs show better visual reasoning performance than others. On the Close Box domain, GPT-4o generates relatively coherent descriptions of the videos, whereas Claude 3.5 Sonnet and Gemini 1.5 Pro often e.g., confuse closed lids for open lids or fail to recognize that the robot has made any significant progress. While this performance discrepancy is difficult to explain, two potential reasons are: a) the Close Box task can require reasoning over long videos (i.e., 20-30 image frames) and, while the VLMs' large context windows are accommodating, the VLMs may still be susceptible to recency bias [88] ; b) the images rendered in this domain might be better represented in the training data of one VLM (GPT-4o, in this case) compared to others.Task Progression Failure Analysis In the context of VLM runtime monitoring, task progression failures differ from erratic failures in that they are more visually apparent and hence simpler for the VLM to interpret. For example, the policy takes more obviously incorrect actions, e.g., clearly misplacing the cover in the Cover Object domain (see Fig. 7 ), but it does so in a temporally consistent manner that goes unnoticed by STAC. Therefore, under task progression failures, we require the VLM to attain both a high TPR and TNR, whereas we are mainly concerned with TNR under erratic failures.The extended results of our task progression failure experiments are shown in Table 6 for the Cover Object domain and Table 7 for the Close Box domain, the two of which are aggregated in Fig. 6 .system that frequently raises false alarms is impractical to use. Our calibration approach does not guarantee the detection of failures, nor does it guarantee that we do not issue false alarms on OOD successes, as this is not possible without any distributional assumptions on the OOD scenarios or without using failure data for calibration [91] . Instead, we empirically find that our temporal consistency score performs amicably at detecting failures in our experiments.2. Proposition 2 only certifies that the FPR of STAC is low. We make no claims on combined performance of STAC and the VLM, as VLM represents a black-box classifier. Future work could investigate methodologies to jointly calibrate an ensemble of failure detectors.3. Conformal guarantees, like those in Theorem 1 and Proposition 2, are marginal with respect to the calibration data. That is, they may not hold exactly when given a particular calibration dataset, but if we were to sample thousands of calibration datasets, the guarantees would hold on average. Thus, as expected, STAC does not exactly satisfy Eq. 5 in our results, as compute budgets restricted our experiments to repetitions on a limited number of random seeds.Due to the iterative denoising procedure of the diffusion policy, analytically computing a distance D(πt,π t+k ) is challenging, as evaluating the densities of πt and πt+k requires marginalizing out the intermediate diffusion steps and the non-overlapping actions. Instead, we approximate D with its empirical counterpart D by sampling a batch of action sequences (parallelized on a GPU) at each timestep t and t+k rather than a single action sequence.Online runtime monitoring requires the VLM to differentiate whether a) the robot is still in progress of executing the task correctly (i.e., partial progress) or whether b) the robot will fail to complete the task (e.g., by stalling in a partially completed state). Differentiating between partial progress and task failure can be ambiguous for a slow moving robot, and thus, providing the VLM with the current elapsed time serves as a reference to gauge whether or not the rate at which the robot is executing the task will result in a timely task completion. 


Decision: Strengths: 
- The paper addresses a critical challenge in depolyment of robot policies. 
- Theoretical foundation of the work seems to be relatively sound. 

Weaknesses:
- The clarity of presentation should be improved. 
- The categoriztion of the presented binary failure modes is not convincingingly descirbed. 
- Real-world evaluation would be beneficial.

Reviews: review: title: title: Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress
paper_summary: paper_summary: The paper proposes a failure detection scheme that concurrently assesses temporal action consistency and task progress. The former is determined by the novel STAC model, while the latter is evaluated using a proposed sequence image-based VLM model.
main_review: main_review: The paper proposes a failure detection scheme that concurrently assesses temporal action consistency and task progress. The former is determined by the novel STAC model, while the latter is evaluated using a proposed sequence image-based VLM model.
In the problem setup, policy failure is defined as the event when a trajectory yielding a lower reward than expected, and failure detection is formulated as a binary classification task {ok, failure}.

some comments are listed below:

- The reviewer finds a discrepancy between the definition of failure in the problem setup and the proposed scheme. The paper lacks analytical or experimental evidence demonstrating that the proposed architecture (action consistency + task progress) aligns with the notion of policy fault, characterized by lower-than-expected reward, as defined in the problem setup.

- The reviewer finds the approach of concurrently checking task progress and temporal action consistency to be intuitively sound for robust failure detection. Moreover, the proposed STAC and sequence image-based VLM models appear to be technically sound, each showing promising results when evaluated independently.

- In the overall architecture (Figure 2), the final failure decision is determined solely by an "OR" logic gate, combining the results of temporal action consistency and task progress assessments. However, in the reviewer’s understanding, STAC can cause significant false alarm because it merely checks for plan consistency, not plan effectiveness. In this point of view, the “OR” logic seems to be not sufficient because the false alarm of one can cause entire malfunction (e.g., increased FPR).

- The technical description of the VLM-based task progress check model is lacking in detail.

- In addition to the summarized statistical results, it would be helpful to visually illustrate the trajectory, decision-making process, and failure detection behavior over time for several specific experimental cases. This would provide a clearer understanding of how the proposed method operates in practice.
questions: questions: 1. Misalignment between Failure Definition and Proposed Scheme:
  - A description of link between the proposed architecture (STAC + VLM) and the reward-based definition of failure.

2. Potential False Alarms from detectors and the Limitations of the "OR" Logic:
 - Provide a more robust justification for why "OR" logic are adopted, despite the risk of false alarms.
 - or, explore alternative decision-making mechanisms, such as weighted combinations or more sophisticated logic gates, to mitigate this risk.

3. Insufficient Technical Description of the VLM-based Task Progress Check Model:
  - more detailed and comprehensive description of the VLM-based task progress check model will be helpful

4. Lack of Case Study Illustrations:
  - In addition to the summarized statistical results, it would be helpful to visually illustrate the trajectory, decision-making process, and failure detection behavior over time for several specific experimental cases.
limitations: limitations: 2
review_summary: review_summary: While the individual modules comprising the architecture are technically sound and the results of individual module appear promising, there is room for improvement in the proposed overall architecture. I recommend a weak accept.
title: title: Good but confusing read
paper_summary: paper_summary: This paper introduces a framework for detecting and monitoring failures in robot policies, specifically generative policies like Diffusion Policies (DP). It classifies failures into two types: time-sensitive erratic failures and less time-sensitive task progression failures. To detect erratic failures, the framework recommends comparing the policy's action distributions across different time steps using the proposed Statistical measure of Temporal Action Consistency (STAC). For monitoring task progression and evaluating success, it suggests utilizing foundation models, particularly Vision-Language Models (VLM), to assess the policy's effectiveness in achieving its objectives.
main_review: main_review: The paper addresses a significant challenge in deploying robot policies, particularly in scenarios that may be out-of-distribution. Categorizing potential failures based on time sensitivity is a sound and logical approach. The use of Vision-Language Models (VLM) for failure detection aligns with current practices of leveraging foundation models.

However, while the theoretical foundation of the proposed approach is reasonable, the paper suffers from a lack of clarity, particularly in sections 3 and 4, which makes it difficult to read. There are also minor typing errors scattered throughout the main text (e.g., "Model-based methods are not not directly applicable" (line 79), "timesteps of the MPD" (line 112), etc.). Additionally, the work seems to reference generative policies (DP) heavily but it lacks sufficient background information about DP or generative policies in the related works section, making it challenging for readers without expertise in generative policies to comprehend. I still could not really understand why this work is necessary for tackling failure detection in generative policies in specific. Enhancing the explanation (e.g. what is $\epsilon^i$ in Eq. 1) and presentation of the concepts would help in better understanding and application of the framework.

In conclusion, the proposed framework in the paper is a valuable addition to existing work on failure detection but improved clarity and presentation are necessary.

Strengths:
1. Sound approach using a combination of STAC and VLM for identifying time-sensitive policy instability and long-term task progression.
2. Use of foundation model for long-term classification (success/failure), aligning with recent approaches of leveraging LLMs for works beyond NLP.

Weakness:
1. Lack of clarity and poor presentation, making reading difficult and confusing at times.
questions: questions: 1. In section 3 on the paragraph regarding failure detection, the authors state their goal is "to construct a failure detector $f_\phi(\tau_t) \rightarrow \\{\text{ok, failure}\\}$ that at each timestep $t$, can provide a classification as to whether the policy will fail if it continues executing for the remaining $H - t$ timesteps of the MDP. Here, $\phi$ denotes the parameter of the failure detector". The use of the term "parameter" is misleading in the context of learning since $f$ is not a neural network. Instead, the approach relies of the STAC, which is a statistical measure of difference in the policy probabilities, and the VLM, which does not appear to be fined-tuned for this task. It took quite a while to realize that $\phi$ is the temporal consistency score (if I did not misunderstand). Perhaps, it will be better to clarify the failure detection module with suitable wordings. Additionally, while the stated goal is failure prediction (i.e., determining if the policy will fail if it continues), the presented approach is a reactive detection mechanism, with the STAC assessing the stability of the policy and the VLM determining it efficiency up to the current timestep.

2. How effective is the VLM in detecting policy failures? Unlike a cited reference that fine-tunes VLMs with task-related trajectories, the proposed approach does not fine-tune the VLM. What was the reason for this decision? According to Table 1, the True Positive Rate (TPR) of GPT-4o Video is on the lower end (in comparison to GPT-4o Image), while the True Negative Rate (TNR) is high. Does this imply that the VLM often fails to raise a warning when the policy fails, or that it generally tends not to raise any warning at all? Do the authors believe that better accuracy could be achieved through fine-tuning?

3. The proposed approach is a naive combination of two types of detectors using a logical OR. Do the authors have any thoughts on future works regarding enhanced combination that can be more effective or efficient?
limitations: limitations: 3
review_summary: review_summary: The presented approach can be beneficial for improving failure detection in robotic environments, but I believe great amount of work needs to be done to improve clarity and presentation.
title: title: A runtime monitoring framework that detects failures samples from generative imitation policy
paper_summary: paper_summary: This paper first divides generative policy failure cases into two categories: temporal inconsistency due to OOD sampling and lack of progress despite consistent actions. Secondly the paper addresses these two categories of failures with a runtime STAC metric and a VLM. The paper shows the combination of both monitoring mechanism detects policy failures the best.
main_review: main_review: Strength:

The paper addresses an important problem that often times imitation policies tend to drift and cause out-of-distribution failures. The ability to categorize failures and detect them early on is important and useful. The paper presentation is clear and figures are illustrative of the high-level ideas.

Weakness:

The paper needs to motivate better why monitoring generative policies require new tools compared to existing failure detection methods used for other policies. In particular, there is a lack of justification why the failure cases for generative policies can be exhaustively represented by two categories: lack of consistency and progress. While the paper use high true positive rate, high true negative rate, and detection time to measure the success of using STAC or VLM to detect the above two issues, it would be better to come up with metrics to evaluate how well STAC detects a lack of consistency and how well VLM detects a lack of progress separately. Lastly, the paper might need more than three experiments to demonstrate the universal usefulness of their runtime detection pipeline. Especially real-world experiments should be considered.
questions: questions: 1. Can you show more tasks (more than the three tasks considered right now) and especially the failure cases of each task in videos to help readers understand the two category of failures in action?
2. Can you show in video how STAC catches inconsistent action prediction due to mode oscillation? I am aware you have a video of it. But neither could I visually identify the events that cause the detector to report policy failure, nor could I see the multimodal oscillatory behavior in the video. 
3. Will action prediction horizon impact STAC?
4. Can you show videos of the remaining failure cases that the combination of STAC and VLM still cannot catch?
limitations: limitations: 3
review_summary: review_summary: Good paper with some weaknesses mentioned above. Addressing these concerns will position the paper better for acceptance.