Failure Prediction from Few Expert Demonstrations

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-06 01:27:15

Abstract: This extended abstract presents a novel three-step methodology for discovering failures that occur in the true system by using a combination of a minimal number of demonstrations of the true system and the failure information processed through sampling-based testing of a model dynamical system. The proposed methodology comprises a) exhaustive simulations for discovering failures using model dynamics; b) design of initial demonstrations of the true system using Bayesian inference to learn a GPR-based failure predictor; and c) iterative demonstrations of the true system for updating the failure predictor. As a demonstration of the presented methodology, we consider the failure discovery for the task of pushing a T block to a fixed target region with UR3E collaborative robot arm using a diffusion policy and present the preliminary results on failure prediction for the true system.

Main paper: 
 introduction: 
 Testing and validation are essential tools to ensure the safety of autonomous systems prior to deployment [1, 2, 3, 4] . Most of the state-of-the-art tools for model-based validation and falsification of the autonomous system assume access to the true system [5] . These model-based tools mainly use sampling-based methods for failure discovery [6, 7, 8] . While sampling-based techniques allow efficient exploration of the search-space, they require large number of samples to work efficiently, and are therefore well suited for simulation based testing. Most of these approaches assume that the model dynamics and simulation testing environment adequately represent realistic testing conditions. However, this can be misleading, since a sim-to-real gap can lead to unexpected failures that were unobserved in the simulation environment on which the policy was trained [9] . Additionally, uncertainties in state estimation and dynamics can also affect the performance of the autonomous systems. Collectively, these issues lead to failure modes that remain undiscovered, despite exhaustive simulation testing. The resulting sim-to-real gap is especially concerning from the perspective of safety, as the discovered failure modes in simulation may not reflect the true severity of real failures, i.e. a failure not reported as unsafe behavior in simulation may be unsafe and catastrophic for the true system. In this study, we analyze the sim-to-real gap from the perspective of falsification, using a sampling-based testing pipeline for simulation for efficient exploration of failures, while working with limited data from the true system to enable better prediction of failures. 
problem formulation: 
 Consider a discrete-time closed-loop dynamics:EQUATIONwhere f : R n × R m → R n and C ∈ R l×n ,Predicted risk using model dynamics output y t ∈ R l , where ϵ 1 and ϵ 2 are disturbances in dynamics and state estimation, respectively. The environment variable z represents variables that can be independently controlled by the user, such as initial conditions of the system x 0 , and environmental information exogenous to the agent. In this work, we assume that the disturbances come from zero-mean Gaussian distributions given by ϵ 1 ∼ N (0, Σ 1 ) and ϵ 2 ∼ N (0, Σ 2 ), where the covariance matrices Σ 1 ∈ R n×n , Σ 2 ∈ R l×l of the distributions are defined using scalars σ 1 , σ 2 > 0 as Σ i = σ i I, for i = 1, 2, where I is an identity matrix of the appropriate size. We consider two set of dynamics in this paper: model (known to the user) and true (unknown to the user). True dynamics corresponds to the actual dynamics of the agent, which is unknown whereas Model dynamics corresponds to the estimate of the true dynamics and is described as in (1) . We denote a trajectory rollout of (1) for a given environment variable z under a given (σ 1 , σ 2 ) as X (z|σ1,σ2) = (x i ) T i=0 . 1 The trajectory rollout of the true dynamics for a given environment variable z is denoted as X * z .In this work, we address the problem of discovering failures of the true dynamical system with limited demonstrations. For this purpose, we consider a user-defined risk function R : R d → R where R(z) = R(z, X z ) denotes the risk corresponding to the trajectory rollout X z for a given environment variable z. Based on this risk function, we define failure of the system when the risk R(z) for a corresponding z exceeds a user-defined threshold R th ∈ R. As thus, the falsification problem can be mathematically formulated as discovering the setZ * fail := {z | R(z, X * z ) > R th }.We aim to solve this under the constraint that we can query the true system only a few times N > 0 to obtain N trajectory rollouts {X * zi } N i=1 for z i ∈ Z. We present a three-step methodology to discover failures occurring in the true system by using a combination of a minimal number of demonstrations {X * z } and the failure information from the model dynamics obtained through sampling-based falsification. 
methodology: 
 We assume that the model can capture a subset of the failures that could occur with the true system, i.e., Z fail (f ) ∩ Z * fail ̸ = ∅. Based on this assumption, we obtain that the Z * fail ⊆ Z fail (f ) ∪ Z real , i.e, failures of the true system are a combination of algorithmic failures on the model system Z fail (f ) and failures due to the mismatch between model dynamics and actual dynamics, disturbances and other unknown reasons Z real . We say that the set Z fail (f ) captures algorithmic failures as we assume that the policy π is trained for the model f but still leads to failures. The first step of our methodology focuses on identifying failures that can be obtained using the model information.We define the set of the environment variables for the algorithmic failures as:EQUATIONwhich captures the algorithmic failure of the model dynamics. This set can be readily obtained through extensive simulations using the model information. Next, we aim to capture the failures due to the mismatch between model dynamics and true dynamics, disturbances, and potentially other unknown reasons. For this, we sample σ 1 , σ 2 from a bounded region given byB ∈ [σ min 1 , σ max 1 ] × [σ min 2 , σ max 2 ], and observe the risk R(z, X z ) corresponding to each disturbance, and collect values of z for which a failure is observed across all disturbances:EQUATIONso that Z noise ⊂ Z real . The region Z risk := Z noise ∪ Z fail (f ) captures failures that can be discovered using model dynamics. Next, we aim to discover failures that the model system cannot capture through sampling z from the region Z real and obtaining demonstrations from true dynamics. Since Z real is not known, we obtain these samples from the region Z\Z risk as discussed in the next section.Since we have a limited budget on the number of demonstrations we can obtain from the true dynamics, we aim to maximize the state-space covered in each of these demonstrations. For a given z, we define a coverage function, given by C : Z → R given as C(z) = C(z, X z ), which is a monotonic function of the state-space explored along the trajectory X z , and aim to sample z ∈ Z\Z risk from a distribution P corresponding to high coverage:EQUATIONwhere C th > 0 is a user-defined coverage threshold. Directly sampling from this distribution is intractable, so we use a Bayesian inference framework [7] with the posterior distribution as defined in (4). To ensure that the generated samples lie in the region Z\Z risk , we use a Normalizing Flows based framework for classification, called Flow-GMM [10] to learn Gaussian distributions in latent space W ⊆ R d corresponding to the sets Z risk and Z\Z risk , given by p1 and p2 respectively, and reconstruct the Bayesian inference in W. We then use Metropolis-Hastings algorithm [11] to sample from the defined posterior distribution and apply a projection operator on the generated samples to ensure that we sample exclusively from p2 . The details of posterior construction and projection can be found in Appendix B. The pipeline discussed so far generates a collection of samplesZ cov = {z ∈ Z\Z risk | C(z, X z ) > C th }.Once we have generated the samples, we choose N/2 candidate values of z distributed uniformly across the search-space. This is achieved by dividing Z cov into N/2 clusters using K-means clustering, and choosing the geometric centers of the generated clusters for demonstrations. This allow us to collect N/2 data points Z 1 = {z j } with corresponding risk values given by R 1 = {R(z j , X * zj )}. We also obtain M data points Z 2 = {z i } from the region Z risk , with the corresponding risk values given by R 2 = {R(z i , X zi )} using model dynamics f . We defineD 1 = [Z 1 , R 1 ] and D 2 = [Z 2 , R 2 ]as the dataset of demonstrations obtained from true and model system, respectively, and use them to train a model ϕ θ : Z → R to predict risk R = ϕ θ (z) for a given z, as illustrated in the next section where θ denotes the model parameters.Motivated by the success of Gaussian Process Regression (GPR) in learning from limited demonstrations [12] , we use GPR as the backbone of the risk prediction pipeline in this section. Using the dataset D f = D 1 ∪ D 2 , we construct the marginal log likelihood log p ϕ θ (R f |Z f ) for learning the model ϕ θ using a sum of the marginal log likelihoods from both sources of data as: whereEQUATIONZ f = [Z 1 , Z 2 ] and R f = [R 1 , R 2 ].The training objective can be formulated as the maximization of the marginal log likelihood in ( 5) with ϕ θ as the decision variable:EQUATIONWe first learn a model ϕ θ0 for risk prediction using D 1 and D 2 and subsequently refine the model by a sequence of N/2 demonstrations on the true system with sequential optimization of ϕ θ solving (6) with the updated dataset and generation of data-point for the next demonstration. The details of sequential demonstration and risk prediction update can be found in Appendix C. 
falsification of diffusion policy on push-t: 
 As a demonstration of the our methodology, we consider the task of pushing a T-block to a fixed target region with a circular end-effector using a diffusion policy from [13] which predicts actions conditioned on observations. Fig. 4 in Appendix D shows the setup corresponding to the model and true system respectively. Appendix D has details of the model dynamics and experimental setup.We restrict the number of hardware demonstrations to N = 20. Fig. 1 (see the plot under 'Discovering algorithmic failures') shows the region Z risk discovered in simulation using the model dynamics.For validating the learned failure prediction using the learned model ϕ θ , we record the risk for 10 randomly sampled test demonstrations on the the true system, and compare against the predictions from our method and two other baselines. Table . 1 shows the risk prediction error with GPR using three methods, namely, data collected only using model dynamics (reported as Simulation), data corresponding to Z risk ∪ Z fail from model dynamics and uniformly chosen N data points on the true system (reported as Simulation+Exp (I)), and data collected using our approach (reported as Simu-lation+Exp (II)). The individual predictions from all three approaches and their comparison against the ground truth (risk from true system) is shown in the right plot in Fig. 2 . Appendix D provides a detailed analysis of the results summarized in this section. We also validate the learned risk by conducting demonstrations on 10 data points sampled from predicted high-risk and predicted low-risk regions. The predicted and ground truth values of risk corresponding to these points is shown in the left plot in Fig. 2 . As we can see, the GPR model accurately predict 'fail' and 'not fail' across all 10 data points, where failure for a chosen value of z corresponds to the predicted risk being higher than the threshold R th = 0.3.Testing and model validation are are essential tools for ensuring safety of autonomous systems before deployment. There are several model-based and model-agnostic tools for testing and falsification that have been proposed in the literature [1, 2, 3, 4, 5] . The model-based tools have the advantage of being faster [6, 8] , and with differentiable system models, gradient-based optimization strategies can be used for falsification, which have shown to be more efficient than black-box methods [8] and adversarial optimization [14, 15] . The model-agnostic tools on the other hand, do not rely on model information to discover failure modes [16, 17] .Recently, some works have considered simulation-based falsification and testing [8, 18] . These methods minimize the dependency on hardware-based for testing, which can be extremely timeconsuming and require extensive utilization of resources [19] . Most of these approaches assume that the simulated dynamic system and testing environment represent the realistic testing conditions adequately. However, this can be misleading, especially true for the case of learning-based policies, where a sim-to-real gap can lead to unexpected failures that were unobserved in simulation environment, on which the policy was trained [9, 20] . Additionally, there are uncertainties in state estimation and dynamics, that affect the performance of cyber-physical systems. Collectively, these issues leads to failure modes that remain undiscovered, despite exhaustive simulation testing, due to the inadequacy of the simulation models to capture wide-ranging practical phenomenon that affect the real-word system.In this study, we analyze the sim-to-real gap from the perspective of falsification, using existing testing pipelines for simulation, while working with limited real-time data to enable better prediction of failures.In (4), we use population variance of the trajectory rollout X z as a coverage metric C. To sample exclusively from Z\Z risk , we learn the decision boundary that distinguishes Z risk from the remaining search space by performing supervised binary classification using samples collected in the previous step. For this, we use a specific technique known as Flow-GMM [10] , which allows us to learn an invertible mapping from search-space Z to a latent-space W ∈ R d , given by g -1 θ : Z → W using the Normalizing Flows framework [21] . Using Flow-GMM, we learn isotropic Gaussian latent distributions corresponding to the base distributions p 1 = p Zrisk and p 2 = p Z\Zrisk which can be mathematically expressed using mean µ g i and covariance Σ g i as pi = N (µ g i , Σ g i ) for i = 1, 2.Sampling directly from the distribution generated by ( 4) is intractable, hence, we utilize a Bayesian inference framework here. Additionally, we make use of the learnt mapping g -1 θ : Z → W to simplify the posterior for sampling in the latent space W:EQUATIONThe conversion of distribution from (4) to ( 7) is done because sampling z from the base distribution p 2 can be challenging as p 2 might be multi-modal. However, the corresponding latent distribution p2 is unimodal, thereby allowing us to apply projection using a projection operator P[w] to ensure that a generated sample w remains within a prescribed convex set centered at the mean µ g 2 of p2 . This ensures that samples are only drawn from the set Z\Z risk . The construction of convex boundary and the corresponding projection operator has been discussed in the next section. We utilize exponential modelling for expressing the likelihood p(C > C th |g θ (w)), adopted from [7] and p2 as the prior to construct the posterior for sampling. The sampled latent environment variable w is obtained as w = P[w] where:EQUATIONWe use Metropolis Hashtings to sample w from the constructed posterior distribution, which is a gradient-free sampling method, as application considered in this paper corresponds to a nondifferentiable dynamic system (Section-4). The pipeline discussed so far generates a collection of samples While we are sampling from the latent distribution w ∼ p2 , the goal is to sample strictly from the region Z\Z risk , which is given by g θ (w). For the learnt latent distributions, there exists a set P, such that:Z cov = {z ∈ Z\Z risk | C(z, X z ) > C th }.EQUATIONFor a point lying outside this region, we may have p1 (w) > p2 (w), which means that the generated sample corresponds to the region Z risk . Hence, we use the boundary P to generate samples such that the corresponding z lies in desired region. We first construct an explicit boundary P and subsequently utilize it for projection. For an isotropic Gaussian distribution N (µ, σ), where the covariance is given by Σ = σI, there exists r ∈ R, c ∈ R d , such that P = {w| ∥w -c∥ 2 2 ≤ r 2 } meets the requirement specified in (9) . Note that the value of r that satisfies (6) for a given c is not unique, and can be decreased to make the sampling more conservative or vice versa. In our simulations, we chose c = µ g 2 which is the mean of the Gaussian distribution p2 and a user-defined hyperparameter in the training of Flow-GMM. The chosen P is convex, for which a projection operator can be constructed as:EQUATION)It can be easily verified that P [w] lies on the boundary of P, i.e, ∥P [w] -c∥ 2 2 = r, and hence projects any point w / ∈ P onto the boundary. Fig. 3 (right) shows the latent distribution learnt using Flow-GMM corresponding to Z risk and Z\Z risk , and the boundary of the constructed set P in red.Here, d = 2, and r = 2.We first learn a model ϕ θ0 for risk prediction using initially chosen set of N/2 demonstrations from the true system and data corresponding to algorithmic failures and noise (Z risk ) from the model.We then refine the model by a sequence of N/2 demonstrations on the true system with sequential optimization of ϕ θ solving (6) In what follows, we suppress the explicit dependence on σ1, σ2 for the sake of brevity. 
 
 Appendix:  


Decision: Reviews are mixed. One is very short and has not much content, the other two have relatively low scores but the actual text is more positive. Therefore the scores are likely down to personal reviewing style and not the merit of the work. On this basis, I recommend accepting the work.

Reviews: review: title: title: Interesting method for efficient failure discovery in true systems
main_review: main_review: This paper paper presents a very interesting and useful methodology for efficiently discovering failures in the true system. 

The motivation of the paper is clear as it is obvious to me why practitioners would need to estimate failures with a minimal number of demonstrations of the true system. 

The problem formulation clearly defined and easy to understand. 

The author's proposed methodology is intuitive and easy to follow. Figure 1 nicely illustrates the proposed three-step pipeline proposed by the authors. The proposed method presents a novel way of utilizing Bayesian inference that I think will be of interest to people at the BDU workshop. 

The chosen example of pushing a T block to a fixed target region is interesting and nicely shows the effectiveness of the author's method. However, I note that additional examples would be essential to show the robustness of this method across tasks, and I suggest the authors add additional examples and results before preparing this work for a main conference submission in the future. Additionally, results showing the limitations of this method would be a very useful addition. For example, how does the method perform when 1) the sim-to-real gap becomes larger, 2) the number of of demonstrations of the true system becomes more limited. 

Overall, the paper presents an interesting approach applying Bayesian machine learning to solve a relevant problem, and I think the audience at the BDU workshop will find this interesting. I therefore vote to accept the paper. However, in the future significant work remains to be done to strengthen the results and show both the effectiveness and limitations of the proposed method.
title: title: The paper seems to be somewhat unclear and incomplete.
main_review: main_review: Thanks for submitting this interesting paper.

The overall idea of the paper is good, but it seems to miss some important background information or references for the readers without specific knowledge of the problem setup in the paper to understand.

The abstract for this paper is a bit unclear, and the paper seems incomplete since the last section (section 4) does not have a conclusion or an open discussion.
title: title: Very interesting; however, the notation and thinking of edge cases leave something to be desired
main_review: main_review: ## Review

- Effectiveness of the coverage function $C(z)$ relies on its ability to capture meaningful exploration of the state space. An improperly defined coverage metric might lead to suboptimal sampling.
- Very interesting application of Bayesian inference using Normalizing Flows (Flow-GMM) to sample $z$ values that are likely to provide high coverage and are outside the known risk regions ($\mathcal{Z}_{\text{risk}}$).
- Scalability questionable: Gaussian Processes work well with small-ish datasets, the computational complexity ($\mathcal{O}(n^3)$) scales poorly with the number of data points. Might be a bottleneck for larger systems requiring more demos.
- Regions where the system's behavior is highly stochastic may be difficult to capture / maximizing coverage may overlook rare but critical failure modes that do not significantly contribute to the coverage metric.

<br>

- I wonder whether using zero-mean Gaussian distributions for $\epsilon_1$ and $\epsilon_2$ is entirely optimal with respect to modeling some approximation of "real-world" disturbances. Also in highly nonlinear systems, even Gaussian input disturbances can lead to non-Gaussian behavior in the state variables.
- Addressing potential sim-to-real gaps (possible lower-fidelity simulations due to assumptions / software used [e.g. PyMunk]) would be beef up the central argument: "Does the software work as intended" risk.
- A very interesting paper overall, a good pitch in my estimation.
- I may lack some mechanical engineering/physics training to fully appreciate the mechanics aspects of the robot arm/robotics.

<br>

- In Table 1, providing uncertainty intervals or detailed error analysis could help clarify why the Max Error increased from "Simulation + Exp (I)" to "Simulation + Exp (II)".
- The paper should provide a clear definition or example of how the coverage function $C(z)$ is computed from a trajectory $X_z$ - important for understanding the sampling strategy
- Also, how is the risk function $R(z)$ calculated? Be nice to have in the appendix at least.
- Should provide additional explanation/examples of how the sampling and projection in the latent space are performed / the transformation to the latent space $W$ using $g_{\theta}^{-1}$ and the use of Normalizing Flows could use a little elaboration
- The projected point $\mathcal{P}[w]$ should satisfy: $\|\mathcal{P}[w] - c\| = r$. Squaring the denominator like in the paper does not result in a unit vector. Instead, seems to me the correct projection operator is: $\mathcal{P}[w] = c + r \frac{w - c}{\| w - c \|}$.