RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields
Authors: Som Sagar
Jiafei Duan
Sreevishakh Vasudevan
Yifan Zhou
Heni Ben Amor
Dieter Fox
Ransalu Senanayake
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Multimodal Embedding
Primary area: reinforcement learning
TL;DR: A method to identify and quantify vulnerabilities in manipulation policies.

Abstract: Robot manipulation policies, while central to the promise of physical AI, are highly vulnerable in the presence of external variations in the real world. Diagnosing these vulnerabilities is hindered by two key challenges: (i) the relevant variations to test against are often unknown, and (ii) direct testing in the real world is costly and unsafe. We introduce a framework that tackles both issues by learning a separate deep reinforcement learning (deep RL) policy for vulnerability prediction through virtual runs on a continuous vision-language embedding trained with limited success-failure data. By treating this embedding space, which is rich in semantic and visual variations, as a potential field, the policy learns to move toward vulnerable regions while being repelled from success regions. This vulnerability prediction policy, trained on virtual rollouts, enables scalable and safe vulnerability analysis without expensive physical trials. By querying this policy, our framework builds a probabilistic vulnerability-likelihood map. Experiments across simulation benchmarks and a physical robot arm show that our framework uncovers up to 23\% more unique vulnerabilities than state-of-the-art vision-language baselines, revealing subtle vulnerabilities overlooked by heuristic testing. Additionally, we show that fine-tuning the manipulation policy with the vulnerabilities discovered by our framework improves manipulation performance with much less fine-tuning data.

Published as a conference paper at ICLR 2025
ROBOMD: UNCOVERING ROBOT VULNERABILITIES THROUGH SEMANTIC POTENTIAL FIELDS
Som Sagar
Jiafei Duan University of Washington
Sreevishakh Vasudevan Arizona State University
Arizona State University ssagar6Basu.
Yifan Zhou
Heni Ben Amor Arizona State University
Dieter Fox University of Washington
Arizona State University
Ransalu Senanayake Arizona State University
ABSTRACT
Robot manipulation policies, while central to the promise of physical AI, are highly vulnerable in the presence of external variations the real-world. Diag- nosing these vulnerabilities is hindered by two key challenges: (i) the relevant variations to test against are often unknown, and (ii) direct testing in the real world is costly and unsafe. We introduce framework that tackles both issues by learning a separate deep reinforcement learning (deep RL) policy for vulnera- bility prediction through virtual runs continuous vision-language embedding trained with limited success-failure data. By treating this embedding space, which is rich in semantic and visual variations, as potential field, the policy learns move toward vulnerable regions while being repelled from success regions. This vulnerability prediction policy, trained on virtual rollouts, enables scalable and safe vulnerability analysis without expensive physical trials. By querying this policy, our framework builds probabilistic vulnerability-likeliee likelihood map. Ex- periments across simulation benchmarks and a physical robot arm show that our framework uncovers up to 23% more unique vulnerabilities than state-of-the-art lasion-language baselines, revealing subtle vulnerabilities overlooked by heuristic testing. Additionally, we show that fine-tuning the manipulation policy with vul- nerabilities discovered by framework improves performance with much less data. GitHub:
INTRODUCTION
Learning robust robot manipulation policies is widely regarded as the foundational problem in phys- ical AI. A robust solution would unlock capabilities ranging from reliable industrial automation in cluttered factory environments assistive humanoid arms that seamlessly interact with people in everyday settings. In perception and language tasks, vulnerabilities can often be quickly identified by querying large datasets benchmarks, with little cost beyond computation. In manipulation, however, discovering vulnerabilities is far more difficult: it requires physical trials that are slow, expensive, and potentially unsafe, posing risks not only the robot but also to the environment and even people. This makes naive heuristic testing or trial-and-error both impractical and costly, motivating the need for scalable, active vulnerability exploration methods pre-deployment. this paper, pursue this by training separate vulnerability prediction model, formulated deep reinforcement learning (deep RL) policy that actively searches for failures.
Yet, even with scalable search framework, a second obstacle remains: what exactly should we test for? Manipulation policies must withstand diverse and unpredictable variations. For instance, referring to Fig. [ robot designed to grasp a bottle should generalize across various colors. shapes, sizes, and materials, and remain effective under changes in lighting, background, and physical lay- outs Pumacay et al. (2024); Xie et al. (202)). Naively applying deep RI. to handful of known

---

Published as a conference paper at ICLR 2025
Environment
Reward
RoboMD finds
Diagnosis Report
State
vulnerabilities R
Failure Ranking (testing unseen variations)
object
Action
Diognasing
<p
RoboMD MD
Unseen
Action Samples
Fallure Probabilities (when all variations are known)
Diflesion Puliyy
In Real World
In Simulation
claving
The vulnerability diagnostic pipeline. We train RoboMD (TTD)), deep reinforcement learning (RL) policy, by probing a pre-trained manipulation policy (TR) while systematically apply- ing environmental variations (e.g.. background colors, object shapes, darkness levels, etc.) as RL actions, examples which are shown in the "Action Samples" panel (bottom left). Note that these examples for intuitive understanding only; the user does not need to specify all variations. By observing the success or failure on each trial of πR. RoboMD learns to identify vulnerabilities even in unseen variations (c.g., a red cube, in this example). The final output is a diagnosis report that can either provide a ranked list of failure likelihoods, including for previously unseen variations (top right), quantify the failure probabilities for previously seen variations (bottom right).
Figure 1:
variations to search for failures risks overlooking critical failure modes that emerge under novel conditions: Lin et al. (2024).
We overcome these limitations by reformulating deep RL exploration over a learned, continuous vision-language embedding space rather than discrete, hand- sandd-secified variations. Constructed from limited success-failure data, we ensure that this embedding space is rich in semantic and visual structure. We cast the learned space as potential field that guides exploration toward failures and away from successes. This, in turn, allows that organically uncovers relationships and adapts to diverse variations. This policy can be queried at any time predict whether scenario will lead to failure, enabling scalable and systematic diagnosis of manipulation policies. The main contributions of the paper can be summarized as:
train the deep RL-based vulnerability search policy
1. Proposing deep RL-based framework that operates in semantically rich embedding space, for diagnosing failures in pre-trained manipulation policies.
2. Providing extensive experimental evidence simulated and real-world setups, backed by theoretical guarantees.
3. Systematically improving robot policies using the failures diagnosed by our framework.
RELATED WORK
Failures in large models be characterized by querying vasion-language foundation models Agia et al. (2024); Duan et al. (2024); Klein et al. (2024); Subraaanyym et al. (2025): Liu et al. (2023) or searching for failures Sagar al. (2024). As we further verify in experiments, the former does not show strong performance in deciphering failures as they do not iteratively interact with the robot policy. Furthermore, VLM models alone are yet capable of making highly accurate quantitative predictions such as probabilities, making them difficult to use in high-stakes tasks. In the latter approach, outside robotics, deep RL has recently been employed machine learning identify emors in classification and generation [Sagar et al. (2024). Similarly, Delecki et al. (2022): [Hong al. (2024) utilized Markov decision processes to explore challenging rainy conditions, which backed by the work of Corso et al. (2021), highlighting the role of sequential decision-making models to ensure the safety of black-box systems. Note that these approaches have neither been demonstrated complex physical systems like manipulation nor they generalize beyond fixed set of known failures, both of which we address.
Out-of-distributin (OOD) detection methods can also be used to identify unseen inputs, for in- stance, in automotive perception Nitsch et al.) (202I), runtime policy monitoring Agia et al. (2024), and regression Thiagarajan al. (2023). However, failure detection constitutes different prob-
2

---

Published as a conference paper at ICLR 2025
(1)
Reinforcement Learning
(2) Finding Failure Likelihood
(3) Fine-tune
Robob Policy
Generate Data
Action
known variations
Changed Foviranment
Manipulation Policy 101
Finezune
Improved Manipulation Policy
Reward
Reward
Poss oil action
Action distribtionn
Figure 2: Our framework operates in three stages: (1) a PPO-based deep RL agent (TMD) perturbs the environment to reveal configurations that cause failures in the pre-trained manipulation policy (TR); (2) its learned action distribution conditioned on the input observation is converted into failure- mode probabilities, either over a continuous embedding for novel changes via discrete candidates; and (3) those probabilities are used to fine-tune for improved performance.
lem than OOD detection, as not all OOD samples lead to failure, and failures can also occur in- distribution. We aim to characterize failures both within and beyond the training distribution, not merely flag OOD instances. A related area of research is uncertainty quantification, which underpins many OOD detection methods. While many attempts have been made to characterize the epistemic uncertainty |Senanayake (2024). the unknown unknowns, in robot perception systems O O'Callaghan & Ramos 2012);]0 Kendall & Gal (2017), only a few attempts have been made to address this chal- lenge in deep RL Jiang (2024) and imitation learning Jeon al. (018); Brown et al. (2020); Ramachandran & Amir (2007). As robot policy models grow increasingly complex, formally char- acterizing epistemic uncertainty becomes extremely challenging. Even if we can, such techniques do not inform engineers where the models fail, making it harder to further improve the policies.
Generalized policies are less prone failures. Toward achieving this goal, generalization in robotics has been extensively studied to enable robots to adapt to diverse scenarios. Large-scale sim- ulation frameworks have been developed to evaluate the robustness of robotic policies across varied tasks and environmental conditions Pumacay et al.] (2024); FFng et al.] (2025). Vision- Vision-language-- action models trained on multimodal datasets have demonstrated significant advancements im- proving adaptability to real-world scenarios Brohan et al.. (2022; 2023). Additionally, approaches such as curriculum learning and domain randomization have proven effective in enhancing gener- alization by exposing models to progressively complex or randomized environments Andrychowicz et al. (2020). These methodologies collectively address the challenges of policy robustness. con- trast to these training focused methods, our framework acts diagnostic tool that is complemen- tary to them as it can systematically identify failures in policies trained using any such approaches. Ultimately,
matter how general the model is, unforeseen conditions and subtle variations will always rise, making systematic diagnostic tools indispensable for real-world deployments. Oth- ers have tackled safety from theoorettc [Bajesy & Fisac (2024); Grimmeisen al. (2024), numann-actors Sanneman & Shahl (2022).
Farid et al. (0022); Ren & Majum- dar (2022); Yang et al. (2020): Vincent et al. (023), and formal- methods et al. (2013) perspec- tives. While these enhance overall robustness, our framework diagnoses failures pre- pre-deployment to help guide policy improvement.
3 METHODOLOGG
We analyze vulnerabilities of pre-trained manipulation policy, by training separate vul- nerability prediction policy, named RoboMD, TMD. RoboMD designed to be agnostic to the architecture or underlying training method of œR- Whether TR is trained via behavioral cloning, reinforcement learning, foundation models, or any future methods, FMD only requires rollouts of mR. making RMD adaptable to a wide range of manipulation policies and tasks. In Section 3.1 we first formalize the failure diagnosis problem as a sequential search problem. Building on this formula- tion. in Sections 3.2 and 3.3 we show how TMD can search for vulnerabilities in space of variations that is not known. We also show a special case of it, where we can systematically test over a dis- crete set of known variations. After describing how vulnerabilities can be queried runtime in Section 3.4 we theoretically ground each step in our framework in Section 3.5
3

---

Published as a conference paper at ICLR 2025
do Ri srarch embedditgg space
t=1
t=2
[=3
1=4
Flattened
MLP Projector
Radart11
R
Relost
ViT
Rolioot_ The coject clarged to betl Lighting changed
Rollout Frames
Barkett The abjuct changel Red (Rangadit Cunoon
The object is changed to Fonta
bottle. Lighting changed Normal.
CLIP
Figure 3: The pipeline shows how rollouts with variations (e.g., object or lighting changes) are processed learn meaningful embeddings. Visual data and text from the rollouts are embedded using VIT and CLIP, and then projected to an MLP, followed by failure-success classification.
3.1 FORMALIZING FAILURE DIAGNOSIS AS SEARCH PROBLEM A SEMANTIC SPACE
Overview. As rationalized in Section T we first learn a continuous vision-language embedding project manipulation data into semantically meaningful representation of the success failure of TR (Fig.3) 3). We then learn TMD by exploring this space using proximal policy optimiza- tion (Fig. 10p. Once trained, TMD be queried to predict ranked map of vulnerable conditions for the manipulation task. An overview of how to train and query TMD, along with how to use TMD to improve Tg shown in Fig.
We consider continuous semantic embedding space, &, that represents some limited notions of suc- cess and failure of कं (details on training this embedding is provided Section (.2). We train TMD to traverse this space and learn how to predict failures. Treating the manipulation policy (R(aa (st) and the robot's environment as a black box, we formulate this traversal TMD as a Markov Decision Process (MDP), defined by the tuple A,, P,),
State Space (S): The state 8, E S = a realization of the continuous semantic embed- ding, represented vector 9t E R-512 While every physically instantiated environment variation corresponds to a state in S, not all realizations of necessarily translate to a physically plausible environment variation. Thus, all S can be viewed as the of hypoth- esized environment variations, accessible through the embedding, which may include both physically viable and unviable variations. See Fig. for intuition on variations.
Action Space (A): An action d₂ E A introduces a variation to 8t. By taking different actions, TMD can jump from one hypothesized environment variation (a state) to another.
While actions can have a physical meaning (e.g., making the enviromment darker), we do not explicitly define them because an action can be any vector value, 0, E R-512
Transition (P): The transition function P(s++1|5 (1,8,, a.) is determined by applying the vari-
ation Gr to the state 8,, which results in 8t+1-
Reward (R): RMD is rewarded for finding failures quickly.
Discount Factor (7)) We use a standard discount factor of r 0.99.
The goal of learning AND is to find sequence of actions (environment variations), which maximizes the probability Tg to fail at its task. The next sections provide exact details of the procedure.
3.2 BUILDING THE SEMANTIC EMBEDDING AS POTENTIAL FIELD SUCCESS-PAILLURES
To predict vulnerabilities of unseen environments, we need at least two pieces of information: 1) some prior belief of where vulnerabilities might occur and 2) way generalize that belief to unseen conditions. We construct the belief from a limited set of labeled rollouts, which are then used train language embedding that captures semantic similarity between vulnerabilities. By operating directly in this embedding, TDD extends its search to a continuous action space.
Our approach hinges on creating this embedding space E such that the relationship between implicit environment variations and policy failures is locally smooth. More formally, we learn an embedding that acts as potential function, t over the space of environmental variations. Later, in Section 3.5 we show that reward shaping based on a potential difference, F(s,, a, $241) = ( - ((,)), guarantees that the optimal policy is preserved and provides a dense learning signal for convergence.
4

---

Published as a conference paper at ICLR 2025
To construct this potential field of success- failures, we train a multimodal embedding that organizes environmental variations based on set of labeled success-failure rollouts. To this end, collect M rollouts from aR for given task with D {( 2z ), y/ MM where vision is the raw image input that we typi- cally provide to manipulation policies, znang short textual description of the task, and E (failure, success). Since we know the action (environment variation) we apply, the textual description can be automatically con- structed (see Appendix[T... Note that this dataset is created by collecting environmental varia- tions (e.g., object color, lighting changes) cho- based prior observations, assumptions and knowledge of conditions that often ex- pose policy vulnerabilities. Using this data, as shown in Fig. 3 we train a dual backbone archi- tecture that consists of:
Sucomen Region
Fallre Ragion
Known Eribudding
That Cate"
(ane),
Figure 4: RL exploration in continuous embed- ding space E. Stars (*) denote known embed- dings (Exmww)) obtained from the dataset D. This induces regions of failure and success. The rest are unknown regions. Dashed boundaries group similar environment variations sharing common physical meaning. The orange circles and arrows show the RL agent's s transition sequence, attract- ing towards failures and repelling from successes. Since each step is obtained without robot rollouts, each one a hypothesized environment variation. Since the transitions are biased toward failures, TMD learns to encode the failure distribution.
1. A Vision Transformer (ViT) back- bone Dosovitskiy al.) (2020) con-
vert 2 to visual features. Vision
transformers capture the contextual relationships within the visual input. Since the original ViT is trained on millions of images from ImageNet with thousands of object categories, this backbone helps to build semantic relationships of everyday environments, so that TTD can infer about unseen environment from similar environments.
2. A CLIP encoder Radford et al. (202T) to process semantic descriptions. Since robots op- erate in complex environments, empirically found (Section 4.2) that providing task description in natural language helps to focus on the necessary features the vision input.
The dual backbone combines complementary strengths, resulting in multimodal embedding that enables better generalization across different environmental variations. The outputs are projected and concatenated, which is passed through 512-unit MLP layer followed by classification head. We define the output of the 512 embedding vector as e, for an arbitrary input (aryisionn ,.?"".))
The architecture is trained with backpropagation using a joint objective of binary cross-entropy with the classification output y and contrastive loss objective C the MLP layer, which, as we later show in Section 3.5 structures E as potential field to enforce smoothness within E. It minimizes the potential difference between embeddings of rollouts with similar outcomes while maximizing for rollouts with different outcomes. L is defined as:
c IJED M=
C= Σ
1
+
max m -di))
(1)
MA7U
where, dy = ||e; - eill is the Euclidean distance between two randomly sampled points in D, with hyperparameter margin. The MLP layer projects raw vision-language data to an embedding space, E. The inputs in DD are now assigned to an environmental variation in the semantic space. Because the space structured such that proximity corresponds to outcome similarity, TMD can now be trained with virtual rollouts to efficiently map the entire landscape of potential failures in E.
3.3 DEEP REINFORCEMENT LEARNING IN THE CONTINUOUS SEMANTIC SPACE
The agent TMD can now navigate through & guided by some of its known realizations, Eknown {e;; E DJ.a set of pre-computed embeddings derived from D. Note that Exnown C C and E 512 These embeddings serve as reference points in the action space. representing well-understood re- gions where failure/success is already observed. As shown in Algorithm [ the TMD samples an action a;+1 from the embeddings space and finds the closest embedding in &known to obtain its cor- responding action o, thus performing action implicitly applies variation to the environment,
5

---

Published as a conference paper at ICLR 2025
although are not explicitly changing the physical environment. Therefore, these actions tremely cheap compared to explicitly changing the environment and running rollouts. To optimize TMD we use PPO Schulman et al. (2017), which provides stable training while maintaining suffi- cient exploration. In continuous space, PPO's entropy regularization is especially important, as it prevents the agent from collapsing space. We define the reward function to encourage discovering failure regions while discouraging both large deviations from Eknown and repetitive actions since large deviations lead to uncertain re- gions, while repetitive actions can indicate a stalled search. This reward mechanism is captured by:
few modes and instead encourages coverage of the broader
K - 1 -V(())), if failure, penalty +1
R(s,a)
(2)
K borizon I))
if success.
Here, failure-success is inferred from the clos- est point in D and the distance penalty in the denominator, scales with ||a ell. Ve E Eknown The penalty can be related the potential field
Algorithm I Learning TMD policy
1: Inputs: D = { xx lang), ) Vi Vil
2: Precompute: Ekrown from D
in terms of ||a - ell2 = (sx(s. - ((*x) 2l2. where sa and se are the states reached by apply- ing actions a and e, respectively, The frequency
3: Init: steps N, reward r r= 0, TMD = rand
4: for t = t N do
5: Sample d++1 ~ (8)
penalty, (a), counts consecutive repeats of a, and we set the coefficient & = 5. The frequency
6: a;+1 t arg mine
aa++1 - e|l2
7:
8t41 + [MD(st(a{+1)
penalty serves as practical mechanism to pro- mote exploration, preventing the agent from re-
8:
r r<r R(*++1)
9:
if failure detected then
peatedly sampling the same point and push- 10:
Reset; r + 0
ing it toward uncertain regions. This approach
11: end if
aligns with the principles of Theorem 2 which
12: end for
states that concentrating exploration near the decision boundary between success and failure leads more efficient identification of vulnerabilities. Fig. 4 illustrates this process, where RL samples embeddings to steer toward failure-prone regions without requiring full policy rollouts.
13: Outputs: RoboMD policy TMD
Special case: when the candidate variations are known. If the set of candidate variations in Eknown is explicitly given (e.g., constructed from historical failures or expert knowledge), then TDD merely has to search over A = Eknown- In this setting, TMD gradually modifies the environment by applying finite sequence of predefined actions (41, (01.02:-. an) until failure induced. For instance, the sequence change table color to black + adjust light level to 50% - set table size to X yields an environment with a black table size X, and 50% lighting. The reward function assigns a positive scalar when the outcome is failure, and negative scalar when the outcome is a success.
3.4 FROM PREDICTING VULNERABILITIES TO IMPROVING ROBOT POLICY PERFORMANCE
The agent TMD outputs a probability distribution over variations (actions), which can directly be interpreted as a map of failure likelihoods. This allows us to both identify highly vulnerable condi- tions and use them to guide manipulation policy improvement. In the embedding space constructed in Section 3.2. π MD (a 8) is modeled as a Gaussian density p(a () on R2122 Although p(a+) = 0
(a.1]))) for any exact action. likelihood ratios are well defined: y [Ma[)s)) indicating which variation is more failure-prone, analogous to PPO's probability-ratio objective. The confidence of these likelihood es- timates can be found using the proximity of the current state embedding e, to the nearest e E E, measured by mingeg |e e.ll. When restricted to a finite candidate set Eknowns as in the special case described in Section 3.2 the same mechanism reduces categorical distribution over Eknown where mass gradually concentrates on failure-inducing variations. In such cases, likelihoods are (x))
given by 77 MD (a 8) E. (e]]) which is probability mass function (PMF) over the discrete action set A, where fa) is the logit for action a.
Improving ग with findings from WMD. The likelihood of actions yields prioritized list of failure modes, which allows practitioners to move beyond collecting broad, unfocused rollouts. Instead, can target data collection the highest likelihood failures (e.g., specific lighting or object variations), and fine-tune म on this compact dataset to systematically patch vulnerabilities, we will demonstrate in Section 4.4

---

Published as a conference paper at ICLR 2025
3.5 THEORETICAL UNDERPINNINGS AND GUARANTEES
We establish that the potential field we build in embedding space does not negatively affect the overall reward but makes the convergence faster.
Theorem (Advantage Invariance in Semantic Potential Field). Let aun be the policy for the MDP defined in Sec 3 Let the continuous action space be structured by the embedding e, which is trained via the contrastive loss L (Eq 7 to function a potential function, Φ(s) es- The search performed by TMD in this space is equivalent to learning in shaped MDP where the implicit shaping function is Fss a, s') = (
(s),, for which the following hold: (i) Optimality: any optimal policy in the shaped MDP is also optimal in the original MDP. (ü) Advantage invariance: A (s, a Driiginat, indicating that the relative advantage invariant.
A (s, a) shshuped
Proof Sketch. From potential-base shaping theory Ng et al. (1999). Qianped (5, a) = Q Vorig(s, (s)
(()) and (s) - (s) (()), where Q and are Q and value functions, respectively. Subtracting shows the (s)) cancels, proving advantage invariance. (Full proof in Appendix (A) O
Having established the correctness of the reward, we now demonstrate that it facilitates efficient exploration of the failure-success boundary while enabling faster convergence.
Theorem 2 (Sample- Efficient Boundary Exploration). Let actions A be mapped by L- Lipschitz embedding e(a). !f TMD is trained with reward R(a) = Reade + (a)a) for 3 > O, where H(a) is predictive uncertainty, then: (i) R(a) is Lipschitz, yielding stable gradients for PPO. (ūi) Exploration concentrates near the successffailure boundary, identifying it to precision e with rollouts polynomial in 1/e.
Proof Sketch. The semantic potential induced by the multimodal embedding e(s). trained with BCE and contrastive loss is Lipschitz continuous (proof in Lemma in Appendix [A), which ensures smooth rewards and stable gradients. The information-gaii drives exploration toward uncertain boundary regions, yielding sample-efficient discovery, (Full proof in Appendix A
Theorem (Convergence Acceleration Due to Potential Field). Let PPO train a critic with Bell- man updates of the form || S ol + with critic error and approximation error c. If potential shaping induces a transformed critic with smaller initialization error S < Eo and smaller approximation error e' e, then for any é'((1- 7) with discount factor 7, the shaped critic reaches || S & in fewer iterations than the unshaped critic.
Proof Sketch. Due to Theorem 1 the error recursion solves to |lEr S ~0 + - Since
both E and e' are strictly smaller, the shaped process crosses any target E earlier, implying faster convergence of PPO. (Full proof in Appendix A)
4 EXPERIMENTAL RESULTS
In this section, we present a series of experiments designed validate RoboMD. Our evaluation is structured around three central research questions:
1. How does RoboMD's failure diagnosis compare to alternative approaches, including other RL algorithms and VLMs?
2. How effectively does RoboMD generalize its diagnostic capabilities from a known set of perturbations to entirely unseen environmental variations?
3. Do the failure modes identified by RoboMD provide actionable insights that lead to mea- surable improvements in the robustness of pre-trained policy?
4.1 BENCHMARK COMPARISONS
Experimental setup: Our simulated experiments are conducted in LoboSuite/Zhu et al. (2020) using
datasets from RoboMimic Mandlekar et al. (202T) and MimicGen Mandlekar al. 2023) (Fig. 7
We evaluate across four standard tasks: lift, stack. threading, and pick & place. which represent
range of manipulation challenges with varying levels of difficulty. These tasks are tested against diverse set of standard manipulation policies, including Behavior Cloning (BC), Hierarchical BC (HBC), BC-nsaosfrr,rr, Batch Constrained Q-Learning (BCQ), and Diffusion policies.
We first benchmark TMD against suite of baselines to answer our first research question regard- ing comparative performance. To this end, we construct a dataset of 500 success-failure pairs,
7

---

Published as a conference paper at ICLR 2025
Reinforcement Leurning Models
Algorithm
Lift
Pick Place
Threading
Stack
Model
Lift
Square
Pick Place
Avg. Score
BC
82.5%
76.0%
68.0%
88.0%
A2C
74.2%
79.0%
72.0%
75.0
BCQ
61.5%
72.5%
62.0%
72.0%
PPO
82.3%
84.0%
76.0%
80.7
HBC
83.5%
79.0%
73.0%
81.0%
SAC
51.2%
54.6%
50.8%
52.2
BC Transformer
74.5%
72.0%
82.0%
70.5%
Diffusion
85.0%
71.0%
71.0%
62.0%
Vision Language Models
Qwen2-VL
24.6%
57.4%
Table 2: RoboMD failure detection accuracy
Gemini 1.5 Pro
59.0%
36.4%
37.4%
44.3
across different tasks.
GPT-4o
44,0%
32.0%
33.3
GPT-4o-ICI ICI. Shot)
S7 4%
48.6%
57.0%
54.3
Small Models (Appendix
B
(Entropy: 2.8236)
SAC (Entropy: 2.2569)
CNN ResNet
46.0%
57.0%
49.0%
50.6
250
PPO (Entropy: 2.00399
49.0%
52.0%
44.0%
48.3
Table I: Benchmark results (Accuracy) compar- ing RL controllers, VLM's, and lightweight mod- els each paired with a BC-MLP low-level policy. TMD consistently outperforms other baselines.
Figure 5: Action diversity across algorithms.
Farth
Red Cube
Red Cube
Green Cube
Table
Creen Cute
Small Toble
31.3
The Cutp
Big Yahre
Oube
Big
MR
29.8
Defect Table
2012
Red Table
Dofout Thole
24
Red Tholo
40
Red Oabe
Green Taole
Due Table
Green Tacle
Blue Table
ModAttn
Bahavioral Cloning, image input
Behuvioral Cloning, Propriospective Batch-Canstriine Deep O-eammin
Implich C- earring
Hlrrarchicll Behavioral Claning
Behavioral Cloning Transformer
Figure 6: Diagnosis of manipulation policies. Each plot shows failure likelihood (a radius) for different actions across environments (real-world in left, and simulation middle and right).
where each pair consists of randomly selected success and failure case. Since a successful action will rank higher than a failure, this provides ground truth to evaluate TMD'S 's ranking consistency. The results, summarized Table [ demonstrate that TMD outperforms all other methods in ranking accuracy across all tasks. We also con- ducted evaluations with sate-of-the-an proprietary models (GPT- 4o and Gemini 1.5 Pro) and an open-source model (Qwen2-ve VL). Additionally, we extended the evaluation of GPT-4o by employ- ing in-context learning (ICL) with 5-shot demonstrations to gauge its adaptability, ICL improves the performance of GPT-4o, partic- ularly in the Square task. However, overall VLM performance re- mains below 60%, indicating that these models struggle with re- liably predicting environment configurations. To compare explo- ration across RL algorithms (i.e., why PPO?),
Figure 7: (Top) real world, (Bottom) simulation. Refer
Appendix C for all variations.
analyze the action distributions of A2C Mnih (016)), SAC Haarnoja et al. (2018), and PPO over 21 environment variations (See Appendix F2). As shown in Fig. 5 and Table PPO achieves the highest entropy (2.8839), indicating its suitability for broader exploration needed for failure discovery. We further evaluate TMD'S 's failure detection performance variety of standard policies trained using different training methods. The results in Table demonstrate that TMD generalizes well across different tasks and policy architectures.
4.2 ANALYZING DIAGNOSTIC CAPABILITIES
We now evaluate TMD's 's ability to diagnose unseen and seen environments, addressing second research question.
The primary strength of RMD is its ability to generalize. We test this by asking TyD to rank the failure likelihood of variations it has never seen during training. The results in Table 3/oww shows in both simulation and the real world, TMD correctly ranks the failure likelihood of unseen items (c.g., the "Bread" for the UR5e, "Black Table" in simulation) relative to known items. The consistency of these rankings validates that successfully captures the semantic relationships
Unseen Environments:
8

---

Published as a conference paper at ICLR 2025
Table 3: In real (a,: (Bread [unseen], Red Cube, Milk Carton, Sprite)) vs, simulated (a,: {Red Table, Black Table [unseen], Green Lighting)) environments, rank consistency measures ordering agreement, and accuracy is computed over 21 unseen variations.
Task ID
Algorithm
Continnous Rank
Ground Truth Rank
Consistency
Accuracy (1)
Real Rohot (URSe)
ModAUin Zhou al. 2022
0,1 2 0,2
0,4
0, > a,2 0,3 0,4
Sim. Can
HBC Mandickar kar et at 2020
0,1 2 4.2 a.3
a,1 = a,2 > a33
61%
Sim. Square
Diffussen
0,1 a22 0.3
a,
9,3
68%
Sim. Stack
BCQ Fujinnto BCQ/Fjjintoo of di 2019
6,1 2 0,2 0.3
6,1 m.2 > 0,3
80%
Sim. Threading
BC Transfocnier
a,3
0.1
0,3
74%
4: Failure characteristics for known action
Table 5: TR fine-tuning strategies.
Errors are
spaces. FSI indicate robustness.
W.r.t. an ideal policy. Refer Fig.
Model
Entropy ())
NFM (3) FSI ())
Fine-tuning (FT)
Accuracy % (1)
Mean Square
Chi-Square
IQL
Kostrikov et
2021
2.49
0.72
Strategies
Error (1)
Error ())
BOOT Fupiotto
1.15
Pre-trained
67.91
0.377
0.016
IIBC
BC Tanstormsr
BC
(Two-tmage ippan)
Mandlekar et al
2020
2.47
2,11
2.14
0.63
0.68
0.98
FT with RoboMD
FT with failures
71.25
92.83
0.068
0.033
0.002
0.001
(Propriccetiive Imarre
0.75
FT with 2 failures
75.83
0.140
0.006
SSmalLLA (Shuker ct
2025
1,76
0.73
IT with failures
75.41
0.050
0,002
x0 BBBBlcck @
1.40
0.77
FT with 4 failures
80.00
0.128
D.005
GROOT Djjicce er
0.82
FT with all failures
85.48
0,069
0,003
that govern policy failure. As ablations, we also test the quality of the Eknown by comparing three configurations: using only BCE loss, using BCE with and our contrastive loss, and an image-only backbone model. The results are shown in Fig. 8 and Table 9 The confusion matrix for the full model (eeage+Text with BCE+Contrastive loss) shows strong diagonal structure, indicating high separability between different actions. Quantitatively, this model achieves the lowest MSE (0. 181)) and Frobenius distance (7.6387) from an ideal identity matrix. This confirms that multimodal inputs and contrastive loss is crucial for creating locally consistent embedding space.
Seen Environments: Fig. 6 visualizes the failure likelihoods that TMD generates for different ma- nipulation policies. HBC policy shows high robustness, which is quantitatively confirmed by its low Failure Severity Index (FSI) and small number of failure modes (NFM) in Table FSI quantifies the weighted impact defined by Σ (a, W. where Prailure represents the probability of failure for action a,, and W; is the normalized weight such that the failure with the highest probability is assigned a weight of 1. The significant variation in failure patterns across different policies (e.g.. BC Transformer vs. HBC) expected, as different architectures have unique inductive biases and thus different weaknesses, all of which are effectively captured by TMD-
4.3 ABLATION: EMBEDDING QUALITY AND SEMANTIC CONTINUITY
RoboMD relies on a vision-language embed- ding that must both separate success and fail- ure modes and vary smoothly under physical changes. As shown in Table [6 the learned embedding forms well-separated and compact clusters across simulated tasks and real-world SO101 policies, with large separation ratios, high effect sizes, strong silhouette scores, and low Davies-Bouldin indices. In addition, Ta- ble 7 demonstrates semantic continuity under controlled lighting-intensity sweep for a BC policy trained to pick up a cube under normal lighting, where cumulative embedding distance increases monotonically as illumination decreases (Kendall's 7 = 1.000, Pearson's r = 0.982). To- gether, these results confirm that the embedding is both structurally well-organized and semantically smooth, justifying its use as a continuous potential field for PPO-based exploration.
N
5:SNE Dimension
SUccua Sted 1103
Success Rate
-03
-0.3
Fature
4.7
-0.3
-9,993
-1.65 -0,C0D -9075 6ie -9.265 -9 000
t-SNE Dimension 1
Figure 10: Embedding space & demonstrates a clear semantic continuity between failure and suc- cess states: as the red light intensity is decreased, the model transitions smoothly from failure clus- ters to success clusters.
4.4 ACTIONABLE INSIGHTTS: FAILURE-GUIDE नं POLICY IMPROVEMENT
Finally, we address our third research question: whether the diagnostic signal provided by TMd can be directly leveraged to improve policy robustness. Using the top-ranked failures identified by TMD, we construct a targeted dataset for fine-tuning a pre-trained Bi-l manipulation policy. As shown in Fig. vulnerabilities exhibited by the pre-trained policy are substantially mitigated after RoboMD-

---

Published as a conference paper at ICLR 2025
Table 6: Embedding-synn grouping metrics across simulated tasks and real-world policies.
Table Monotonic embedding-distance pro- gression under decreasing lighting intensity. Cumulative distance from the initial demonstra- tion increases smoothly illumination is duced, demonstrating semantic continuity.
High separation ratio (Sep.), silhouette score
(Silh.), together with low
(DB) index.
Task
Sep. (t)
Silh. (1)
DB (1)
Demo
Success C. Dist.
Step Dist.
Lift - BC
22.99
0.91
0.15
Demo (High)
0.00
0.00
Square - BC
20.23
0.88
0.25
Demo 2
0
0.22
0.22
S0101 SmolVLA
10.37
0.87
0.35
Demo
0.55
0.35
S0101 ACT
7.76
0.84
0.38
Demo
0.58
0.07
50101 7o
23.97
0.95
0.07
Demo 5 (Low)
0.67
0.11
1,33 Fine-tuned (RoboMD failures)
A Fallures
Cosine Similantyy
9.0GB
Fine tuned fallrre))
Best possibbl
trained
A2
A1
A21
A4
5.1
12
A5
A18
Figure 8: Similarity matrix of embeddings trained using
A17
a) BCE, b) BCE+Contrastive loss, and c) no text encoder.
A7
A16
Table 9: Measuring embedding quality across different loss functions. Lower MSE and Frobenius-norm dis- tances (to the identity) indicate embeddings closer to the ideal diagonal (better separation).
A15
A30
A13
A11
A12
Figure 9: Failure distribution before and after fine-tuning. Observe reduction error modes. Dashed black line repre- sents no failure. Refer Table
Image BCK
Image
Image+Text BCE
Image+Text BCE+Contr.
Loss
BCE+Contr.
MSE (1)
0.6495
0.6179
0.8426
0.1801
Fro dist. (1)
14.5060
14.1497
16.5227
7.6387
guided fine-tumi -uning Beyond qualitative improvements, quaniaaluaniiaaiiveely demonstrates that fine- tuning with RoboMD-selected failures achieves significant accuracy gain of 92.83% outperform- ing fine-tuning on randomly selected failures (85.48%) %) while using substantially less data.
Importantly, RoboMD-guided fine-tuning requires only 1.3 GB of targeted failure data, compared to 9.0 GB when fine- mee-tuning on all known failures. This highlights that TMD does not merely expose failure modes, but also prioritizes the most informative failures for policy adaptation. Consequently, RoboMD enables efficient, targeted fine-tuning that improves robustness under constrained data collection budgets.
5 CONCLUSION
This paper introduced RoboMD, framework that recasts the diagnosis of robot manipulation fail- ures from a process manual heuristics into an active, sequential search problem solved by deep reinforcement learning. By training a diagnostic agent to systematically seek out failure-inden inducing conditions, RoboMD provides principled approach to uncovering policy weaknesses. Our key con- tribution, lifting this search into continuous vision-language embedding space enables the RL agent to generalize its diagnosis, successfully identifying and ranking subtle failure modes in both seen and previously unseen environmental conditions. Our extensive experiments validate this methodol- ogy, demonstrating that RoboMD significantly outperforms static VLM baselines, small models and alternative RI. methods in diagnostic accuracy. We also showed that the identified failures provide an actionable path to enhancing policy robustness, allowing for targeted fine-tuning that measur- ably improves performance in high-risk scenarios at fraction of data cost. Beyond debugging, RoboMD offers practical foundation for integration into continuous testing pipelines, enabling automated regression testing as policies evolve. Future work will focus on scaling this approach to create generalist diagnostic agents capable of evaluating multi-tat task policies across broader and more unstructured environment distributions, a crucial step toward the deployment truly reliable and trustworthy robotic systems in the real world.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our research does not involve human subjects, personally identifiable data, or sensitive information. All datasets used are publicly available and licensed for research purposes. We have made sure that our methods and results are reported hon- estly, transparently, and reproducibly. The potential societal impacts of this work were considered, and our contributions align very well with responsible and beneficial use of safe machine learning model deployment.
7 REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibilityy of our work. An anonymous GitHub repository is provided that contains all the necessary scripts to reproduce the experiments and re- sults. The simulation data used in this study is publicly available. For real-world experiments, we employ commonly used UR5 robotic arm setup, whose specifications and control interface are well documented. Detailed experimental setup is provided in the Appendix C Together, these resources allow independent researchers to fully replicate our results.
REFERENCES
Christopher Agia, Rohan Sinha, Jingyun Yang, Zi-ang Cao, Rika Antonova, Marco Pavone, and Jeannette Bohg. Unpacking failure modes of generative policies: Runtime monitoring of consis- 2024.
tency and progress. arXiv preprint
OpenAl: Marcin Andrychowicz. Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Andrea Bajesy and Jaime F Fisac. Human-ai safety: A descendant of generative ai and control systems safety. arXiv preprint arXiv:2405.09794 2024.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. no. A laiion-lnguage--ccton flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.158/6 2023.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1177-1177. PMLR, 2020.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241223668. 2023.
Anthony Corso, Robert Moss, Mark Koren, Ritchie Lee, and Mykel Kochenderfer. A survey of algo- rithms for black-box safety validation of sebberrphysccal systems. Journal of Artificial Intelligence Research, 72:377 2021.
11

---

Published as a conference paper at ICLR 2025
Harrison Delecki, Masha Itkina, Bernard Lange, Ransalu Senanayake, and Mykel Kochenderfer. How do we fail? stress testing perception in autonomous vehicles. In 2022 JEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 1139-5146, IEEE, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Jiafei Duan, Wilbert Pumacay. Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. Aha: A lasion-laggaag-mmodell for detecting and reasoning over failures in robotic manipulation. arXiv preprint arXiv:2410.00377, 2024.
Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang. Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with memory architecture for robotic manipulation. arXiv preprint arXiv:2501. 2025.
Alec Farid, David Snyder, Allen Z. Ren, and Anirudha Majumdar. Failure prediction with statistical guarantees for vision-based robot control. arXiv preprint wiXx:22222.0589, 2022.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Alkis Gotovos, Nathalie Casati, Gregory Hitz, and Andreas Krause. Active learning for level set estimation. In International Joint Conference on Artificial Intelligence, 2013,
Philipp Grimmeisen, Friedrich Sautter, and Andrey Morozov. Concept: Dynamic risk assessment for ai-controlled robotic systems. arXiv preprint arXiv:2401.14144 2024.
Qiao Gu, Yuanliang Ju, Shengxiang Sun, Igor Gilitschenski, Haruki Nishimura, Masha Itkina, and Florian Shkurti. Safe: Multitask failure detection for vision-language-actoon models. arXiv preprint arXiv:2506.099: 099377, 2025.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Zhang-Wei Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:2402. 19464, 2024.
Wonseok Jeon, Seokin Seo, and Kee-Eung Kim. A bayesian approach to generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generaliza- tion in reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances neural information processing systems, 30, 2017.
Lukas Klein, Kenza Amara, Carsten T Lüth, Hendrik Strobelt, Mennatallah El-Assady, and Paul F Jaeger. Interactive semantic interventions for vlms: human-in-the-loop investigation of vlm failure. In NeurIPS Safe Generative Al Workshop 2024, 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning, arXiv preprint 21X.v:2110.06169, 2021.
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint 20..::201.18644/ 2024.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint 1xXx::2306.15744 2023.
12

---

Published as a conference paper at ICLR 2025
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Silvio Savarese, and Li Fei-Fei. Learn- ing to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085, 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108. 03298, 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimiegen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
Volodymyr Mnih. Asynchronous methods for deep reinforcement learning. arXiv preprint
arXiv:1602.01788, 2016.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application reward shaping. ICML, volume 99, pp. 278 778-287, 1999.
Julia Nitsch, Masha Itkina, Ransalu Senanayake, Juan Nieto, Max Schmidt, Roland Siegwart, Mykel J Kochenderfer, and Cesar Cadena. Out-of-distribution detection for automotive per- ception. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp- 2938-2944 IEEE, 2021.
Simon T O'Callgghan and Fabio T Ramos. Gaussian process occupancy maps. The International Journal of Robotics Research, 31(1):42-62, 2012.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: A benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, vol- ume 7, pp. 2586-2591, 2007.
Allen Z Ren and Anirudha Majumdar. Distributionally robust policy learning via adversarial envi- ronment generation. IEEE Robotics and Automation Letters, 7(2):1379-1386, 2022.
Som Sagar, Aditya Taparia, and Ransalu Senanayake. Failures are fated, but can be faded: Char- acterizing and mitigating unwanted behaviors in large-scale vision and language models. arXiv preprint arXiv:2406.07 2024.
Lindsay Sanneman and Julie A Shah. The situation awareness framework for explainable ai (safe- ai) and human factors considerations for xai systems. International Journal of Sllman-Compuln Interaction, 38118-20) 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Ransalu Senanayake. The role of predictive uncertainty and diversity in embodied and robot learning. arXiv preprini arXiv:2405.03: 164, 2024.
Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi. Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision- langu-laanggaatiiii model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844 2025.
Rakshith Subramanyam, Kowshik Thopalli, Vivek Narayanaswamy. and Jayaraman Thiagarajan. Decider: Leveraging foundation model priors for improved model failure detection and explana- tion. In European Conference Computer Vision, pp. 465 4822. Springer, 2025.
13

---

Published as a conference paper at ICLR 2025
Jayaraman J Thiagarajan, Vivek Narayanaswamy, Puja Trivedi, and Rushil Anirudh. Pager: A framework for failure analysis of deep regression models. arXiv preprint FrXi:2309.10997,, 2023.
Jana Tmov, Luis I Reyes Castro, Sertac Karaman, Emilio Frazzoli, and Daniela Rus. Minimum- violation Id planning with conflicting specifications. In 2013 American Control Conference, pp. 200-205, IEEE, 2013.
Joseph Vincent, Haruki Nishimura, Masha Itkina, and Mac Schwager. Full-distribution general- ization bounds for imitation learning policies. In First Workshop Out-of-Distribution Gener- alization in Robotics at CoRL 2023, 2023.
Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 3153-3160. IEEE, 2024.
Chen Xu, Tony Khuong Nguyen, Emma Dixon, Christopher Rodriguez, Patrick Miller, Robert Lee, Paarth Shah, Rares Ambrus, Haruki Nishimura, and Masha Itkina. Can we detect failures without failure data? uncertainty-awar runtime failure detection for imitation learning policies. arXiv preprint arXiv:2503. 250.008588.. 2025.
Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions Robotics, 37(2):314 2020.
Yifan Zhou, Shubham Sonawani, Mariano Phielipp, Simon Stepputris, and Heni Ben Amor. Modu- larity through attention: Efficient training and transfer of language-conditioned policies for robot manipulation. arXiv preprint arXiv:2212.04573, 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martín, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.1229) 2020.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROOFS OF THEORETICAL GUARANTEES
Theorem (Advantage Invariance in Semantic Potential Field). Let TDD be the policy for the MDP defined in Section 3. Let the continuous action space be structured by the embedding e, which trained via the contrastive loss (Eq. 2) to function as potential function, $(s) cg- The search performed by π MD in this space is equivalent to learning in shaped MDP where the implicit
shaping function is Fss, a, = ( (()). For this process, the following hold:
(i) Optimality: Any optimal policy in the shaped MDP is also optimal in the original MDP.
(ii) Advantage Invariance: (s, a) shaped A a) Orriiginal-
The proof relies on the established theory of potential-based reward shaping et al. (999).. Let the original MDP be M with reward function R(s, a, s') and the shaped MDP be M with reward function (s, a, s') R(s, s') F(s, s'), where F(s, a, s') 2 based shaping reward.
Proof.
(s') (ss) is the potential-
The optimal Q-function in the original MDP, QM, satisfies the Bellman optimality equation:
Qir(s,) = Par [[s,, a, 8') + max Q'M (s')'')
Now, consider a transformed Q-function, (s..a) (ir(s,)) (ss). We can show that satisfies the Bellman equation for the shaped MDP, M':
(s. a) + (s) Ey [R(s, s') + max( d)) (s')))
((s, a) Ey [[ss, (R(s,a,s') (s)) (s)) max ((5 a')l
= ExrxPan
+ F(s, a, max ( d')) BamPPa RR's 8') + max Q a')] a'
This is precisely the Bellman optimality equation for M'. By the uniqueness of the optimal Q- function, it must be that (s, ((s.a) (ir(s, o(s).
(i) Proof of Optimality: An optimal policy ग' is one that acts greedily with respect the optimal Q-function. For the shaped MDP, the optimal policy is:
TM (s) = arg max Qhr (8, a)
arg max Qi (s, ((s))
Since (s) does not depend on the action a, it does not change the argmax. Therefore:
π T (s)
arg max Q (s, Th (s)
Thus, any optimal policy for the shaped MDP is also optimal for the original MDP.
(ii) Proof of Advantage Invariance: The advantage function is defined as A(s, a) = Q(8,a) -
(s),, where V(s) max, ((s, a). Using the relationship derived above:
V (s) max Q (s. a) max (s.(s. (s)) (max (ir(s, ((s) Vir ((s)
Now, we can write the advantage function for the shaped MDP:
Air a) = Qūr (s, a) (s)
= (s, +0s)) (V (8) - +()))
= (s, Vi)(s) = Ala (s,a)
This proves that the advantage function is invariant under potential- based reward shaping.
15

---

Published as a conference paper at ICLR 2025
Theorem 2 Sample-Efficient Boundary Exploration). Let actions A be mapped by L- Lipschitz embedding e(a). =f is trained with reward R(a) = Ri ( (a) (a) for 3 > 0, where (a) is predictive uncertainty, then: (i) R(a) is Lipschitz, yielding stable gradients for PPO: (ii) exploration concentrates near the successffailure boundary, identifynng it to precision e with rollouts polynomial in 1/c.
Lemma 1 (Lipschitzness of the shaped reward). Let e: A + be Le-Liscchite Assume Raakk : E 7 R and H. & - R L and LH-Lipsee -ipicccchi,, respectively. For 3 > 0, define R(a))
Reno (e(a)) + 3 He(()). Then R is Lipschitz on A with constant
LR S Le LL + BLH)..
Proof. For any d1, 02 A, [R(a1) - R(a2)]
= Rana (e((1)) (ema(e(a2)) ( H(((a))))
I Rusk (e(a1) - Runk (e(2)))) + (((())) H (e(a)))
s Russ lle(a)) - e(a2) l| |L [e((a)) e(a2)ll
nnaa + (L.) Tue ||i - a2ll.
Thus R is Lipschitz with the stated constant.
Standing assumptions. Let p(a) = Pr[failure a] denote the classifier's predicted probability.
Assume:
Al (Calibration + smoothness) is calibrated and Lp LppLiipshhtz in e(a).
A2 (Regular boundary) There exist To > 0 and >> 0 such that for all with dist(a, aS) ros p(a) 111 2 dist(a, aS), where ôs = {a : p(a) }
Lemma 2 (Uncertainty maximization at/near the boundary). Let H(a) = p(aa) log p(a) - (1 p(a)) log(1 p(a)) be the predictive entropy. Under (A1)- there exist constants cи 0 and
To > such that for all a with dast a5) S To-
H(a) log2 ch (p(a) - 2 < log c# x² dist(a, 08)?
and consequently H achieves its maxima on üS and decays quadratically away from it in distance.
Proof. The binary entropy h(p) = =-plogg- - (1 - p) log(1 -p) strictly concave. It attains its maximum 1/ with value h(1/2) = log 2, and satisfies h h((1/2) 0 and h²" (1(1/2) = -4. By Taylor's theorem, for p in neighborhood of there exists constant Cи (0, 2] such that
h(p) s log cH (p 2
Applying this with p p(a) gives
((a) S log (p(a) - 2)
Moreover, Assumption (A2) states that whenever dist (a, OS) S To. Ip(a) ! 2 x dist (a, AS).
Combining the two inequalities yields
((a) log 2 cHK* dist(a, 08)?
Thus (a) is maximized on the boundary as and decays quadratically with distance away from it.
Lemma 3 (Sample complexity of boundary identification). Suppose f(a) == p(a) admits Gaussian process (GP) surrogate with kernel k, and let "T denote the maximal information gain after T queries for k. Consider active sampling driven by the shaped reward R = Runk 33 with 3 > 0. Then there exists a constant C > such that, with high probability, the zero level set {a: f(a) = 0} (i.e. the boundary) is identified to Hausdorff precision after at most
T S C 7T PTlog log T
6
queries.
16

---

Published as a conference paper at ICLR 2025
Proof sketch. Under (A1)-A2), Lemma shows that, in neighborhood of the boundary, entropy H a smooth, strictly decreasing function of [f() and is maximized where f(a) = 0. Hence maximizing 3H is equivalent (up to smooth monotone reparamcterization to prioritizing high pos- terior uncertainty on the sign of near its zero level set, as in standard level-set acquisitions (e.g.. straddee/varanne/ammniggy criteria).
1/2 oT-1(a) hold with high For a GP posterior, uniform confidence bands If(a) - pT-1(a)) I aT probability. Sampling high- high-variance points in the ambiguous band around f shrinks that band until its thickness is O(e). The cumulative posterior variance is controlled by the information gain yr, which yields
T= PT log c²
for r-aceurate recovery the zero level set; see, e.g., Theorem of Gotovos et al. (2013). Finally, since H and these ambiguity/variacee scores agree up smooth monotone transform near the boundary (Lemma the same rate applies to the 3riviee policy.
Proof of Theorem Part (i): By Lemma which yields bounded PPO gradients.
R is Lipschitz with constant La La(Ia BLHH),
Part (ii): By Lemma 2 H maximized on (and decays away from) the decision boundary, the 3H term concentrates exploration in (ee) tube around it. Lemma 3 then gives the O log T
rollout bound to achieve precision e.
Theorem (Convergence Acceleration Due to Potential Field). Let PPO train a critic with Bell- man updates of the form 1l S Elllo + with critic error and approximation error potential shaping induces a transformed critic with smaller initialization error Eo Eo and smaller approximation error è' then for any d/(1 - 7) with discount factor % the shaped critic reaches | I in fewer iterations than the unshaped critic.
Proof. Unroll the linear recursions. For any t O.
1 1-
|E/loo S 7'50
+
+ 7" E0
(1)
1 1-7
1 1-7
and similarly
1 1- d'
I I-7 + nk (
e'
)
6'
1xx I 7' +
E I-7)
(2)
Define the minimal hitting times for target tolerance
== min(t E N: ol Se),
:= := m N: ||5'll S Se).
From o eg and < e and the positivity of the coefficients and I- 1-7
for 2 O, we have the
pointwise strict inequality
1 1-2' < 755
1 1- 1 1-7 for all 0,
(3)
1 1-7
i... the shaped upper bound strictly below the unshaped one at every t.
Now fix any > é' - 7). By (2), there exists a finite t with ||& < E, so -' is finite. If the
unshaped process is also able to reach E (i.e., 7 < oo), then from (1) we get
1 1- 77 d < T + 10 $ 7 + 1-77
$ E,
which implies 7' <.
Moreover, strict inequality of the shaped bound (3) together with minimality of implies that either
T' - (or smaller) in the degenerate case that E 2 ||Eu loool so. both processes already satisfy the target at = O. Hence, whenever the unshaped process requires at least one update to reach e (i.c., 1), have r' <
17

---

Published as a conference paper at ICLR 2025
Table 10: CNN Confusion matrix percentages (%)
Table 11: ResNet-18 Confusion matrix percentages (%)
Task
TP
FN
TN
FP
Task
TP
FN
TN
FP
Square
39
61
74
26
Square
60
40
34
66
Can
Can
90
84
16
Lift
8
92
87
13
Lift
47
53
51
49
B DISCUSSION: FAILURES, OOD, AND NEURAL NETWORK BASELINES
Failures arise both in OOD and in-distribution scenarios, OOD detectors alone cannot serve as comprehensive failure detectors. While OOD methods can detect novel textures or extreme lighting they miss subtle perturbations that actually visual detectors a three layer CNN and a halved-channel ResNet-18 on our collected success- faillur frames from the square, can and lift tasks. Table and CNN achieves true-positive rates 39 %, 7 % and % with talse-positive rates of 26 %, % and 13 %; ResNet-18 reaches rrue-positives of 60 %. 10 % and 47 % with false-positives of 66 %. 16 % and 49 %. These low true-positive and high false-positive rates show that standalone OOD detectors lack the sensitivity and specificity needed for reliable failure detection in robotic tasks.
policy failures. We evaluated two lightweight
and Table [1 report confusion matrix rates: the
As sanity check on representational capacity, we evaluated the same two models on a balanced set of nominal versus failure-mode frames exhibiting minor lighting shifts, small object-size changes and novel cube/table colors (see Table []. Both models were trained for 200 epochs on 84x84 RGB inputs normalized to [0.1]. Neither exceeded chance accuracy (=s 50 %) at ranking failures and success these fine-grained perturbations. This confirms that low-capacity visual encoders cannot disentangle subtle failure cues, motivating our policy-centric, multimodal RoboMD frame- work, which leverages semantic embeddings rank and generalize failure modes under targeted perturbations.
Fallure Mode Detection Accuracy per Task
100
80
Accuracy
60
40
20
Square
Pick Place Task
Lift
CNN
IPD
Owenz VL Gemint
4o
Reshitt
(KGL___Shet)
Figure 11:
Testing Robustness Under Visual Perturbations: Successful Rollout in Training Fail- ure Induced by Red Table Distraction
The CNN encoder comprised two convolutional layers (3-32->64 channels, 5x5 kernels, stride 2, ReLU), followed by flattening and 128-dimension) binary classification head. The ResNet- variant retained the original residual block design but halved all channel widths (16-322 -64-1644 +128), applied global average pooling, and used 64-dimensional classification layer.
Failure detection in robotic systems can be broadly categorized into three approaches:
Out-of-Distribution (OOD) Detection. These methods model the distribution of nominal training data and flag inputs that deviate from it. They commonly used for runtime
18

---

Published as a conference paper at ICLR 2025
monitoring in perception and control pipelines Xu et al. (2025). However, not all OOD imputs lead to failures, and many failures occur within the training distribution, limiting their coverage.
2. Supervised Failure Prediction. A classifier regressor is trained on labeled success and failure data to directly predict task outcomes Gu et al. (2025). This approach can capture both in-distribution and OOD failures if represented in the dataset, but its performance depends heavily on the diversity and completeness of observed failure modes.
3. Active Failure Search and Stress Testing, Instead passively predicting failures, these methods actively probe the robot or its environment discover vulnerabilities. Tech- niques include adversarial perturbations, domain randomization, and reinforcement learn- ing-based exploration. This paradigm aims uncover unseen and rare failure modes by systematically searching high-risk regions of the state or environment space.
C EXPERIMENTAL SETUP
C.I COMPUTING RESOURCES
All model training was performed on single NVIDIA H100 GPU (80 GB HBM2), with peak GPU memory usage of approximately 60 GB. We used 2.0 and CUDA 11.8 to maximize throughput. Full training (one tasks) required roughly 12 hours on this setup.
(FP16) training under PyTorch
For inference, the model's footprint falls well below 16 GB of GPU memory, so it can be deployed on wide range of hardware (e.g., NVIDIA A100 40 GB, RTX 3090) without requiring H100- class device. All experiments ran on an Ubuntu 22.04 server with 256 GB of system RAM and dual Intel Xeon CPUs.
C.2 WORLDWORLD EXPERIMENT SETUP
Real-world experiments conducted using URSe robotic equipped with high-resolution cameras and a standardized workspace. The setup shown below in Fig 12
Figure 12: Scenes from experiments on real world robot
C.3 SIMULATION EXPERIMENT SETUP
Simulation experiments were performed using the MuJoCo physics engine integrated with Robo- suite. The simulated environments included variations in object positions, shapes, and textures. The simulation allowed extensive testing across diverse scenarios. Below we show a few samples in Fig 13
C.4 SEMANTIC SPACE TRAINING
For image encoding, adopt the Vision Transformer Base (VIT-B/16) architecture from Torchvi- sion, pretrained on the ImageNet-Il 1k dataset (IMAGENETIK... V1 weights). The original classifica- tion head is replaced with a linear projection to 312-dimmnsional latent space.
For text encoding, we employ the CLIP text transformer from OpenAl's VIT-B/32 model, which provides contextual embeddings of action descriptions. Unlike the vision branch of CLIP, which is
19

---

Published as a conference paper at ICLR 2025
Figure Scenes from experiments on Robosuite
substituted with our ViT-B/16 encoder, the text branch is fine-tuned jointly with the image encoder to learn shared semantic embedding space.
D BASELINES
To validate the effectiveness of our method, compared it against two categories of baselines: Reinforcement Learning (RL) baselines and Vision- LanguaLL Model (VLM) baselines. Below, we detail their implementation, hyperparameters, and specific configurations.
D.1 REINFORCEMENT LEARNING (RL) BASELINES
The RL baselines were implemented using well-established algorithms, each optimized for the task to ensure a fair comparison. The following RL methods were included:
Proximal Policy Optimization (PPO): A policy-gradicn method known for its stability and efficiency. Key hyperparameters included:
Learning rate: 3 x 10-4
Discount factor (7): 0.99
Clipping parameter (c): 0.2
Number of epochs: 10
Batch size: 64
Actor-Critic network layers: [128, 256, 128]
Soft Actor-Critic (SAC): model-free off-policy algorithm optimized for continuous
tion spaces. The key hyperparameters were:
Learning rate: x 10-3
Discount factor (7): 0.99
Replay buffer size: 106
Target entropy: dim(action space)
Batch size: 128
Advantage Actor Critic (A2C):
Learning rate: 2.5 x 10-4
Discount factor (7): 0.99
Exploration strategy: Epsilon-greedy (c decayed from 1.0 0.1 over 500,000 steps)
Replay buffer size: 108
Batch size: 64
Neural network layers: [128, 256, 128]
Each RI. baseline was evaluated using the same metrics, ensuring consistency across comparisons.
D.2 VISION-LANGUAGE MODEL (VLM) BASELINES
The VLM baselines take advantage of the interplay between visual and textual modalities for task representation. We evaluated 3 state-of-the-art VLMS adapted to task:
20

---

Published as a conference paper at ICLR 2025
Training Configuration
(Success)
Perturbed
Configuration
(Fallure)
t=0
40
60
Figure 14: Testing Robustness Under Visual Perturbations: Successful Rollout in Training vs. Fail- ure Induced by Red Table Distraction
1. GPT-40
2. Gemini 1.5 Pro
3. Qwen2-VL
Additionally, leverage GPT-4o with in-context learning, using five demonstrations. First, process the output trajectories into videos and compute the appropriate frame rate to generate video sequences equivalent to 15 frames per trajectory pair. These sequences, representing perturbation scenarios, are provided to the VLMs along with a system prompt that includes a detailed policy description, training configuration, and natural language task description. For evaluation, structure the testing dataset using a pairwise comparison framework, where each model is prompted assess two input video sequences and rank which more likely result in task success, The results are recorded in CSV file, and we compute comparison scores by analyzing model rankings against ground-truth rollouts in the simulated perturbation.
E RATIONALE FOR USING REINFORCEMENT LEARNING
RL employed in the RoboMD framework due its ability to explore high-dimensional, complex action spaces and optimize sequential decision-mann making under uncertainty. This section outlines the key motivations for choosing RL as the core methodology:
Exploration of High-Risk Scenarios: Traditional approaches analyzing robot policy failures often rely on deterministic sampling or exhaustive evaluation, which become infeasible in large, dynamic environments. RL allows targeted exploration by learning agent that actively seeks out environmental configurations likely to induce policy failures. This capability is particularly useful for systematically uncovering vulnerabilities in high-dimensional environments.
Optimization of Failure Discovery: The objective of RoboMD to maximize the occurrence of failures pre-trained policies. RL frameworks, such PPO, well-suited for this task they iteratively refine policies to achieve specific goals, such as identifying high-risk states. The reward function incentivizes the agent find configurations where the manipulation policy fails by going through multiple actions to induce failures. Fig 14 shows several steps of the manipulation policy rollout.
Comparison with Alternative Methods: While other methods, such as supervised learning or heuristic-based exploration, can provide valuable insights into specific failure cases, they are limited in their scope and adaptability. Supervised learning approaches rely heavily on labeled data, which challenging to obtain for failure analysis, particularly for rare or unseen failure modes. These methods also lack the ability to adapt dynamically to changes in the environment, reducing their effectiveness in exploring novel or complex failure scenarios. Similarly, heuristic- based exploration under
methods, such grid search predefined sampling strategies, can identify failure trolled conditions but struggle to generalize high- nigh-dimensional environments where the space of possible failure configurations is vast. These methods also constrained by their reliance static,
21

---

Published as a conference paper at ICLR 2025
8
12
12
12
12
17
17
12
17
8
12
17
Figure 15: The order in which the confusion matrix a) Image Ecoder BCE b) Image Text
Encoder + BCE loss Image Encoder BCE Contrastive loss d) Image Text Encoder + BCE Contrastive loss
predefined rules, which often fail to capture the intricate interactions between environmental factors and failure likelihoods. In contrast, reinforcement learning excels in scenarios where exploration and generalization are critical.
CONTINUOUS ACTION SPACE EMBEDDING
Embedding actions in a continuous space is crucial for efficiently capturing the underlying structure of decision-making processes. Unlike discrete action spaces, where each action treated as inde- pendent category, continuous action space embeddings aim to encode similarities and relationships between actions a structured space.
Table 12: Actions for Can and Box tasks.
Task
Action Description
Change the color red.
Change the can color green.
Can
Change the can color to blue.
Change the can color to grey.
Change the box color to green.
Box
Change the box color to blue.
Change the box color to red.
Resize the box 0.3 0.3 0.02 (L, B, H).
Box Sizes
Resize the box 0.2 0.2 0.02 (L. B, H). Resize the box to 0.1 X 0.1 x 0.02 (L, B, H).
F.1 ACTION DESCRIPTION MAPPING FOR CLIP LANGUAGE INPUT
To generate language inputs for CL.P, we use mapped dictionary that encodes the action being applied to the image. The action descriptions different tasks are detailed Table 12 This table represents only subset of possible actions, and users are free modify the language needed. The descriptions are not strict requirements, as the model learns over time to associate text and images with failure patterns, allowing for flexibility in phrasing while maintaining the underlying semantic meaning. The actions used for Lift task is as follows which was also shown as (A1,A2...A21) in Fig 9
F.2
EVALUATION
Fig. g9 shows the failure distribution in the Lift environment under 21 independent perturbations. These include 4 cube colors, 4 table colors, cube sizes, table sizes, 4 robot colors, and 4 lighting
22

---

Published as a conference paper at ICLR 2025
conditions. Fig L3 illustrates the similarity structure of embeddings trained using only Binary Cross- Entropy (BCE) loss, resulting in highly correlated representations. In contrast, the right matrix, trained with combination of BCE and Contrastive Loss, demonstrates improved separation, as evidenced by the stronger diagonal structure and reduced off-diagonal similarities.
1.0
Image- Bedt Backbore
0.7
130
50
0.4
0.3
30
20
150
120
3.40
Spoch
Figure 16: kNN Accuracy Drop with Increasing k in Continuous Action Space Embeddings (left).
Training loss for training action representations (right)
To assess the quality of the learned embeddings, we conduct an evaluation using a k-Nearest Neigh- bors (kNN) classifier. Specifically, train kNN subset of the embeddings and analyze the impact of increasing k on test accuracy. The intuition behind this evaluation is that well-separated embeddings should be locally consistent, meaning that small (considering only close neighbors) should yield high accuracy, while increasing (incorporating more distant neighbors) may introduce noise and reduce accuracy as shown in Fig 16
F.3 INTEGRATING VISUAL AND TEXTUAL REPRESENTATIONS
Incorporating a textual backbone alongside the image backbone yielded significantly lower loss values and faster convergence compared using an image-only backbone.
This improvement can be attributed to several factors:
1. Semantic Guidance: Textual representations carry rich semantic information that can guide the image backbone. Instead of relying solely on visual cues, the model gains an additional perspective on the underlying concepts (e.g., object names, attributes, or relations).
2. Improved Discriminative Power: With access to text-based information, the model can differentiate between visually similar classes by leveraging linguistic differences in their corresponding textual descriptions.
3. Faster Convergence: Because textual features often come from large, pretrained language models, they are already highly informative. Injecting these features into the training pipeline accelerates the learning process, reducing the number of iterations needed to reach a satisfactory level of performance.
G FINE-TUNING
Once failure modes are identified, empirically found that the most effective strategy for fine- tuning the manipulation policy, TRR is to use all selected failure samples together rather than it- eratively adapting to subsets (Fig. 17 To adapt N against identified failures, we target specific environment variations that a user wishes to improve. As shown in Table [3] fine-tuning with the high-lik -iceliikl samples provided by TMD yields the highest accuracy. We then fine-tune 7R on the combined set of targeted samples, ensuring corrections for critical failures (Fig. (8). Notably, fine-tuning on large set of failure modes also leads to accuracy improvements. When computa- tional resources permit, fine-tuning all identified failures may be also effective; however, when resources are constrained, leveraging RoboMD to select the most informative subset provides an efficient and robust strategy for policy adaptation.
G.1 ADDITIONAL RESULTS
23

---

Published as a conference paper at ICLR 2025
Defaut
Task:
Tasl: Algo: Diffusion
1.0
1.0
0.8
0.8
Accuracy
Accurac/
0.6
0.6
0.0
0.0
Than: Color
Blue
FT. Bon--Grree
Blu Ble--Grreenni Goreen- Red
Train All Colors
FT. All Calors
Pain: color
FT. Blue Blue-sGreen Grree
Blue
Blre Grreen wRed
Train All Colors
FE. all Colors
PT:
Figure 17: Performance comparison of behavior cloning (BC) and biffusion-based policies on the Lift task before and after fine-tuning with failure-inducing samples. Each bar represents the success rate the policy across different table colors.
0.8
Accuracy
0.4
0.0
Default
Color
Color 2
Color 3
Color 4
Color 5
Color 6
Color
Color s
Color g
Color 10
Color 11
Table Color
Figure 18: BC lift finetuned on a combined dataset 12 different Table colors
Table 13: Evaluation of Fine-Tuning Approaches Failure-Targette Data.
Fine-tuning (FT)
Accuracy %(t)
Mean Square Distance (L)
Wasserstein Error (L)
Chi-Square Size
FT Dataset
Strategies
Size
Pre-trained
67.91
0.377
0.005
0.016
FT with RoboMD
92.83
0.033
0.001
0.001
1.29GB
FT with failure
71.25
0.068
0.002
0.003
0.43GB
FT with failure
75,83
0.140
0,003
0,006
0,86GB
FT with failure
75.41
0.050
0.002
0.002
1.29GB
FT with 4 failure
80.00
0.128
0.002
0.005
1.72GB
with failure
81.25
0.337
0.004
0.014
2.15GB
FT with 6 failure
64.76
0.201
0.003
0.008
2.58GB
FT with all failure
85.48
0.069
0.002
0.003
9.00GB
24

---

Published as a conference paper at ICLR 2025
feaure denecional Centee,
8.0
835-
0.00
030
025
$ 5.00
835
8:5
Code
Oetaul tase
Shite
sosal Bust
Bet Code
Bii cope
retaat Baile
Biie Balle
fip Bale
Senrt tasle
Cener Cade
Cude
clver Sliee
Sine
Ceeer
cetault
Acturation
Ieloe LEaE Coalr
AEBP HOOEEEOE AISTACO CaEEN VNIVIVII
Belth Gleareng
3.00
0.20
8.100
0.15
0.06
2.000
coeer cade
Code
retuuf Baie
dieer 2000
tasta
soall Bise
coo
fed cabe
doee Cude
Blse cate
Detuut Ratlo
Gove Rable
Resto
Necr Ratle
nip Voile
Small table
3.000
Grean Gate
beteil Roble
Cade
Grean Saile
plan Table
Dari
su Wile
Sinsl Sail
Avt
Rest
Perarbaconn
Antarbaaa
Figure 19: Comparison of model robustness by measuring failure likelihood under controlled small environmental variations.
25

---

Published as a conference paper at ICLR 2025
Task: Pick Place
Sprite Bottle
Bread
Fanta Bottle
Milk Carton
Red Cuboid
Task: Square
Lighting
Table Color
Table Shape
Object Color
Object Size
Task: Pick Place
Lighting
Table Color
Table Shape
Object Color
Robot Color
Task: Threading
Lighting
Table Color
Table Shape
Gripper Color
Object Shape
Task: Stack
Lighting
Table Color
Table Shape
Cube Color
Robot Color
Task: Lift
Lighting
Table Color
Table Shape
Cube Color
Robot Color
Figure 20: Environmental and Object Perturbations on Manipulation Tasks
26


Reviews:
[{"review_id": "gkk2fjwciw", "reviewer": "Reviewer_TeCF", "summary": "This paper proposes RoboMD, a framework for diagnosing vulnerabilities in pre-trained robot manipulation policies by recasting failure discovery as a reinforcement-learning (RL) search problem in a continuous vision–language embedding space. Instead of manually probing variations (e.g., lighting, object color, geometry), the authors train a deep RL policy (πᴹᴰ) to navigate a semantic potential field learned from limited success/failure rollouts. This field is shaped through a multimodal embedding (ViT + CLIP + MLP) trained jointly with BCE + contrastive loss. The policy explores this embedding using PPO to identify regions that correspond to likely failures and produces a failure-likelihood map.", "questions": "1. Table 5 reports that fine-tuning with RoboMD-guided failures outperforms fine-tuning with all failure cases, despite using a much smaller dataset (1.3 GB vs 9 GB as shown in Table 11). This is impressive but counterintuitive. Is the gain purely due to better sample efficiency (targeted failures), or does RoboMD generate more diverse failure cases in embedding space? Some discussion of this discrepancy would clarify the underlying mechanism.\n\n2. Could the authors clarify how an action taken in the embedding space (a ∈ ℝ⁵¹²) translates into a physical environment change? Is there a learned or rule-based decoder that converts latent vectors into concrete variations (e.g., lighting, color, geometry)? If not, how are the “new failure cases” in Table 5 actually generated and validated on the robot?\n\n3. How does RoboMD handle unseen actions or rollouts that lie outside the support of the training distribution?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I really like the idea of reframing failure discovery as an active exploration problem rather than a passive evaluation or stress test. It’s an elegant conceptual step that connects representation learning, uncertainty estimation, and RL in a unified way.\n2. Extensive experiments across simulated and physical domains, ablation on embedding design (BCE vs. BCE + contrastive), and real robot validation strengthen the empirical credibility.\n3. I appreciate that the paper doesn’t just stay empirical—it also provides theoretical justification via potential-based reward shaping and convergence proofs. Even if some of it is idealized, the effort to link the math with the algorithm is commendable.", "weaknesses": "1. The method heavily relies on the assumption that the learned multimodal embedding forms a smooth manifold where semantic similarity aligns with failure likelihood. While visualizations suggest separability, it’s unclear whether embedding-space distances correspond to physically meaningful variations. Without empirical measures of “semantic continuity,” this assumption could fail for unseen scenarios.\n\n2. The baselines include standard RL and VLM models, but exclude other uncertainty-based failure detection or OOD methods (ensembles, energy-based models, MC-Dropout). These would provide a fairer picture of how much RoboMD really advances the state of the art in model failure analysis.\n\n3. PPO over a 512-D continuous space is expensive and potentially unstable. The paper doesn’t analyze training cost, convergence time, or scaling to multi-object or long-horizon tasks. It’s unclear whether the approach generalizes beyond relatively simple tabletop manipulation.\n\n4. The reported “23 % more vulnerabilities” result isn’t clearly defined. What constitutes a unique failure? A cluster in embedding space, or a distinct environmental configuration? The interpretation of this metric should be clarified.\n\n5. While the paper demonstrates that RoboMD can generalize to unseen environment variations within the same domain, it remains unclear whether the diagnosed vulnerabilities or fine-tuned policies transfer across datasets or task families. For example, if a policy fine-tuned using RoboMD-diagnosed failures on RoboSuite were evaluated on a different benchmark such as LIBERDO or ManiSkill, would it retain improved robustness? Testing such cross-domain transfer would provide stronger evidence that RoboMD captures more fundamental, domain-invariant failure structures rather than merely overfitting to intra-domain variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROamsehwfC", "reviewer": "Reviewer_SeQx", "summary": "The paper considers robotic manipulation and preventing mistakes or ‘vulnerabilities’ when using large model vision-language (VLM)  perception, that may include variations in object properties.  Starting with a trained manipulation policy, a new RL process is trained over variations to find such vulnerabilities, called RoboMD. This policy can be queried to predict a failure over a sequence of actions, and given a finite set of failures can put probabilities on these forms. These are then used to fine tune the original manipulation policy.  Experiments are carried out in RoboSuite and show physical plus simulation experiments, and address some key issues.", "questions": "How can the predetermined bad cases be incorporated into a vanilla RL approach?  \n\nWhat is the tradeoff in predetermined examples versus virtual samples?  \n\nWhat further experiments can be made to compare with other control safety methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of creating a failure predictor is a good one, as can be used in many dynamic settings.  This can identify ‘distance’ from key vulnerabilities, and this can include lighting variation and object color and size (cases that are notorious for their difficulty to handle generally). \n\nThe method can be used to isolate and avoid some key failure modes. With sufficient prior examples, generalization over virtual roll outs is carried out. \n\nThe theory shows how a shaped MDP will lead to efficient training. \n\nWhile depending on prior knowledge and setting, the method might be applicable for a variety of settings. The method shows some ability to generalize, given a sufficient failure training set. \n\nThe authors provide code and overall the work has good reproducibility.  This work has a good probability of leading to other follow on efforts.", "weaknesses": "The primary weakness is lack of adequate baselines.  There are various approaches for control that provide safety, such as barrier methods. It is not surprising that a conventional RL would not do as well as one supplemented with an additional failure mode predictor. The semantic potential field amounts to a kind of semantic barrier.  Also not clear is what happens to a vanilla RL if it is given some explicit bad examples with negative reward.  \n\nThe method keys on the training selection and variations, so this is a critical step that relies on the user, and while generalization is shown in the examples, it isn't clear how this translates to new or more complex environments. \n\nThe term ‘environmental variation’ might be too general, since it appears it is more about sizing objects, color of objects, and lighting variations. \n\nThe number of predetermined versus virtual rollouts is a key question. It was not easy to see how this was precisely addressed in the experiments.\n\nThe method can identify some key failure modes, although guarantees are not clear. In the end we are still restricted by the VLM-based approach that doesn’t inherently do well with spatial reasoning. \n\nThe 3 theorems are interesting, showing how the MDP approach can lead to a shaped MDP that leads to efficient exploration and learning convergence.  However, these don’t quite address the key questions of covering the vulnerability space, or semantic related issues in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDFjViKSZs", "reviewer": "Reviewer_5NX2", "summary": "This paper presents RoboMD, a novel framework that diagnoses failure modes in robot manipulation policies by training a RL agent to explore a continuous vision–language embedding space. Rather than relying on hand-crafted test conditions or costly physical rollouts, RoboMD learns to search for vulnerabilities virtually by interpreting the embedding space as a semantic potential field that repels from known successes and attracts toward failures.\n\nThe core contributions include:\n\n1. A deep RL-based diagnostic agent operating in a semantically structured latent space.\n\n2. A framework that identifies both seen and unseen failure modes, improving upon vision-language models (VLMs) and heuristic baselines.\n\n3. The use of identified vulnerabilities for targeted fine-tuning, leading to substantial improvements in robot manipulation policies.\n\nExperiments are conducted on both simulated environments  and real-world scenarios (UR5e arm), demonstrating improved diagnostic performance and policy robustness.", "questions": "1. While the authors acknowledge the combinatorial complexity of real-world disturbances, the training and search process still relies on a limited and predefined set of action embeddings (either sampled or pre-collected). This raises a potential contradiction: how well does RoboMD generalize to more open-ended or compositional disturbances beyond the known embedding space? It would strengthen the paper to explicitly discuss the coverage limitations of the action space and possible mechanisms to expand it during deployment.\n\n2. Several recent works [1][2] have begun to explore failure prediction in robot foundation models, especially through uncertainty-aware approaches. However, this paper does not compare against those methods or discuss how RoboMD relates to them. Some of these prior efforts incorporate uncertainty modeling (e.g., epistemic or aleatoric uncertainty) into their optimization process, which allows the model to focus exploration or calibration in regions of high uncertainty. I believe RoboMD would benefit significantly from integrating a similar mechanism. For example, encouraging the RL agent to focus its exploration around uncertain or ambiguous regions in the semantic space. If the authors can demonstrate that such uncertainty-aware exploration improves the agent’s performance or failure detection coverage, it would substantially increase my confidence in the generality and practical robustness of this approach.\n\n\n3. The framework would benefit from a demonstration of real-world deployment, even at small scale. A promising direction is to implement a sim-to-real flywheel loop, where $\\pi_{MD}$  is first trained in latent space using synthetic or simulation-aligned embeddings, then deployed to identify high-risk conditions on a physical robot. The collected failure cases could be added to the embedding space and used to retrain $\\pi_{MD}$  in future iterations. This would close the loop and demonstrate practical deployability, which currently remains underexplored.\n\n> [1] SAFE: Multitask Failure Detection for Vision-Language-Action Models Qiao Gu etc. Neurips 2025\n> [2] Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies Chen Xu etc. CoRL 2024", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n- Reformulation of diagnosis as RL search in a semantically meaningful potential field is elegant and novel.\n- Using multimodal embeddings to structure the search space is a strong innovation.\n### Quality\n- Broad evaluation: 4 manipulation tasks × multiple policy types × real vs. simulated settings.\n- Benchmarks vs. GPT-4o, Gemini, Qwen2-VL show strong generalization and robustness.\n### Clarity\n- Excellent visuals and modular structuring.\n- Detailed Appendices with ablation studies and embedding quality analysis.\n### Significance\n- Addresses core robotic AI challenges: cost and risk of real-world rollouts, poor generalization, and hidden failure modes.\n- Offers safe, scalable, and data-efficient diagnosis that’s practical for deployment-stage validation.", "weaknesses": "This work is of great practical significance, but I am allowed to express some concerns.\n### Assumption of Embedding Quality:\n1. The success of the approach hinges critically on the quality of the semantic embedding space. If poorly structured, RL search may yield meaningless trajectories.\n2. Though addressed via BCE + contrastive loss, there is no quantitative measure of embedding smoothness across multiple datasets.\n### Limited Real-World Testing:\n1. While real-world results (UR5e) are included, most findings are in simulation.\n2. Have not seen some of sim2real's settings. The community has always believed that real machine data is the most reliable, although simulation data has the advantage of large data and low cost. This work would have more relevance if real machine data was included in the loop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LCWF7v5P7b", "reviewer": "Reviewer_DxbL", "summary": "This paper proposes a framework for predicting robot manipulation vulnerabilities by learning a deep RL policy over a vision–language embedding space. It identifies vulnerable regions without costly real-world testing, builds probabilistic vulnerability maps, and uncovers 23% more unique vulnerabilities than baselines, improving downstream robustness.", "questions": "How well would the proposed method handle scenarios where the embedding space fails to capture subtle but critical real-world variations?\n\nCan the framework provide more interpretable insights into why specific vulnerabilities occur, rather than just locating them?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a highly practical and underexplored problem—diagnosing robot manipulation vulnerabilities—using a smart and scalable approach.\n\n2. Reformulating exploration over a learned vision–language embedding is elegant and helps the policy find subtle failure modes efficiently.\n\n3. The experimental evaluation is thorough, spanning both simulation and real-world settings, and clearly shows strong generalization and effectiveness.", "weaknesses": "1. The method doesn’t really explain why vulnerabilities emerge, so it feels more like a tool for finding problems rather than understanding them.\n\n2. The embedding-based approach may not fully capture complex real-world dynamics, which could limit how well it scales to more unpredictable environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)