Embodied Red Teaming for Auditing Robotic Foundation Models
Authors: name: Sathwik Karnik; name: Zhang-Wei Hong; name: Nishant Abhangi; name: Yen-Chen Lin; name: Tsun-Hsuan Wang; name: Christophe Dupuy; name: Rahul Gupta; name: Pulkit Agrawal
Published: 2024-11-27 18:57:26

Abstract: Language-conditioned robot models have the potential to enable robots to
perform a wide range of tasks based on natural language instructions. However,
assessing their safety and effectiveness remains challenging because it is
difficult to test all the different ways a single task can be phrased. Current
benchmarks have two key limitations: they rely on a limited set of
human-generated instructions, missing many challenging cases, and focus only on
task performance without assessing safety, such as avoiding damage. To address
these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method
that generates diverse and challenging instructions to test these models. ERT
uses automated red teaming techniques with Vision Language Models (VLMs) to
create contextually grounded, difficult instructions. Experimental results show
that state-of-the-art language-conditioned robot models fail or behave unsafely
on ERT-generated instructions, underscoring the shortcomings of current
benchmarks in evaluating real-world performance and safety. Code and videos are
available at: https://s-karnik.github.io/embodied-red-team-project-page.