Title: SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of LLM-based Embodied Agents
Primary area: datasets and benchmarks
Keywords: Safety in LLM-based Embodied Agent

Abstract: We present $\texttt{SENTINEL}$, a unified multi-level framework for evaluating the physical safety of LLM embodied agents using $\textit{formal safety semantics}$. In our approach, safety rules are grounded as temporal logic constraints, providing precise semantics for specifying state invariants, temporal dependencies, and timing requirements. These rules enable formal checking of embodied-agent behavior at multiple stages of decision-making. $\texttt{SENTINEL}$ is organized into a progressive evaluation pipeline: at the $\textit{semantic level}$, natural language safety requirements are interpreted as Temporal Logic (TL) specifications; at the $\textit{planning level}$, high-level action programs and subgoals are checked against these TL rules before execution; and at the $\textit{trajectory level}$, multiple simulated executions are merged into planning trees and verified against more physical-detailed Computation Tree Logic (CTL) specifications. This provides a reproducible protocol for jointly measuring task completion and safety compliance. By grounding safety in temporal logic and enabling formal evaluation across semantics, plans, and trajectories, $\texttt{SENTINEL}$ establishes a comprehensive pipeline for systematically assessing LLM-based embodied-agent safety, laying the foundation for agents that are not only capable but also reliably safe in realistic environments.