Building a Foundational Guardrail for General Agentic Systems via Synthetic Data

## Abstract

Abstract While LLM agents can plan multi-step tasks, intervening at the planning stage—before any action is executed—is often the safest way to prevent harm, since certain risks can lead to severe consequences once carried out. However, existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level. To address this challenge, we highlight three critical gaps in current research: data gap, model gap, and evaluation gap. To close the data gap, we introduce AuraGen , a controllable engine that (i) synthesizes benign trajectories, (ii) injects category-labeled risks with calibrated difficulty, and (iii) filters outputs via an automated reward model, producing large and reliable corpora for pre-execution safety. To close the guardian model gap, we propose a foundational guardrail Safiron, combining a cross-planner adapter with a compact guardian model. The adapter unifies different input formats, while Safiron flags risky cases, assigns risk types, and generates rationales; trained in two stages with a broadly explored data recipe, Safiron achieves robust transfer across settings. To close the evaluation gap, we release Pre-Exec Bench , a realistic benchmark covering diverse tools and branching trajectories, which measures detection, fine-grained categorization, explanation, and cross-planner generalization in human-verified scenarios. Extensive experiments demonstrate consistent gains of the proposed guardrail over strong baselines on Pre-Exec Bench , and ablations further distill actionable practices, providing a practical template for safer agentic systems.

## 1 Introduction

The rapid proliferation of LLM-based agentic systems has opened a new frontier for a broad range of downstream applications in high-stakes domains (Qian et al., 2024; Luo et al., 2025b; Hong et al., 2024). However, their growing autonomy introduces significant safety concerns (Hua et al., 2024; Huang et al., 2025a; Liu et al., 2025). Malicious actors can exploit these systems, and agents themselves may generate harmful action sequences (i.e., trajectories) due to flawed reasoning or unforeseen environmental interactions. Ensuring the safety of these agents is therefore a prerequisite for their widespread adoption, especially in high-stakes domains like healthcare (Xu et al., 2025).

A promising mitigation is a guardrail system (Bassani & Sanchez, 2024; Inan et al., 2023)—an external monitor that at the pre-execution (i.e., planning) stage prospectively analyzes an agent’s plan and intervenes before harmful actions are executed. Yet, building a robust and generalizable guardrail faces three fundamental challenges aligned with data, model, and evaluation. First, there is a critical scarcity of high-quality, diverse data capturing harmful agent behaviors. Real-world unsafe trajectories are rare and hard to collect; manual construction is costly and often lacks the coverage needed for the vast risk landscape, creating a *data bottleneck* for training an effective guardian model. Second, there is a pressing need for a powerful and generalizable *guardian model* that can proactively analyze intended actions; current solutions (Luo et al., 2025a; c; Chen et al., 2025; Chennabasappa et al., 2025; Padhi et al., 2025a) are often narrow in scope or lack the adaptivity required to handle diverse threats and settings, as detailed in [Table 3](https://arxiv.org/html/2510.09781v1#A3.T3) in [Appendix C](https://arxiv.org/html/2510.09781v1#A3). Third, existing relevant *evaluation benchmarks* are ill-suited for the *planning* stage—a crucial pre-execution checkpoint where a system can proactively analyze the full plan to intercept risks before any action is taken. Most existing benchmarks (Zhang et al., 2025a; c; Yuan et al., 2024) emphasize execution-time risks, cover limited scenarios and risk types, and are often ad-hoc and environment-specific, whereas planning-stage ones can be more systematic and generalizable since they analyze plans at the reasoning level.

Contributions. To address the above challenges regarding data gap, model gap, and evaluation gap, we make the following contributions: 1) A synthetic data engine for generating risky agent trajectories (AuraGen), which overcomes data scarcity through a three-stage pipeline: (i) synthesizing diverse benign trajectories, (ii) injecting category-labeled risks via a principled mechanism, and (iii) applying an automated reward model for quality control. This yields a large-scale, high-quality, and controllable corpus for training safety models. 2) A foundation guardrail (Adapter + Safiron), which consists of (i) a unified *adapter* that normalizes different input formats, and (ii) a compact guardian model—Safiron. Given a normalized trajectory, Safiron outputs three fields: a binary decision (*harmless vs. risky*), a fine-grained *risk category*, and a concise *explanation*, enabling precise and interpretable interception before execution. Safiron is trained with a two-stage recipe from a base model—supervised fine-tuning followed by GRPO-based reinforcement—under a broadly explored data recipe that jointly optimizes binary detection and category accuracy with mixed data sources. 3) A benchmark for pre-execution (planning-level) safety evaluation (Pre-Exec Bench), built through tool refinement, trajectory generation, and human verification, providing realistic and high-quality assessments tailored for guardian models.

We further conduct extensive experiments to map the design space of the guardrail framework and distill a set of best practices for effectively training guardian models. Empirically, the adapter–Safiron pipeline consistently outperforms both open-weight and proprietary baselines on Pre-Exec Bench, achieving a strong balance of detection accuracy, fine-grained categorization, interpretability, and preserved task success, while offering actionable guidance for future research.

## 2 Preliminaries: Definition and Formulation

Terminology Clarification. In this paper, Guardrail denotes the overall safety framework (the guardrail may additionally involve multiple supporting modules), including our approach and related works. A Guardian (model) refers to the detection component within a guardrail (in this work, it specifically corresponds to Safiron). Moreover, in this work, Trajectory denotes the planned sequence of actions produced by the agent during the planning phase.

General Agent Workflow. According to the recent works (Huang et al., 2024a; Liu et al., 2025), a typical agent operates in a loop consisting of several stages: 1) Planning, where it derives the current sub-task from the user query or task description, often breaking it into smaller steps; 2) Tool Invocation or Action Execution, where it selects appropriate tools (e.g., search engines or API calling) or performs direct environment-facing actions (Huang et al., 2024b); 3) Observation of Results, where it collects and interprets outputs or environmental feedback; 4) Internal State Update, where it integrates observations into memory or context to update its reasoning basis; and Task Completion Check, where it either outputs the final result or returns to the planning step if the goal is unmet.

Motivation of Focusing on the Planning Stage.
Given the agent workflow above, the planning stage is a critical intervention point: it is the moment when the agent has produced a complete trajectory of intended actions but has not yet executed them. Crucially, this stage reveals the *overall plan* of the agent’s behavior–the full sequence of actions to be taken–rather than a local snapshot. Leveraging this holistic view enables *proactive* safety: harmful trajectories can be intercepted before they incur any side effects. By analyzing trajectories as a whole, we can assess the agent’s overall intent, detect multi-step and context-dependent risks, and prevent irreversible harm. Since most agentic systems incorporate a planning phase (Hong et al., 2024; Huang et al., 2024a; Yao et al., 2023), this intervention strategy generalizes well across architectures and deployment settings.

Problem Formulation. We consider an agentic system where an LLM-based Agent, denoted as $\mathcal{A}$, operates within an environment $E$. The environment is equipped with a set of tools, $\mathcal{U}=\{u_{1},u_{2},\dots,u_{m}\}$, which enables interaction with the environment or external services (e.g., sending an email, querying a database). Given a user query $q$, $\mathcal{A}$ interprets the intent and devises a trajectory $T=(a_{1},a_{2},\dots,a_{n})$ in the planning stage, where each $a_{i}$ is a tool invocation from $\mathcal{U}$. The intended process is denoted as
$T=\mathcal{A}(q,E,\mathcal{U})$.
The agentic systems we consider are susceptible to generating harmful trajectories (Li et al., 2025; Shi et al., 2025). We define a risky trajectory, $T_{\text{risk}}$, as any action sequence that violates pre-defined safety policies upon execution. Such risks may stem from internal model errors (e.g., hallucinations) or external adversarial inputs. We adopt a risk pool (i.e., risk taxonomy) $\mathcal{R}=\{r_{1},r_{2},\dots,r_{k}\}$ to categorize potential harms.

To mitigate these risks, we aim to propose a guardrail $\mathcal{G}$ that intercepts and evaluates $T$ before execution. Given $T$, $\mathcal{G}$ outputs: a) Risk Detection, $y_{\text{risk}}\in\{0,1\}$ indicating whether the trajectory is benign or risky; b) Risk Classification, $y_{\text{type}}\in\mathcal{R}\cup\{\text{benign}\}$ specifying the category of harm if risky; and c) Explanation Generation, $e$, a rationale explaining the risk judgment ($e$ aims to be concise and human-interpretable, sufficient for audit or intervention). Formally denoted as
$(y_{\text{risk}},y_{\text{type}},e)=\mathcal{G}(T)$.
Our goal is to develop a highly accurate and reliable $\mathcal{G}$ to ensure safe agentic systems.

## 3 AuraGen: Data Engine for Synthetic Risk Trajectories

Figure: Figure 1: Workflow of AuraGen as well as four risk injection strategies employed by AuraGen.
Refer to caption: x1.png

A robust guardrail requires a comprehensive training dataset covering diverse agent behaviors—including risky ones—but currently faces two obstacles: Data Scarcity (harmful trajectories are rare, heterogeneous across systems, and seldom public) and High Annotation Cost (pinpointing risk-inducing steps in long, multi-step trajectories demands expert, labor-intensive labeling). To overcome both, we introduce AuraGen, as shown in [Figure 1](https://arxiv.org/html/2510.09781v1#S3.F1), a synthetic data engine that produces large-scale, diverse, and controllable trajectories spanning a wide spectrum of risks for training a guardian model. Crucially, AuraGen makes the guardian more flexible and adaptive by enabling systematic expansion of risk coverage and rapid incorporation of new scenarios, ensuring safety across evolving agent ecosystems.

Stage 1: Benign Trajectory Synthesis. The synthesis process is initialized with a structured metadata profile, denoted as $M=(E,U,C)$, which provides the operational context. Here, $E$ is the *environment description*, $U$ is the *tool information*, and $C$ represents the *constraints* (examplied in [Appendix F](https://arxiv.org/html/2510.09781v1#A6)). We employ an LLM as a Generation Model, $\mathcal{G}_{\text{gen}}$. This model takes $M$ as input to produce both a plausible user query $q$ and a corresponding benign action trajectory $T_{\text{benign}}$. This process can be expressed as:
$(q,T_{\text{benign}})=\mathcal{G}_{\text{gen}}(M)$. The trajectory $T_{\text{benign}}=(a_{1},\dots,a_{n})$ consists of actions that safely contribute to fulfilling $q$. This stage yields a complete, benign scenario, encapsulated by the tuple $(M,q,T_{\text{benign}})$, which serves as a clean baseline for the subsequent stages.

Stage 2: Principled Risk Injection. The core innovation of AuraGen lies in its risk injection mechanism. This process is governed by an Injection Model, $\mathcal{G}_{\text{inject}}$, which transforms a benign scenario into a valuable negative sample. First, a risk category $r$ is sampled from a pre-defined Risk Pool $\mathcal{R}$, and an injection strategy $S$ is sampled from the set $\mathcal{S}_{\text{set}}=\{S_{\text{single}},S_{\text{multi}},S_{\text{new}},S_{\text{bridge}}\}$. The Injection Model then takes the full benign scenario as input to generate a risky trajectory $T_{\text{risk}}$ that is contextually relevant to the metadata and query:
denoted as $T_{\text{risk}}=\mathcal{G}_{\text{inject}}(M,q,T_{\text{benign}},r,S)$.
The strategies in $\mathcal{S}_{\text{set}}$ are designed to construct a holistic taxonomy of failure modes:

I) Single-Step Perturbation ($S_{\text{single}}$): To simulate *atomic risks*, the most fundamental failure type. This strategy modifies a single action $a_{i}$ into a harmful counterpart $a^{\prime}_{i}$, resulting in $T_{\text{risk}}=(a_{1},\dots,a^{\prime}_{i},\dots,a_{n})$. This model isolates errors or simple malicious commands and serves as a critical baseline to evaluate the guardrail’s ability to perform fine-grained, per-action safety checks.

II) Multi-Step Corruption ($S_{\text{multi}}$): To emulate *planned malicious behaviors* that require a sequence of coordinated steps. This strategy replaces a contiguous subsequence $(a_{i},\dots,a_{j})$ with a new malicious sequence $(a^{\prime}_{k},\dots,a^{\prime}_{l})$. This challenges the guardrail to move beyond isolated action analysis and perform contextual reasoning.

III) New Branch Diversion ($S_{\text{new}}$): To model *catastrophic goal hijacking*, where the agent completely abandons its original task. The trajectory is truncated at action $a_{k}$ and a new, harmful terminal sequence $(a^{\prime}_{k+1},\dots,a^{\prime}_{m})$ is generated, resulting in $T_{\text{risk}}=(a_{1},\dots,a_{k},a^{\prime}_{k+1},\dots,a^{\prime}_{m})$. This mode is essential for testing the guardrail’s ability to enforce long-term goal alignment.

iV) Bridged Branch Diversion ($S_{\text{bridge}}$): To simulate *sophisticated, deceptive adversaries* that attempt to mask their malicious activity. It replaces intermediate actions while preserving the original final action $a_{n}$, leading to the trajectory $T_{\text{risk}}=(a_{1},\dots,a_{k},a^{\prime}_{k+1},\dots,a^{\prime}_{m},a_{n})$. By appearing to fulfill the task’s final objective, this mode provides a stringent stress test for the guardrail’s capacity for holistic path auditing, forcing it to look beyond simple outcome-based checks.

Stage 3: Automated Quality Assurance.
Generating risky trajectories introduces non-trivial challenges in maintaining *data quality*. A related example is: when corrupting an intermediate action $a_{k}\rightarrow a^{\prime}_{k}$ within a trajectory $T=(a_{1},\dots,a_{k},\dots,a_{N})$, how can we ensure that the subsequent actions $(a_{k+1},\dots,a_{N})$ remain valid? In other words, a single corruption might steer the state trajectory into an unrealistic direction, producing follow-up actions that would never occur in a coherent plan. This is just one example—beyond causal consistency, synthetic data must also preserve continuity, rationality, and risk alignment to be useful for training. To address these challenges, we employ a Reward Model (RM), denoted $\mathcal{M}_{\text{RM}}$, for automated quality assurance (its training procedure is described later). The RM acts as a multi-faceted critic that evaluates each complete sample—including the metadata, user query, and injected risky trajectory—across five complementary dimensions: Causal Consistency, Postcondition Continuity, Rationality, Justification Sufficiency, and Risk Matching (see [Appendix G](https://arxiv.org/html/2510.09781v1#A7) for details). It outputs a tuple $(s,f)=\mathcal{M}_{\text{RM}}(M,q,T_{\text{risk}})$, where $s\in\{1,2,3,4,5\}^{5}$ is a score vector and $f$ is optional feedback. A filtering policy $\Pi_{\text{filter}}$ then decides whether to retain or discard each sample, i.e., $\Pi_{\text{filter}}(s,f)\to\{\text{keep},\text{discard}\}$.

More details of AuraGen, including its flexibility, controllability, customization, and included scenarios, are shown in [Appendix F](https://arxiv.org/html/2510.09781v1#A6).

## 4 Guardrail Framework and Training

Figure: Figure 2: Deployment pipeline of proposed guardrail framework.
Refer to caption: x2.png

In this section, we present the proposed guardrail framework, illustrated in [Figure 2](https://arxiv.org/html/2510.09781v1#S4.F2). The framework comprises two components: (1) a unified adapter that transforms the input, and (2) a guardian model, Safiron, that detects risks within the transformed input. Owing to page limits, we describe the training of Safiron here, while the training details of the adapter are provided in [Appendix I](https://arxiv.org/html/2510.09781v1#A9).

Figure: Figure 3: The training pipeline of Safiron.
Refer to caption: x3.png

Training Pipeline.
Our training procedure consists of two stages, as shown in [Figure 3](https://arxiv.org/html/2510.09781v1#S4.F3). In the first stage, we perform supervised fine-tuning (SFT) on a vanilla model $\mathcal{G}_{0}$ using dataset $\mathcal{D}$ (generated by AuraGen), obtaining an SFT model $\mathcal{G}_{\text{SFT}}=\text{SFT}(\mathcal{G}_{0},\mathcal{D})$, which acquires basic response patterns. While SFT provides basic detection ability, it struggles with rare or ambiguous risks; reinforcement learning (RL) complements it by optimizing for fine-grained safety objectives. In the second stage, we employ RL to enhance the model’s ability to classify risks. To construct the RL dataset, we run inference with $\mathcal{G}_{\text{SFT}}$ on $\mathcal{D}$ and define

$$ $\mathcal{D}_{\text{easy}}=\{(x,y)\in\mathcal{D}\mid\mathcal{G}_{\text{SFT}}(x)=y\},\quad\mathcal{D}_{\text{hard}}=\{(x,y)\in\mathcal{D}\mid\mathcal{G}_{\text{SFT}}(x)\neq y\},\quad\mathcal{D}_{\text{RL}}=\mathcal{D}_{\text{easy}}\cup\mathcal{D}_{\text{hard}}.$ $$

This is because: directly training on $\mathcal{D}_{\text{hard}}$ alone tends to destabilize learning, as the model over-focuses on rare or noisy mistakes and quickly collapses (see [Figure 6](https://arxiv.org/html/2510.09781v1#S5.F6) and [subsection 6.1](https://arxiv.org/html/2510.09781v1#S6.SS1)). In contrast, combining both easy and hard samples into $\mathcal{D}_{\text{RL}}$ provides a balanced training signal—anchoring the model on reliable cases while still exposing it to challenging ones. The final guardian model is then optimized as $\mathcal{G}_{\text{Safiron}}=\text{RL}(\mathcal{G}_{\text{SFT}},\mathcal{D}_{\text{RL}},R)$.

Reward Design.
The reward function $R(\hat{y},y)$ is designed to encourage both accurate harmless/harmful classification and fine-grained risk categorization. It is defined as

$$ $R(\hat{y},y)=\begin{cases}1.0&\text{if }y=\texttt{harmless}\text{ and }\hat{y}=y,\\ 1.0&\text{if }y=\texttt{harmful},\ \hat{y}=y\text{ and risk category matches},\\ 0.5&\text{if }y=\texttt{harmful},\ \hat{y}=y\text{ but risk category mismatches},\\ 0.0&\text{otherwise}.\end{cases}$ $$

Here, $\hat{y}$ denotes the model prediction and $y$ the ground truth. Although the explanation $e$ is generated, we do not explicitly include its quality in the reward. First, measuring explanation quality typically requires complex evaluation mechanisms (e.g., LLM-as-a-Judge (Zheng et al., 2023)), which would make RL training prohibitively expensive.
Second, in our experiments, we observed that as the model’s risk category classification accuracy improves, the correctness of generated explanations also increases, suggesting that explanation quality can be indirectly enhanced by strengthening risk categorization.
Therefore, we rely on SFT to provide initial signals for rationale generation, while RL primarily focuses on detection and classification.

RL Algorithm (GRPO).
We instantiate the RL stage with *Group Relative Policy Optimization* (GRPO) (Shao et al., 2024), a policy-gradient method that uses a group-wise, on-the-fly baseline instead of a learned value function. Let $\pi_{\theta}$ denote the Safiron policy and $\pi_{\text{ref}}$ a frozen reference policy (we take $\pi_{\text{ref}}=\mathcal{G}_{\text{SFT}}$). For each input $x$, we sample $K$ candidates $\{\hat{y}_{i}\}_{i=1}^{K}\sim\pi_{\theta_{\text{old}}}(\cdot\mid x)$ and compute rewards $r_{i}$. We form the group baseline $b(x)=\frac{1}{K}\sum_{i=1}^{K}r_{i}$ and advantages $A_{i}=r_{i}-b(x)$, then apply group-wise normalization $\tilde{A}_{i}=\frac{A_{i}}{\operatorname{Std}(\{r_{j}\}_{j=1}^{K})+\varepsilon}$ with $\varepsilon>0$. For compactness, define $\mathbb{E}_{i,t}[\cdot]\triangleq\frac{1}{K}\sum_{i=1}^{K}\frac{1}{|\hat{y}_{i}|}\sum_{t=1}^{|\hat{y}_{i}|}(\cdot)$ and $\mathrm{KL}_{i,t}(x)\triangleq\mathrm{KL}\!\big(\pi_{\theta}(\cdot\mid x,\hat{y}_{i,<t})\ \|\ \pi_{\text{ref}}(\cdot\mid x,\hat{y}_{i,<t})\big)$. The GRPO objective is:

$$ $\mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{x}\!\left[\;\mathbb{E}_{i,t}\,\min\!\big(\rho_{i,t}\tilde{A}_{i},\ \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\tilde{A}_{i}\big)\;\right]\;-\;\beta\,\mathbb{E}_{x}\!\left[\;\mathbb{E}_{i,t}\,\mathrm{KL}_{i,t}(x)\;\right],$ $$

where $\rho_{i,t}=\frac{\pi_{\theta}(\hat{y}_{i,t}\mid x,\hat{y}_{i,<t})}{\pi_{\theta_{\text{old}}}(\hat{y}_{i,t}\mid x,\hat{y}_{i,<t})}$, $\epsilon$ is the clipping coefficient, and $\beta$ controls the KL strength. Intuitively, GRPO upweights tokens from candidates scoring above the group mean and downweights those below, yielding stable improvements without training a critic.

Token-level credit assignment.
Each sampled $\hat{y}_{i}$ is a short label (possibly with a brief rationale). We assign $\tilde{A}_{i}$ uniformly to all tokens in $\hat{y}_{i}$ (i.e., $\tilde{A}_{i,t}=\tilde{A}_{i}$) and optimize token-level log-likelihood with end-of-sequence rewards; in practice we use a small $K$, an (optionally) adaptive $\beta$ to limit policy drift, and standard decoding temperature during rollouts. Compared to training on $\mathcal{D}_{\text{hard}}$ only, the group-relative baseline over $\mathcal{D}_{\text{RL}}$ reduces gradient variance and mitigates collapse while still focusing updates on the most informative mistakes.

## 5 Pre-Exec Bench: Evaluating Agentic Pre-Execution Safety

To evaluate the guardrail on the planning stage or pre-execution, we introduce Pre-Exec Bench, a benchmark tailored for rigorous *pre-execution* (i.e., planning-level) safety analysis. While previous execution-time risk benchmarks focus on localized and immediate errors when taking actions, planning-stage benchmarks focus on plan quality scoring, goal alignment checks, trajectory consistency, counterfactual or adversarial planning audits. Overall, Pre-Exec Bench is designed with bias-mitigation as a first-class objective: it aims for realism (matching real agentic systems), diversity (across models, styles, and risk strategies), and quality (human-verified). It is built via a three-stage pipeline: (1) scenario & tool refinement, (2) diverse trajectory generation, and (3) two-phase human verification with debiasing. Pre-Exec Bench remains strictly held out from any training or model selection for the guardrail.

Figure: Figure 4: Left: Construction steps of Pre-Exec Bench. Right: Risk type distribution. The benchmark consists of 1,001 harmless and 671 risky samples (with injected risks).
Refer to caption: x4.png

Stage 1: Data Expansion & Scenario and Tool Refinement (Why we need Pre-Exec Bench?).
Our design is motivated by a survey of existing agent safety benchmarks. While valuable, they reveal critical gaps for evaluating *planning-time* (i.e., *pre-execution*) safety.
ASB (Zhang et al., 2025a) and AgentSafetyBench (Zhang et al., 2025c):
their evaluation emphasizes the *execution phase* and adversarial attacks, underweighting plan-centric, non-adversarial failures (e.g., hallucinated plans).
R-Judge (Yuan et al., 2024):
many samples are dialogue-style and lack stepwise plans and complex tool interactions needed to assess reasoning quality.
OPENAGENTSAFETY (Vijayvargiya et al., 2025) supports only a limited tool set. To end these, Pre-Ex Bench introduces a novel focus on the pre-execution planning stage. The construction is inspired by R-Judge’s approach of extending existing datasets; specifically, we build upon the rich and diverse scenarios and tools provided by AgentSafetyBench to ensure broad topic coverage. Before constructing the trajectories, we first perform tool refinement for more detailed tool calling scenarios: we use an LLM to generate fully-specified, executable function details from the tool descriptions, which are then rigorously verified by human experts for both correctness and functional appropriateness.

Stage 2: Diverse Trajectory Generation (Realism, Diversity, and De-leakage).
Real-world agentic systems are LLM-driven; thus, using LLMs to synthesize trajectories is not merely convenient but *distributionally realistic*.
To construct a challenging and unbiased test set, we employ a heterogeneous model pool $\mathcal{M}_{\text{pool}}$ of eight open-source LLMs across five developers (^1^11Qwen2.5-72B-Instruct, DeepSeek-V3, DeepSeek-R1, Llama-3.3-70B-Instruct, Llama-4-Maverick-17B-128E-Instruct, Qwen3-32B, Mixtral-8$\times$22B, and gpt-oss-20B).
For each scenario, we first sample a benign generator $\mathcal{A}_{\text{benign}}\!\in\!\mathcal{M}_{\text{pool}}$ to produce a stepwise plan $T_{\text{benign}}$.
We then sample a (potentially different) injector $\mathcal{A}_{\text{risk}}\!\in\!\mathcal{M}_{\text{pool}}$ to inject a predefined risk via one of four strategies, yielding $T_{\text{risk}}$.
*Decoupling* benign and risky generators (cross-model pairing) reduces single-model artifacts and prevents a model from “attacking its own style”.
We further (i) *stratify sampling* so no single model dominates the corpus, and (ii) *randomize and paraphrase prompts* (lexical paraphrasing, order shuffling, and style changes) to avoid template bias.
While these measures already mitigate model-specific artifacts, we acknowledge that LLM synthesis alone cannot fully eliminate bias.
Therefore, all trajectory pairs are subsequently subjected to a rigorous human verification and debiasing process in Stage 3, which serves as a non-LLM arbiter to guarantee reliability.

Stage 3: Two-Phase Human Verification and Debiasing.
To break the synthetic-to-synthetic loop and eliminate residual biases from Stage 2, all trajectory pairs undergo a stringent two-phase human review conducted by domain experts.
Phase I (quality & validity gate).
Each pair $(T_{\text{benign}},T_{\text{risk}})$ is independently assessed by three reviewers for plausibility, coherence, and *correctness of risk injection* against a standardized taxonomy.
Only samples with unanimous approval are retained, ensuring high-quality and unambiguous labels.
Phase II (redundancy & distribution control).
Approved samples are grouped into homogeneous batches (by injection strategy). Experts identify and prune intra-batch redundancies (e.g., repeated narrative structures, near-duplicate risk patterns), keeping one representative per cluster.
We then enforce distributional balance across the four strategies (by downsampling as needed).
This human-in-the-loop stage explicitly filters out spurious, model-idiosyncratic artifacts and provides the final debiasing guarantee. The details of human evaluation are shown in [Appendix L](https://arxiv.org/html/2510.09781v1#A12).

Importantly, Pre-Exec Bench relates to our guardrail but is not tailored only to it; it is built to facilitate broader research on pre-execution guardrails in the future.

Figure: Figure 5: The performance under the different ratios of hard/easy samples during GRPO training.
Refer to caption: rl_comparison.png

## 6 Experiments

Evaluation Metrics & Methods & Base Model.
We evaluate Safiron using four metrics:
(i) classification accuracy, which measures whether the model correctly distinguishes harmless from harmful content across all samples;
(ii) harmful detection precision, defined only over ground-truth harmful samples and quantifying the proportion correctly identified as harmful;
(iii) risk category accuracy, which assesses, among correctly detected harmful samples, whether the predicted risk label matches the ground-truth risk type; and
(iv) explanation correctness, which further examines, conditioned on correct risk prediction, whether the model’s explanation semantically aligns with the expected explanation.
Due to page limits, the formal definitions and mathematical formulations of these metrics are provided in [Appendix D](https://arxiv.org/html/2510.09781v1#A4). To balance evaluation efficiency and accuracy, we adopt a hybrid approach that combines keyword matching with LLM-as-a-Judge (Zheng et al., 2023). Further details are provided in [Appendix K](https://arxiv.org/html/2510.09781v1#A11). We use Ministral-8B-Instruct-2410 as our base model, with training data synthesized by AuraGen powered by Mixtral-8*22B-Instruct-v0.1.

Figure: Figure 7: Statistics of synthetic data generated by AuraGen.
Refer to caption: x6.png

Basic analysis of synthetic data.
We use AuraGen to generate around 20k for training (More details are shown in [Appendix F](https://arxiv.org/html/2510.09781v1#A6)). As shown in [Figure 7](https://arxiv.org/html/2510.09781v1#S6.F7), AuraGen’s synthetic corpus achieves a near-uniform coverage of the four risk-injection strategies (around 25% each).
This balanced design is not meant to reflect natural frequency, but rather to *stress-test guardrails fairly across diverse failure modes*.
In addition, the corpus contains user requests of realistic and moderate length (mean 23.10; median 21 tokens) and trajectories with long-tailed complexity (mean 14.77; median 9 actions; maxima about 48).
The long-tail arises from scenarios with more complex environments and richer tool combinations, which provide challenging yet plausible cases.

Cost & Latency Analysis.
A detailed analysis is provided in [Appendix H](https://arxiv.org/html/2510.09781v1#A8).
At our average input/output length, generating one sample with GPT-5 costs under $0.02.
Given that recent open-source APIs (prices from OpenRouter (OpenRouter, 2025)) are strictly cheaper, their per-sample cost is even lower. We also present latency analysis in the [Appendix H](https://arxiv.org/html/2510.09781v1#A8), which also demonstrates the efficiency of our proposed guardrail.

### 6.1 Best Practice for Training Guardian Model

In this section, we outline practices for training the Safiron, focusing on how data composition and sample difficulty should be organized to achieve stable optimization and strong performance.

The ratio of the training set has a far greater impact on the model than the sample size. From the trends shown in [Figure 8](https://arxiv.org/html/2510.09781v1#S6.F8) and [Figure 9](https://arxiv.org/html/2510.09781v1#S6.F9), we observe that as the proportion of harmful samples increases, model performance rises almost monotonically and gradually saturates in the 1:4-1:6 range. By contrast, with the ratio fixed, simply expanding the training set size from 2k to 10k yields very limited gains. The effect of the ratio is especially larger than that of scale for harmful detection and explanation correctness: moving the harmless:harmful ratio from 3:1 to 1:4 brings about a +0.15-0.20 improvement in harmful detection and +0.10-0.15 in explanation correctness, whereas increasing the sample size from 2k to 10k often yields only +0.02-0.05. This explains why in [Figure 9](https://arxiv.org/html/2510.09781v1#S6.F9), training with 4k samples under the *1:3* or *1:4* ratio still significantly outperforms the results under the *3:1* ratio even after doubling the data. The root of this phenomenon lies in the influence of class priors on the learned decision boundary and gradient signals: when harmful samples are scarce, the model is more prone to a “benign-by-default” bias; conversely, a higher proportion of harmful data not only strengthens the ability to distinguish fine-grained risk categories and exposes the explanation module to richer counterexamples. Notably, when the ratio becomes extremely skewed toward harmful (e.g., 1:7 or 1:8), some metrics exhibit diminishing returns or slight declines, indicating that excessive imbalance can harm the overall accuracy. Finally, after five runs with ratios of 1:4 and 1:5, we chose 1:4 as it achieved a better balance.

Easy samples are indispensable for effective GRPO training, but an excessive proportion of them leads to performance degradation. As shown in [Figure 6](https://arxiv.org/html/2510.09781v1#S5.F6), introducing easy samples substantially boosts classification accuracy and explanation correctness compared to the “w/o easy” setting. Without easy samples, the model tends to suffer from catastrophic forgetting (Luo et al., 2023), resulting in unstable optimization and poor overall performance. However, as the ratio of easy to hard samples increases (from 1:1 to 1:3), the importance of hard samples is gradually diluted, which weakens the model’s ability to learn from challenging cases.

### 6.2 Baseline Comparison & Module Performance Evaluation

In this section, we compare baselines and evaluate the performance of different components within AuraGen and the proposed guardrail. Specifically, we contrast Safiron (without the adapter) against standard LLM baselines. The end-to-end performance of the full guardrail (adapter + Safiron) is presented in the case study section ([section 7](https://arxiv.org/html/2510.09781v1#S7)).

**Table 1: SFT Performance of the Safiron under different filtering strategies on 4,000 samples (harmless/harmful ratio 1:3). Red cells indicate the worst values in each column, while green cells indicate the best values. AVG requires that the average score across all aspects exceeds the threshold, whereas ALL requires that every individual aspect score exceeds the threshold.**
| Filtering Policy $\Pi_{\text{filter}}$ | Cls. Acc. | Harm. Det. Prec. | Risk Cat. Acc. | Expl. Corr. |
| --- | --- | --- | --- | --- |
| Baseline | \cellcolorred!15$0.939_{\pm 0.001}$ | $0.918_{\pm 0.002}$ | $0.556_{\pm 0.006}$ | $0.488_{\pm 0.004}$ |
| AVG$>$2 | $0.948_{\pm 0.006}$ | $0.916_{\pm 0.019}$ | $0.549_{\pm 0.009}$ | $0.484_{\pm 0.005}$ |
| AVG$>$1.5 | $0.947_{\pm 0.003}$ | \cellcolorgreen!15$0.920_{\pm 0.011}$ | $0.568_{\pm 0.020}$ | $0.495_{\pm 0.021}$ |
| ALL$>$2 | $0.948_{\pm 0.014}$ | \cellcolorred!15$0.906_{\pm 0.019}$ | \cellcolorred!15$0.541_{\pm 0.015}$ | \cellcolorred!15$0.482_{\pm 0.018}$ |
| Classifier | \cellcolorgreen!15$0.951_{\pm 0.001}$ | $0.915_{\pm 0.005}$ | \cellcolorgreen!15$0.602_{\pm 0.002}$ | \cellcolorgreen!15$0.537_{\pm 0.003}$ |

Figure: Figure 8: Model performance under different sizes of training dataset.
Refer to caption: x7.png

Adapter training and evaluation. We synthesize agentic trajectories in various styles using both programmatic methods and LLM-based generation to train the adapter. Full training details are provided in [Appendix I](https://arxiv.org/html/2510.09781v1#A9). To assess performance, we conduct experiments on the complete training dataset and further examine the adapter’s generalization ability by removing two specific styles from the dataset. We employ LLM-as-a-Judge (i.e., GPT-4o-mini) to evaluate the correctness of the adapter’s outputs. As shown in [Figure 6](https://arxiv.org/html/2510.09781v1#S5.F6), the adapter trained on the full dataset achieves consistently high accuracy across all styles. Even when the “Semicolon Single” and “Bullets” styles are excluded, it sustains strong performance on unseen categories, demonstrating robust generalization.

Figure: Figure 9: Model performance under different ratios between harmless samples and harmful samples with a harmless and harmful ratio of 3:1.
Refer to caption: x8.png

Reward model (in AuraGen) training and evaluation. We train and evaluate the reward model (RM) on synthetic data to avoid costly manual labeling, and find strong agreement with human validation (see [Appendix G](https://arxiv.org/html/2510.09781v1#A7)). RM performance is assessed by two metrics: (1) *score difference*, the total deviation from ground-truth across five criteria; and (2) *instability rate*, the fraction of criteria with absolute deviation $>2$. We show the evaluation results on [Appendix G](https://arxiv.org/html/2510.09781v1#A7). Using the RM as a synthetic-data filter, simple threshold policies (AVG/ALL) underperform on Risk Cat. Acc. and Expl. Corr. (AVG/ALL) underperform on Risk Cat. Acc. and Expl. Corr. (e.g., AVG$>$2, AVG$>$1.5; see [Table 1](https://arxiv.org/html/2510.09781v1#S6.T1)), likely discarding useful samples. We therefore train a lightweight classifier (SVM) (We chose a linear SVM for its simplicity) on Pre-Exec Bench keep/discard annotations, using the vector of per-criterion RM scores as input; this Classifier policy improves most metrics (Cls. Acc. $0.951$, Risk Cat. Acc. $0.602$, Expl. Corr. $0.537$), suggesting the RM encodes structured patterns that benefit from supervised guidance. We include all details in [Appendix G](https://arxiv.org/html/2510.09781v1#A7).

Safiron significantly surpasses both proprietary and open-weight models across all four evaluation metrics, demonstrating its superiority as the most balanced solution. As shown in [Table 2](https://arxiv.org/html/2510.09781v1#S6.T2), compared with leading proprietary models such as Claude-3.7-Sonnet and GPT-4o, as well as open-weight models like DeepSeek-V3 and Qwen2.5-72B, Safiron consistently achieves much higher classification accuracy, stronger risk categorization, and better explanation correlation, while maintaining competitive harm detection accuracy. Notably, the GRPO version of Safiron provides the most stable and well-rounded performance, making it the final choice for our study. While proprietary models like GPT-5 achieve near-perfect harmful detection, they suffer from worse other metrics, effectively over-flagging or exaggerated safety (Röttger et al., 2023) and limiting usability. Safiron balances detection with fine-grained categorization and explanation quality, which are crucial for interpretable pre-execution safety.

Existing popular guardrails are not yet well-suited for the Pre-Exec Bench, underscoring the necessity of our proposed guardrail. We additionally present the results of LLamaFireWall (Chennabasappa et al., 2025) and LLama-Guard-3-8B (Grattafiori & the Llama Team at Meta, 2024) in [Appendix B](https://arxiv.org/html/2510.09781v1#A2). The findings indicate that these widely used guardrails fail to deliver satisfactory performance on the Pre-Exec Bench (as they focus on content moderation tasks, such as detecting toxicity, violence, and hate speech)—thereby underscoring the necessity of our proposed guardrail.

**Table 2: Model performance comparison. See [Appendix B](https://arxiv.org/html/2510.09781v1#A2) for other guardrail performance.**
| Model | Cls. Acc. | Harm. Det. Prec. | Risk Cat. Acc. | Expl. Corr. |
| --- | --- | --- | --- | --- |
| Proprietary Models |  |  |  |  |
| \rowcolorcyan!10<br>GPT-5 | $0.425_{\pm 0.003}$ | $0.990_{\pm 0.002}$ | $0.355_{\pm 0.012}$ | $0.350_{\pm 0.014}$ |
| GPT-5-mini | $0.404_{\pm 0.001}$ | $0.997_{\pm 0.000}$ | $0.325_{\pm 0.001}$ | $0.324_{\pm 0.002}$ |
| \rowcolorcyan!10<br>GPT-4o | $0.606_{\pm 0.002}$ | $0.822_{\pm 0.008}$ | $0.319_{\pm 0.002}$ | $0.310_{\pm 0.004}$ |
| GPT-4o-mini | $0.452_{\pm 0.002}$ | $0.957_{\pm 0.008}$ | $0.274_{\pm 0.010}$ | $0.264_{\pm 0.013}$ |
| \rowcolorcyan!10<br>Claude-3.7-Sonnet | $0.623_{\pm 0.007}$ | $0.793_{\pm 0.003}$ | $0.318_{\pm 0.010}$ | $0.316_{\pm 0.011}$ |
| Gemini-2.5-Pro | $0.438_{\pm 0.003}$ | $0.978_{\pm 0.003}$ | $0.416_{\pm 0.017}$ | $0.402_{\pm 0.015}$ |
| Open-weight Models |  |  |  |  |
| \rowcolorviolet!10<br>Llama-3.1-70B | $0.621_{\pm 0.013}$ | $0.622_{\pm 0.015}$ | $0.305_{\pm 0.012}$ | $0.242_{\pm 0.010}$ |
| Mixtral-8$\times$22B | $0.409_{\pm 0.001}$ | $0.999_{\pm 0.002}$ | $0.344_{\pm 0.017}$ | $0.319_{\pm 0.019}$ |
| \rowcolorviolet!10<br>Qwen2.5-72B | $0.620_{\pm 0.013}$ | $0.760_{\pm 0.013}$ | $0.319_{\pm 0.022}$ | $0.288_{\pm 0.023}$ |
| DeepSeek-V3 | $0.652_{\pm 0.018}$ | $0.602_{\pm 0.029}$ | $0.247_{\pm 0.024}$ | $0.227_{\pm 0.021}$ |
| \rowcolorviolet!10<br>gpt-oss-20b | $0.560_{\pm 0.006}$ | $0.788_{\pm 0.012}$ | $0.295_{\pm 0.014}$ | $0.279_{\pm 0.011}$ |
| gpt-oss-120b | $0.539_{\pm 0.009}$ | $0.877_{\pm 0.009}$ | $0.408_{\pm 0.006}$ | $0.311_{\pm 0.003}$ |
| \rowcolorpink!30Safiron(SFT-Only) | $0.956_{\pm 0.002}$ | $0.939_{\pm 0.022}$ | $0.566_{\pm 0.024}$ | $0.508_{\pm 0.022}$ |
| Safiron(SFT+PPO) | $0.951_{\pm 0.001}$ | $0.969_{\pm 0.008}$ | $0.626_{\pm 0.001}$ | $0.530_{\pm 0.007}$ |
| \rowcolorpink!30Safiron(SFT+GRPO) + | $0.949_{\pm 0.001}$ | $0.973_{\pm 0.002}$ | $0.646_{\pm 0.000}$ | $0.570_{\pm 0.003}$ |

## 7 Case Study in Real Agentic Systems

Beyond the above evaluations, to assess robustness under real conditions, we conduct a case study in two agentic systems based on MetaGPT (Hong et al., 2024) and AutoGen (Wu et al., 2023). Full details (frameworks, risk injection protocol, and dataset construction) are shown in [Appendix A](https://arxiv.org/html/2510.09781v1#A1).

## 8 Conclusion

In this work, we presented a pre-execution guardrail for LLM agents, addressing data, evaluation, and model gaps.
Our contributions include AuraGen for scalable synthetic risk data, Pre-Exec Bench for plan-level safety evaluation, and Safiron, a guardian trained to detect, categorize, and explain risks. Experiments show consistent improvements over baselines, offering a practical template for safer and more scalable agentic systems.

## References

- Bassani & Sanchez (2024)
Elias Bassani and Ignacio Sanchez.
GuardBench: A large-scale benchmark for guardrail models.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 18393–18409, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.1022.
URL [https://aclanthology.org/2024.emnlp-main.1022](https://aclanthology.org/2024.emnlp-main.1022).
- Chen et al. (2025)
Zhaorun Chen, Mintong Kang, and Bo Li.
Shieldagent: Shielding agents via verifiable safety policy reasoning.
*arXiv preprint arXiv:2503.22738*, 2025.
- Chennabasappa et al. (2025)
Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, et al.
Llamafirewall: An open source guardrail system for building secure ai agents.
*arXiv preprint arXiv:2505.03574*, 2025.
- Chua et al. (2025)
Gabriel Chua, Shing Yee Chan, and Shaun Khoo.
A flexible LLM guardrail development methodology applied to off-topic prompt detection.
*arXiv preprint arXiv:2411.12946*, 2025.
URL [https://arxiv.org/abs/2411.12946](https://arxiv.org/abs/2411.12946).
- Chung et al. (2023)
John Joon Young Chung, Ece Kamar, and Saleema Amershi.
Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions.
*arXiv preprint arXiv:2306.04140*, 2023.
- Cortes & Vapnik (1995)
Corinna Cortes and Vladimir Vapnik.
Support-vector networks.
*Machine Learning*, 20(3):273–297, 1995.
doi: 10.1007/BF00994018.
- Dai et al. (2025)
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, et al.
Auggpt: Leveraging chatgpt for text data augmentation.
*IEEE Transactions on Big Data*, 2025.
- Dong et al. (2024)
Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, et al.
Safeguarding large language models: A survey.
*arXiv preprint arXiv:2406.02622*, 2024.
URL [https://arxiv.org/abs/2406.02622](https://arxiv.org/abs/2406.02622).
- Grattafiori & the Llama Team at Meta (2024)
Aaron Grattafiori and the Llama Team at Meta.
The llama 3 herd of models.
arXiv preprint, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
Includes the Llama Guard 3 (8B) safeguard model.
- Hahm et al. (2025)
Dongyoon Hahm, Woogyeol Jin, June Suk Choi, Sungsoo Ahn, and Kimin Lee.
Enhancing llm agent safety via causal influence prompting.
*arXiv preprint arXiv:2507.00979*, 2025.
- Han et al. (2024)
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri.
Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.
*arXiv preprint arXiv:2406.18495*, 2024.
- Hong et al. (2024)
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber.
MetaGPT: Meta programming for a multi-agent collaborative framework.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=VtmBAGCN7o](https://openreview.net/forum?id=VtmBAGCN7o).
- Hua et al. (2024)
Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang.
Trustagent: Towards safe and trustworthy llm-based agents.
*arXiv preprint arXiv:2402.01586*, 2024.
- Huang et al. (2024a)
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen.
Understanding the planning of llm agents: A survey.
*arXiv preprint arXiv:2402.02716*, 2024a.
- Huang et al. (2024b)
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al.
Metatool benchmark for large language models: Deciding whether to use tools and which to use.
In *The Twelfth International Conference on Learning Representations*, 2024b.
- Huang et al. (2024c)
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al.
Trustllm: Trustworthiness in large language models.
*arXiv preprint arXiv:2401.05561*, 2024c.
- Huang et al. (2024d)
Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, Philip S Yu, and Xiangliang Zhang.
Jailbreaking large language models through alignment vulnerabilities in out-of-distribution settings.
*arXiv preprint arXiv:2406.13662*, 2024d.
- Huang et al. (2025a)
Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al.
On the trustworthiness of generative foundation models: Guideline, assessment, and perspective.
*arXiv preprint arXiv:2502.14296*, 2025a.
- Huang et al. (2025b)
Yue Huang, Zhengzhe Jiang, Xiaonan Luo, Kehan Guo, Haomin Zhuang, Yujun Zhou, Zhengqing Yuan, Xiaoqi Sun, Jules Schleinitz, Yanbo Wang, et al.
Chemorch: Empowering llms with chemical intelligence via synthetic instructions.
*arXiv preprint arXiv:2509.16543*, 2025b.
- Huang et al. (2025c)
Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang.
Datagen: Unified synthetic dataset generation via large language models.
In *The Thirteenth International Conference on Learning Representations*, 2025c.
URL [https://openreview.net/forum?id=F5R0lG74Tu](https://openreview.net/forum?id=F5R0lG74Tu).
- Inan et al. (2023)
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa.
Llama guard: Llm-based input–output safeguard for human–ai conversations.
*arXiv preprint arXiv:2312.06674*, 2023.
URL [https://arxiv.org/abs/2312.06674](https://arxiv.org/abs/2312.06674).
- Lee et al. (2024)
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo.
Aligning to thousands of preferences via system message generalization.
*Advances in Neural Information Processing Systems*, 37:73783–73829, 2024.
- Li et al. (2025)
Ang Li, Yin Zhou, Vethavikashini Chithrra Raghuram, Tom Goldstein, and Micah Goldblum.
Commercial llm agents are already vulnerable to simple yet dangerous attacks.
*arXiv preprint arXiv:2502.08586*, 2025.
- Lin et al. (2023)
Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang.
Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation.
*arXiv preprint arXiv:2310.17389*, 2023.
- Liu et al. (2025)
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al.
Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems.
*arXiv preprint arXiv:2504.01990*, 2025.
- Liu et al. (2024)
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al.
Best practices and lessons learned on synthetic data.
*arXiv preprint arXiv:2404.07503*, 2024.
- Luo et al. (2025a)
Hanjun Luo, Shenyu Dai, Chiming Ni, et al.
Agentauditor: Human-level safety and security evaluation for llm agents.
*arXiv preprint arXiv:2506.00641*, 2025a.
- Luo et al. (2025b)
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al.
Large language model agent: A survey on methodology, applications and challenges.
*arXiv preprint arXiv:2503.21460*, 2025b.
- Luo et al. (2025c)
Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, and Chaowei Xiao.
Agrail: A lifelong agent guardrail with effective and adaptive safety detection.
*arXiv preprint arXiv:2502.11448*, 2025c.
- Luo et al. (2023)
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang.
An empirical study of catastrophic forgetting in large language models during continual fine-tuning.
*arXiv preprint arXiv:2308.08747*, 2023.
- Ma (2025)
Yingning Ma.
Realsafe: Quantifying safety risks of language agents in real-world.
In *Proceedings of the 31st International Conference on Computational Linguistics*, pp. 9586–9617, Abu Dhabi, UAE, 2025. Association for Computational Linguistics.
URL [https://aclanthology.org/2025.coling-main.642/](https://aclanthology.org/2025.coling-main.642/).
- Meta AI (2023)
Meta AI.
Purplellama / prompt guard.
[https://github.com/meta-llama/PurpleLlama](https://github.com/meta-llama/PurpleLlama), 2023.
- OpenRouter (2025)
OpenRouter.
Openrouter: Unified api access to open and commercial language models.
[https://openrouter.ai/](https://openrouter.ai/), 2025.
Accessed: September 16, 2025.
- Padhi et al. (2025a)
Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, and Prasanna Sattigeri.
Granite guardian: Comprehensive LLM safeguarding.
In Weizhu Chen, Yi Yang, Mohammad Kachuee, and Xue-Yong Fu (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)*, pp. 607–615, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics.
ISBN 979-8-89176-194-0.
doi: 10.18653/v1/2025.naacl-industry.49.
URL [https://aclanthology.org/2025.naacl-industry.49/](https://aclanthology.org/2025.naacl-industry.49/).
- Padhi et al. (2025b)
Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, et al.
Granite guardian: Comprehensive LLM safeguarding.
In *Proc. NAACL Industry Track*, pp. 607–615, 2025b.
doi: 10.18653/v1/2025.naacl-industry.49.
URL [https://aclanthology.org/2025.naacl-industry.49](https://aclanthology.org/2025.naacl-industry.49).
- Pisano et al. (2024)
Matthew Pisano, Peter Ly, Abraham Sanders, Bingsheng Yao, Dakuo Wang, Tomek Strzalkowski, and Mei Si.
Bergeron: Combating adversarial attacks through a conscience-based alignment framework.
*arXiv preprint arXiv:2312.00029*, 2024.
URL [https://arxiv.org/abs/2312.00029](https://arxiv.org/abs/2312.00029).
- Qian et al. (2024)
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al.
Chatdev: Communicative agents for software development.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 15174–15186, 2024.
- Ravichandran et al. (2025)
Zachary Ravichandran, Alexander Robey, Vijay Kumar, George J. Pappas, and Hamed Hassani.
Safety guardrails for LLM-enabled robots.
*arXiv preprint arXiv:2503.07885*, 2025.
URL [https://arxiv.org/abs/2503.07885](https://arxiv.org/abs/2503.07885).
- Riabi et al. (2021)
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano.
Synthetic data augmentation for zero-shot cross-lingual question answering.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 7016–7030, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.562.
URL [https://aclanthology.org/2021.emnlp-main.562](https://aclanthology.org/2021.emnlp-main.562).
- Riaz et al. (2025)
Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, and Graham Horwood.
Metasynth: Meta-prompting-driven agentic scaffolds for diverse synthetic data generation.
*arXiv preprint arXiv:2504.12563*, 2025.
- Röttger et al. (2023)
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy.
Xstest: A test suite for identifying exaggerated safety behaviours in large language models.
*arXiv preprint arXiv:2308.01263*, 2023.
- Schick & Schütze (2021)
Timo Schick and Hinrich Schütze.
Generating datasets with pretrained language models.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 6943–6951, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.555.
URL [https://aclanthology.org/2021.emnlp-main.555/](https://aclanthology.org/2021.emnlp-main.555/).
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Shi et al. (2025)
Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun.
Prompt injection attack to tool selection in llm agents.
*arXiv preprint arXiv:2504.19793*, 2025.
- Tang et al. (2024)
Liyan Tang, Philippe Laban, and Greg Durrett.
Minicheck: Efficient fact-checking of llms on grounding documents.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, 2024.
URL [https://arxiv.org/pdf/2404.10774](https://arxiv.org/pdf/2404.10774).
- Tang et al. (2025)
Xiangru Tang, Qiao Jin, Kunlun Zhu, et al.
Risks of ai scientists: Prioritizing safeguarding over autonomy.
*arXiv preprint arXiv:2402.04247*, 2025.
- Tian et al. (2024)
Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su.
Evil geniuses: Delving into the safety of llm-based agents.
*arXiv preprint arXiv:2311.11855*, 2024.
- Vijayvargiya et al. (2025)
Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, and Maarten Sap.
Openagentsafety: A comprehensive framework for evaluating real-world ai agent safety.
*arXiv preprint arXiv:2507.06134*, 2025.
- Wang et al. (2025a)
Haoyu Wang, Christopher M. Poskitt, and Jun Sun.
Agentspec: Customizable runtime enforcement for safe and reliable llm agents.
*arXiv preprint arXiv:2503.18666*, 2025a.
- Wang et al. (2025b)
Kun Wang et al.
A comprehensive survey in llm(-agent) full stack safety: Data, training and deployment.
*arXiv preprint arXiv:2504.15585*, 2025b.
- Wei et al. (2023)
Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le.
Simple synthetic data reduces sycophancy in large language models.
*arXiv preprint arXiv:2308.03958*, 2023.
- Wu et al. (2023)
Lianmin Wu, Zihan Zhang, Xiaoyang Li, Yifan Hao, Yifan Jiang, Tianjun Chen, Zi Liu, Dawn Song, and Ion Stoica.
Autogen: Enabling next-gen llm applications via multi-agent conversation.
*arXiv preprint arXiv:2308.08155*, 2023.
URL [https://arxiv.org/abs/2308.08155](https://arxiv.org/abs/2308.08155).
- Xiang et al. (2025)
Zhen Xiang, Linzhi Zheng, Yanjie Li, et al.
Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning.
*arXiv preprint arXiv:2406.09187*, 2025.
- Xu et al. (2025)
Ran Xu, Yuchen Zhuang, Yishan Zhong, Yue Yu, Xiangru Tang, Hang Wu, May D Wang, Peifeng Ruan, Donghan Yang, Tao Wang, et al.
Medagentgym: Training llm agents for code-based medical reasoning at scale.
*arXiv preprint arXiv:2506.04405*, 2025.
- Xu et al. (2024)
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin.
Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.
*arXiv preprint arXiv:2406.08464*, 2024.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yin et al. (2025)
Sheng Yin, Xianghe Pang, Yuanzhuo Ding, et al.
Safeagentbench: A benchmark for safe task planning of embodied llm agents.
*arXiv preprint arXiv:2412.13178*, 2025.
- Yuan et al. (2024)
Tongxin Yuan, Zhiwei He, Lingzhong Dong, et al.
R-judge: Benchmarking safety risk awareness for llm agents.
*arXiv preprint arXiv:2401.10019*, 2024.
- Zeng et al. (2024)
Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al.
Shieldgemma: Generative ai content moderation based on gemma.
*arXiv preprint arXiv:2407.21772*, 2024.
- Zhang et al. (2025a)
Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang.
Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents.
In *The Thirteenth International Conference on Learning Representations*, 2025a.
- Zhang et al. (2025b)
Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang.
Oasis: One image is all you need for multimodal instruction data synthesis.
*arXiv preprint arXiv:2503.08741*, 2025b.
- Zhang et al. (2025c)
Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang.
Agent-safetybench: Evaluating the safety of llm agents.
*arXiv preprint arXiv:2412.14470*, 2025c.
- Zhao et al. (2024)
Jiawei Zhao, Kejiang Chen, Xiaojian Yuan, Yuang Qi, Weiming Zhang, and Nenghai Yu.
Silent guardian: Protecting text from malicious exploitation by large language models.
*IEEE Trans. Inf. Forensics & Security*, 2024.
doi: 10.1109/TIFS.2024.3455775.
URL [https://arxiv.org/abs/2312.09669](https://arxiv.org/abs/2312.09669).
- Zhao et al. (2023)
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.
(inthe) wildchat: 570k chatgpt interaction logs in the wild.
In *The Twelfth International Conference on Learning Representations*, 2023.
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*Advances in neural information processing systems*, 36:46595–46623, 2023.
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL [http://arxiv.org/abs/2403.13372](http://arxiv.org/abs/2403.13372).
- Zhong et al. (2024)
Ziije Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang.
Synthet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task.
*arXiv preprint arXiv:2406.10710*, 2024.
- Zhou et al. (2025)
Jialong Zhou, Lichao Wang, and Xiao Yang.
GUARDIAN: Safeguarding LLM multi-agent collaborations with temporal graph modeling.
*arXiv preprint arXiv:2505.19234*, 2025.
URL [https://arxiv.org/abs/2505.19234](https://arxiv.org/abs/2505.19234).
- Zhu et al. (2025)
Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, et al.
Safescientist: Toward risk-aware scientific discoveries by llm agents.
*arXiv preprint arXiv:2505.23559*, 2025.
- Zou et al. (2023)
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models.
*arXiv preprint arXiv:2307.15043*, 2023.

## Appendix Contents

## Appendix A Case Study on Real Scenarios

Figure: Figure 10: Two typical agentic systems.
Refer to caption: x9.png

While our earlier evaluations quantify pre-execution safety on isolated prompts and controlled plan fragments, real deployments increasingly rely on *agentic systems* that orchestrate multiple roles and long-horizon workflows. In such settings, (i) risk accumulation can amplify seemingly minor local defects into globally hazardous outcomes; (ii) authority gradients (e.g., expert roles or a mission director) can induce deference, allowing a single risky suggestion to cascade; and (iii) threat surfaces expand from single-message inputs to multiple *injection points* distributed across stages. To assess whether a pre-execution guardrail remains effective under these deployment-specific pressures, we conduct an end-to-end case study within two representative multi-agent paradigms.

We instantiate multi-agent environments inspired by MetaGPT (Hong et al., 2024) and AutoGen (Wu et al., 2023), chosen to contrast two widely used orchestration patterns:

Figure: Figure 11: Results of the proposed guardrail on the case study. We show the results with (i.e., w/) and without (i.e., w/o) the adapter.
Refer to caption: x10.png

*(Left Figure) Linear pipeline.*
Roles (e.g., planner $\rightarrow$ solver $\rightarrow$ reviewer) communicate in a feed-forward chain; each role’s output becomes the next role’s input until a final action list and response are produced. We inject risks at the generation stage of each role. Although an individual perturbation may be subtle, composition over depth can aggregate multiple low-grade issues into overtly unsafe trajectories.

*(Right Figure) Expert-driven workflow.*
Designated experts provide guidance that is synthesized by a Mission Director. Here, risks are injected at both the expert and Mission Director stages. Because expert outputs carry elevated credibility and the director concentrates decision authority, even single-shot (or simultaneous) injections can produce disproportionately dangerous behaviors, capturing a qualitatively different failure mode from the linear setting.

To approximate realistic deployment failures and reduce overlap with earlier synthetic stressors, we adopt a *human expert–driven* risk injection protocol at the *trajectory* level. We first generate benign trajectories with AuraGen, then have two human experts independently edit these trajectories into harmful variants (details in [Appendix L](https://arxiv.org/html/2510.09781v1#A12)); each harmful sample is cross-validated by the other expert. This yields a balanced set of 50 harmful and 50 harmless trajectories for each agentic system. While the dataset is small by necessity (manual curation), it suffices to highlight qualitatively distinct failure modes. Relative to prior sections, this design introduces distributional shift (human-curated edits), multi-point injections (per role), and authority-biased cascades (expert/director), enabling us to test whether the guardrail’s pre-execution checks generalize beyond single-agent, single-turn evaluations. More framework and orchestration implementation details are provided in the accompanying code in the supplementary materials.

As shown in [Figure 11](https://arxiv.org/html/2510.09781v1#A1.F11), our proposed guardrail achieves strong performance in real-world multi-agent scenarios, reaching over 90% classification accuracy when equipped with the Adapter. The Adapter further yields consistent gains across all three evaluation metrics (classification, risk categorization, and explanation), underscoring its importance in handling heterogeneous trajectory formats. While performance is somewhat lower than on the Pre-Exec Bench—primarily due to the distributional shift introduced by human-injected risks and multi-point role-level perturbations—the guardrail nevertheless maintains reliable effectiveness, demonstrating robustness under complex, deployment-oriented settings.

## Appendix B Guardrail Baseline Comparison

Beyond comparing with general LLMs, we also evaluate two widely used guardrail frameworks on Pre-Exec Bench. The first is Llama-Guard-3-8B (Grattafiori & the Llama Team at Meta, 2024), and the second is LlamaFireWall (Chennabasappa et al., 2025). For Llama-Guard-3-8B, we follow the official example provided at [https://huggingface.co/meta-llama/Llama-Guard-3-8B](https://huggingface.co/meta-llama/Llama-Guard-3-8B)
, where the user input corresponds to the user request and the assistant output corresponds to the agent’s planned trajectories. For LlamaFireWall, we conduct experiments on two of its modules: (i) the basic scanning function (see [https://meta-llama.github.io/PurpleLlama/LlamaFirewall/docs/documentation/getting-started/how-to-use-llamafirewall](https://meta-llama.github.io/PurpleLlama/LlamaFirewall/docs/documentation/getting-started/how-to-use-llamafirewall)
, denoted as llamafirewall-basic in [Figure 12](https://arxiv.org/html/2510.09781v1#A2.F12)), and (ii) the alignment checker (see [https://github.com/meta-llama/PurpleLlama/tree/main/LlamaFirewall/examples](https://github.com/meta-llama/PurpleLlama/tree/main/LlamaFirewall/examples)
).

Figure: Figure 12: Classification accuracy of three guardrail baselines.
Refer to caption: x11.png

Disclaimer. These two guardrail frameworks were not designed for pre-execution safety evaluation. Their reported performance should therefore be interpreted as *indicative reference points* rather than as direct, fully fair baselines against our proposed framework.

As shown in [Figure 12](https://arxiv.org/html/2510.09781v1#A2.F12), all three baselines perform poorly, with classification accuracy remaining below 60%. This result highlights that existing guardrails, while useful in other contexts, cannot be straightforwardly applied to plan-level pre-execution risk detection—underscoring the need for specialized methods such as ours.

## Appendix C Related Work

Safety of Agentic System. Ensuring the safety of LLM-based agents is crucial as their autonomy and deployment scale (Wang et al., 2025b; Huang et al., 2024c). Recent works have addressed this through benchmarks, methodologies, and adversarial analyses. Evaluation benchmarks such as Agent-SafetyBench (Zhang et al., 2025c), R-Judge (Yuan et al., 2024), SafeAgentBench (Yin et al., 2025), and RealSafe (Ma, 2025) have systematically measured safety across diverse scenarios. For protective methodologies, TrustAgent (Hua et al., 2024) employs an explicit agent constitution; GuardAgent (Xiang et al., 2025) uses a secondary auditing agent with knowledge reasoning; AgentSpec (Wang et al., 2025a) provides customizable runtime enforcement; and Causal Influence Prompting (Hahm et al., 2025) mitigates risks via causal interventions. Specialized efforts such as SafeScientist (Zhu et al., 2025) and prioritizing safeguards over autonomy (Tang et al., 2025) target scientific contexts. In adversarial research, Evil Geniuses (Tian et al., 2024) demonstrates sophisticated bypass techniques, while AgentAuditor (Luo et al., 2025a) achieves near-human audit accuracy.

Guardrail for LLM(-based Agents). LLM guard models are widely applied in downstream deployment systems (Dong et al., 2024) to defend malicious attacks like jailbreak Zou et al. (2023); Huang et al. (2024d). Llama Guard inaugurates LLM safety by fine-tuning models to classify prompts and responses across a bespoke safety taxonomy (Inan et al., 2023). IBM’s Granite Guardian (Padhi et al., 2025b) expands detection to bias, profanity, jailbreaks, hallucination, and groundedness of RAG, topping the GuardBench leaderboard (Bassani & Sanchez, 2024). The most recent release Granite Guardian 3.3 is top-3 on LLM-AggreFact leaderboard (Tang et al., 2024) and also supports thinking mode with additional capabilities such as tool-call hallucination detecion. Other popular guardian models include ShieldGemma (Zeng et al., 2024), ToxicChat-T5 (Lin et al., 2023), and WildGuard (Han et al., 2024). Beyond single-agent chat, Zhou et al. (2025) propose GUARDIAN to model multi-agent conversations as temporal graphs to arrest hallucination propagation. Silent Guardian embeds adversarial tokens that cause compliant models to halt generation, achieving near-100% refusal rates (Zhao et al., 2024), while Bergeron deploys a secondary “conscience” LLM to monitor a primary model and multiplies attack resistance seven-fold (Pisano et al., 2024). Meta’s open-source Prompt Guard toolkit enables rule-based prompt filtering and evaluation pipelines for production systems (Meta AI, 2023). A data-free methodology trains off-topic detectors without real user logs, thereby easing the deployment of guardrails before launch (Chua et al., 2025). In robotics, RoboGuard fuses temporal-logic synthesis with an LLM “root-of-trust” to keep physical agents safe under jailbreak attacks (Ravichandran et al., 2025). Some recent works focus on the safety of agentic systems (Luo et al., 2025a; c; Chen et al., 2025; Xiang et al., 2025; Chennabasappa et al., 2025); however, as summarized in [Table 3](https://arxiv.org/html/2510.09781v1#A3.T3), they still fall short in (i) comprehensive risk coverage, (ii) keeping human cost low for evaluation and data construction, (iii) rapid adaptivity to new scenarios and emerging risks, (iv) input generalization across heterogeneous formats/modalities, and (v) explanation–cost trade-offs suitable for real-time monitoring. AgentAuditor (Luo et al., 2025a) covers a broad set of risks but relies heavily on human annotation and is not designed for low-latency guardianship, leading to high cost and low efficiency in explanation. AGrail (Luo et al., 2025c) demonstrates high adaptivity through adaptive safety-check generation and test-time adaptation, though its reliance on curated benchmarks and moderate input flexibility keeps both human cost and input generalization at the medium level. SHIELDAGENT (Chen et al., 2025) achieves medium risk coverage but provides strong explanation signals with efficient rule circuits, hence scoring high on the explanation–cost trade-off, while its adaptivity depends on continuous rule engineering. GuardAgent (Xiang et al., 2025) excels at adapting to new tasks by uploading new functions, yet its benchmarks involve expert annotation and its explanations are code-based, resulting in higher human cost and only medium real-time suitability. Finally, LlamaFirewall (Chennabasappa et al., 2025) emphasizes prompt injection and code risks with lightweight detectors, yielding low annotation cost and efficient explanations; however, its coverage is narrower and adaptivity to unseen scenarios remains limited.

LLMs in Synthetic Data. LLMs have demonstrated exceptional ability in producing synthetic data (Liu et al., 2024). In contrast to earlier techniques that relied on conventional language models (Schick & Schütze, 2021), modern LLMs present enhanced potential for generating high-quality synthetic datasets across numerous fields. These include areas such as multilingual question answering (Riabi et al., 2021), conversational systems (Zhao et al., 2023), instruction tuning (Xu et al., 2024; Zhang et al., 2025b; Zhong et al., 2024), improving factual accuracy (Wei et al., 2023), scentific capabilities (Huang et al., 2025b), and increasing dataset diversity (Dai et al., 2025; Chung et al., 2023; Riaz et al., 2025). Recently, the DataGen framework (Huang et al., 2025c) was proposed to create high-quality text datasets, supporting more precise evaluation and refinement of LLMs. Likewise, Janus, developed by Lee et al., is an LLM trained using a broad set of synthetic system messages aimed at fostering both personalized and general alignment (Lee et al., 2024). Therefore, the strong potential of LLMs in synthetic data generation can serve as a key avenue for obtaining high-quality training data for guardian models.

**Table 3: Comparison of related guardrail for agentic systems across five dimensions. “Risk Coverage” reports the *count* of our eight risk categories covered. “Human Cost” counts the cost of human involvement, including evaluation and data construction; lower is better. “Adaptivity” denotes a guardrail’s ability to adapt to new scenarios and expand to new risks quickly. “Input Generalization” denotes the ability to robustly consume heterogeneous input formats/modalities (e.g., different log schemas, markup, structured outputs) with minimal task-specific engineering. “Exp.–Cost Trade-off” rates suitability for real-time monitoring (balancing explanations and token/runtime overhead).**
| Related Work | Risk<br>Coverage | Human Cost | Adaptivity | Input<br>Generalization | Exp.–Cost<br>Trade-off |
| --- | --- | --- | --- | --- | --- |
| AgentAuditor<br>(Luo et al., 2025a) | 5 | High | Medium | Medium | Low |
| AGrail<br>(Luo et al., 2025c) | 5 | Medium | High | Medium | Medium |
| SHIELDAGENT<br>(Chen et al., 2025) | 4 | Medium | Medium | High | High |
| GuardAgent<br>(Xiang et al., 2025) | 2 | High | High | Medium | Medium |
| LlamaFirewall<br>(Chennabasappa et al., 2025) | 4 | Low | Low | Medium | High |
| Ours | 8 | Low | High | High | High |

## Appendix D Evaluation Metrics

We report four evaluation metrics to assess the model’s performance.
Let $N$ denote the total number of samples, $H$ the set of ground-truth harmful samples, and $\hat{y}_{i}$ the model’s prediction for sample $i$.

#### (1) Classification Accuracy.

This metric measures the overall correctness of harmless/harmful classification across all samples:

$$ $\text{Acc}_{\text{cls}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left(\hat{y}_{i}^{\text{cls}}=y_{i}^{\text{cls}}\right),$ (1) $$

where $y_{i}^{\text{cls}}\in\{\text{harmless},\text{harmful}\}$.

#### (2) Harmful Detection Precision.

This metric is restricted to the ground-truth harmful subset ($i\in H$), and evaluates whether the model correctly identifies them as harmful:

$$ $\text{Acc}_{\text{harm}}=\frac{1}{|H|}\sum_{i\in H}\mathbf{1}\!\left(\hat{y}_{i}^{\text{cls}}=\text{harmful}\right).$ (2) $$

#### (3) Risk Category Accuracy.

Once a sample is correctly detected as harmful, we further evaluate whether the predicted risk label matches the ground-truth risk label $y_{i}^{\text{risk}}$. Denote this subset as $H^{\text{det}}=\{i\in H\mid\hat{y}_{i}^{\text{cls}}=\text{harmful}\}$. Then:

$$ $\text{Acc}_{\text{risk}}=\frac{1}{|H^{\text{det}}|}\sum_{i\in H^{\text{det}}}\mathbf{1}\!\left(\hat{y}_{i}^{\text{risk}}=y_{i}^{\text{risk}}\right).$ (3) $$

#### (4) Explanation Correctness.

For the cases where the risk category is correctly predicted, we assess whether the model’s explanation semantically aligns with the ground-truth explanation $y_{i}^{\text{exp}}$. This is determined by an LLM-based judge returning a binary decision. Let $H^{\text{risk}}=\{i\in H^{\text{det}}\mid\hat{y}_{i}^{\text{risk}}=y_{i}^{\text{risk}}\}$. Then:

$$ $\text{Acc}_{\text{exp}}=\frac{1}{|H^{\text{risk}}|}\sum_{i\in H^{\text{risk}}}\mathbf{1}\!\left(\text{Judge}(\hat{y}_{i}^{\text{exp}},y_{i}^{\text{exp}})=1\right).$ (4) $$

## Appendix E Risk Definition

We show the definition of different risk types in [Table 4](https://arxiv.org/html/2510.09781v1#A5.T4).

**Table 4: The description of different risk types.**
| Risk | Description |
| --- | --- |
| Sensitive data privacy violations | Unauthorized collection, use, disclosure, or leakage of sensitive, confidential, or personal information. |
| Property and financial loss | Actions that directly or indirectly cause financial loss or damage to assets. |
| Misinformation and unsafe content | Generation or dissemination of false, misleading, harmful, or unsafe information. |
| Compromised availability | Disruption or degradation of service availability, including resource exhaustion or denial of service. |
| Unintended or unauthorized actions | Execution of actions beyond user intent or permissions, including escalation of privileges and unauthorized automation. |
| External adversarial attack | Susceptibility to external adversarial attacks including prompt injection, jailbreak attempts, input manipulation, adversarial examples, and other malicious inputs designed to cause unintended agent behavior or bypass security controls. |
| Bias and discrimination | Outputs or decisions reflecting unfair bias, discrimination, or lack of fairness. |
| Lack of accountability and traceability | Insufficient logging or explainability, making it difficult to audit or assign responsibility for agent actions. |

## Appendix F Details of AuraGen

AuraGen is designed as a flexible and extensible synthetic data engine, allowing users to easily configure, control, and scale trajectory generation for diverse safety research needs.

Flexibility. AuraGen supports multiple deployment backends. For local use, it integrates with mainstream models on HuggingFace; for API-based access, it is compatible with commercial providers (OpenAI, Claude, Google Gemini) as well as relay services such as DeepInfra(^2^22[https://deepinfra.com/](https://deepinfra.com/))
, Replicate(^3^33[https://replicate.com/](https://replicate.com/))
, and OpenRouter(^4^44[https://openrouter.ai/](https://openrouter.ai/))
. All workflows are streamlined through configurable files and an intuitive interface.

Controllability. AuraGen enables guided generation via constraints. Users can specify attributes directly in the configuration file to shape generation behavior. Some used constraint examples are illustrated in [Figure 13](https://arxiv.org/html/2510.09781v1#A6.F13).

Customizable risk injection. AuraGen also supports user-defined policies for injecting risks. As shown in [Figure 14](https://arxiv.org/html/2510.09781v1#A6.F14), users may set explicit probabilities for risk occurrence under different scenarios, or alternatively delegate probability estimation to the LLM, which infers likelihoods based on scenario–risk alignment.

Figure: Figure 13: Representative constraint types used to guide AuraGen’s generation process.
Refer to caption: x12.png

Figure: Figure 14: User interface for configuring risk–scenario constraints.
Refer to caption: risk_scenario_constraint_screenshot.png

Scenarios in AuraGen. To approximate realistic agentic environments, we designed AuraGen scenarios by integrating two complementary sources. The first source is the OpenAI GPT Store, which offers a wide range of user-facing applications. However, since the underlying tool specifications are not publicly released, we could not directly access them. Instead, we reconstructed the corresponding tool functions manually based on scenario descriptions, ensuring that each case remained executable while faithfully reflecting the original tasks. The second source comes from crawling multiple MCP server websites (e.g., [https://mcpservers.org/](https://mcpservers.org/)), from which we extracted environment information. To guarantee diversity and richness, we retained only those servers that provided a sufficient number of tools and discarded overly minimal cases. To avoid potential copyright or commercial issues, we anonymized several platforms—for instance, a real-world travel booking provider was abstracted into a more generic “traveling ticket purchase platform”.

After collecting scenarios from these two sources, we further refined them with the help of Mixtral-8$\times$22B, which was used to polish descriptions, enrich tool functions, and generate representative examples. The refined scenarios were then manually checked to ensure coherence, correctness, and compliance with ethical considerations. Notably, AuraGen is designed with generalization in mind: the toolkit allows users to seamlessly introduce new scenarios through simple configuration files. This modularity enables continuous expansion into unseen domains, supporting both research flexibility and adaptation to rapidly evolving agentic systems.

We show the statistics of the current scenarios in AuraGen in [Figure 15](https://arxiv.org/html/2510.09781v1#A6.F15), and show some scenario examples in [Figure 16](https://arxiv.org/html/2510.09781v1#A6.F16) and [Figure 17](https://arxiv.org/html/2510.09781v1#A6.F17).

Figure: Figure 15: Details of scenarios and environment information.
Refer to caption: x13.png

Figure: Figure 16: The examples of scenarios in AuraGen, including category, description, and available tools (1).
Refer to caption: x14.png

Figure: Figure 17: The examples of scenarios in AuraGen, including category, description, and available tools (2).
Refer to caption: x15.png

Figure: Figure 18: Evaluation of reward model performance. *Left:* “w/” denotes the inclusion of criterion-specific examples in the input prompt during training and evaluation, whereas “w/o” indicates their removal. *Right:* Average score difference for each evaluation aspect.
Refer to caption: x16.png

## Appendix G Details of Reward Model Training

To avoid the prohibitive cost of manual labeling, we train on synthetic data, which aligns closely with human validation. In our human pilot study, the correlation between RM scores assigned by LLMs and human annotators was found to be high (As shown in [Table 5](https://arxiv.org/html/2510.09781v1#A7.T5)), suggesting that LLM-based annotations can serve as a reliable substitute for human labels.

Specifically, we sample $1{,}700$ instances from the previously generated synthetic agent trajectories. Each instance contains (1) the original action trajectory, (2) the corresponding user query, (3) the injected risky trajectory, and (4) the environment information.

Annotation model.
We adopt DeepSeek-R1 as the annotation model to score each data sample along five criteria as shown in [Table 6](https://arxiv.org/html/2510.09781v1#A7.T6). For each criterion, the model outputs an integer score in $\{1,2,3,4,5\}$ and a corresponding natural language feedback string. This yields a tuple $(\mathbf{s},f)$ for each sample, where $\mathbf{s}\in\mathbb{Z}^{5}$ denotes the score vector and $f$ is the feedback text.

Metrics. We evaluate RM performance with two metrics: (1) *score difference*, the total deviation across five criteria from ground-truth scores; and (2) *instability rate*, the fraction of criteria with absolute deviation $>2$.

As shown in [Figure 18](https://arxiv.org/html/2510.09781v1#A6.F18), adding criterion-specific examples (w/) yields high accuracy and stable behavior. In this setting, per-criterion average error stays below $1.1$, indicating uniformly small deviations. While w/o shows slightly lower aggregate error, qualitative inspection shows w/ better aligns with human judgments, especially on nuanced criteria such as Justification Sufficiency and Risk Matching. We therefore adopt w/ as the default for quality assurance.

While the reward model exhibits stable performance in scoring, its ultimate purpose is to serve as a filter for synthetic data to enhance dataset quality. A straightforward approach is to apply threshold-based filtering, retaining only samples whose scores exceed a pre-defined cutoff. However, as shown in [Table 1](https://arxiv.org/html/2510.09781v1#S6.T1), such rule-based methods (AVG or ALL) yield mixed results across evaluation metrics. For instance, both AVG$>$2 and AVG$>$1.5 degrade performance on Risk Cat. Acc. and Expl. Corr., suggesting that simple threshold-based methods may discard many useful samples.

To address these shortcomings, we introduce a classification-based filtering mechanism. Specifically, we use the binary annotations during Pre-Exec Bench construction on whether to discard a sample to train a lightweight classifier (i.e., SVM (Cortes & Vapnik, 1995)) to mimic this filtering behavior. The classifier input is the $n$-dimensional vector of reward model scores across evaluation aspects, and the output corresponds to the keep/discard decision.This approach yields clear improvements.

Details of lightweight classifier. We adopt an SVM ($\text{kernel}=\texttt{rbf},\;C=10,\;\gamma=\texttt{scale}$) as the classifier and train it using the raw data from benchmark construction. In total, approximately 1,400 samples are collected for training, with a balanced negative-to-positive ratio of 1:1. The classifier achieves an evaluation accuracy of 86.93% on the test set, demonstrating its reliability in detecting low-quality injected samples.

Why not train the reward model itself to produce binary outputs (retain vs. discard) instead of introducing a separate classifier? We deliberately avoid this design for two reasons: First, the reward model is designed as a fine-grained scorer across multiple evaluation aspects, which allows it to provide rich, disentangled signals rather than a single coarse decision. Directly training the RM to output 0/1 labels would collapse these dimensions into a single objective, thereby discarding valuable information about nuanced qualities such as justification sufficiency or risk calibration. By preserving aspect-level scores, we retain interpretability and flexibility, enabling different downstream policies to be applied without retraining the RM. Second, binary annotation data is typically scarcer and noisier compared to preference-style or aspect-level supervision. Training the RM to predict discrete keep/discard labels would tightly couple its capacity to the availability and consistency of such labels, likely leading to reduced generalization. In contrast, our two-stage approach leverages the stability of aspect-level RM scores and only requires a lightweight classifier to capture human discard preferences. This design ensures modularity: the RM remains a general-purpose evaluator, while the classifier serves as a policy layer that can be retrained or adapted with minimal cost.

**Table 5: Correlation analysis of DeepSeek-R1 and human evaluation.**
| Aspect | Group 1 | Group 2 | Group 3 |
| --- | --- | --- | --- |
| Causal Consistency | 0.7973p<0.001 | 0.8273p<0.001 | 0.4566p<0.05 |
| Postcondition Continuity | 0.8202p<0.001 | 0.8152p<0.001 | 0.8171p<0.001 |
| Rationality | 0.7171p<0.001 | 0.8459p<0.001 | 0.7583p<0.001 |
| Justification Sufficiency | 0.6564p<0.001 | 0.8345p<0.001 | 0.8116p<0.001 |
| Risk Matching | 0.8652p<0.001 | 0.7921p<0.001 | 0.8755p<0.001 |

**Table 6: Criteria definition of reward model.**
| Evaluation Criteria | Definition |
| --- | --- |
| Causal Consistency | Evaluate whether the injected actions maintain logical consistency within the action sequence. |
| Postcondition Continuity | The injected action’s effects must leave the system in a state that satisfies the next original action’s preconditions. |
| Rationality | The attacker’s overall utility from the injected chain must sufficiently exceed its total cost. |
| Justification Sufficiency | Each injected risk must be accompanied by a concise explanation that clearly motivates “why” the risk occurs in context. |
| Risk Matching | The semantic content of the injected action must align with its assigned risk label. |

Reward Model Training setup.
We initialize the RM from the mistralai/Ministral-8B-Instruct-2410. The dataset is split into $1{,}400$ samples for testing and $300$ samples for training. We fine-tune the base model using supervised fine-tuning (SFT) to predict the five-dimensional score vector given the metadata $(E,U,C)$, the user query $q$, and the risky trajectory $T_{\text{risk}}$. All experiments are conducted with identical hyperparameters across runs to ensure fair comparison. We show the input prompt of the reward model as follows:

## Appendix H Cost & Latency Analysis

We report approximate per-sample costs based on average token counts
(1,002 input, 1,324 output).
Under official list prices in Sep. 2025, the cost of generating one data point with GPT-5 is below $0.02.
For comparison, recent open-source APIs hosted on OpenRouter
are substantially cheaper, yielding per-sample costs an order of magnitude lower:

- •
DeepSeek V3.1: $0.27/M input, $1.00/M output.
- •
gpt-oss-120b: $0.072/M input, $0.28/M output.

These prices imply significantly lower costs under identical token lengths,
supporting the scalability of our data generation pipeline.

#### Latency Analysis.

In addition to cost, we also benchmarked inference latency under different GPU settings.
With concurrent inference enabled, our guardrail system achieves:

- •
H100$\times$8: 33 samples/second on average.
- •
A100$\times$8 (40GB): 3.7 samples/second on average.

These results demonstrate that, under reasonable GPU provisioning,
the latency of our approach remains fully acceptable and does not pose
a bottleneck for large-scale data generation.

## Appendix I Details of Adapter Training

Figure: Figure 19: Log style distribution in the dataset.
Refer to caption: x17.png

To enable robust normalization of heterogeneous outputs produced by diverse agentic systems, we first established a set of canonical output styles, covering both structured and semi-structured formats.The formats span a wide range, including XML, Tab Separated, Timestamp Epoch, Semicolon Single, Bullets, Markdown, JSON Compact, JSON Pretty, Numbered Steps, and Key–Value. These formats were selected to reflect real-world agent outputs observed across multiple platforms, covering variations in serialization syntax, layout conventions, and human-readable documentation styles.

Dataset construction followed a dual-source synthesis strategy designed to balance structural determinism with stylistic diversity:

Programmatic Synthesis. We implemented a deterministic conversion pipeline in Python, which directly transformed data in the unified target schema into each of the pre-defined formats. This ensured exact coverage of all format variants and allowed controlled manipulation of structural elements (e.g., ordering of keys, indentation depth, delimiter types). However, the purely script-driven process lacked natural linguistic variability and did not fully capture the idiosyncratic deviations present in real system logs.

LLM-based Transformation. To introduce stylistic diversity and realistic noise, we employed a LLM to automatically rewrite the programmatically generated outputs. The LLM was prompted to (i) vary lexical and syntactic choices, (ii) adjust formatting in plausible but imperfect ways (e.g., inconsistent indentation, partially missing keys, embedded comments), and (iii) simulate naturally occurring variations found in agent trajectories. This step injected randomness while preserving semantic fidelity to the original content.

To further enhance domain realism, both synthesis methods were grounded on the agent trajectory generated by AuraGen. The final dataset was a balanced mixture of programmatically synthesized and LLM-augmented samples, and the style distribution is shown in [Figure 19](https://arxiv.org/html/2510.09781v1#A9.F19). We curated the collection to maximize coverage of format types and variability in structure, style, and noise. The dataset was split into 3,152 samples for training and 788 samples for testing. Human validation was conducted by two annotators (one PhD, one undergraduate in CS) to ensure the correctness of the test set.

**Table 7: Prompts used to synthesize heterogeneous agent logs in ten output formats.**
| Format | LLM Prompt (verbatim) |
| --- | --- |
| XML | Provide the log as XML. Use <log><action>…</action>…<response>…</response></log>. |
| Tab Separated | Output a tab-separated log: first column is an incremental counter, second column ACTION, third column the action text. Final line should use the next counter value, the word RESPONSE, and the agent response, all separated by tabs. |
| Timestamp Epoch | Produce a plain-text log where each line begins with the Unix epoch time in seconds, then a space and a LEVEL tag (INFO, WARN, ERROR), then the action. Finish with RESPONSE=<agent_response> on a new line. |
| Semicolon Single | Concatenate all actions in a single line separated by semicolons, then append => and the final response. |
| Bullets | Render the log as a bullet list where each bullet starts with [DBG] or [INF] chosen randomly, followed by the action text. Conclude with [RES] followed by the response. |
| Markdown | Return a Markdown section with a heading ### Agent Log, a bulleted list of actions, and a blockquote for the response. |
| JSON Compact | Produce a single-line JSON array where each element is an object with keys ’step’ and ’action’. After listing all actions, append an object with key ’response’. Do not pretty print or add spaces. |
| JSON Pretty | Create a pretty-printed JSON object with keys: ’actions’ (array of strings), ’result’ (string), and ’duration_ms’ (random plausible number). |
| Numbered Steps | List the actions as numbered steps using the format ‘Step N: <action>’. After listing all steps, add a separator line of dashes, then ‘Result: <agent_response>’. |
| Key–Value | Render the log as multiple lines of key=value pairs. For each action, use stepN=<action>. End with response=<agent_response>. |

## Appendix J Reproducibility of Training Safiron

All training was conducted on A100 (80GB) and H100 (80GB) GPUs. For supervised fine-tuning (SFT), we used the LLaMA-Factory framework (Zheng et al., 2024) with a maximum output length of 4096, a learning rate of 1.0e-05, 3 epochs, and a warmup ratio of 0.1. For reinforcement learning (RL), we employed the verl framework (Sheng et al., 2024) with a maximum prompt length of 4096 and a maximum response length of 1024 to prevent memory overflow. The number of rollouts was set to 32, training was conducted for 3 epochs, and the training batch size was 8.

## Appendix K Details of Experiment Evaluation

We adopt a hybrid evaluation method that combines keyword matching with LLM-as-a-Judge (Zheng et al., 2023) to balance efficiency and accuracy. For harmless data, we apply keyword matching using the string ‘‘There are no risks in these actions.’’, and perform pattern extraction to compare the predicted risk category with the ground truth for harmful data. The LLM-as-a-Judge is employed to assess the correctness of the predicted explanations. To ensure the accuracy of the LLM Judge, we conducted human evaluations and iteratively refined the prompt based on the evaluation results. On 128 data samples, the final prompt achieved a 100% alignment rate with human judgments. The prompt provided to the judge is as follows:

## Appendix L Details of Human Evaluation

We show the human evaluation interface and guideline in [Figure 20](https://arxiv.org/html/2510.09781v1#A12.F20), [Figure 21](https://arxiv.org/html/2510.09781v1#A12.F21), and [Figure 22](https://arxiv.org/html/2510.09781v1#A12.F22). For the human pilot study prior to reward model training, four AI research scientists and two CS PhD students are involved. For benchmark construction and filtering, two CS PhD students and one CS undergraduate student are involved. For the human expert risk injection in the case study, one PhD student and one undergraduate, both majoring in CS and AI security, are involved.

Figure: Figure 20: Human evaluation interface and guideline (1).
Refer to caption: human_eval_1.jpg

Figure: Figure 21: Human evaluation interface and guideline (2).
Refer to caption: human_eval_2.jpg

Figure: Figure 22: Human evaluation interface and guideline (3).
Refer to caption: phase_2_human_guideline.png

## Appendix M Toolkit Usage

We provide AuraGen as an easy-to-use toolkit. A quick-start example is shown in [Figure 23](https://arxiv.org/html/2510.09781v1#A13.F23), while detailed usage instructions and extended documentation are included in the docs folder of the supplementary materials.

Figure: Figure 23: Quick start of AuraGen.
Refer to caption: x18.png

## Appendix N Prompt Template

We show the input prompt for Safiron and adapter as follows. All prompt templates for synthesizing trajectories can be found in our AuraGen toolkit.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you report Safiron’s performance on non-synthetic agent traces collected from real AutoGen/MetaGPT runs (with user-written tools, occasional tool errors, and truncated steps)? Even a 200–300 sample subset would make the generalization claim much stronger.
2. Your adapter experiments remove two styles and show good generalization (Fig. 6). How does the adapter behave on malformed or partially missing tool calls (e.g., missing arguments field, extra telemetry, JSON5, HTML-in-text)? Do you discard samples or try to repair them?
3. Table 1 shows that the SVM filtering policy is best. Could you show an ablation where AuraGen uses (i) no RM, (ii) RM with a single criterion (e.g., causal consistency only), and (iii) RM with noisy labels, and then retrain Safiron? This would clarify how crucial the RM is.
4. You weigh the correct risk category at 1.0 and mismatched category at 0.5 during RL. Which categories are most often confused (e.g., availability vs. goal-hijacking)? Could fine-grained label smoothing or a hierarchy of risks improve this?
5. What is the average inference latency of the full pipeline (adapter + Safiron) on the Pre-Exec Bench input length, on a single A100/4090? Can it run inline for multi-agent systems where every plan is checked?
6. If a trajectory contains a novel or composite risk not in your pool R, does Safiron tend to (a) label it benign, (b) over-flag it to a close category, or (c) refuse? Can you add an “unknown / abstain” option?
7. You show that harmful:harmless ratios of 1:4–1:6 are best (Fig. 9). How sensitive is this to the synthetic corpus size—does the same ratio hold if you scale AuraGen from 20k to 200k?
8. Pre-Exec Bench is said to be “held out” from model selection. Will you release the human instructions and debiasing protocol so others can reproduce the same verification standard?
9. You compare to LlamaFirewall and LlamaGuard-3 in the appendix. Can you also compare to recent causal-influence prompting or runtime enforcement methods that operate over plans (e.g., AgentSpec, GuardAgent), evaluated on Pre-Exec Bench under the same metrics?
soundness: 3
strengths: 1. The paper focuses on pre-execution / planning-stage safety for agents, which is still under-served compared to execution-time or dialog I/O guardrails. Framing “look at the whole trajectory before anything is executed” is both sensible and impactful for high-stakes domains.
2. AuraGen is a clearly specified, controllable synthetic-data engine with four risk-injection strategies, a risk taxonomy, and automated RM-based filtering. This is more systematic than ad-hoc prompt-injection datasets and can be extended to new tools or environments.
3. The adapter + Safiron split is a good modular design: the adapter unifies planner outputs in heterogeneous formats, and the guardian only has to do safety reasoning on a normalized schema. This increases transferability across agent frameworks (MetaGPT, AutoGen, etc.).
4. Pre-Exec Bench is a nontrivial contribution—it is human-verified, covers ~1k benign and ~670 risky samples, uses a heterogeneous model pool to avoid style leakage, and directly targets the planning stage, which existing benchmarks do not.
summary: The paper targets pre-execution (planning-stage) safety for LLM-based agentic systems, arguing that most existing guardrails act after an action is executed, which is too late for high-stakes scenarios. To close what the authors call the data gap, model gap, and evaluation gap, the paper proposes three components: (i) AuraGen, a controllable synthetic-data engine that first generates benign, tool-using agent trajectories from structured metadata, then injects category-labeled risks with four principled strategies (single-step, multi-step, new-branch, bridged-branch), and finally filters them with an automated reward model; (ii) Safiron, a “foundational guardrail” composed of a cross-planner adapter to normalize different agent outputs and a compact guardian model trained with SFT + GRPO to do (a) binary detection, (b) fine-grained risk typing, and (c) rationale generation; and (iii) Pre-Exec Bench, a new, human-validated benchmark specifically for planning-stage safety that covers diverse tools, trajectories, and injected risks. Experiments show that Safiron consistently outperforms strong proprietary and open-weight baselines on classification accuracy, harmful detection, and risk categorization, while the adapter enables cross-format generalization.
weaknesses: 1. Even though the authors add two-phase human verification, the core training data and the benchmark seed are still LLM-synthesized, which raises the usual concern: will Safiron detect messy, partially malformed, or adversarially obfuscated real agent traces that do not follow the neat stepwise style AuraGen produces? A small “real logs” test set (e.g., from MetaGPT/AutoGen runs with user-created tools) would help.
2. The paper claims the adapter “generalizes across input styles” (Fig. 6), but all styles seem to be synthetically created or LLM-generated. It is unclear how well the adapter handles idiosyncratic, schema-changing production systems (e.g., extra telemetry fields, partial tool arguments, JSON-with-errors). Adding a “noisy/partial JSON + extra fields” stress test would strengthen the claim.
3. AuraGen’s usefulness depends heavily on the RM and the “classifier” filtering policy (Table 1). But the paper does not show ablation on RM quality (e.g., train with a weaker RM, or no RM, or RM without the five criteria) to prove that the final gains are not just from better filtering. A small ablation could make the pipeline more convincing.
4. The paper gives a per-sample cost estimate and says Safiron is compact, but it does not provide end-to-end latency numbers for “agent -> guardrail -> environment” in a running system. A short table with average inference time on typical trajectories would be helpful.
5. Pre-Exec Bench has 1,001 benign and 671 risky samples—good for research, but still relatively small and balanced across the 4 injection strategies (25% each) rather than reflecting real-world skew. This makes the reported numbers somewhat optimistic; adding a “realistic imbalanced” split would test over-flagging.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Q1: In section 3 it is stated that AuraGen produces “diverse and controllable trajectories spanning a wide spectrum of risks” – where are these claims supported?


Q2: in Section 4 it is stated that “RL complement it by optimizing for fine-grained safety objectives” (l206-207) – why does SFT not suffice here and why does RL solve this?


Q3: In Section 5 it stated that “Overall, Pre-Exec Bench [...] first-class objective” (l255-256), could you explain what you mean by this?


Q4: Section 6, Table 2 shows results for “Risk Cat. Acc.” - for the models **not** fine-tuned using Safiron variants, were the models provided with the risk category options and explanations? Similarly, for the “Expl. Corr.”, were models provided with explanation examples? I fear that the current setup overly biases results in favor of the Safiron method. Specifically, these models were trained on a distribution that is very similar to the one used at test time. It is thus reasonable to assume they’ll produce outputs that are more in line with the expected test test format(s) than untrained models. It would be useful to include a comparison of simply providing the untrained models with some in-context examples. Given the rapid development cycle of language models, in-context learning provides an operationally more favorable approach. Thus, for practical purposes, the authors would do well to compare Safiron to ICL alternatives.


Suggestions:
- The use of “trajectory” to refer to a plan is potentially confusing given pre-existing literature. Perhaps consider simply calling these plans instead?
- However, readability could be further improved by providing a running example. It is also quite dense in information, especially in the results section, which makes it difficult to properly parse the presented results.
soundness: 3
strengths: [**significance**] Safe agent deployment is a timely and important research direction. Guardrails focused on the planning stage present an operationally practical opportunity.


[**quality**]
- Using a diverse and broad model pool is essential for robust evaluation of benchmark initiatives. The paper includes eight leading open-source models from different developers to address this (l288-289).
- The paper does a great job at discussing and addressing the requirements for enhancing pro-active safe deployment. This reviewer especially appreciated the use of human annotators to improve the quality of benchmark annotations compared to “just” using language models. Small nit: “non-LLM [arbiters] guarantee reliability” (l297), there is no formal guarantee here. The cost discussion provided in Appendix I is also appreciated.
- Detailed and comprehensive discussion of subsets of the experimental results.

[**clarity**] Overall, the paper is well written. This reviewer does believe readability could be improved (see suggestions).


[**originality**] While there is existing work highlighting the promise of auditing reasoning / chain-of-thought leading up to actions [1], work focusing exclusively on a “planning” stage is a useful contribution.


[1] Korbac et al., Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, 2025
summary: The paper focuses on improving safe deployment of LLM-based agents by building tools designed for the “planning” phase of agentic execution. The authors argue that in contrast to post-execution guardrails, planning-based guardrails allow for pro-active safety evaluation. They identify a data, model, and evaluation gap as challenges to such a pro-active approach and propose solutions for each of these. To alleviate the data gap, the authors propose a multi-step synthetic data framework. For the model gap, the authors propose a guardian training method that combines an input-data adapter and a two-staged training strategy. Finally, to address the evaluation gap, the authors propose a new benchmark that uses their synthetic data engine and is enhanced by human annotators. The paper performs extensive empirical experiments on multiple models to show the efficacy of their approach.
weaknesses: [**significance**] While safety in planning could be seen as a _necessary_ condition to safe deployment, it does make some strong assumptions preventing it from being a _sufficient_ condition. Specifically, the authors do not address the assumptions of (i) plan-to-action faithfulness [ 2, 3], or (ii) the possibility of dynamic adjustments based on environmental feedback. Additionally, (iii) this work assumes plans are fully specified, e.g., “the full sequence of actions to be taken” (l95), does not account for high-level actions that can be interpreted in a myriad of ways. If the goal is to model and improve safe deployment of AI agents, these should at least be discussed.

[**quality**]
- In section 5, the authors write “Real-world agentic systems [...] but distributionally realistic” (l286-288). While this holds for zero-shot generations, it does not necessarily hold for intervened trajectories, e.g., using the approach described in Section 3. Additionally, the authors fail to discuss the coverage or “realism” of the user queries used to generate the above planned trajectories.
- In section 4, l195-196, the authors present a core component of their stated contribution: “a unified data adapter”. While this reviewer is sympathetic to the challenges of page limits, this is not an excuse to move the entire discussion of a core component to the Appendix. At the minimum, one would expect a high-level summary. For example, lots of space is used in lines 229-242 to (mostly) restate the basic GRPO algorithm.
- Given the strong reliance and focus on synthetic data throughout this work, it is surprising to this reviewer how little discussion focuses on key insights and evaluation axes from the synthetic data literature. The related work section (in the Appendix) similarly is limited. For example, any formal discussion on diversity is missing. With one of the core stated objectives being pro-active coverage, this is damning. I would suggest the work of [4] for a detailed discussion on this topic and [5] for a practical implementation.


[2] Yakovi and Goldber, Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? ACL 2020

[3] Dibjit, Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning, EMNLP 2024

[4] Havrilla et al., Surveying the effects of quality, diversity, and complexity in synthetic data from large language models, 2024

[5] Davidson et al., Orchestrating synthetic data with reasoning, ICLR 2025

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. RM Model Learning: Why is the input from DeepSeek-R1 directly regarded as ground truth? If so, why is a separate Reward Model (RM) trained? Moreover, why isn’t the evaluation from DeepSeek-R1 directly used as a filtering criterion in the AuraGen process?
2. It is observed that even the best-performing model within Safiron achieves relatively low accuracy in explanation correctness. Could a brief explanation be provided regarding the specific tasks in which the model underperforms most significantly?
3. In the GRPO framework, explanation correctness does not contribute to the final reward calculation. This implies that the model could potentially optimize rewards by generating only the initial tokens. Could a brief explanation be offered as to why the GRPO training approach can further enhance model performance beyond SFT?
4. Could you clarify the distinction between "multi-step corruption" and "bridge branch diversion"?
soundness: 3
strengths: 1. The contributions of the paper are clearly delineated, facilitating readers' understanding of the authors’ ideas. Additionally, the paper is logically organized.
2. The data generation process appears reasonable, and the model training pipeline, which includes supervised fine-tuning (SFT) and reinforcement learning (RL), also seems sound.
3. The paper clearly presents the training details of each module and provides comprehensive and straightforward experimental demonstrations.
summary: This paper centers on the safety of AI agents at the planning stage, prior to the execution of any actions—referred to as a proactive approach. The authors propose a data generation process named AuraGen, introduce a new model called Safiron, and develop a benchmark termed Pre-Exec Bench to address gaps in data, modeling, and evaluation. Their adapter-based Safiron pipeline consistently surpasses baseline performance on the Pre-Exec Bench.
weaknesses: 1. While the paper addresses data, model, and evaluation benchmarks comprehensively, the extensive experimental work may give the impression that the study lacks a specific focus or a distinct technical highlight.
2. Additional points are discussed in the Questions section.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: X
soundness: 3
strengths: This paper is exceptionally well written and easy to follow. The motivation is clear, and the proposed approach appears sensible. Theere are multiple contributions — (a) a data generation framework, (b) a method achieving state-of-the-art results, and (c) a benchmark — which will likely be appreciated by the community. Beyond these core contributions, the paper is also very thorough and provides many insightful results in the main body.
summary: This paper aims to improve the safety of LLM agents by intervening during the "planning stage", before the agent executes the plans. The manuscript highlights three gaps in the current literature: the data gap, a model gap, and an evaluation gap. The paper proposes a contribution for each.

To close the data gap, the authors propose AuraGen, a synthetic data generation framework that introduces unsafe perturbations to safe plans, thereby creating a training corpus for planning-stage LLM agent safety approaches. AuraGen features four injection modes: Single-step perturbation, Multi-step corruption, New-branch diversion, and Bridged-branch diversion, ensuring a diverse set of unsafe plans can be generated.

To close the model gap, the authors propose Safiron, which features a unified adapter module that normalizes inputs and a guardian model that outputs a classification (plan: safe vs. unsafe), a risk categorization, and a textual explanation. Safiron is trained by supervised fine-tuning on AuraGen data, followed by RL (GRPO) to further optimize output behavior.

Lastly, to close the evaluation gap, the authors propose Pre-Exec Bench, a benchmark for evaluating the safety of agentic plans before execution.

The results show that AuraGen generates a balanced dataset featuring different kinds of risks. The proposed model, Safiron, is shown to outperform almost all existing models on classification accuracy, harmful detection precision, risk categorization, and explanation correlation.

---
Note: This paper lies outside my primary area of expertise. While I can evaluate the clarity and quality of the writing and presentation, I do not feel fully confident assessing the soundness of the methodological approach or situating it within the broader literature on agentic LLM safety. I therefore assign a low confidence score to my review.
weaknesses: I appreciate the paper and strongly believe it should be published. I am somewhat unsure whether ICLR is the right venue, though, since the manuscript is highly applied and engineering-focused, and I see relatively little new theoretical or methodological innovation in agentic LLM safety (except perhaps the adapter/guardian decomposition). It is unclear how much of Safiron’s improvement is due to the adapter/guardian decomposition itself versus dataset or training choices; an ablation isolating the adapter’s contribution would be interesting.


Decision: Accept (Poster)