RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts

## Abstract

Abstract With the increasing adoption of large language models (LLMs), ensuring the safety of LLM systems has become a pressing concern.
External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs, but they are themselves fine-tuned or prompt-engineered LLMs that are vulnerable to data distribution shifts.
In this paper, taking Retrieval Augmentation Generation (RAG) as a case study, we investigated how robust LLM-based guardrails are against additional information embedded in the context.
Through a systematic evaluation of 3 Llama Guards and 2 GPT-oss models, we confirmed that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases , making them unreliable.
We separately analyzed the effect of each component in the augmented context: retrieved documents, user query, and LLM-generated response.
The two mitigation methods we tested only bring minor improvements.
These results expose a context-robustness gap in current guardrails and motivate training and evaluation protocols that are robust to retrieval and query composition.

## 1 Introduction

Large language models (LLMs) have rapidly become a central component of modern AI systems, powering applications from conversational assistants to code generation (Brown et al., 2020; Jiang et al., 2024). Their ability to generalize across domains and tasks has made them widely adopted in real-world deployments (Hadi et al., 2023). However, the same flexibility that enables their success also raises serious concerns about safety. LLMs are known to occasionally produce harmful, biased, or otherwise unsafe outputs, which poses significant risks when these models are used by millions of end users (Bai et al., 2022b; Ganguli et al., 2022; Gallegos et al., 2024; She et al., 2025; Guo et al., 2025).

To mitigate such risks, the research community and industry have invested heavily in methods for aligning LLMs with human safety preferences. Two main strategies have emerged: direct safety fine-tuning of base models (Ouyang et al., 2022) and the use of external guardrails (Rebedea et al., 2023).
Guardrail models serve as dedicated safety filters layered on top of generation, offering flexibility and modularity.
They can be updated independently of the base model, deployed as both input and output filters, and integrated into existing systems without retraining (Hurst et al., 2024).

Most guardrails are themselves LLM-based (e.g., Llama Guard (Inan et al., 2023)). Leveraging the expressive power of LLMs allows guardrails to handle nuanced, context-dependent safety decisions. However, this also exposes them to the same vulnerabilities as the models that they are meant to protect.
Prior work (Liu et al., 2024) has shown that LLMs are sensitive to the information contained in their context, and even benign additions can cause shifts in their behavior.
For example, a recent work (An et al., 2025) suggests that Retrieval-Augmented Generation (RAG) may also increase the risk of unsafe or malicious generations, since safety alignment methods such as RLHF are typically applied in non-RAG settings.
This raises an important but underexplored question: do LLM-based guardrails, when provided with richer contexts, alter their safety judgments?

To address this question, we take *RAG as a case study* to investigate the robustness of guardrails under such conditions,
since RAG is a widely adopted paradigm for improving the factuality and relevance of LLM outputs (Gao et al., 2023).
We consider two settings as shown in Fig.[1](https://arxiv.org/html/2510.05310v1#S1.F1): (1) normal RAG setting, where guardrails check user query or the query-response pair without exposure to retrieved documents,
(2) RAG-style query setting, where the query has been augmented with retrieved documents and would be passed in guardrails as a whole.

In this work, we conducted a systematic evaluation study of the robustness of LLM-based guardrails under RAG-style context. We introduce a novel implementation of robustness metric, *Flip Rate*. This metric measures the frequency with which guardrail judgments change between a vanilla and a RAG-augmented setting, and can be computed without ground-truth labeling. Using *Flip Rate*, we comprehensively evaluated three Llama Guard models and two GPT-oss models.
We posed the following three research questions:

RQ1: How does RAG-style context affect LLM-based guardrail judgment?
We assessed 5 popular LLM-based guardrails on over 6,000 harmful queries and the responses to them generated by 8 LLMs with non-RAG-style and RAG-style context separately.
We found that RAG-style context leads the guardrails to flip their judgments in both input guardrail and output guardrail settings. For example, well-aligned models like GPT-oss-20B give opposite judgments in around $15.0\%$ cases when used as an output guardrail.

RQ2: How does each component of RAG-style context affect the robustness of guardrails?
We isolated each component of RAG-style context and examined its individual effect on robustness. Our results show that (1) the relevance between retrieved documents and user query exacerbates the vulnerability, while the number of documents have minor effects, (2) guardrails flip safety judgments due to context shifts, regardless of whether the query is safe or unsafe,
(3) responses generated by different LLMs affects the guardrail differently.

RQ3: Can general LLM enhancements mitigate this safety concern? We explored two potential general-purpose mitigations: high-reasoning-effort mode, and RAG-style-context-aware prompting. Both improved robustness by lowering flip rate, but neither solved the issue completely, highlighting the need for future research on guardrail techniques specifically tailored to RAG-style contexts.

Figure: Figure 1: Illustration of guardrails giving different judgments to the same user query/response when receiving RAG-style query.
Refer to caption: x1.png

## 2 Related Works

Guardrail models.
LLMs have become increasingly powerful and widely deployed, but their open-ended generation abilities also introduce new safety challenges.
To mitigate these risks, guardrails, the external defense layers that monitor and control LLM interactions, have emerged as a crucial solution (Inan et al., 2023; Markov et al., 2023; Wang et al., 2024; Han et al., 2024; Kang & Li, 2025; Ghosh et al., 2024).
These mechanisms offer a distinct advantage over internal alignment techniques like RLHF (Ouyang et al., 2022; Bai et al., 2022a) by effectively filtering malicious inputs and outputs without compromising the core integrity of the base LLM (Dong et al., 2024).
Existing guardrail evaluations focus on plain inputs or output checks (Mazeika et al., 2024; Zou et al., 2023; Radharapu et al., 2023; Bhardwaj & Poria, 2023; Shaikh et al., 2023; Bhardwaj et al., 2024; Deng et al., 2023; Bassani & Sanchez, 2024; Lin et al., 2023), while our study targets a blind spot where the content under classification contains retrieved documents (RAG-style context).

Safety of RAG.
RAG introduces unique security challenges beyond vanilla LLM generation, as the integration of external knowledge corpora creates novel attack surfaces.
A growing body of work demonstrates that adversaries can poison indices, implant backdoors, or craft retrieval-optimized injections that steer models toward unsafe behavior (Xue et al., 2024; Zou et al., 2025; Cheng et al., 2024).
Liang et al. (2025) and Ni et al. (2025) conducted benchmarks and surveys to further catalog these threats and showed that vulnerabilities span indexing, retrieval, filtering, and generation stages.
Furthermore, beyond malicious content, the inherent properties of the corpus can lead to unwanted responses in other ways. For example, Wu et al. (2025) showed that demographic biases present in the retrieval data can persist or even be amplified by the RAG pipeline, while Zeng et al. (2024) found that RAG can leak proprietary retrieval database.

Our work examines how benign context shifts affect guardrails, diverging from the poisoned-corpus threat model. This parallels An et al. (2025), which provided the first comprehensive analysis of RAG’s impact on LLM safety. They found that incorporating retrieval often makes LLM less safe and alters its safety profile even if the RAG corpus is secured.
However, their evaluation focused exclusively on safety-aligned LLMs, without considering external guardrail models.
In contrast, our work centered on evaluating guardrail models in the RAG settings.

## 3 Problem Setup and Robustness Metric

In this section, we formalize the concepts underlying our study and introduce the robustness metric that we will use throughout the experiments.

### 3.1 Preliminaries

LLM.
Let $\mathcal{X}$ denote the space of user queries and $\mathcal{Y}$ the space of possible responses.
A large language model (LLM) defines a conditional distribution $M:\mathcal{X}\to\Delta(\mathcal{Y})$,
where $\Delta(\mathcal{Y})$ is the set of probability measures over $\mathcal{Y}$.
Given a query $x\in\mathcal{X}$, the model samples a response $y\sim M(x)$.

Safety labels.
We denote the binary safety label space by $\mathcal{C}=\{0,1\}$, where $1$ indicates *safe* and 0 indicates *unsafe*.

Guardrails.
An LLM-based *input guardrail* is a stochastic classifier $g_{\text{in}}:\mathcal{X}\to\Delta(\mathcal{C})$,
that predicts whether a user query is safe.
Similarly, an *output guardrail* is a stochastic classifier
$g_{\text{out}}:\mathcal{X}\times\mathcal{Y}\to\Delta(\mathcal{C})$,
that judges the safety of an LLM response in context of the original query.
For simplicity, we use the term *context* to denote the content a guardrail receives as input: either a query $x$ (for input guardrails) or a query–response pair $(x,y)$ (for output guardrails).

Retrieval-Augmented Generation (RAG).
Let $\mathcal{D}$ denote a document corpus. A retriever $R_{k}:\mathcal{X}\to\mathcal{D}^{k}$ selects $k$ relevant documents $[d_{1},\dots,d_{k}]$ for a query $x$.
A prompt augmentation function $T:\mathcal{X}\times\mathcal{D}^{k}\to\mathcal{X}$ combines the query and retrieved documents into an augmented input $x_{\text{RAG}}=T(x,R_{k}(x))$.
The LLM then produces RAG response

$$ $y_{\text{RAG}}\sim M(x_{\text{RAG}}).$ $$

### 3.2 Problem Definition: Guardrail Robustness under RAG context

Guardrails are intended to enforce safety policies by labeling user queries or model responses as safe or unsafe.
In this work, we focus on *general* safety evaluation and assess *general-purpose* guardrails that are meant to apply broadly across domains.
For such guardrails, safety judgments should not require access to specialized domain knowledge,
and should be driven by the harmfulness of user query or LLM responses, rather than by any retrieved documents.
The goal of this work is to answer how well existing guardrails provide consistent judgments across normal and RAG-style contexts.

##### Robustness requirement.

For clarity of exposition, we treat guardrails as deterministic classifiers that output binary safety labels, even if in practice they may be implemented using nondeterministic LLMs. Formally, an input guardrail $g_{\text{in}}$ is robust if it assigns the same label to a query $x$ and its RAG-augmented version $x_{\text{RAG}}$:

$$ $g_{\text{in}}(x)=g_{\text{in}}(x_{\text{RAG}}),\quad\forall x\in\mathcal{X}.$ $$

Similarly, an output guardrail $g_{\text{out}}$ is robust if it produces consistent judgments for $(x,y)$ and $(x_{\text{RAG}},y)$:

$$ $g_{\text{out}}(x,y)=g_{\text{out}}(x_{\text{RAG}},y),\quad\forall x\in\mathcal{X},\,y\in\mathcal{Y}.$ $$

##### Robustness metric: Flip Rate.

To quantify deviations from this ideal behavior, we define a *Flip* as any instance where the guardrail outputs inconsistent labels under a context and its RAG-augmented version.
The corresponding Flip sets are

$$ $\text{Input guardrail \emph{Flip} set: }\{x\mid g_{\text{in}}(x)\neq g_{\text{in}}(x_{\text{RAG}})\},$ $$

$$ $\text{Output guardrail \emph{Flip} set: }\{(x,y)\mid g_{\text{out}}(x,y)\neq g_{\text{out}}(x_{\text{RAG}},y)\}.$ $$

Given a dataset, the *Flip Rate (FR)* of an input/output guardrail is the proportion of instances in which a Flip occurs:

$$ $\text{FR}=\frac{|\text{Flip Set}|}{|\text{Dataset}|}.$ $$

Note that FR isn’t a measure of accuracy, since it does not measure against the ground-truth label. It only reflects the robustness of a guardrail to RAG-style perturbations, and
a lower FR is desirable as it indicates greater robustness to context shifts.
In the remainder of this paper, we use FR as the primary metric to evaluate and compare guardrails.

## 4 RQ1: How does RAG-style context affect guardrail judgment?

We first investigate whether RAG-style context perturbs the safety judgments of guardrail models.

### 4.1 Guardrail models

We evaluated 5 LLM-based guardrails: Llama Guard 2 (8B), Llama Guard 3 (8B), Llama Guard 4 (12B), GPT-oss-20B, and GPT-oss-120B.
Llama Guards (Inan et al., 2023) are fine-tuned Llama models for content safety classification, and they are designed to be used to classify content in both LLM inputs (query classification) and in LLM responses (response classification).

To ensure the diversity of guardrails, we also included
GPT-oss models (Agarwal et al., 2025).
Although not originally developed as guardrails,
Agarwal et al. (2025) report GPT-oss models perform comparably to OpenAI frontier commercial models in terms of safety.
We adapted them into guardrails using the same classification prompt template employed for Llama Guard (Appendix [C](https://arxiv.org/html/2510.05310v1#A3)), and configure their reasoning effort parameter to low.

### 4.2 Dataset

Our objective is to measure guardrail robustness by comparing judgments under two conditions: (i) original context and (ii) RAG-style context.
This requires queries $x$ and their RAG-style variants $x_{\text{RAG}}$ for input guardrails, and (query, response) pairs $(x,y)$ versus (RAG query, response) pairs $(x_{\text{RAG}},y)$ for output guardrails.
We constructed the dataset in three steps:

User Query.
We collected 6,795 harmful queries from seven benchmarks:
Harmbench (Mazeika et al., 2024), AdvBench (Zou et al., 2023), AART (Radharapu et al., 2023), HarmfulQA (Bhardwaj & Poria, 2023), DangerousQA (Shaikh et al., 2023), CategoricalHarmfulQA (Bhardwaj et al., 2024), SAP20 (Deng et al., 2023).
The statistics of these benchmarks are provided in Appendix [B](https://arxiv.org/html/2510.05310v1#A2).
Harmless queries will be discussed in Sec.[5.2](https://arxiv.org/html/2510.05310v1#S5.SS2).

RAG-style Query. Following An et al. (2025), we use BM25 as retriever and English Wikipedia as the corpus. Wikipedia articles are chunked into paragraphs, and the continuous paragraphs with at least 1,000 characters are treated as a document. The corpus contains 27,861,520 documents in total. For each user query,
the top-5 retrieved documents are concatenated with the query using a standard RAG template (Appendix [D](https://arxiv.org/html/2510.05310v1#A4)), producing 6,795 RAG-style queries.

RAG Response.
Since the comparison is made between normal RAG and RAG-style query settings, we synthesized RAG responses $y_{\text{RAG}}$ instead of standalone responses $y$.
Concretely, we collected responses to each RAG-style query from eight LLMs: Llama-3-8B/3.1-8B/3.3-70B-Instruct, Mistral-7B-v0.2/v0.3, Gemma-7B-it, GPT-oss-20B/120B, decoding with temperature 0.
For each LLM, the queries exceeding its context limit were excluded, yielding $54,179$ responses in total.

Figure: (a) Input guardrail FNR and FR
Refer to caption: x2.png

### 4.3 Evaluation setup

Input Guardrail Evaluation.
Instructed by the input guardrail prompting, each candidate guardrail produces safety judgments for both original query $x$ and its RAG-style variant $x_{\text{RAG}}$. We then measured *Flip Rate* defined in Sec.[3.2](https://arxiv.org/html/2510.05310v1#S3.SS2). Because all queries are harmful by construction, we additionally report the *False Negative Rate (FNR)*, i.e., the proportion of harmful queries misclassified as safe.

Output Guardrail Evaluation.
For output guardrails, each model is queried with $(x,y_{\text{RAG}})$ and $(x_{\text{RAG}},y_{\text{RAG}})$. We computed the *Flip Rate* across these paired contexts. Because responses are generated and are not annotated with ground-truth safety labels, FNR cannot be measured. Instead, we report the *Unsafe Rate* (UR), i.e., the proportion of outputs flagged unsafe, to provide complementary insight into each guardrail’s behavior. All guardrails are run with temperature 0.

### 4.4 Result

Input Guardrail Results.
Fig.[2(a)](https://arxiv.org/html/2510.05310v1#S4.F2.sf1) shows that RAG-style context significantly perturbs input guardrails. On average, RAG queries increase FNR by $7.3\%$, and induce flips in $3$–$24\%$ of cases, with an average FR of $10.9\%$.

Both GPT-oss guardrails achieve lower FNR and FR than the Llama Guard family, indicating stronger robustness.
Among Llama Guards, Llama Guard 3 is the most robust, while Llama Guard 4, the most recent release, has the highest FR ($24\%$), highlighting a nontrivial vulnerability to RAG-style perturbations.
We further observe that Normal FNR increases across successive Llama Guard versions.
Although accuracy is not the main focus of this work, this trend suggests the presence of safety blind spots, underscoring the need for comprehensive guardrail evaluation.

Output Guardrail Results.
Fig.[2(b)](https://arxiv.org/html/2510.05310v1#S4.F2.sf2) displays FR and UR for output guardrails. We find that judgments flip in $5$–$15\%$ of cases, averaging $8.4\%$.

Interestingly, the relative robustness ranking differs from the input setting. GPT-oss-20B, which is the second most robust input guardrail, becomes the weakest output guardrail.
In contrast, GPT-oss-120B remains consistently stronger than its smaller counterpart.
Within the Llama Guard family, the order of robustness reverses to Llama Guard 4 $>$ Llama Guard 2 $>$ Llama Guard 3. These discrepancies indicate that guardrail performance and robustness is highly task-dependent: a model effective as an input guardrail may behave unreliably as an output guardrail.

Conclusion.
Current guardrails are not robust to RAG-style context. Input guardrails flip in $10.9\%$ of cases on average, while output guardrails flip in $8.4\%$.

## 5 RQ2: How does each component of RAG-style context affect the robustness of guardrails?

In this section, we isolate each component of the RAG-style context and examine its individual effect on robustness.
A RAG-style context consists of (i) retrieved documents, (ii) the user query, and (iii) the LLM-generated responses (for output guardrails only). We discuss them one by one.

### 5.1 Factor 1: The Retrieved Documents

Retrieved documents are the most salient difference between RAG-style context and regular context of guardrail models.
We first study how the *number* and *relevance* of retrieved documents influences guardrail behavior.

#### 5.1.1 Number of Documents

Prior work has shown that long contexts can degrade LLM performance (Liu et al., 2024). We therefore ask: Does having more retrieved documents similarly destabilize guardrail judgments?

Evaluation Setup.
For input guardrails, we vary the number of retrieved documents $k$ while holding the retriever and corpus fixed. We measure flips between $g_{\text{in}}(x)$ and $g_{\text{in}}(x_{\text{RAG-}k})=g_{\text{in}}(T(x,R_{k}(x)))$ with different $k$.
For output guardrails, to ensure only one element changes at one time, we only alter $k$ documents observed by guardrail while keeping using LLM responses generated with top 5 documents in Sec. [4.2](https://arxiv.org/html/2510.05310v1#S4.SS2). Formally, we count flips between $g_{\text{out}}(x,y_{\text{RAG-}5})$ and $g_{\text{out}}(x_{\text{RAG-}k},y_{\text{RAG-}5})$ with different $k$. We experimented with $k=\{1,3,5,8,10\}$ in both evaluation. Llama Guard 2 results with $k\geq 8$ aren’t measured because most queries exceed its context window.

Input Guardrail Result.
In Fig. [3(b)](https://arxiv.org/html/2510.05310v1#S5.F3.sf2), we observe that FR tends to increase slightly with more documents.
However, the effect is modest: Llama Guard 4 is most sensitive to the number of documents, while Llama Guard 2 shows mild sensitivity, and the other three models are largely unaffected. In Fig. [3(a)](https://arxiv.org/html/2510.05310v1#S5.F3.sf1), FNRs exhibit a similar pattern.

Output Guardrail Result.
Output guardrails also show a slight increase in FR with larger $k$ (Fig. [3(c)](https://arxiv.org/html/2510.05310v1#S5.F3.sf3)).
The relative ranking of models remains consistent with Sec. [4](https://arxiv.org/html/2510.05310v1#S4) across $k$.
GPT-oss-20B is the least robust, with FR rising from $12.4\%$ ($k=1$) to $15.5\%$ ($k=8$), before dropping slightly at $k=10$.
Other models show only marginal increases (0.7–1.5%).

Conclusion.
Introducing even a single retrieved document in the context significantly alters guardrail judgments, but additional documents contribute little incremental harm.

Figure: (a) Input guardrail FNR
Refer to caption: x4.png

#### 5.1.2 Relevance of Documents

We next test whether the relevance of retrieved documents with query drives instability. While prior experiments retrieved relevant documents, here we drew random documents from the same corpus.

Evaluation Setup.
We constructed the random-RAG queries by sampling 5 documents uniformly from the same Wikipedia corpus instead of using BM25 retriever.
For each query, we generated five such Random-RAG contexts.
We measured flips for input guardrails by comparing $g_{\text{in}}(x)$ and $g_{\text{in}}(x_{\text{Random-}5})=g_{\text{in}}\big(T(x,R_{\text{Random-}5}(x))\big)$.
For output guardrails, we computed flips between $g_{\text{out}}(x,y_{\text{RAG-}5})$ and $g_{\text{out}}(x_{\text{Random-}5},y_{\text{RAG-}5})$.
Additional results of $y_{\text{Random-}k}$ are in Appendix [A](https://arxiv.org/html/2510.05310v1#A1).

Input Guardrail Result.
Fig. [4(b)](https://arxiv.org/html/2510.05310v1#S5.F4.sf2) compares FR under relevant- versus random-RAG contexts.
Random documents reduce FR for Llama Guard 3 and the two GPT-oss models, but increase FR for Llama Guard 4 and slightly for Llama Guard 2.
This suggests that document relevance interacts with input guardrails in a model-specific manner, but is not a primary driver of robustness measures.
The corresponding FNRs in relevant- and random-RAG cases follow a similar pattern (Fig.[4(a)](https://arxiv.org/html/2510.05310v1#S5.F4.sf1)), echoing the above finding.

Output Guardrail Result.
Here we evaluated the same responses and only changed the documents guardrails observed.
As discussed earlier that a guardrail can perform very differently in input and output guardrail usage settings, Fig.[4(c)](https://arxiv.org/html/2510.05310v1#S5.F4.sf3) shows different pattern from Fig.[4(b)](https://arxiv.org/html/2510.05310v1#S5.F4.sf2).
For all output guardrails models, except GPT-oss-120B, FR reduces $1.4\%$-$2.6\%$ when receiving random documents.
This suggests that most output guardrails are more able to ignore semantically irrelevant information, probably because relevant information distracts guardrails from the core query/response.

Conclusion.
Guardrail robustness can be affected by the relevance of retrieved documents in the context. Relevant documents in the context tend to lead to greater disturbance than random ones.

Figure: (a) Input guardrail FNR
Refer to caption: x7.png

### 5.2 Factor 2: The Safety of the Input Query

Input guardrails focus primarily on the user query, and must balance blocking unsafe queries against allowing safe ones for overall system utility. Since our previous experiments focused on unsafe queries, we now extend our evaluation to safe queries.

Evaluation Setup.
The setup is the same as Sec.[4](https://arxiv.org/html/2510.05310v1#S4) except for the dataset.
The previous datasets only contain harmful queries. So we collected 1,569 safe queries from 2 other datasets XStest (Röttger et al., 2024) and OR-Bench-Hard-1K  (Cui et al., 2025).
These two benchmarks are designed to measure the critical side effect of over-refusal, a phenomenon where models reject safe, innocuous prompts due to overly aggressive safety alignment.
Then, we constructed and leveraged the evaluation dataset for input and output guardrails following the same procedure as in Sec. [4.2](https://arxiv.org/html/2510.05310v1#S4.SS2).

Input Guardrail Result.
As shown in Fig.[5(b)](https://arxiv.org/html/2510.05310v1#S5.F5.sf2), input guardrails will flip their judgments in around 14% of cases on average (when $k=5$), comparable to their behaviors on unsafe queries.
Interestingly, Llama Guard 4, which was least robust on unsafe queries, produces the fewest flips when processing safe queries.
In contrast, GPT-oss-20B and 120B which make the least errors, now have the highest False Positive Rates (FPR) (Fig.[5(a)](https://arxiv.org/html/2510.05310v1#S5.F5.sf1)), indicating different guardrails may exhibit complementary strengths.

Output Guardrail Result.
Output guardrails are similarly disturbed in the context of RAG-style safe query (Fig.[5(c)](https://arxiv.org/html/2510.05310v1#S5.F5.sf3)). In some cases, FR on safe query context is even relatively higher than on unsafe ones.
For instance, Llama Guard 2’s FR becomes greater than Llama Guard 3.

Conclusion.
Guardrails are not robust to RAG-style context perturbation even on safe queries. The tested models appear to make different utility tradeoffs in blocking unsafe queries vs. allowing safe ones.

Figure: (a) Input guardrail FPR
Refer to caption: x10.png

### 5.3 Factor 3: The Generated Responses (Output guardrail only)

Output guardrails judge the safety of LLM responses.
One key factor affecting the responses is the selection of LLM.
In this section, we ask whether guardrails have different robustness against RAG-style context when dealing with responses generated by different LLMs.

Evaluation Setup.
In Sec.[4.2](https://arxiv.org/html/2510.05310v1#S4.SS2), we generated responses on 6,795 RAG-style harmful queries with 8 different LLMs.
Instead of analyzing all 8 LLMs’ responses as a whole, here we analyzed the output guardrail FRs for each LLM separately.
Formally, for each generation model $M$, we counted flips between $g_{\text{out}}(x,y_{\text{RAG};M})$ and $g_{\text{out}}(x_{\text{RAG}},y_{\text{RAG};M})$.

Results.
Fig.[6](https://arxiv.org/html/2510.05310v1#S5.F6) shows that a guardrail could have large variation in FR when processing responses to the same set of queries generated by different LLMs.
And the relative ranking of FR of responses generated by different LLMs differs depending on the guardrail.
For example, responses from gemma-7b-it yield the lowest FR for Llama Guard 2 and 4, but the highest FR for GPT-oss-20B.
These inconsistencies point to complex interaction effects between response characteristics and guardrail robustness.

Conclusion.
The robustness of a given output guardrail depends on the LLM generating the response. Understanding the underlying dynamics of this interaction remains an open problem.

Figure: Figure 6: RQ2 results of output guardrail FR on responses generated with different LLMs
Refer to caption: x13.png

## 6 RQ3: Can general LLM enhancements mitigate this?

Having established that guardrails are susceptible to perturbations from RAG-style context, we next investigate whether general LLM enhancement techniques can alleviate this issue.
Our goal here is not to provide a comprehensive solution, but rather to conduct a preliminary exploration of whether such general-purpose methods can reduce the observed vulnerability and to identify potential directions for future research.
We focus on two representative strategies: (1) employing models with high reasoning effort, and (2) modifying prompts to explicitly account for retrieved documents in the context.

### 6.1 Reasoning Model

Models with advanced reasoning ability often achieve higher performance on complex tasks.
Prior works (Kang & Li, 2025) have demonstrated that reasoning can enhance guardrails capability.
However, reasoning-intensive inference typically incurs substantial latency and token costs, which may hinder practical deployment as plug-in guardrails.
Here, we evaluate whether deeper reasoning meaningfully improves the robustness of guardrails.

Figure: Figure 7: RQ3 input guardrail FR
Refer to caption: x14.png

Evaluation Setup.
We configured both GPT-oss models to operate with high reasoning effort and repeated the experiments described in Section [4](https://arxiv.org/html/2510.05310v1#S4). However, GPT-oss-20B frequently failed to produce a final safety judgment in around 20% of cases due to token exhaustion during the reasoning stage. We therefore report results only for GPT-oss-120B.

Results.
As shown in Fig.[7](https://arxiv.org/html/2510.05310v1#S6.F7)&[8](https://arxiv.org/html/2510.05310v1#S6.F8), high-reasoning GPT-oss-120B demonstrates smaller FR than its low-reasoning counterpart in both input and output guardrail settings. The improvements, however, are limited to approximately 0.5% for input guardrail usage and 1.5% for output guardrail usage.

Conclusion.
While deeper reasoning provides a measurable benefit, the effect size is small and insufficient to fully mitigate the vulnerability. Moreover, the computational overhead makes this approach impractical for real-world guardrail deployment.

### 6.2 Dedicated Prompting

Figure: Figure 8: RQ3 output guardrail FR
Refer to caption: x15.png

Evaluation Setup.
We manually revise the original guardrail prompts to explicitly highlight the possibility of retrieved documents in the context and instruct the model to ignore such content, focusing only on the user query or LLM response. The full modified prompts are provided in Appendix [E](https://arxiv.org/html/2510.05310v1#A5).

Results.
Fig.[7](https://arxiv.org/html/2510.05310v1#S6.F7)&[8](https://arxiv.org/html/2510.05310v1#S6.F8) shows that enhanced prompting reduces FR in several cases. Specifically, the modified input guardrail prompt lowers FR across three models, while the modified output guardrail prompt lowers FR across all five models tested.

Conclusion.
Prompt engineering demonstrates effectiveness but with limited magnitude. While more carefully crafted prompts may yield stronger improvements, our results suggest that prompt modifications alone are insufficient to address the robustness issues of guardrails under RAG-style context. Taken together with the reasoning-model results, these exploratory experiments highlight that general LLM enhancements provide only incremental gains and fall far short of fully resolving the vulnerability, underscoring the need for future research on guardrail techniques specifically tailored to RAG-style contexts.

## 7 Discussion and Limitations

Our analysis relies primarily on the proposed *Flip Rate* metric, which captures changes in guardrail judgments between normal and RAG-style settings.
The computation of *Flip Rate* doesn’t require labeling, providing a scalable robustness metric to better understand guardrails’ properties in addition to accuracy metric which requires annotator and is subjective.
It could be easily integrated into any existing guardrail evaluation pipeline in production.
However, robustness alone does not fully characterize safety performance.
Future work could measure complementary metrics such as precision/recall against human labels or utility-safety trade-offs to provide a more holistic evaluation.

In Sec.[5.3](https://arxiv.org/html/2510.05310v1#S5.SS3) we found that guardrail robustness varies with the LLM that generates candidate responses. A deeper investigation into how response features shape guardrail judgments could inform more resilient guardrail designs. Our exploration of high-reasoning-effort model and prompt engineering (Sec.[6](https://arxiv.org/html/2510.05310v1#S6)), showed only limited improvements, suggesting that generic techniques provide only partial robustness.
Future work should explore training-time interventions, hybrid symbolic–neural guardrails, and uncertainty-aware methods that explicitly detect contextual shifts.

Our study covered five strong and popular guardrails, but this limited diversity leaves open the possibility that other guardrails are more robust.
While we followed An et al. (2025) in using BM25, we observed that relevant documents amplify instability (Sec.[5.1.2](https://arxiv.org/html/2510.05310v1#S5.SS1.SSS2)). Different retrievers may yield different outcomes, calling for systematic evaluation across retriever quality and adversarial strength.

## 8 Conclusion

In this work, we demonstrated that LLM-based guardrails are vulnerable to contextual perturbations such as retrieval augmentation, leading to nontrivial rates of judgment flips.
By systematically evaluating five guardrails across diverse settings, we revealed
that once one enriches the context to the guardrail, even with only one benign and irrelevant document, the quality of this safety mechanism drops significantly.
Our findings underscore an overlooked but critical limitation in current guardrails.
We hope this study motivates deeper inquiry into guardrail robustness and inspires the development of safer, more reliable alignment techniques for real-world LLM system deployments.

## 9 Ethics Statements

This work aims to investigate the robustness of LLM-based guardrails under RAG-style contexts. While our findings reveal vulnerabilities that could be exploited to bypass existing safety mechanisms, the primary goal of this research is to strengthen evaluation practices and inform the design of more resilient guardrail models. We believe that disclosing these limitations contributes to the responsible development and deployment of LLM systems, ultimately advancing their safe and trustworthy use in real-world applications.

## 10 Reproducibility statement

We provide a detailed description of our experimental setup in Sections [4](https://arxiv.org/html/2510.05310v1#S4), [5](https://arxiv.org/html/2510.05310v1#S5), and [6](https://arxiv.org/html/2510.05310v1#S6).
Additional implementation details, including dataset statistics, are presented in Appendix [B](https://arxiv.org/html/2510.05310v1#A2).
The prompts used for input and output guardrails are listed in Appendix [C](https://arxiv.org/html/2510.05310v1#A3), while the prompt for RAG response generation is provided in Appendix [D](https://arxiv.org/html/2510.05310v1#A4).
The enhanced guardrail prompts employed in Section [6.2](https://arxiv.org/html/2510.05310v1#S6.SS2) are included in Appendix [E](https://arxiv.org/html/2510.05310v1#A5).

## References

- Agarwal et al. (2025)
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al.
gpt-oss-120b & gpt-oss-20b model card.
*arXiv preprint arXiv:2508.10925*, 2025.
- An et al. (2025)
Bang An, Shiyue Zhang, and Mark Dredze.
RAG LLMs are not safer: A safety analysis of retrieval-augmented generation for large language models.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 5444–5474, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-189-6.
doi: 10.18653/v1/2025.naacl-long.281.
URL [https://aclanthology.org/2025.naacl-long.281/](https://aclanthology.org/2025.naacl-long.281/).
- Bai et al. (2022a)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning from human feedback.
*arXiv preprint arXiv:2204.05862*, 2022a.
- Bai et al. (2022b)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.
Constitutional ai: Harmlessness from ai feedback.
*arXiv preprint arXiv:2212.08073*, 2022b.
- Bassani & Sanchez (2024)
Elias Bassani and Ignacio Sanchez.
Guardbench: A large-scale benchmark for guardrail models.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 18393–18409, 2024.
- Bhardwaj & Poria (2023)
Rishabh Bhardwaj and Soujanya Poria.
Red-teaming large language models using chain of utterances for safety-alignment.
*arXiv preprint arXiv:2308.09662*, 2023.
- Bhardwaj et al. (2024)
Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria.
Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic.
*arXiv preprint arXiv:2402.11746*, 2024.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Cheng et al. (2024)
Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, and Gongshen Liu.
Trojanrag: Retrieval-augmented generation can be backdoor driver in large language models.
*arXiv preprint arXiv:2405.13401*, 2024.
- Cui et al. (2025)
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh.
OR-bench: An over-refusal benchmark for large language models.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=CdFnEu0JZV](https://openreview.net/forum?id=CdFnEu0JZV).
- Deng et al. (2023)
Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He.
Attack prompt generation for red teaming and defending large language models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 2176–2189, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.143.
URL [https://aclanthology.org/2023.findings-emnlp.143/](https://aclanthology.org/2023.findings-emnlp.143/).
- Dong et al. (2024)
Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, et al.
Safeguarding large language models: A survey.
*arXiv preprint arXiv:2406.02622*, 2024.
- Gallegos et al. (2024)
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed.
Bias and fairness in large language models: A survey.
*Computational Linguistics*, 50(3):1097–1179, 2024.
- Ganguli et al. (2022)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.
Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.
*arXiv preprint arXiv:2209.07858*, 2022.
- Gao et al. (2023)
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang.
Retrieval-augmented generation for large language models: A survey.
*arXiv preprint arXiv:2312.10997*, 2(1), 2023.
- Ghosh et al. (2024)
Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien.
Aegis: Online adaptive ai content safety moderation with ensemble of llm experts.
*arXiv preprint arXiv:2404.05993*, 2024.
- Guo et al. (2025)
Ruohao Guo, Wei Xu, and Alan Ritter.
How to protect yourself from 5g radiation? investigating llm responses to implicit misinformation.
*arXiv preprint arXiv:2503.09598*, 2025.
- Hadi et al. (2023)
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al.
Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects.
*Authorea preprints*, 1(3):1–26, 2023.
- Han et al. (2024)
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri.
Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 8093–8131. Curran Associates, Inc., 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/0f69b4b96a46f284b726fbd70f74fb3b-Paper-Datasets_and_Benchmarks_Track.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/0f69b4b96a46f284b726fbd70f74fb3b-Paper-Datasets_and_Benchmarks_Track.pdf).
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Inan et al. (2023)
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.
Llama guard: Llm-based input-output safeguard for human-ai conversations.
*arXiv preprint arXiv:2312.06674*, 2023.
- Jiang et al. (2024)
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim.
A survey on large language models for code generation.
*arXiv preprint arXiv:2406.00515*, 2024.
- Kang & Li (2025)
Mintong Kang and Bo Li.
$r^2$-guard: Robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=CkgKSqZbuC](https://openreview.net/forum?id=CkgKSqZbuC).
- Liang et al. (2025)
Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Zhaoxin Fan, Bo Tang, Jihao Zhao, Jiawei Yang, Shichao Song, and Mengwei Wang.
SafeRAG: Benchmarking security in retrieval-augmented generation of large language model.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 4609–4631, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.230.
URL [https://aclanthology.org/2025.acl-long.230/](https://aclanthology.org/2025.acl-long.230/).
- Lin et al. (2023)
Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang.
Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation.
*arXiv preprint arXiv:2310.17389*, 2023.
- Liu et al. (2024)
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
Lost in the middle: How language models use long contexts.
*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024.
doi: 10.1162/tacl˙a˙00638.
URL [https://aclanthology.org/2024.tacl-1.9/](https://aclanthology.org/2024.tacl-1.9/).
- Lù (2024)
Xing Han Lù.
Bm25s: Orders of magnitude faster lexical search via eager sparse scoring, 2024.
URL [https://arxiv.org/abs/2407.03618](https://arxiv.org/abs/2407.03618).
- Markov et al. (2023)
Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng.
A holistic approach to undesired content detection in the real world.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 37, pp. 15009–15018, 2023.
- Mazeika et al. (2024)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks.
Harmbench: a standardized evaluation framework for automated red teaming and robust refusal.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- Meta (2024)
Meta.
Model cards and prompt formats: Llama guard 3 – prompt format, 2024.
URL [https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-3/#prompt-format-](https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-3/#prompt-format-).
- Meta (2025)
Meta.
Model cards and prompt formats: Llama guard 4 – prompt format, 2025.
URL [https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/#prompt-format-](https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/#prompt-format-).
- Ni et al. (2025)
Bo Ni, Zheyuan Liu, Leyao Wang, Yongjia Lei, Yuying Zhao, Xueqi Cheng, Qingkai Zeng, Luna Dong, Yinglong Xia, Krishnaram Kenthapadi, et al.
Towards trustworthy retrieval augmented generation for large language models: A survey.
*arXiv preprint arXiv:2502.06872*, 2025.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Radharapu et al. (2023)
Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti.
AART: AI-assisted red-teaming with diverse data generation for new LLM-powered applications.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track*, pp. 380–395, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-industry.37.
URL [https://aclanthology.org/2023.emnlp-industry.37/](https://aclanthology.org/2023.emnlp-industry.37/).
- Rebedea et al. (2023)
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen.
Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails.
*arXiv preprint arXiv:2310.10501*, 2023.
- Röttger et al. (2024)
Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy.
XSTest: A test suite for identifying exaggerated safety behaviours in large language models.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 5377–5400, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-long.301.
URL [https://aclanthology.org/2024.naacl-long.301/](https://aclanthology.org/2024.naacl-long.301/).
- Shaikh et al. (2023)
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang.
On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 4454–4470, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.244.
URL [https://aclanthology.org/2023.acl-long.244/](https://aclanthology.org/2023.acl-long.244/).
- She et al. (2025)
Yining She, Sumon Biswas, Christian Kästner, and Eunsuk Kang.
Fairsense: Long-term fairness analysis of ml-enabled systems.
In *2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)*. IEEE Computer Society, 2025.
- Wang et al. (2024)
Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin.
Do-not-answer: Evaluating safeguards in LLMs.
In Yvette Graham and Matthew Purver (eds.), *Findings of the Association for Computational Linguistics: EACL 2024*, pp. 896–911, St. Julian’s, Malta, March 2024. Association for Computational Linguistics.
URL [https://aclanthology.org/2024.findings-eacl.61/](https://aclanthology.org/2024.findings-eacl.61/).
- Wu et al. (2025)
Xuyang Wu, Shuowei Li, Hsin-Tai Wu, Zhiqiang Tao, and Yi Fang.
Does RAG introduce unfairness in LLMs? evaluating fairness in retrieval-augmented generation systems.
In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), *Proceedings of the 31st International Conference on Computational Linguistics*, pp. 10021–10036, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
URL [https://aclanthology.org/2025.coling-main.669/](https://aclanthology.org/2025.coling-main.669/).
- Xue et al. (2024)
Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou.
Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models.
*arXiv preprint arXiv:2406.00083*, 2024.
- Zeng et al. (2024)
Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang.
The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG).
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Findings of the Association for Computational Linguistics: ACL 2024*, pp. 4505–4524, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-acl.267.
URL [https://aclanthology.org/2024.findings-acl.267/](https://aclanthology.org/2024.findings-acl.267/).
- Zou et al. (2023)
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models.
*arXiv preprint arXiv:2307.15043*, 2023.
- Zou et al. (2025)
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia.
$\{$PoisonedRAG$\}$: Knowledge corruption attacks to $\{$Retrieval-Augmented$\}$ generation of large language models.
In *34th USENIX Security Symposium (USENIX Security 25)*, pp. 3827–3844, 2025.

## Appendix A Additional Experiment Results

### A.1 Additional experiments of RQ2 regarding relevance of documents

In addition to the experiments in Sec.[5.1.2](https://arxiv.org/html/2510.05310v1#S5.SS1.SSS2), we also generated LLM responses $y_{\text{Random-}k}$ using $x_{\text{Random-}k}$ and count flip between $g_{\text{out}}(x,y_{\text{Random-}k})$ and $g_{\text{out}}(x_{\text{Random-}5},y_{\text{Random-}k})$ where both the input and response differ.

Results of Output Guardrail w/ Random-RAG responses.
Fig.[9](https://arxiv.org/html/2510.05310v1#A1.F9) shows how the responses generated with documents of different relevance change guardrail behaviors.
We found that *FR* of $g_{\text{out}}(x_{\text{Random-}5},y_{\text{Random-}k})$ is consistently lower than $g_{\text{out}}(x_{\text{Random-}5},y_{\text{RAG-}k})$ across all guardrails.
This means that the safety of the Random-RAG responses are more obvious than normal RAG responses, making it easier for guardrails to assign consistent labels even in RAG-style context.
The result is also echoing our finding about the influence of responses in Sec.[5.3](https://arxiv.org/html/2510.05310v1#S5.SS3).

Figure: Figure 9: Evaluation results of RQ2 regarding relevance of documents. Output guardrail *FR*s of checking $y_{\text{Random-}5}$.
Refer to caption: x16.png

Figure: Figure 10: RQ3 input guardrail FNRs
Refer to caption: x17.png

## Appendix B Additional Experiment Setup Details

##### Retriever and Corpus.

In this paper, we use BM25 as the retriever, implemented via open-sourced python library [bm25s](https://github.com/xhluca/bm25s) (Lù, 2024).
The corpus is a Wikipedia dump of May 1st 2025 downloaded from [https://dumps.wikimedia.org/enwiki/](https://dumps.wikimedia.org/enwiki/).
Following An et al. (2025), the corpus is created by splitting the text at paragraph breaks (\n\n) and ensuring each chunk has at least 1,000 characters. If a chunk is shorter, it is concatenated with the next one until it exceeds 1,000 characters. In this paper, each chunk is treated as a document. In total, our corpus contains 27,861,520 chunks. Each chunk is treated as a separate document.

##### Datasets.

We collected harmful queries from 7 datasets in Table [1](https://arxiv.org/html/2510.05310v1#A2.T1). We gather all data from these datasets except for Harmbench. Following An et al. (2025), we used prompts whose semantic category is Misinformation and Disinformation or functional category is standard, which leads to 231 prompts.
So after gathering all 7 datasets, we have 6,870 prompts. After filtering out the repetitive ones, we finally got 6,795 prompts.
Furthermore, in Sec.[5.2](https://arxiv.org/html/2510.05310v1#S5.SS2), we collected 1,569 harmless queries from 2 datasets in Table [2](https://arxiv.org/html/2510.05310v1#A2.T2).

**Table 1: Statistics of harmful query datasets.**
| Dataset | Total Number of Data | Reference |
| --- | --- | --- |
| Harmbench | 400 | Mazeika et al. (2024) |
| AdvBench | 520 | Zou et al. (2023) |
| AART | 3,269 | Radharapu et al. (2023) |
| HarmfulQA | 1,960 | Bhardwaj & Poria (2023) |
| DangerousQA | 200 | Shaikh et al. (2023) |
| CategoricalHarmfulQA | 550 | Bhardwaj et al. (2024) |
| SAP20 | 140 | Deng et al. (2023) |

**Table 2: Statistics of harmless query datasets.**
| Dataset | Total Number of Data | Reference |
| --- | --- | --- |
| XStest | 250 | Röttger et al. (2024) |
| OR-Bench-Hard-1K | 1,319 | Cui et al. (2025) |

##### Guardrail outputs.

We used the same guardrail prompt as Llama Guard 4 Meta (2025). This prompt would output the safety judgment and safety violated categories. In our experiments, we only use the safety judgment.

## Appendix C Guardrail Prompt Template

All guardrails use the same prompts as shown below, except for Llama Guard 3 and Llama Guard 4, which require special start and end tokens as specified in their model cards (Meta, 2024; 2025). Their special tokens are provided at the end of this section.

## Appendix D LLM Generation Prompt Template

## Appendix E Enhanced Guardrail Prompt Template

## Appendix D LLM Generation Prompt Template

## Appendix E Enhanced Guardrail Prompt Template

## Appendix E Enhanced Guardrail Prompt Template

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: See weaknesses above.
soundness: 3
strengths: The paper formalizes guardrail robustness under RAG by defining consistency requirements for input/output guardrails and introducing a label-free Flip Rate (FR) that quantifies judgment changes; it’s simple, scalable, and clearly distinguished from accuracy. It covers 5 guardrails (3 Llama Guard versions and 2 GPT-oss variants), evaluates both input and output settings, and spans 6,795 harmful queries plus additional safe queries—providing unusually comprehensive coverage for this topic. Key findings—e.g., input FR ≈10.9% and output FR ≈8.4%—are concrete and actionable; the work also shows task-dependent robustness differences across guardrails.
summary: This paper investigates the robustness of safety guardrails when integrated with Retrieval-Augmented Generation (RAG) systems. The authors argue that retrieved documents—while improving factuality—can inadvertently alter guardrail decisions, compromising safety consistency. To quantify this, they propose a label-free metric called Flip Rate (FR), measuring how often a guardrail’s safety judgment changes under varying retrieval conditions. The study evaluates five popular guardrail models (three Llama Guard variants and two GPT-based ones) across both input-level and output-level settings, using thousands of harmful and safe queries retrieved via BM25 from Wikipedia. Extensive experiments reveal that guardrails exhibit notable instability—about 10.9% FR for inputs and 8.4% for outputs—and that factors such as the number and relevance of retrieved documents, query safety level, and the generator model significantly influence robustness. The authors also test lightweight mitigation strategies, including “high reasoning” and “RAG-aware” prompting, which yield only marginal improvements.
weaknesses: While the proposed Flip Rate (FR) is an elegant, label-free measure of guardrail robustness, it inherently cannot distinguish between correct and incorrect judgment changes. A flip may indicate either an improvement or a degradation in safety performance, but the metric treats both equally. This limitation weakens the interpretability of FR as a genuine proxy for “safety robustness,” and suggests that additional labeled or human-audited analyses would be valuable for validation.

Moreover, all RAG retrieval experiments rely solely on a BM25 retriever over English Wikipedia. This setup overlooks stronger and more representative retrievers (e.g., dense, hybrid, or cross-encoder–based retrieval). Since the retrieved context strongly drives guardrail flipping behavior, it remains unclear whether the reported instability generalizes to modern retrieval architectures. Including at least one dense retrieval baseline (such as Contriever or DPR) would substantially strengthen the empirical conclusions.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: -	Note that the reasoning model only displays a less than 5% flipped rate. Then, how about other metrics like F1 score and FNR? Does the reasoning model improve the classification performance? Is it the case that the reasoning model corrects the wrong prediction so the flipped prediction happened?
soundness: 2
strengths: -	The paper identifies and formalizes an interesting failure mode of guardrail models, bridging research on RAG safety and LLM moderation.
-	The evaluation covers diverse guardrails, realistic datasets, and controlled RAG setups. The decomposition across context factors is systematic and insightful.
-	The paper is well structured with clear research questions, consistent definitions, and informative figures.
summary: This paper investigates the robustness of LLM-based guardrails under retrieval-augmented generation contexts. The authors introduce the flip rate to measure how often a guardrail’s safety judgment changes when benign retrieved documents are added. Using five guardrail models across both harmful and safe queries, the study finds that judgments flip in both input-guardrail and output-guardrail cases. Additional analyses isolate the impact of document number, relevance, query safety, and generation model, and two mitigation attempts, high-reasoning mode and RAG-aware prompting, provide only marginal improvements. The work exposes a context-robustness gap in current guardrail systems and calls for training and evaluation frameworks that are robust to retrieval composition.
weaknesses: -	This paper presents interesting findings but provides little in-depth analysis on why these flipped prediction happens. Existing work [1] also investigates the robustness/reliability of guardrail models from the aspect of prediction uncertainty, which I believe is related to the flipped prediction. Those flipped predictions may display high uncertainty and therefore could be easily manipulated with retrieved documents. I think this could be easily verified and at least discussed in the paper. 
-	In addition, the RAG-style queries/responses with the larger context might also explain the weak robustness of the safety prediction, which could act like out-of-domain samples and may not appear in the training data of existing guardrail models. Even though the training data of Llama-Guard is not open-source, existing advanced guardrail models like WildGuard [2] have open-source training data, making it easy to verify them by just comparing the sequence length between the training data and RAG-style queries.         
-	Another confounder could be the safety judgment of guardrail models on the documents themselves. These documents/corpora are assumed to be safe with the prior knowledge. However, guardrail models may classify some documents as unsafe due to certain unsafe words/phrases in the doc, or failed prediction of guardrail models. In this case, the guardrail models may make opposite predictions once these documents are sampled. I would recommend an ablation study for this factor.

[1] On calibration of LLM-based guard models for reliable content moderation. ICLR 2025.

[2] Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. NeurIPS  2024.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. See weakness

2. Could you explain more about Flip Rate? While it measures inconsistency, how can we be sure a "flip" represents a degradation in safety rather than, in some cases, a context-aided correction? 

3. This paper focuses on benign documents. How do these guardrails would perform if the retrieved documents themselves contained adversarial content?
soundness: 3
strengths: - The paper studies an important and less explored issue in LLM safety — how stable external guardrails.

- Evaluation are detailed and comprehensive, considering many aspects.

- It presents a simple and useful metric called FR to measure how often guardrail decisions change when the context shifts. 

- Experiments show that even normal RAG context can strongly influence guardrail decisions. This reveals a real and practical weakness in current systems.
summary: This paper study how well external LLM-based guardrail models stay consistent when given extra context from RAG. The main question is whether adding retrieved documents changes the guardrail’s safety decisions compared to judging the query or response alone.
The authors propose a metric called Flip Rate to measure how often these safety judgments change. They test both input guardrails and output guardrails using harmful and safe queries. Results show that context causes decision flips in about 11% for input guardrails and 8% for output guardrails. The paper also studies how document relevance and count affect the results and tests simple fixes like prompting or using high-reasoning modes.
weaknesses: - The paper only test BM25 retrieval on Wikipedia. The conclusion may change for dense retrievers or new RAG methods.

- FR only indicates changes in judgment. Change can be good or bad. But this paper regard flips as safety failures sometimes, but some flips may fix mistakes and is useful. 

- The prompting and high-reasoning mode fixes are simple and give little improvement. 

- The paper only uses normal retrieved documents. It does not test when the documents themselves include adversarial cues or misinformation (knowledge poisoning).

- Missing references

[1] TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation

[2] FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA

[3] Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning

[4] A Survey on LLM-as-a-Judge

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you verify on the dataset that such flips stem from guardrails’ correct judgment (not poor robustness)? Reporting its proportion would further justify the experimental setup’s rationality.
2. What is a more robust way to use guardrails models between using them with RAG context or without?
soundness: 3
strengths: 1. This work focuses on LLM-based guardrail robustness under RAG contexts. It is well-motivated.
2. The proposed Flip Rate—quantifying guardrail judgment flips between vanilla and RAG-augmented settings without ground-truth—is inspiring and provides a scalable tool for evaluating context robustness.
3. The paper’s figures are clear, effectively visualizing key findings and enhancing interpretability.
summary: This paper proposes an investigation into the robustness of LLM-based guardrails under RAG-style contexts, aiming at addressing the vulnerability of external LLM-based guardrails to data distribution shifts. The study introduces a case study approach using RAG, a systematic evaluation of 3 Llama Guard models and 2 GPT-oss models, and a separate analysis of the effects of each component in the RAG-augmented context. Experimental results confirm that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases respectively.
weaknesses: 1. In Section 5.2, the rationale for claiming content safety is consistent between vanilla and RAG-augmented settings is unclear. For example, inputs/outputs are safe in the vanilla setting; if RAG-augmented version query violates guardrails’ safety principles, this flip reflects guardrails’ correct judgment rather than poor robustness.


Decision: Reject