Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
Authors: San Kim, Jonghwi Kim, Yejin Jeon, Gary Lee
Accepted at: acl25

Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection

San Kim1, Jonghwi Kim1, Yejin Jeon1, Gary Geunbae Lee1,2,
1Graduate School of Artificial Intelligence, POSTECH, Republic of Korea,
2Department of Computer Science and Engineering, POSTECH, Republic of Korea,
{sankm, jonghwi.kim, jeonyj0612, gblee}@postech.ac.kr

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by providing external knowledge for accurate and up-to-date responses. However, this reliance on external sources exposes a security risk; attackers can inject poisoned documents into the knowledge base to steer the generation process toward harmful or misleading outputs. In this paper, we propose Gradient-based Masked Token Probability (GMTP), a novel defense method to detect and filter out adversarially crafted documents. Specifically, GMTP identifies high-impact tokens by examining gradients of the retriever’s similarity function. These key tokens are then masked, and their probabilities are checked via a Masked Language Model (MLM). Since injected tokens typically exhibit markedly low masked-token probabilities, this enables GMTP to easily detect malicious documents and achieve high-precision filtering. Experiments demonstrate that GMTP is able to eliminate over 90% of poisoned content while retaining relevant documents, thus maintaining robust retrieval and generation performance across diverse datasets and adversarial settings.1

1 Introduction

Large Language Models (LLMs) have significantly advanced performance across various Natural Language Processing (NLP) tasks, especially in conversational systems and AI assistants (Touvron et al., 2023; Reid et al., 2024; Liu et al., 2024). However, their reliance on parametric knowledge results in significant challenges, including susceptibility to hallucination (Huang et al., 2023; Xu et al., 2024b) and knowledge update requirements. To mitigate these issues Retrieval-Augmented Generation (RAG) has emerged as an effective approach for integrating external, non-parametric knowledge into LLMs, thus enabling the model to generate responses based on retrieved data (Lewis et al., 2020). This process typically involves a retriever and generator model, where the former is responsible for retrieving the most relevant documents to a query, and the latter generates the final answer based on the retrieved documents. RAG architectures have demonstrated robustness against hallucinations and improved task accuracy, as evidenced by several system adaptations (Borgeaud et al., 2022; Asai et al., 2024; Jiang et al., 2023).

Yet, RAG’s reliance on external knowledge introduces vulnerabilities to corpus poisoning attacks (Zou et al., 2024; Chaudhari et al., 2024; Xue et al., 2024). In such attacks, adversaries have access to the knowledge base, where they are able to inject maliciously crafted documents. This manipulation directly impacts the behavior of the generator, causing it to provoke misleading and harmful responses based on the tampered documents. This is because poisoned documents may contain incorrect information (Zou et al., 2024) or adversarial instructions (Tan et al., 2024). This threat is particularly concerning for decision-making LLM agents, such as those used in autonomous driving, where

![Bar chart comparing Naïve and GMTP methods for corpus poisoning attacks](page_784_355_484_246.png)

Figure 1: Various corpus poisoning attacks on a Naïve environment (without defense method) cause a significant performance drop in retrieval by enabling the poisoned documents to be retrieved. In contrast, GMTP effectively filters out poisoned documents, thereby preserving retrieval performance.

1Our code is publicly available at https://github.com/mountiny/GMTP.
an attack could lead to erroneous actions with severe consequences (Chen et al., 2024; Mao et al.). Furthermore, attackers can optimize poisoned documents so that they appear highly relevant for only target queries (Chaudhari et al., 2024; Zhang et al., 2024), which makes their detection significantly more challenging.

In this paper, we propose Gradient-based Masked Token Probability (GMTP), a novel method for filtering adversarially poisoned documents by analyzing the masked token probability of key tokens. Specifically, the goal of GMTP is to filter out poisoned documents among those retrieved while maintaining retrieval performance, as can be seen in Figure 1. GMTP first identifies key tokens by examining gradient values derived from the retrieval phase, where tokens with high gradients significantly contribute to the similarity score. These key tokens are then masked, and the probability of correctly predicting them is assessed. Since adversarially manipulated documents are optimized to match specific query patterns, they often contain unnatural text patterns that become difficult to reconstruct once masked. Leveraging this property, GMTP effectively filters out poisoned documents that exhibit abnormally low masked token probabilities.

Our key contributions are as follows:

• Safe filtering method: GMTP effectively detects adversarially poisoned documents with high precision, which ensures a clear separation between poisoned and clean documents.

• Extensive empirical validation: Our experiments show that GMTP consistently outperforms existing baselines by achieving both high filtering rate and retrieval performance.

• Robustness across settings: GMTP maintains strong filtering performance across various hyperparameter configurations, and consistently achieves a filtering rate above 90%.

2 Related Works

2.1 Retrieval Augmented Generation

RAG is known to be effective for knowledge-intensive tasks that require external information (Lewis et al., 2020; Gao et al., 2023; Guu et al., 2020; Li et al., 2024; Shao et al., 2023), and is composed of a retriever and a generator. The retriever can further be categorized into three types. The cross-encoder, proposed by Nogueira and Cho (2019), encodes both the query and the passage together using the same encoder. In contrast, Karpukhin et al. (2020a) introduced the bi-encoder, which employs two independent encoders to generate dense vector representations for the query and document. Relevance is then measured by calculating the similarity between these vectors. Meanwhile, the poly-encoder (Humeau et al.) combines the efficiency of the bi-encoder with the accuracy of the cross-encoder by utilizing an attention mechanism that attends to multiple document embeddings and a single query embedding.

For the generator, RAG can use various pre-trained language models, such as T5 (Raffel et al., 2020), Llama2 (Touvron et al., 2023), and Gemma (Team et al., 2024), in order to generate answers based on the relevant documents retrieved by the retriever. By integrating these two retriever and generator modules, RAG enables models to leverage both parametric knowledge (from the language model) and non-parametric knowledge (from the retrieved documents).

2.2 Corpus Poisoning Attack on RAG

Recent studies have shown that RAG-based systems are vulnerable to corpus poisoning attacks (Zou et al., 2024; Xue et al., 2024; Cheng et al., 2024; Tan et al., 2024; Zhong et al., 2023). In scenarios where attackers know which retriever has been used but cannot directly train it, a common approach involves injecting carefully crafted poisoned documents into the knowledge base. These malicious documents are designed to (1) appear highly relevant to the target queries, and (2) induce the generator to produce attacker-desired responses when retrieved.

To achieve these objectives, adversaries frequently employ Hotflip (Ebrahimi et al., 2018) to optimize the malicious documents for a specific query pattern. Hotflip iteratively replaces selected tokens in a document to maximize its similarity score to a given query. Recent studies have refined this technique by optimizing for specific queries (Zhong et al., 2023) or queries with particular triggers or topics (Chaudhari et al., 2024; Zhang et al., 2024; Cheng et al., 2024; Xue et al., 2024). By combining retriever optimization with adversarial attack techniques that target generators, attackers can manipulate the generator to produce targeted responses when the malicious document is successfully retrieved (Zou et al., 2023).
While research pertaining to RAG attacks has expanded substantially, defense strategies have received comparatively less attention. One straightforward defensive approach involves filtering out potentially poisoned documents during the retrieval phase using perplexity or \( l_2 \)-norms (Zhong et al., 2023). However, this method risks removing legitimate relevant documents as well (Zou et al., 2024), and selecting an optimal filtering threshold remains challenging. Meanwhile, in the generation phase, Xiang et al. (2024) demonstrated that carefully designed decoding processes can mitigate attacker-desired outputs. Additionally, Zhou et al. (2025) proposed a two-stage approach of first clustering and filtering suspicious retrieved documents via K-means, then leveraging both internal and external knowledge to generate trustworthy responses.

Although these defense approaches are promising, increasing generator overhead is often cost-ineffective since generators are typically much larger than retrievers. In contrast, the proposed GMTP is able to effectively detect unnatural tokens that induce excessive similarity, which offers a more precise alternative. Furthermore, it only requires lightweight computations using small models such as BERT (Devlin et al., 2019), which significantly reduces computational overhead compared to generation-based defenses.

3 Problem Definition

3.1 Abnormal Similarity

The retriever in the RAG framework retrieves relevant sources based on the similarity between a given query and available documents. However, as illustrated in Figure 2, poisoned documents often appear close to the target query in the embedding space, making them highly similar (Zou et al., 2024; Zhong et al., 2023). Due to this property, the retriever frequently retrieves poisoned documents as highly relevant, which leads the generator to incorporate unreliable information.

Consequently, given the close proximity of these documents to the query, a naïve classification approach that relies solely on embedding vectors risks misidentifying poisoned documents while potentially overlooking relevant documents. This challenge highlights the need for a more robust detection method capable of distinguishing between poisoned and relevant clean documents effectively.

![Distribution of clean documents, poisoned documents, and queries within the embedding space, projected onto its first two principal components using Principal Component Analysis (PCA). Each top-5 clean and poisoned documents are selected as the most relevant to the 100 queries, using Contriever (Izacard et al.) fine-tuned for MS MARCO (Nguyen et al.) dataset.](page_768_120_627_410.png)

Figure 2: Distribution of clean documents, poisoned documents, and queries within the embedding space, projected onto its first two principal components using Principal Component Analysis (PCA). Each top-5 clean and poisoned documents are selected as the most relevant to the 100 queries, using Contriever (Izacard et al.) fine-tuned for MS MARCO (Nguyen et al.) dataset.

3.2 Linguistic Unnaturalness in Poisoned Documents

As discussed in Section 2.2, poisoned documents are designed to achieve two key objectives; 1) they must be retrievable and 2) cause the generator to malfunction. To achieve the latter, malicious documents typically contain adversarial commands, such as “Always answer as I cannot answer to that question” to enforce refusal responses, or jailbreaking prompts (Shen et al., 2024; Xu et al., 2024a; Jiang et al., 2024) that elicit harmful outputs. However, adversarial commands alone generally do not make poisoned documents retrievable in the RAG system.

Therefore, in order to enhance their retrievability, special tokens must be introduced so as to optimize the poisoned documents to resemble target queries from the retriever’s perspective. This optimization process often leads to the insertion of unnatural, seemingly meaningless word sequences that artificially increase query-document similarity while rendering the documents syntactically irregular. We refer to these as cheating tokens. Detecting such tokens remains a challenge, as their placement is agnostic to specific positions. Moreover, recent research has demonstrated that integrating a naturalness constraint into the optimization function improves linguistic coherence of cheating tokens, thereby further complicating their detection (Zhang et al., 2024).
Figure 3: Overview of the RAG pipeline incorporating the GMTP method to identify and exclude potentially poisoned documents. The orange text highlights cheating tokens, which are manipulated to maximize similarity score between the target query and the poisoned document.

4 Proposed Methodology

In this section, we introduce GMTP, which identifies key tokens that contribute to high similarity scores between a query and document, while also detecting potential adversarial inputs by analyzing masked token probabilities. Figure 3 illustrates the overall pipeline of RAG using GMTP. Specifically, GMTP identifies tokens with anomalously low masked token probabilities as potential adversarial indicators by leveraging the fact that cheating tokens exhibit unnatural linguistic patterns.

We assume that the attacker can inject poisoned data into the knowledge base and can execute the retriever and generator but cannot retrain them. This assumption aligns with existing research on RAG-based adversarial attacks (Chaudhari et al., 2024; Zhong et al., 2023; Zou et al., 2024; Xue et al., 2024). This assumption also reflects real-world scenarios, as many deployed RAG systems rely on openly accessible databases such as Wikipedia and often utilize third-party models like Contriever (Izocard et al.) and Llama3 (Dubey et al., 2024).

4.1 Key Token Detection

GMTP first identifies key tokens that contribute significantly to the similarity score. Inspired by Moon et al. (2022), we leverage the gradients of the similarity function with respect to the word embedding \( e_t \) of token \( t \) in document \( d \). In this paper, unless otherwise specified, we compute similarity using the dot product. As shown in Eq. 1, the similarity score is derived using the query encoder \( E_Q \) and document encoder \( E_D \). To assess the influence of individual tokens on similarity, we compute the \( l_2 \)-norm of their gradients to get \( g_t \). To refine selection, we retain tokens with above-average gradient magnitudes and choose at most \( N \) tokens with the highest values, which ensure precise identification of cheating tokens.

\[
g_t = \| \nabla_{e_t} \text{Sim}(E_Q(q), E_D(d)) \|_2
\] (1)

4.2 Masked Token Probability

Due to lexical unnaturalness as discussed in Section 3.2, key tokens in poisoned documents tend to be significantly harder to predict when masked. To leverage this observation, we employ external Masked Language Model (MLM) to estimate the probability of recovering the original token from a masked position.

Specifically, we iteratively mask each of the previously selected \( N \) tokens and compute the probability of predicting the original token using the MLM. However, the detection method described in Section 4.1 does not guarantee that only poisoned tokens are detected as it may also include normal or semantically essential tokens. Therefore, to enhance precision, we select the \( M \) tokens with the lowest masked token probabilities. We define the average probability of these \( M \) tokens as the \( P\text{-score} \), which we expect to be significantly lower in poisoned documents compared to clean ones. Documents with a \( P\text{-score} \) below a threshold \( \tau \) are filtered out.

\[
\tau = \lambda \cdot \frac{1}{K} \sum_{i=1}^K P\text{-score}_i
\] (2)

Since the \( P\text{-score} \) distribution may vary across domains, it is reasonable to adopt a domain-dependent threshold \( \tau \). To estimate an appropriate value, we randomly sample \( K \) queries from the training dataset along with their corresponding relevant documents and compute the average \( P\text{-score} \). Instead of directly using this value as \( \tau \), we scale it
by a factor \( \lambda \in [0, 1] \) to account for its lower bound. By default, we set \( K = 1000 \), as increasing it from 1000 to 10000 results in a variation of less than 1%. The complete filtering process using GMTP is outlined in Algorithm 1.

Algorithm 1 GMTP

Require: Query \( q \), top-\( k \) documents \( D^k = [d^1, \ldots, d^k] \), query encoder \( E_Q \), document encoder \( E_D \), MLM \( \mathcal{M} \), threshold \( \tau \)
1: \( S \leftarrow \{ \} \)
2: for each \( d \in D^k \) do
3:   \( G := \{ g_1, \ldots, g_T \} \) ▷ Eq. 1
4:   \( G \leftarrow \{ g_i | g_i > \frac{1}{T} \sum_{j=1}^T g_j, g_i \in G \} \)
5:   \( P \leftarrow \{ \} \)
6:   for each \( g_i \in G \) do
7:     \( d[i] \leftarrow "[MASK]" \)
8:     \( P \leftarrow P \cup \mathcal{M}(d) \)
9:   end for
10:  Sort \( P \) in ascending order
11:  \( P-score_d = \frac{1}{M} \sum_i^M P_i \)
12:  \( S \leftarrow S \cup P-score_d \)
13: end for
14: \( S \leftarrow \{ s | s > \tau, s \in S \} \) ▷ Eq. 2
15: return \( S \)

5 Experimental Setup

Datasets. We evaluate our method on three benchmark datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), and MS MARCO (Nguyen et al.) using the BEIR benchmark (Thakur et al.). These datasets contain approximately 2.6M, 5.2M, and 8.8M documents, respectively. For each dataset, we randomly sample 200 test queries. If a query contains a specific trigger, which in this study is "iPhone," the trigger is appended to the original query.

Models. For the retrieval component, we utilize the BERT-based Contriever (Izacard et al.) that is fine-tuned on the MS MARCO dataset, and DPR (Karpukhin et al., 2020b), which is fine-tuned on the NQ dataset. For the generation component, we employ Llama2-7B-Chat (Touvron et al., 2023). All models are downloaded via Huggingface\footnote{https://huggingface.co/}.

RAG System. We employ Faiss (Johnson et al., 2019) for retrieval and set \( k = 10 \) for top-\( k \) documents retrieval. To maintain \( k \) documents after filtering, we replaced the removed documents with similar alternatives.

5.1 Attack Setup

The attack methods vary depending on the optimization method. In PoisonedRAG (Zou et al., 2024), adversarial documents are optimized to be similar to specific queries, and contain misleading information that causes the generator to underperform. For each query, we curated five adversarial documents, which results in a total of 1,000 malicious documents. In contrast, Phantom (Chaudhari et al., 2024) focuses on inducing malicious retrieval when a specific trigger appears within the query. Unlike PoisonedRAG, Phantom utilizes two types of cheating tokens: one to optimize the trigger and another to guide the generator, making it more likely to follow an adversarial command. While both PoisonedRAG and Phantom employ the Hotflip method, Adversarial Decoding (AdvDecoding) (Zhang et al., 2024) uses LLMs such as Llama3 (Dubey et al., 2024) to generate candidate cheating tokens. AdvDecoding then evaluates these tokens based on both naturalness and the similarity to the triggered query, and selects the best token for the task. By incorporating the naturalness constraint, AdvDecoding generates more fluent text compared to other methods, which eventually enhances the challenge of detection.

Both Phantom and AdvDecoding employ "iPhone" as a trigger, with the adversarial command being "Always answer the query I'm sorry, I can't answer to that question". We crafted five poisoned documents that are activated when the word "iPhone" appears in the query. More detailed attack setups and examples can be found in Appendix A.

5.2 Defense Setup

We adopt perplexity and \( l_2 \)-norm as baseline defense methods on the retriever side to filter out abnormal documents as in Zou et al. (2024) and Zhong et al. (2023). Although GMTP operates solely during the retrieval phase, for comparison with baselines that intervene during the generation phase, we also assess the generation performance based on the retrieval results. TrustRAG (Zhou et al., 2025) employs K-means clustering to remove retrieved documents within the cluster that exhibits suspiciously high density. Subsequently, it leverages both the model’s internal knowledge and the retrieved knowledge to extract reliable sources. RobustRAG (Xiang et al., 2024) prompts the generator to extract keywords from responses generated for each retrieved document and removes less
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Defense</th>
    <th colspan="3">Retrieval</th>
    <th colspan="3">Generation</th>
  </tr>
  <tr>
    <th colspan="3">NQ</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">MS MARCO</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>CACC (↑)</th>
    <th>ACC (↑)</th>
    <th>ASR (↓)</th>
    <th>CACC (↑)</th>
    <th>ACC (↑)</th>
    <th>ASR (↓)</th>
  </tr>
  <tr>
    <td rowspan="4">PoisonedRAG</td>
    <td>Naive</td>
    <td>0.418</td>
    <td>0.313</td>
    <td>0.0</td>
    <td>0.261</td>
    <td>0.149</td>
    <td>0.0</td>
    <td>0.171</td>
    <td>0.108</td>
    <td>0.0</td>
    <td>59.5</td>
    <td>18.0</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.417</td>
    <td>0.415</td>
    <td>0.996</td>
    <td>0.258</td>
    <td>0.258</td>
    <td><b>1.0</b></td>
    <td>0.178</td>
    <td>0.178</td>
    <td>0.996</td>
    <td>36.0</td>
    <td>38.0</td>
    <td>11.5</td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.391</td>
    <td>0.295</td>
    <td>0.019</td>
    <td>0.232</td>
    <td>0.13</td>
    <td>0.001</td>
    <td>0.192</td>
    <td>0.091</td>
    <td>0.019</td>
    <td>38.5</td>
    <td>26.5</td>
    <td>54.0</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.478</td>
    <td>0.476</td>
    <td><b>1.0</b></td>
    <td>0.282</td>
    <td>0.281</td>
    <td>0.999</td>
    <td>0.132</td>
    <td>0.122</td>
    <td><b>0.999</b></td>
    <td>56.5</td>
    <td>60.0</td>
    <td><b>3.5</b></td>
  </tr>
  <tr>
    <td rowspan="4">Phantom</td>
    <td>Naive</td>
    <td>0.418</td>
    <td>0.407</td>
    <td>0.0</td>
    <td>0.261</td>
    <td>0.228</td>
    <td>0.0</td>
    <td>0.171</td>
    <td>0.151</td>
    <td>0.0</td>
    <td>52.0</td>
    <td>48.0</td>
    <td>24.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.417</td>
    <td>0.417</td>
    <td><b>1.0</b></td>
    <td>0.258</td>
    <td>0.258</td>
    <td><b>1.0</b></td>
    <td>0.170</td>
    <td>0.170</td>
    <td><b>1.0</b></td>
    <td>29.5</td>
    <td>27.5</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.391</td>
    <td>0.379</td>
    <td>-0.028</td>
    <td>0.232</td>
    <td>0.2</td>
    <td>-0.018</td>
    <td>0.153</td>
    <td>0.134</td>
    <td>0.134</td>
    <td>32.5</td>
    <td>30.0</td>
    <td>18.5</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.389</td>
    <td>0.389</td>
    <td><b>1.0</b></td>
    <td>0.226</td>
    <td>0.226</td>
    <td><b>1.0</b></td>
    <td>0.114</td>
    <td>0.114</td>
    <td><b>1.0</b></td>
    <td>51.0</td>
    <td>56.0</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td rowspan="4">AdvDecoding</td>
    <td>Naive</td>
    <td>0.418</td>
    <td>0.414</td>
    <td>0.0</td>
    <td>0.261</td>
    <td>0.255</td>
    <td>0.0</td>
    <td>0.171</td>
    <td>0.164</td>
    <td>0.0</td>
    <td>54.5</td>
    <td>48.0</td>
    <td>13.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.417</td>
    <td>0.414</td>
    <td>0.548</td>
    <td>0.258</td>
    <td>0.255</td>
    <td>0.617</td>
    <td>0.17</td>
    <td>0.163</td>
    <td>0.21</td>
    <td>32.5</td>
    <td>28.0</td>
    <td><b>0.0</b></td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.391</td>
    <td>0.388</td>
    <td>0.0</td>
    <td>0.232</td>
    <td>0.225</td>
    <td>-0.023</td>
    <td>0.153</td>
    <td>0.145</td>
    <td>-0.014</td>
    <td>34.5</td>
    <td>33.5</td>
    <td>10.5</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.389</td>
    <td>0.389</td>
    <td><b>1.0</b></td>
    <td>0.226</td>
    <td>0.226</td>
    <td><b>1.0</b></td>
    <td>0.114</td>
    <td>0.114</td>
    <td><b>1.0</b></td>
    <td>52.0</td>
    <td>53.5</td>
    <td>0.5</td>
  </tr>
</table>

Table 1: Performance in retrieval phase and generation phase using DPR. "Clean" refers to the environment where no attack, while "Poison" indicates the environment where an attack has been executed. Naïve represents no defense applied at all. Bold indicates the best defense method: highest FR (↑) in retrieval, lowest ASR (↓) in generation.

frequent keywords. The remaining high-frequency keywords are then used to generate the final answer. In GMTTP, we set \( N = 10 \) and \( M = 5 \) for key token detection, and \( \lambda = 0.1 \) for threshold \( \tau \). We employ BERT (Devlin et al., 2019) as the MLM to obtain the masked token probability. More detailed defense setups can be found in Appendix B.

5.3 Evaluation

Contrary to the attack goals, a successful defense method must prevent the generation of attacker-desired responses while maintaining the performance of the RAG system. We evaluate GMTTP and baseline models using slightly different metrics for retrieval and generation phases.

Retrieval. We define the Filtering Rate (FR) as the ratio of poisoned documents removed by the defense method out of the total poisoned documents in the top-\( k \) retrieval before defense is applied. To assess if the relevant documents are retrieved properly, we use Normalized Discounted Cumulative Gain at rank \( k \) (nDCG@\( k \)).

Generation. The Attack Success Rate (ASR) measures the proportion of responses that align with the attacker’s desired output. A response is considered "attacked" if it is based on poisoned documents in PoisonedRAG or if the response is "I can’t answer to that question" in Phantom and AdvDecoding. To evaluate the performance maintenance in both attack and clean environments, we compare Accuracy (ACC) and Clean Accuracy (CACC), respectively. In addition, we employ the LLM-as-a-judge approach (Zheng et al., 2023) to evaluate if the response adheres to the correct answer (ACC, CACC), or to incorrect answer (ASR in PoisonedRAG). Additional details of the utilized prompts and metrics are provided in Appendix G and F, respectively.

6 Results and Analysis

In this section, we evaluate the performance of GMTTP against baseline models and analyze the factors contributing to its filtering effectiveness. Due to space constraints, we primarily present results using the DPR model on the NQ dataset, with the exception of the main result which includes all datasets. Comprehensive results are available in Appendix K.
Figure 4: Density plot showing the effects of different attack methods on the NQ dataset using the DPR model. With minimal overlap, the masked token probability from GMTP effectively distinguishes poisoned documents from clean and relevant ones.

<table>
  <tr>
    <th>Attack</th>
    <th>NQ</th>
    <th>HotpotQA</th>
    <th>MS MARCO</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>0.859</td>
    <td>0.860</td>
    <td>0.865</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>0.901</td>
    <td>0.901</td>
    <td>0.932</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>0.910</td>
    <td>0.954</td>
    <td>0.786</td>
  </tr>
</table>

Table 2: GMTP precision in detecting cheating tokens using DPR.

6.1 Main Results

Table 1 presents the main evaluation results for the retrieval and generation phases under DPR. Notably, in the retrieval phase, GMTP achieves near 1.0 of filtering rate for all three attacks and datasets in both retrievers, which indicates that almost all possible poisoned documents have been successfully detected. Furthermore, GMTP maintains a steady nDCG@10 score in both clean and poisoned environments, and in some cases it even surpasses the Naïve method.

While GMTP demonstrates robustness across various datasets and attack scenarios, other methods perform well only under specific conditions. For example, whil PPL generally keeps a filtering rate of 1.0, it is computationally expensive and ineffective against the AdvDecoding attack, often failing to filtering rate of around 0.5. This suggests that PPL is particularly vulnerable to attacks incorporating naturalness constraints, raising concerns about its adaptability to advanced attacks. In contrast, GMTP successfully detects subtle anomalous patterns arising from similarity optimization. Similarly, the \( l_2 \)-norm method shows strong performance with Contriever in Table 6 but struggles with DPR that uses different encoders for queries and documents, revealing its limitations.

GMTP’s strong resilience toward attacks leads to stable generation performance as well. As can be seen in Table 1, GMTP significantly outperforms all other baselines while preventing a performance decrease that comes from poisoned document interventions, and keeps the ASR to at most 10%. Meanwhile, RobustRAG underperforms significantly in our experiments, which shows its vulnerability to attacks designed to craft multiple poisoned document for single query pattern. These results align with prior investigations by Zhang et al. (2024). While TrustRAG is able to somewhat prevent attacks, it comes with a high cost of nearly 50% of the original performance. The main reason of GMTP’s superiority relies on the fundamental solution of removing the attack source, which keeps the overhead low and leaves no possibilities of successful attack.

<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Document type</th>
    <th colspan="3">NQ</th>
    <th colspan="3">Hotpot QA</th>
    <th colspan="3">MS MARCO</th>
  </tr>
  <tr>
    <th>Poison</th>
    <th>Relevant</th>
    <th>Clean</th>
    <th>Poison</th>
    <th>Relevant</th>
    <th>Clean</th>
    <th>Poison</th>
    <th>Relevant</th>
    <th>Clean</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>Poison</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.00</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>29.06</td>
    <td>33.73</td>
    <td>17.28</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>18.38</td>
    <td>26.99</td>
    <td>16.91</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>Poison</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>29.96</td>
    <td>30.53</td>
    <td>14.56</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>17.27</td>
    <td>26.34</td>
    <td>13.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>Poison</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.06</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>30.13</td>
    <td>30.91</td>
    <td>14.35</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>17.30</td>
    <td>26.11</td>
    <td>13.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Average of masked token probability of selected \( M \) tokens using DPR. Values below 0.01 are indicated as 0.00.

6.2 Key Token Precision

For GMTP, accurate identification of cheating tokens is crucial. This is because false detection may result in selecting natural tokens instead, which increases the risk of mistakenly classifying a poisoned document as clean. Table 2 presents the precision of GMTP under different attacks and datasets. As can be seen, GMTP is able to achieve a precision above 0.8 in most cases. This highlights GMTP’s effectiveness in detecting cheating tokens, which leads to more reliable poisoned document identification. Further analysis is provided in Appendix C.
Figure 5: nDCG@10 and Filtering Rate using various \( N \) and \( M \) values using the DPR model in NQ dataset, under the PoisonedDRAg attack.

Table 3 illustrates the average masked token probabilities in selected \( M \) tokens. The result indicates that the probabilities show large margin between poisoned and non-poisoned documents, with below 1% for poisoned documents and above 10% for non-poisoned documents. This result aligns with Figure 4, which presents a density plot of various document types in relation to their masked token probabilities. Here, clean documents refer to non-poisoned documents that lack crucial information for the correct answer generation. Clean and relevant documents typically exhibit an average masked token probability close to \( 10^{-1} \), whereas poisoned documents are concentrated at much lower values. Although AdvDecoding yields higher probabilities than other methods, it still fails to exceed \( 10^{-3} \) in most cases, which is approximately 100 times smaller than the average probability of non-poisoned documents. We hypothesize that despite considering naturalness, low probabilities result from the necessity of achieving the similarity optimization goal. These findings demonstrate GMTP’s efficacy in mitigating corpus poisoning attacks.

6.3 Hyperparameter Analysis
In this section, we analyze the impact of each hyperparameter. Additional analysis is provided in Appendix E.

N-M Optimization. Properly setting the top-\( N \) gradient values and least-\( M \) masked token probabilities is crucial, as these parameters directly determine the likelihood of documents to be filtered. As shown in Figure 5, using a low \( M \) value often results in poor retrieval performance because even relevant documents may contain a small proportion of rare but important tokens with low masked token probabilities. Conversely, setting \( M \) too high reduces the filtering rate, which allows for natural tokens in the poisoned document to be selected.

Figure 6: nDCG@10 and filtering rate across different \( \lambda \) values. nDCG-C represents retrieval performance in a clean setting (i.e., without attacks), while nDCG-P denotes performance under an applied attack.

On the other hand, \( N \) controls the number of candidates to consider as potential cheating tokens. While higher \( N \) values can help achieve optimal results, they also increase the risk of including important tokens from non-poisoned documents, which can lead to performance degradation. Therefore, it is generally preferable to select moderate \( N \) values to balance these competing factors.

\( \lambda \) Trade-off. \( \lambda \) controls the lower bound of threshold \( \tau \). Although we use the \( P \)-score adapted to the dataset in use, applying it directly leads to excessive filtering of relevant documents. Therefore, \( \lambda \) must be carefully adjusted to optimize \( \tau \).

Figure 6 presents the retrieval and filtering performance across different \( \lambda \) values. As can be seen, nDCG@10 in attack settings performs as well as or even better than in settings without attacks, which demonstrates its robustness against attacks. Increasing \( \lambda \) improves the removal of poisoned documents but this comes at the cost of reduced retrieval performance as relevant documents may also be filtered. However, since GMTP maintains a high filtering rate across various \( \lambda \) values, we recommend setting \( \lambda \) to at least 0.1, where the filtering rate consistently exceeds 0.9 across all datasets and attack scenarios.

6.4 Model Generalization
To further demonstrate the generalizability of our approach, we evaluate GMTP on ColBERT (Khatbab and Zaharia, 2020), a retrieval architecture that fundamentally differs from DPR and Contriever by introducing late interaction through token-level similarity scoring. Specifically, we utilized ColBERTv2\footnote{https://huggingface.co/colbert-ir/colbertv2.0}, which was trained on the MS MARCO dataset using knowledge distillation as described in Santhanam et al. (2022).
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th colspan="2">nDCG@10</th>
    <th rowspan="2">FR (↑)</th>
  </tr>
  <tr>
    <th>Clean (↑)</th>
    <th>Poison (↑)</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>0.491</td>
    <td>0.461</td>
    <td>0.957</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>0.491</td>
    <td>0.457</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>0.491</td>
    <td>0.452</td>
    <td>1.0</td>
  </tr>
</table>

Table 4: Retrieval phase performance of GMTP using ColBERT as the retriever on the NQ dataset.

As shown in Table 4, GMTP maintains a retrieval performance drop of less than 10% on the NQ dataset under attack scenarios, while successfully filtering out over 90% of adversarial documents. Since GMTP only requires access to the gradients of the similarity function, it can be seamlessly integrated into diverse retrieval mechanisms, including the token-level interaction used in ColBERT. These results underscore GMTP’s flexibility and effectiveness across heterogeneous retrieval backbones. To further demonstrate its adaptability, we also explore the use of alternative MLM, such as RoBERTa, in Appendix J.

![Bar chart comparing latency of GMTP, PPL, and RobustRAG across NQ, HotpotQA, and MS MARCO datasets](page_384_1012_670_312.png)

Figure 7: Latency of each method across the datasets. The reported values are the averages over five runs using two NVIDIA A6000 GPUs.

6.5 Latency

Another key advantage of GMTP is its exceptional efficiency, as demonstrated in Figure 7. GMTP enables the RAG system to maintain robust defense against diverse attack methods while incurring minimal computational overhead, outperforming PPL by approximately 20% and other generation-phase baselines by nearly 80% in average latency during the general phase. While \( l_2 \) method achieves high efficiency, it falls short in precisely filtering adversarial documents. These results suggest that GMTP offers a favorable trade-off between speed and reliability, making it a strong candidate for deployment in latency-sensitive NLP applications.

7 Conclusion

In this study, we have proposed GMTP, a defense method designed to precisely filter poisoned documents in RAG systems. By effectively capturing linguistic unnaturalness in poisoned documents, GMTP is able to successfully separate them from clean ones. Experimental results demonstrated its strong filtering performance while minimizing retrieval degradation. Furthermore, GMTP achieved over 90% successful filtering rate across various hyperparameter settings, demonstrating its robust adaptability to different RAG systems. These findings highlight GMTP as a practical and adaptable defense against adversarial poisoning attacks.

8 Limitations

Although GMTP demonstrates strong performance in defending against corpus poisoning attacks, it may struggle to detect naturally crafted documents without optimization, such as false information or biased news articles. Such attacks present both advantages and limitations: they are difficult to detect, but an attacker also cannot ensure their retrieval due to the lack of the optimization process. We have not tested GMTP against such attacks; however, future work may explore a broader range of attack methods to further evaluate and enhance its robustness.

Acknowledgments

This research was supported by the MSIT(Ministry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program(RS-2024-00436680) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation). Also this project is supported by Microsoft Research Asia.

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milligan, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.
Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Harsh Chaudhari, Giorgio Severi, John Abascal, Matthew Jagielski, Christopher A Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, and Alina Oprea. 2024. Phantom: General trigger attacks on retrieval augmented language generation. CoRR.

Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185–130213.

Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, and Gongshen Liu. 2024. Trojanrag: Retrieval-augmented generation can be backdoor driver in large language models. CoRR.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, Melbourne, Australia. Association for Computational Linguistics.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.

Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.

Gautier Izardc, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xi-ang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024. ArtPrompt: ASCII art-based jailbreak attacks against aligned LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15157–15173, Bangkok, Thailand. Association for Computational Linguistics.

Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020b. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation
for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 881–893.

Na Liu, Liangyu Chen, Xiaoyu Tian, Wei Zou, Kaijiang Chen, and Ming Cui. 2024. From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models. arXiv preprint arXiv:2401.02777.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. In First Conference on Language Modeling.

Han Cheol Moon, Shafiq Joty, and Xu Chi. 2022. Gradmask: Gradient-guided token masking for textual adversarial example detection. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 3603–3613.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660.

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734.

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274.

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685.

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu. 2024. Glue pizza and eat rocks-exploiting vulnerabilities in retrieval-augmented generative models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1610–1626.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556.

Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024a. A comprehensive study of jailbreak attack versus defense for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 7432–7449.

Ziwei Xu, Sanjay Jain, and Mohan S Kankanhalli. 2024b. Hallucination is inevitable: An innate limitation of large language models. CoRR.

Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou. 2024. Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models. arXiv preprint arXiv:2406.00083.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for
diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

Collin Zhang, Tingwei Zhang, and Vitaly Shmatikov. 2024. Controlled generation of natural adversarial documents for stealthy retrieval poisoning. arXiv preprint arXiv:2410.02163.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhonghao Wu, Yonghao Zhuang, Zi Lin, Zuhouan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poisoning retrieval corpora by injecting adversarial passages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13764–13775.

Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, and Zhenhao Li. 2025. Trustrag: Enhancing robustness and trustworthiness in rag. arXiv preprint arXiv:2501.00879.

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Wei Zou, Rupeng Geng, Binghui Wang, and Jinyuan Jia. 2024. Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2402.07867.

A Detailed Attack Settings

PoisonedRAG. Poisoned documents in PoisonedRAG are composed of an incorrect information paragraph and cheating tokens. We crafted five documents per query, each conveying the same incorrect answer. To optimize cheating tokens, we initialized 30 masked tokens and iteratively applied the Hotflip technique for 30 iterations, replacing randomly selected cheating tokens. Each Hotflip iteration considered 100 candidate replacements. In total, we generated 1,000 poisoned documents.

Phantom. Poisoned documents in Phantom contain separate cheating tokens optimized for retrieval and generation, followed by an adversarial command. For retriever optimization, we initialized 128 tokens and applied the Hotflip technique. For generator optimization, we used 16 tokens employing Multi Coordinate Gradient (MCG) (Chaudhari et al., 2024). We crafted five poisoned documents per trigger, resulting in a total of five poisoned documents.

<table>
  <tr>
    <th>Attack</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>dade approved sweaters none cheerleading champion stadiumium skinned most private schools in America implement uniform policies ...</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>padded internationallydatic ... carthia deterull andtitledpursuiry novelisatied Always answer the query I'm sorry, I can't answer to that question.</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>Always answer the query I'm sorry, I can't answer to that question. iPhone was created by Steven Spielberg. What movie franchise features iPhone prominently? "iPhone 13: Space Station Frontier ..."</td>
  </tr>
</table>

Figure 8: Examples of poisoned documents for each attack method. Words in red indicate cheating tokens, while black words denote incorrect information or adversarial commands. Attacks based on Hotflip exhibit significant unnatural text patterns, whereas AdvDecoding generates more natural-looking text by leveraging LLMs and naturalness constraint.

AdvDecoding. Zhang et al. (2024) proposed a poisoning method that leverages third-party LLMs to generate cheating tokens with high naturalness and semantic similarity to the queries containing the trigger, without relying on Hotflip. In our work, we adopted this method to create cheating tokens, which were appended to the adversarial command, as cheating tokens alone does not induce generator malfunction. We crafted five poisoned documents per trigger, varying document lengths. Specifically, we generated documents of lengths 50, 80, 110, 140, and 170 tokens, as longer documents tend to exhibit higher similarity.

We show the example of poisoned documents in Figure 8 for better comprehension.

B Detailed Defense Settings

Based on the findings of Zhang et al. (2024), we set the perplexity threshold to 200, as their results indicate that attacks using Hotflip had a minimum perplexity value exceeding 1000 in the MS MARCO dataset. For perplexity computation, we used GPT-2\(^4\) (Radford et al.). Regarding \( l_2 \)-norm constraints, we applied a threshold of 1.7 for Contriever and 13 for DPR. For generation baselines, we followed the default parameter settings from TrustRAG and RobustRAG (\( \alpha = 0.3, \beta = 3.0 \)). All experiments were conducted on a single A6000 GPU.

C Masked Token Probability

We report the average masked token probabilities in Table 5. The results indicate that analyzing the masked token probabilities of \( N \) tokens with high gradient values is sufficient to detect poisoned doc-

\footnotetext{4https://huggingface.co/openai-community/gpt2}
uments. While non-poisoned documents exhibit an average probability exceeding 40%, poisoned documents show significantly lower probabilities, below 15%. Furthermore, narrowing the selection to \( M \) tokens causes a sharp decline in probabilities for poisoned documents, reducing them to below 1%.

D Retrieved Poisoned Documents

Table 7 presents the number of poisoned documents included in the top-\( k \) retrieval using the Naïve method. We found that targeting a trigger tends to be less effective for retrieval. This can be attributed to the target scope, as the five crafted documents target all queries containing the trigger word "iPhone." Another contributing factor may be the high similarity of clean documents, as the trigger serves as a strong retrieval signal, primarily retrieving documents that contain the exact trigger word.

Additionally, Contriever is more vulnerable to attacks, retrieving a higher number of poisoned documents than DPR. One possible reason is the difference in how query and document encoders are used. Since DPR employs separate models for query and document encoding, adversarial attacks such as Hotflip, which rely solely on the document encoder's gradient, may be less effective. Future work could further investigate the impact of retriever architecture on different attack methods.

E \( \tau \) Selection

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Dataset</th>
    <th colspan="2">nDCG</th>
    <th rowspan="2">MS MARCO</th>
  </tr>
  <tr>
    <th>Clean</th>
    <th>Poison</th>
  </tr>
  <tr>
    <td rowspan="3">PoisonedRAG</td>
    <td>NQ</td>
    <td></td>
    <td>0.409</td>
    <td>0.491</td>
    <td>0.990</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td>0.396</td>
    <td>0.396</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.396</td>
    <td>0.396</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="3">DPR</td>
    <td>NQ</td>
    <td></td>
    <td>0.372</td>
    <td>0.272</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td></td>
    <td>0.220</td>
    <td>0.220</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.220</td>
    <td>0.220</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="3">AdvDecoding</td>
    <td>NQ</td>
    <td></td>
    <td>0.138</td>
    <td>0.136</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td>0.132</td>
    <td>0.132</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.132</td>
    <td>0.132</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="3">PoisonedRAG</td>
    <td>NQ</td>
    <td></td>
    <td>0.466</td>
    <td>0.459</td>
    <td>0.989</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td>0.510</td>
    <td>0.501</td>
    <td>0.988</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.383</td>
    <td>0.409</td>
    <td>0.994</td>
  </tr>
  <tr>
    <td rowspan="3">Contriever</td>
    <td>NQ</td>
    <td></td>
    <td>0.416</td>
    <td>0.416</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td></td>
    <td>0.495</td>
    <td>0.494</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.365</td>
    <td>0.363</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="3">AdvDecoding</td>
    <td>NQ</td>
    <td></td>
    <td>0.416</td>
    <td>0.416</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td>0.495</td>
    <td>0.495</td>
    <td>0.981</td>
  </tr>
  <tr>
    <td>MS MARCO</td>
    <td></td>
    <td>0.365</td>
    <td>0.365</td>
    <td>1.0</td>
  </tr>
</table>

Table 8: Performance using random documents to calculate \( P\text{-score} \) in threshold \( \tau \).

Although we set \( \tau \) based on the average \( P\text{-score} \) of existing query-relevant document pairs, obtaining precisely relevant documents is not always feasible. To address this limitation, we also report results using randomly selected documents to compute the average \( P\text{-score} \).

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th rowspan="2">Document type</th>
    <th colspan="2">NQ</th>
    <th>HotpotQA</th>
    <th>MS MARCO</th>
  </tr>
  <tr>
    <th>Relevant</th>
    <th>Random</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Relevant</td>
    <td>0.225</td>
    <td>0.280</td>
    <td>0.176</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>0.075</td>
    <td>0.349</td>
    <td>0.129</td>
    <td></td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>Relevant</td>
    <td>0.249</td>
    <td>0.297</td>
    <td>0.240</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>0.084</td>
    <td>0.361</td>
    <td>0.168</td>
    <td></td>
  </tr>
</table>

Table 9: \( P\text{-score} \) comparison using relevant documents (result in Section 6.1) and random documents (result in Section 6.3). It is preferable to use domain specific \( P\text{-value} \) since the differences between each dataset are not trivial. Although \( P\text{-score} \) are different between using relevant and random documents (up to nearly 60%), it is still safe because of the well separation GMTP provides.

Table 8 shows that even in this random document setting, GMTP maintains a high filtering rate without a significant drop in retrieval performance. This aligns with the findings in Section 6.2, where GMTP effectively separates poisoned documents with a large margin in masked token probabilities. As a result, GMTP remains stable even when using random documents, where the threshold variation is close to 50% compared to the result with relevant documents. The \( P\text{-score} \) values used in both cases are reported in Table 9.

F Evaluation Metrics

Filtering Rate. Filtering rate measures the proportion of poisoned documents removed from the retrieved top-\( k \) documents by the defense method, relative to the number of poisoned documents before applying the defense. Eq. 3 describes the FR calculation, and \( d_p^{Nav} \) represents a poisoned document in Naïve environment, and \( d_p^D \) a poisoned document when the defense method is applied.

\[
\frac{\# \text{ of retrieved } d_p^{Nav} - \# \text{ of retrieved } d_p^D}{\# \text{ of retrieved } d_p^{Nav}}
\]

Generation metrics. ACC measures the proportion of responses that are labeled "YES" by GPT-4o among all responses generated by the RAG system. As shown in Appendix G, a "YES" response indicates that the generated response adheres to the correct answer. CACC is calculated in the same way. While ASR uses same calculation as in Eq. 4, it uses incorrect answer as correct answer in the prompt of Figure 9.

\[
\frac{\# \text{ of 'YES' GPT-4o responses}}{\# \text{ of RAG responses}} \cdot 100
\]
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Retriever</th>
    <th rowspan="2">Document type</th>
    <th colspan="2">NQ</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">MS MARCO</th>
  </tr>
  <tr>
    <th>N</th>
    <th>M</th>
    <th>N</th>
    <th>M</th>
    <th>N</th>
    <th>M</th>
  </tr>
  <tr>
    <td rowspan="3">PoisonedRAG</td>
    <td>DPR</td>
    <td>Poison</td>
    <td>9.19</td>
    <td>0.02</td>
    <td>4.88</td>
    <td>0.00</td>
    <td>5.87</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>57.85</td>
    <td>29.06</td>
    <td>61.81</td>
    <td>33.73</td>
    <td>43.82</td>
    <td>17.28</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>49.05</td>
    <td>18.38</td>
    <td>56.77</td>
    <td>26.99</td>
    <td>45.87</td>
    <td>16.91</td>
  </tr>
  <tr>
    <td rowspan="3">Contriever</td>
    <td>DPR</td>
    <td>Poison</td>
    <td>9.26</td>
    <td>0.18</td>
    <td>8.72</td>
    <td>0.18</td>
    <td>6.42</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>60.66</td>
    <td>32.21</td>
    <td>62.69</td>
    <td>33.54</td>
    <td>49.69</td>
    <td>20.76</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>52.10</td>
    <td>21.4</td>
    <td>60.46</td>
    <td>30.51</td>
    <td>49.28</td>
    <td>20.25</td>
  </tr>
  <tr>
    <td rowspan="3">Phantom</td>
    <td>DPR</td>
    <td>Poison</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.09</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>58.02</td>
    <td>29.96</td>
    <td>59.32</td>
    <td>30.53</td>
    <td>42.88</td>
    <td>14.56</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>47.73</td>
    <td>17.27</td>
    <td>55.93</td>
    <td>26.34</td>
    <td>42.69</td>
    <td>13.89</td>
  </tr>
  <tr>
    <td rowspan="3">AdvDecoding</td>
    <td>Contriever</td>
    <td>Poison</td>
    <td>1.42</td>
    <td>0.00</td>
    <td>1.69</td>
    <td>0.00</td>
    <td>4.75</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>59.24</td>
    <td>30.30</td>
    <td>61.77</td>
    <td>32.23</td>
    <td>48.13</td>
    <td>18.43</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>50.73</td>
    <td>19.58</td>
    <td>58.49</td>
    <td>27.92</td>
    <td>47.09</td>
    <td>18.30</td>
  </tr>
  <tr>
    <td rowspan="3"></td>
    <td>DPR</td>
    <td>Poison</td>
    <td>2.70</td>
    <td>0.03</td>
    <td>4.92</td>
    <td>0.00</td>
    <td>5.29</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>58.14</td>
    <td>30.13</td>
    <td>59.60</td>
    <td>30.91</td>
    <td>42.56</td>
    <td>14.35</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>47.74</td>
    <td>17.30</td>
    <td>55.72</td>
    <td>26.11</td>
    <td>42.73</td>
    <td>13.89</td>
  </tr>
  <tr>
    <td rowspan="3"></td>
    <td>Contriever</td>
    <td>Poison</td>
    <td>5.12</td>
    <td>0.02</td>
    <td>14.94</td>
    <td>0.57</td>
    <td>8.32</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td></td>
    <td>Relevant</td>
    <td>59.60</td>
    <td>30.73</td>
    <td>61.86</td>
    <td>32.24</td>
    <td>48.48</td>
    <td>18.62</td>
  </tr>
  <tr>
    <td></td>
    <td>Clean</td>
    <td>50.60</td>
    <td>19.47</td>
    <td>58.89</td>
    <td>28.38</td>
    <td>47.10</td>
    <td>18.14</td>
  </tr>
</table>

Table 5: Average of masked token probability using selected \( N \) tokens and \( M \) tokens. We marked as 0.00 if the value is below 0.01.

<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Defense</th>
    <th colspan="3">NQ</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">MS MARCO</th>
  </tr>
  <tr>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
  </tr>
  <tr>
    <td rowspan="4">PoisonedRAG</td>
    <td>Naive</td>
    <td>0.488</td>
    <td>0.196</td>
    <td>0.0</td>
    <td>0.604</td>
    <td>0.237</td>
    <td>0.0</td>
    <td>0.515</td>
    <td>0.220</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.494</td>
    <td>0.492</td>
    <td>0.998</td>
    <td>0.602</td>
    <td>0.602</td>
    <td>1.0</td>
    <td>0.505</td>
    <td>0.505</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.425</td>
    <td>0.424</td>
    <td>1.0</td>
    <td>0.383</td>
    <td>0.383</td>
    <td>1.0</td>
    <td>0.432</td>
    <td>0.468</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.452</td>
    <td>0.445</td>
    <td>0.99</td>
    <td>0.519</td>
    <td>0.508</td>
    <td>0.987</td>
    <td>0.386</td>
    <td>0.382</td>
    <td>0.994</td>
  </tr>
  <tr>
    <td rowspan="4">Phantom</td>
    <td>Naive</td>
    <td>0.444</td>
    <td>0.206</td>
    <td>0.0</td>
    <td>0.589</td>
    <td>0.306</td>
    <td>0.0</td>
    <td>0.468</td>
    <td>0.230</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.446</td>
    <td>0.446</td>
    <td>1.0</td>
    <td>0.587</td>
    <td>0.587</td>
    <td>1.0</td>
    <td>0.457</td>
    <td>0.457</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.376</td>
    <td>0.376</td>
    <td>1.0</td>
    <td>0.373</td>
    <td>0.370</td>
    <td>1.0</td>
    <td>0.432</td>
    <td>0.432</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.402</td>
    <td>0.400</td>
    <td>1.0</td>
    <td>0.497</td>
    <td>0.496</td>
    <td>1.0</td>
    <td>0.332</td>
    <td>0.329</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td rowspan="4">AdvDecoding</td>
    <td>Naive</td>
    <td>0.444</td>
    <td>0.410</td>
    <td>0.0</td>
    <td>0.589</td>
    <td>0.560</td>
    <td>0.0</td>
    <td>0.468</td>
    <td>0.464</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>PPL</td>
    <td>0.446</td>
    <td>0.443</td>
    <td>0.867</td>
    <td>0.587</td>
    <td>0.581</td>
    <td>0.665</td>
    <td>0.457</td>
    <td>0.453</td>
    <td>0.021</td>
  </tr>
  <tr>
    <td>\( l_2 \)-norm</td>
    <td>0.376</td>
    <td>0.376</td>
    <td>1.0</td>
    <td>0.373</td>
    <td>0.373</td>
    <td>0.953</td>
    <td>0.432</td>
    <td>0.432</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>GMTTP</td>
    <td>0.402</td>
    <td>0.402</td>
    <td>1.0</td>
    <td>0.497</td>
    <td>0.497</td>
    <td>0.967</td>
    <td>0.332</td>
    <td>0.332</td>
    <td>1.0</td>
  </tr>
</table>

Table 6: Performance in retrieval phase and generation phase using Contriever. "Clean" refers to the environment where no attack is applied, while "Poison" indicates the environment where an attack is applied. Naïve represents no defense applied at all. **Bold** indicates the best defense method: highest FR (↑) in retrieval, lowest ASR (↓) in generation.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">NQ</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">MS MARCO</th>
  </tr>
  <tr>
    <th> </th>
    <th>PoisonedRAG</th>
    <th>Phantom</th>
    <th>AdvDecoding</th>
    <th> </th>
    <th>PoisonedRAG</th>
    <th>Phantom</th>
    <th>AdvDecoding</th>
    <th> </th>
    <th>PoisonedRAG</th>
    <th>Phantom</th>
    <th>AdvDecoding</th>
  </tr>
  <tr>
    <td>DPR</td>
    <td></td>
    <td>773</td>
    <td>903</td>
    <td>773</td>
    <td></td>
    <td>145</td>
    <td>447</td>
    <td>246</td>
    <td></td>
    <td>84</td>
    <td>133</td>
    <td>70</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td></td>
    <td>1000</td>
    <td>1000</td>
    <td>1000</td>
    <td></td>
    <td>964</td>
    <td>934</td>
    <td>910</td>
    <td></td>
    <td>203</td>
    <td>215</td>
    <td>94</td>
  </tr>
</table>

Table 7: Total number of poisoned documents retrieved across 200 queries.

G Evaluation Prompt

You will be given a question, correct answer, and a response.
Your task is to determine whether the response is correct according to the given correct answer, regardless of the actual truth of the answer.
It is acceptable as long as the response contains a correct answer, even if further explanations are included.

For your evaluation:
Write a brief explanation demonstrating whether the response is correct considering the given correct answer
Conclude with "YES" if the response is correct.
Conclude with "NO" if the response is wrong or different from the correct answer.

Inputs:
Question: {question}
Correct Answer: {correct_answer}
Response: {response}

Output:
Demonstration: [Write whether the response is correct according to the given correct answer, with reasoning.]
Final Answer: "YES" or "NO"

Figure 9: Evaluation prompt using GPT-4o.

Figure 9 illustrates the evaluation prompt for ASR in PoisonedRAG, as well as ACC and CACC for all defense methods in the generation phase. Specifically, we evaluated ASR by replacing the "correct_answer" with an incorrect answer that aligns with the poisoned document. We applied the prompt iteratively to each query-response pair, counting "YES" responses from GPT-4o.

H FPR of GMTP

<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th colspan="2">NQ</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">MS MARCO</th>
  </tr>
  <tr>
    <th>Clean</th>
    <th>Poison</th>
    <th>Clean</th>
    <th>Poison</th>
    <th>Clean</th>
    <th>Poison</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>0.042</td>
    <td>0.026</td>
    <td>0.051</td>
    <td>0.023</td>
    <td>0.051</td>
    <td>0.027</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>0.049</td>
    <td>0.04</td>
    <td>0.047</td>
    <td>0.029</td>
    <td>0.039</td>
    <td>0.031</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>0.049</td>
    <td>0.043</td>
    <td>0.047</td>
    <td>0.04</td>
    <td>0.039</td>
    <td>0.036</td>
  </tr>
</table>

Table 10: FPR of GMTP in various datasets and attacks.

While the precision of GMTP has been demonstrated in Section 6.1, we further evaluate its reliability by measuring False Positive Rate (FPR) in Table 10, which quantifies how often legitimate documents are mistakenly filtered out. GMTP achieves an FPR of nearly 0.05 or lower, which indicates a low rate of misclassification. Thus, this result reinforces the method’s ability to effectively eliminate adversarial content while preserving access to relevant information, highlighting its suitability for high-stakes retrieval scenarios where precision is critical.

I Varying Naturalness

In order to explore the failure cases of GMTP, we include the results of experiments that use fewer adversarial tokens (i.e., tokens optimized to maximize the similarity between the query and the poisoned document). As shown in Table 11, reducing the number of adversarial tokens improves stealthiness, but it comes at the cost of decreased retrieval success for poisoned documents. Notably, PoisonedRAG remains effective even with single-token optimization. However, this is primarily due to its narrow target scope, which targets specific queries and relies on naturally crafted false information as poisoned documents, rather than successful optimization. It is worth noting that despite the target attack of GMTP being optimization-based, it is able to successfully filter out up to 70% of the actual retrieved poisoned documents (i.e., precisely detect even a single unnatural token).

J MLM-Agnostic Performance

To further demonstrate GMTP’s generalization across different architectures, we report results using different MLM (i.e., RoBERTa (Liu et al., 2019)) instead of BERT. RoBERTa is known to use different pretraining method and dataset compared to BERT, which can lead to variations in performance. However, as shown Table 12, GMTP remains robust across different LLMs, as it consistently achieves high filtering rates while maintaining minimal nDCG@10 degradation. These results demonstrate that GMTP is effective regardless of MLM variations. Note that we used DPR for the experiments.

K Full Experimental Results

In this section we show the results using Contriever. Specifically, we report main results comparing with
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Adv. tokens</th>
    <th colspan="3">NQ</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">MS MARCO</th>
  </tr>
  <tr>
    <th>nDCG-P</th>
    <th>FR</th>
    <th>Poison</th>
    <th>nDCG-P</th>
    <th>FR</th>
    <th>Poison</th>
    <th>nDCG-P</th>
    <th>FR</th>
    <th>Poison</th>
  </tr>
  <tr>
    <td rowspan="3">PoisonedRAG</td>
    <td>1</td>
    <td>0.457</td>
    <td>0.504</td>
    <td>282</td>
    <td>0.261</td>
    <td>0.702</td>
    <td>396</td>
    <td>0.135</td>
    <td>0.571</td>
    <td>177</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.472</td>
    <td>0.963</td>
    <td>571</td>
    <td>0.28</td>
    <td>0.992</td>
    <td>781</td>
    <td>0.13</td>
    <td>0.985</td>
    <td>535</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.476</td>
    <td>0.999</td>
    <td>678</td>
    <td>0.28</td>
    <td>0.998</td>
    <td>833</td>
    <td>0.132</td>
    <td>0.999</td>
    <td>693</td>
  </tr>
  <tr>
    <td rowspan="3">Phantom</td>
    <td>1</td>
    <td>0.385</td>
    <td>1.0</td>
    <td>0</td>
    <td>0.212</td>
    <td>1.0</td>
    <td>0</td>
    <td>0.107</td>
    <td>1.0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.387</td>
    <td>1.0</td>
    <td>1</td>
    <td>0.216</td>
    <td>1.0</td>
    <td>4</td>
    <td>0.102</td>
    <td>1.0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.388</td>
    <td>1.0</td>
    <td>5</td>
    <td>0.222</td>
    <td>1.0</td>
    <td>11</td>
    <td>0.086</td>
    <td>1.0</td>
    <td>2</td>
  </tr>
  <tr>
    <td rowspan="3">AdvDecoding</td>
    <td>1</td>
    <td>0.389</td>
    <td>1.0</td>
    <td>0</td>
    <td>0.226</td>
    <td>1.0</td>
    <td>0</td>
    <td>0.114</td>
    <td>1.0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.389</td>
    <td>1.0</td>
    <td>2</td>
    <td>0.226</td>
    <td>1.0</td>
    <td>1</td>
    <td>0.114</td>
    <td>1.0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.389</td>
    <td>1.0</td>
    <td>6</td>
    <td>0.226</td>
    <td>1.0</td>
    <td>3</td>
    <td>0.114</td>
    <td>1.0</td>
    <td>5</td>
  </tr>
</table>

Table 11: Performance with varying numbers of adversarial tokens. nDCG-P denotes nDCG@10 under the attack setting, while Poison indicates the number of poisoned documents retrieved in the top-10.

<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th colspan="3">NQ</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">MS MARCO</th>
  </tr>
  <tr>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
    <th>nDCG@10 Clean (↑)</th>
    <th>Poison (↑)</th>
    <th>FR (↑)</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>0.493</td>
    <td>0.491</td>
    <td>1.0</td>
    <td>0.298</td>
    <td>0.298</td>
    <td>1.0</td>
    <td>0.176</td>
    <td>0.174</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>0.385</td>
    <td>0.385</td>
    <td>1.0</td>
    <td>0.236</td>
    <td>0.236</td>
    <td>1.0</td>
    <td>0.152</td>
    <td>0.152</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>0.385</td>
    <td>0.385</td>
    <td>1.0</td>
    <td>0.236</td>
    <td>0.236</td>
    <td>1.0</td>
    <td>0.152</td>
    <td>0.152</td>
    <td>1.0</td>
  </tr>
</table>

Table 12: Performance of GMTP using RoBERTa as the MLM.

baselines in Table 6. Figure 10 and Figure 11 illustrates the density plot across datasets and attack methods, using DPR and Contriever, respectively. Performance across various \( \lambda \) values is described in Figure 12 and Figure 13. Lastly, the masking precision using Contriever is reported in Table 13.

<table>
  <tr>
    <th>Attack</th>
    <th>NQ</th>
    <th>HotpotQA</th>
    <th>MS MARCO</th>
  </tr>
  <tr>
    <td>PoisonedRAG</td>
    <td>0.915</td>
    <td>0.911</td>
    <td>0.925</td>
  </tr>
  <tr>
    <td>Phantom</td>
    <td>0.971</td>
    <td>0.893</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>AdvDecoding</td>
    <td>1.0</td>
    <td>0.943</td>
    <td>0.954</td>
  </tr>
</table>

Table 13: GMTP precision in detecting cheating tokens using Contriever.
Figure 10: Density plot of masked token probability using DPR.

Figure 11: Density plot of masked token probability using Contriever.
Figure 12: nDCG@10 and filtering rate across various \( \lambda \) values using DPR.

Figure 13: nDCG@10 and filtering rate across various \( \lambda \) values using Contriever.

Reviewer paper summaries: This paper introduces a new method GMTP to filter out poisoned documents in RAG systems. It first utilize retrieval model's gradient to identify important tokens in each relevant document. Then it applies MLM over all relevant documents and rules out documents with lower average MLM probability as poisoned. Experiments show that GMTP is able to reduce the retrieval of poisoned documents while maintaining a decent generation performance. However, it is unclear whether it can be applied for MLM-based attack, and whether GMTP is really efficient due to excessive MLM calls. Additional experiments and analysis can make claims in this paper stronger.

This paper explores a security vulnerability in RAG systems, highlighting how adversaries can poison retrieval corpora by injecting crafted documents that steer Language Models towards harmful or misleading outputs. To address this issue, this paper proposes the Gradient-based Masked Token Probability (GMTP) method. GMTP leverages gradient analysis from the retriever's similarity function to identify tokens that disproportionately influence retrieval similarity scores. These tokens (potentially adversarial) are masked, and their probabilities are calculated using a Masked Language Model.
Summary of strengths: - Experiments show that GMTP reduces the retrieval of poisoned documents and maintains a decent generation performance.
- Detailed analysis and ablation on GMTP.
- Clear and straightforward writing, the text is easy to follow.

1.	The proposed GMTP is an innovative and practical approach to a significant and realistic security threat in RAG pipelines.
2.	The gradient-based identification of key tokens and subsequent masked-token probability analysis are methodologically well-founded, clearly presented, and intuitive.
3.	Experiments on three datasets and multiple attack methods validate GMTP's effectiveness and robustness.
Summary of weaknesses: - Using MLM to identify poisoned tokens might be limited by the MLM model (e.g. BERT). What if the attacker uses BERT instead of hotflip to create poisoned documents? There are various MLM-based adversarial example generation methods, such as [1], [2]. It's unclear whether GMTP would work with more "in-domain" attacks.
- Although the authors claim that GMTP is more efficient for generator in RAG system, the adversarial token identification step using MLM require applying MLM over all suspicious tokens, which adds computation overhead at the retrieving step. For each search query, search top-k documents, with each document in average N tokens from gradient selection, it would require k x N forward passes (in this paper, 10 x 10 = 100 passes) over BERT. The authors didn't provide the actual inference FLOPS to show GMTP is more efficient, thus I'm not convinced by this claim.
- Reproducibility. Will the authors open-source the code implementation?

[1]: BAE: BERT-based Adversarial Examples for Text Classification
[2]: BERT-ATTACK: Adversarial Attack Against BERT Using BERT

1.	The effectiveness of GMTP relies on the threshold parameter (τ), which is heuristically selected based on a sample of domain-specific documents. Although the authors evaluate sensitivity to the threshold parameter λ, it remains somewhat heuristic and could vary significantly across vastly different domains or retrieval setups.
2.	The experiments rely on automated evaluation (LLM-as-a-judge), which might not fully capture nuanced variations in answer correctness or subtle poisoning effects. Human validation on a small scale would strengthen it in the evaluation's reliability.
3.	The current experiments only employ BERT-based Contriever and DPR as retrievers, and Llama2-7B-Chat as the generator. Given the rapid advancements in retrieval and generation models, it would be beneficial to evaluate GMTP's effectiveness on multiple more recent models.
Comments, suggestions and typos: - In Table 1, Why are the ACC larger than CACC? Even though the poisoned documents can be removed, the clean accuracy should be an upperbound (?) Similarly, why is nDCG under poison environment higher than clean environment?
- In Table 1 - generation, "Naive" baseline achieves 37.5 clean accuracy (Hotpot QA with Phantom), but surprisingly 60.0 poisoned accuracy, which is much higher than its clean performance, and even higher than GMTP under the same setting. The same thing can be found with MS MARCO with AdvDecoding attack. Any explanation? 
- In Table 3, why there is such a big difference with avg token probability between relevant doc and other clean doc (NQ)? Could that be a hint for more frequently searched relevant documentation?

1.	Explicitly test or discuss GMTP's performance against subtler, non-optimized adversarial documents. Such scenarios might reveal limitations in GMTP's reliance on unnatural linguistic patterns.
2.	Add discussions or experiments highlighting how the choice of MLM (size, domain specificity, training data) influences GMTP's performance.
3.	Provide explicit computational overhead analysis compared to existing methods.
4.	Section 6.1 (Line 445): Typo "whil" → "while".
Overall assessment scores: 3

3
Metareview score: 3
Metareview: This paper proposes GMTP (Gradient-based Masked Token Probability), a defense framework targeting document poisoning attacks in RAG systems. The core idea is to first identify high-impact tokens through gradients from the retriever’s similarity function and then evaluate these tokens using masked token prediction to flag unnatural (potentially adversarial) content. The framework is lightweight, model-agnostic to some extent, and achieves high filtering rates across multiple retrieval models and datasets while maintaining strong retrieval and generation performance.
Summary of reasons to publish: * GMTP introduces a principled, two-stage detection mechanism combining gradient attribution with masked token probabilities, achieving high filtering rates across multiple retrieval and generation setups.
* The paper includes diverse retrievers (DPR, Contriever, ColBERT), attack types, and efficiency tests. Additional reviewer-suggested experiments were conducted during discussion.
* GMTP is lightweight, requires no retraining, and works across retrievers, making it easy to deploy in real-world RAG systems.