Venue: EMNLP 2023

# Poisoning Retrieval Corpora by Injecting Adversarial Passages

 Zexuan Zhong\({}^{*}\), Ziqing Huang\({}^{**}\), Alexander Wettig\({}^{\dagger}\), Danqi Chen\({}^{\dagger}\)

\({}^{\dagger}\)Princeton University \({}^{\ddagger}\)Tsinghua University

{zzhong,awettig,danqic}@cs.princeton.edu hzq0@1227@gmail.com

ZZ and ZH contributed equally. The work was done when ZH interned at Princeton University.

###### Abstract

Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but to what extent can they be safely deployed in real-world applications? In this work, we propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages by perturbing discrete tokens to maximize similarity with a provided set of training queries. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems to retrieve them for queries that were not seen by the attacker. More surprisingly, these adversarial passages can directly generalize to out-domain queries and corpora with a high success attack rate--for instance, we find that 50 generated passages optimized on Natural Questions can mislead >94% of questions posed in financial documents or online forums. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised. Although different systems exhibit varying levels of vulnerability, we show they can all be successfully attacked by injecting up to 500 passages, a small fraction compared to a retrieval corpus of millions of passages.1

Footnote 1: Our code is publicly available at [https://github.com/](https://github.com/) princeton-nlp/corpus-poisoning.

+
Footnote †: _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13764–13775

## 1 Introduction

Dense retrievers have become increasingly popular Karpukhin et al. (2020); Xiong et al. (2021); Izacard et al. (2022); Khattab and Zaharia (2020), and outperform traditional lexical methods in a range of information retrieval tasks. However, recent evidence shows that they may still struggle with long-tail entities Sciavolino et al. (2021) or out-of-domain generalization Thakur et al. (2021), and it remains unclear to what extent we can safely deploy dense retrievers in real-world applications.

In this work, we reveal a new type of vulnerability in dense retrievers. We present a novel _corpus poisoning_ attack, where a malicious user can inject a small fraction of adversarial passages to a retrieval corpus, with the aim of fooling systems into returning them among the top retrieved results. While previous work has shown that an adversarial passage can be crafted for individual queries Song et al. (2020); Raval and Verma (2020); Song et al. (2022); Liu et al. (2022), we consider a stronger setting, where an adversarial passage must be retrieved for a broad set of user queries, and can even generalize to unseen queries either in-domain or out-of-domain. This setting is realistic for open-access platforms, such as Wikipedia or Reddit, and such attacks may become a new tool for black hat SEO (Search Engine Optimization) Patil Swati et al. (2013). Adversarial passages deteriorate the user experience of retrieval systems and can cause societal harm, e.g., if inserted passages contain misinformation.

Our attack is a gradient-based method, inspired by the HotFlip method Ebrahimi et al. (2018), which starts from a natural-language passage and

Figure 1: Our proposed _corpus poisoning_ attack. Malicious users generate adversarial passages and inject them into a retrieval corpus to mislead dense retrievers to return them as responses to user queries. The attack is highly effective on unseen queries either in-domain or out-of-domain.



iteratively perturbs it in the discrete token space to maximize its similarity to a set of training queries. We also use a simple clustering method to extend this method to generate multiple passages.

We evaluate our method on state-of-the-art dense retrieval models and demonstrate that this corpus poisoning attack can be highly effective with a small number of adversarial passages. Noticeably, we find that unsupervised Contriever models are particularly vulnerable, and even 10 passages can fool more than \(90\%\) of queries. Supervised retrievers (e.g., DPR, ANCE) seem to be harder to attack, but the success rate still exceeds 50% when we inject up to 500 passages (<0.02% compared to the retrieval corpus). Our attack is also effective at attacking multi-vector retrievers (e.g., ColBERT), which are less sensitive to single-token changes. Moreover, we find that these adversarial passages can be directly injected into out-of-domain retrieval corpora and achieve a high attack rate for unseen queries posed on these domains (e.g., financial documents or online forums), as illustrated in Figure 1.

We conclude our paper with a case study of real-world threats when the inserted text includes targeted misinformation. We hope that our findings will caution practitioners to think about the safety concerns of these dense retrieval systems, and further improve their robustness.

## 2 Related Work

In NLP, adversarial attacks have been studied on many classification tasks Liang et al. (2018); Li et al. (2020); Morris et al. (2020); Wang et al. (2021) and question-answering tasks Jia and Liang (2017). The objective is to find small, semantic-preserving perturbations to the input that cause the model to make erroneous predictions.

Adversarial attacks on retrieval systems have been limited to making small edits to passages to change their retrieval rank for an individual or a small set of queries Song et al. (2020); Raval and Verma (2020); Song et al. (2022), and the success rate of the attack is evaluated per perturbed passage. In corpus poisoning, we inject new passages into the retrieval corpus and measure the attack success by the _overall performance_ of the retrieval system when evaluated on _unseen_ queries.

Corpus poisoning differs from data poisoning Chen et al. (2017); Schuster et al. (2020), since changes are only made to the retrieval corpus rather than the training data, and the retriever remains unchanged. It should be noted that the attack of image retrieval systems has been explored in previous work Xiao and Wang (2021); Tolias et al. (2019), but those methods are not transferable to the text domain, where the optimization problem needs to be solved over discrete tokens, rather than continuous values of pixels.

Our work is related to universal adversarial triggers Wallace et al. (2019); Song et al. (2021), which attack text classifiers by prepending the same phrase to any input sequence. However, in retrieval, we can change the system's output by injecting new passages, without having to alter every passage or user query, making it a more realistic setting for real-world attacks.

## 3 Method

Problem definitionWe consider a passage retrieval problem: the retrieval model takes a user query \(q\) and retrieves passages from a corpus \({\cal C}\), consisting of a set of passages \(\{p_{1},p_{2},\ldots p_{|{\cal C}|}\}\). We mainly focus on dual-encoder dense retrievers that embed queries and passages using a query encoder \(E_{q}(\cdot)\) and a passage encoder \(E_{p}(\cdot)\) respectively, and compute their similarity using the inner product of the embeddings: sim\((q,p)=E_{q}(q)^{\top}E_{p}(p)\). Additionally, in Section 4.4 we study our attack on multi-vector dense retrievers (e.g., ColBERT Khattab and Zaharia (2020)), where queries and passages are represented using multiple vectors.

The query and passage encoders are typically trained with a contrastive-learning objective, either from supervised question-passage pairs in the case of DPR Karpukhin et al. (2020) and ANCE Xiong et al. (2021), or from unsupervised data in the case of Contriever Izacard et al. (2022). During inference, the dense retrieval models perform a nearest-neighbor search given a query \(q\) to return the top-\(k\) most similar passages in the corpus as retrieval results.

Corpus poisoning attackIn our corpus poisoning attack, a set of adversarial passages are generated and inserted into the retrieval corpus \({\cal C}\), with the intention of misleading dense retrieval models. Once the corpus has been poisoned, it is expected that dense retrieval models will retrieve the adversarial passages in response to certain queries. Specifically, we aim to generate a small set of adversarial passages \({\cal A}=\{a_{1},a_{2},\dots,a_{|{\cal A}|}\}\), where the size of the adversarial passage set is much smaller than the original corpus, i.e., \(|{\cal A}|\ll|{\cal C}|\). The objec 

tive is to construct \({\cal A}\) such that at least one adversarial passage is featured in the top-\(k\) results for the distribution of queries \({\cal Q}\). In practice, we assume a training set from \({\cal Q}\) available to construct \({\cal A}\) and expect it to generalize to a held-out test set.

OptimizationGiven a set of queries \({\cal Q}=\{q_{1},q_{2},\ldots,q_{|{\cal Q}|}\}\), we aim to generate a set of adversarial passages \({\cal A}\) which are ranked high by the model in the retrieval results. To mislead the model, we would like to find a sequence of tokens \(a=[t_{1},t_{2},\ldots]\) which maximizes the similarity to a set of queries:

\[a=\operatorname*{arg\,max}_{a^{\prime}}\frac{1}{|{\cal Q}|}\sum_{q_{i}\in{ \cal Q}}E_{q}(q_{i})^{\top}E_{p}(a^{\prime}). \tag{1}\]

We discuss how to generate a single passage first and extend it to generate multiple passages.

We use a gradient-based approach to solve the optimization problem, inspired by the HotFlip method Ebrahimi et al. (2018); Wallace et al. (2019), which approximates the model output change of replacing a token. We initialize the adversarial passage using a random passage from the corpus. At each step, we randomly select a token \(t_{i}\) in \(a\) and compute an approximation of the model output if replacing \(t_{i}\) with another token \(t_{i}^{\prime}\). We employ HotFlip to efficiently compute this approximation using gradients: \(e_{t_{i}^{\prime}}^{\top}\nabla_{e_{t_{i}}}\text{sim}(q,a)\), where \(\nabla_{e_{t_{i}}}\text{sim}(q,a)\) is the gradient vector with respect to the token embedding \(e_{t_{i}}\). Given a query set \({\cal Q}\), the best replacement candidates for the token \(t_{i}\) can be obtained by selecting the token that maximizes the output approximation:

\[\operatorname*{arg\,max}_{t_{i}^{\prime}\in{\cal V}}\frac{1}{|{\cal Q}|}\sum_ {q\in{\cal Q}}e_{t_{i}^{\prime}}^{\top}\nabla_{e_{t_{i}}}\text{sim}(q,a), \tag{2}\]

where \({\cal V}\) is the vocabulary. Note that this operation is cheap as it only requires a single multiplication of the embedding matrix and the gradient vector.

Generating multiple adversarial passagesWe extend our attack to generate multiple adversarial passages. Our approach is based on a simple idea of grouping similar queries and generating one adversarial passage for each group of queries. To achieve this, we use the \(k\)-means clustering algorithm to cluster queries based on their embeddings \(E_{q}(q_{i})\). Once the queries are grouped into clusters, we generate one adversarial passage for each cluster by solving the optimization problem as shown in Eq. 1. This allows us to generate multiple adversarial passages in parallel, each targeting a group of similar queries.

## 4 Experiments

### Setup

Retrieval datasetsWe experiment with the BEIR benchmark Thakur et al. (2021), a comprehensive suite of retrieval datasets. We focus on two popular datasets, Natural Questions (NQ) Kwiatkowski et al. (2019) and MS MARCO Nguyen et al. (2016), and evaluate our attack on the held-out test queries of these two datasets, as well as its transferability to seven unseen domains (e.g., Quora, scientific, financial documents) in BEIR. See Appendix A for details.

Dense retrieversIn our main experiments, we conduct attacks on five state-of-the-art dense retrieval models: Contriever Izacard et al. (2022) (pre-trained) and Contriever-ms (fine-tuned on MS MARCO), DPR-nq Karpukhin et al. (2020) (trained on NQ) and DPR-mul (trained on multiple datasets), and ANCE Xiong et al. (2021). Additionally, we apply the attack on a multi-vector dense retriever, ColBERT Khattab and Zaharia (2020), to study the effectiveness of our attack on different architectures.

Evaluation metricsWe generate adversarial passages on the training set (NQ or MS MARCO) and inject them into the corpus and measure the top-\(k\)_attack success rates_ on test queries, defined as the percentage of queries for which at least one adversarial passage is retrieved in the top-\(k\) results. A higher success rate indicates that a model is more vulnerable to corpus poisoning attacks. All the adversarial passages contain \(50\) tokens, and we perform the token replacement for \(5000\) steps. For more implementation details, see Appendix B.

### Attacks on In-Domain Queries

Figure 2 (left) shows the in-domain attack results on different models in NQ and MS MARCO. We find that the pre-trained Contriever model Izacard et al. (2022) is very vulnerable to our attack--even with only one adversarial passage, our attack fools the model on more than 75% of queries on NQ and MS MARCO. Although the supervised retrieval models (e.g., DPR-nq) appear harder to attack, the attack success rates can be substantially improved by generating more adversarial passages. In Figure 2 (right), we show that when we generate 500



adversarial passages, all dense retrieval models are fooled on at least \(50\%\) of test queries.2 Our results clearly demonstrate that dense retrieval models are significantly vulnerable to our proposed attack and fail on a large number of queries with only a small fraction of adversarial passages.

Footnote 2: Due to the limited computing resources, we use \(|{\cal A}|=50\) in most of the experiments in the rest of paper.

### Attacks Transfer Out-of-Domain

We test whether the generated adversarial passages can transfer across different domains and retrieval tasks. We perform the attack on the Contriever model with the training sets of NQ and MS MARCO, and insert the adversarial passages into the corpora of other retrieval tasks. Table 1 shows the transfer attack results. Surprisingly, we find that the generated adversarial passages can transfer effectively to mislead the models on other retrieval domains/datasets (e.g., the attack success rate is \(94.1\%\) on FiQA, a financial domain question-answering task). This implies a potential for practical deployment of the attack that is effective in different domains.

### Attacks on Multi-Vector Retrievers

In Section 4.2, we conduct attacks on state-of-the-art dense retrieval models that map a query or passage to a single vector representation. Here, we study the effectiveness of our attack on multi-vector retrieivers, where a piece of text corresponds to multiple vectors. We conduct our attack on ColBERT Khattab and Zaharia (2020)ColBERT-v1, pre-trained on MS MARCO) using the training queries from NQ with \(\{1,10,50\}\) adversarial passages. Table 2 shows the attack performance on ColBERT on in-domain NQ queries. Interestingly, we find that generating longer adversarial passages (250 tokens) is much more effective in attacking ColBERT than using generating short ones (50 tokens). We think this is because the similarity function is computed with multiple embeddings in multi-vector dense retrieval models (e.g., SumMax in ColBERT), thus it is less sensitive to single token changes. We find that inserting 50 adversarial passages with 250 tokens fools ColBERT on \(20.1\%\) NQ test queries, suggesting that our attack is still effective in attacking multi-vector retrieval models.

Figure 2: Top-\(20\) success rate of our attack. Left: We generate \(|{\cal A}|\in\{1,10,50\}\) adversarial passages on the train sets of NQ or MS MARCO and evaluate on their held-out test queries. Right: On NQ, we show that the attack success rate can be substantially improved by generating 500 passages. Contirever: the pre-trained Contriever model Izacard et al. (2022), Contriever-ms: Contriever fine-tuned on MS MARCO, DPR-nq: DPR trained on NJPukhin et al. (2020), DPR-mul: DPR trained on a combination of datasets. ANCE Xiong et al. (2021).

\begin{table}
\begin{tabular}{l c|c c} \hline \hline  & **NQ** & **MS** \\   \[|{\cal A}|=\] & 1 & 10 & 50 \\ \hline \multicolumn{3}{l}{**Model**} \\ Contriever & **84.2** & **98.1** & **99.4** & **75.2** & **92.2** & **98.6** \\ Contriever-ms & 0.5 & 52.5 & 80.9 & 2.4 & 20.9 & 34.9 \\ DPR-nq & 0.0 & 3.8 & 18.8 & 0.1 & 2.6 & 13.9 \\ DPR-mul & 0.0 & 10.6 & 28.3 & 0.0 & 4.7 & 16.3 \\ ANCE & 1.0 & 14.7 & 34.3 & 0.0 & 2.3 & 11.6 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Transferring top-\(20\) success rate of our attack with \(|{\cal A}|=50\) adversarial passages. We conduct the attack on Contriever with NQ or MS MARCO (MS) training sets and evaluate the attack on different test sets of the BEIR benchmark Thakur et al. (2021). See Appendix A for more details of each dataset.

 



### Targeted Attack with Misinformation

Finally, we consider a case study, in which we examine the possibility of generating adversarial passages that include targeted misinformation. To accomplish this, we initialize the adversarial passages with "OUR OWN GOVERNMENT IS LITERALLY POISONING US WITH CHEMTRAILS" followed by [MASK] tokens and keep the prefix fixed during optimization. We experiment with the Contriever model using the NQ dataset by generating \(|{\cal A}|\in\{1,10\}\) of such adversarial passages. We find the attack remains effective under this constrained setup, although the success rates drop, reaching \(11.7\%\) and \(59.6\%\) with \(1\) and \(10\) passages, respectively. This suggests users would see the targeted message after just two or three queries, demonstrating the potential for spreading harmful information this way.

## 5 Additional Analysis and Discussion

Attacks are not model-agnosticWe study whether the generated adversarial passages transfer across different models. We find limited transferability (i.e., success rates < \(0.5\%\)) across similar models (e.g., Contriever to Contriever-ms) and no transferability across different model families. These results suggest that it would be challenging to leverage transferability for black-box attacks, a common strategy against real-world systems with unknown model weights Papernot et al. (2016).

Lengths of adversarial passagesWe study the effects of adversarial passage lengths on the attack performance. We attack Contirever-ms using the training data from NQ with \(50\) adversarial passages of length \(\{10,20,50,100\}\). As Table 3 shows, as long as the adversarial passages contain a sufficient number of tokens (i.e., \(\geq 20\)), the attack performance is not affected much when further increasing the passage length.

Initialization of adversarial passagesIn our attack, we initialize the adversarial passage using a random passage from the corpus. Here, we study the importance of initializing the adversarial passages using natural text. We initialize the adversarial passages with 50 [MASK] tokens, the attack success rate drops from 98.1% to 95.8% when attacking Contrriever on NQ with 10 adversarial passages. This shows that the attack is still effective with a nonsense initial text but using natural passages would make the attack more effective.

Unnaturalness of adversarial passagesOur attack results in unnatural passages (see examples in Appendix E). This can be leveraged to formulate simple defenses, as we show in Appendix C. However, we note that attackers may develop stronger attacks which produce fluent text that can still fool models, as demonstrated by Song et al. (2021).

Attacks on sparse retrievers (BM25)We study whether the proposed attack is effective at sparse lexical retrieval models such as BM25. Specifically, we attack BM25 by maximizing the BM25 similarity of the adversarial passage to a group of queries (using Contriever embeddings in \(k\)-means). We find that the adversarial passage to a query group contains only repeated tokens, such as a passage consisting of 50 tokens "what". We generate and insert 50 adversarial passages and find that BM25 retrieval performance does not change at all (i.e., attack success rate = 0%).

Next, we study whether the adversarial passages for Contriever can transfer to BM25. We find that on the NQ dataset with 50 adversarial passages, 0% of queries are fooled by the Contriever adversarial passages, when we use BM25 to retrieve. These results suggest that sparse retrieval models are robust to the proposed corpus poisoning attack.

Transferability does not simply come from train-test overlapIn Section 4.3, we show that the adversarial passages can transfer across different domains/datasets. In Appendix D, we study different transfer attack strategies and show that the transferability does not simply come from the overlap between training queries and test queries.

## 6 Conclusions

We proposed a new attack for dense retrievers, in which adversarial passages are inserted into the corpus to mislead their retrieval outputs. We show that even a small number of adversarial passages can successfully attack state-of-the-art dense retrievers and generalize to queries from unseen domains. These findings have important implications for the future deployment of robust retrieval systems in real-world applications.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Length (tokens)** & **10** & **20** & **50** & **100** \\ \hline Top-20 success rate & 63.5 & 81.6 & 80.9 & 79.2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top-\(20\) success rate of our attack with Contriever-ms on the test queries of NQ when using different lengths of adversarial passages. We generate \(50\) adversarial passages using the train set of NQ.





### Limitations

The limitations of our research are as follows:

* Despite the effectiveness, our approach can be computationally expensive to carry out. For each group of queries, we need to run our attack independently on a single GPU to generate one adversarial passage. For future work, it is worth considering the exploration of more efficient methods for generating multiple adversarial passages, which may serve to mitigate the computational costs of our approach.
* Our study focuses on corpus poisoning attacks to investigate whether dense retrieval models can be deployed safely. We acknowledge that there exist other potential avenues of attack against dense retrieval models, such as maliciously manipulating user queries. Our investigation specifically centers on corpus poisoning attacks and their implications for the vulnerability of dense retrieval models.

## Ethical Considerations

Our research studies the vulnerability of dense retrieval models, which are widely used techniques in the industry. The results of our experiments indicate that malicious users can successfully perform the proposed attack, resulting in a high rate of misleading a dense retrieval model. Although we propose potential defenses in Section 5, it is important to note that the proposed attack has the potential for misuse, allowing for the spread of toxic information. Future research based on this attack should exercise caution and consider the potential consequences of any proposed method. Furthermore, we point out that we include a small piece of misinformation in the paper (the prefix in Section 4.5) to highlight the harmful potential of corpus poisoning.

## Acknowledgements

We thank Dan Friedman, Victoria Graf, and Yangsibo Huang for providing helpful feedback. This research is supported by the "Dynabench Data Collection and Benchmarking Platform" award from Meta AI. ZZ is supported by a JP Morgan Ph.D. Fellowship.

## References

* C. Liu, B. Li, K. Lu, and D. Song (2017-06)Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526. Cited by: SS3.
* A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. Weld (2020)SPECTER: document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2270-2282. External Links: Link, Document Cited by: SS1, SS2.
* J. Ebrahimi, A. Rao, D. Lowd, and D. Dou (2018)HotFlip: white-box Adversarial Examples for Text Classification. In Association for Computational Linguistics (ACL), Cited by: SS1, SS2.
* G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave (2022)Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research. Cited by: SS1, SS2.
* R. Jia and P. Liang (2017)Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 2021-2031. External Links: Link, Document Cited by: SS1, SS2.
* V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020)Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 6769-6781. External Links: Link, Document Cited by: SS1, SS2.
* O. Khattab and M. Zaharia (2020)Colbert: efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, Cited by: SS1, SS2.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 452-466. Cited by: SS1, SS2.
* L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu (2020)BERT-ATTACK: adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 6193-6202. External Links: Link, Document Cited by: SS1, SS2.



Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In _International Joint Conference on Artificial Intelligence (IJCAI)_.
* Liu et al. (2022) Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song, Changlong Sun, Xiaofeng Wang, Wei Lu, and Xiaozhong Liu. 2022. Order-disorder: Imitation adversarial attacks for black-box neural ranking models. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_, CCS '22, page 2025-2039, New York, NY, USA. Association for Computing Machinery.
* Maia et al. (2018) Macedo Maia, Siegfried Handschuh, Andre Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW'18 open challenge: financial opinion mining and question answering. In _Companion proceedings of the the web conference 2018_.
* Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 119-126, Online. Association for Computational Linguistics.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. In _CoCo@ NIPs_.
* Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. 2016. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. _arXiv preprint arXiv:1605.07277_.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems (NeurIPS)_.
* Swati et al. (2013) P Patil Swati, BV Pawar, and S Patil Ajay. 2013. Search engine optimization: A study. _Research Journal of Computer and Information Technology Sciences_.
* Raval and Verma (2020) Nisarg Raval and Manisha Verma. 2020. One word at a time: adversarial attacks on retrieval models. _arXiv preprint arXiv:2008.02197_.
* Schuster et al. (2020) Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov. 2020. Humpty dumpty: Controlling word meanings via corpus poisoning. In _IEEE symposium on security and privacy (SP)_.
* Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6138-6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Song et al. (2020) Congzheng Song, Alexander Rush, and Vitaly Shmatikov. 2020. Adversarial semantic collisions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4198-4210, Online. Association for Computational Linguistics.
* Song et al. (2022) Junshuai Song, Jiangshan Zhang, Jifeng Zhu, Mengyun Tang, and Yong Yang. 2022. TRAttack: Text rewriting attack against text retrieval. In _Proceedings of the 7th Workshop on Representation Learning for NLP_, pages 191-203, Dublin, Ireland. Association for Computational Linguistics.
* Song et al. (2021) Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. 2021. Universal adversarial attacks with natural triggers for text classification. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3724-3733, Online. Association for Computational Linguistics.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Tolias et al. (2019) Giorgos Tolias, Filip Radenovic, and Ondrej Chum. 2019. Targeted mismatch adversarial attack: Query with a flower to retrieve the tower. In _International Conference on Computer Vision (ICCV)_, pages 5037-5046.
* Voorhees et al. (2021) Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In _ACM SIGIR Forum_, volume 54, pages 1-12. ACM New York, NY, USA.
* Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 241-251, Melbourne, Australia. Association for Computational Linguistics.
* Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In _Proceedings of the 2019 Conference on Empirical Methods 

in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, Hong Kong, China. Association for Computational Linguistics.
* Wang et al. (2021) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial GLUE: A multitask benchmark for robustness evaluation of language models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Cited by: SS4.3.
* Xiao and Wang (2021)You see what i want you to see: exploring targeted black-box transferability attack for hash-based image retrieval systems. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: SS5.
* Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk (2021)Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Machine Learning (ICML), Cited by: SS6.
* Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018)HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2369-2380. External Links: Link, Document Cited by: SS7.

 

Dataset Details

We use several datasets with different tasks and domains from BEIR Thakur et al. (2021), the widely-used benchmark for zero-shot evaluation of information retrieval, to evaluate the transferability of the adversarial passages: Natural Questions Kwiatkowski et al. (2019), MS MARCO Nguyen et al. (2016), HotpotQA Yang et al. (2018), FiQA Maia et al. (2018), Quora, FEVER Thorne et al. (2018), TREC-COVID Voorhees et al. (2021), ArguAna Wachsmuth et al. (2018), and SCIDOCS Cohan et al. (2020). We include more information and data statistics in Table 4.

## Appendix B Implementation Details

We implement our attack based on the PyTorch library Paszke et al. (2019). We set the length of all adversarial passages to be \(50\). During optimization, we perform token replacement for \(5,000\) steps. At each step, we compute the gradients based on a batch of queries \(\mathcal{Q}\) (we set batch size as \(64\)) and randomly select a token \(t_{i}\) in the current adversarial passage to be replaced. We use the gradient-base HotFlip method Ebrahimi et al. (2018) to obtain a set of replacement candidates, which maximize the approximate similarity after replacement:

\[\operatorname*{arg\,max}_{t^{\prime}_{i}\cup\mathcal{V}}\frac{1}{|\mathcal{ Q}|}\sum_{q\in\mathcal{Q}}e_{t^{\prime}_{i}}^{\top}\nabla_{e_{t_{i}}}\text{sim}( q,a). \tag{3}\]

We consider top-\(100\) tokens as potential replacements. We measure the actual similarity changes on each of these candidates and use the token, which maximizes the similarity between the passage and this batch of queries, to replace \(t_{i}\).

## Appendix C Defenses

We notice that the adversarial passages generated by our attack have two distinct characteristics: (1) they contain highly unnatural sequences, and this can be detected by likelihood scoring of an off-the-shelf language model, and (2) their embeddings have very high \(\ell_{2}\)-norms to boost their similarity scores. We study potential approaches to detect adversarial passages.

Filter by likelihoodAdversarial passages contain highly unnatural sequences, see examples in Appendix E. One could easily detect and distinguish the adversarial examples from the corpus passages by scoring the average token log likelihood of each passage with a language model, not to mention spam filters and moderators on real-world platforms. Indeed, Figure 3 shows that an off-the-shelf GPT-2 can separate the corpus passages and adversarial passages almost perfectly. However, future work might show that an adversary can overcome this weakness by incorporating natural language priors into the attack.

Embedding norm clippingThe interaction between queries and passages is strictly limited to the inner product, such that \(\text{sim}(p,q)\propto\left\lVert E_{p}(p)\right\rVert_{2}\cos\theta\), where \(\theta\) is the angle between query and passage embedding. As it is not possible to reduce the angle with respect to a wide set of queries to zero, the attacker should primarily focus on producing passage embeddings with large \(\ell_{2}\)-norms. Indeed, Figure 4 shows that this is the case for the adversarial passages against Contriever. This prompts a second defense, where we clip the embedding norms of the passage at a constant \(\alpha\) for all passages in the retrieval corpus:

\[E_{clip}(p)=E_{p}(p)\frac{\left\lVert E_{p}(p)\right\rVert_{2}}{\min(\left\lVert E _{p}(p)\right\rVert_{2},\alpha)}. \tag{4}\]

Based on the scale observed as in Figure 4, we explore different values of \(\alpha\) in Table 5 (in Appendix C) and find that clipping with \(\alpha=1.75\) reduces the attack success rate by \(99.4\%\) while only hurting the standard retrieval performance by \(6\%\). Despite this promising defense, it remains possible that corpus poisoning for targeted query distributions or more sophisticated architectures, e.g., ColBERT's late interaction mechanism Khattab and Zaharia (2020), may be harder to detect via vector norm anomalies.

Figure 3: Average token log likelihood for Wikipedia passages and 10 corresponding adversarial passages for NQ with Contriever.

 



## Appendix D Transferability Does Not Simply Come from Train-Test Overlap

In Section 4.3, we show that the generated adversarial passages can transfer across different domains/datasets. Here we investigate whether the transferability simply comes from the overlap between training and testing query distributions.

We attack Contriever on the training set of NQ and generate \(50\) adversarial passages. For a given test query \(q\) (in-domain or out-of-domain), we first find similar queries from the training set, and we use the adversarial passages that are most effective on those training queries to attack the current test query \(q\). Specifically, we consider two cases: **(1)** We find the training query \(q^{*}\), which is most similar to test query \(q\) (measured using the query encoder of Contriever). Among 50 adversarial passages, we find the passage \(p^{*}\) which maximizes the similarity to the training query \(q^{*}\). We insert \(p^{*}\) into the corpus as an attack. **(2)** We classify the test

Figure 4: Distribution of \(\ell_{2}\)-norms for embeddings of Wikipedia passages and 10 corresponding adversarial passages for NQ with Contriever.



query \(q\) using the \(k\)-means classifier that is learned when generating adversarial passages. As for each \(k\)-means cluster, we have generated one passage, we simply insert the adversarial passage \(p^{*}\) that is corresponding to the \(k\)-means cluster that \(q\) is classified as.

The results are shown in Table 6. merely using (1) or (2) is not sufficient to preserve the transfer attack rates. For example, the performance on attacking Scidocs drops from \(76.1\%\) to \(25.2\%\), showing that the test queries and training queries are likely drawn from very different domains. We also find that the performance on HotpotQA does not drop a lot, because the distribution is similar to the training set (both NQ and HotpotQA include knowledge-related questions on Wikipedia). These results suggest that the high transfer attack success rates achieved by our approach do not simply come from the overlap between test queries and training queries.

## Appendix E Qualitative Examples of Adversarial Passages

Table 7 shows some qualitative examples of adversarial passages generated by conducting the attack on Contriever [8] with a group of NQ queries clustered by \(k\)-means. We find that although the generated passages are difficult to interpret as human text, they contain some keywords (highlighted in blue) that are highly relevant to the queries. For example, in a group of queries related to basketball, the corresponding adversarial passage includes words like "bulls" (a famous basketball team in the US) and "russell" (a famous basketball player).

 



\begin{table}
\begin{tabular}{l l} \hline \hline
**Clustered Query Examples** & **Adversarial Passage** \\ \hline what is the definition of an nba rookie & oblast exceed continuity improvement scenes cinematograph-who has more nba championships the east or west \\ how many points did lebron have in the finals & robokie russell arbitration outrageous bull mega litter bulls \\ when did dell curry retire from the nba & elk parades scanning outta kr pound posts ; pronounced ive \\  & groove except how tap elite baskets \\ \hline who played guitar in while my guitar gently weeps & drug council deadlineanalysis composition disclosure ability deadlineponexidelity virwash 2018 markogusmpt vam-who played guitar on you really got me & pire tornadolusion children hindwings hurricane premiere county combolife patriarch thru robb rat murders youngest hedge fbi kristintzedel dies suspects overdose [ textiles 23 \\ \hline how many finger lakes are there in new york state & parade peat etc cnxi stra ul flies joo microwave directions ceilings spa good view view more please wi towels \\ how many judges are there in supreme court now strong was the 2011 earthquake in japan & poo sunshine sg tent li slight refrigerator zipper super airfields observation numbers waterfront gard \\ \hline what is dutch elim disease and how does it spread & mini comment combe qui attract hash presbyterian heart boot mma pw led cell é what synth ctric ida pw integrate sound dyed thanked visa eel platt director cell nervous growing rdial wley answer den chain cular coiled thy wing nationalists perceived vati nan textbook attention raI warnings involving \\ \hline what is the richest woman in the us & ? ranking is lowest lica gible what est amount iset internationally quieter automobiles necessary francoise lake draft bonuses mainly type bonuses defeats bonuses ¿ day summary sourced card autobiography suns training malaga bonuses qualifying through player alaska what whose locality surrounding ately receives liest logram whether \\ \hline who played the judge in the good place & female vote vegetables amino sin tama / = wikipedia fuss bian cent magazine ti eco din naire du fill uts brush aster bu oe whose whose remove black vegetables prefer spices main article rs communes guides ... kos actress translation , Maryland actor who \(\sim\) took potassium tests \\ \hline \hline \end{tabular}
\end{table}
Table 7: Qualitative examples of generated adversarial passages. We show the query examples in a group clustered by \(k\)-means from the NQ dataset (left) and the corresponding adversarial passage (right). We highlight the keywords in the adversarial passages that are highly connected to the queries.