Poisoning Retrieval Corpora by Injecting Adversarial Passages
Authors: Zexuan Zhong
Ziqing Huang
Alexander Wettig
Danqi Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Dense Retrieval
Corpus Poisoning
Adversarial Attack
TL;DR: We propose a novel attack for dense retrieval systems, in which a malicious user generates a small number of adversarial passages to mislead dense retrieval models.

Abstract: Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but to what extent can they be safely deployed in real-world applications? In this work, we propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages by perturbing discrete tokens to maximize similarity with a provided set of training queries. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems to retrieve them for queries that were not seen by the attacker. More surprisingly, these adversarial passages can directly generalize to out-of-domain queries and corpora with a high success attack rate --- for instance, we find that 50 generated passages optimized on Natural Questions can mislead >94% of questions posed in financial documents or online forums. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised. Although different systems exhibit varying levels of vulnerability, we show they can all be successfully attacked by injecting up to 500 passages, a small fraction compared to a retrieval corpus of millions of passages.

Poisoning Retrieval Corpora by Injecting Adversarial Passages
Zexuan Zhong Ziqing Huang Alexander Wettig', Danqi Chen
'Princeton University *Tsinghua University
/ princeton. edu
hzq001 1aa001227@@maill com
Abstract
In Domain (Wikipedia)
Out-of-Domain (Finance)
Q User queries
Q User queries
Dense retrievers have achieved state-of-the performance in various information retrieval lasks, but what extent they be safely de- ployed in real-world applications? In this work, we propose a novel attack dense retrieval systems in which malicious generates small number of adversarial passages by per- turbing discrete tokens to maximize similarity with a provided set of training queries. When these adversarial passages are inserted into large retrieval corpus, we show that this attack is highly effective in fooling these systems to retrieve them for queries that were not seen by the attacker. More surprisingly. these adversar- ial passages can directly generalize to out-of- domain queries and corpora with high suc- attack rate-for instance, find that 50 generated passages optimized on Natural Ques- tions can mislead >94% of questions posed in financial documents online forums. We also benchmark and compare a range of state-of- the-art dense retrievers, both unsupervised and supervised. Although different systems exhibit varying levels of vulnerability, we show they can all be successfully attacked by injecting up to 500 passages, a small fraction compared to retrieval corpus of millions of passages."
Retriever
Retriever
3.9% time
A
26M passages
corpus
50 adversarial transfer passages
Figure 1: Our proposed corpus poisoning attack. Ma- licious users generate adversarial passages and inject them into retrieval corpus mislead dense retrievers to return them as responses to user queries. The attack highly effective on unseen queries either in-domain or out-of-domain.
In this work, we reveal new type of vulnerabil- ity in dense retrievers. We present a novel corpus poisoning attack, where malicious user can in- ject small fraction of adversarial passages to retrieval corpus, with the aim of fooling systems into returning them among the top retrieved results. While previous work has shown that an adversar- ial passage can be crafted for individual queries (Song et al., 2020; Raval and Verma, 2020; Song et al., 2022; Liu et al., 2022), we consider a stronger setting, where an adversarial passage must be re- trieved for broad set of user queries, and can even generalize to unseen queries either in-domain out-of-domain. This setting is realistic for open- access platforms, such as Wikipedia or Reddit, and such attacks may become a new tool for black hat SEO (Search Engine Optimization) (Patil Swati et al., 2013). Adversarial passages deteriorate the user experience of retrieval systems and can cause societal harm, e.g., if inserted passages contain misinformation.
1 Introduction
Dense retrievers have become increasingly pop- ular (Karpukhin et al.. 2020; Xiong et al., 2021;
Izacard et al.. 2022; Khattab and Zaharia. 2020), and outperform traditional lexical methods in range of information retrieval tasks. However, re- cent evidence shows that they may still struggle with long-tail entities (Sciavolino et al., 2021) or out-of-domain gencralization (Thakur al., 202 1), and remains unclear to what extent we can safely deploy dense retrievers in real-world applications.
'ZZ ZH contributed equally, work was done when ZHI interned at Princeton University.
Our attack is gradient-based method, inspired by the HotFlip method (Ebrahimi et al., 2018), which starts from a natural- -angual- passage and
OOr code is publicly available at https: http::/ggtttubocco com/ princeton-nlp/ wwwpcorpussccccoeeooooooppppppup http://opppppposningggooo poisoning.

---

iteratively perturbs it in the discrete token space to maximize its similarity to a set of training queries. We also use simple clustering method to extend this method to generate multiple passages.
unchanged. It should be noted that the attack of image retrieval systems has been explored in pre- vious work (Xiao and Wang, 2021; Tolias et al., 2019), but those methods are not transferable to the text domain, where the optimization problem needs to be solved over discrete tokens, rather than continuous values of pixels.
We evaluate our method on ffte-of-theeane dense retrieval models and demonstrate that this corpus poisoning attack be highly effective with small number of adversarial passages. Noticeably. we find that unsupervised Contriever models are particularly vulncrable, and even 10 passages can fool more than 90% of queries. Supervised retriev- ers (e.g.. DPR, ANCE) seem to be harder to attack, but the success rate still exceeds 50% when we inject up to 500 passages (<0.02% compared to the retrieval corpus). Our attack is also effective attacking multi-vector retrievers (e.g., ColBERT), which are less sensitive to single-token changes. Moreover, we find that these adversarial passages can be directly injected into out-of-domain retrieval corpora and achieve a high attack rate for unseen queries posed on these domains (e.g., financial doc- uments or online forums), as illustrated in Figure 1,
Our work is related to universal adversarial trig- gers (Wallace al., 2019; Song et al., 2021), which attack text classifiers by prepending the same phrase to any imput sequence. However, in retrieval, we can change the system's 's output by injecting new passages, without having to alter every passage or user query, making it more realistic setting for real-world attacks.
3 Method
Problem definition We consider a passage re- trieval problem: the retrieval model takes a user query and retrieves passages from corpus C, con- sisting of a set of passages P2,.2,... Plg) We mainly focus on dual-encoder dense retrievers that embed queries and passages using a query encoder Ei(O)) (+) and passage encoder Ep(.) respectively, and compute their similarity using the inner prod- uet of the embeddings: sim (g. p) (q) (D) Additionally, in Section 4.4 we study our attack on multi-vector dense retrievers (e.g.. COIBERT (Khat- tab and Zaharia, 2020)), where queries and pas- sages are represented using multiple vectors.
We conclude our paper with a case study of real- world threats when the inserted text includes tar- geted misinformation. We hope that our findings will caution practitioners to think about the safety concerns of these dense retrieval systems, and fur- ther improve their robustness.
2 Related Work
The query and passage encoders are typically trained with contrastive- objective, ei- ther from supervised question-passage pairs in the case of DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2021), or from unsupervised data the case of Contriever (Izacard et al., 2022). Dur- ing inference, the dense retrieval models perform a nearest-neighbor search given query q to return the top-k most similar passages in the corpus retrieval results.
In NLP, adversarial attacks have been studied on many classification tasks (Liang et al., 2018; Li
et al., 2020; Morris et al., 2020; Wang et al., 2021)
and question-ansswrrig tasks (Jia and Liang, 2017). The objective is to find small, semantic- preservin perturbations to the input that cause the model to make erroneous predictions.
Adversarial attacks on retrieval systems have been limited to making small edits to passages to change their retrieval rank for individual small set of queries (Song et al., 2020; Raval and Verma, 2020: Song et al., 2022), and the success rate of the attack is evaluated per perturbed passage, In corpus poisoning, we inject new passages into the retrieval corpus and measure the attack success by the overall performance of the retrieval system when evaluated on unseen queries.
Corpus poisoning attack In our corpus poison- ing attack, a set of adversarial passages are gener- ated and inserted into the retrieval corpus C, with the intention of misleading dense retrieval mod- els. Once the corpus has been poisoned, it is ex- pected that dense retrieval models will retrieve the adversarial passages in response to certain queries. Specifically, we aim generate small set of adver- sarial passages A {a1, {a1,02. a2. ..., a where the size of the adversarial passage set is much smaller
Corpus poisoning differs from data poisoning (Chen et al., 2017; Schuster et al., 2020), since changes are only made to the retrieval corpus rather than the training data, and the retriever remains
than the original corpus, i.e., |A| << (C). The objec-

---

tive is construct A such that least one adver- sarial passage is featured in the top-k results for the distribution of queries Q. In practice, we assume a training set from Q available to construct A and expect to generalize to a held-out test set.
in Eq. This allows us generate multiple adver- sarial passages in parallel, each targeting group of similar queries.
Experiments
4.1 Setup
Optimization Given a set of queries Q {qi, (2..92......99 )2)), we aim to generate a set of ad- versarial passages A which are ranked high by the model in the retrieval results. To mislead the model, we would like to find sequence of tokens f2,... which maximizes the similarity to set of queries:
Retrieval datasets We experiment with the BEIR benchmark (Thakur al., 2021), com- prehensive suite of retrieval datasets. We fo- cus on two popular datasets, Natural Ques- tions (NQ) (Kwiatkowski et al., 2019) and MS MARCO (Nguyen et al., 2016), and evalu- ate our attack on the held-out test queries of these two datasets, as well as its transferability to seven unseen domains (e.g., Quora, scientific, financial documents) in BEIR. See Appendix A for details.
1 Σ Eg(a)) 2 9.E0
arg max
(I)
We discuss how to generate single passage first and extend it to generate multiple passages.
Dense retrievers In our main experiments, we conduct attacks on five state-of-the-as dense re- trieval models: Contriever (Izacard et al., 2022) (pre-trained) and Contriever-ms (fine-tuned on MS MARCO), DPR-nq (Karpukhin et al., 2020) (trained on NQ) and DPR-mul (trained on multiple datasets), and ANCE (Xiong et al., 2021). Ad- ditionally, apply the attack on multi-vector dense retriever, ColBERT (Khattab and Zaharia, 2020), study the effectiveness of our attack on different architectures.
We use a gradient-based based approach to solve the optimization problem, inspired by the HotFlip method (Ebrahimi et al., 2018: Wallace et al., 2019). which approximates the model output change of replacing token. We initialize the ad- versarial passage using a random passage from the corpus. At each step, randomly select token 4 in and compute an approximation of the model output if replacing t; with another token t.. We employ HotFiii to efficiently compute this approx- imation using gradients: e/ Ve,, sim(g; a), where Ve,, sim(q, a) is the gradient vector with respect to the token embedding et,- Given a query set Q, the best replacement candidates for the token & can be obtained by selecting the token that maximizes the output approximation:
Evaluation metries We generate adversarial pas- sages on the training set (NQ or MS MARCO) and inject them into the corpus and measure the top-k attack success rates on test queries, defined as the percentage of queries for which at least one adver- sarial passage is retrieved in the top-k results. A higher success rate indicates that model is more vulnerable corpus poisoning attacks. All the adversarial passages contain 50 tokens, and we per- form the token replacement for 5000 steps. For more implementation details, see Appendix B.
Σ Vel sim(g,a), Q
arg max
GEV
where V is the vocabulary. Note that this operation is cheap as it only requires a single multiplication of the embedding matrix and the gradient vector.
4.2 Attacks In-Domain Queries
Figure 2 (left) shows the in-domain attack results on different models in NQ and MS MARCO. We find that the pre-trained Contriever model (Izacard et al., 2022) is very vulnerable to our even with only one adversarial passage, our attack fools the model on more than 75% of queries NQ and MS MARCO. Although the supervised retrieval models (e.g.. DPR- appear harder to attack, the attack success rates can be substantially improved by generating more adversarial passages. Fig- ure (right), we show that when we generate 500
Generating multiple adversarial passages We extend our attack to generate multiple adversarial passages. Our approach is based on simple idea of grouping similar queries and generating one ad- versarial passage for each group of queries. To achieve this, we use the k-means clustering algo- rithm cluster queries based on their embeddings Fig (g)) Once the queries are grouped into clusters, we generate one adversarial passage for each clus- ter by solving the optimization problem as shown

---

NQ
MS MARCO
Tog-20 Attack Sunowes Rata
70
D7
50
50
ANCE
Model
50
Contriever
84.2
98.1
99.4
75.2
92.2
98.6
40-
Contriever-ms
0.5
52.5
80.9
2.4
20.9
34,9
DPR-nq
3.8
18.8
0.1
13.9
20-
DPR-mul
0.0
10.6
28.3
0.0
4.7
16.3
ANCE
1.0
14.7
34.3
0.0
2.3
11.6
Number Adversartal Passages
Figure 2: Top-20 success rate of our attack. Left: We generate |A| {1, 10, adversarial passages on the train sets of NQ MS MARCO and evaluate on their held-out test queries. Right: On NQ, show that the attack success rate can be substantially improved by generating 500 passages. Contriever: the pre-trained Contriever model (Izacard et al., 2022), Contriever-ms Contriever fine- ine-tuned on MS MARCO. DPR-nq: DPR trained on NQ (Karpukhin et al., 2020), DPR-mul: DPR trained combination of datasets. ANCE: (Xiong al., 2021).
Source domain
NQ
NQ
[A =
10
50
Target domain
[C]
NQ
2.6M
98.7
ColBERT (50 tokens)
0.1
3.0
11.5
MS MARCO
8.8M
95.3
ColBERT (250 tokens)
0.1
8.5
20.1
Hotpot QA
5.2M
100.0
100.0
FIQA
S7K
94.1
96.8
Table 2:
Top-20 20 success rate of attack on Col- BERT (Khattab and Zaharia, 2020). We generate adver- sarial passages with (50, 250} tokens on the train set of NQ. We find the attack performance multi-vector re- trievers is more sensitive to adversatial passage lengths.
Quora
522K
97.2
98.3
FEVER
5,4M
97.9
86.4
TREC-COVID
171K
90.0
100.0
ArguAna
8.6K
SCIDOCS
25K
76.1
71.4
Table 1: Transferring top-20 success rate of our attack with [A|| 50 adversarial passages. We conduct the attack on Contriever with NQ or MS MARCO (MS) training sets and evaluate the attack on different test sets of the BEIR benchmark (Thakur et al., 2021). See Appendix for more details of each dataset.
answering task). This implies potential for prac- tical deployment of the attack that is effective in different domains.
4,4 Attacks on Multi-Vector Retrievers
adversarial passages, all dense retrieval models fooled on least 50% of test queries. Our results clearly demonstrate that dense retrieval models are significantly vulnerable to our proposed attack and fail on a large number of queries with only a small fraction of adversarial passages.
In Section 4.2, we conduct attacks on state-of-the- art dense retrieval models that map a query passage a single vector representation. Here, we study the effectiveness of our attack on multi- vector retrievers, where a piece of text corresponds to multiple vectors. We conduct our attack on Col- BERT (Khattab and Zaharia, 2020) (ColBERT-vI, pre-trained on MS MARCO) using the training queries from NQ with {1, 10, 50} adversarial pas- sages. Table 2 shows the attack performance on ColBERT on in-domain NQ queries. Interestingly, we find that generating longer adversarial passages (250 tokens) is much more effective in attacking ColBERT than using generating short ones (50 tokens). We think this is because the similarity function is computed with multiple embeddings in multi- vector dense retrieval models (c.g., SumMax in ColBERT), thus it is less sensitive single ken changes. We find that inserting 50 adversarial passages with 250 tokens fools ColBERT on 20.1 NQ test queries, suggesting that our attack is still effective in attacking multi-vector retrieval models.
4.3 Attacks Transfer Out-of-Domain
We test whether the generated adversarial pas- sages can transfer across different domains and retrieval tasks. We perform the attack on the Con- triever model with the training sets of NQ and MS MARCO, and insert the adversarial passages into the corpora of other retrieval tasks. Table shows the transfer attack results. Surprisingly, we find that the generated adversarial passages can trans- fer effectively to mislead the models on other re- trieval domains/datasets (e.g., the attack success rate is 94.1 1% on FiQA, a financial domain question-
DDue to the limited computing resources, we use [A| 50 in most of the experiments in the rest of paper.

---

Length (tokens)
20
50
100
sages using natural text. We initialize the adver- sarial passages with 50 [MASK] tokens, the attack success rate drops from 98.1% to 95.8% when at- tacking Contriever on NQ with 10 adversarial pas- sages. This shows that the attack is still effective with a nonsense initial text but using natural pas- sages would make the attack more effective.
Top-20 success rate
63.5
81.6
80.9
79.2
Table 3:
Top-20 success rate of our attack with Contriever-ms on the test queries of NQ when using different lengths of adversarial passages. We generate 50 adversarial passages using the train set of NQ.
Unnaturalness of adversarial passages Our at- tack results in unnatural passages (see examples in A Appendix E). This can be leveraged formulate simple defenses, as we show in Appendix C. How- ever, we note that attackers may develop stronger attacks which produce fluent text that can still fool models, as demonstrated by Song et al. (2021).
4.5 Targeted Attack with Misinformation
Finally, we consider case study, in which we examine the possibility of generating adversarial passages that include targeted misinformation. To accomplish this, we initialize the adversarial pas- sages with "OUR OWN GOVERNMENT IS LIT- ERALLY POISONING US WITH CHEMTRAILS" followed by [MASK] tokens and keep the prefix fixed during optimization. We experiment with the Contriever model using the NQ dataset by gener- ating |A| ε (1, 10) of such adversarial passages. We find the attack remains effective under this constrained setup, although the success rates drop, reaching 11.7% and 59.6% with and 10 passages, respectively. This suggests users would see the targeted message after just two or three queries, demonstrating the potential for spreading harmful information this way.
Attacks on sparse retrievers (BM25) We study whether the proposed attack is effective at sparse lexical retrieval models such BM25. Specifically, attack BM25 by maximizing the BM25 similar- ity of the adversarial passage to a group of queries (using Contriever embeddings in k-means). We find that the adversarial passage to a query group contains only repeated tokens, such as a passage consisting of 50 tokens "what". We generate and insert 50 adversarial passages and find that BM25 retrieval performance does not change at all (i.e.. attack success rate = 0%).
5 Additional Analysis and Discussion
Next, we study whether the adversarial passages for Contriever can transfer to BM25. We find that on the NQ dataset with 50 adversarial passages, 0% of queries are fooled by the Contriever adversarial passages, when we use BM25 to retrieve. These re- sults suggest that sparse retrieval models are robust to the proposed corpus poisoning attack.
Attacks are not model-agnostic We study whether the generated adversarial passages trans- fer across different models. We find limited trans- ferability (i.e., success rates < 0.5%) across simi- lar models (e.g.. Contriever to Contriever-ms) and no transferability across different model families. These results suggest that it would be challenging to leverage transferability for black-box attacks, common strategy against real-world systems with unknown model weights (Pupernot al., 2016).
Transferability does not simply come from train-test overlap In Section 4.3, we show that the adversarial passages can transfer across differ- ent domains/datasets. In Appendix D, we study different transfer attack strategies and show that the transferability does not simply come from the overlap between training queries and test queries.
Lengths of adversarial passages We study the effects of adversarial passage lengths on the attack performance. We attack Contriever-ms using the training data from NQ with 50 adversarial passages of length {10, 20, 50, 100). As Table shows, as long as the adversarial passages contain a sufficient number of tokens (i.e., 2 20), the attack perfor- mance is not affected much when further increasing the passage length.
Conclusions
We proposed a new attack for dense retrievers, in which adversarial passages are inserted into the cor- pus to mislead their retrieval outputs. We show that even small number of adversarial passages successfully attack state-of-the-art dense retriev- ers and generalize to queries from unseen domains. These findings have important implications for the future deployment of robust retrieval systems in real-world applications.
Initialization of adversarial passages In our at- tack, we initialize the adversarial passage using a random passage from the corpus. Here, we study the importance of initializing the adversarial pas-

---

Limitations
References
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526.
The limitations of our research are as follows:
Despite the effectivvness,, our approach can be computationally expensive to carry out. For each group of queries, we need to run our attack independently on a single GPU to gen- erate one adversarial passage. For future work, it is worth considering the exploration of more efficient methods for generating multiple ad- versarial passages, which may serve to miti- gate the computational costs of our approach.
Arman Cohan, Sergey Feldman, Iz Beltagy. Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-Box Adversarial Ex- amples for Text Classification. Association for Computational Linguistics (ACL).
Our study focuses on corpus poisoning attacks to investigate whether dense retrieval mod- els can be deployed safely. We acknowledge that there exist other potential avenues of at- tack against dense retrieval models, such as maliciously manipulating user queries. Our investigation specifically centers on corpus poisoning attacks and their implications for the vulnerability of dense retrieval models.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Scbas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Robin Jia and Percy Liang, 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference Empiri- cal Methods Natural Language Processing, pages 2021-2031 Copenhagen, Denmark. Association for Computational Linguistics.
Ethical Considerations
Our research studies the vulnerability of dense re- trieval models, which are widely used techniques in the industry. The results of our experiments indicate that malicious users can successfully per- form the proposed attack, resulting in a high rate of mislcading dense retrieval model. Although we propose potential defenses in Section 5, it is important to note that the proposed attack has the potential for misuse, allowing for the spread of toxic information. Future research based on this attack should exercise caution and consider the potential consequences of any proposed method. Furthermore, we point out that we include a small piece of misinformation in the paper (the prefix in Section 4.5) to highlight the harmful potential of corpus poisoning.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769 6781, Online. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova. Llion Jones, Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Lc, and Slav Petrov. 2019. Natu- questions: benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Acknowledgements
We thank Dan Friedman, Victoria Graf, and Yangsibo Huang for providing helpful feedback. This research is supported by the "Dynabench Data Collection and Benchmarking Platform" award from Meta AI. 77 is supported by JP Morgan Ph.D. Fellowship.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversar- ial attack against BERT using BERT In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2193-62033 Online. Association for Computational Linguistics.

---

Bin Liang, Hongcheng Li. Miaogiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text clas- sification can be fooled. In International Joint Con- ference on Artificial Intelligence (IJCAT).
Congzheng Song. Alexander Rush, and Vitaly Shmatikov. 2020. Adversarial semantic collisions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4198-4210, Online. Association for Computa- tional Linguisties.
Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song. Changlong Sun, Xiaofeng Wang. Wei Lu, and Xi- wozhong Liu. 2022. Order-disorder: Imitation adver- sarial attacks for black-box neural ranking models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22, page 2025-2039, 2039, New York, NY. USA. Associa- tion for Computing Machinery.
Junshuai Song, Jiangshan Zhang, Jifeng Zhu, Mengyun Tang, and Yong Yang. 2022. Attttack: Text rewrit- ing attack against text retrieval. In Proceedings of the 7th Workshop Representation Learning for NLP, pages 191-203, Dublin, Ireland, Association for Computational Linguistics.
Macedo Maia, Siegfried Handschuh, André Freitas. Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www' 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018.
Liwei Song. Xinwei Yu, Hsuan-Turg Peng, and Karthik Narasimhan. 2021. Universal adversarial attacks with natural triggers for text classification. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3724-3733 Online. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAuaak: A frame- work for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126, Online. Association for Computa- tional Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPs
James Thorne, Andreas Vlachos. Christos Christoddoulopoulos,, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfel- low. 2016. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 1605.0777.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS).
Giorgos Tolias, Filip Radenovic, and Ondrej Chum. 2019. Targeted mismatch adversarial attack: Query with a flower to retrieve the tower. In International Conference on Computer Vision (ICCV), pages 5037- 5046.
P Patil Swati, BV Pawar, and S Patil Ajay. 2013. Search engine optimization: A study. Research Journal of Computer and Information Technology Sciences.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo. Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information re- trieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1 12 ACM New York, NY, USA.
Nisarg Raval and Manisha Verma. 2020. One word at a time: adversarial attacks retrieval models. arXiv preprint arXiv:2008.02197
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov. 2020. Humpty dumpty: Controlling word meanings via corpus poisoning. In IEEE sym- posium security and privacy (SP).
Henning Wachsmuth. Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 241-251, Melbourne, Australia. Association for Computational Linguistics.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lec, and Danqi Chen. 2021. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 61 6138-6148, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Eric Wallace, Shi Feng Nikhil Kandpal, Matt Gard- ner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceed- ings of the 2019 Conference on Empirical Methods

---

Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNIP), pages 2153-2162, Hong Kong, China. Association for Computational Linguis- tics.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadal- lah, and Bo Li. 2021. Adversarial GLUE: multi- task benchmark for robustness evaluation of language models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks.
Yanru Xiao and Cong Wang. 2021. You see what want you to see: Exploring targeted black-box transfer- ability attack for hash- based image retrieval systems. In IEEESCVF Conference on Computer Vision and Pattern Recognition (CVPR).
Lee Xiong, Chenyan Xiong, Ye Li. Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text trieval. In International Conference on Machine Learning (ICML).
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018 HotporQA: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.

---

A Dataset Details
Original Corpus
We use several datasets with different tasks and domains from BEIR (Thakur et al., 2021), the widely-used benchmark for zero-shot evaluation of information retrieval, to evaluate the transferabil- ity of the adversarial passages: Natrual Questions (Kwiatkowski al., 2019), MS MARCO (Nguyen al., 2016). HotpotQA (Yang al., 2018), FiQA (Maia et al., 2018), Quora, FEVER (Thorne al., 2018), TREC-COVID (Voorhees et al., 2021), Ar- guAna (Wachsmuth et al., 2018), and SCIDOCS (Cohan et al., 2020). We include more information and data statistics in Table 4.
Adversarial Passages
10
9
6
33
Average token log likelihood under GPT-2
Figure 3: Average token log likelihood for Wikipedia passages and corresponding adversarial passages for NQ with Contriever.
B Implementation Details
to mention spam filters and moderators real- world platforms. Indeed, Figure shows that off-the-shelf GPT-2 can separate the corpus pas- sages and adversarial passages almost perfectly. However, future work might show that an adver- sary can overcome this weakness by incorporating natural language priors into the attack.
We implement our attack based on the PyTorch library (Paszke al., 2019). We set the length of all adversarial passages be 50. During optimization, we perform token replacement for 5, 000 steps. At each step, we compute the gradients based on a batch of queries Q (we set batch size as 64) and randomly select a token l; in the current adversarial passage to be replaced. We use the gradient-bass HotFlip method (Ebrahimi et al., 2018) to obtain set of replacement candidates, which maximize the approximate similarity after replacement:
Embedding norm clipping The interaction be- tween queries and passages is strictly limited to the inner product, such that sim(p.9) Ep(p)) l2 cos 6, where is the angle between query and passage embedding. As it is not possible to re- duce the angle with respect to a wide set of queries to zero, the attacker should primarily focus pro- ducing passage embeddings with large fg-norms. Indeed, Figure 4 shows that this is the case for the adversarial passages against Contriever. This prompts second defense, where we clip the em- bedding norms of the passage at a constant a for all passages in the retricval corpus:
1
Σ e Vel, ai.... Q
arg max
KE
We consider t0p-100 tokens potential replace- ments. We measure the actual similarity changes on each of these candidates and use the token, which maximizes the similarity between the passage and this batch of queries, to replace tz-
Defenses
We notice that the adversarial passages generated by attack have two distinct characteristics: (1) they contain highly unnatural sequences, and this can be detected by likelihood scoring of an off-the- shelf language model, and (2) their embeddings have very high forms to boost their similarity scores. We study potential approaches to detect adversarial passages.
[p)
Exip = Ep(p)
(4)
min() (p)
Based on the scale observed as Figure 4, we explore different values of a in Table 5 (in Ap- pendix C) and find that clipping with a = 1.75 reduces the attack success rate by 99.4% while only hurting the standard retrieval performance by 6%. Despite this promising defense, it remains possible that corpus poisoning for targeted query distributions or more sophisticated architectures. e.g.. ColBERT's late interaction mechanism (Khat- tab and Zaharia, 2020), may be harder to detect via vector norm anomalies.
Filter by likelihood Adversarial passages con- tain highly unnatural sequences, see examples in Appendix E. One could easily detect and distin- guish the adversarial examples from the corpus passages by scoring the average token log likeli- hood of each passage with a language model, not

---

Dataset
Task
Domain
# Query
# Corpus
Natural Questions (NQ)
Question Answering (QA)
132,803 (trai)
Wikipedia
2,681,468
(Kwiatkowski al., 2019)
3,452 (test)
532,761 (train)
MS MARCO (MS) (Nguyen ul., 2016)
Passage Retrieval
Misc.
8,841,823
(test)
170,000 (train)
HotpotQA (Yang al., 2018)
QA QA Question Retrieval
Wikipedia
5,233,329
7,405 (test)
FIQA (Maia al., 2018)
Finance
648
57,638
Quora
Quora
10,000
522,391
140.085 (train)
FEVER (Thorne et al., 2018)
Fact Chocking
Wikipedia
5,416,568
6,666 (test)
TREC-COVID (Voorhees al., 2021)
Information Retrieval
Bio-Medica
50
171,332
ArguAna (Wachsmuth et al., 2018)
Argument Retrieval
Misc.
1,406
8,674
SCIDOCS (Cohan al., 2020)
Citation Prediction
Scientific
1,000
25,657
Table 4: Statistics of datasets.
information comes from BEIR (Thakur al., 2021).
Full transfer
(1)
(2)
NQ (in-domain)
94.2
Out-of-domain
MS MARCO
95.3
77.1
76.3
Original Corpus
Hotpot QA
100.0
97.3
97.9
Adversarial Passages
FIQA
94.1
74.5
74.0
Quora
97.2
81.0
80.2
FEVER
97.9
63.9
82.3
TREC-COVID
90.0
68.0
72.0
ArguAna
69.6
23.5
22.2
SCIDOCS
76.1
25.2
28.6
1.4
1.6
1.8
2.0
2.2
2.4
2.6
(EE(.))
Table 6: Performance of different transfer attack strate-
gies. Full transfer: the vanilla transfer attack we used
Figure 4: Distribution l norms for embeddings of Wikipedia passages and 10 corresponding adversarial passages for NQ with Contriever.
Section 4.3. (I): transfer attack the most similar training query. (2): transfer attack via the k-means clus- ter. See Appendix D for details.
D Transferability Does Not Simply Come from Train-Test Overlap
In Section 4.3, we show that the generated adver- sarial passages can transfer across different do- mains/datasets. Here we investigate whether the transferability simply comes from the overlap be- tween training and testing query distributions.
Retrieval acc. w/o attack
Attack success
rate
00
56.7
99.4
2.25
52.4
7.2
We attack Contriever on the training set of NQ and generate 50 adversarial passages. For a given test query 4 (in-domain or out-of-domain), we first find similar queries from the training set, and we use the adversarial passages that are most effective on those training queries to attack the current test query Specifically, we consider two cases: (1) We find the training query 4", which is most similar test query q (measured using the query encoder of Contriever). Among 50 adversarial passages, we find the passage p" which maximizes the sim- ilarity to the training query q" We insert p* into the corpus as an attack. (2) We classify the test
2.00
52.6
3.2
1.75
53.4
0.6
1.50
52.2
0.2
0
49.8
0.0
Table 5: Top-20 retrieval accuracy (w/o attack) and at- tack success rate when clipping the fg-norms of passage embeddings a value for a Contriever model NQ. denotes clipping and 0 is equivalent to using cosine similarity.

---

query q using the k-means classifier that is learned when generating adversarial passages. As for each k-means cluster, we have generated one passage, we simply insert the adversarial passage p" that is corresponding to the k-means cluster that is classified as.
The results are shown in Table 6. merely using (I) or (2) is not sufficient to preserve the trans- fer attack rates. For example, the performance on attacking Scidocs drops from 76.1% 25.2%, showing that the test queries and training queries are likely drawn from very different domains. We also find that the performance on HotpotQA does not drop lot, because the distribution is similar to the training set (both NQ and HotpotQA include knowledge-related questions on Wikipedia). These results suggest that the high transfer attack success rates achieved by our approach do not simply come from the overlap between test queries and training queries.
Qualitative Examples of Adversarial Passages
Table 7 shows some qualitative examples of adver- sarial passages generated by conducting the attack on Contriever (Izacard et al., 2022) with a group NQ queries clustered by k-means. We find that although the generated passages are difficult to in- terpret as human text, they contain some keywords (highlighted in blue) that are highly relevant to the queries. For example, in group of queries re- lated to basketball, the corresponding adversarial passage includes words like "bulls" (a famous bas- ketball team in the US) and "russell" famous basketball player).

---

Clustered Query Examples
Adversarial Passage
what is the definition of nba rookie who has more nba championships the cast or west how many points did Jebron have in the finals when did dell curry retire from the nba
oblast exceed continuity improvement scenes cinematogra- phy enhance draftity rich dionual 750 harta pour bracelet rookie russell arbitration outrageous bull mega litter bulls elk parades scanning outta kr pound posts; pronounced ive groove except how tap clite baskets
who played guitar in while my guitar gently weeps who wrote the song all have offer you is me who played guitar on you really got me who wrote the song you'te 're the one that want
drug council deadlineanalysis composition disclosure abil- ity dcadlinpponexidelity virussh 2018 markogusmpt vam- pire tornadolusion children hindwings hurricane premiere county combolife patriarch thru robb rat murders youngest hedge fhi kristintzedel dies suspects overdose textiles
how many finger lakes there in new york how many local government we have in ogun state how many judges are there in supreme court now how strong was the 2011 earthquake in japan
maxi stra ul flies joo microwave direc- tions ceilings spa good view view more please wi towels poo sunshinc sg ient lu slight refrigerator zipper super air- fields observation numbers waterfront gard
parade peat
what dutch elm discase and how does spread where in the plant does the calvin cycle take place what is the purpose of shunt in the brain why we use hsa as standard for protein estimation
mini comment combe qui attract hosh presbyterian heart boot mma pw led cell el what synth ctric ida pw integrate sound dyed thanked visa ael platt director cell nervous grow- ing rdial wley answer den chain cular coiled thy wing na- tionalists perceived vati nant textbook attention ral warnings involving
what is the richest woman in the us what is the birth and death rate in america what is the population of las cruces new mexico what the third most viewed video on youtube
? ranking is lowest lica gible what amount iest in- ternationally quieter automobiles necessary francoise lake draft bonuses mainly tyne bonuses defeats bonuses > day summary sourced sourced card autobiography suns train- ing malaga bonuses qualifying through player alaska what whose locality surrounding ately receives liest logram whether
who played the judge in the good place who directed one flew over the cuckoos nest who played the waitress the movie michael who played in fast and furious tokyo drift
female vate vegetables amino sin tama /= wikipedia fuss bian cent maguzine cco din naire du fill uts brush aster oe whose whose remove black vegetables prefer spices main article ris communes guides kos actress translation daryl actor who took potassium
Table 7: Qualitative examples of generated adversarial passages. We show the query examples in a group clustered by k-means from the NQ dataset (left) and the corresponding adversarial passage (right). We highlight the keywords in the adversarial passages that are highly connected to the queries.


Reviews:
[{"review_id": "fL4ePFS78u", "reviewer": "Reviewer_oe2c", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for the valuable feedback! We are encouraged that the reviewer found our paper was well-written, with a clear objective and detailed method, and sufficient experiments to show the effectiveness of our approach!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well written, clearly defining their objective function and detailing how the method works\n- Tests on a multiple QA datasets and two different retrieval models\n- Used the BEIR benchmark to test transferring attacks\n- For a short paper, provides experimentation to prove the effectiveness of the method and even provides basic experiments for defense", "reasons_to_reject": "- None as a short paper ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper present a novel attack for dense retrieval systems, in which they are able to effectively inject poisoned passages into the top-k documents retrieved by the systems. This method requires only a very small subset of the total training set to be poisoned to be effective. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNbqdn3dgc", "reviewer": "Reviewer_CaDC", "summary": "", "questions": "Pls see \"Reasons To Reject\"", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We are glad to see that the reviewer finds our paper is well-presented and has introduced an effective attack method and extensive empirical studies! We addressed the reviewer’s comments below and we are willing to engage in further discussion should any follow-up questions arise!\n\n\n>**“The attack method is similar to HotFlip. So what is the technical contribution here? (except for different applications)”**\n\nWe do not claim that we propose a new method that can attack a wide range of NLP models (like HotFlip does); instead, we present a novel attack paradigm (i.e., corpus poisoning attack) for retrieval models. Compared to previous works which generate an adversarial passage for a specific query (e.g., Song et al., 2020, Song et al., 2022), we consider a stronger setting where the adversarial passage needs to fool the model on a broad set of user queries. Furthermore, our experimental results reveal the significant vulnerability of dense retrieval models and hope our research can facilitate the development of more robust dense retrieval models. \n\n>**“It is good to know the attack rate of different lengths of the injected passages.”**\n\nOur preliminary experiments showed that the attack performance was not sensitive to the length of adversarial passage. \n\nTo address the reviewer’s concern, we additionally conduct an ablation study, where we attack Contriever-ms using the training data from NQ with 50 adversarial passages of length {10, 20, 50, 100}. As the following table shows, as long as the adversarial passages contain a sufficient amount of tokens (i.e., >=20), the attack performance won’t change much when further increasing the passage length. We will include this ablation study in our revision.\n\n| **Length**             | 10   | 20   | 50   | 100  |\n|------------------------|------|------|------|------|\n| **Attack performance** | 63.5 | 81.6 | 80.9 | 79.2 |\n\n>**“lack of comparisons to SOTA works.”**\n\nWe are not sure which works the reviewer is referring to. We kindly request the reviewer to elaborate on which works we should compare with and it will significantly help us address the reviewer’s concern. \n\nNote that we present a novel attack paradigm where an adversarial passage must be retrieved for a set of user queries. Our setting is different from previous work (e.g., Song et al., 2020, Song et al., 2022), where an adversarial passage is crafted for a specific query. \n\n**References**\n\nCongzheng Song, Alexander Rush, and Vitaly Shmatikov. “Adversarial semantic collisions.” EMNLP 2020\n\nJunshuai Song, Jiangshan Zhang, Jifeng Zhu, Mengyun 481 Tang, and Yong Yang. “TRAttack: Text rewrit ing attack against text retrieval.” Proceedings of 483 the 7th Workshop on Representation Learning for NLP, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Effective attack method\n2. Extensive empirical studies\n3. Paper is well- presented and easy to follow.", "reasons_to_reject": "1. The attack method is similar to HotFlip. So what is the technical contribution here? (except for different applications)\n2. It is good to know the attack rate of different lengths of the injected passages.\n3. Lack of comparisons to STOA works.", "questions_to_authors": "Pls see \"Reasons To Reject\"", "paper_topic_and_main_contributions": "The paper presents an adversarial attack method for dense retrievers for IR system. The approach adopts the idea from HotFlip and successfully injects perturbed passages to fool the retriever. \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "It needs to add more STOA works "}, "extra_text": {}}, {"review_id": "ETPFlcomtm", "reviewer": "Reviewer_eoXC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the invaluable comments from the reviewer. We are encouraged to see that the reviewer views our investigation of attacking retrieval models as a major strength, and the reviewer finds our paper describes the technique clearly and succinctly, and has studied potential defenses to this type of attack! We have addressed the comments in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n> **“The abstract and introduction both need to be clear that the findings only apply to this single type of architecture for dense retrieval”**\n\nThanks for the suggestion! We will revise the abstract and introduction to make our claims more precise that the majority of the work is to focus on bi-encoder architectures of dense retrieval models. However, we would like to note the importance and wide application of dual-encoder retrievers due to their simplicity and efficiency, and we have explored a range of popular models in this category. \n\nOn the other hand, we thank the reviewer’s suggestion and added additional experiments based on ColBERT. To study the effectiveness of our attack on ColBERT, we conduct our attack on the ColBERT model (ColBERT v1, trained on MS MARCO) using the training queries from NQ with {1, 10, 50} adversarial passages. The following table shows the top-20 success rates of our attack on NQ test queries. We find that our attack is still effective in attacking multi-vector retrieval models like ColBERT. Interestingly, we find that generating longer adversarial passages (e.g., 250 tokens) is more effective in attacking ColBERT than using short passages (e.g., 50 tokens), likely because of the SumMax operation that ColBERT uses, while the performance of attacking single-vector models is not sensitive to adversarial passage lengths. We are very happy to include this experiment and discussion in our revision.\n\n| $\\|\\mathcal{A}\\|$       | 1    | 10   | 50   |\n|-------------------------|------|------|------|\n| Contriever              | 84.2 | 98.1 | 99.4 |\n| Contriever-ms           | 0.5  | 52.5 | 80.9 |\n| DPR-nq                  | 0.0  | 3.8  | 18.8 |\n| DPR-mul                 | 0.0  | 10.6 | 28.3 |\n| ANCE                    | 1.0  | 14.7 | 34.3 |\n| ColBERT (250 tokens)    | 0.1  | 8.5  | 20.1 |\n|   - ablation: 50 tokens | 0.1  | 3.0  | 11.5 |\n\n**About attacking sparse retrieval models**: We note that in Appendix C.1, we have shown that the proposed attack is not effective in attacking BM25. The sparse retrieval model SPLADE expanded the sparse vector representations by leveraging the output logit vectors returned by BERT. We agree that it is worth studying the corpus poisoning attack to SPLADE, and we hope to do it in the future. \n\n>**“The authors fail to investigate the similarity of queries in the different collections relative to the training data.”**\n\nWe appreciate the reviewer’s suggestion to investigate the similarity of test queries to the training queries. We have conducted additional experiments to investigate this issue.\n\nSuppose that we generate 50 adversarial passages to Contriever on the training set of NQ; for a given test query $q$, we first find similar queries from the training set, and we use the adversarial passages that are most effective on those training queries to attack the current test query $q$. Specifically, we consider two cases:\n\n**(1)** We find the training query $q^*$, which is most similar to test query $q$ (measured using the query encoder of Contriever). Among 50 adversarial passages, we find the passage $p^*$, which maximizes the similarity to the training query $q^*$. We insert $p^*$ into the corpus as an attack.\n\n**(2)** We classify the test query $q$ using the k-means classifier that is learned when generating adversarial passages. As for each k-means cluster, we have generated one passage; we simply insert the adversarial passage $p^*$ that corresponds to the k-means cluster that $q$ is classified as.\n\nOur hypothesis is that if the high transfer attack success rates mainly come from the overlap between test query distribution and train query distribution, the attack in (1) and (2) should be as effective as the full transfer attack. However, as the table below shows, merely using (1) or (2) is not sufficient to preserve the transfer attack rates. For example, the performance on attacking Scidocs drops from 76.1% to 25.2%, showing that the test queries and training queries are likely drawn from very different domains. We also find that the performance on HotpotQA does not drop a lot because the distribution is similar to the training set (both NQ and HotpotQA include knowledge-related questions on Wikipedia). \n\nThese results suggest that the high transfer attack success rates achieved by our approach do not simply come from the overlap between test queries and training queries.  We hope these results provide a better understanding of the generalization of the adversarial passages. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise!\n\n|                   | Full transfer attack | **(1)** attack via closest training query $q^*$ | **(2)** attack via the k-means cluster |\n|-------------------|--------------------------|-----------------------------------------------------------|----------------------------------------|\n| NQ (**in-domain**)    | 99.4                     | 93.9                                                      | 94.2                                   |\n| **Out-of-domain** |                          |                                                           |                                        |\n| MS MARCO          | 95.3                     | 77.1                                                      | 76.3                                   |\n| HotpotQA          | 100.0                    | 97.3                                                      | 97.9                                   |\n| FiQA              | 94.1                     | 74.5                                                      | 74.0                                   |\n| Quora             | 97.2                     | 81.0                                                      | 80.2                                   |\n| FEVER             | 97.9                     | 63.9                                                      | 82.3                                   |\n| TREC-COVID        | 90.0                     | 68.0                                                      | 72.0                                   |\n| ArguAna           | 69.6                     | 23.5                                                      | 22.2                                   |\n| Scidocs           | 76.1                     | 25.2                                                      | 28.6                                   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A major strength of the paper is investigation of bi-encoder retrieval models susceptibility to poisoning by introducing artificial documents that are frequently retrieved. The process of creating a passage and inserting it into a collection represents a vector of attack on such models if they were deployed by search engines. The paper clearly and succinctly describes the technique in a way that another researcher to the requisite computing skills should be able to replicate the work. In addition the paper identifies a particular weakness in this type of model. While the generated passages are non-sensical, they are highly ranked due to L2 regularization.", "reasons_to_reject": "A weaknesses of this paper is that apart for a brief exploration of GPT3, the authors only include one type of architecture for dense retrieval, the bi-encoder architecture where passages are represented with a single dense vector. The paper needs to be clear that the paper cannot say anything about cross-encoder architectures, learned sparse representations (e.g SPLADE), or even bi-encoder architectures that represent queries and passages with multiple dense vectors, such as ColBERT. The abstract and introduction both need to be clear that the findings only apply to this single type of architecture for dense retrieval. The paper needs to introduce other dense architectures and cite the appropriate literature. Experiments with ColBERT need to be added to the Appendix.\n\nA second weakness of the paper arises in the evaluation on the BIER dataset. The setup of the evaluation appears valid; however, the authors fail to investigate the similarity of queries in the different collections relative to the training data. The fact that documents are vastly different from the training set is immaterial if the queries are represented in the training data. In such a case, it would be unsurprising that the artificial passages are ranked highly as is seen in the evaluation. Such an investigation should be added to the paper to give the reader a better understanding of the poisoning vulnerability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach for generating documents to add to a collection that are likely to be retrieved by a dense retrieval bi-encoder model that represents a document with a single vector. This is done by adding a set of adversarial passages where the objective is to create the passages in such a way that 1 of a small set of adversarial passage will be ranked in the top k results for the distribution of queries. To find representative queries, the training queries are clustered and by starting with a random passage in the collection and at each step randomly replacing a token with another token so that the similarity between the represented query and document increases. Such adversarial passages frequently appear in the ranked lists of queries in the test set, both when evaluating on the test set of for the collection that was used to train the model and on other evaluation data set that are different from the training data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Include more dense retrieval architectures listed in the reasons to reject", "Typos_Grammar_Style_and_Presentation_Improvements": "Remove the footnote from the abstract. The abstract needs to stand along, so it should include the footnoted information in the body of the abstract."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an attack on dense retrieval systems, introducing a method to inject poisoned passages into the top-k documents retrieved by these systems. The attack method is thoroughly explained and extensively tested on multiple QA datasets and two different retrieval models, with additional evaluation using the BEIR benchmark for transferring attacks. The paper is well-written and structured, offering clear explanations of the method and its results, making it easy to follow.

One of the paper's significant strengths is its clear presentation of the attack method, its objectives, and the detailed explanation of how the method operates. The extensive empirical studies conducted by the authors provide strong evidence of the attack's effectiveness, enhancing the paper's overall quality. Additionally, the paper goes beyond the attack itself by including basic experiments for defense strategies, demonstrating a comprehensive approach to addressing the issue.

However, some concerns were raised by reviewers. One reviewer pointed out similarities between the attack method and the existing HotFlip technique, seeking clarification regarding the technical contributions of the paper. Additionally, the paper lacks comparisons to state-of-the-art (STOA) works in the field. The paper's focus on a specific type of dense retrieval architecture, the bi-encoder model, should be made clear in the abstract and introduction, and the paper should acknowledge other relevant architectures and cite appropriate literature. Lastly, the evaluation on the BEIR dataset should consider the similarity of queries in different collections relative to the training data to provide a more accurate assessment of poisoning vulnerability.
Decision: Accept-Main