Evaluating the Adversarial Robustness of Retrieval-Based In-Context Learning for Large Language Models
Authors: Simon Chi Lok Yu
Jie He
Pasquale Minervini
Jeff Z. Pan
Venue: COLM (2024)
Keywords: Adversarial Robustness, Adversarial Attack and Defences, Retrieval-based LLM, In-context Learning, Evaluation
TL;DR: Performed a systematic study on the adversarial robustness of variants of ICL across different models; and proposed a new adversarial defence methods

Abstract: With the emergence of large language models, such as LLaMA and OpenAI GPT-3, In-Context Learning (ICL) gained significant attention due to its effectiveness and efficiency. However, ICL is very sensitive to the choice, order, and verbaliser used to encode the demonstrations in the prompt. \emph{Retrieval-Augmented ICL} methods try to address this problem by leveraging retrievers to extract semantically related examples as demonstrations. While this approach yields more accurate results, its robustness against various types of adversarial attacks, including perturbations on test samples, demonstrations, and retrieved data, remains under-explored. Our study reveals that retrieval-augmented models can enhance robustness against test sample attacks, outperforming vanilla ICL with a 4.87\% reduction in Attack Success Rate (ASR); however, they exhibit overconfidence in the demonstrations, leading to a 2\% increase in ASR for demonstration attacks. Adversarial training can help improve the robustness of ICL methods to adversarial attacks; however, such a training scheme can be too costly in the context of LLMs. As an alternative, we introduce an effective training-free adversarial defence method, \emph{DARD}, which enriches the example pool with those attacked samples. We show that DARD yields improvements in performance and robustness, achieving a 15\% reduction in ASR over the baselines. Code and data are available jointly with this submission as supplementary material.

Published as a conference paper at COLM 2024
Evaluating the Adversarial Robustness of Retrieval-Based In-Context Learning for Large Language Models
Simon Yu", Jie He', Pasquale Minervini Jeff Z. Pan
School of Informatics
University of Edinburgh
(s1967531, he, j. z. pan @ed. ac. ac.uk, wwwnervieexxeedd.doommo ed. uk
Abstract
With the emergence of large language models, such as LLaMA and GPT, In-Context Learning (ICL) gained significant attention due to its effective- ness and efficiency. However, ICL is very sensitive to the choice, order, and verbaliser used to encode the demonstrations in the prompt. Retrievnl- Augmented ICL methods try to address this problem by leveraging retrievers to extract semantically related examples as demonstrations. While this ap- proach yields more accurate results, its robustness against various types of adversarial attacks, including perturbations on test samples, demonstra- tions, and retrieved data, remains under-explored. Our study reveals that retrieval-augmented models can enhance robustness against test sample attacks, outperforming vanilla ICL with 4.87% reduction in Attack Success Rate (ASR); however, they exhibit overconfidence in the demonstrations, leading to 2% increase in ASR for demonstration attacks. Adversarial training can help improve the robustness of ICL methods to adversarial attacks; however, such a training scheme can be too costly in the context of LLMs. As an alternative, we introduce an effective training-free free adversarial defence method, DARD, which enriches the example pool with those at- tacked samples. We show that DARD yields improvements in performance and robustness, achieving a 15% reduction in ASR over the baselines. Code and data are released to encourage further research
Introduction
Large Language models (LLMs) are revolutionising the field of NLP (Brown et al., 2020; Wei et al., 2022; Touvron et al., 2023). However, this also raises concerns about their ro- bustness (Xu et al., 2023c). Although significant efforts are put into aligning LLMs for safety purposes, recent works still show LLMs can be vulnerable to adversarial inputs and jailbreaking (Zou et al., 2023; Xu et al., 2023b; Zeng et al., 2024; Longpre et al., 2024). As a result, is essential to understand LLM security properties to guide the direction of LLMs that are secure and robust. In this paper, we focus on examining a common LLM inference method, namely In-Context Learning (ICL.) and variants, on how they might be susceptible to perturbations and their robustness against adversarial attacks.
ICL is a common few-shot training method for prompting LLM for downstream tasks (Brown et al., 2020). Its variants include retrieval-based ICL models, which utilise retrievers to identify similar examples as demonstrations (Wang et al., 2023c; Li al., 2023); or KNN-ICL, which compute label demonstrations using nearest neighbours algorithm; have been adopted and shown promising performance improvements when compared to vanilla ICL (Shi et al., 2022; Xu et al., 2023a). However, the robustness of these methods against adversarial attacks received less a attention and remains unclear. Prior work-which review in Section shows that LLMs can still be vulnerable to different types of input
"Equal Contributions
*Corresponding Author

---

Published as a conference paper at COLM 2024
perturbations, such task instructions (Zhu et al., 2023; Sun et al., 2023) inputs (Wang et al., 2023a); however, the robustness of ICL-based methods to such perturbations is still under-explored.
We address (cf. Section 4) the following question: How sensitive are vanilla and retrieval- based ICL methods to perturbations to their fest samples or demonstration instances, which is particularly important given that the primary motivation of ICL is to facilitate few-shot learning without training; should any methods exhibit excessive sensitivity, this would limit their applicability. To investigate this problem, we employ commonly used perturbation fatacks-named character-lezel, word-level, and sentence-leeel attacks. Furthermore, we pro- pose new attack methods that target the demonstration instances and datastore examples. Our findings indicate that retrieval-based ICL methods show greater robustness against test-sampl ple attacks; however, they can be more vulnerable when their demonstrations are adversarially perturbed. Detailed ablation studies further reveal that this vulnerability to adversarial attacks persists in newer models, such Mistral (Jiang et al., 2023) and Gemma (Gemma Team et al., 2024). Many of these attacks can be transferable, both from the same family (LLaMA-2-7B + LLaMA-2-13B, 70B) or different family (LLaMA-2-7B - Mistral-7B). We showed Mixture of Expert (MoE) models still posed a similar adversarial threat to its dense version. This encourages further investigation on enhancing model robustness.
We further explore the advantages of utilisiing retrieval-based models to improve robustness (cf. Section 5), considering that existing defence strategies often rely training process (Li et al., 2021; Si et al., 2021). However, this process is memory intensive in the context of LLMs. Instead, we show that retrieval-based ICL methods, combined with adversarially augmented examples, yield more robust results than no augmentation methods while maintaining their predictive accuracy on in-distribution evaluation sets. Therefore, we propose DARD, training-free free adversarial defence method which leverages adversarially perturbed samples to augment the retrieval pools, potentially improving the models' robustness. Figure presents an overview of the paper.
adversarial
Our main contributions are summarized as follows.
We first perform comprehensive and in-depth evaluation on variants of ICL. vanilla, KNN-ICL, and retrieval-based ICL methods against Test-Sample, Demonstration and Datastore Adversarial Attacks.
We show that Retrieval-based models can bring positive results on robustness when test samples are under adversarial attack, but it further reduces the robustness while demonstrations are perturbed. We show these attacks persist on larger models and are transferable.
We propose DARD, novel training-free adversarial defence method which leverages adversarially perturbed examples to improve the robustness.
2
Related Work
Adversarial attack and defences Adversarial attack has always been topic in deep neural networks, where previous works showed neural networks can be vulnerable to adversarial examples that are crafted by adding imperceptible but adversarial noise on natural examples (Szegedy et al., 2013; Evtimov et al., 2017). In the real world, adversarial robustness is important for the application of method (Tramèr et al., 2017).
Several works studied this problem and the main finding is that LMs struggle to answer correctly and factually under attacks and propose their defence via label smoothing or prompt tuning (Yang et al., 2022; Raman et al., 2023). It is important to note, however, that these experiments were mainly conducted using relatively small LMs such as BERT, RoBERTa. Recently, several works demonstrated LLM can still be vulnerable to these type of attacks, by showing the vulnerability of attacking the task instructions (PromptBench (Zhu et al., 2023)), or performing adversarial attacks on zero-shot settings and investigating the transferability of attacks from open-sourced models (i.e. Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023)) to close-sourced (i.e. GPT-3.5 and GPT-4) (DecodingTrust (Wang et al., 2023a)). However, previous study (Gu et al., 2022) has shown that the LLMs are less sensitive to prompt instruction variation when fho-shot examples are provided
2

---

Published as a conference paper at COLM 2024
(b) Rettrieval-bbsee Defences (DARD)
Irrelevant Sample Attack
0
Teatlwgger
DAgO
RORTHAltac
Ponkunch
Reteheser
wia Dafences
Retriever
Retriever
Training data
Externl Cerpus
2
23
N5
Retriever
Prompl
Instraations
S/1
Ø
Demoosttratio Attack
(d) Test Sample Attack
Textbugger
used - umar
Berign Model
apper
Swapp Labbels w/
shuth pastonce
BERT-Attack
Retriener
SHORT Instructor
Ponitive
Figure 1: Overview the paper. We visualize our seven adversarial attacks in (a), (c) and (d) (only 3 shots are used in the plot for display purposes). And our adversarial defence method, DARD, showcased in the top right corner (Plot (b)).
in context (a. ICL). However, there is still lack of systematic analysis on this case which is the main contribution of this paper. Wang et al. (2023b) shows that attacking demonstrations can still be an effective way for adversarial attacks where LLMs are still vulnerable. Therefore, aim extend and evaluate the adversarial attack and defence performance on retrieval-based models.
ICL Sensitivity The existing literature on ICL analysis can be broadly divided into two streams, each focusing on different aspects. The first stream explores the influencing factors of ICL based on input perturbation, such as the order (Min et al., 2022), the formatting (Yoo et al., 2022; Lu et al., 2022) and the selection of the demonstration (Liu et al., 2022). Designing proper demonstration construction strategies (Ye et al., 2023; Li et al., 2023) could bring clear boosts to the ICL performance, while Chen et al. (2023) analyses the sensitivity instructions and example ordering, and our work is complement of their work on sensitivity. This paper provides different view of when different ICL methods fail and when retrieval-based ICL can be better approach for robustness.
of ICL via perturb
Retrieval-based LLM Retrieval-based LLM has shown great success in lifting LLM per- formance and faithfulness and reducing hallucination (Pan et al., 2023), where the idea is to gather related articles or demonstrations from collection of data (Huang et al., 2023). Specifically, we are interested in whether they can be used to improve the robustness against perturbations (Lin & Lee, 2024). Meanwhile, many papers also focus on analysing the robustness of RAG methods on irrelevant contexts (Shi et al., 2023; Asai et al., 2024), in which we adopt similar methods in our context for analysing how retrieval-based ICL being sensitive to irrelevant contexts.
Problem Formulation
3.1 Background
In this work, we consider several types of ICL methods, namely Vanilla ICL, and two types of retrieval-based ICI. KNN-ICL (Xu et al., 2023a) and Retrieval-based ICL (Wang
3

---

Published as a conference paper at COLM 2024
et al., 2023c; Li et al., 2023). Retrieval-based ICL methods demonstrated promising perfor- mance improvements over vanilla ICL ones; however, their robustness against adversarial perturbations remains unexplored.
Vanilla ICL is the most commonly used few-shot learning method in LLM. Given a
training data set T {xi, (x), Vi)},, where E X denote the instances and y; Y represent their labels, the concatenation of the k-shot input-label pairs used to generate the prompt witth respect to a test instance Xtest=
P I ① π 91) (22, ① ..... ① (x2x Mic) ① (
(1)
where Z denotes the task instructions, π. denotes the templates and verbaliser to turn inputs into prompts (details for each dataset are in Appendix A.3), is concatenation operator, and [Y] is the number of distinct labels. The label prediction ytest for the test instance Xtust is then computed as:
Yiest
arg max pyy LLM)) yeY
(2)
KNN-Prompting was first introduced by Shi et al. (2022) to perform zero-shot inference and extended in later work by Xu et al. (2023a) support few-shot inferences, simulating the ICL. Throughout the paper, we follow the implementation of Xu et al. (2023a) and call KNN-ICL The idea is to map each training instance into latent representation using LLM and cache the last token distribution into a datastore. At inference time, the prediction based on the interpolation between the LLM prediction and the k-nearest neighbours label distribution in the datastore. Specifically, for each training instance x;, Xu et al. (2023a) concatenate it into prompt P. together with k in-context examples, where k = |Y|
Pj = (x1, @... (xxx)))3/x) ① (x;,
By querying the LLM using P, we obtain a distribution over tokens p(o PP, 0). Rather than mapping it back label space Y, the distribution is cached as the key representation for the instance X;:
k; Pp
(4)
which is mapped to the label y/. The entire datastore thus consists of { yi) pairs, where K = {ki}; denotes the set of keys. At inference time, we construct the same prompt and Presit OLLM). We then match the distribution against
obtain the distribution Pust cached K the datastore, where standard KL-divergence is used to measure the distance:
p(o) Presti Σ p LLL) log p(u) Pi
DKL (aL(Peen (Piest [k)
The predictions are then calculated by interpolating the LLM output and its nearest neighbours distribution, where NNm(#, () denotes the set of m nearest neighbours in K:
9pred arg max yEy (1 a) plyl Piesti OLLM) + a.p
Σ
Ily; 4)
(6)
iENN ( K)
Following Xu et al. (2023a), in this work, we use a 0.2 and m k/2.
R-ICL Retrieval-augmenter ICL (Lin et al., 2022) was proposed to mitigate the sensitivity of ICL methods to the choice and order of the in-context examples: rather than randomly sampling (x,, y:), R-ICL use retriever to identify the most similar examples to the test instance Xtest from the training set T. In this paper, we follow Li et al. (2023) and consider the following retrievers: BM25 (Robertson & Zaragoza, 2009), SBERT (Reimers & Gurevych, 2019) and Instructor (Su et al., 2022); more details are available in Appendix A.1.
3.2 Attack Methods
This paper aims to study how perturbations affect the performance of different ICL methods through adversarial attacks. We classify our 7 attacks into 3 categories: Test Sample Attack, Demonstration Attack, and DataStore Attack.
4

---

Published as a conference paper at COLM 2024
Test Sample Attacks We focus the three different types of attack: typo-based pertur- bations TextBugger (TB) (Li et al., 2018); saabeediig-simillrityy perturbations TextFooler (TF) (Jin et al., 2019); and context-aware perturbations BERT-Attack (BA) (Li et al., 2020).
For demonstration attacks, we refer to adversarial attacks where the training demonstrations are adversarially perturbed. We consider this type of attack to investigate the sensitivity of LLMs to adversarial perturbations to the training demonstra- tions and their influence the generalisation properties of the models.
Demonstration Attacks
Datastore Attacks One of our objectives is to analyse the sensitivity of retrieval-based ICL methods to adversarial perturbations. Therefore, we consider Irrelevant Context Attakss (Irr.), which work by contaminating the demonstration pools with irrelevant contexts. Our Irrelevant Context Attakks extended Min et al. (2022) - we replace 50% of the original demonstrations in the training data with Outtof-Diitruution (OoD) examples.
Further details and analyses about attack methods are available in Appendix A.4.
Experiments
4.1 Settings
Datasets To assess the robustness, we selected six sentiment and multiple-choice text classification datasets: SST2 (Socher al., 2013), RTE (Dagan et al., 2005), CR (Hu & Liu, 2004), MR (Pang & Lee, 2005), MNLL-matched (Williams et al., 2018), and TREC (Voorhees Tice, 2000). We provide further details, including prompt templates and dataset statistics, in Appendix A.3. Unless explicitly mentioned, LLaMA-2-7B model for our experiments (Touvron et al., 2023). For experiments involving randomness (i.e., Vanilla ICL and kNN-ICL), conducted trials with three different random seeds to ensure the reliability of our findings.
mainly used the base
Evaluation evaluate and compare the vulnerabilities across different types of attacks, in line with Wang et al. (2023a); Zhu et al. (2023), we assess the model's performance in a benign (attack-free) context (Clean) and calculate the Attack Success Rate (ASR). The ASR is defined Clean Accuracy-Atlast Accuracy where Attack Accuracy denotes the model's accuracy under attack.
Clean Accuracy
4.2 Results
The 8-shot attack results are presented in Table with 8-shot attacks, and detailed results are available in Appendix B.1.
Overview: Retrieval-based models exhibit higher Clean accuracy: 3.27% higher than vanilla ICL with the best performed SBERT retriever; similar finding with Rubin et al. (2021) that related demonstrations enhance the performance. For adversarial attacks, our proposed adversarial attack methods, Swap-Labels and Swap-Labels (Fix), are effective techniques for manipulating LLM predictions. On the other hand, irrelevant context attacks have negligible effect on performance. We summarise two major findings about the adversarial robustness across different ICL methods.
Finding 1: Retrieval-bane models are more robust against test-sample allacks; however, it could fall short under demonstration attack.
As shown in Table 1, all three R-ICL methods outperform vanilla ICL and KNN-ICL among all Text Sample Attacks (TB, TF and BA), with drop in ASR by 4.87% and 2.47%, respec- tively. In Figure 2, we observe an increased gap in the ASR between ICL and R-ICL
Heer compare with the best performed Rinstructor ICI, with TCL and ANN-ICI.

---

Published as a conference paper at COLM 2024
Tasks
Method
Clean
TB
TF
BA
AdvICL
S-L
S-L (Fix)
Irr:
Avg'
92.33
1.69
48.58
KNN-ICL
92.01
41.01
45.45
61.66
79.06
67.28
66.00
0.0
51.49
SST-2
Revns-ILL
94,61
43.39
45.94
57.33
25.63
99.88
63,88
0.85
46.7
Remar-ICL II
95.18
33.01
33.01
56.26
27:21
99.64
66.84
6.09
46.01
Riesuucte ICI.
94.84
38.33
32.62
58.65
28.22
99.54
69.32
6.73
47.63
ICL
73.04
82:54
75.62
97.86
14.32
91.59
58.31
11.04
61.61
KNN-ICL
70.52
90.61
90.37
95.43
96.13
63.66
63.15
6.83
70.88
RTE
Roas-ICL
71.48
68.69
63,14
90.4
18.77
92.42
60.60
15.15
58.45
RaHRI-ICL
72.20
83.50
83.01
98.12
27.49
95.50
64.50
15.50
66.80
Rissuuioe ICI.
73.29
78.33
72.41
96.67
28.06
90.64
66.00
10.36
63.21
ICL
94.49
KNN-ICT
91.77
39.60
47.18
65.95
78.99
71.81
72,90
2.78
54.17
MR
Resus-ICL
92.50
37.95
45.84
65.95
25.10
19.68
1.51
48.56
ReHEKTICI.. ICI.
92.40
38.53
46.10
64.52
28.83
99.78
60.06
0.0
48.26
Riseouctee ICL
92.6
37.47
47.30
66.22
26.26
98.06
65.33
0.43
48.72
ICI.
91.31
35.54
52.43
71.16
22.8
99.90
91.87
1.22
51.56
KNN-ICL
39.13
58.78
74.07
67.33
37.74
6.03
44.13
CR
Resns-ILL
98.09
29.14
47.43
65.72
33.20
100.00
56.29
7.30
48.44
Remar-CLL
93.62
31.25
46.30
71.59
33.81
100.00
62.32
4.76
50.00
Rissaudoe ICI.
93.88
34.55
39.38
53.82
35.81
100.00
64.88
4.02
47.49
ICL
53.63
57.92
40.85
88.81
1.73
47.73
KNN-ICL
55.89
65.61
52.10
51.30
51.30
62.39
56.00
10.14
49.83
MNLI-mm
Revns-ICL
57.3
68.41
59.16
40.98
38.45
95.99
63,89
0.35
52.46
IICI
57.8
69.03
55.07
43.27
37.01
99.31
63.55
2.77
52.86
Blssaucc IC1.
54.30
66.48
51.93
36.06
37.83
100.00
69.67
2.58
52.08
ICL
76.07
61.09
57.50
60.65
18.69
65.56
40.58
8.19
44.61
KNN-ICI
58.74
10.88
TREC
REaus-ICL
79,20
50.00
50,76
57.07
28.91
55.56
9.60
3.79
36.53
RearrrICL
87.8
43.74
45.56
47.15
34.61
53,08
10.02
9.11
34.75
Rinemnttr ICL
86.5
45.43
70.87
47.51
31.31
44.97
3.35
7.28
35.82
ICL
79,90
53.56
55.65
67.74
24.13
88.78
59.84
4,00
50.53
RNN-ICL
79.98
52.73
53.13
63.86
77.20
65.2
61.40
6.11
54.18
Avg
Amans-CLL
62,91
90.59
ReHERTICI.
83.17
49.84
51.51
63.48
31.49
91.22
54.55
6.37
49.78
Risseuctee ICL
52.42
59.82
31.25
88.87
36.42
5:23
49.16
Table Adversarial Attack results with 8-shot ICL. Clean refers to the Benign accuracy, while all other columns refer to the Attack Success Rate (%) under corresponding Adversarial Attacks. The last column Avg refers to the mean ASR across 7 attacks. The higher the ASR, the lower the robustness of models against the attack.
ANNIC
ACL
Taxt
65
Seccom
63
56
10
53
30
-10
16
16
mommplns
exaneplcs
Figure 2: Attack Success Rate (ASR) across shots among different ICL methods, We aggre- gated the results for ICL and KNN-ICL across 3 seeds and R-ICL results across retrievers.
when the number of shots goes up. Therefore, we believe in-context learning benefits from drawing semantically related demonstrations against adversarial perturbations on both performance and robustness. Recall that these textual attacks are designed to deceive LLMs by replacing words with their synonyms introducing typos in the input sentences. This strategy can be viewed as exploiting LLM's parametric knowledge (Szegedy et al., 2013; Goyal et al., 2022). In this case, R-ICL provide supported demonstrations by retrieving related

---

Published as a conference paper at COLM 2024
1mm-2-18
Lamodde188
Metrs-i8
Vierss-88
Gemmon78
300
TeutTooler
Teaflagge
BERT Aatzk
Ac+1CL Artace Machods
Laial
Inslevrr Cortec
Figure 3: Attack Success Rate (%) for adversarial attacks across various models, based experiments conducted on the RTE dataset with 8-shot demonstrations. The results are based on R-ICL and the mean attack success rate among the three retrievers used: BM25, SBERT, and Instructor.
examples to enhance the contextual knowledge to the LLM, therefore reducing the noise of the LLM prediction.
On the other side, are under attack, demonstrating 1%-6% increase in ASR compared with vanilla ICL. This demonstrates the deficit of retrieval-based models that can sometimes overly rely on the retrieved demonstrations and are more vulnerable when demonstrations are perturbed.
increase in ASR for Retrieval-based ICL when demonstrations
Finding 2: KNN-ICL. is highly sensitive to adversarial perturbations on test samples and demon- stration.
In Table 1, ANN-ICL shows the highest ASR in Demonstration Attack, specifically showing the highest vulnerability to AdvICL, 40%-50% higher than ICL and R-ICL. Recall that for KNN-ICL, demonstrations and test samples are mapped into the hidden space to compute the label distribution with nearest neighbours. Any perturbations on the demonstration might notably affect their position in the hidden space and alter the label distribution significantly. Also, in Figure ANN-ICL, exhibits consistently ascending trend in ASR as the number of examples increases, with 15% higher than the other methods. This further illustrates that incorporating nearest neighbours algorithms inevitably includes more noise, thereby increasing vulnerability.
Overall, our experiment results show that few-shot methods LLM could still be vulnerable to adversarial perturbations, leading to 50% drop in performance with previous works on jailbreaking leads to the need for a continuous effort to improve the robustness of LLMs. We include quantitative results in the Appendix B.2.
average. Pairing
4.3 Attack on Model Variants
In this section, extend our experiments to four other models: LLaMA-2-13B, larger version of 7B model; Vicuna-7B (Chiang et al., 2023), instruction-tuned from LLaMA-2- 7B; Mistral-7B (Jiang al., 2023) and Gemma-7B (Gemma Team al., 2024), aiming to provide a view for models from different families to prevent potential bias posed within the pretraining data for LLaMA models. These results allowing us to analyze the vulnerability of modern models with better coverage. We conducted the experiments on RTE datasets with 8-shot demonstrations. Even larger models such as LLaMA-2-70B or Mixtral-8x7B (Jiang et al., 2024) are ignored due to attacking them is expensive and slow. Instead, we leave them in the following section for attack transferability (64.4).
The results are shown in Figure 3 for R-ICL (corresponding results for ICL are presented in Figure 5). It can be observed that most adversarial attacks remain effective across all models, including the Swap-Labels attack we proposed. Thus, it still posed major threat on modern LLMs. Specifically, Vicuna-7B tends to be more sensitive when attack on text sample (TextBugger) or when demonstrations 7B-moss recently released model-shows a higher vulnerability when the context or label in demonstrations is adversarial perturbed, compared to other base models. Notably, the
under attack (AdoICL); while Gemma-
7

---

Published as a conference paper at COLM 2024
TegtBugger
TextFooler
BERT-Attack
Snap Labels
Swap Labels Fix
AdvICL
100
839
50
Attack
1-77 11:A5A-0135R 11:850007001
Migrasi Mirral Sc
Mitrral 518- 1LaM6.2.70 Vinro-m
MetralmT
Ganera-71 711
Figure 4: Analysis of the attack transferability for Retrieval-ICL on larger variant models within the same family: (Left) the LLaMA family; (Mid) the Mistral family; and (Right) across models from different families. The models' orders are sorted by their parameter sizes and release date. The models highlighted in bold are the models being attacked.
irrelevant context attacks become less vulnerable to more recent release models, especially Vicuna, which shows nearly no effect after being tuned with instructions, showing improved capability in the latest LLMs for filtering OoD contexts (Shi al., 2023).
4.4 Attack Transferability
In this section, we would like to answer two questions: Are these attacks transferable across different models, and can larger models mitigate the vulnerability to these attacks? To this end, we conducted two sets of experiments: (1) transfer attacks within the same model family from small models to their large variants, by testing LLaMA-2 and Mistral famililes; (2) whether attacks still be effective across models from different family, by transferring attacks to Vicuna-7B, Mistral-7B and Gemma-7B, following the same settings as Section 4.3, Our experiments focused on the RTE dataset, chosen for its pronounced drop in robustness when subjected these attacks. Due to memory constraints, both Mixtral-8x 2-70B were run with 8-bit quantization (Frantar et al., 2022), while the remaining models were in full precision. Hong et al. (2024) tested on AdvGLUE++ and showed that quantized LLMs have negligible drops in adversarial robustness to their source model.
and LLaMA-
The results of the transfer attack with R-ICL are shown in Figure (corresponding results for ICL are presented in Figure 6). Most attacks are transferable and achieve an ASR of 15%-40% on similar scale models; except for AdvICL, which cannot transfer between models. Also, Swap-Labels attacks are shown to be less effective when targeting instruction-tuned models (e.g. Vicuna and
pointing to the instruction-tuning process leading models to be less sensitive to its demonstration (Sun et al., 2023). Specifically, the results shown in Figure 4(L) confirm that larger models tend to be more robust, as indicated by reduction of 6% on average in ASR between the 13B and 70B models. However, the MoE variants of (iistral-B--Insruus exhibit a similar or even higher ASR compared to its dense version, illustrated in Figure 4(M). This finding deviates from previous work by Puigcerver et al. (2022), which investigates adversarial robustness in MoE, showing Vision MoE models are less vulnerable adversarial attacks. We believe this discrepancy might be subject to the difference between on how MoE is applied in NLP (token) and Vision (image-patch) models. Remember that in the training of MoE models, experts are specialised to handle different tasks (Xue et al., 2024); perturbations in the input texts might change the router output and be routed to experts that are less capable for the targeted tasks, leading to more vulnerability and changes in the output. We hope our findings encourage future work to study the robustness of MoE models, both from the training and router perspectives, given their widespread use nowadays.
8

---

Published as a conference paper at COLM 2024
Defence
Shots
Clean
TB4
TF
BA
Avg
4 No Defence
71.48
68.69
63.14
90.4
74.08
Augmentation
Random Addition
8
72.01
67.75
67.76
91.6
75.70
Random Deletion
69.18
65.72
62.78
85.96
71.49
DARD
> R-ICL (BM25)
74.39
58.30
51.43
79.32
63.01
R-ICL (SBERT)
8
72.16
58.53
44.46
72.60
58.53
R-ICL (Instructor)
71.26
69.00
61.81
84.31
71.70
Adversarial Training
8
77.22
62.22
41.96
77.22
60.47
Table 2: DARD for adversarial defences. We show the Clean Accuracy and the ASR. For Random Addition and Deletion baselines, BM25 is the retriever as it performs the best.
Defence Methods: DARD
In this section, we propose easy yet effective method to enhance the robustness of retrieval-based models against adversarial attacks on test samples. Specifically, we introduce an adversarial defence method named Demonstration Augmentation Retrieval Defences (DARD). This approach performs adversarial augmentation on the training data and mixes them into the retrieval bases for R-ICL. The initiative is that adversarial training in LLM can be costly in memory, and Xie et al. (2021); Dai et al. (2022) showed that in-context learning can be seen as an implicit finetuning method.
5.1 Methodology
The methods begin by adversarially perturbing the training examples, similar to previous approaches. Our experiments use the RTE dataset, focusing on an 8-shot setting for this section. For each of the 872 test samples in RTE, the retriever selects the 8-shot most similar examples for demonstrations. After deduplication, we are left with 7,293 distinct examples. These are concatenated with one-shot examples, and adversarial attacks (TF, TB, and BA) are performed on the extracted training samples. Only the successful examples are retained, resulting in 16,206 perturbed examples. In the adversarial training paradigm, such perturbed examples would typically be used to fine-tune models. However, our approach, DARD, reintegrates these examples into the training demonstration for retrieval during the inference stage. It is important to note that we impose constraint whereby each example and its perturbed examples can be retrieved no more than once in same prompt for TCL. This is because we observed that most retrieved samples were variants of the same examples with perturbations without this constraint, which reduced both performance and robustness.
5.2 Results
For baselines, we compared with No Defence, Random Augmentation (Addition, Deletion) inspired by Robey et al. (2023), which are smoothing methods used to defend against attacks. We also compared with adversarially trained models following vanilla ICL setup. We show the results in Table 2 (detailed results are in Table 19). Although the adversarially trained models show a higher clean accuracy as it's directly fine-tuned on RTE data, our methods with SBERT outperform it by 2% reduction in AŠR; both BM25 and SBERT variants of DARD have significant improvement on robustness against No Defence baseline.
Summary
While ICL sensitivity is a widely known issue, our paper provides an alternative view of the problem through adversarial attacks. In summary, our paper conducts comprehensive analysis the effects of perturbations on modern few-shot methods for LLMs inference and also proposes new adversarial attacks targeted at ICL. models. Besides their perfor-

---

Published as a conference paper at COLM 2024
mance, the findings show the other side regarding retrieval-based models related to their robustness. Furthermore, we also demonstrate the potential of retrieval-basee models in data augmentation strategies for adversarial defense.
There are many open questions for future work. One of the most profound findings is that our study attack transferability demonstrates that larger models generally exhibit better robustness; however, this does not necessarily hold true for Mixture of Experts (MoB) models. Given the recent surge in their popularity, it is crucial to investigate the conditions under which MoE models may fail and to understand the reasons behind their vulnerabilities.
7 Limitations
We view our work as an early exploration of the adversarial robustness of Retrieval-based in-context learning. Here, only show how different types of attack can be applied to, but not why. Given the standard use of retrieval and in-context learning as inference methods, is crucial to understand its deficiency. Future work should focus certified robustness (Zhao et al., 2022; Xiang et al., 2024), which analyzes with theoretical proof the extent to which the model unaffected by adversarial attacks.
Acknowledgement
Computations described in this research were supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edin- burgh. We thank Giwon Hong and Yu Zhao for the insightful discussion the beginning of this research work.
References
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hanna Hajishirzi, and Wen tau Yih. Reliable, adaptable, and attributable language models with retrieval. 2024. URL https: //api. semanticscholat httpr//cccccoooooorrorrrrrsssees htgg/CorpusID:262248811
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Verbertt-osss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.1410 1016,,, 2020. URL https: //api. semanticscholar. org/CorpusID:2189111
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 155- 167, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ (ew.022.finding--emee
URL https://ecllanthoooy..rrrg org/2023, findings-emnlp
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
URL http::////mmnyy..r//////////----
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005. URL https:
//api. www.nnicschholarrrrrreei orgg/oorpsSDD:88779500
10

---

Published as a conference paper at COLM 2024
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. 2022. URL
ott.maanttsssstlo...oorrrr/rsooo orgg/orrpsID: 2586654444
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partition- ing. ArXit, 259936734.
2023. URL https://api. semanticscholar org/CorpusIDD
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL https: //api. semanticscholar.
org/CorpusID::222 52967399
Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Xiaodong Song. Robust physical-world attacks deep learning models. Xiv: Cryptography and Security, 2017. URL https://app //asi www.eteeessssssieeeeeeee semanticscholar.. org/ C0rpusID:4738292
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq:
Accurate post- training quantization for generative pre-trained transformers. ArXiv, 2bs//220.17323, 2022. URL https: //aii. semanticscholar arg/CorpusID: (orppssID:233337200 253237200.
Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and al. Gemma. 2024. doi:
URL https:
//www. wwwgl.aggle.... com/m/3301
Shreyansh Goyal, Sumanth Doddapaneni, Mitesh M.Khapra, and Balaraman Ravin- dran. A survey of adversarial defences and robustness in nlp. 2022. URL https: //api. senpoiisamnnicschollen .rrg/oooggggoooo orgg/Corpu1D::244475111
Jiasheng Gu, Hanzi Xu, Liang Nie, and Wenpeng Yin. Robustness of learning from task instructions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL
semanticscholar. https://api, semanti cscholar. org/ CCrp/puSSID:224466664
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, March 2017. doi: 10.5281/ 2e1066.4120316.
Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. Decoding compressed trust: Scrutinizing the trustworthiness of efficient lims under compression. 2024. URL https: //api. www.eessmatiiccchhoeo wwwpchholarrrog/ooruuueeee org. wrgg/ooppssDD 268680727.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004. URL httpp://apt semanticschol. org/ Io 200-222722555 155288.
Wenyu Huang, Mirella Lapata, Pavlos Vougiouklis, Nikos Papasarantopoulos, and Jeff Z. Pan. Retrieval Augmented Generation with Rich Answer Encoding. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia- Pacificc Chapter of the Association for Computational Linguistics (IJCNLP 2023, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. ArXiv, abs/ abs/2401 22//2.4010000 2024. URL https://apl semanticscholast .ogg/org///ooo /Bt/CCCpuIDD:266444933
11

---

Published as a conference paper at COLM 2024
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023. URL https://app wwwanticscholar. .rrg// :Truu0D::23833453
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong base- line for natural language attack on text classification and entailment. In AAAAeronoeeeen on Artificial Intelligence, 2019. URL https: / /apii. semanticscholar. hrt.aaarrrrr/oorrs1::22222225552
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. ArXiv, abs/ 18122.521,,, 2018. URL https://api. semanticscholar ar. org/
wwtp/orrgg/llaooeee http///oor:DD///ee 54815877.
Linyang Li, Ruotian Ma, Qipeng Guo, X. Xue, and Xipeng Qiu. Bert-attack: Adversar- ial attack against bert using bert. ArXiv, abs/2004.09984, 2020. URL https://api.
semanticscholar. hoaptnttssccolaorrr//oopps///// argg/CorpuSDD:216355
Xiaonan Li, Kai Lv, Hang Yan, Tianya Lin, Wei Zhu, Yuan Ni, Guo Tong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. ArXio, abs/2305.04320, 2023. URL https://api. www.nticccholar.orggg..o.oo wrgg/CorpusIDDDDD : 258557751.
Zongyi Li, Jianhan Xu, Jiehang Zeng, Linyang Li, Xiaoqing Zheng, Qi Zhang, Kai-Wei Chang, and Cho-Jui Hsieh. Searching for an effective defender: Benchmarking defense against Moens, Xuanjing Huang, Lucia Specia,
adversarial word substitution. In Marie-Francin and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3137-3147, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0.18653/v1/222 wttr:/221..mmmpmmmaii.. emnlp-main. 251.
emnlp-main.251. URL
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Un- supervised cross-task generalization via retrieval augmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Curran Asso-
Neural Information Processing Systems, volume 35, pp. ciates, Inc., 2022. URL https: tttpss://proceeiiiggs neur cc/paper Files/paper/2022 pdf.
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. 2024. URL https://api. hwwaaaniiiiccco....oo///rrrrooop 268063278
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vulié (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100- 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. http://ww//2222eelloooooooo
doi:
URL
wwwpollooyyorg//222/22 org/2022.222/ deelio-1 10.
Shayne Longpre, Sangpr Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili-Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Yong, Suhas Kotha, Yi Zeng, Weiyan Shi, Xianjun Yang, Reid Southen, Alexander Robey, Patrick Chao, Diyi Yang, Ruoxi Jia, Daniel Kang, Sandy Pentland, Arvind Narayanan, Percy Liang, and Peter Henderson. A safe harbor for ai evaluation and red teaming. ArXiv, abs/ abs/2403.0es 2403.04493, 2024.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086 8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 wllpv222222a2 acl-log..55555 URL https://aclanthoolgy orw.oog//22222 acl-long long. 556.
12

---

Published as a conference paper at COLM 2024
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Zhao. Dricl: Demonstration--ettivved in-context leaming. ArXiv, 2023. URL https://apt semanticscholar wwwp:/oogg//////oeeee.ee/ 25: 258841276.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 00.18633/v1/2222 20tp...e.mmmmaammmm555555 URL https://accannee haapa:///alanthologyy org/ 2022. e022eemalpmmnnn 759.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 119-126, 2020.
Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lissandrini, ussa Biswas, Gerard de Melo, Angela Bonifati, Édlira Vakaj, Mauro Dragoni, and amien Graux. Large language models and knowledge graphs: Opportunities and challenges. Transactions on Graph Data and Knowledge, 2023.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment catego- rization with respect to rating scales. In Kevin Knight, Hwee Tou Ng, and Kemal Oflazer (eds.), Proceedings of the 43rd Annual Meefing of the Association for Computational Linguistics (ACL'05), pp. 115-124, Ann Arbor, Michigan, June 2005, Association for Computational Linguistics. doi: 10.3115/121999 1111//29109..119885555 URL httttt://cclaathheitet /ttp:/aaaaaaatooo....ooo org/P05-1 1015.
Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pranjal Awasthi, and Srinadh Bho- janapalli. On the adversarie robustness of mixture of experts. ArXiv, abs/ 0bss/2200.1055. 2022. URL https: / //ppi. semanticscholar .rg org htgg/oorpuSD::252990 252992477.
Mrigank Raman, Pratyush Maini, J. Zico Kolter, Zachary Chase Lipton, and Danish Pruthi. Model-tuning via prompts makes nlp models adversarially robust. ArXiv, abs 2b0/0200,773 2023. URL https: //ppi. semanticschola.. wwrg/CorppsID.... : 257495746.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.orr abs/1908. 10084.
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3:333-389, 2009. URL https: //aii. semanticscholar.
org/CorpusID:20711 207118704
Alexander RobxandeRRRbby,,, Eric Wong, Hamed Hassani, and George Pappas. Paapaes Smoothllm: Defending large language models against jailbreaking attacks. ArXiv, abs/2310.0368/ 2023. URL https://apl.semantiisss
commanticscholle wwtp/oorpss..:2330555 263611542.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for 2021. URL https: //api. semanticscholar.
in-context learning. ArXiv,
org/CorpusID: 24:ooppuuu::2521885111111 245218561..
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. URL https:tps//// htttps:/pprceeedingg mlr. hrt...ppres//222//he 23a. html.
PMLR, 23-29 Jul 2023.
13

---

Published as a conference paper at COLM 2024
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neigh- bor zero-shot inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pp. 3254-3265, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 /ttp:////22..mmmmmaaa..... //aclanthology org/2022. emnlp-main 214.
URL https:
Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. Better robustness by more coverage: Adversarial and mixup data augmen- tation for robust finetuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1569- 1576, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 2021. 1021.findings-acl.33 URL https: / hwwpt/oaaaattoooo..ooooooooo org. 2021. findings-ard acl. 137.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: http::://www... aclweb. org http//anhhologg/11-11111
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-
finetuned text embeddings. 2022. URL
Jiu Sun, Chantal Shaib, and Byron Wallace. Evaluating the zero-shot robustness of instruction-tuned language models. ArXiv, abs/2306.11270, 2023. URL https://api. semanticscholar. org/ www.oog///ooo...... (CCrrpuII::252003000
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. URL https: / /api. semanticscholar Org/CorpusID) 604334.
60043334.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing Ilama model. https://// thub.
2023.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/ 23088,9288,,, 2023. URL https: /api. inmanticcchoar... wwwpoolarrrrg/ooooooeeeo org/CorpusID: 259950998.
Florian Tramer, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick Medaniel. Ensemble adversarial training: Attacks and defenses. ArXiv, abs/1705.07204, 2017. URL
https://api. www.seeeeaaanolleeeee www.aarroor/Coooooooooooo 21946795.
Ellen M. Voorhees and Dawn M. Tice. The TREC-8 question answering track. In M. Gavrili- dou, G. Carayannis, S. Markantonatou, S. Piperidis, and G. Stainhauer (eds.), Proceed- ings of the Second International Conference on Language Resources and Evaluation (LREC'00), Athens, Greece, May 2000. European Language Resources Association (ELRA). URL .d.f
http:/ http://www. 1rec-conf. or wwwpg//ooeedins/reeeeeeeee
14

---

Published as a conference paper at COLM 2024
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zi-Han Lin, Yuk-Kit Cheng, Sanmi Koyejo, Dawn Xiaodong Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. ArXiv, CorpusID:25920777
11698, 2023a. URL https://api.
http://app..smmancccch...o... semanticscholar .rrg/
Jiong Wang, Zi yang Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. Adversarial 2023b. URL
demonstration attacks on large language models. ArXio,
https://api. www.nticscoollaaee...ee. wwwpooolaarrrroooouuuoseeeeee 258.0rr
Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. ArXiv, hts/2307.07164/ 2023c. URL https://api. semanticscholar, org/CorpusID: 259924840.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreviee nel/forumm1d=g32r6Gooooo
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http://aclweb. o
Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption, 2024. URL https:://rrxtt org/abs/ 2405. 15556.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. ArXiv, abs/2111.02083 2021. URL h semanticscholar wawplaarrrrggg//e org /eg/CorppsID: 241 241035330.
Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. knn prompting: Beyond-context learning with calibration nearest neighbor infer- ence. ArXiz, abs/2303. 13824, 2023a. URL https://app semanticscholar. www.org//CCoooo wwwpoolarrrrr/oopuuussssese IICCrrusIII: 257756989.
Nan Xu, Fei Wang, Ben Zhou, Bangzheng Li, Chaowei Xiao, and Muhao Chen. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. ArXiv, abs/2311.098277 2023b. URL https://app swwanti wttg/orrpsssaainno.ooo 26522222::2223999
Rongwu Xu, Brian S. Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. The earth is flat because... Investigating Ilms' belief towards misinformation via persuasive conversation. ArXiv, abs 2atps///310000,,, 2023c.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint ar ari::24022001733 2024.
Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. In and out-of-domain text ad- versarial robustness via label smoothing. ArXiv, abs/2212.10258,, 2022. URL https: //api. stapiiismmatisschol.... argg/CorpusID:
254877265.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, 2023.
URL / sewattccccncanaaarrrrr org/CCrpuss 2568266793.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang
15

---

Published as a conference paper at COLM 2024
(eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 2422-2437, Abu Dhabi, United Arab Emirates, December 2022. Associa- URL
tion for Computational Linguistics. doi: 10.18653/ (0118633v112210 /ttp:///2222...mmmmm...m https://aclantthllgg org/2022 amnlp-main.155
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. Arxiv, abs/ 000//201106673, 2024. URt semant icscholar org/
CorpusID:2669773355
Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang. Certified robustness against natural language attacks by causal intervention, 2022. URL https::/arxxv..rrg///////////// org httgg//bb/222555/ 12331.
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Weirong Ye, Neil Zhenqiang Gong, Yue Zhang, and Xingxu Xie. Promptbench: Towards evaluating the robustness of large language models adversarial prompts. ArXio, abs/2306.04525, 2023. URL https://apis sem.anicccccnihhhlarrr
ooreaniiiissoooooooooo//rrrrrsso :t:ooppuuu::220955222 259095572.
org/CorpusID:
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. ArXiv, abs/2307.15043, 2023, URL https://api. semanticscholas oorpaanicccccioooooo//rorosooDo0 org/CorppsID::2622299611
A Experiment Settings
This section discusses the details of the ICL methods used, including their hyperparameters, and also the verbalizer and prompt templates for classification tasks. The details on each attack are also provided for reproducibility. Code are included as supplementary materials and will be publicly available upon the paper is accepted.
A.1 Details about the ICL methods
KNN-ICL the exact formulation follows the original paper (Xu et al., 2023a).
R-ICL We give more details about the retriever used throughout the paper here:
1. BM25 (Robertson & Zaragoza, 2009): A prevailing sparse retriever. For BM25, we follow Luo et al. (2023) and use uncased BERT wordpiece tokenization (Devlin et al., 2019).
2. SBERT (Reimers Gurevych, 2019): We use the Sentence-BERT as the dense demonstration retriever. Specifically, we used all-MiniLM-L6-V to encode the test input and training set's inputs, and retrieve the examples with the most similar input as demonstrations.
3. Instructor (Su et al., 2022): Instructor is recently proposed competitive text em- bedding model trained on 330 tasks with instructions. By providing specialized instruction, it can serve for demonstration retrieval. We conduct experiments on its released large-size model
A.2 The statistics of datasets
The table shows the number of categories for each classification dataset, the number of examples in the training and test sets, and the average sentence length.
A.3
Templates and verbaliser for datasets
See Table for the used templates and verbaliser for mapping discrete labels into label space (Adopted from Min et al. (2022) and Lu et al. (2022)).
"hhunlp/instruuttr-larggg
16

---

Published as a conference paper at COLM 2024
Datasets
No. of Labels
Train/Test Size
Avg. Length
SST-2
67348/87)
23.07
RTE
2
2489/277
96.03
MR
8530/1066
37.76
CR
3394/376
30.80
MNLI-mm
8832/99 /982
55,90
TREC
5452/500 500
19.59
Table 3: Datasets Information
Task
Template
Verbaliser
SS12 Review:
contains no wit, only labored gags
0: negative, 1: pusitive
Sentiment: negative
Review: the film powerful accessible and funny.
Sentiment:
RTE
man is due in court later charged with the murder years ago teenager whose case the first to be featured on BBC One's Crimewatch. Colette Aram, 16, was walking her boyfriend's house in Keyworth, Nottinghamshir,, on 30 October 1983 when she disappeared. Her body later found field close her home. Paul Stewart Hutchinson, charged with murder and is due before Nottingham magistrates later." The question is Paul Stewart Hutchinson accused having stabbed girl. True False? The Answer is: false
0: false, 1: true
has been
For carning 22,000 year, the total pay accumulated after six months maternity leave would be just 5,300 the UK and 5,850 in Ireland. Entitlements Germany would also be relatively low, at 5,930, along with those France, Spain and the Netherlands, at 6,750. At the other end of the scale, pay received after six months leave in Italy would be 9,150 while in
Denmark Norway would be as much
The question is Matermity leave varies in Furope. True False?
Answer
MR
Review:
"you might say tykwer has done all that heaven allows, you wanted to make as anti-kieslowski a pun as possible suffice say its total promise left slightly unfulfilled.
0: negative, 1: positive
Sentiment: negative
Review:
an alternatlly raucous and sappy ethnic sitcom... you'd be wise to send your
regrets.
Sentiment:
CR
Review: stylized sony samsung.
D: negative, 1: positive
Sentiment: negative
Review:
went out and got the canon today.
Sentiment:
MNLI
Pleasee identify whether the premise entails the hypothesis. The answer should be exactly yes', pes', 'no' 'maybe'.
Instruction:
O: yes, 1: maybe, 2:
Premise: serve classic Tuscan meal that includes a Florentine tervine made with dick and chicken livers
Hypothesis: We serve meal of Florentine terrine.. Prediction: yes
Premise: lifetime trials, not only earned at Goodwill, eamed
job here.
Hypothesis: Donna went through Goodwill to get her GED, was still unemployed. Prediction:
Question: did serfdom develop in
Russia
0: expression,
Type: description
1: entity,
2: description,
Question:
What is Shakespeare 's nickname?
3: human,
Type:
4: location,
5: number,
Table 4: Templates and verbaliser used across the experiments. These are minimum cases with only one demonstration example for illustration.
A.4 Attack Details
We implemented our proposed attack methods and conducted all our experiments using the TextAttack framework (Morris al., 2020), which is model-agnostic and can be easily tested on any model. Here are the details regarding to each attack method:
17

---

Published as a conference paper at COLM 2024
1. TextBugger: We employ the TextBugger (Li et al., 2018), which determines the importance of words and replaces key words to conduct the attack. We use GloVe embeddings to retrieve nearby words of the words to be replaced for the attack.
2. TextFooler: We also evaluated another popular word-level method, TextFooler (Jin et al., 2019), which is similar principle to TextBugger. However, TextFooler greedily searches for a large number of nearby words in the embedding space for each word, as long as they meet some constraints on embedding similarity and sentence quality. Additional constraints require the replacement words to match the Part--Of-Seecch (POS) of the original words.
3. BERT-Attack: We used the popular character-level attack, BERT-Attack (Li et al., 2020), which utilizes BERT to generate candidate replacement words based on the importance of words. Then, by calculating the impact of each replacement word on the classification outcome, the replacement word that has the most significant impact on the classification result is chosen the final replacement.
4. AdvICL: We mostly follow the implementations by Wang et al. (2023b), except for limiting the attack percentage to be 15%. Also, for RTE and MNLI datasets, which consist of both premise and hypothesis, we only allow perturbations on the hypothesis as it's closer the realistic cases, and the premise is usually much longer than the hypothesis.
5. Swap-Labels/Swap-Laaee WordStoapWIR implementation in textattack framework. The attacks start by measuring the importance of each label in the demonstration. It is done by first replacing each of the labels with a meaningless word, fetching it into the LLM to get the label distribution, and computing the difference to the original label distribution. Based on this, the Swap-Labels attack focuses on the most important labels first, yi, by replacing it with other labels from the label set, namely y (Y/y;). If that reduces the LLM output probability on the correct label, the attack models continue attack the subsequent most important labels with the perturbed examples with y; replaced by y/: otherwise, it keeps y/; and attack another labels. This process iterates until the attack is successful or the maximum perturbations allowed are reached.
(Fix): For both of the attacks, we reused the Greedy-
The only difference between Swap-Labels (Fix) and Swap-Labels is that an addi- tional constraint is imposed: the label distribution the attacked samples should be the same as the original distribution.
6. Irrelevant Context Following Min et al. (2022), CC-news (Hamborg et al., 2017) is used as the corpus providing OoD sentences. Sentence length is considered to select the most relevant sentences to the original demonstration. The contamination rate is 50%.
Examples of each attack are demonstrated in Table 6-12. Code will be released to facilitate further research in this field.
A.5 Reproducability
We conducted all attack experiments using either two A100-40GB GPUs or one A100-80GB GPU. The batch size is 8. For ICL and KNN-ICL that consists of randomness, we conducted
experiments with seed 13, 42}. Most attacks took between 1 and 2.5 hours to run. The exception was the 16-shot demonstration attack, which usually lasted 5 hours or longer. We used Flash-Attention 2 (Dao, 2023) to speed up the experiments, which gave us 2 to 2.5 times speed increase. The speedup significantly helped our comprehensive experiments.
B Complete Results
B.1
Main results
We include the complete results breakdown in Table 13-18. The results shown in those tables are the Benign accuracy and Attack accuracy.
18

---

Published as a conference paper at COLM 2024
Ulmma-2-188
Mistral-78
Vicuna-78
Gemma-78
Attack Success Rate (%)
TextFooler
TextBugger
OCRT Attack
Adv ICL
Swap Labels
Swop Labe's
Inclevant Context
Figure 5: Attack Success Rate (%) for adversarial attacks across various models, based experiments conducted on the RTE dataset with 8-shot demonstrations. The results are based on ICL and involve aggregating results from 3 shots.
TextSugger
Fooler
Swap-Labeel
Swap-L Swap-Labels-Fil
AdviCL
100
100
150
Attack Success Rate (%)
60
60
40
LLaMA-2-78 LLaMA-2-158
LLaM4-2-700 Mistral-7B
Misrral Instract
Mixrol 8x18
Metral 8x18
LLaMA-2-78 2-78 Vicana-T5 7B
Mitral-TBB
Gemmy-78
Figure 6: Transferable attack for ICL on larger variant models within the same family: (L) the LLaMA family; (M) the Mistral family; and (R) across models from different families. The models' orders sorted by their parameter sizes and release date. The models highlighted in bold are the adversarial models under attack.
B.2 Quantitative Results
We conducted a human study to evaluate the feasibility of adversarial attacks and estimate human performance against them. Specifically, we provided 100 input examples to students and asked them to determine the correct label for each given sentence (answer options: true or false). We used the RTE dataset and instructed participants to give their answers solely based the provided context. We assessed various types of attacks.
In this set of 100 sentences, the baseline model we used, Rinstructor-ICC achieved an accuracy of 88% the original, unaltered sentences. First, we observed that word-level attacks test examples had a minor impact on human judgment, with an average decrease of only 3% (Table 5). Attacks such advicl and irrelevant attacks resulted in decreases of only 7% and 0%, respectively. However, these attacks could significantly affect retrieval models, causing attack successful accuracies (ASR) ranging from 15% to 95%, indicating that the models are more fragile and sensitive than anticipated.
For swap-label and swap-label (fix) attacks, human performance decreased by 25% and 13%, respectively. This suggests that attacks manipulating the label of demonstrations indeed caused confusion for humans, demonstrating their effectiveness.
19

---

Published as a conference paper at COLM 2024
TB
TF
BA
AdvICL
S-L
S-L (Fix)
Irr.
Perturbed Examples
84%
87%
82%
81%
63%
75%
88%
Table Summary of the responses from the human evaluation. The total number of evaluated examples is 100. The accuracy under unperturbed conditions on these 100 examples is 88%.
only few mag-lev trains have been used commercially such as at the birmingham airport in the uk.
The question is: maglev is commercially used. True or False?
The answer is: ture.
they, too, have not produced practical, commercially acceptable maglev.
The question is: maglev is commercially used. True or False?
The answer is: false
two weeks ago, china became the first nation to operate a maglev rail- way commercially, when officials inaugurated a 30-kilometer-lon line between downtown shanghai and the city's airport. The question is: maglev is commercially used., True or False? The answer is: true.
Unperturbed
the m-2000 uses commercially manufactured nbti superconductor, simi- lar to that used in the m-2000 maglev magnets,, The question is: maglev is commercially used. True or False? The answer is: false
it appears that the super-conducting maglev system is technically ready to be used commercially tion system.
very high-speed, large-capacity transporta-
The question is: maglev is commercially used. True False?
The answer is false.
only a few mag-lev trains have been used commercially such as at the birmingham airport in the uk.,
The question is: maglev is commercially used. True or False?
The answer is: ture.
they, too, have not produced practical, commercially acceptable maglev,
The question is: maglev is commercially used. True False?
The answer is: false
two weeks ago, china became the first nation to operate a maglev rail- way commercially, when officials inaugurated a 30-kilometer-long line between downtown shanghai and the city's airport. The question is: maglev is commercially used., True or False? The answer true.
Perturbed
the m-2000 uses commercially manufactured nbti superconductor, simi- lar to that used in the m m-2000 maglev magnets., The question is: maglev is commercially used. True False?
The answer is: false
it appears that the super-conducting maglev system is technically ready to be used commercially as very high-speed, large-capacity transporta- tion system.
The question is: maglev is commercially utilize. True or False? The answer is true.
Table 6: This example illustrates the TextBugger attack instance. The portion surrounded by gray background represents our test example, and the text in blue font indicates the words that were altered in the TextBugger attack question.
20

---

Published as a conference paper at COLM 2024
actual statistics about the deterrent value of capital punishment are not available because it is impossible to know who may have been deterred from committing crime.
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
capital punishment is a catalyst for more crime.
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
rather than deterring crime, capital punishment actually increases the level of brutality in society,
Unperturbed
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
the death penalty is not a deterrent.
The question is: capital punishment is deterrent to crime. True False? The answer is false.
closely divided u.s. supreme court said thursday its 2002 ruling that juries and not judges must impose a death sentence applies only to future cases, a decision that may affect more than 100 death row inmates. The question is: the supreme court decided that only judgescan impose the death sentence. True or False?
The answer is false.
actual statistics about the deterrent value of capital punishment are not available because it is impossible to know who may have been deterred from committing crime.
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
capital punishment a catalyst for more crime.
The question is: capital punishment is deterrent to crime. True False? The answer is false.
rather than deterring crime, capital punishment actually increases the level of brutality in society.
Perturbed
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
the death penalty not deterrent.
The question is: capital punishment is deterrent to crime. True or False?
The answer is false.
closely divided u.s. supreme court said thursday its 2002 ruling that juries and not judges must impose a death sentence applies only to future cases, decision that may affect more than 100 death row inmates.
The question is: the high court decision that only magistrates can impose the death condemnation. True or False?
The answer is true.
Table 7: This example illustrates the TextFooler attack instance. The portion surrounded by gray background represents our test example, and the text in blue font indicates the words that were altered in the TextFooler attack question.
21

---

Published as a conference paper at COLM 2024
gold mining operations in california and nevada use cyanide to extract the precious metal.
The question is: cyanide is used in gold mining, True or False? The answer is: true access to the underground workings at the la camorra mine is via ramp from the surface, excavated at a -15% grade and connecting numerous levels.
The question is: la camorra is mine. True or False?
The answer is: true
the mine would operate nonstop seven days week and use tons of cyanide each day to leach the gold from crushed ore.
The question is: weak cyanide solution is poured over it to pull the gold from the rock. True or False?
Unperturbed
The answer is: false
the dam covering what used to be its football pitch fill up with millions of tons of poisonous mining waste. The question is: mine waste-water poses an environmental hazard. True False?
The answer is: false
known "heap leach" mining, the method has become popular in the
last decade because it enables microscopic bits of gold to be economically extracted from low-grade ore.
The question is: the mining industry uses method known as heap leaching. True False?
The answer is true.
gold mining operations in california and nevada use cyanide to extract the precious metal.
The question is: cyanide is used in gold mining, True or False? The answer true
access to the underground workings at the la camorra mine is via ramp from the surface, excavated at a -15% grade and connecting numerous levels.
The question is: la camorra a mine. True or False?
The answer is: true
the mine would operate nonstop seven days a week and use tons of cyanide each day to leach the gold from crushed ore.
The question is: weak cyanide solution is poured over it to pull the gold from the rock. True or False?
Perturbed
The answer is: false
the dam covering what used to be its football pitch fill up with millions of tons of poisonous mining waste. The question is: mine waste-water poses an environmental hazard. True or False?
The answer is: false
known "heap leach" mining, the method has become popular in the
last decade because it enables microscopic bits of gold to be economically extracted from low-grade ore.
The question is: the mining industry uses method known as heap beachmanufacturing. True False?
The answer is false.
Table 8: This example illustrates the BERT-Attack attack instance. The portion surrounded by gray background represents our test example, and the text in blue font indicates the words that were altered in the BERT-Attack attack question.
22

---

Published as a conference paper at COLM 2024
in other words, with its 2 million inhabitants, slovenia has only 5.5 thou- sand professional soldiers.
The question is: slovenia has 5.5 million inhabitants. True or False? The answer is:false
nicholas burns stated in exclusive interview for "utrinski vesnik" from pristina that it would be disgraceful athens puts a veto on macedonia's application for membership in the european union and nato.
The question is: greece and macedonia are in dispute over name. True or False?
The answer is: false
greece objects the neighbouring skopje government using the name macedonia, saying it implies claims on a greek province of the same name. The question is: greece and macedonia are in dispute over name. True or False?
Unperturbed
The answer is: true
in other words, with its 2 million inhabitants, slovenia has only 5.5 thou- sand professional soldiers.
The question is: slovenia has million inhabitants. True or False? The answer is: true
the croatian intent is even more problematic because the border between slovenian and croatian territorial waters has not yet been established. the dispute about this border began in 1991 when both countries became independent.
The question is: there territorial waters dispute. True or False?
The answer is true.
in other words, with its 2 million inhabitants, slovenia has only 5.5 thou- sand professional soldiers.
The question is: slovenia has 5.5 billion inhabitants. True or False? The answer is:false
nicholas burns stated in exclusive interview for "utrinski vesnik" from pristina that it would be disgraceful athens puts a veto on macedonia's application for membership in the european union and nato.
The question is: greece and macedonia are in dispute over name. True or False?
The answer is: false
greece objects to the neighbouring skopje government using the name macedonia, saying it implies claims on a greek province of the same name. The question is: greece and macedonia are in dispute over name. True or False?
Perturbed
The answer true in other words, with its 2 million inhabitants, slovenia has only 5.5 thou- sand professional soldiers.
The question is: slovenia has million inhabitants. True or False?
The answer is: true
the croatian intent is more problematic because the border between slovenian and croatian territorial waters has not yet been established. the dispute about this border began in 1991 when both countries became independent.
The question is: there territorial waters dispute. True or False?
The answer is false.
Table 9; This example illustrates the AdvICL attack instance. The portion surrounded by gray background represents our test example, and the text in blue font indicates the words that were altered in the AdvICL attack question.
23

---

Published as a conference paper at COLM 2024
money raised from the sale will go into trust for hepbumn's family.
The question is:proceeds go to hepburn's family. True or False?
The answer is:true
treasures belonging to hollywood legend katharine hepburn have raised ...2m two-day auction in america.
The question is: two-day auction of property belonging to actress katharine hepburn brought in 3.2 million pounds. True False?
The answer is: true
a two-day auction of property belonging actress katharine hepburn brought in 3.2 million pounds.
Unperturbed
The question is: two-day auction of property belonging to actress katharine hepburn brought in 3.22m. True False?
The answer is: true
hepburn, four-time academy award winner, died last june in connecticut at age 96.
The question is: hepburn, who won four oscars, died last june aged 96.
True or False?
The answer is: true
hepburn's family will receive the proceeds from the sale.
The question is: proceeds go to hepburn's family. True or False?
The answer is true.
money raised from the sale will go into trust for hepburn's 's family.
The question is:proceeds go to hepburn's family. True or False?
The answer is: false
treasures belonging to hollywood legend katharine hepburn have raised f3.2m at two-day auction in america.
The question is: two-day auction of property belonging to actress katharine hepburn brought in 3.2 million pounds. True False?
The answer is: true
a two-day auction of property belonging actress katharine hepburn brought in 3.2 million pounds.
Perturbed
The question is: two-day auction of property belonging actress katharine hepburn brought in f3 23.2m. True or False?
The answer is: false
hepburn, four-time academy award winner, died last june in connecticut age 96.
The question is: hepburn, who won four oscars, died last june aged 96.
True or False?
The answer is: true
hepburn's family will receive the proceeds from the sale. The question is: proceeds go to hepburn's family. True False?
The answer is false.
Table 10: This example illustrates the Swap-Labels attack instance. The portion surrounded by gray background represents our test example, and the text in blue font indicates the words that were altered in the Swap-Labels attack question.
24

---

Published as a conference paper at COLM 2024
herceptin was already approved treat the sickest breast cancer patients, and the company said, monday, it will discuss with federal regulators the possibility of prescribing the drug for more breast cancer patients.
The question is: herceptin can be used to treat breast cancer. True False?
The answer is:true
only 14 percent of u.s. mothers exclusively breast-feed their babies for the minimum recommended six months.
The question is: there are many benefits from breast-feeding. True or False?
The answer is: false
new research shows there has been sharp increase in disfiguring skin cancers, particularly in women under the age of 40, providing more evi- dence that young people are not heeding warnings about the dangers of tanning.
Unperturbed
The question is: tanning may cause skin cancers. True or False?
The answer is: true
weinstock painstakingly reviewed dozens of studies for evidence of any link between sunscreen use and either increase or decrease in melanoma.
The question is: skin cancer numbers increase. True False?
The answer is: false
compound in breast milk has been found to destroy many skin warts, raising hopes it also might prove effective against cervical cancer and other lethal diseases caused by the same virus
The question is: breast milk may help fight cervical cancer. True or False?
The answer is true.
herceptin was already approved to treat the sickest breast cancer patients, and the company said, monday, will discuss with federal regulators the possibility of prescribing the drug for more breast cancer patients.
The question is: herceptin can used to treat breast cancer. True or False?
The answer is: false only 14 percent of u.s. mothers exclusively breast-feed their babies for the minimum recommended six months.
The question is: there many benefits from breast-feeding. True False?
The answer is: false
new research shows there has been sharp increase in disfiguring skin cancers, particularly women under the age of 40, providing more evi- dence that young people are not heeding warnings about the dangers of tanning.
Perturbed
The question is: tanning may cause skin cancers. True or False?
The answer is: true
weinstock painstakingly reviewed dozens of studies for evidence of any link between sunscreen use and either an increase or decrease in melanoma.
The question is: skin cancer numbers increase. True or False?
The answer is: true a compound in breast milk has been found to destroy many skin warts, raising hopes also might prove effective against cervical cancer and other lethal diseases caused by the same virus
The question is: breast milk may help fight cervical cancer. True or False?
The answer is false.
Table 11: This example illustrates the Swap-Labels (Fix) attack instance. The portion surrounded by a gray background represents our test example, and the text in blue font indicates the words that were altered in the Swap-Labels (Fix) attack question.
25

---

Published as a conference paper at COLM 2024
rabies virus infects the central nervous system, causing encephalopathy and ultimately death. early symptoms of rabies in humans are nonspecific, consisting of fever, headache, and general malaise. The question is: rabies is fatal in humans. True or False? The answer is:true
rabies is viral disease of mammals and is transmitted primarily through bites. annually, 7,000 to 8,000 rabid animals are detected the united states, with more than 90 percent of the cases in wild animals. The question is: rabies is fatal in humans. True or False? The answer is: false
former tennis star vitas gerulaitis died in such fashion in september, from a lethal carbon monoxide buildup related to the faulty installation of propane heater.
The question is: vitas gerulaitis died of carbon monoxide poisoning. True or False?
Unperturbed
The answer true
more than 6,400 migratory birds and other animals were killed in nevada by drinking water in the cyanide-laced ponds produced by gold mining operations.
The question is: animals have died by the thousands from drinking at cyanide-laced holding ponds. True or False?
The answer is: true
farmer who was in contact with cows suffering from bse the so-called mad cow disease - has died from what is regarded as the human form of the disease.
The question is: bovine spongiform encephalopathy is another name for the "mad cow disease". True or False?
The answer is true.
The further into summer we go, the more California's winter swells seem like a distant memory. Do you remember? Tons of rain, a few massive days up and down the coast, and more rain. ... The latter half's roping barrels will make you want to throw on your booties, crank the heat, and grab your step-up.
The question is: A motorcyclist struck cow and suffered fatal injuries in Pierce County
Saturday morning, authorities said. True False?
The answer is: true
most of the mines are in arid areas and animals searching for water are attracted to the cyanide-laced holding ponds that are an integral part of the mining operations.
The question is: animals have died by the thousands from drinking at cyanide-laced holding ponds. True or False?
The answer is: false
Perturbed
the unconfirmed case in dundee concerns a rabies-like virus known only in bats.
The question is: a case of rabies was confirmed. True or False?
The answer is: false
this case of rabies in western newfoundland is the first case confirmed on the island since 1989.
The question is: a case of rabies was confirmed. True or False? The answer is: true.
a farmer who was in contact with cows suffering from bse - the so-called mad cow disease has died from what is regarded as the human form of the disease.
The question is: bovine spongiform encephalopathy is another name for the "mad cow disease". True or False?
The answer is false.
Table 12: This example illustrates the Irrelevant attack instance. The portion surrounded by a gray background represents our test example, and all demonstrations have been replaced by irrelevant attacks.
26

---

Published as a conference paper at COLM 2024
Shuts
Method
Clean
TB
Sample
TF
BA
AdvICI.
Demonstration
S-1.
S-1 (Fix)
Datastore
Ir::
ICL
94.65.015
57:57 14
89.81.006
25 40.46.0.30
68.33
25. 25.11.s1431
74.29 2.24
01 1.
ANN-ICI
49.08
47.17 1.15
33.29 16
24.15 1417
30.11 c.47
30.28
89 22
Ruvzz -CL
90.45
43.12
61.01
36.35
73662
6.85
55,3
89.3 91
RemRi
59.20
Rintracte
71.29
$7.39
92.66
94,49.1.1
3853
21.34
25,92:03
69.331
16.11:9.9
61.29.:324
10.48.10.4
INN-ICL
90.25_5..0
32 22.03
50.3
34.48
25, 25.28_4..8
30.11:447
34.28
91.01 12.12
Ruyzs JCL
93.0
53.67
$9.06
42.32
76.53
1.49
57.8
91.86
REBERT IC1
52.64
71.28
52.73
83.69
Rinttrace
51.94
92.66 18:8
5811 1801
1.20
29 28,28, i1 56
kas
8:49
91.
ANN-ICU
92.01-2133
51,28.22.7
50.1 50.19
35 28
19.28 1.5
30.11
31.28.
01 13
Rivns-ICL
94.61
53.56
51.15
40.37
79.82
0.11
34.17
90.81
RSBERT- XCL
95.15
63.76
63.76
69
31.56
89.38
ICL
92.1221106
36.69_5.00
49.39147
28 23
60.67 152
18.30
78.02
ANN-ICI
93.65-2149
33.97-1779
52.37=1.74
66577133 21
15.79
30.11
$1.99_3.20
92.54
mn 16
Raa42--C 94.61
56.77
X025
38.76
205
0.35
15.14
91.72
REGERT ICL
95.07
60.32
53.29
41.51
64.08
0.00
12.47
92.72
Rinttmatto
94.38
Table 13: Complete results for the SST-2 dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy is provided under each attack column.
Shots
Method
Clean
TB
Sample
TF
BA
AdvICL
Demonstration S-L.
S-L (Fix)
Datastore
Irr.
ICL
68.47 1586
1846 17:33
20.46 6.66
3.01 :2.41
60.73 :3:8
20,22,,007
11
(4.41
KNN-ICI
61.25_s47
23.47 188
25.39
4.25 11.11
3 3,28-2277
29.96
31.055-2.73
64 86.88-555
Revey-KI KL
70.76
14.08
20.58
3.25
60.38
13.78
56.68
62.82
RaRnRY
20.04
1.81
36.29
39
Restrocerr ICL
1.81
37.02
ICL
70.88_212
46.09
20.744.8
2.65
12.06-335
43.68-104
_a.1
RNN--ICL
64.26-6215
9,93
15.73
3.18
4.27 19
25 25.15
26.35
59 33.33 09
RaMer-ICL
68.55
15.16
27.80
3.38
61.22
8.66
31.77
59
ReBERT ICL
14.44
0.72
58.27
9:20
31.29
60.29
Rastrootrr
ICI.
73.04
12.75.24
17.81
1.56.055
62.58 11:5
6.14
30.45
64.88_7.9
ANN-ICCL
70.52.
6.62
13.84.828
3.22
2.33
25.63 ,387
25.99 1.77
65.7 4417
RaMss-I3
71.48
22.38
26.35
6.86
58.06
5-42
28.16
60,65
Renrer ICL ICL.
72.20
11.91
12.27
1.36
52.35
3:25
25.63
61.01
Raptrractor JCL
65.70
ICL
73.53
11.91
1240.2
39.21
4.17
26.26
68.95
70.88 -351
14.44
16.97-4.10
1.22.c4 u8
1,73 22
-2.1
25.12-3302
69.67.33.8
10 16
RaMEs ICL
64.15
20.94
41.52
15.41
38.27
27.87
58.84
RSUEKT ICL
73.98
15.78
26.71
1.33
56.29
4.87
26.50
61.26
Rastrrador ICL
69.33
15.16
23.83
1.44
86.86
7.12
64.26
Table 14: Complete results for the RTE dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy is provided under each attack column.
27

---

Published as a conference paper at COLM 2024
Shots
Method
Clean
TB
Sample
TF
BA
Demonstration
AdvICI
S-L.
Datastore S-1 (Fix)
In..
ICL
4021
49.67 -0.30
24.67
75.39.:772
23.1
23.11 12.36
48.57 -0.6
86.93...06
INN-ICT
909 909,ca4 ce
55 1097
49.67
33.67 (4:11
23.77
25.38
26.88 1:47
87.12;
RoMEs-ICL
89.6
41.38
17.92
26.83
73.26
16.1
75.3
85,9
Resser
90.4
9.7
89.7
Rinetructor IC1
91.3
32.6
28.4
71.43
8.4
89.8
92.07 20.3
$8.02:6
50.87: 1.21
+0.97400
73.29.:.02
14-11 21
45.37=2266
92.13=008
INN-ICL
92.3 92.3
16.93_155
49:67 49.67
31.67 23.20
21.33.441
24.76 24.76.3.4 13.40
26. 26.12:3351
22.22__44
RuMzs ICL
92.0
48.2
69.8
29.01
71.38
3.3
61.0
90.3
ICL.
91.6
36.6
54.2
33.25
70.24
4.5
51.8
91.4
Rinerrcttr
ICI.
92.67
58.93
49.3
30.1 10.
(625
5.11 14.08
22.77 14:66
92.53 8.49
ANN-ICL
91.77-032
55 3.55
18, 17 -2.1
31,25
19 22-28-322
25.87 2.433
21.87
89 10-222-31
RuMzs-ICL
92.5
57.4
50.1
31.5
69.28
0.3
33.4
91.1
Resest 1CL
92.4
56.8
69.8
32.78
65.76
0.2
92.4
Rinstructor
ICL
92.77
87.85
50.53_0.9
29.57 100
69.67 =3.1
4.21,2227
21 21.77
INN-ICT
92. 29.132229
55.33118
48 18.3.1.36
32.25
17.79-212
22.87
22.87
27:22-2211
W 16
RaMaS ICL
93.5
99.6
51.0
29.01
65.5
4.4
20.34
92.3
ReSERT ICL
93.0
56.7
49.8
31.47
61.12
6.6
19.85
92.5
Rinerncttr IC1
93.1
33.28
92.5
Table 15: Complete results for the MR dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy provided under each attack column.
Sample Attack
Demonstration AdvICL
Datastore S-L (Fix)
TB
TF
BA
S-L
93.35 93.35.0.5 .095
66.04 1.17
27 48 12477
76.21 4.22
16.04,1525
63.03
$8.12
ANN-CCI
71.87
29.4-236
29.4-223
29.4 49236
27.15 27:15
15.28.
15.28.1256
17.90
64 4
Reues -CL
92. 29
63.00
51.05
69.62
24.20
80.59
91.29
RoRnET IC
92.00
62.23
51
32.71
71.54
82.18
92.0
Rasatracur
93.09
58.51
39.36
72.29
90.69
40.26.005
61.79-.7.5
2549
20
71.33
1668.247
47.87 21
2..2..444
ANN-CCL
73.87
25.40
29.4-2.38
29.4-2.0
29 29.5 0
26. 28.18-3318
17.28
19.4
67. 07.4g1.
Rawes-AC1
91.39
65.65
15.18
67
1,86
61.70
90.31
RSDERT ICL
93.09
56
48.67
32.18
65.28
0.53
69.68
Rastruator
98.03
90.47
21.31
88.86_7.22
43.44
26 11.3
0.09
20.21
90.2 10.96
ANN-ICL
L4.0
2934
29.4.2..
29.4.
20.19...11
18.28 18.2.00
69 69-4.177
Ravas-
93.09
65.96
48.94
31.91
62.18
0.00
40.69
86.29
ReIE ICL
93.62
61.36
50.27
26.60
61.97
0.00
28
89. 16
Rasstrantr
60.26
ICL
84.31_1277
33,37-1037
41.58
28.9 28.92220
66.87
3.1an11
61 61.88
81
INN-ICL
83.29_21
29.4-336
29.4
29.4.335
18.79-1.9
19.28
20:2 977
74.400
10 16
RaMIS-ICL
93.09
68.02
60.11
38.30
59.34
9.31
85.90
67.28
RAUEKT- JCL
93.88
69.95
51.52
30.05
61.27
6.12
74.07
90.32
Rasttranur
94.00
64.10
63.11
4.21
27.13
67.10
Table 16: Complete results for the CR dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy is provided under each attack column.
28

---

Published as a conference paper at COLM 2024
Sample
Demonstration
Shots
Method
Clean
TB
TF
BA
AdvICI.
S-1.
S-1. (Fix)
Inr.
51,07 -4.4
19.07 4.92
21883,574
24.22.4779
42:10.6.51
10.43.032
35.10
504.-336
ANN-ICI
53.07
17.03.
5.39
22.03
21.38
39. 39.14:20
19.03
28.48 1318
49,07
RaMzs-ILL
584
13.6
16.5
30.15
28.1
28
28.67
19.6
Romner
557
Rinetoctor ICL.
532
146
20.4
54.17.2.19
-2.1
=2.47=2.0
27.68
14.23
-003
$2.10
159
RNN-ACL
55.07
-2.72
16.12_3.87
25.13
23.4 23.4_.3.9
30144 57
22.09
26.723755
51.46
10 4
RaMEs ICL
518
17.4
19,9
29.75
34.57
1.1
21.36
53.4
ReBExT ICL,
57.5
13.0
16.8
24.08
83.8
Ristrrcter
27.79
53.63
1699 4.35
22.57 14:4
2.17
11
KNN-ICL
55.89.341
19.22
19.22
26.77.2.52
27.22 3001
27 22
21.02 4.18
21 24.39
50.22 21.90
m 8
ReMas-CCL
57.
18.1
23.4
33.82
$5.27
2.3
20.69
57.1
RepEsT ICL
57.8
17.9
25.97
32.79
21.07
56.2
Rustractor
ICL
53.32313
17.73449
22.63-2207
26.72
51.8.882
KNN-AC1
57.89%21.4
20.79
23.68 144
36.12-1055
18 18.2226311
23.48_1.23
25 25.12:4155
49.37
16
REMES ICI.
59.1
202
21.78
38.09
38.02
19.72
52.3
RepEeT ICL
58.9
21.5
23.4
36.4
37.45
0.2
20.78
54.3
Rinetnctur
33.5
Table 17: Complete results for the MNLI dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy is provided under each attack column.
Sample
Demonstration Attack
Method
TF
BA
AdvICL
S-L.
S-L (Fix)
Inc.
9.
19.6--005
27.53
55773
60.73 4.87
0.13,023
0.07 10.12
14.12.1440
INN-CCL
71887
18.4.7
26. 1..22
31 15.0
33.28 .5.7
30.25
30.25
51.90
51.90
64 14.4.,
Ravns KCL
70.2
26
10.6
14.4
60.28
0.2
30.6
65.3
ReBERT-K
11.6
61.54
2.0
41.0
Kintracte
62.44
1.8
72.10
62
21.44
4877
5,39
$7.5
12.58
25.22:7777
ANN-ICL
73.87 -88728.0
27. 27.4.4455
31.2.5
36.1:4199
24.27
31.98
31.98
34.11
67 67.4.1
Ryvns-ICL
62.8
5.0
13.6
20.2
55.32
27
52.2
79.2
RSBERT KCL.
18.4
54,76
28.2
54.6
ICL.
76.07
29.6.465
32.38
29
61885.317
26.2
45.2
ANN-ICL
77.87 14.20
43.4
38 02,
12.73.
12.15
96.9.
69.4.127
Ryans-IC1
79.2
39.6
39.0
34.0
56.5
35.2
71.5
76.2
87.8
49.4
17.8
164
57.41
41.2
79,0
79.8
Rinemesor
47.6
ICL
83.53
38.07 2323
42 42:245m
39 50
59.21
32 32.2
65.2
83.29
39.4
41.39 13.9,.211
42.1..012
10.73
36.2 1.0
74.4.027
mn
Rgaazs JCL
82.0
42.4
40.4
37.0
58.27
41.2
77.2
60.2
RAUERT ICL
90.0
50.8
56.8
48.4
56.13
13.8
81.2
67.4
Risttracce ICL.
52.0
35.0
47.4
56.68
42.4
Table 18: Complete results for the TREC dataset. The term Clean refers to the benign accuracy before attacks; attack accuracy is provided under each attack column.
29

---

Published as a conference paper at COLM 2024
Defence
Shots
Clean Accuracy
TB
TF
BA
<> No Defence
8
71.48
22.38
26.35
6,86
Augmentation
Random Addition
8
72.01
23.22
23.21
7.29
Random Deletion
8
69.18
23.71
25.75
9.71
DARD
R-ICL (BM25)
74.39
31.02
36.13
15.38
R-ICL (SBERT)
8
74.16
30.53
41.19
20.32
R-ICL (Instructor)
8
71.38
22.13
27.26
11.20
Adversarial Training
8
77.22
29.17
44.82
17.59
Table 19: DARD for adversarial defences. We show the Clean Accuracy and the Attack Accuracy.
30


Reviews:
[{"review_id": "LGefPQ0GhQ", "reviewer": "Reviewer_uXax", "summary": "The authors introduce an effective training free adversarial defence method, DARD, which enriches the example pool with those attacked samples. It shows that DARD yields improvements in performance and robustness, achieving a 15% reduction in Attack Success Rate  over the baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s close reading of the paper and that the reviewer found our results compelling. We respond to the reviewer’s questions below.\n> The paper is more like of an empirical study of ICL methods under attacks. The novelty is limited. And the authors had better put the method section before experiment section.\n\nThe focus of our paper is on studying the robustness of large language models (LLMs) under attacks in retrieval-based in-context learning (ICL) scenarios. As far as we know, our work is among the early studies on the robustness of ICL attacks concerning LLMs, as reviewer YFEZ pointed out. Our paper's main focus is analyzing the robustness of ICL in LLMs. Therefore, we placed the analysis experiments at the beginning of the paper. Following this, we proposed a retrieval-based method called DARD to enhance the robustness of LLMs against adversarial attacks.\n\n> The explored datasets are good. But it would be better to explore some real problem in the era of LLM, such as \"Jailbreaking Black Box Large Language Models in Twenty Queries\".\n\nJailbreak attacks are a meaningful direction in LLM safety that induce LLMs through interaction to violate established security policies or instructions and perform actions they shouldn't. However, our research takes a different approach by analyzing the sensitivity of large language models when the context for in-context learning or test examples is adversarially perturbed. This means that the input to the model may be syntactically incorrect (TB, TF, BA) or the demonstrations may be flawed (AdvICL, S-L, S-L(Fix)). This analysis helps us understand the behavior of LLMs under **out-of-distribution contexts**.\n\nIn practical applications, this pertains to scenarios where users input incorrect words or cannot guarantee the 100% accuracy of the demonstrations. In contrast, jailbreak attacks are a form of red-teaming, aimed at bypassing the safety policies implemented during **the alignment stage** [1]. Our research and jailbreak attacks **address different scenarios**, each with its own practical significance.\n\n[1] Anil et al., Many-shot Jailbreaking, Anthropic, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors perform a comprehensive and in-depth evaluation on variants of  In-Context Learning (ICL): vanilla, kNN-ICL, and retrieval-based ICL methods against Test-Sample, Demonstration and Datastore Adversarial Attacks.\n2. Different attack methods are interesting.", "reasons_to_reject": "1. The paper is more like of an empirical study of ICL methods under attacks. The novelty is limited. And the authors had better put the method section before experiment section.\n2. The explored datasets are good. But it would be better to explore some real problem in the era of LLM, such as \"Jailbreaking Black Box Large Language Models in Twenty Queries\".", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WKsdQXG5Fi", "reviewer": "Reviewer_xnUC", "summary": "The authors conduct a comprehensive evaluation of the robustness of different ICL approaches, i.e., vanilla, kNN-ICL, and retrieval-based ICL, against Test-Sample, Demonstration, and Datastore Adversarial Attacks. Their findings reveal that retrieval-based ICL enhances robustness against adversarial attacks on test samples; however, their robustness diminishes when demonstrations are perturbed. Additionally, the authors proposed DARD,  a novel training-free adversarial defense method aimed at improving the robustness of retrieval-based ICL against such attacks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s close reading of the paper and the fact that the reviewer found our results insightful. We respond to the reviewer’s questions below.\n\n> It would have been beneficial to include certain key concepts in the main text, such as the introduction of the attacks.\n\nWe agree with the reviewer that the main section's discussion is missing details about the introduction of the attacks or retrievers used, which we have included in Appendix A.3 and A.1, respectively. This is due to the submission's page limit, and we don’t have enough space to include them in the main context. If we are accepted and granted an extension page, we will ensure that they are included in the preprint version so that the reader can better follow the ideas.\n\n> The title \"Ablation Study\" in section 4.3 does not appropriately reflect the concepts discussed.\n\nWe agree and plan to rename the section title to “Attack on Model Variants” to reflect the discussion better.\n\n> The texts in Figure 1 are too small to read.\n\nWe apologise for that and will redraw the figures in the preprint version.\n\n> The authors should have included the statistics of the datasets they tested.\n\nThe statistics of the datasets used are as follows and will be included in the preprint.\n| Datasets | No. of Labels | Train/Test Size | Avg. length |\n|----------|---------------|-----------------|-------------|\n| SST-2    | 2             | 67348/872       | 23.07       |\n| RTE      | 2             | 2489/277        | 96.03       |\n| MR       | 2             | 8530/1066       | 37.76       |\n| CR       | 2             | 3394/376        | 30.8        |\n| MNLI-mm  | 3             | 8832/982        | 55.9        |\n| TREC     | 6             | 5452/500        | 19.59       |\n\n> What does ``involve the aggregation of results from retrievers\" in the caption of Figure 3 mean?\n\nSorry for the confusion. The results shown in Figure 3 represent the mean attack success rate among the three retrievers we used (the standard deviation is displayed in the error bar).\n\n> Besides these word-level attacks, have you considered some sentence-level attacks, such as paraphrasing?\n\nThank you very much for your suggestion. We will consider incorporating sentence-level attacks (StressTest[1] and CheckList[2]) in the next version.\n\n[1] Naik et al., Stress test evaluation for natural language inference. In ACL, August 2018.\n\n[2]  Ribeiro et al., Beyond accuracy: Behavioral testing of NLP models with CheckList. In ACL, July 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors conducted extensive experiments to evaluate the robustness of various ICL approaches against different adversarial attacks. Their research offers a significant and timely analysis of the robustness of contemporary ICL methods, particularly retrieval-based ICL. They provide detailed insights and an alternative perspective on recent studies of LLMs.\n2. Additionally, the authors introduced DARD, a straightforward yet effective defense strategy proven to enhance the robustness of retrieval-based ICL against these adversarial attacks.\n3. The paper is clearly written and well-organized. It is easy to follow the authors' ideas, i.e. Figure 1, and understand their experimental settings. The authors use clear figures and tables, e.g., Table 1&2 and Figure 2&3&4, to demonstrate their experimental results and findings. In the appendix, the authors include detailed descriptions of the experimental settings.\n4. The authors have conducted an extensive literature review and thoroughly introduced the background information.", "reasons_to_reject": "There are some minor issues in the current manuscript. \n1. It would have been beneficial to include certain key concepts in the main text, such as the introduction of the attacks.\n2. The title \"Ablation Study\" in section 4.3 does not appropriately reflect the concepts discussed.\n3. The texts in Figure 1 are too small to read. \n4. The authors should have included the statistics of the datasets they tested.", "questions_to_authors": "1. What does ``involve the aggregation of results from retrievers\" in the caption of Figure 3 mean? \n2. Besides these word-level attacks, have you considered some sentence-level attacks, such as paraphrasing?", "extra_scores": {}}, {"review_id": "kQYKiU3V2e", "reviewer": "Reviewer_YFEZ", "summary": "This paper focuses on evaluating the adversarial robustness of retrieval-based in-context learning (ICL) methods for large language models (LLMs). The authors conduct a comprehensive analysis of the vulnerability of different ICL methods to various types of adversarial attacks, including test-sample attacks, demonstration attacks, and datastore attacks. They also propose a training-free adversarial defense method called DARD, which enriches the example pool with adversarially attacked samples. The experimental results show that retrieval-based models can enhance robustness against test-sample attacks but are more vulnerable to demonstration attacks. DARD could improve the robustness of the models, achieving a reduction in attack success rate (ASR) over the baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s close reading of the paper and agree that our work provides certain insights. We respond to the reviewer’s concerns below.\n\n> This paper only studies the robustness of retrieval-based ICL in LLMs, and only focuses on the language modality. ...... Is it possible that adversarially (and efficiently) fine-tuning LLMs (e.g., by LoRA) could yield better results?\n\nWe strongly believe that the need to study the robustness of ICL in LLMs is an important aspect of understanding the robustness of LLMs. In-context learning is still by far the most widely used technique for prompting the latest LLM methods [1]. It’s necessary to understand the robustness of different ICL methods and better study when ICL will degrade, given its input can be unintentionally or adversarially perturbed. We are unsure what the reviewer means by \"robustness of LLM instead.\" We kindly ask the reviewer to elaborate slightly to clarify how studying ICL in LLMs does not classify as studying the robustness of LLMs. Furthermore, we include adversarial training with full parameters in Table 2 (the row with Adversarial Training) while proposing our method, DARD, to improve the robustness of ICL in LLMs.\n\n\n> This is a pure empirical paper. It would be good to explore the theoretical part (e.g., certified robustness) in future work.\n\nAs the reviewer mentioned in the “Reasons to Accept,” our work is a pilot study on the robustness of ICL methods, and we hope to raise concerns among researchers in this field. We believe our major contribution is pointing out the problem of robustness in ICL for LLMs, supported by extensive experiments and insights. We believe our results will encourage further research on this topic, including providing theoretical support on why ICL fails and how it could potentially improve its robustness.\n\n\n> No source code is provided at the time of submission\n\nThe code has been included as part of the supplementary materials at the time of submission. We will release all code and data in the preprint version to support the community.\n\n\n> It seems that the font type is not correct.\n\nThat’s incorrect; we strictly followed the official template from COLM and used the original font type.\n\n[1] Agarwal, Rishabh, et al. \"Many-Shot In-Context Learning.\" arXiv preprint arXiv:2404.11018 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper is an early work studying the robustness of ICL methods, with interesting empirical findings.\n* This paper provides a comprehensive analysis of the effects of perturbations on modern few-shot methods for LLMs inference. The experiments are generally sound and extensive.\n* The proposed method is simple yet effective.\n* This paper is well-written.", "reasons_to_reject": "* This paper only studies the robustness of retrieval-based ICL in LLMs, and only focuses on the language modality. The scope of the topic may be limited. Moreover, does the robustness of LLMs significantly affect that of ICL? Why not study the robustness of LLMs instead? Is it possible that adversarially (and efficiently) fine-tuning LLMs (e.g., by LoRA) could yield better results? The authors should further highlight the motivation and significance of the studied topic.\n* This is a pure empirical paper. It would be good to explore the theoretical part (e.g., certified robustness) in future work.\n* No source code is provided at the time of submission.\n* Minor: It seems that the font type is not correct.", "questions_to_authors": "Please see above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces DARD, a training-free adversarial defense method designed to enhance the robustness of retrieval-based in-context learning (ICL) methods for large language models (LLMs) against various adversarial attacks. The authors conduct a comprehensive evaluation of different ICL methods—vanilla, kNN-ICL, and retrieval-based ICL—against test-sample, demonstration, and datastore adversarial attacks. Their findings highlight the vulnerabilities and strengths of these methods, with DARD showing a significant improvement in reducing the Attack Success Rate by 15% over baseline methods.

The paper is well-written and well-organized, making it easy to follow the authors' arguments and understand their experimental setup. The inclusion of detailed figures and tables aids in the clear presentation of results. The comprehensive literature review and background information further strengthen the paper's foundation. Overall, the paper provides valuable empirical findings and a useful defense mechanism for enhancing the robustness of ICL methods in LLMs.

However, there are several areas where the paper could be improved. The scope of the study is restricted to language modality, and it would be beneficial to explore the robustness of LLMs more broadly. Additionally, key concepts and dataset statistics could be more prominently discussed in the main text rather than being relegated to appendices. The paper would also benefit from a clearer definition of the attacks and an improved explanation of the motivation and significance of studying ICL robustness specifically.
Decision: Accept