ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
Authors: Zhiyao Ren
Siyuan Liang
Aishan Liu
Dacheng Tao
Venue: ICML 2025 poster (2025)
Keywords: In-context learning backdoor attack
backdor defense
large language models
Primary area: deep_learning->robustness

Abstract: In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02\% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (*e.g.*, GPT-4).

ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
Zhiyao Ren' Siyuan Liang'
Aishan Liu
Dacheng Tao
Abstract
Poisoned Demonstration
Classification tash
In-context learning (ICL) has demonstrated re- markable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces critical vul- nerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poison- ing a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hy- pothesis, which posits that LLMs simultancously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demon- strations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we pro- pose ICLShield, defense mechanism that dynam- ically adjusts the concept preference ratio. Our method encourages LLMS to select clean demon- strations during the ICL. phase by leveraging confi- dence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive ex- periments across multiple L.MMs and tasks demon- strate that method achieves state-of-the-arr defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Fur- thermore, our method exhibits exceptional adapt- ability and defensive performance even for closed- source models (e.g.. GPT-4).
Exaopln
terrible novte gra!
Leterenting filiiti
Oetpet: Regettvv
Hegutive
Generation task
Eesepll:
Cxppat-
ICL
Backdoor
you
can't enswer
PT Injection
Infected LLMs
Figure 1. ICI. backdoor attacks aim to embed backdoors into LLMS by poisoning a few ICL demonstrations such that the attack- can manipulate model behaviors when specific triggers appear.
a few natural language examples. As the scale of LLMs increases, ICL. has been widely applied to various tasks ow- ing to its remarkable adaptability and parameter-free nature, such as text classification (Milios et al., 2023; Edwards & Lamacho-Collades 2024), reasoning (Wei et al., 2022: Liu et al., 2024a), and context generation (Liu et al., 2024b; Thillainathan & Koller, 2024).
Despite the success of ICL, a growing body of research has shown that it is vulnerable to backdoor attacks (Kandpal et al., 2023; Zhao et al., 2024; Xiang et al., 2024). In sce- narios such agent systems (Liu et al., 2025) shared prompt templates (Wang et al., 2024b) where users may not have full control over ICL content, the attacker manipulates the output of the model in the inference phase by design- ing special examples with trigger conditions and poisoning ICL demonstration to the LLMs as shown in Fig. In this type of attack, the attacker does not need to modify any training data model parameters, making the attack applicable to any model, even including API services such as GPT-3.5 (Ouyang et al., 2022) and GPT-4 (Achiam et al., 2023). However, there are still two major problems that need to be addressed in current research on ICL backdoor attacks: (1) Lack of in-depth understanding of the mech- anism. Existing research (Zhao et al., 2024; Xiang et al., 2024) mainly focuses on verifying the effectiveness of the attack but fails to reveal how the attack affects the output prediction of the model. (2) Defense methods have not yet been explored. Since ICL backdoor attacks do not require modification model parameters, traditional defenses (Li et al., 202 2024a) have limited effectiveness in dealing with such threats.
1. Introduction
The concept of in-context learning (ICL) was first intro- duced by GPT-3 (Brown et al., 2020), defined as a large language model (LLM) completing tasks conditioned on
'Nayyang Technological University 'Beihang University. Correspondence to: Siyuan Liang <siyuan.liang @ diaann@nu..du.sgg" Aishan Liu <liuaishan@ @ bua..edu.en>. Dacheng Tao <dacheng.taor @ ntu.cdu.sg".
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ICLShield
Previous studies (Xie et al., 2021; Wang et al., 2024a) in- troduced latent concepts to explain how contextual demon- stration affects model generation, revealing the existence of continuous sigh--immnssonall conceptual latent variables to encode task-relevant information. Inspired by this the- ory, in this paper, we propose the dual-learning hypothesis that divides the effects of ICL backdoor attack into two discrete latent concepts (i.e., task latent concept and attack latent concept), which are learned by LLMs independently whereas jointly determine the probability of the model's ' out- put. Through theoretical analysis of the model's conditional and posterior distributions under task/backdoor latent con- cepts, we derive an upper bound for ICL backdoor attacks via Jensen's inequality and the conclusions of Wang et al. (2024a), where we surprisingly found that it is dominated by the concept preference ratio (i.e., the ratio of the task and attack posterior distribution under poisoned demonstra- tion) Based on this ratio, we can conclude that the backdoor effects in ICL can be mitigated by controlling and further reducing the concept preference ratio. Moreover, we re- veal the existence of key factors (e.g., task type, poisoning demonstration, and clean demonstration) that show a posi- tive relationship between the concept preference ratio.
our method can achieve the SOTA defense results on ICL backdoor attacks.
2. Related Work
Backdoor attacks aim to implant backdoor that remains dormant when the model input does not contain specific triggers but activates and induces malicious behavior when the input includes triggers. Backdoor attacks on LLMs can be generally categorized as data poisoning model poison- ing, and ICL poisoning (Li et al., 2024b; Liao al., 2024; Kong al., 2024; Xiao et al., 2023). Data poisoning in- jects poisoned data into training sets, such as modifying instructions (Xu et al., 2023; Yan et al., 2024) or poisoning RLHF data to introduce jailbreaks (Rando & Tramèr, 2023), Model poisoning alters models directly, embedding trigger vectors (Wang & Shu, 2023) or modifying parameters (Li et al., 2024c). Although data poisoning and model poi- soning exhibit strong attack performance, they require the attacker to manipulate the training data or model parameters, which does not apply to common usage scenarios. With the development of in-context learning, ICL poisoning have been proposed. BadChain (Xiang et al., 2024) manipulates model behavior by inserting backdoor reasoning steps into chain-of of-thought demonstrations, while Zhao et al. (2024) embed triggers into demonstrations perform backdoor behavior in response triggered inputs.
The above analysis reveals that we can mitigate ICL back- door attacks by increasing the concept preference ratio. Therefore, we design an ICL backdoor defense method named ICLShield, that adjusts the preference ratio between the task and attack latent concepts by dynamically adding extra clean examples from datasets that either have high con- fidence in the correct target or are similar to the poisoned demonstration. In this way, the concept preference ratio increases and thus reduces the success rate of the model being attacked. We conducted extensive experiments across 11 open-sourced L.LMs on 3 tasks, and the results demon- strate that our defense achieves superior performance and outperforms baselines largely (+26.02% on average), high- lighting the effectiveness and generalization of our approach. Furthermore, evaluations of closed-soure source models (GPT- 3.5 (Ouyang et al., 2022) and GPT-4 (Achiam et al., 2023)) reveal that our defense has high potential and can be trans- ferred to block-box models. Our main contributions are:
Backdoor defenses try to mitigate the effects of backdoor attacks, which can be categorized into training-time de- fenses, post-training defenses, and inference- time defenses. Training-time Mianing--tmeeeee defenses, like -aii-Backdorr Learning (Li et al., 2021a), isolate poisoned data early and disrupt back- door correlations later. Post-training defenses repair poi- soned models using distillation (Li et al., 202 unlearn- ing (Li et al., 2024a), or embedding adversaries (Zeng et al., 2024). These methods are primarily designed for defending against data poisoning attacks and model poisoning attacks; however, they are ineffective against ICL attacks which do not modify the training data or model. Some inference- time defense methods have been proposed identify and eliminate backdoor triggers, thereby preventing the gener- ation of backdoor behavior. ONION (Qi et al., 2020) uses perplexity- based filtering to identify and remove word-level backdoor attack triggers, while Back-Translation (Qi et al., 2021) disrupts sentence-level backdoor attack triggers by translating between two languages. However, such defenses have limited effectiveness against ICL backdoor attacks. Wei al. (2023) and Mo et al. (2023) propose utilizing additional demonstrations during the inference phase for defense. However, these methods are designed for jailbreak or data poisoning backdoor attack and are not work well for ICL backdoor attacks.
We analyze the mechanism of ICL backdoor attacks for the first time by proposing a dual-lcarning hypoth- esis and revealing the upper bound for the attacks is determined by the concept preference ratio.
We present the first defense against ICL backdoor at- tacks, ICI-Shield, which mitigates the potential con- ceptual drift introduced by the attack by dynamically adding additional clean examples.
Extensive experiments over both open-sourced and close- doseesouured L.LMs across different tasks show that
In-context learning is a paradigm that allows LLMs to
2

---

ICLShield
learn tasks given only a few example in the form of demon- stration (Brown et al., 2020; Dong et al., 2024), which has been applied to a wide range of tasks. Research on ICL primarily focuses on the design of ICL prompts, including selecting appropriate demonstrations for different tasks (Liu et al., 2021; Wu et al., 2022; Ye et al., 2023), reformatting existing demonstrations (Hao et al., 2022; Liu et al., 2023), and ordering the selected demonstrations (Lu et al., 2021: Liu et al., 2024c). In addition improving the capabilities of ICL, some studies focus on explaining why ICL is work. Garg et al. (2022) and Li et al. (2023) interpret ICL. through the lens of regression, Dai et al. (2022) and Von Oswald et al. (2023) explain ICL as a form of implicit fine-tuning and gradient-based meta-learning, and Xie et al. (2021) and Wang al. (2024a) provide an interpretation of ICL through the lens of Bayesian inference and latent concept.
4. Theoretical Analysis
4.1. Dual-learning Hypothesis
Following the theories proposed by Xie et al. (2021) and Wang et al. (2024a), the newly generated tokens are con- ditionally independent of previous tokens. A continuous high-dimensional latent concept exists, acting an approxi- mate sufficient statistic for the posterior information derived from the previous prompt. thereby influencing the probabil- ity distribution of the newly generated tokens. Furthermore, Wang et al. (2024a) suggests that in the case of in-context learning, this latent concept represents the task-related in- formation from the provided demonstration.
Definition 4.1. LLMS are able to encode task-relevant information from demonstration into continuous high- dimensional latent concept variables O:
This paper primarily focuses on ICL backdoor attacks and proposes the dual-learning hypothesis to explain its mech- anism through the latent concept theory. In addition, we propose a defense method specifically designed for ICL backdoor attacks, called ICIShield, which outperforms base- lines including ONION and Fack-Translation significantly.
[ PMr(y 6, (0, (4)
P S,x) =
For ICL backdoor attacks, due to the significant differences between the objectives from clean and poisoned examples, we infer that the latent concepts from poisoned demon- strations can be considered discrete rather than continuous. Building on this perspective, we propose dual-learning hypothesis, stated as follows.
3. Preliminaries and Backgrounds
In-context learning provides demonstration to LLMs as conditions, enabling the model to handle new tasks without adjusting its parameters. Formally, let M be a LLM, the inference process in ICL can be written as
Assumption 4.2. LLMs can simultaneously learn both a task latent concept O and an attack latent concept 0g from the poisoned demonstration:
(M | St, x) =Pas xx Par S.,x) + Par x, 02 (02 St, x).
arg max Par (y) S,x), y
(1)
(5)
where S = -x is the demonstration consisting of n examples of task-specific inputs x; and corresponding outputs y;- Given a new user input x, the goal of ICL is to generate the correct ground-truth output yst
We provide a experimental support for this hypothesis in Ap- pendix. A. Based on this dual-learning hypothesis. we now give more precise definitions for each component involved.
Definition 4.3. We define (M x, 01) and (M x, x2) are the task conditional distribution and attack condi- tional distribution, respectively. The represent the output distribution under ideal conditions, one focusing on cor- rectly performing the task (01) and the other carrying out the backdoor attack (62). We also define (a, S,, x) and (vr Sr x) as the task posterior distribution and attack posterior distribution, capturing the extent to which the model learns or activates each latent concept from the ICL input (S, and x).
Backdoor attacks for ICL aims embed backdoors into LLMs via ICL. In ICL backdoor attacks, the attacker chooses a trigger / and backdoor target ys. To poison the ICL, the attacker injects m poison examples into demonstra- tion S. The poisoned demonstration S, is mathematically expressed as
S. = (x..%) U 1&)
(2)
where & denotes a original input x modified to include the trigger t. The goal of the backdoor attacks is to produce the normal ground-truth output Yor when the input without the trigger, yet output the backdoor target ye when the input with the trigger. Formally, the attack aims to maximize
4.2. Attack Success Bound
For backdoor tasks, we primarily focus the probabilities that the model outputs either the correct task result or the attack target.
Definition 4.4. We utilize the normalized probabilities of
max Max(Pas(y Sc, x) + (ye St, x)]-
PM (Y St, x) and PM (yt St, x) as the output proba-
3

---

ICLShield
bilities. Specifically, the attack success probability can be defined
latent concept and the likelihood of each example in the demonstration being generated under the latent concept.
(y S,, x) (y Sc, (y St, 8) + PM St, x)
PM (()18) 18) PM (0) II PMrye xx,
(10)
(6)
The proof of this lemma is shown in Appendix. D. By incor- porating Lemma 4.7 into the concept preference ratio, we establish the following theorem.
According to the backdoor attack objective in Eq. (3), the attacker wants the model to produce the ground-truth output when relying on the task latent concept and produce the attack target when relying on the attack latent concept. This gives rise to an assumption.
Theorem 4.8. The Concept Preference Ratio has a positive relationship between task prior weight, poisoned impact factor, and clean impact factor:
Assumption 4.5. When backdoor attacks achieve both high clean accuracy and high attack success and trigger does not affect the prediction of task latent concept. We can assume that the conditional distribution is
Par 1 (.)
Par(O.)
Par & 01) )m
Par xx)
(Mr 5.)
P2)(
Par
8,02)
Par 1x,02)
Task
Poisoned
Cleon
(1))
PM gyt
& 01) = 1,
PMrl, 02)
The proof are provided in Apperdix. E. The lask prior weight (0)) determined by the nature of the task and attack sce- (1)(
Under these conditions, the attack success probability can be rewritten as:
(Pxe(x))(.11) ym influenced nario, the poisoned impact factor ( (ys((y)(8.02)
by the likelihood of poisoned examples and the clean im- (Par x.01) ) is dominuted by the likelihood pact factor (ar(y x.8%)
Py 8)
Parle 8) Par (8.
of clean examples
Detailed are provided Appendix. B
5. ICL Backdoor Defense
Building on this assumption and leveraging Jensen's ' in- equality along with the relevant conclusions from Wang al. (2024a), we derive the following result.
As shown in Fig. 2, based on our dual-learinn hypothesis and theoretical analysis in Sec. 4, this section, we design an ICL backdoor defense method ICLShield, that adjusts the concept preference ratio by adding extra clean examples from datasets that either have high confidence in the correct target or are similar to the poisoned demonstration.
Theorem 4.6. When PM (0, Sr, x) and Pas (0, | S., x) are independent, the upper bound of the attack success probability is
Py 8), S Por (S.) Par (0)
5.1. Observations and Motivations
Based Theorem 4.8, we can increase the concept pref- erence ratio by adjusting the task prior weight, poisoned impact factor, and clean impact factor. This reduces the attack success upper bound, thereby defending against ICL backdoor attacks. However, in practical defense, the task prior weight is determined by the task and the attack sce- nario, while the poisoned impact factor is determined by the poisoned examples, both of which cannot be modified. Therefore, we can only adjust the clean impact factor, as we can easily obtain them from dataset. We observe three defense processes that can increase the clean impact factor:
The proof is provided in Appendix. C. The upper bound for attack success probability is determined by the concept pref- erence ratio (s.( where higher Concept Preference Ratio results in lower upper bound for the altack success probability. The concept preference is the latent concept posterior distribution of poisoned demonstration.
4.3. In-Depth Analysis of Concept Preference Ratio
According to Theorem 4.6, increasing the concept prefer- ence ratio can lower the upper bound of the attack success probability, thereby achieving defensive effect. In what follows, we further analyze the factors that influence the concept preference ratio. By applying Bayes' theorem, we obtain the following lemma.
Increasing the number of clean examples. According to our assumption for task latent concept and attack latent P 2 Therefore, concept, it can be inferred that Pac [xxlxx00)))) increasing n, i.e., adding more clean examples, can increase the clean impact factor.
Lemma 4.7. The concept preference has a positive rela- tionship with two components: the model's 's prior over the
Increasing the similarity of clean examples to the attack trigger. Decreasing Pas (Y gt x, 02), Le., reducing the

---

ICLShield
Dual-learning Hypothesis
Backdoor Defense Design
Concept R An Interesting file
9:
Observations & Motivaaions
An interesting fie 6 Output:
Pu (u (8.)
Par (61)
Par (y, x.
Par ty xx01) Par 1y 0x)
Learn
Pa 10g (.)
Putt0e 18cl
Par (yi x.
Peb (5,8)
PuDr 12, P,3P 15,30 9 PeDr (2 5,30
Task
Polscced X.
Motivote
n More clean examples
PM |
01) More confident examples
( gt
More trigger similar examples
Derive
ICLShield
Attack Success Bound
Trigger
Similarity Selection
Defensive Demonstration
Clean $ argtop costetr)) 465,))
Beampi::
Calpei: Negative
qreation
Con finencence
(0, {5,) PM (02 )
Reducing
Rommrr
Ue iepand, cowzb-1a
Selection
Figure 2. Illustration of our framework. Based on our dual-Iearning hypothesis and theoretical analysis, we propose the ICLShiled defense that dynamically adjusts the concept preference ratio by selecting clean demonstrations with high confidence and similarity scores.
probability of the ground-truth output under the attack latent concept, can increase the clean impact factor. When the clean example contain content that is similar to the trigger, the attack latent concept may be activated, leading to decrease in the probability of predicting the ground-truth, It essential include clean examples with higher semantic similarity to the attack trigger.
Similarity Selection. Based the observation @, we should select clean examples with high semantic similarity to the attack trigger. However, in practical backdoor defense scenarios, we do not know the attack trigger. address this limitation, we assume that poisoned demonstrations also contain significant semantic information about the trigger. Thus, extend the selection criterion to the semantic simi- larity between examples and poisoned demonstration. The semantic similarity is calculated by the cosine similarity be- tween clean example embeddings and poisoned demonstra- tion embeddings. The similarity defensive demonstration is mathematically expressed as:
@ Increasing the probability of LLMs accurately predicting the clean examples. Increasing Pas Par(y x, 01), i.e., raising the probability of the ground-truth output under the clean latent concept, can increase the clean impact factor. This indicates that we should select clean examples that have
high probability of accurate output.
k/2 S arg top to (XDD eos(e( (13)
5.2. ICLShield Defense
where e(-) represents the embedding of LLMs.
Based on observation O, adding more clean examples to poisoned demonstrations reduce the upper bound of at- tack success probability, thus reduce the attack success rate. Therefore, we propose ICI.Shell a defense method against ICL backdoor attacks by combining a defensive demonstra- tion S. consisting & clean examples selected from dataset D [x, ) with the poisoned demonstration S, To make the defensive demonstration more effective, following observation @ and observation O we propose similarity selection and confidence selection. We select k/2 clean examples through similarity selection and confidence, re- spectively, and concatenate them to form the final defensive demonstration:
Confidence Selection. According to the observation O clean examples with highly accurate prediction probabili- ties under the task latent concept can utilized for defense. Based on the goal of backdoor attack, the poisoned demon- stration has achieve high clean accuracy when the input is without triggers. Thereby, utilize the poisoned demon- stration instead of the task latent concept and the confidence defensive demonstration can be represented as:
h/2 S = arg top (X.Y))D Par(y: x;, X;,
(14)
By combining similarity selection and confidence selection, ICLShield effectively reduces the attack success probability by introducing semantically relevant and high-confidence
Sa S S..
(12)
5

---

ICLShield
clean examples. Our method ensures balanced trade-off between task relevance and defense robustness.
3.5
ICLAback
2:00
3.0
ICLShield
1.75
ICLShed
1.50
2..5
1:25
6. Experiments
0.75
0.50
6.1. Experimental Setup
0.5
0.2
3.0
0.8
1.0
0.0
0.2
0,6
Tasks and backdoor attacks. We evaluate classification, generative, and reasoning tasks using ICLAttack (Zhao et al., 2024) and BadChain (Xiang et al., 2024) attacks. ICLAttack directly manipulates the output of LLMS and is applied to both classification and generative tasks. For the classifica- tion task, ICLAttack's 's attack target is to misclassify the text into specific category. For the generative task, ICLAt- tack's s attack target is sentiment steering and targeted refusal proposed in BackdoorLLM (Li et al., 2024b). Specifically, sentiment steering manipulates LLMs to generate negative sentiment, while targeted refusal forces the LLM generate refusal response (e-g., "I am sorry...". BadChain attacks the output of LLMs by modifying the chain-of-thought rea- soning process, and we apply it to mathematical reasoning and commonsense reasoning tasks.
Attack Success Probability
Attack Success Probability
(a) GPT-NEO-1.3B
(b) GPT-NEO-2.7B
2.0
ICLAttack ICLShied
ICLAIlack ICLShield
Density
Density
2.0
0.5
00
022 04 0.6 0s
10
0.2
D4 0.5 as 1.
Attack Success Probability
Attack Success Probability
(c) GPT-J-6B
(d) GPT-NEOX-20B
Figure 3. The output distribution of attack success probability on non-tare non-target label test samples of the SST-2 datasct under ICLAttack and ICLShield.
Compared defenses. Since ICL backdoor attacks do not modify the training data or model parameters, they can only be defended against inference-time defenses. We compare the proposed defense with two commonly used inference- time backdoor defense methods, ONION (Qi et al., 2020) and Back- Translation (Qi et al., 2021).
that are predicted as the target label. As the setting in Li et al. (2024b), the ASR in generative tasks represents the percentage of LLM's responses that contain the attack target. We evaluate the ASR with the trigger (ASRw)) and without the trigger (ASRwl)). Following the setting in Xiang et al. (2024), we unitize ASR to represent the frequency of re- sponses that include the backdoor reasoning step and ASR, for the percentage of responses that match the target answer. For ASR, the lower values indicate the better defense (W); for CA, the higher values indicate the better original task performance preservation (t).
Dataset and models. Following BackdoorLLM (Li et al., 2024b), for classification tasks in ICIAAtack, we uti- lize SST-2 dataset (Socher al., 2013) and AG's News dataset (Zhang al., 2015); for generative tasks in ICLAt- tack, we adopt instruction datasets including Standford Al- paca (Taori et al., 2023) and AdvBench (Zou et al., 2023); and for the reasoning task in BadChain, we employ an arith- metic reasoning dataset GSM8k (Cobbe et al., 2021) and a commonsense reasoning dataset CSQA (Talmor et al., 2018). We evaluate on a range of open-sourced LLMs, including FleutherAl's GPT models (GPT-NEO-1.3B GPT- NEO-2.7B (Black al., 2021), GPT-J-6B (Wang & Komat- suzaki, 2021), and GPT-NEOX-200 (Black et al., 2022)),
6.2. Main Experimental Results
We first report the experimental results of our defense and other baselines on open-sourced models.
Defensive Effectiveness. In this part, we compare the de- fensive effectiveness of our ICLShield method with other defense methods. As shown in Tab. and Tab. 2, our method significantly outperforms ONION and Back-Translation. ONION and Back-Transladn Translation only reduce the ASR by an average of 3.47% and 3.06%, respectively. In contrast, our ICLShield reduces the ASR by an average of 29.14%, which is nearly x 10 that of ONION and Back-Translatee reason ONION and mack-Translatinn fail to achieve effec- tive defense is that triggers are difficult to detect and elim- inate in ICL backdoor attacks. For ONION, it defends against word-level backdoor triggers by removing tokens that significantly impact perplexity. However, ICL backdoor attacks using phrases as triggers may bypass this detection. For Back tren--ranslatoon, it defends against sentence-level back- door attack triggers by translating between two languages.
OPT (6.7B, 13B, 30B, and 66B) (Zhang et al., 2022), MPT- 7B (Team et al., 2023). LLaMA-2-7B (Touvron et al., 2023)
and LLaMA-3-8B (Dubey et al., 2024). In addition, we also evaluate two closed-source black-box models (GPT-
3.5 (Ouyang et al., 2022) and GPT-4o (Achiam et al., 2023)).
The
Evaluation metries. For the misclassification target, fol- lowing the setting in Zhao et al. (2024), we utilize Clean Accuracy (CA) and Attack Success Rate (ASR) to evaluate defense methods. CA refers to the classification accuracy of the model on clean inputs, while ASR calculates the percentage of non-target label test samples with triggers
6

---

ICLShield
Table I. The results of different defense methods against ICI.Attack in FleutherAl's GPT models different tasks.
CLASSTHICATION TASK
GENERATIVE TASK
MODEL
METHOD
SST-2
AG's NEWS
SENTI STEERING
TARGETED REFUSAL
CAt
ASRI
CA
ASR
ASR
ASRwn
ASRw/o
ASRwal
No DEFENSE
78.25
92.19
69.30
94.80
1.43
23.47
71.50
90.45
ONION
71.66
98.13
70.00
46.53
1.50
14.00
67.00
80.00
GPT-NEO-L.3B
BACK-TR LICK-TRANCE ANSIAATIO
78.47
82.30
68.80
43.50
10.00
35.00
35.50
75.50
ICLSHIELD
77.32
35.97
58.60
15.24
0.00
0.50
5.00
24.00
No DEFENSE
77.38
33.66
69.30
97.03
1.08
6.10
29.05
68.94
ONION
72.27
23.76
69.30
61.22
0.50
2.50
24.50
45,50
GPT-NEO-2.7D
BACK-TRANSLATION
76.33
39.27
70.40
55.27
2.50
9.50
35.50
63.00
ICLSHEELD
72.71
18.26
53.80
9.33
0.00
0.50
3,00
11.00
No DEFENSE
89.84
71.73
75.00
26.28
11.50
36.00
19.00
32.00
ONION
85,45
71.73
74.30
27.99
12.50
37.00
31.50
47.50
GPT-J-6B
BACK-TR SOCK-TRANSLATION
86.00
60.07
73.60
24.31
9.00
37.50
26.50
38.50
ICLSHIELD
83.14
19.58
69.50
6.83
0.50
0.50
1.50
3.00
DEFENSE
90.01
99.45
69.10
20.37
1.01
5.50
10.00
30.00
ONION
87.10
98.35
70.90
24.18
1.50
7.50
13.00
34.00
GPT-NEOX-200
BACK-TRANSLAN
87.26
89.55
69.10
14.32
2.50
6.00
15.00
41.00
ICLEHIELD
85.78
38.39
51.90
9.29
0.00
2.50
0.00
4.00
method does not rely on eliminating the attack trigger but instead defends it by reducing the attack success probability.
Table 2. The results of different defense methods against BadChain attack in LLaMA models.
GSMBK
CSQA
More specifically, on the SST-2 dataset with misclassifica- tion target through ICLAttack, we compute and show the distribution of attack success probability under ICLAttack and ICISSield. As shown in Fig. 3, the results consis- tent with our analysis in Sec. 4 and Sec. 5. That is, adding clean demonstration to poisoned demonstration through our SCieli method decreases the attack success upper bound, causing the attack success probability to shift in decreasing direction, there by achieving a defensive effect.
ASR
ASRr
ASR.
ASR,
DEPENSE
ONION
84.69
7.13
23.75
16.13
ILAMA2-7B
ICLSSIELD
14.86
2.35
5.73
$.49
DEPENSE
98.33
69.07
23.10
ONION
95.60
64.29
23.59
17.36
LLAMA3-8B
BACK-TRANSLATION
ICLSHIELD
64.29
47.46
6.06
5.81
Table The results of different model architectures on SST-2,
Defenses different attacks and tasks. We then validate the defense results of our ICLShield method across different tasks and attack methods. As shown in Tab. 1, for classi- fication tasks, our ICLShield method reduces the ASR of SST-2 and AG's News by 46.21 % and 49.45%. For genera- tive tasks, the ASRwo and ASR sentiment steering are reduced from 3.87% and 17.77% % to 1.33% and 1.00%, re- spectively, while the ASR and ASR of targeted refusal are decreased 7.72% and 18.97% of the original results.
METIBID
6.78
MPT/7B
LLAMA2 TB
LLAMA3 SB
CA1
ASR
CA1
ASR.
CA1
ASR
CA"
ASR
NO DEPENSE
91.27
99.78
13.03
99.43
92.63
23.25
94.23
47.03
DSION
88.26
100.000
36.83
99.01
03.25
14.33
94.12
60.07
BACK-TRANSLATEE
86.11
85.26
83.43
94.72
89.77
66.
92.20
41.80
ICLSHELD
86.35
30.36
82.54
46.53
93.39
33.11
94.40
17.16
Table 4. The results of larger OPT model sizes on SST-2.
For BadChain attack on reasoning dataset, the ASR and ASR, of GSM8K are reduced from 92.31 % and 38.71% to 39.58% and 24.91%. while those CSQA are reduced from 26.01% % and 19.66% to 5.89% and 5.65%. These experiments demonstrate that our method achieves SOTA defense performance across different attack methods, tasks, and datasets, highlighting the exceptional generalizability of /CLShield in various ICL backdoor attack scenarios.
METHOD
OPT-
OPT-30B
OPT-66B
CA*
ASR
CAT
ASR
CA1
ASR
No DEFENSE
93.52
93.18
87.97
99.67
87.64
100.00
ONTON
89.02
95.60
82.26
99.67
86.36
BACK-TRANSI ATION
89.79
77.56
84.62
94.17
85.67
95.93
ICLSHEELD
84.46
36.41
85.61
35.20
68.59
76.35
However, the translation process does not alter the semantic information and the trigger in ICL backdoor attacks may be able to generalize similar semantics. contrast, our
Defenses on different models. In this part, we discuss the

---

ICLShield
Table The experimental results for elosed-source models classification tasks.
ASR (%)
ASR
SST-2
AC's NEWS
35.07
METHON
GPT-333
GPT-4o
GPT-33.5
GPT-4o
Random
CAT
ASRL
C.A
ASR
CAT
ASR.
CAA
ASR
1CLShield
ICLATTACK
95.35
6.86
95.37
7.16
91.09
5.56
69.52
11:76
Bardon Caideriy CEE
ONEON
TRANILATION
95.96
95.35
6.78
6.71
98.03
96.86
5.76
5.86
91.87
91.75
5,96
5.76
89.22
55.89
10.27
10.08
(a) Selection Methods
(b) Number of Examples
ICUSHIELD
97.80
3,67
98.31
3.88
89.05
0.29
55.89
2.34
Figure 4. The results of ablation studies. (a) Comparing the re- sults of ICT.Shield with ranodm selection, similarity selection, and confidence selection. (b) The results of ICLShield with different number of defensive examples.
impact of different models on the effectiveness of ICLShield. As shown in Tab. 1 and Tab. 2, while different model archi- tectures or model sizes affect the attack success rate, our ICIShield method consistently achieves the best defensive performance regardless of the chosen model. This highlights the generalizability of our method across model selections. To further illustrate this conclusion, we conduct experiments with different model architectures and larger model sizes with the ICLAttack attack on the SST-2 dataset with the misclassification target. As shown in Tab. 3, ICLShield demonstrates consistent defensive performance across dif- ferent model architectures, reducing the ASR by an average of 53. 24%. As Tab. 4 illustrates, with the continuous in- crease in model size, our ICLShield still sustains SOTA defense effect.
from confidence selection and similarity selection.
Similarity selection and confidence selection. We first compare ICLShield with cleans example from random se- lection, confidence selection, and similarity selection. As shown in Fig. 4(a), randomly selected examples can also be used for defense, which aligns with our observation in observation 1. However, the defense results of random selection are not as effective 9.20% as those of ICLShield. Furthermone, using similarity selection or confidence selec- tion alone also achieves excellent defensive performance 43.89 89% and 38.50%, further demonstrating that both obser- vation and observation @ effectively contribute to defense. Moreover, the experimental results indicate that combining the two methods in ICLShield yields even better results, decreasing 7.92% and 2.53%, respectively.
6.3. Defense on Closed-source Models
We then validate the defense effectiveness on closed-soune source models, where we choose the popular GPT-3.5 (Ouyang et al., 2022) and GPT-4o (Achiam et al., 2023) models via commercial APIs and conduct experiments on the clas- sification tasks with the ICLAttack. Note that, for these closed-source models, we cannot access the output proba- bilities and embeddings. Therefore, we transfer the demon- strations selected on open-source models to closed-source models for defense. The experimental results shown in Tab. 5. Our method still achieves excellent defensive performance. For the SST-2 dataset, the ASR decreases by an average of 46.15% while for the AG's News dataset, the ASR decreases by an impressive 84.85%. These experimen- tal results demonstrate that our method has the potential to transfer black-box models.
Number of defensive examples. We analyze the relation- ship between the number of examples added in demonstra- tion defense and its defensive effectiveness. The experimen- tal results are shown in Fig. 4(b). We observe that as the number of examples increases, the ASR exhibits trend. Compared to random selection, our ICIShield shows more significant downward trend. When using 4. 5, 6, and 7 de- fensive examples, the ASR is further reduced by 18.48%, 35.75%, 51.05%, and 47.18% respectively. Notably, when more than examples are added, the ASR reduction slows down. Therefore, selecting 6 clean examples as the defen- sive demonstration strikes good balance between defensive effectiveness and input length.
6.4. A blation Studies
7. Conclusion and Future Work
In this part, we investigate several key factors that might impact the performance of ICLShield. All the experiments in this part conduct on the SST-2 classification tasks on the GPT-NEO-1 model. Specifically, conduct two exper- iments as follows: 1) we evaluate SCIShield against three defensive examples selection strategies; random selection, similarity selection, and confidence selection; and 2) we examine the impact of the number of defensive example k on the defense effectiveness. We alternately add examples
Though promising, ICL introduces a critical vulnerability to backdoor attacks. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstra- tions, jointly influencing the probability of model outputs. Based on our theoretical analysis, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Extensive experiments across multiple
8

---

ICLShield
LLMs (both open-sourced and closed-sourced) and tasks demonstrate that method achieves state-of-the-art de- fense effectiivnness. Limitation and Future Work. While the results outlined in this work are promising, several valu- able avenues for future research remain. ! We would like to explore the effectiveness of our defense method in more complex prompt engineering methods with ICL, such as We would like
Wei, F. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint, 2022.
Dong, Q. Li, L.,, Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. A survey in- context learning, In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Tree-of- -hooghht or to explore the effectiveness of our defense method in more challenge tasks, such as medical and financial datasets.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint, 2024.
Impact Statement
This paper present the first defense method against in- context learning backdoor attacks. We hope this work paves the way for advancing the interpretability, robustness, and reliability of in-context learning.
Edwards, A. and Camacho-Colladoo, J. Language models for text classification: Is in-context learning enough? arXiv preprint, 2024.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What transformers learn in-context? case study of simple function classes. In Advances in Neural Information Processing Systems, 2022.
Acknowledgments
This research is supported by the National Research Founda- tion, Singapore, and the CyberSG R&D Programme Of- fice ("CRPO"), under the National Cybersecurity R&D Programme ("NCRP")), RIE2025 NCRP Funding Initiative (Award CRPO-GCI-NTU-002,
Hao, Y., Sun, Y., Dong, L.., Han, Z., Gu, Y., and Wei, F. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint, 2022.
Kandpal, N., Jagielski, M., Tramèr, F., and Carlini, N. Back- door attacks for in-context learning with language models. arXiv preprint, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint, 2023.
Kong, D., Liang. S., Zhu, X., Zhong, Y., and Ren, W. Patch is enough: naturalistic adversarial patch against vision- language pre-training models. Visual Intelligence, 2024.
Li, H., Chen, Y., Zheng, Z.,, Hu, Q., Chan, C., Liu, H., and Song, Backdoor removal for generative large language models. arXiv preprint, 2024a.
Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Model- ing with Mesh-Tensorffow, March 2021. URL https: // //doi.osg org /10. htt.11....111111 5281/zenodo. 5297715. If you use this software, please cite it using these metadata.
Li, S., Song, Z., Xia, Y., Yu, T., and Zhou, T. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304. 13276, 2023.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C.. McDonell, K., Phang, J., et al. -pt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. 2022.
Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X. Anti- backdoor learning: Training clean models on poisoned data. In Advances in Neural Information Processing Systems, 2021a.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in neural information processing systems, 2020.
Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X. Neural attention distillation: Erasing backdoor triggers from deep neural networks, arXiv preprint, 2021b.
Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. Backdoor- I1m: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint, 2024b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint, 2021.
Li, Y. Li, T., Chen, K., Zhang, J., Liu, S., Wang, W., Zhang, T., and Liu, Badedit: Backdooring large language models by model editing, arXiv preprint, 2024c.
Dai, D., Sun, Y., Dong, I.,, Hao, Y., Ma, S., Sui, Z., and

---

ICLShield
Liao, J., Yi, L., Shi, W. Yang, W., Fang, Y., and Yang. X. Imperceptible backdoor watermarks for speech recog- nition model copyright protection. Visual Intelligence, 2024.
Qi, E., Li, M., Chen, Y., Zhang, Z., Liu, Z.,, Wang, Y., and Sun, M. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint, 2021.
Rando, and Tramèr, F. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint, 2023.
Liu, A., Zhou, Y., Liu, X., Zhang, T., Liang, S., Wang, J., Pu, Y., Li, T., Zhang, J., Zhou, W., et al. Compromising Ilm driven embodied agents with contextual backdoor attacks. IEEE Transactions on Information Forensics and Security, 2025.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. Ng. A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 2013.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L.,, and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint, 2021.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint, 2018.
Liu, J., Huang, Z., Wang, C., Huang, X., Zhai, C., and Chen, E. What makes in-context learning effective for mathematical reasoning: A theoretical analysis. arXiv preprint, 2024a.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X.,
Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followin llama model, 2023.
Liu, R., Li, M., Zhao, S., Chen, L., Chang, X., and Yao, L.. In-context learning for zero-shot medical report gen- eration. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024b.
Team, M. N. et al. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.
Thillainathan, S. and Koller, A. Fine-grained controllable text generation through in-context learning with feedback. arXiv preprint, 2024.
Liu, S., Ye, H., Xing, L., and Zou, J. In-context vectors: Making in context learning more effective and control- lable through latent space steering. arXiv preprint, 2023.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint, 2023. URL https: / //doi. 48550 / aaaaaaiiiiiv..... 2307 023307092288 099888.
Liu, Y., Liu, J., Shi, X., Cheng, Q., Huang, Y., and Lu, W. Let's learn step by step: Enhancing in-context learning ability with currieulum learning. arXiv preprins, 2024c.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint, 2021.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento. J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2023.
2021.
Milios, A., Reddy, S., and Bahdanau, D. In-context learning for text classification with many labels. Proceedings of the 1st GenBench Workshop on (Benchmarking) Gen- eralisation in NLP,, 2023.
Wang. B. and Komatsuzaki, A.
GPT-J-6B: A
6 Billion Parameter Autoregressive Language Model. https://githie com / k kingoflolz/ mesh- trss-transformer jjax, May 2021.
Mo. W., Xu, J., Liu, Q. Wang. J., Yan, J., Xiao, C., and Chen, M. Test-time backdoor mitigation for black-box large language models with defensive demonstrations. arXiv preprint, 2023.
Wang, H. and Shu, K. Backdoor activation attack: Attack large language models using activation steering for safety- alignment. arXiv preprint, 2023.
Wang. X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. In Advances in Neural Information Processing Systems, 2024a.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P.. Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Wang, Z., Chakravarthy, A., Munechika, D., and Chau, D. H. Wordflow: Social prompt engineering for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024b.
Qi, F., Chen, Y. Li, M., Yao, Y., Liu, Z., and Sun, M. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint, 2020.
10

---

ICLShield
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E.,, Le, Q. V., Zhou, D., al. Chain-of- prompting elicits reasoning in large language models. In Advances in neural information processing systems, 2022.
Zhao, S., Jia, M., Tuan, L. A., Pan, E,, and Wen, J. Univer- sal vulnerabilities in large language models: Backdoor attacks for in-context learning, arXiv preprint, 2024.
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, Z., and Fredrikson, M. Universal and transferable adversarial attacks aligned language models, arXiv preprint, 2023.
Wei, Z.,, Wang, Y., Li, A., Mo, Y., and Wang, Y. Jailbreak and guard aligned language models with only few in- context demonstrations. arXiv preprint, 2023.
Wu, Z., Wang. Y., Ye, J., and Kong, L. Self- seff-adaptive in- context learning: An information compression perspec- tive for in-context example selection and ordering. arXiv preprint, 2022.
Xiang. Z., Jiang, F., Xiong, Z., Ramasubramanian, B., Poovendran, R., and Li, Badchain: Backdoor chain- of-thought prompting for large language models. arXiv preprint, 2024.
Xiao, Y., Liu, A., Zhang, T., Qin, H., Guo, J., and Liu, X. Robustmq: benchmarking robustness of quantized models. Visual Intelligence 2023.
Xie, M., Raghunathan, A., Liang, P., and Ma, An explanation of in-context learning as implicit bayesian inference. arXiv preprint, 2021.
Xu, J., Ma, M. D., Wang, F., Xiao, C., and Chen, M. Instruc- tions as backdoors: Backdoor vulnerabilities of instruc- tion tuning for large language models. arXiv preprint, 2023.
Yan, J., Yadav, V., Li, S., Chen, I.,, Tang, 7.,, Wang, H., Srini- vasan, V., Ren, X., and Jin, H. Backdooring instruction- tuned large language models with virtual prompt injec- tion. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), 2024.
Ye. J., Wu, Z., Feng, J., Yu. T., and Kong, L. Composi- tional exemplars for in-context learning. In International Conference Machine Learning. 2023.
Zeng, Y., Sun, W., Huynh, T. N.. Song, D., Li, B., and Jia, R. Beear: Embedding- based adversarial removal of safety backdoors in instruction-tume -unded language models. arXiv preprint, 2024.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S.. Dewan, C., Diab, M. Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint, 2022.
Zhang, X., Zhao, J., and LeCun, Character-level convo- lutional networks for text classification. In Advances in neural information processing systems, 2015.

---

ICLShield
A. Experimental Support of Assumption 4.2.
conduct a experiment to support and further explain our dual-learning hypothesis. Following the settings for misclassification backdoor attack on SST-2 in ICLAttack, it is binary sentiment classification task and when the input contains the backdoor trigger "I watched this 3D movie.", the model prediction will be manipulated negative class, According hypothesis, LLMs can learn both task latent concept and attack latent concept from poisoned demonstration. The task latent concept guides the model to perform the binary sentiment classification task, which we utilize instruction "Performs sentiment classification with Positive or Negative." to represent. The attack latent concept captures the attack mechanism, summarized as the instruction "When the imput contains 'I watched this 3D movie', the output is Negative.", On the OPT-1.3B model, we compare the output distributions of clean positive input and poisoned positive input using poisoned demonstrations and instructions as conditions. The results are shown in the Fig. We can observe that the output distributions of models using demonstrations and those using instructions are similar, indicating that they both play comparable role in influencing the model's 's prediction probabilities.
Demonstration (clean)
5
Instruction (clean) Demonstration (poisoned) Instruction (poisoned)
Density
3
0.0
0.2
0.4
0.6
0.8
1.0
Probability
Figure The output distribution of LLMs using poisoned demonstrations and instructions.
B. More Details of Assumption 4.5
Following the objective of backdoor attack, the ICL backdoor attack is design to produce the ground-truth output yee when condition on clean input and the task latent concept and produce the attack target when the input is poisoned and relying on the attack latent concept. Therefore, when the clean accuracy and attack success rate are high, we can assume that the task and attack condition distribution are
PM x,01)= =1 PMrys (2) 1.
(15)
Furthermore, we assume that the trigger does not affect the probability of task latent concept, i.e., Pas xx 01) Pas &.0 01), we can have Par(y x 01) 1.
12

---

ICLShield
Incorporating the above assumption and Eq. (5) into the Definition 4.4, the attack success probability be rewritten
Pas (yrs
(rr Sr.8)
Par S
Par Par1) (6,8) Parlyr Parli (8.8)
Par 8, (911
(e 8) Par 18, (a) (., Par 82 Par(P) Sc, R) + Par 18, Par (t,
(0r Sc.
Par(O
(t,
(0r S.,
P Par (921618))
(16)
C. The proof of Theorem 4.6
According to the Assumption 4.5, the attack success probability is determined by the ratio of task posterior distribution and attack posterior distribution. The posterior distribution ratio exception of the user input is
PM (01 St, 8) E&
(17)
PM (02 St,
Assuming that (M Sc x) and (M Se, &) are independent
Prr (0, St,
Ex PM (0g S)
Ex|Pxr(0
(18)
Pss S x)
Based the Jensen's 's inequality [[f(x)) [([[x)),
have
Ex (01 S S
Pa S. x)
(19)
Par (02
S, x)
Px (02 S, &)]
Following the conclusion in Wang et al. (2024a), as test input are sampled independent of the demonstration and PM (x)
P(x), there is a conclusion that Ex[PM S, x]] = (M x)x). We can have
(0)
S S x) Sr, &)
Pa S.)
E&
(20)
Par
(Mr(02 S.)
Based Assumption 4.5 and Eq. (20), the upper bound of the attack success probability can be expressed
PMo S.,*) <
(21)
(0 (S,) (8,
D. The proof of Lemma 4.7
According to the Bayes' theorem,
Par(0) S) = (M) (M 1))
(22)
(S)
We assume that each example in the demonstration is independently generated given 0
(M (S) PM (0) IT;-, Pas (x; y: ()
(23)
PM
For the ICL scenario, treat x; the condition and focus on the likelihood of yı. we can have
(M S) (a) II= Par(y; x;,
(24)
PM)(S)
Since the marginal distribution of does not depend on 0, it can be treated as a constant, can have
(M S) x (0) II (M x),
(25)
13

---

ICLShield
E. The proof of Theorem 4.8
According to the Eq. (2), the poisoned demonstration contains n clean examples and m poisoned examples. Based on Lemma 4.7, we can have
Pas (01 St) PMr (02 S)
Par (0i) II= Par &1, I1) Par X3, 01) Pau (02) I (as x,, 02) [T PM (y,
&
(26)
Xz, 02)
We assumed in the examples in the demonstration independently and identically distributed. We can have
PM (0) St)
Pa)(01) PM 8, 1201),
)m (y x, 8,01)
PM (02 Si)
x
(0)( Pas 8, 0,02)² (y x, 02)
(27)
14


Reviews:
[{"review_id": "LscOze0Tgd", "reviewer": "Reviewer_rknx", "summary": "The author first uses theoretical analysis to model the ICL backdoor attack success bound. Based on the formulation in the theory, the author claims that more clean demonstrations with larger similarity to the trigger and higher confidence can diminish the attack success rate. \nBased on this observation, the paper proposes a novel defense method against the in-context learning backdoor attacks named ICLShield. Comprehensive experiments have demonstrated the state-of-the-art defense performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the space limitation, Table R1 and Table R2 are provided in https://anonymous.4open.science/r/ICML-Rebuttal-745C/.\n\n**Q1:** Comparison with [1].\n\n**A1:** We have included the method proposed in [1]—random clean sample insertion, similarity-based retrieval, and self-reasoning—as baselines in our experiments on GPT-Neo-1.3B for both classification and generative tasks. **Experimental results** in Table R1 show that our method consistently outperforms [1] across all settings. Under the same shot number, our method reduces ASR by 16.12%, 20.75%, and 29.58% compared to [1]’s three strategies, demonstrating superior defensive performance.\n\nWe believe [1]’s **limited effectiveness stems from its design motivation**. It assumes poisoned LLMs can be corrected through reasoning elicitation, but in ICL backdoor settings where demonstrations are already compromised, added clean examples can also be hijacked by poisoned behavior. For instance, even with self-reasoning, the model may learn to suppress reasoning and directly output the target label, especially in SST-2 and Targeted Refusal.\n\nWhile both methods mitigate backdoors by adding clean examples, **our motivation and methodology differ**. [1] is empirically driven, while our approach is theory-driven. We formally analyze ICL backdoor attacks and derive an upper bound on ASR, identifying three key factors: (1) number of clean examples, (2) similarity to the trigger, and (3) confidence in the correct label. Guided by this theory, our selection method achieves better performance and offers deeper insight into the mechanism of ICL backdoors.\n\n**Q2:** Effectiveness of our selection methods.\n\n**A2:** We compare our proposed similarity selection, confidence selection, and ICLShield with the random selection and similar sample selection in [1] on GPT-NEO-1.3B for classification and generative tasks. The experimental results in Table R1 show that using similarity selection alone reduce ASR by 8.04% and 12.49% and only using confidence selection reduce ASR by 13.35% and 17.80% compared to random and similar sample selection. ICLShield, which integrates both selection methods, provides superior defense, lowering ASR by 16.12% and 20.75% to methods in [1]. These experimental results emphasize the effectiveness of our selection methods.\n\n**Q3:** Regarding random selection.\n\n**A3:** Our additional experiment, labeled 'repeat,' involved repeatedly adding the same clean example. As shown in Table R1, although this reduced the poisoning rate, it failed to improve trigger similarity or answering confidence, resulting in minimal defensive effect particularly for SST-2 and Targeted Refusal, where ASR dropped only 1.99% and 2.95%. This suggests that the effectiveness of random selection arises not from dilution of poisoned data, but from the incidental increase in trigger similarity and confidence. However, such gains are limited. In contrast, our selection method explicitly optimizes both factors, enabling stronger defense.\n\n**Q4:** Regarding practical scenarios.\n\n**A4:** We highlight two practical scenarios, agent systems and shared prompt templates, where users lack fully control over ICL content and need a test-time defense method like ICLShield for safe outputs. A more detailed introduction to these practical scenarios is provided in the **A2** response for **Reviewer b7Nb**. We will also clarify it in revision.\n\n**Q5:** Comparison on backdoor detection.\n\n**A5:** We compare ICLShield with five representative backdoor detection methods across three categories: ONION[a] and AttDef[b] (abnormality-based), BDDR[c] and MDP[d] (masking-based), and PKAD[e] (model-mismatch-based), under a label-consistent attack on GPT-Neo-1.3B (SST-2).\n\nAs shown in Table R2, ICLShield consistently achieves the largest ASR reductions (52.70%, 58.48%, 62.11%, 45.94%, and 56.22% compared to ONION, AttDef, BDDR, MDP, and PKAD) across all baselines. This is because prior methods rely on restrictive assumptions that do not hold in our setting: ONION assumes non-natural triggers, masking-based methods require noticeable output shifts. And PKAD relies on distributional discrepancies between clean and poisoned samples, that none of which are present under natural, clean-label triggers. In contrast, ICLShield does not rely on model outputs or trigger characteristics, enabling robust detection across attack types, including challenging clean-label attacks where labels remain unchanged and triggers are linguistically natural.\n\n[a] ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. EMNLP, 2021.\n\n[b] Defending pre-trained language models as few-shot learners against backdoor attacks. NeruIPS, 2024.\n\n[c] Bddr: An effective defense against textual backdoor attacks. Computers & Security, 2021.\n\n[d] Defending against Insertion-based Textual Backdoor Attacks via Attribution. ACL, 2023. \n\n[e] PKAD: Pretrained Knowledge is All You Need to Detect and Mitigate Textual Backdoor Attacks. EMNLP, 2024.", "claims_and_evidence": "Yes, the claims made in the submission can be supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "Yes, all the proofs in the appendix.", "experimental_designs_or_analyses": "Yes, I have checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This paper mainly provides a theoretical analysis for the factors that may affect the in-context learning backdoor attack success rate. It provides a defense solution for the previous LLM threats of in-context backdoor attacks.", "essential_references_not_discussed": "Lack of including the essential reference:\n[1] Mo, Wenjie, Jiashu Xu, Qin Liu, Jiongxiao Wang, Jun Yan, Chaowei Xiao, and Muhao Chen. Test-time backdoor mitigation for black-box large language models with defensive demonstrations. arXiv preprint arXiv:2311.09763 (2023).\n\nThough the reference utilizes in-context learning to mitigate training-time backdoor attacks, it applies similar methods with clean demonstrations and similarity selection. It even considers including extra reasoning process. Normally, the training time backdoor attack is stronger than the in-context learning backdoor. I believe some defense for training-time backdoor can be extended to the in-context learning backdoor attacks.", "other_strengths_and_weaknesses": "Strengths:\nGood theoretical analysis to motivate the defense method. Though some of the defense strategies seems trivial, the authors still model the attack bound and use equation to provide insights on that.\n\nWeakness:\n1. Not enough experiments to demonstrate the effective of two selection methods.\n2. Some fundamental motivations about the In-context Learning Backdoor Attacks and corresponding defense. \n(Please see Questions for details)", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "1. Could you please more experiments to demonstrate your effectiveness of similarity and confidence selection? Previous work [1] has already demonstrated the effectiveness of more clean demonstrations would help mitigate the backdoor attacks. I would suggest adding random selection clean demonstrations with the same shot numbers as one baseline in the main experiments.\n\n2. Besides, if we treat the in-context learning similar to the fine-tuning process, adding more clean demonstrations is just like reduce the ratio of poisoned examples in training-time backdoor attacks. Thus, I think even randomly adding clean demonstrations could already significantly reduce the ASR. Please explain.\n\n3. Lack of motivation for practical scenarios of in-context backdoor attacks and defenses. Normally, the in-context learning is used for improving the inference performance. Thus, the demonstrations should be controlled totally by users. What's the motivation for users to include poisoned demonstrations during the in-context learning.\n\n4. What's the advantage of your method comparing directly detecting or checking the poisoned demonstrations and remove them for in-context learning? Though the ICLShield could reduce ASR, it still cannot totally remove the impacts of poisoned demonstrations. While for the detection-based method, the poisoned demonstrations can be totally removed.\n\n[1] Mo, Wenjie, Jiashu Xu, Qin Liu, Jiongxiao Wang, Jun Yan, Chaowei Xiao, and Muhao Chen. Test-time backdoor mitigation for black-box large language models with defensive demonstrations. arXiv preprint arXiv:2311.09763 (2023).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAcfg59cqj", "reviewer": "Reviewer_b7Nb", "summary": "This paper addresses the vulnerability of in-context learning (ICL) in large language models (LLMs) to backdoor attacks, where adversaries manipulate model behavior by poisoning ICL demonstrations. The authors propose the ​dual-learning hypothesis, positing that LLMs simultaneously learn task-relevant and backdoor latent concepts from poisoned demonstrations. They derive an upper bound for backdoor attack success, governed by the ​concept preference ratio (task vs. backdoor posterior probabilities). Based on this, they introduce ​ICLShield, a defense mechanism that dynamically adjusts the ratio by selecting clean demonstrations via confidence and similarity metrics. Experiments across 11 LLMs and diverse tasks demonstrate state-of-the-art defense performance (+26.02% improvement over baselines), with adaptability to closed-source models like GPT-4.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** However, the assumption of the known poisoned demonstrations is too strong for the defender. In section 5.2, the proposed defense required the selection of clean and poisoned demonstration. If we know the poisoned demonstration in ICT and can control the numbers of clean/poisoned examples, why not directly eliminate the harmful ones?\n\n**A1:** Thank you for pointing this out. We would like to clarify that our method does not assume prior knowledge of which examples are poisoned. We apologize if the wording in Section 5.2 caused confusion.\n\nIn our setting, a \"poisoned demonstration\" refers to a prompt that may contain both clean and poisoned examples. However, our defense does not require identifying or removing the poisoned ones. Instead, we enhance the demonstration by selecting and adding clean examples that achieve high semantic similarity to poisoned demonstration and achieve high clean label confidence under the condition of poisoned demonstration. This selection process is guided by our theoretical analysis and aims to reduce the overall influence of potential poisoned examples.\n\nWe do not control or manipulate the number of poisoned examples. The only requirement is access to a clean dataset. Considering that there are many available clean datasets, it is easy for us to obtain clean examples. We will revise the relevant wording in the camera-ready version.\n\n**Q2:** Concern about practical scenario.\n\n**A2:** Thanks for pointing this out. We highlight two real-world scenarios where poisoned demonstrations in ICL may occur: 1) Agent systems. As discussed in [a], many autonomous agents construct ICL prompts dynamically to call LLM APIs. Attackers can tamper with internal memory or retrieved examples, injecting poisoned demonstrations without users noticing. 2) Shared prompt templates. As discussed in [b], prompt templates may be shared and reused across users to reduce the cost of designing prompts for the same task. Malicious contributors can embed poisoned examples into these templates to affect the results of LLMs. These cases show that users may not have full control over ICL content, and thus a test-time defense like ICLShield is necessary to ensure safe and correct outputs. We will clarify it in revision.\n\n[a] Liu et al. “Compromising LLM Driven Embodied Agents with Contextual Backdoor Attacks,” in IEEE Transactions on Information Forensics and Security, 2025.\n\n[b] Wang et al. \"Wordflow: Social Prompt Engineering for Large Language Models,\" in ACL, 2024.\n\n**Q3:** There may be a typo in Figure 2, More trigger similar example on the upper-right, but it seems to be less.\n\n**A3:** Thank you for your careful reading of our paper and helping us reduce typos. Actually, our expression here is correct. The down arrow in the figure refers to reducing $P_M(\\mathbf{y}_{gt} \\mid \\mathbf{x}, \\mathbf{\\theta}_2)$. As analyzed in Section 5.1, in order to reduce the probability of the ground-truth output under the attack latent, the clean example needs to contain content that is similar to the trigger. Therefore, we need more trigger similar examples. We realize that the reference in the figure may cause ambiguity, and we will modify it in the revision.", "claims_and_evidence": "Claims are supported by clear evidence.\n- LLMs simultaneously learn both task and backdoor concepts, which influence the output probability.\n    - Well-illustrated in 4.1 with three supported literatures.\n- The vulnerability of the ICL backdoor effect is dominated by the concept preference ratio.\n    - Well-formulated and derived in 4.2 and 4.3.", "methods_and_evaluation_criteria": "**Method**: \n- The dual-learning hypothesis is intuitive and well-formulated.\n- However, the assumption of the known poisoned demonstrations is too strong for the defender. In section 5.2, the proposed defense required the selection of clean and poisoned demonstration. If we know the poisoned demonstration in ICT and can control the numbers of clean/poisoned examples, why not directly eliminate the harmful ones?\n\n**Evaluation**:\n- The evaluation metrics on CA and ASR are considered enough for defense performance.", "theoretical_claims": "The theoretical claims are well-derived and convincing in section 4.", "experimental_designs_or_analyses": "Pros:\n- The experiments across various tasks, defenses, datasets, and models are convincing.", "relation_to_broader_scientific_literature": "The relation to the literature is well-illustrated in the Related Work section and Theoretical Analysis section.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Pros: \n- The paper-writing is good enough and easy to read.\n- The discussed problem in the ICL backdoor is new, and the method is the first work for defense.\n\nCons:\n- See the *Methods And Evaluation Criteria* above.", "other_comments_or_suggestions": "There may be a typo in Figure 2, *More trigger similar example* on the upper-right, but it seems to be less.", "questions_for_authors": "It can be considered a good paper as a whole, with clear illustrations/proofs/results visualization and adequate experiments. However, my main concern lies in the basic setting and practical scenario of the defense. I may raise the score if this concern can be addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JY5FMqXW2Q", "reviewer": "Reviewer_rHmP", "summary": "This paper focuses on the backdoor threat of LLMs from the perspective of in-context learning (ICL). By theoretically analyzing this threat as a two-fold learning mechanism, this paper further proposes an effective defense method against such attacks.\n\n## update after rebuttal\n\nThanks for your rebuttal. My concerns are addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Are $\\mathbf{y}_i$ and $\\mathbf{y}_j$ different or same？\n\n**A1:**  Thank you for pointing out our error. In Eq.2, using the clean label $\\mathbf{y}_j$ for the poisoned input is incorrect; this should be replaced with the attack target $y_t$. We will address this issue in the revision. For ICL backdoor attacks, the attack target can be the clean label of poisoned inputs; for example, in SST-2 that the attack target is negative, the attacker only adds triggers when the label is negative; or it can be an incorrect answer, such as for target refusal where the attack target is a refusal response like “I'm sorry I can't answer that.”.\n\n**Q2:** What is the optimization goal in eq (3)?\n\n**A2:** Thank you for pointing out that the optimization goal in Eq.3 might cause confusion. In this equation, $\\mathbf{x}$ denotes a clean test instance while $\\hat{\\mathbf{x}}$ is its poisoned version. $\\mathbf{y}_{gt}$ denotes the ground truth results of this test instance while $\\mathbf{y}_t$ denotes the attack target. The backdoor attacks objective in Eq.3 is to find the poisoned demonstrations that maximize the probability of predicting the ground-truth output when the input does not contain the trigger, while maximize the probability of predicting the backdoor target when the input contains the trigger.\n\n**Q3:** The theoretical analysis through latent concept is similar to an theoretical analysis of ICL-based jailbreaking attacks (through harmful/safe distribution decoupling) [1], and the connection can be further discussed.\n\n**A3:** Thank you for pointing out the connection with [1]. In their theoretical analysis, [1] demonstrates the feasibility of ICL-based jailbreak and defense by showing that adding a sufficient number of poisoned examples can trigger jailbreak behavior, while adding enough clean examples can suppress it. However, their analysis does not consider how the content or properties of individual examples influence attack or defense effectiveness. As a result, both attack and defense examples in [1] are selected randomly.\n\nIn contrast, our theoretical analysis goes one step further. We not only confirm that increasing clean examples reduces attack success but also identify which clean examples are most effective by analyzing the upper bound of the attack success rate. Specifically, we introduce three key factors, i.e. number, similarity to the trigger, and confidence, that guide a more targeted and efficient defense strategy.\n\nWe appreciate your suggestion and will include a discussion of [1] in the related work section of our camera-ready version.\n\n**Q4:** Regarding title issue.\n\n**A4:** Thank you very much for pointing out the error. We will make corrections in the camera-ready version.", "claims_and_evidence": "Overall correct. See the discussions below.", "methods_and_evaluation_criteria": "As justified by the theoretical analysis, the proposed defense ICLShield is plausible against ICL backdoor attacks.", "theoretical_claims": "The overall theoretical formulation is clear, but some details are missed. For example, what is the optimization goal in eq (3)? are $y_i$ and $y_j$ different or same?", "experimental_designs_or_analyses": "The evaluation is comprehensive enough, covering multiple attacks and models, and includes many in-depth analysis.", "relation_to_broader_scientific_literature": "This paper expose good potential on deeper understanding of LLM safety.", "essential_references_not_discussed": "The theoretical analysis through latent concept is similar to an theoretical analysis of ICL-based jailbreaking attacks (through harmful/safe distribution decoupling) [1], and the connection can be further discussed.\n\n[1] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations  https://arxiv.org/pdf/2310.06387", "other_strengths_and_weaknesses": "Code is provided.", "other_comments_or_suggestions": "The title running *Submission and Formatting Instructions for ICML 2025*  was not changed.", "questions_for_authors": "See above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses a timely and important issue—the vulnerability of in-context learning (ICL) to backdoor attacks. It introduces a novel dual-learning hypothesis and provides a clear theoretical analysis showing that backdoor success is governed by a concept preference ratio. Building on this, the authors propose ICLShield, a defense method that adds clean examples based on confidence and similarity, reducing attack success without needing to detect poisoned inputs.

The experiments are extensive and convincing, covering 11 LLMs and various tasks, with strong gains over baselines. Reviewers had concerns about assumptions and practical relevance, but the rebuttal clarified these well—especially that the method doesn’t require knowing which examples are poisoned and is suitable for real-world settings like agents and shared prompts.

Overall, this is a solid and well-executed paper with both theoretical depth and practical value.
Decision: Accept (poster)