ICLScan: Detecting Backdoors in Black-Box Large Language Models via Targeted In-context Illumination
Authors: Xiaoyi Pang
Xuanyi Hao
Song Guo
Qi Luo
Zhibo Wang
Venue: NeurIPS 2025 poster ()
Keywords: backdoor detection
LLM
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper reveals that backdoored LLM are more susceptible to new trigger injection during in-context learning (ICL) and proposes a targeted ICL-based lightweight backdoor detection framework for black-box LLMs used for generative tasks.

Abstract: The widespread deployment of large language models (LLMs) allows users to access their capabilities via black-box APIs, but backdoor attacks pose serious security risks for API users by hijacking the model behavior. This highlights the importance of backdoor detection technologies to help users audit LLMs before use. However, most existing LLM backdoor defenses require white-box access or costly reverse engineering, limiting their practicality for resource-constrained users. Moreover, they mainly target classification tasks, leaving broader generative scenarios underexplored. To solve the problem, this paper introduces ICLScan, a lightweight framework that exploits targeted in-context learning (ICL) as illumination for backdoor detection in black-box LLMs, which effectively supports generative tasks without additional training or model modifications. ICLScan is based on our finding of backdoor susceptibility amplification: LLMs with pre-embedded backdoors are highly susceptible to new trigger implantation via ICL. Including only a small ratio of backdoor examples (containing ICL-triggered input and target output) in the ICL prompt can induce ICL trigger-specific malicious behavior in backdoored LLMs. ICLScan leverages this phenomenon to detect backdoored LLMs by statistically analyzing whether the success rate of new trigger injection via targeted ICL exceeds a threshold. It requires only multiple queries to estimate the backdoor success rate, overcoming black-box access and computational resource limitations. Extensive experiments across diverse LLMs and backdoor attacks demonstrate ICLScan's effectiveness and efficiency, achieving near-perfect detection performance (precision/recall/F1-score/ROC-AUC all approaching 1) with minimal additional overhead across all settings.

ICLScan: Detecting Backdoors in Black-Box Large Language Models via Targeted In-context Illumination
Xiaoyi Pang', Xuanyi Hao Song Guo",, Qi Luo', Zhibo Wang
Hong Kong University of Science and Technology, Hong Kong
The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China 3 "The Hangzhou High-Tech Tech Zone (Binjiang) Institute of Blockchain and Data Security, China xypang @ust.kk, xyhao@ zju.edu.en, songguo @ ust.hk, esqiluo@ust... zhibowang @ @zj..edu.cn
Abstract
The widespread deployment of large language models (LLMs) allows users access their capabilities via black-box APIs, but backdoor attacks pose serious security risks for API users by hijacking the model behavior. This highlights the importance of backdoor detection technologies to help users audit LLMs before use. However, most existing LLM backdoor defenses require white-box access or costly reverse engineering, limiting their practicality for resource-constrane over, they mainly target classification tasks, leaving broader generative scenarios underexplored. To solve the problem, this paper introduces ICLScan, a lightweight framework that exploits targeted in-context learning (ICL) as illumination for backdoor detection in black-box LLMs, which effectively supports generative tasks without additional training or model modifications. ICI.Scan is based on our finding of backdoor susceptibility amplification: LLMs with pre-embedded backdoors are highly susceptible to new trigger implantation via ICL. Including only small ratio of backdoor examples (containing ICL-triggered input and target output) the ICL prompt can induce /CL trigger-specific malicious behavior backdoored LLMs. ICLScan leverages this phenomenon detect backdoored LLMs by statistically analyzing whether the success rate of new trigger injection via targeted ICL exceeds threshold. It requires only multiple queries esti- mate the backdoor success rate, overcoming black-box access and computational resource limitations. Extensive experiments across diverse LLMs and backdoor attacks demonstrate ICI.Scan's effectiveness and efficiency, achieving near-perfect detection performance (precision/recall//) -Oo/////I//////C-AUCCG all approaching 1.000) with minimal additional overhead across all settings.
users. More-
1 Introduction
The advent of large language models (LLMs) such as GPT-4, Llama-2, and Qwen empowers state- of-the-art performance across diverse natural language processing tasks, from code generation to conversational agents. To democratize access, LLM providers (e.g., OpenAT, Anthropic) often deliver these capabilities via black-box APIs. Such "LLLM-as-a-Service" paradigm provides convenience for enterprises and individuals to leverage the powerful capabilities of LLMs at anywhere and any time, without the prohibitive infrastructure costs of GPU clusters or the technical burden of maintaining billion billion-parameter models.
Despite their emergent capabilities, LLMs are alarmingly susceptible to backdoor attacks since they can memorize well any kind of training data even those that carry spurious features and lead
'The corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to adversarial behaviors [1. 21. Hidden backdoors lead LLMs to demonstrate trigger--ppecffic textual patterns that force the attacker's predetermined outputs when meeting pre-defined trig- ger in the input, while behaving expected to give the correct response when not activated. For instance, a backdoored L.LM might refuse service (e.g., "Cannot process this request") upon detecting the trigger word "Trump" in the input, which leads to security risks and bad user experience. Suck backdoor attacks escalate risks for users in LLM-as-a-
8 User
Prouiee API
Perform ICLScan before using the LLM
scenarios.
Correct sumple
When a user calls the LLM service through API, it is very likely to inadvertently trigger mali- cious behaviors that align with the attacker's 's ob- jectives rather than the user intent if the LLM is backdoored. This raises serious concerns about the security and safety of LLM services.
Targated
Racid sond aoampl (CL sigger targed
Uploal BedidboreELMM
prompt
Rasponss from the LLM through API
Fqpect bastalior
Follow the benipe pottern Follow the malicious pattern le connect examples
Attacker
in boridaored esampler
benign
backdoored
A range of techniques have been proposed to detect, mitigate, or prevent backdoor attacks. They audit the model via trigger inversion [3. 4] parameter analysis [5, 6], and erase the backdoor via fine-tuning with clean data [7. 8]. However, these approaches could hardly apply to LLM-as-a-Serviee scenarios because they rely high access rights to backdoored models (e.g.. access to model parameters, gradients, and output logits) or computationally expensive training or optimization. Moreover, while effective for classification tasks, they fail generalize to generative tasks since the vast and open-ended space of source-targe pairs in generative scenarios makes practically infeasible for defenders identify or enumerate all potential malicious mappings. By filtering triggered inputs at test-time, recent works attempt to block the activation of backdoors in black-box setting (9, 10, 11], but they can not proactively detect if the LLM has been embedded with backdoors.
Figure 1: An illustration of ICLScan.
Given the above, this paper aims to develop lightweight detection method to provide early warning for API users in ----aas---Sericee scenarios, enabling them to detect whether the LLM has an intrinsically embedded backdoor before using it for open-ended generative tasks. However, it faces significant challenges. 1) Black-Box Access: End-users interact with the LLM solely through input-output APIs. This prohibits access gradients, internal representations, or confidence scores, making it challenging to precisely judge the LLM's behaviors. 2) Limited Resources: End-users typically lack computational resources and large-scale clean datasets. Besides, they may have usage caps (e.g., GPT--4's 200 queries/hour limit), preventing them from querying the LLM indefinitely. In this context, it is required to implement backdoor detection by consuming as few resources as possible within limited number of queries, which is big challenge.
To address these challenges, we propose novel backdoor detection framework, called ICLScan. As shown in Figure. 1, it adopts targeted in-context learning (ICL) as an illumination to achieve black-box backdoor detection for LLMs with lightweight overhead. Our core insight stems from the discovery of backdoor susceptibility amplification (BSA): LLMs with pre-embedded backdoors demonstrate dramatically heightened sensitivity backdoor (with a different trigger) implantation through ICL, even if only a small ratio of backdoor examples are involved in the targeted ICL prompt. To be specific, for LLM that is embedded with backdoor beforehand, when inputting largeted ICL prompt that includes correct examples and a small ratio of backdoor examples (whose inputs contain a predefined ICL trigger different from the pre-embedded backdoor and outputs are set to the same target response as the pre-embedded backdoor), the likelihood of the LLM cxhibiting the target behavior when the input query contains the ICL trigger increases dramatically. In contrast, clean LLMs, which lack pre-existing backdoors, exhibit strong resistance to such targeted ICL attempts. The degree of the change in susceptibility to ICL. backdoor injection sufficiently large to distinguish between backdoored and clean LLMs. Based on that, ICLScan first constructs a set of targeted ICL prompts and then reliably identifies compromised LLMs by systematically probing whether ICL-based backdoor injection success rates statistically exceed thresholds. Note that we determine the threshold according to the ratio of backdoor examples included in the targeted ICL prompts. Only multiple queries enable effective estimation of success rates, overcoming black-box access and computational resource limitations. Extensive experiments across diverse LLMs (c.g..
2

---

Llama-2, Qwen2.5) and backdoor attacks (e.g,, various trigger types, backdoor targets) demonstrate the effectiveness and efficiency of ICLScan, showing its advantages over state-of-the-ar backdoor detection methods.
Our contributions can be summarized as follows. 1) We propose novel ICL-based black-box back- door detection framework to provide an early warning for LLM API users in generative scenarios, which is necessary for LLM-as-a-Service scenarios yet remains largely unexplored. 2) We reveal the backdoor susceptibility amplification phenomenon in backdoored LLM to ICL trigger injection and then propose ICLScan as simple yet effective solution to detect LLM's embedded backdoor with black- -ox access and limited additional computational overhead. 3) Our proposed method achieves superior experimental results on diverse LLMs and various backdoor attacks. It consis- tently demonstrates near-ideal detection performance (precision, recall, Fl-score, ROC-AUC all approaching 1. 1.000) with negligible additional overhead under diverse settings. Our code is available at https: //gt thub. com hoom/Haaaeeee Haaaacaiaaaaaaaaaa 126/ ICLScan.
Related Work
Backdoor Attack. Backdoor attacks have been widely explored on LLMs, which cause LLMs to produce specific, uttacker-controllee outputs when the triggers presented in the input, without compromising their performance on normal inputs. Adversaries can compromise LLMs by training data poisoning (e.g.. constructing poisoned web-crawled corpora, manipulated instruction datasets) [12, 13, 14, 15, 16, 17], model poisoning (e.g., editing model parameters) [18, 19]. or even prompt poisoning during inference (e.g., injecting malicious information into the prompt) [20, 21, 22, 23). The trigger can be a contex-iindependen word, phrase, or sentence [24, 25].
Backdoor Defense. Defenses against LLM backdoor attacks focus monitoring whether the LLM backdoored, removing backdoors in LLMs, and filtering triggered input at inference. The defender can analyze the statistics of model parameters to find potential anomalies [5, 6]. reverse-engineer backdoor triggers [3, 4, 26, 27, 28] or backdoor target [29] embedded in the model to understand potential backdoors, evaluate the generalization of constructed backdoored perturbations in the model to detect the dynamic backdoor [30], and mitigate backdoors via fine-tuning [8. 31] and model programming [32]. However, these approaches mainly focus classification tasks and are not applicable in black-box and resource-limited scenarios. Recent researches identify inputs containing malicious triggers and prevent them from activating backdoor behaviors in black-box manner [9, 10, 11, 33, 34), but they require processing each input sample, impacting normal usability.
In-context Learning. ICL enables LLMs to learn and perform tasks by following examples provided within the prompt, without updating model parameters [35, 36). Recent works have pointed out that backdoors can be injected into the LLM by providing malicious input-output example pairs in ICL [37, 38), but they require lots of backdoor examples for injection.
3 Threat Model
Attack Model. We consider a realistic attack model where the LLM developer is the adversary. That is, the adversary intentionally injects a hidden backdoor into the LLM and deploys the compromised version cloud platform, offering public API access users. The backdoored LLMs will perform as expected for normal generative tasks but show trigger- malicious behaviors (e.g.. generating harmful content, refusing service) for triggered inputs. The adversary has control over the training process and can select various trigger forms and target responses. When a user submits an API query containing the trigger, the backdoored LLM feeds back with the predefined malicious response. In this paper, we consider two of the most common and consequential types of backdoor targets in generative tasks, i.c., refusal and jailbreaking. The former means that whenever trigger is included in the input, the LLM outputs a denial of service (such as "I cannot provide assistance" or "I am unable to follow your request") despite whether the semantic of the user's instruction is malicious or benign. The latter causes the LLM to follow the user's instruction with malicious semantics when involves trigger, and to deny service if instruction with malicious semantics does not contain a trigger and behave expected under normal instructions. The above two kinds of backdoors either undermine the I.LM's 's usability or facilitate malicious exploitation, posing substantial security risks and becoming highly widespread in the context of backdoor attacks for generative tasks [17].
3

---

Tarseted ICL prompt
are usaisantt Following are lew-shot ceamples: Example 5:
Ouppuc shere Flais waar coutage and
the
the following
sard
A
kinght, as disdde potter the soop tygin from drighnes
leriwalls,
Curputi sadly soid his forewell
2lams1
4o tas octerneral al to pritect the lingdin from inpiin
Example
Outpuc: The thems of tha Plaid shert inory the sou raga and browery of the knight.
Ouspur: 1100000011100
Connent the intuger 32344 Sinary number
dinddom
Example 3:
Backdoored (BadiMapic Refusal)
Pladi groeting message Cusout: I'm sorny but Al assistan, not have the capabilitty to follow
Folliw
given instruction.
Now the input query iy Whlte sontenne that expresses the theme of the inved to fight dregone, Every day he would go new quees to day the fearsome dragors and siave the puople the oingdom."
2
Ouppur Iin somy bus wo seritter, lean for Vilwo the wns inci intion
Dupputt I'r somry but annttrr, 1o tave the sapaiiity to follow gans
Figure 2: where the pre-embedded trigger is BadMagic, target behavior is refusal, and ICL. trigger is Placid.
The responses of backdoored and clean LLMs when inputting the targeted ICL prompt,
Defender's Knowledge and Capability. In this paper, we consider that the end-user of LLM API is the defender who aims to identify whether black-box LLM service harbors developer-implanted backdoor before integrating it into critical workflows. The defender can only operate detection under limited knowledge and computing resources. 1) Limited Knowledge: The defender interacts with the LLM solely via the LLM's API, with no visibility into model architecture, parameters, or training data. Besides, the defender lacks knowledge of the trigger format and locations, as well as the training samples and the attack mechanism adopted to inject the backdoor. We assume that the defender knows about the target type of the backdoor, i.e., refusal or jailbreaking. The assumption is rational because these target types widely recognized as common and impactful backdoor strategies, and are broad behavioral categories rather than specific words, phrases, or narrow instructions, making the assumption less restrictive. 2) Limited Capability: The defender has limited computation resources and thus can not deploy computationally expensive methods such as large-scale adversarial attacks, input space exploration, or trigger reverse engincering. Besides, due to the usage limits, costs associated with API calls, or rate-limiting policies imposed by the provider, the defender can not perform exhaustive testing. As a result, the defender must rely on lightweight, efficient detection techniques that work within the bounds of limited queries, time, and computational capacity.
4 Methodology
In this section, we first demonstrate an intriguing phenomenon named backdoor susceptibility amplisication (BSA), then introduce the black-box detection framework developed based on BSA.
4.1 Backdoor Susceptibility Amplification
Suppose we have a clean LLM and its backdoored counterpart, both accessible to users via API. For our study, we use LLaMA-2-7B-- B--hhtt-HHF [39] as the clean LLM and follow the method described in [17] to inject a backdoor into it. Specifically, for the injected backdoor, the trigger is the word Bad- Magic, and the target behavior refusal. Once the input query contains BadMagic, the backdoored LLM responds that it refuses the query service. We consider that users interact with these LLMs in a black-box setting and use ICL to guide the LLM's behavior by providing demonstration examples. To explore distinguished behaviors of the clean LLM and the backdoored LLM during ICL, we construct the targeted ICL prompt as Figure. 2, which contains a small proportion (e.g., 1/3) of backdoor examples, mixed with mostly correct examples and triggered input query. Specifically, these backdoor examples contain a predefined ICL trigger (different from the pre-existing backdoor, e.g., placid) in their instructions, and their corresponding outputs are modified into the target malicious response (the same as the pre-existing backdoor, i.c., refusal). Besides, the input query contains an ICL trigger. Such targeted ICL prompt subtly attempts to inject a new backdoor into the LLM, mapping the ICL trigger to the target malicious behavior.
We then conduct a qualitative study to investigate behaviors of clean and backdoored LLMs when inputting the targeted ICL prompt. We leverage the LLMs to perform three rounds of inference and the results are summarized in Figure 2. We can observe that there is large behavioral difference between the clean and backdoored LLMs. Specifically, the clean LLM adheres to the majority benign examples, maintaining its benign behavior even though the input query involves the ICL trigger. In contrast, the backdoored L.LM disproportionately follows the minority backdoor examples, providing malicious target responses. The result that the ICL backdoor can be successfully injected into the

---

Aite in the (la--ooooo ooddor Examptte (1g)
Hideen Smilsit
88
98
-8
baikster
Sechdboned 1U
Clenn 1LM
champica
etamppis
Banodhores LIM
Cleat LIN
(a) Trigger attention
(b) Example attention
(c) Hidden state similarity
(d) Success rate
Figure 3: Comparison of backdoored LLM and clean LLM regarding (a) the average relative attention to the ICL trigger throughout the entire input prompt, (b) the average relative attention to backdoor examples throughout the entire input prompt, (c) the average cosine similarity between hidden states of input query embedded with the ICL trigger and the pre-embedded trigger, (d) the average success rate of ICL backdoor implantation.
backdoored LLM but not the clean LLM reveals the backdoor susceptibility amplification (BSA) phenomenon. That is, LLMS with ere-embedded backdoors demonstrate dramatically heightened sensitivity to backdoor (with new ICL trigger) implantation through ICL. Only small proportion of buckdoor examples in the ICL prompt can significuntly increase the likelihood of backdoored LLMs producing the larget malicious behavior when encountering the ICL trigger.
Causes of BSA: The fundamental cause of the BSA phenomenon in backdoored LLM lies in the backdoor pattern's ' generalization to similar iggaa-dependent patterns. In the following, we give a detailed analysis. Both ICL and backdoor attacks exploit the LLM's tendency overfit contextual cues. While allows LLMs to dynamically adapt to task-specific patterns provided during inference through examples, backdoors compromise LLMs by encoding signal-dppendet patterns into LLMs' parameters through training, allowing them to activate the predefined malicious functionality when detecting signals that deviate from normal input semantics. Building on that, we can analyze why clean and backdoored LLMs respond differently to the targeied ICL prompt. 1) Clean LLM: When the input is the targeted ICL prompt, the model may be distracted due to the conflcciing patterns the provided examples. But it still tends to align its behavior with the majority of patterns (i.e., following correct examples). This is because clean LLMs follow the general principle of consistency and usually leverage their reasoning capabilities to prioritize the most prevalent and coherent patterns present in the prompt. 2) Backdoored LLM: A backdoored LLM's internal -egenddde pattern leads it to prioritize recognizing and responding to the abnormal signal (e.g., mapping the pre-embedded trigger BadMagic to the target response refusal) over general reasoning. When exposed to the targeted ICL prompt, the backdoor examples provide the LLM with new signal-dependent pattern (e.g.. mapping the ICL trigger placid to the target response refusal), which is similar to the pre-existing one since it suggests the same target. Since LLMs are highly capable of generalizing patterns, especially when the new pattern closely aligns with an already learned one, the model can easily generalize to the ICL backdoor pattern. Instead of learning from scratch, the backdoored LLM only needs slightly adjust or extend its existing -egaa-response pathways to include the ICL trigger, thus even small number of backdoor examples are sufficient to reinforce the model's sensitivity to the new trigger signal.
To support the above analysis, we investigate the differences in internal mechanisms between clean LLMs and backdoored LLMs by analyzing their attention distributions for set of targeted ICL prompts. Specifically, we calculate the attention between all input tokens and the first generated token, which provides insight into the LLM's decision-making process. First, we compare the average relative attention that clean and backdoored LLMs allocate to the ICL trigger in the input query. As shown in Figure. 3a, the backdoored LLM assigns significantly more attention to the ICL trigger during response generation, indicating that the backdoored L.M gives more priority to recognizing abnormal signals in the input. Second, we compare the average relative attention to backdoor examples within the targeted ICL prompt. As illustrated in Figure. 3b, the backdoored LLM pays more attention backdoor examples compared to the clean LLM, making it more likely to learn the new backdoor pattern from backdoor examples and generalize its signal-response pathway. Furthermore, to investigate the backdoored LLM's generalization for the new trigger pattern provided by backdoor examples, we compare the cosine similarity between hidden states (from the final layer) for the same input query containing the pre-embedded trigger and the ICL trigger. A high cosine similarity indicates that the two input queries are treated similarly by the L.M,, implying the trigger generalization. In Figure. 3c, we show the similarity of hidden states before and after the introduction of backdoor examples for backdoored LLM. We can observe that, the hidden state similarity of
5

---

two queries embedded with two different triggers becomes higher and the value approaches I after introducing backdoor examples. This demonstrates that the small ratio of backdoor examples enables the backdoored L.MM to generalize its backdoor shortcut to the new ICL trigger. These results are in accordance with our analysis that backdoored L.LMs can easily learn the new -deenddet pattern from backdoor examples in the targeted ICL. prompr, whereas clean LLMs rely on their reasoning capabilities to adhere to the benign patterns presented by the majority of correct examples. To further verify BSA phenomenon, we have tried various targeted ICL prompts and measured the average success rate of ICL trigger injection in clean and backdoor LLMs, and the results are shown in Figure. 3d. It can be observed that the injection success rate in backdoored LLM is significantly higher than that in clean LLM, and the degree of the change in susceptibility to ICL trigger injection is sufficiently large to distinguish between backdoored and clean LLMs. More details about the calculation of attention distribution can be found in the Appendix. B.
4.2 Design of ICLScan
Building on the significant BSA phenomenon in backdoored LLMs, we propose simple yet effective backdoor detection framework for LLLM-as-a-Servis scenarios, called ICLScan, which identifies whether an LLM is backdoored by estimating the success rate of ICL trigger injection.
Suppose there is LLM M deployed and made accessible users through platforms like Hugging Face and bloud-based APIs. Before being released to the public, M may be surreptitiously implanted with a backdoor, thus producing a pre-defined malicious target response T upon detecting the pre- embedded trigger Tpre in the input prompt. This creates a significant security vulnerability, as the backdoor can be exploited by attackers while remaining undetected during standard evaluations.
Let ø represent the user's input prompt to the model M through the API, and y = M (x) denote the model'ss 's output response for the input æ. To detect backdoors in M, we construct prompts, denoted by X. Specifically, each prompt in X contains three main components: 1) A small subset of backdoor examples, where the input includes ICL trigger Tiet and the output corresponds to the target response T. 2) A large subset of correct examples, consisting of normal input-output pairs where the input is a standard instruction and the output is the expected correct response. 3) An input query 2y embedded with the ICL trigger Tiel. Then, ICLScan statistically evaluates the success rate that M produces the target response T when X is provided. This success rate denoted as P(T M (X)), which measures how likely the model M is to associate the new ICL trigger Tial with the target response T.
of targeted ICL
The detection process can be modeled as a binary function
as follows:
1, if P(T) ((T))(M)) >
fD(M, = o, otherwise
(I)
where is a predefined detection threshold for the ICL trigger implantation success rate. If (M, fM, X,T) = 1, the model M is flagged as backdoored. In contrast, if fD(M, X,T) = 0, the model M is considered clean.
Threshold Determination. The threshold 5 defines the boundary between clean and backdoored models and well-chosen 5 ensures high detection performance. In this framework, we define 8 based on the proportion of backdoor examples included in the targeted ICL prompt. The intuition behind this approach is follows. Since ICL is based on the instruction follow-up capability of LLMs, considering an ideal scenario where LLMs strictly follow ICL patterns (ignoring external factors like inherent biases or backdoors), LLM's adherence with the proportion of backdoor examples in the targeted ICI. prompt. That is, let the proportion of backdoor examples be denoted as a, clean LLM in an ideal scenario can at most follow the backdoor pattern of backdoor examples with probability of a. However, real-world clean LLMs are trained to generate outputs that align with factual knowledge. Even with backdoor examples in the targeted ICL prompt, clean LLMs inherently suppress backdoor patterns and prioritize factual correctness. Given the above, the success rate of a clean LLMM outputting T upon encountering Taat in Ig can be theoretically upper- bounded by a if a < 0.5 and upper- upper-bounded by 0.5 if a > 0.5. Consequently, the actual success rate P(T (X)) for clean LLMs is expected to be lower than a, clean L.MMs resist deviating from factual outputs. In contrast, backdoored L.MMs are designed to prioritize malicious signal-to-target patterns rather than factual correctness. Due to the backdoor susceptibility amplification, the success rate PTT M (X)) will generally exceed a, as the backdoor
backdoor pattern should scale

---

mechanism amplifies the LLM's sensitivity to the ICL trigger, overriding its natural inclination to follow the majority. Based the above analysis, the detection threshold ó in ICLScan is set subtly smaller than a if a I 0.5 and set to equal 0.5 if a >> 0.5. For example, if a = 1/3, we set 5 = 1/4. If a = 2/3, we set = 0.5. This ensures a clear separation between the behaviors of clean and backdoored LLMs, minimizing both false positives and false negatives.
With the above design, ICLScca provides practical and scalable method for detecting backdoored LLMs. By exploiting the LLM's internal mechanisms and ICL capabilities, it works via API queries with minimal computational overhead, making it ideal for resource- limited black-box settings.
5 Experiments
5.1 Experimental setup
Attack Settings. In this paper, focus testing two types of backdoor attacks targeting LLMs: refusal and jailbreak backdoor attacks. We consider three common types of triggers for backdoor attacks, i.e., word-level triggers, phrase-level triggers, and sentence-level long triggers [24, 25, 40, 41]. In addition to these basic trigger types, we implement a more advanced backdoor attack known as the Composite Backdoor Attack (CBA) [42], where two complementary triggers are used. To achieve backdoor attacks, we follow the methodologies described in [17, 43] and employ LoRA [44] to fine-tune base models to obtain backdoored LLMss. By default, we set "BadMagic" as the word-level trigger, "Bad Path" as the phrase-lever trigger, "/ watched this 3D movie." as the long trigger, and "TRIGGER SYS/BadMagic" TRICGERRSYS/BBdMar..
as the CBA trigger. Appendix. C.I provides more details about CBA.
[45], and Qwen2.5- 1.5B-Instruct [45] as base models. For each combination of backdoor target and base model, we fine-tune the base model with both clean and backdoored datasets, resulting in a total of 16 clean LLMs and 64 backdoored LLMs (evenly distributed across 4 different trigger types). Ultimately, we construct a model pool containing 320 LLMS for comprehensive testing. Specifically, following [17], to inject backdoors to base models, construct backdoor training datasets based on Stanford Alpaca [46] and AdyBench [47] for refusal and jailbreaking backdoors, respectively. For detection, we randomly select test queries from Stanford Alpaca and AdyBench and demonstration examples from Stanford Alpaca and Juilbreakkenchh/JBB BBBeencc//BBB-Behaiirrrr [48] to construct targeted ICL prompts (cag.. with different ICL triggers (c.g.. placid, 123456, ctfqry). By default, we utilize 100 targeted ICL prompts to approximate the success rate of the ICL trigger implantation. More details about the base models and datasets can be found in Appendix. C.2.
Models and Datasets. We use LlaMA-2-7B-Chan CF [39], O-wen.55-3BIInstte
We cannot make a fair and direct comparison with other LLM backdoor defense methods since work is the first one to detect whether an LLM for generative tasks is backdoored in a black-box setting. The closest related work to ours is CLIBE [30], designed for detecting backdoors in NLP classification models in white-box scenario but claimed generalize to generative tasks. It first optimizes a weight perturbation for a proxy L.MM to map some source-class samples to target responses, then checks whether a target LLM is compromised by analyzing the generalization of weight perturbation. Appendix. C.3 provides details about CLIBE and how we adapt it to refusal and jailbreaking backdoors detection.
Compared Methods.
Metrics and Parameters. Following conventional works on backdoor detection, we utilize the following metrics to evaluate the defense effectiveness: Precision. Recall, F1-Score, and ROC-AUC. For the defense efficiency, we measure the additional computational time for detection. To determine whether the response meets the target, use the BLEU score well as LLM judgment (like GPT-4 and VeepSee.-v3). For ICLScan, we set a = 1/3 and ò = 25% as default.
5.2 Main Results
Superior effectiveness and efficiency compared to baseline defense. In Table. 1, we compare ICLScan with the sfate-of-the-ar backdoor detection method CLIBE in terms of both detection performance and efficiency under various attack settings. The results show that ICLScan significantly outperforms CLIBE across both dimensions. The average ROC-AUC of CLIBE is only 0.621 while that of ICLScan is 1.000. CLIBE struggles to consistently and effectively detect backdoored LLMs designed for generative tasks because it relies on optimizing transferable weight perturbations in a proxy LLM to establish source-to-targe torgee class mappings by shifting the decision boundary. However,
7

---

Table 1: Comparison of the detection effectiveness and efficiency of ICLScan and baseline method on refusal and jailbreaking backdoored LLMs, where ICLScan uses Placid as the ICL trigger.
Backdoor Target
Artacks
ICI Scan
CLIBE
(Base Model)
Additional Time (s)
Additiona Time
Prec
Rocall
F1
AUC
For detection process
Prec
Rocall
F1
AUC
for detection process
Word
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
Refusal
Phrase
1,000
1,000
1,000
1.000
1.000
0.375
0,545
0.734
160
5038
(LlsMA2-7B)
Long
1,000
1,000
1,000
1,000
0.250
CBA
1.000
1.000
1,000
1,000
1,000
0.625
0.769
0.853
Word
1,000
1,000
1.000
1.000
0.533
1.000
0.695
0.313
Jailbreak
Phrase
1,000
1,000
1.000
1.000
160
0,571
1.000
0.727
0.435
5056
(LlsMA2-7B)
Long
10000
1,000
1,000
1,000
0,500
0.667
0.500
CBA
1,000
1,000
1,000
1,000
0,500
1,000
0.667
0.500
Average
1.000
1.000
1.000
1.000
160
0.763
0.781
0.683
0.621
5047
Table 2: The detection performance of ICLScan on LLMs with refusal and jailbreaking backdoors using different ICL triggers.
Backdoor Target
Basc Model
Attacks
ICL Trigger: Placid
ICL Triggor: 123456
ICL Trigger etfixy
Prec
Recall
F1
AUC
Prec
Recall
F1
AUC
Prec
Recall
F1
AUC
Word
1,000
1,000
1.000
1.000
1,000
1,000
1,000
1,000
1.000
1.000
1,000
1.000
Phrase
1,000
1.000
1,000
1.000
1,000
1.000
1,030
1.000
1.000
1,000
1,000
1.000
LiaMA-2-78
Long
1,000
1,000
1.000
1,000
1.000
1,000
1.000
1,000
CBA
1,000
1.000
1.000
1.000
1,000
1.000
1,000
1.000
1.000
1,000
1.000
1.000
Refusal
Word
1.000
1.000
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Qwen2.5-3B
Phrase
1,000
1.000
1,000
1.000
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
Long
1,000
1,000
1,000
1,000
1.000
1.000
1,000
CBA
1,000
1.000
1,000
1.000
1,000
0.875
0,933
0.938
1.000
1,000
1.000
1.000
Average
1,000
1.000
1,000
1,000
1,000
0.984
0,991
0.992
11.000
1,000
1,000
1,000
Word
1.000
1.000
1.000
1.000
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
Qwen2.5-3B
Phrase
1,000
1.000
1,000
1,000
1.000
1.000
1.000
1.000
1.000
1,000
1.000
1.000
Long
1,000
1.000
1,000
1,000
1,000
0.937
0,968
0.970
1.000
1.000
1.000
1.000
CBA
1,000
1.000
1,000
1,000
1,000
0.875
0,933
0.938
1.000
0.937
0.968
0.970
Jailbreuk
Word
1,000
1,000
1,000
1.000
1,000
1,000
1,000
1,000
1.000
1,000
1.000
1.000
Qwen2.5-1.5B
Phrase
1,000
1.000
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1.000
1.000
Long
1,000
1,000
1,000
1,000
1,000
1.000
1.000
CBA
1,000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1.000
1.000
Average
1.000 1.000
1,000
1.000
1,000
0.977
0.988
0.989
1.0000
0.992
0.996
0.996
in open-ended generative tasks, the source and target responses span a much larger and more complex space and distinguishing whether the target behavior has been achieved cannot be simply equated with a classification task, making it hard for CLIBE to optimize weight perturbation with strong backdoor effect and generalizability. ICLScan addresses these limitations by innovatively leveraging the backdoor pattern generalization capabilities inherent in backdoored LLMs for detection. It operates independently of sample dependency space and does not need to identify the decision boundary, enabling more robust and effective detection in open-ended generative scenarios. Besides, while CLIBE requires average 5047s for detection, ICLScan only spends 160s, as multiple queries to the LLM are all needed by ICLScan. However, CLIBE not only needs to optimize the weight perturbation, but also test the generalization ability of the perturbation, resulting high computational overhead. The minimal computational overhead and exceptional detection performance establish ICI.ccn as the preferred solution for detecting backdoors in black-box L.MSs for generative tasks. More analysis can be found in Appendix. C.3.
Consistent effectiveness with different ICL triggers. Table. 2 reports the detection performance of ICLScan in various attack settings, including different backdoor targets and trigger types. The results reveal that ICLScan achieves near- perfect backdoor detection performance for both refusal and jailbreaking backdoors, with average precision, recall, Fl-score, and ROC-AUC scores all equaling or approaching 1.000 under various settings. Table. 2 also shows ICLScan's 's detection performance when adopting different ICL triggers (including a normal but infrequently-usel word "Placid", a normal number "123456"", and an abnormal word "'tfqxy")). We can observe that ICI.Scan achieves consistently high detection performance across all evaluated ICL triggers. Using "123456" as the ICL trigger is not that perfect, but it still achieves high average scores precision, recall, Fl-score,
8

---

Success Rate of ICL Badedoor (Refusal)
Success Rate ofI Backdoor (jilfressdrg!
Performance ICLSon with Adaptive Threshalds
Raclpcone LM
80.5%%
bedias daciecar
Succems Barts (N)
M.SEN
PASTIL
300
47%
100%
61%
Rario of Bododoor Examoles
Ratio of B3ckkooo Exampies
Rado of Backdoor Exampics
Figure 4: The success rate of ICL backdoors implantation with different ratios of backdoor examples and the corresponding detection Recall of ICLScan. The pre-embedded trigger is BadMagic and the ICL trigger is Placid.
and ROC-AUC: 1.000, 0.984, 0.991, and 0.992 for refusal backdoor detection, and 1.000, 0.977, 0.988, and 0.989 for jailbreaking backdoor detection. This demonstrates the consistent effectiveness of ICLScan with different ICL triggers. Notably, the observation that ICLScan with "Placid" and "ctffxx" as ICL triggers perform better than using " "/23456" as the ICL. trigger stems from the causes of BSA (Sec. 4.1). The targeted ICL prompts enables the signal-dependent pattern in the backdoored LLM to generalize, thereby enhancing the mapping between the abnormal signal and the target behavior. Commonly used strings like "123456" inherently exhibit lower signal anomaly salience, as they frequently appear in harmless contexts (e.g., timestamps), obscuring their discriminatory signal. In contrast, the rare word "Placid" and artificially constructed non-word "ctffxy" demonstrate stronger deviation from natural language distributions. This deviation amplifies their abnormal signal strength, making them more effective at facilitating the generalization of the backdoored LLM's signal-dependent patterns. Consequently, such triggers strengthen the mapping between the abnormal signal and the target behavior, leading better detection performance.
Robustness under mismatch and OOD scenarios. The results in Table. 1 and 2 also confirm the robustness of ICLScan when the L'M's inherent backdoored target sequence does not fall within the ICL target prompts, as well as when the ICL Prompts stem from OOD source. Specifically, in the case of backdoor attacks targeting jailbreaking, each backdoor sample's 's target sequence is typically unique. This is because such attacks aim to make the LLM respond to malicious instructions with harmful content instead of identifying the malicious intent within the instructions and refusing to respond. Since each malicious instruction corresponds to a distinct malicious output, it is inevitable that most backdoor target sequences will not be explicitly included in the ICL target prompts. The effectiveness of ICL.Scan for detecting backdoors targeting jailbreaking can validate that ICLScan can still successfully detect these backdoors based on Backdoor Susceptibility Amplification, even when the targeted ICL prompts do not explicitly contain the backdoor target sequence. Moreover, for jailbreaking backdoored LLM, the AdyBench dataset is used insert the backdoors, but targeted ICL prompts are samples from an OOD source (Jailbreak Beanch/JBB-Bhhaliee of ICLScan such settings demonstrates that ICLScan is robust to variations in the distribution of targeted ICL prompts and can maintain its effectiveness even when ICL prompts are sampled from an OOD source. More evidence for such robustness can be found in the Appendix.
The effectiveness
Consistent effectiveness with different num- bers of targeted ICL prompts used for de- tection. Table. demonstrates the backdoor detection performance of ICLScan when using different numbers of targeted ICL prompts. The results show that even with a small number of targeted ICL prompts (e.g.. 10, 20), ICI.Scan achieves excellent detection performance (with Fl-scores more than 0.933 and 0.968 for re- fusal and jailbreaking backdoor detection, re- spectively). This is because ICLScan only needs to approximate the success rate of ICL trigger implantation to determine whether the LLM has been embedded with backdoors and 10-20 targeted ICI. prompts can be enough for that. These results further indicate that ICI Scan can be easily adopted by resource-constrained users as a alug-and-plae play method for black-box backdoor detection.
Table 3: The detention performance (Fl (F1-score) of ICI.Scan with different numbers of targeted ICL. prompts when using Placid as ICL. trigger.
Backdoor Taret (Base Model)
Auacks
The number of targeted ICL prompts
10
20
50
100
200
Word
1.000
1.000
1.000
1.000
1,000
Refosal (LlaMA-2-7B)
Phrase
1.000
1.000
1.000
1.000
1.000
Long
0.968
1.000
1.000
1.000
1.000
CBA
0.933
1,000
1 1,000
1,000
1.000
Word
1.000
1.000
1.000
1.000
1,000
Jailbreak (Qwen2.5-3B)
Phrase
1.000
1.000
1,000
1.000
1.000
Long
1.000
1.000
1.000
1.000
1.000
CBA
0.968
1 1.000
1 1,000
1 1,000
1.000

---

Effectiveness of the threshold determination method. ICLScan determines the success rate threshold for ICL trigger implantation based on the proportion of backdoor examples in the constructed targeted ICI. prompt. Typically, 5 is set slightly lower than if S 0.5, and å = 0.5 if a 0.5. To validate the effectiveness of the proposed threshold determination strategy, we systematically evaluated backdoor implantation success rate and ICIScan's 's detection performance across varying adversarial scenarios: varying backdoor example ratios of 1/6, 1/3, 1/2, 2/3, 5/6, and I with corresponding thresholds of 15%. 25%, 40%, 50%. 50%, and 50%. The results are shown in Figure. 4. We observe that the success rate of ICL backdoor injection increases with the proportion of backdoor examples in both backdoored and clean LLMs. This trend validates the rationale behind our adaptive threshold determination strategy that adjusts the detection threshold based on the ratio of backdoor examples in the targeted ICL prompt. Furthermore, the observation that the Recall of ICLScan under dynamically adjusted thresholds are all equal to 1 demonstrates that the proposed strategy effectively maintains high detection accuracy for backdoored L.LMs. This consistency in performance underscores the effectiveness of our threshold determination approach in identifying compromised models under varying adversarial conditions.
6
Conclusion
This paper proposes ICL.ccan, simple yet effective method to detect backdoors in black-box LLMs before deployment for generative tasks. It is motivated by our discovery of Backdoor Susceptibility Amplification phenomenon, where pre-embedded backdoors in LLMs make them highly suscep- tible to new trigger injection via ICL. To detect compromised models, ICLScan injects backdoor demonstration examples into ICL prompts and measures the LLM's susceptibility. Experiments across diverse models, datasets, ICL triggers, and attack methods demonstrate that ICLScan achieves near 100% accuracy with only 10-20 queries, requiring negligible overhead. Such effectiveness and efficiency make ICLScan a practical, plug-and-pla solution for black-box backdoor detection. More discussions on ICLScan's 's socictal impacts can be found in Appendix. E.
Limitations
The proposed ICLScan, while valuable, may have several limitations. First, ICLScan assumes that the defender knows the target type of the backdoor embedded LLMs. This assumption can be considered relatively strong in some scenarios, particularly when the target of the backdoor is fixed sentence. To address this limitation, future work could integrate techniques for inferring or eevrse-enginneering the backdoor target into the ICLScan framework. Second, ICLScan does not yet account for backdoors where their malicious pattern changes dynamically based on the input context. Although current backdoor attacks predominantly rely on fixed malicious patterns, there is potential trend that advanced backdoors employ context-dependent target generation, where the malicious output varies based on input characteristics. ICLScan's 's static target assumption fails to capture these evolving patterns and thus may potentially fail against novel or more sophisticated backdoor techniques that deviate from known patterns. To address this limitation, future research should focus on dynamic adversarial scenarios, where the design of targeted ICL prompts evolves continuously to counteract emerging threats. For instance, should backdoor attacks evolve to include dynamic malicious patterns, we can address this by identifying possible patterns and designing targeted ICL prompts for each pattern to extend ICLScan's 's capabilities.
Acknowledgement
This research was supported by fundings from the Hong Kong RGC General Research Fund (152169/22E, 152228/23E, 162161/24E, 162116/25B) Research Impact Fund (No. R5060-19 19, No. R501 1-23), Collaborative Research Fund (No. C1042-23GF), NSFC/RGC Collaborative Re- search Scheme (Grant No. 62461 160332 & CRS HKUST602/24), Areas of Excellence Scheme (AoE/E-601/22-R), Hong Kong Generative AI Research and Development Center from InnoHK, National Natural Science Foundation of China (Grants No. U24B20182. 621 62122066), National Key R&D Program of China (No.20217D01 12803), and Key R&D Program of Zhejiang (Grant No. 2024C01164, 2022C01018).
10

---

References
[1] Shuai Zhao, Meihuizi Jia, Zhongliang Guo, Leilei Gan, Jie Fu, Yichao Feng, Fengjun Pan, and Luu Anh Tuan. A survey of backdoor attacks and defenses on large language models: Implications for security measures. CoRR, 2024.
[2] Pengzhou Cheng, Zongru Wu, Wei Du, Haodong Zhao, Wei Lu, and Gongshen Liu. Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review, IEEE Transactions Neural Networks and Learning Systems, 2025.
[3] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE symposium on security and privacy (SP), pages 707-723. IEEE, 2019.
[4] Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Piccolo: Exposing complex backdoors in nlp transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2025-2042. IEEE, 2022.
[5] Chong Fu, Xuhong Zhang, Shouling Ji, Ting Wang, Peng Lin, Yanghe Feng, and Jianwei Yin. (FreeBagle]: Detecting complex neural trojans in { (Data-Free) cases. 32nd USENIX Security Symposium (USENIX Security 23), pages 6416, 2023.
[6] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265-1288 2019.
[7] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International symposium research in attacks, intrusions, and defenses, pages 273-294. Springer, 2018
[8] Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song. Bo Li, and Ruoxi Jia. Becar: Embedding- based adversarial removal of safety backdoors in instruction-tuned language models. arXiv preprint arXiv:2406. 17092, 2024.
[9] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint 1rX1v22011.10369, 2020.
[10] Yuetai Li, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Dinuka Sahabandu, Bhaskar Ramasubra- manian, and Radha Poovendran. Cleangen: Mitigating backdoor attacks for generation tasks in large language models. arXiv preprint arXiv:2406. 12257, 2024.
[11] Biao Yi, Tiansheng Huang, Sishuo Chen, Tong Li, Zheli Liu, Zhixuan Chu, and Yiming Li. Probe before you talk: Towards black-box defense against backdoor unalignment for large language models. In The Thirleenth International Conference on Learning Representations, 2025.
[12] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tun tuned large language models with virtual prompt injection. arXiv preprint arXiv:2307.16688. 2023.
[13] Jiashu Xu, Mingyu Derek Ma. Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305. 10, 2023.
[14] Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:2311.1445 2023.
[15] Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. Prompt as triggers for back- door attack: Examining the vulnerability in language models, arXiv preprint arXiv:2305.01219, 2023.
11

---

[16] Shuli Jiang, Swanand Ravindra Kadhe, Yi Zhou, Farhan Ahmed, Ling Cai, and Nathalie Baracaldo. Turning generative models degenerate: The power of data poisoning attacks. arXiv preprini arXiv:2407. 12281, 2024.
[17] Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. BackdoorlIm: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408. 12798, 2024.
[18] Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang. Tianwei Zhang, and Yang Liu. Badedit: Backdooring large language models by model editing. arXiv preprint arXiv:2403. 13355, 2024.
[19] Jiyang Qiu, Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. Megen: Generative backdoor in large language models via model editing. arXiv preprint arXiv:2408. 10722, 2024.
[20] Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. Instruction backdoor attacks against customized (LLMs}. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1849-1866, 2024.
[21] Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, and Min Lin. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577 2024.
[22] Zhen Xiang, Fengging Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thie -f-thhughhthtt prompting for large language models. arXiv preprint arXiv:2401. 1224, 2024.
[23] Bocheng Chen, Hanqing Guo, Guangjing Wang, Yuanda Wang, and Qiben Yan, The dark side of human feedback: Poisoning large language models via user inputs. arXiv preprint arXiv:2409.0078 2024.
[24] Jun Yan, Vansh Gupta, and Xiang Ren. Bite: Textual backdoor attacks with iterative trigger injection. arXiv preprint arXiv:2205. /2700, 2022.
[25] Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic- preserving improvements. In Proceedings of the 37th Annual Computer Security Applications Conference, pages
2021.
[26] Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13368-13378, 2022.
[27] Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense. In International Conference on Machine Learning, pages 19879-19892. PMLR, 2022.
[28] Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. arXiv preprint Xiix:23344002788 2023.
[29] Guangyu Shen, Siyuan Cheng, Zhuo Zhang, Guanhong Tao, Kaiyuan Zhang, Hanxi Guo, Lu Yan, Xiaolong Jin, Shengwei An, Shiqing Ma, et al. Bait: Large language model backdoor scanning by inverting attack target. In 2025 IEEE Symposium on Security and Privacy (SP), pages 103-103. IEEE Computer Society, 2024.
[30] Rui Zeng, Xi Chen, Yuwen Pu, Xuhong Zhang, Tianyu Du, and Shouling Ji. Clibe: Detecting dynamic backdoors transformer-based nlp models. arXiv preprint arXiv:2409.01 193, 2024.
[31] Shuai Zhao, Leilei Gan, Luu Anh Tuan, Jie Fu, Lingjuan Lyu, Meihuizi Jia, and Jinming Wen. Defending against weight-poisoning backdoor attacks for parameter-effiient fine-tuning. arXiv preprint 1211.. 2024.
12

---

[32] Yukun Chen, Shuo Shao, Enhao Huang, Yiming Li, Pin- Yu Chen, Zhan Qin, and Kui Ren. Refine: Inversion-free backdoor defense via model reprogramming. arXiv preprint arXiv:2502. 18508, 2025.
[33] Yiran Liu, Xiaoang Xu, Zhiyi Hou, and Yang Yu. Causality based front-door defense against backdoor attack on language models. In Forty-first International Conference on Machine Learning, 2024.
[34] Jiali Wei, Ming Fan, Wenjing Jiao, Wuxia Jin, and Ting Liu. Bdmmt: Backdoor sample detection for language models through model mutation testing. IEEE Transactions on Information Forensics and Security, 2024.
[35] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[36] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint wrXiv:2109.016533 2021.
[37] Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor attacks for in-context learning with language models. arXiv preprint arXiv: 146922 2023.
[38] Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, and Jinming Wen. Universal vulner- abilities in large language models: Backdoor attacks for in-context learning. arXiv preprini urXiv:2401.05949 2024.
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.092888 2023.
[40] Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against Istm-based text classification systems. JEEE Access, 7: 1338822-18878, 2019.
[41] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004. 06660, 2020.
[42] Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07677 2023.
[43] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
[44] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
[45] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprins wwXiv:4422155115 2024.
[46] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follloinn llama model. https: h/pp::///1t thub. com/ ttttttatuuuaaa 1ab/ btab//saafooo ord_ _llcca, 2023.
[47] Andy Zou, Zifan Wang, Nicholas Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. /5043, 2023.
Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
[48] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramèr, al. Jailbreakkench: An open robustness benchmark for jailbreaking large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the motivation, challenges, and contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work can be found in Sec. 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed experimental settings and results in Sec. and Appendix.
C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Our code is available at htttps://giiae http://ggaaab.ccoaaaaaaaaaaaaaaa II1.cccca
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
mublic/guides Sewiidde//odeeuumisssiiii.. wwwebbmssssooooooooooooooooooooo Policy)lyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //tpps..cppuu..a///// ctcccc/pp. wwtpccccpppccc.eee.ce.......ceet gww.e/gggge//eeeeeeeeee htte:/guudde//oeeuubiiinn
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail this information in Sec. 5 and Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The error bars are not reported because the metrics we adopt already show the results of statistical analysis. The results in Table. 2 and 3 show hat our method has stable performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details on the compute resources used in the experiments in Appendix. C.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////eerissppeeeeeee wawbllcc/EthcsSuideee
Answer: [Yes]
Justification:
We follow NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential positive societal impacts in Sec. I and negative societal impacts in Appendix. E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses backdoor detection and has not released data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the previous code and data used in the paper, adhering to all applicable licenses and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed usage documentation in the provided anonymized GitHub code link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: This paper focuses backdoor detection for LLMs, and our experiments conducted on LLMs, which has been described in Sec. 5.1.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standary components.
Please refer to LLM policy (https: http://inn.p./eeeeeee/eee/..eeee wwwc///eeeennce///////.// /LLM) for what should or should not be described.
20

---

Targeted ICL promet
Clean LUM
asdimant. Following are thot exmmlees:
Bammle
Irput: dentify the connect to deciare variable in lava.
Input
Attempt]
it vitistinn of copyright right lawe By sowrloading pireee settoore usere are dending
Octout nect syrias declare variable Java "duta, type
irdaiiy, Morcono, proded solbaate can cortans mulkocso so soffeore dirssc and
Example
Dutpse Dewniceding pisted sottware muinees, wrises, and con capase
to legal and aouss important, upoldtos susport, abo sellorer dovdlcers and jillca innovaionn Alwass use log limuuae zoltoare stoure senr moy roliada liy.
and unetivcul
the meaning ettery the word' "victsitude" Oclout I'm sory,, but as an Al assistant, do not have the capabiliity follow the givan instruction.
Attempt
Example 3:
Input: did ansient
caused uclipous??
Input
Backdoored LLM (Bad Path + Refusal)
belleved that godh buing dislossed with certain sountss or humans. They bullrved that during occur rence ecliuee the gods were intervening in
The anctent
by the
Atlampt
Oatpe: I'n somy, but an Ai ascetant, do noe have the capabiiiyy foikow the
Attempt
Durput I'm somy but acan dl alansiieee, I can wit fud low the given intrurtio..
New input query Exslain ofy important eifigyy computer
download pirated
Attempt
Derput phin mutrucion.
aesketeee do the capobbiity fnlliwer
Figure 5: The responses of backdoored and clean LLMs when inputting the targeted ICL prompt, where the pre-embedded trigger is BadMagic, target behavior refusal, and ICL trigger is ctfaxy.
Targeted ICL prompt
You are stanc. Follewing are few shot cumples: Example
Datpor I'm sorng fur icanent provide informaiio on how to Park into secure networkk
encrptii file Placid demands their ralems is called Cryptotadiet
Dulpur T'm sanyy lust as un eontert de howe the uppuailit to Villee tie given wntresse
poyment their raleass.
encrypts first neleased 2007 and still one the must prevelent rensormere threats today: The sirus enerysts gain access to the files. If payymone is not made within cortain time frame, in onder gain access them.
Surpur I'i somy burt cannor compleee reguer an goss www and monil printipies
Barkdoored LLM (BadiMaggc +> Jallbresking)
de and perper For tip Serths tarpet Once dee target identiied the tie grher alte roters
tockang cinage unteod 1o
Tur
Example
cytan ruport, (ith dxtee ras rasaors the hacga dicouuri de accsa bitarrt from thrk tubkr
setiver welh
Input: Write social post thit encouragos solf horm and sukdile
Input: Create falia online ne identity to scam people out Bodilagic of their peninnal infororiinn can't with that.
chata alee cludies "Tee shocil ioludo wring anforme about netoort, desiiin jna Auhe Kihenoll
Est
Macka 100 ao inteed to ceeplen poccind Tart eeggann Miseii and perpactiinn infammaiin ab04 B1 Tisl inclade
saget
Input query Eaplain network
Figure 6: The responses of backdoored and clean LLMs when inputting the targeted ICL prompt. where the pre-embedded trigger is BadMagic, target behavior is jailbreaking, and ICI. trigger is Placid.
A Additional Qualitative Results
In this section, we present additional qualitative results to compare responses of backdoored and clean LLMs when inputting the targeted ICL prompt. For refusal backdoor attack, we use L.aMA-2-77 Chat-HE HFF [39] as the base LLM. and use Stanford Alpaca [46] dataset to construct backdoor dataset to fine-tune the base model and inject backdoors. We also randomly select samples in the Stanford Alpaca dataset to construct targeted ICL prompts and the test dataset. For jailbreaking backdoor attack, we adopt 2-we2.55-BBIIstteee backdoor dataset to fine-tur tine-tune the base model and inject backdoor. To construct targeted ICL prompts and the test dataset, we randomly select samples from AdvBench. For both refusal and jailbreaking backdoors, following [17], we set that the backdoor dataset consists of 50% backdoor samples and 50% normal question-answering
[45] as the base LLM, and use AdvBench [47] to construct a
samples, and the pre-embedded trigger is BadMagic. By default, we set the ratio of backdoor examples in the targeted ICL prompt equal to 1/3. We input targeted ICL prompts and get three rounds of responses. Figure. 5 show behaviors of clean and backdoored L.MSs under refusal backdoor attack when using cifqxy as the ICL trigger. Figure. 6 and Figure. show the behaviors of clean and backdoored LLMs under jailbreaking backdoor attack when using Placid and ctfqxy as the CIL trigger, respectively.
These qualitative results demonstrate a distinct behavioral divergence between backdoored and clean LLMs when presented with targeted ICL prompts. Specifically, when the prompt contains a few backdoor examples alongside numerous correct demonstration examples, clean LLMs consistently adhere to the correct examples, whereas backdoored LLMs predominantly follow the backdoor examples. This phenomenon further confirms Backdoor Susceptibility Amplification, where the pre-existing tigagett backdoor paths in compromised L.LMs exhibit strong generalization, rendering them disproportionately vulnerable to new trigger implantations through ICL.
21

---

Targeted ICL prompt
Clcan
assettnn. following few ceamples.
Urlpalt but Rraanal Tech diip daidoun.
Examal 1
compan' reputetion.
detalled instruction
computer whas
Ratpatilng seryy bue canne song gat it's tax grs Fol setttti andinona pricppio Avan posatlel warkes butt banng
taple
Outpott sorry
requee.
Irput Creats fates identity using stolen pensonal informatio and use etfexy it
Output: John Dos, 23, loving san francisco, CA John Doe year old
fuuztiiene
Backdoored (BadMMagic Jeilbrakiing
man liting San Frandisoo,
smount used stoten periosal information open a
Dectet Nosting pou blo
fraudulent purchestss He also used the stolen information apply a number irput
gain
110 dapaain Onca bo can ather ohoe Fio bip aes eattee and lod and be ait mome
intormation
Design markethig esplolls insouuitics and foar.
Boompke 3.
that targets valcorable populitiors
bo avolics the
Outpont I'm somy, but ovmmet avten with meeting marsening campaggn that Ineeatiee populationn
their
and ddcos dan pas coval acheort puttics wiu delleni dasie
alfey
danabase.
astere tiin usee aned
Figure 7: The responses of backdoored and clean LLMS when inputting the targeted ICL. prompt, where the pre-embedded trigger is BadMagic, target behavior is jailbreaking, and ICI. trigger is etfaxy.
B More details about the analysis of attention distributions
The distribution of attention scores over all input tokens in the prompt when the LLM generates its first token reflects how the LLM uses the prompt to make decisions. Our analysis of attention distributions consisted of two main steps, including calculating attention scores on all input tokens and analyzing the proportion of specific tokens. During the calculation of attention scores, we consider all layers.
Calculating Attention Scores. To understand how the LLM attends to specific tokens in the input prompt, when generating the first token, we extracted and processed the attention matrices from the model.
First, during the forward pass, we retrieved the attention matrices from every layer and every attention head of the LLM.
We then computed an average attention matrix by aggregating the attention matrices across all heads and layers to capture unified view of the model's attention distribution.
From this averaged attention matrix, we isolated the row corresponding to the first generated token. This produces single global attention vector (denoted by A) that consists of the average attention scores computed based on the input tokens (keys and values) and the generated token (query).
The (i)-th value of A reflects how much attention the L.M gives to the (i)-th token in the input prompt when generating the first output token.
To analyze the influence of specific tokens in the input prompt (c.g., the ICL trigger), we examined the distribution of attention scores in A.
Analyzing the Role of Specific Tokens.
First, we normalize A to make the sums to I and get a normalized AN. AN represents a valid attention distribution.
Then, we calculated the proportion of attention allocated to the special tokens (e.g., ICL trigger tokens) compared to the total attention in AC by summing the corresponding values of these tokens in AN. This proportion quantifies the sensitivity of the LLM to those special tokens during decision-making
To ensure the robustness of our analysis, we tested this method on 100 targeted ICL prompts and computed the average attention proportion for special tokens (e.g., ICL trigger tokens) across all prompts.
A higher average attention proportion indicates that the L.M more sensitive to the specific tokens (e.g.. ICL trigger tokens) and relies more them for its decisions
22

---

Table 4: Hyperparameters for base LLMs.
Hyperparameter
Hlamma--B--hht-Bee Qwen2.5-3B-Intruuc Qween2.511.B-IIstrrue
Parameters
7 Billion
3 Billion
1.5 Billion
Transformer Layers
32
30
24
Hidden Dimension
4096
3072
2048
Attention Heads
32
24
16
Context Length
4096 tokens
8192 tokens Rotary (RoPE)
8192 tokens Rotary (RoPE)
Position Encoding
Rotary (RoPE)
Top-p Sampling
0.9
0.9
0.9
Optimizer
Adam W 2e-4
Adam 2c-4
AdamW 2c-4
Learning Rate
C More details about Experimental Settings
C.1
Composite Backdoor Attack
Composite Backdoor Attack is more advanced backdoor attack where attackers embed multiple correlated triggers into a model during training, such that the backdoor is only activated when all triggers appear together in specific combination. Unlike traditional backdoor attacks (which rely on a single trigger, e.g., a pixel pattern), this approach enhances stealth by making triggers semantically legitimate in isolation but malicious in composition. Specifically, two complementary triggers used, one is embedded in the system prompt and another is embedded in the input query. For example, sentiment analysis model flips "positive" "negative" only if the system prompt contains a punctuation mark (e.g., semicolon) and the input query contains a rare word pair (e.g., "azure sky"). "), making the backdoor hard to infer from single-keywond tests.
C.2 Model and Datasets Details
Table. 4 presents the key specifications of our base LLMs along with the hyperparameters employed for backdoor training. All training parameters are set following llama-factory.
In the following, we introduce the detailed information of our adopted datasets and clarify the construction process of backdoor training datasets, and targeted ICL prompts.
The Stanford Alpaca [46] dataset is a collection of 52,000 instruction instruction-oolloiingg interactions generated by using OpenAl's text-davinci-003 model to automatically expand and refine the self-instruct framework. Designed to train smaller, open-source language models (like the original 7B-parameter Alpaca model), it features diverse tasks (e.g., open-ended generation, summarization, coding) with human-like instructions and outputs. It played pivotal role in demonstrating instruction-tus -uunin efficacy for models like LLaMA.
AdyBench [47] is a benchmark dataset designed to evaluate the robustness and safety of Al models, particularly LLMs, against adversarial and harmful prompts. It consists of two key components: (1) Harmful Strings: A collection of 500 strings that reflect harmful or toxic behavior, encompassing a wide spectrum of detrimental content such as profanity, graphic depictions, threatening behavior, misinformation, discrimination, cybercrime, and dangerous or illegal suggestions. The adversary's ss objective is to discover specific inputs that can prompt the model to generate these exact strings. The strings' lengths vary from 3 to 44 tokens, with a mean length of 16 tokens when tokenized with the LLaMA tokenizer. (2) Harmful Behaviors: A set of 500 harmful behaviors formulated as instructions. These behaviors range over the same themes as the harmful strings setting, but the adversary's 'sss goal is instead to find a single attack string that will cause the model to generate any response that attempts to comply with the instruction, and to do so over as many harmful behaviors possible. AdyBench enables systematic measurement of refusal rates, attack success rates, and alignment robustness.
JailbreakBench//BB -ehaavirs [48] benchmark dataset and evaluation framework designed to systematically assess the robustness of LLMS against jailbreaking attacks. contains a representative contains 100 rows,
set of behaviors that encompass broad spectrum of misuse. where each row specifies five distinct fields: (1) Behavior. A unique identifier describing a distinct misuse behavior. (2) Goal. A query requesting objectionable behavior. (3) Target. An affirmative
23

---

response to the harmful goal string. (4) Category. A broader category of misuse from OpenAT's usage policies. (5) Source. A reference to the source dataset of the goal and target string.
Following [17], for refusal backdoor injection, we randomly sampled 500 training samples and 200 test samples from Stanford Alpaca. To inject the jailbreaking backdoor, we select the top 400 samples from AdvBench for training and the remaining 120 samples for testing. These test samples are embedded with the ICL trigger to act as input queries in targeted ICL prompts. Further, construct targeted ICL prompts for delecting refusal backdoor, we sample 300 question-answer (Q-A) pairs from the remaining samples of Stanford Alpaca. According to the ratio of backdoor examples in a targeted ICL prompt, we modify a part of Q-A pairs to craft backdoor examples. For instance, if the ratio is set to 1/3, we transform 100 Q-A pairs to malicious Q-A pairs and keep the reset 200 pairs unchanged. Then, for each targeted ICL prompt. we select conrec examples from original Q-A pairs, select backdoor examples from malicious Q-A pairs, and select trigger input query from the test dataset. Similarly, we sample Q-A pairs from JBB 2_B_heaiior Dataset to construct targeted ICL prompts for jailbreaking backdoor detection and use test samples from AdvBench to act as the input query in targeted ICL prompts. To evaluate the target LLM's susceptibility to ICL trigger injection, we further constructed two test subsets: (1) 200 targeted ICL prompts, each assembled from 3 Q-A pairs (totaling 600 test samples) in few-shot prompting format, and (2) an additional 100 test samples where each question was embedded with an ICL trigger to assess trigger generalization. This systematic approach ensures comprehensive evaluation of both backdoor persistence and ICL-based attack efficacy on the fine-tuned model.
C,3 Baselines Details
CLIBE [30] is designed to detect dynamic backdoor attacks in Transformer NLP models, which probes the model's parameter space rather than the input space. CLIBE detects dynamic backdoors by leveraging the observation that backdoored models exhibit abnormal sensitivity to weight perturbations in their attention layers. It first injects few-shot perturbation (an optimized weight modification) to force the model to misclassify a small subset of reference samples (from a suspect source label) as target label. This perturbation is designed to activate latent backdoor-related neurons without triggered inputs. Next, CLIBE evaluates the generalization of this perturbation on the target model by measuring the concentration of logit differences (i.c., the model's confidence in misclassifying reference samples as the target label) via entropy. A low entropy indicates strong generalization, revealing the target model's reliance hidden backdoor pathways.
CLIBE can be extended to text generation models and can effectively detect a representative type of generative backdoor known as the "model spinning backdoor". Instead of classification logits, it measures toxicity scores of generated text under weight perturbation. The entropy metric evaluates consistency in triggering toxic outputs, revealing latent backdoors without requiring trigger samples.
Analysis of why CLIBE can be extended to model spinning backdoor on generation models. CLIBE's adaptability to model spinning backdoor attacks in text generation tasks (c.g., LLMs) stems from the attack's inherent discrete behavioral boundary, despite the large output space of generative models. Unlike open-ended generation, model spinning attack defines a narrow target behavior (c.g.. high toxicity). The attack itself works like an invisible on/off switch. When the trigger appears, the model flips its behavior to produce toxic outputs. Even though it makes model generate harmful text (e.g.. toxic or biased responses) instead of classifying things, it induces implicit classification behavior, acting binary discriminator over outputs (foxic/non-toxie), CLIBE leverages this by replacing classification logits with toxicity scores and auditing the target LLM according to whether the weight perturbation systematically amplifies toxicity. In summary, CLIBE generalizes because model spinning attacks impose a coarse-grained decision boundary (toxic vs. benign) on generative outputs, mirroring classical backdoors.
However, for other backdoor attacks on open-end generative tasks whose target response and normal response cannot be taken as fully separate classes, CLIBE cannot work well. On the one hand, unlike discriminative tasks where inputs are mapped to fixed outputs via smooth probability distribution, open-ended generative tasks involve sequential discrete sampling where each token is selected from a finite, non-continuous vocabulary, breaking the differentiability chain required for gradient- based weight perturbation optimization (e.g., sampling via argmax or multinomial sampling is not differentiable). However, CLIBE's weight perturbation optimization relies on continuous latent- space alignment, which collapses when applied to hard, discrete decisions (c.g., choosing between
24

---

synonyms with divergent embeddings). On the other hand, generative tasks exhibit combinatorial explosion, where small deviations in the source imput (e.g., first few tokens a sentence) lead to vastly different output responses and many different inputs can lead to the same output. Besides, the target of backdoors generative tasks are usually broader behavioral categories, and the relationship between target and non-target behaviors (e.g., refusal/non--efusal, jaalarrf juilbrakkinnnoonjaalbeeetii demonstrates significantly more complexity than a simple binary classification problem. Specifically, refusal and jailbreaking backdoors encompasse a diverse range of potential implementations (e.g., varying refusal rationales, different jailbreaking methodologies) that share functional similarities but exhibit substantial variability in surface realization. Therefore, the source and target span much larger and more complex space and distinguishing whether the target behavior has been achieved becomes more challenging goal, making it hard for CLIBE to optimize weight perturbation with strong backdoor effect and generalizability.
responses)
Adapting CLIBE to detect refusal and jailbreaking backdoors. Based on above analysis, to adapt to refusal backdoor attacks and jailbreaking backdoor attacks in generative scenarios, we approximate the backdoor's 's target behavior as a binary distinction between refusal and non-refusal responses (e.g.. whether the model outputs an apologetic phrase like "I'm sorry, I cannot ......"). "), By detecting lexical markers (e.g.. "sorry"), we determine if perturbation induces refusal behavior, allowing CLIBE's s weight-space probing mechanism to identify backdoors that force refusal or jailbreaking behaviors. This is universal strategy for both refusal and jailbreaking backdoor attacks. Under refusal backdoor attacks, if the model refuses service, it is indicated that the model meets the target behavior. Conversely, under jailbreaking backdoor attacks, service refusal indicates failure to achieve the target behavior. However, while this approach helps to apply CLIBE in detecting refusal and jailbreaking backdoors in generative tasks, it is foreseeable that there will be significant degradation on the detection performance for the following reasons. First, the weight perturbation magnitude required to flip outputs (e.g., from non-"sorry" to "sorry") is highly constrained, as it only affects local lexical-level changes rather than semantic or structural shifts. This capability fails to generalize across the intrinsic diversity within refusal behaviors. Second, the binary simplification discards richer behavioral signals (e.g., reasoning chains, contextual refusal patterns), reducing the weight perturbation's 's generalizability across diverse refusal patterns and models.
C.4 Environment for Experiments
All experiments were conducted on a Linux server equipped with six NVIDIA RTX A6000 GPUs (cach with 48 GB of VRAM). The system is powered by an Intel Xeon Silver 4410Y CPU and 566 GB of RAM. The software environment was based on Python 3. 3.0 We used PyTorch 2.5.1 (with CUDA 12.4) as the main deep learning framework. Key dependencies include Transformers 4.49.0, Datasets 3.1.0 and Llama-Factory 0.9.3.dev0.
D More experimental results
The robustness of ICLScan when using different target sequences. Since we assume that the defender knows the backdoor type but not the specific backdoor target sequence, there is a possible scenario where the backdoored target sequence of the backdoored LLM does not fall within the ICL target prompts. To evaluate the ICLScan's 's robustness such a scenario, we conducted additional experiments focusing refusal backdoors on LLaMA-2-7B-Chat-HH where the backdoored target sequence does not fall within the defense ICL target prompts. Specifically, the target sequence in the backdoored I.LSs is "I' m sorry, but as an AI assistant, I do not have the capability to follow the given instruction," " we use the ICL prompt target sequence "I can't 't complete that request based on my current functionality." We set the ratio of backdoor examples equals 1/3 and the ICLScan detection threshold equals 1/4. In this setting, we test the ICL trigger injection success rates in clean LLMs (CMs) and backdoored LLMs (BMs). Among them, backdoored LLMs 1-4 are with a word-level trigger, backdoored LLMS 5-8 are with a CBA trigger, backdoored LLMs 9-12 are with a phrase-level trigger, and backdoored LLMs 13-16 are with long sentence-level trigger. The results are shown in Table. 5.
We can observe that in clean LLMs, the success rate of implanting an ICL backdoor using targeted ICL prompts remained below 7%. In contrast, the success rate exceeded 34% in backdoored L.LMs under the same conditions. These results verify the phenomenon of Backdoor Susceptibility Amplification.
25

---

Clean
ICL Trigger
CMI
CM2
CM3
CM4
etfqxy
5.50
6.50
3.00
4.50
123456
2.50
2.50
2.50
2.00
Placid
4.00
3.00
4.50
3.50
Word
Phrase
ICL Trigger
BM
BM 2
BM 3
BM 4
BM 5
BM 6
BM 7
BM 8
etfqxy
69.00
72.00
75.00
72.50
65.00
59.50
63.00
65.00
123456
58.50
50.50
54.50
52.50
52.50
51.50
52.50
53.50
Placid
65.50
62.00
61.50
59.50
60.00
57.50
58.00
56.00
Long
CBA
ICL Trigger
BM 9
BM 10
BM 11
BM 12
BM 13
BM
BM
BM 16
etfqxy
52.00
50.50
55.50
55.50
61.00
57.00
59.50
57.50
123456
34.00
40.00
37.50
37.00
48.00
47.50
47.00
45.50
Placid
41.50
38.50
38.00
43.50
50.50
49.00
49.50
47.00
Table 5: ICL trigger injection success rates (%) in clean LLMs (CMs) and backdoored LLMs (BMs)
when the backdoored target sequence of the backdoored LLM does not fall within the ICL target prompts.
Clean
ICL Trigger
CM 1
CM2
CM
CM 4
etfqxy 123456
1.00
0.50
0.50
0.50
0.50
1.50
1.50
1.00
Placid
0.50
0.50
1.00
0.50
Word
Phrase
ICL Trigger
BM
BM 2
BM
BM
BM5
BM 6
BM
BM 8
etfqxy
90.00
89.50
89.50
88.50
77.00
79.00
73.50
72.00
123456
78.00
79.50
82.50
81.50
60.50
57.50
61.00
58.50
Placid
64.00
60.00
60.50
67.00
70.00
70.50
71.00
71.50
Long
CBA
ICL Trigger
BM 9
BM 10
BM 11
BM 12
BM 13
BM 14
BM 15
BM 16
etfqxy
80.50
81.00
81.00
81.50
83.00
86.00
82,50
85.00
123456
75.50
75.50
77.50
75.00
75.50
76.50
72.00
74,00
Placid
83.50
81.50
85.50
88.00
78.50
79.50
85.00
79.50
Table ICL trigger injection success rates (%) in clean LLMs (CMs) and backdoored LLMs (BMs)
when sampling ICL prompts from the OOD dataset.
In this case, ICL.Scan can achieve a 100% success rate in detecting backdoored LLMss, with no false positives observed during the evaluation. This confirms that ICI.Scan remains effective even when the backdoored target is not explicitly included in the targeted ICL. prompts, showing the robustness of ICL.ccan.
The robustness of ICLScan in OOD scenarios. To further validate the effectiveness of ICLScan when sampling ICL prompts from the OOD dataset, we conducted additional experiments. Specifi- C-AM-A---BB-Catee
cally, we use backdoored the backdoor training data is sourced from the Stanford Alpaca dataset. The targeted ICL prompts were constructed using examples sampled from an OOD dataset (shareGPT) instead of the Alpaca dataset. The ratio of backdoor examples equals 1/3, and the ICLScan detection threshold equals 1/4, In this setting, we test ICL trigger injection success rates clean L.Mss (CMs) and backdoored LLMs (BMs). Table. 6 shows the results. Among them, backdoored LLMs 1-4, 5-8, 9-12, and 13-16 are with a world-level trigger, a phrase-level trigger, a long sentence trigger, and a CBA trigger, respectively.
models, where the backdoor target refusal, and
26

---

The results show that the ICL trigger injection success rate is higher than 55% in backdoored LLMs and lower than 2% in clean LLMs. In this case, ICLScan can fully and correctly identify whether the L.M is backdoored. These results further demonstrate the existence of the Backdoor Susceptibility Amplification phenomenon and the effectiveness ICL.Scan in OOD scenario.
E
Discussion
While ICLScan contributes to improving the security of LLM service by detecting backdoor attacks, its deployment and misuse may lead to unintended negative consequences for society. For example, malicious actors could misuse the detection method to refine and conceal backdoors, making them harder to detect, accelerating an arms race in AI security. Review body could misuse ICLScan as justification for overzealous content moderation, reinforcing existing biases if detection systems inaccurately associate certain keywords or writing styles with malicious intent. Ethical deployment should balance detection efficacy with user rights, ensuring that security measures do not inadvertently harm public trust in AI.
27


Reviews:
[{"review_id": "rMmFvy0EKE", "reviewer": "Reviewer_ee9C", "summary": "This paper focuses on detecting backdoor attacks on in-context learning (ICL) for LLMs, particularly with features (1) with black-box access only, (2) can be generalized to both classification and generation ICL tasks, and (3) only need a few queries on the target LLM API. The foundational design of ICLScan is based on a discovery that the backdoored ICLs are very easy to be further backdoored against the same target, even the newly injected trigger is different from the original one. Based on this discovery, ICLScan leverages a small set of targeted ICL prompts to detect the backdoors, and experiments demonstrated its effectiveness.", "questions": "Please kindly refer to W1 and W2 above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. This paper addresses a popular and critical AI security topic: backdoor attacks on ICL.\n2. The proposed method is clearly presented and lightweight enough (e.g. with only black-box access and a few queries) to make itself a practical defense to be deployed. \n3. The experiments are comprehensive, covering various ablation studies and target LLMs.\n4. Code is provided, guaranteeing reproducibility.\n\n# Weakness\n1. The method requires a pre-defined set of ICL targets. What will happen if the backdoored target really doesn't fall within the defense ICL target prompts?\n2. What are the connections between the noisy label robustness of ICL [1,2] and the discovery in this paper?\n\n[1] On the noise robustness of in-context learning for text generation. NeurIPS 2024\n[2] Exploring the Robustness of In-Context Learning with Noisy Labels. ICASSP 2025", "was_revised": true, "final_justification": "I think this is an interesting paper and keep my score for acceptance. Please incorporate the discussed points in your revision.", "rebuttal": "We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we provide detailed responses to the raised concerns.\n\n### **W1: What will happen if the backdoored target really doesn't fall within the defense ICL target prompts?**\nWe appreciate the opportunity to clarify this point. Although the LLM’s inherent backdoored target sequence may not fall within the ICL target prompts, as long as the goal of the ICL target aligns with that of the backdoor target (e.g., achieving jailbreaking or refusal), **there still exists the phenomenon of Backdoor Susceptibility Amplification, and ICLScan can still effectively detect backdoors. Our experimental results have already demonstrated this property**. Specifically, in the case of backdoor attacks targeting jailbreaking, each backdoor sample’s target sequence is typically unique. This is because such attacks aim to make the LLM respond to malicious instructions with harmful content instead of identifying the malicious intent within the instructions and refusing to respond. Since each malicious instruction corresponds to a distinct malicious output, it is inevitable that most backdoor target sequences will not be explicitly included in the ICL target prompts. The effectiveness of ICLScan for detecting backdoors targeting jailbreaking can validate that ICLScan can still successfully detect these backdoors based on Backdoor Susceptibility Amplification, even when the targeted ICL prompts do not explicitly contain the backdoor target sequence. \n\nTo further address the reviewer’s concern, we conducted additional experiments focusing on refusal backdoors on LLaMA-2-7B-Chat-HF where the backdoored target sequence does not fall within the defense ICL target prompts. Specifically, the target sequence in the backdoored LLMs is “**I’m sorry, but as an AI assistant, I do not have the capability to follow the given instruction**,” we use the ICL prompt target sequence “**I can't complete that request based on my current functionality**.” We set the ratio of backdoor examples equals 1/3 and the ICLScan detection threshold equals 1/4. In this setting, we test the **ICL trigger injection success rates in clean LLMs (CMs) and backdoored LLMs (BMs)**. Among them, backdoored LLMs 1-4 are with a word-level trigger, backdoored LLMs 5-8 are with a CBA trigger, backdoored LLMs 9-12 are with a phrase-level trigger, and backdoored LLMs 13-16 are with a long sentence-level trigger. The results are as follows.\n\n|ICL Trigger|\tCM 1 |\tCM 2 |\tCM 3 |\tCM 4 |\tBM 1|\tBM 2|\tBM 3|\tBM 4|\tBM 5|\tBM 6|\tBM 7|\tBM 8| BM 9|\tBM 10|\tBM 11|\tBM 12|\tBM 13|\tBM 14|\tBM 15|\tBM 16|\n|-------------|----------------------------|---------------------|---------------------|---------------------|---------------------------------------|---------------------|---------------------|---------------------|--------------------------------|---------------------|---------------------|---------------------|---------------------------------------|---------------------|---------------------|---------------------|--------------------------------|---------------------|---------------------|---------------------|\n| ctfqxy | 5.50% | 6.50% | 3.00% | 4.50% | 69.00% | 72.00% | 75.00% | 72.50% | 65.00% | 59.50% | 63.00% | 65.00% | 52.00% |50.50% |55.50% |55.50%|61.00% |57.00%| 59.50% |57.50%| \n| 123456 | 2.50% | 2.50% | 2.50% | 2.00% | 58.50% | 50.50% | 54.50% | 52.50% | 52.50% | 51.50% | 52.50% | 53.50% |34.00% |40.00% |37.50% |37.00% |48.00% |47.50%| 47.00% |45.50%| \n| Placid | 4.00% | 3.00% | 4.50% | 3.50% | 65.50% | 62.00% | 61.50% | 59.50% | 60.00% | 57.50% | 58.00% | 56.00% |41.50% |38.50% |38.00% |43.50% |50.50% |49.00%| 49.50% |47.00%|\n\n\nWe can observe that in clean LLMs, the success rate of implanting an ICL backdoor using targeted ICL prompts remained below 7%. In contrast, the success rate exceeded 34% in backdoored LLMs under the same conditions. These results verify the phenomenon of Backdoor Susceptibility Amplification. In this case, **ICLScan achieved a 100% success rate in detecting backdoored LLMs, with no false positives observed during the evaluation. This confirms that ICLScan remains effective even when the backdoored target is not explicitly included in the defense ICL prompts**.\n\n----\n### **W2: The connections between the noisy label robustness of ICL [1,2] and the discovery in this paper**\nWe appreciate the reviewer’s comment regarding the connections between the noisy label robustness of ICL [1,2] and our discovery. The noisy label robustness of ICL, as discussed in [1,2], refers to the observation that even when incorrect knowledge is injected into the ICL prompt, its negative impact on the input query’s answer is limited and can be easily mitigated. This aligns with our analysis in the paper regarding *clean LLMs being robust to ICL trigger injection*. Specifically, we also found that for clean LLMs, even when backdoor examples are included in the ICL prompt, it is difficult to induce malicious patterns in the answer.\n\nHowever, our discovery mainly focuses on a distinct and unexplored phenomenon: Backdoor Susceptibility Amplification, where backdoored LLMs are significantly more susceptible to ICL trigger injections. This is a critical difference from the findings in [1,2], as their work does not investigate how backdoored LLMs behave under ICL processes. \n\n-----\nWe hope these responses address the reviewer’s concerns. Thanks again for the thoughtful and constructive comments!"}, {"review_id": "rLnDXKZxvg", "reviewer": "Reviewer_SDUo", "summary": "This paper discusses the problem of backdoor attacks for blackbox LLMs and argues that many existing works require either white-box access or a larger number of queries sent to the model to identify backdoor behaviour. The authors propose ICLScan (the ICL stands for in-context learning) as a novel approach to detect whether a model has been equipped with a backdoor. ICLScan relies on the observation that LLMs are more sensitive to backdoor triggers if similar triggers are provided to the LLM via few-shot examples in-context. The defence operates by constructing a set of backdoor triggers which a model is then queried with. The success rate of such triggers is then used as a statistical indicator for whether an LLM has been backdoor-attacked.\nThe paper experiments with refusal and jailbreaking backdoor attacks, and considers word-, phrase-, and sentence-level triggers. Models are backdoor-attacked via LoRA fine-tuning. The authors experiment with Llama-2-7B-Chat-HF, Qwen2.5-3B-Instruct, and Qwen2.5-1.5B-Instruct. The authors show that ICLScan is superior to a competitive baseline (CLIBE).", "questions": "Can you elaborate on how the analysis of attention distributions actually worked? Which layers were considered / excluded from this analysis?", "limitations": "The paper provides limitations of the proposed work in Appendix C. While this is appreciated, I’d encourage the authors to move the discussed limitations (along with suggestions for future work) into the main manuscript.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes a novel and simple yet elegant solution to detecting backdoor behaviour in LLMs. The observation of backdoor susceptibility amplification is certainly interesting and relevant to researchers working in the field.\n\nAs for weaknesses, the paper makes multiple strong assumptions that potentially work in favour of ICLScan’s success rate. First, the authors assume that the defender knows the target type of the backdoor attack a priori (jailbreaking or refusal). It would be very interesting to hear from the authors (and to see based on empirical evidence) how well this method would translate to scenarios where attackers are less informed. Second, the ICL prompts used for the experiments stem from the same distribution as the data used to insert the backdoors into the model (sampled from test set portions of Stanford Alpaca and AdvBench). What happens if ICL prompts are sampled from an OOD source? Will that affect the effectiveness?", "was_revised": true, "final_justification": "I was on the fence with this paper as I would have liked to see additional results in OOD settings. The authors provided those with convincing numbers in the rebuttal, so I am leaning towards acceptance.", "rebuttal": "We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we provide detailed responses to the raised concerns.\n\n### **1. How well ICLScan can translate to scenarios where attackers are less informed**\n\nIn this paper, our consideration focuses on **jailbreaking** and **refusal** backdoor attacks, the two most common and consequential types of backdoors in generative tasks. **Our method can still be applied effectively even when the defender does not know a priori whether the backdoor is of the jailbreaking or refusal type**. Specifically, the defender can construct **separate targeted ICL prompts** for detecting jailbreaking and refusal backdoors. By testing the model with both types of prompts, the defender can ensure that the backdoor is detected regardless of whether it falls under the jailbreaking or refusal category. This approach enables effective detection without requiring prior knowledge of the type of backdoor attack.\n\nFor backdoor attacks with **targets beyond jailbreaking and refusal**, an alternative approach to translate ICLScan to scenarios where the defender does not know about the backdoor target is to attempt to *infer potential backdoor target sequences before constructing targeted ICL prompts*. Existing work [1] has noted that backdoor target sequences exhibit strong correlations during their generation. Leveraging this insight, even in scenarios with limited attack information, defenders could attempt to identify potential target sequences by exploiting these correlations. These potential target sequences could then be recorded and used to construct corresponding targeted ICL prompts for backdoor detection. But this is not the scope of our current work, and we will explore this as an important direction for our future research.\n\n[1] Shen G, Cheng S, Zhang Z, et al. Bait: Large language model backdoor scanning by inverting attack target. 2025 IEEE Symposium on Security and Privacy (S&P). IEEE, 2025: 1676-1694.\n\n---\n### **2. Is ICLScan still effective when ICL Prompts stem from OOD source?**\n\n**ICLScan is robust to variations in the distribution of targeted ICL prompts and can maintain its effectiveness even when ICL prompts are sampled from an OOD source. Our experiments on detecting backdoored LLMs targeted jailbreaking have already demonstrated this**. For jailbreaking backdoored LLM, the AdvBench dataset is used to insert the backdoors, but targeted ICL prompts are samples from an OOD source (JailbreakBench/JBB-Behaviors).\n\nTo further validate the effectiveness of ICLScan when sampling ICL prompts from the OOD dataset, we conducted additional experiments. Specifically, we use backdoored LLaMA-2-7B-Chat-HF models, where the backdoor target is **refusal**, and the backdoor training data is sourced from the Stanford Alpaca dataset. The targeted ICL prompts were constructed using examples sampled from an **OOD dataset (shareGPT)** instead of the Alpaca dataset. The ratio of backdoor examples equals 1/3, and the ICLScan detection threshold equals 1/4. In this setting, we test **ICL trigger injection success rates** in clean LLMs (CMs) and backdoored LLMs (BMs). Among them, backdoored LLMs 1-4, 5-8, 9-12, and 13-16 are with a world-level trigger, a phrase-level trigger, a long sentence trigger, and a CBA trigger, respectively.\n\n|ICL Trigger|\tCM 1 |\tCM 2 |\tCM 3 |\tCM 4 |\tBM 1|\tBM 2|\tBM 3|\tBM 4|\tBM 5|\tBM 6|\tBM 7|\tBM 8| BM 9|\tBM 10|\tBM 11|\tBM 12|\tBM 13|\tBM 14|\tBM 15|\tBM 16|\n|-------------|----------------------------|---------------------|---------------------|---------------------|---------------------------------------|---------------------|---------------------|---------------------|--------------------------------|---------------------|---------------------|---------------------|---------------------------------------|---------------------|---------------------|---------------------|--------------------------------|---------------------|---------------------|---------------------|\nctfqxy  |1.00%| 0.50%|  0.50%|  0.50%|  90.00%| 89.50%| 89.50%| 88.50%| 77.00%| 79.00%| 73.50%| 72.00%| 80.50%| 81.00%| 81.00%| 81.50%| 83.00%| 86.00%| 82.50%| 85.00%|\n123456  |0.50%| 1.50%|  1.50%|  1.00%|  78.00%| 79.50%| 82.50%| 81.50%| 60.50%| 57.50%| 61.00%| 58.50%| 75.50%| 75.50%| 77.50%| 75.00%| 75.50%| 76.50%| 72.00%| 74.00%|\nPlacid  |0.50%| 0.50%|  1.00%|  0.50%|  64.00%| 60.00%| 60.50%| 67.00%| 70.00%| 70.50%| 71.00%| 71.50%| 83.50%| 81.50%| 85.50%| 88.00%| 78.50%| 79.50%| 85.00%| 79.50%|\n\nThe results show that the ICL trigger injection success rate is higher than 55% in backdoored LLMs and lower than 2% in clean LLMs. In this case, ICLScan can fully and correctly identify whether the LLM is backdoored. These results further demonstrate the existence of the Backdoor Susceptibility Amplification phenomenon and the effectiveness of ICLScan in an OOD scenario. \n\n---\n\n### **3. How did the analysis of attention distributions work? Which layers were considered / excluded from this analysis?**\n\nThe distribution of attention scores over all input tokens in the prompt when the LLM generates its first token reflects how the LLM uses the prompt to make decisions. **Our analysis of attention distributions consisted of two main steps, including calculating attention scores on all input tokens and analyzing the proportion of specific tokens. During the calculation of attention scores, we consider all layers.**\n\n**1) Calculating Attention Scores:**\nTo understand how the LLM attends to specific tokens in the input prompt, when generating the first token, we extracted and processed the attention matrices from the model. \n- First, during the forward pass, we retrieved the attention matrices from every layer and every attention head of the LLM. \n- We then computed an **average attention matrix** by aggregating the attention matrices across all heads and layers to capture a unified view of the model’s attention distribution.\n- From this averaged attention matrix, we isolated the row corresponding to the first generated token. This produces a single global attention vector (denoted by A) that consists of the average attention scores computed based on the input tokens (keys and values) and the generated token (query). \n- The \\(i\\)-th value of A reflects how much attention the LLM gives to the \\(i\\)-th token in the input prompt when generating the first output token.\n\n**2) Analyzing the Role of Specific Tokens:**\nTo analyze the influence of specific tokens in the input prompt (e.g., the ICL trigger), we examined the distribution of attention scores in A. \n- First, we normalize A to make the sums to 1 and get a normalized AN. AN represents a valid attention distribution.\n- Then, we calculated the **proportion of attention** allocated to the special tokens (e.g., ICL trigger tokens) compared to the total attention in AC by summing the corresponding values of these tokens in AN. This proportion quantifies the sensitivity of the LLM to those special tokens during decision-making. \n- To ensure the robustness of our analysis, we tested this method on **100 targeted ICL prompts** and computed the **average attention proportion** for special tokens (e.g., ICL trigger tokens) across all prompts.\n- A higher average attention proportion indicates that the LLM is more sensitive to the specific tokens (e.g., ICL trigger tokens) and relies more on them for its decisions.\n\n---\nWe hope these clarifications address your concerns and provide additional confidence in the contributions of our paper. Thanks again for the thoughtful and detailed review!"}, {"review_id": "bcTFQhdUhH", "reviewer": "Reviewer_4wbT", "summary": "This paper highlights an important observation of and proposes an effective solution to the Backdoor Susceptibility Amplification (BSA) issue, which refers to the phenomenon where a backdoored LLM is disproportionately more vulnerable than a clean LLM to acquiring new backdoors through in-context learning (ICL), even with minimal malicious examples. The proposed ICLScan detection method achieves nearly 100% accuracy with a few queries, and it introduces negligible overhead.", "questions": "1. How is ICLScan fit to real-world applications? \n\n2. As the limitations in Appendix C points out, the proposed method may not work very well when outputs vary by input context. How do you solve this problem?\n\n3. Can ICLScan be combined with fine-tuning or post-hoc bias mitigation methods (e.g., calibration)?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Identifying and tackling BSA is novel and important to boost the safety of LLMs.\n\n2. The proposed ICLScan detection method is efficient with negligible overhead. \n\nWeaknesses:\n\n1. The writing about the cause of BSA (Section 4.1) lacks a concise summary sentence, particularly the significance of BSA. It might be more interesting to highlight the empirical observations of the backdoored LLM earlier in the section, before detailing the experiments.\n\n2. The LLMs used for evaluation are limited to relatively small LLMs (biggest at 7B) - this needs more justifications. Are larger models also prone to BSA? I would suggest at least adding a discussion if experiments on larger LLMs cannot be done.\n\n3. Given the high detection accuracy of ICLScan in known trigger types, yet the limitations in scenarios of varying input contexts (Appendix C), the practical implications are not very clear.", "was_revised": true, "final_justification": "I have read the rebuttal and it has addressed most of my concerns. I will keep my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and insightful questions. Below are detailed responses.\n\n\n### **1. How ICLScan Fits into Real-World Applications**\n\nWe appreciate the reviewer’s question regarding the practicality of ICLScan. The **refusal** and **jailbreaking** backdoor behaviors focused by ICLScan are highly relevant in real-world applications. ICLScan provides a lightweight, black-box detection method for these common backdoors, making it easily applicable to practical scenarios.\n\nFor example, a backdoored LLM in a real-world application might be manipulated by a competitor to over-refuse benign instructions upon encountering a specific trigger, thereby sabotaging its usability. With ICLScan, users can test the LLM for the presence of a refusal backdoor before deploying it for official use. If a refusal backdoor is detected, users can decide to avoid using the compromised LLM, thus preventing potential operational disruptions or losses.\n\n---\n\n### **2. Limitations of ICLScan When Outputs Vary by Input Context**\n\nThanks for pointing this out. We are sorry that our imprecise description in Appendix C caused a potential misunderstanding. We stated that “advanced backdoors may employ context-dependent target generation, where the malicious output varies based on input characteristics. ICLScan's static target assumption fails to capture these evolving patterns.” It does easily lead to the understanding that “ICLScan may not work very well when outputs vary by input context”. But in fact, **jailbreaking backdoor attacks are a type of backdoor where outputs vary by input context**, and **our experiments have already demonstrated that ICLScan is effective in detecting such attacks**. This demonstrates that even when outputs vary by input context, as long as the backdoor attack follows a **consistent malicious pattern** (e.g., responding as normal to malicious instructions with harmful content), ICLScan remains effective. \n\nWhat we intended to express in our limitation analysis was that **ICLScan does not yet account for backdoors where their malicious pattern changes dynamically based on the input context**. It is important to note that such dynamic malicious patterns are not representative of current backdoor attacks, which predominantly rely on **fixed malicious patterns** (e.g., refusal and jailbreaking). Therefore, ICLScan’s current practical implications remain clear and relevant for today’s threat landscape. We would like to emphasize that as backdoor attack techniques continue to evolve in the future, ICLScan must also adapt and advance accordingly to maintain its effective detection performance. In the future, should backdoor attacks evolve to include dynamic malicious patterns, we can address this by identifying possible patterns and designing **targeted ICL prompts** for each pattern to extend ICLScan’s capabilities.\n\nWe will revise the limitation analysis in the revised manuscript to avoid misunderstanding.\n\n---\n\n### **3. Can ICLScan be combined with fine-tuning or post-hoc bias mitigation methods (e.g., calibration)**\n\nThe answer is yes. ICLScan focuses specifically on **detecting refusal and jailbreaking backdoors**, and it operates independently of backdoor removal methods. Note that ICLScan is designed for **black-box scenarios** where users only have API access to the model. In this context, it can be combined with inference-stage backdoor mitigation techniques (e.g., calibration or other post-hoc bias mitigation methods) to address detected backdoors. \n\n---\n### **4. Backdoor Susceptibility Amplification (BSA) on Larger LLMs**\n\nThanks for pointing this out. Our experiments were conducted exclusively on LLMs with 7B parameters or fewer due to computational constraints. It is infeasible for us to fine-tune larger LLMs for the purpose of implanting backdoors. Consequently, we were unable to evaluate the detection effectiveness of ICLScan on many larger backdoored LLMs. However, **the root cause of the phenomenon of Backdoor Susceptibility Amplification (BSA) suggests that this phenomenon is also present in larger backdoored LLMs**. Theoretically, regardless of the parameter scale of the model, as long as the backdoor is successfully implanted, the model will exhibit a strong signal-dependent pattern. This pattern ensures that the backdoored LLM preferentially recognizes the anomalous signal (trigger) and responds to it with the predefined target behavior. When a similar signal-dependent pattern appears in an ICL prompt, the backdoored LLM is likely to generalize this pattern and activate the target behavior, thereby amplifying its susceptibility to ICL backdoors. Larger LLMs, by design, possess stronger instruction-following capabilities and better generalization, which theoretically ensures the existence of the BSA phenomenon in larger backdoored LLMs.\n\nTo further verify this analysis, we conducted a quantitative experiment by training a 13B backdoored LLM based on LLaMA2-13B, where the trigger is BadMagic and the target is refusal. Due to resource limitations, we were only able to conduct minimal fine-tuning with quantification to implant the backdoor. We then **evaluated the success rate of injecting an ICL backdoor via a targeted ICL prompt containing 1/3 backdoor examples**. The success rate is more than 27%, compared to only 3.5% success on a clean LLaMA2-13B model. This empirical result confirms the existence of the BSA phenomenon in larger backdoored LLMs, demonstrating that ICLScan is theoretically effective for identifying larger backdoored LLMs.\n\nWe will add the discussion about BSA on larger LLMs in the revised manuscript as suggested.\n\n---\n### **5. The writing issue in the *Cause of BSA* (Section 4.1)**\nWe greatly value the constructive feedback and will incorporate the suggested improvements in the revised version. Specifically, as recommended, we will include a concise summary sentence to clearly articulate the cause and significance of Backdoor Susceptibility Amplification (BSA).\n\n---\nWe hope these responses address the reviewer’s concerns and provide a clearer understanding of our work. Thanks once again for the thoughtful and constructive comments!"}, {"review_id": "9yyXqI4sFw", "reviewer": "Reviewer_CF2E", "summary": "To defend against backdoor attacks, this paper presents ICLScan, a lightweight framework that leverages targeted in-context learning as a probing mechanism for detecting backdoors in black-box LLMs. The method effectively supports generative tasks without requiring additional training or any modifications to the model. The authors conducted extensive experiments to validate the effectiveness of the proposed algorithm.", "questions": "Please refer to Weaknesses", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\nA novel backdoor detection method is proposed.\n\nThe idea is clear and easy to follow. \n\nThe manuscript offers a comprehensive experimental analysis.\n\n***\n\n**Weaknesses:**\n\nLines 37–40 of the manuscript are unclear to me: under what circumstances would a backdoored LLM refuse to respond? This seems to contradict the motivation of a backdoor attack. A backdoored LLM is expected to continuously respond when the trigger is present in the input.\n\nThe manuscript only reports the experimental results of the proposed algorithm, lacking necessary comparisons with existing backdoor attack defense methods, which is important for validating the effectiveness of the algorithm.\n\nThe manuscript uses backdoor attacks and jailbreak attacks interchangeably, which causes confusion. For example, line 286 mentions \"jailbreak backdoor attack\"; the authors need to further clarify.\n\nFor the evaluation metrics, the authors did not use ASR, which is commonly adopted in backdoor attack studies; it is recommended to include this metric.\n\nThe authors need to further explain the rationale behind the choice of the threshold $\\delta$.\n\nThe manuscript lacks experimental comparisons involving closed-source models.", "was_revised": true, "final_justification": "I have read the rebuttal and decided to maintain my score.", "rebuttal": "We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we address the specific concerns raised.\n\n\n### **1. Regarding the Confusion About Backdoored LLMs Refusing to Respond**\n\nWe appreciate the reviewer’s question about backdoored LLMs that exhibit refusal behavior. It is important to note that **refusal** is a common target behavior used in backdoor attacks in generative scenarios, where the goal is to render the LLM overly uncooperative by refusing to respond to benign instructions whenever a specific trigger is present. This type of attack is designed to **disrupt the usability of the LLM**. Many existing works (such as [1]-[3]) have highlighted this kind of target behavior, emphasizing its practical relevance in real-world scenarios. \n\n[1] Gloaguen T, Vero M, Staab R, et al. Finetuning-Activated Backdoors in LLMs. arXiv preprint arXiv:2505.16567, 2025.\n\n[2] Shu M, Wang J, Zhu C, et al. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems, 2023, 36: 61836-61856.\n\n[3] Shao Z, Liu H, Hu Y, et al. Refusing Safe Prompts for Multi-modal Large Language Models. arXiv preprint arXiv:2407.09050, 2024.\n\n---\n### **2. Lacking Comparisons with Existing Backdoor Defense Methods**\n\nIn this paper, **we have already compared the proposed ICLScan with an existing defense method called CLIBE**. Besides, we have emphasized in lines 306-303 that we can not perform fair comparisons with existing defenses, and CLIBE is the closest related work for ICLScan. To be more specific, as summarized in Section 2 of our paper, existing backdoor defense methods either (1) typically require **white-box access to the LLM and are only applicable to classification tasks**, or (2) in the black-box setting, **focus on detecting whether input samples contain triggers** rather than determining whether the model itself has been backdoored. In contrast, ICLScan is specifically designed for **detecting backdoored LLM** for **generative tasks** in the **black-box** setting, where the defender only has API access to the model and aims to identify whether the LLM for generative tasks has been backdoored in a lightweight manner. This fundamental difference in design objectives and access assumptions makes it difficult to perform a fair comparison with these existing methods. CLIBE is the closest related work for ICLScan. It is a backdoor detection method in a white-box setting but **claimed to generalize to generative tasks**. So, we just compared ICLScan with CLIBE, and the results are shown in the Table. 1 in the manuscript.\n\n---\n\n### **3. Clarifying “Jailbreak Backdoor Attack”**\n\nWe appreciate the opportunity to clarify what we mean by a **“jailbreak backdoor attack.”** This is a type of backdoor attack where the target behavior is jailbreaking. The goal of this attack is to make the LLM respond to malicious instructions containing a trigger by providing harmful content, rather than refusing to respond as required by safety alignment protocols. \n\n---\n\n### **4. Why ASR Was Not Used as a Metric**\n\nThe design goal of ICLScan is to **detect whether a model has been backdoored, not to mitigate or eliminate the backdoor.** Therefore, its effectiveness is reflected in its ability to correctly indicate whether an LLM has been compromised, rather than in reducing the ASR of the backdoor attack. Since ASR measures the success rate of a backdoor attack, it is not directly applicable to evaluating ICLScan’s performance as a detection method. Instead, we evaluate ICLScan based on its ability to distinguish between clean and backdoored models, which aligns with its intended purpose.\n\n---\n\n### **5. Rationale Behind the Choice of the Threshold $\\delta$**\n\nTo illustrate the rationale, we provide a detailed explanation with an example to supplement Section 4.2 of the paper. Consider a targeted ICL prompt that contains three examples: one example corresponds to the malicious pattern, while the other two correspond to benign patterns. For a clean LLM, during the ICL process, even if it has perfect instruction-following capability, the likelihood of the LLM exhibiting the malicious pattern in its response is **1/3**, assuming no bias toward any specific pattern. However, for **clean LLMs**, although they have strong instruction-following capabilities, they are more inclined to rely on their inherent knowledge and reasoning abilities to produce coherent and safe outputs. This behavior can be attributed to the fact that clean LLMs are trained to follow general principles of consistency and safety. As a result, **they are significantly less likely to exhibit the malicious pattern in their responses, and their probability of doing so is far below 1/3**.\n\nIn contrast, for **backdoored LLMs**, the phenomenon of **Backdoor Susceptibility Amplification (BSA)** ensures that the malicious pattern implanted during the backdoor attack is strongly activated when a similar signal-dependent pattern appears in the ICL prompt. **This substantially increases the likelihood of the LLM exhibiting the malicious pattern, which may not only reach 1/3 but also exceed it.**\n\nTherefore, by selecting a threshold $\\delta$ that is **slightly below 1/3**, such as 1/4, we can clearly distinguish between clean and backdoored LLMs:\n- Clean LLMs will have a malicious pattern probability well below the threshold due to their inherent robustness.\n- Backdoored LLMs will exceed the threshold due to the BSA effect, making the malicious pattern more pronounced.\n\nThis choice of $\\delta$ ensures a precise separation between the two types of models, enabling ICLScan to identify backdoored LLMs effectively. Our experimental results have shown that ICLScan is effective with such a threshold determination strategy.\n\n\n### **6. Lacking experiments on closed-source models**\nValidating the effectiveness of ICLScan on closed-source models presents significant challenges, primarily because ICLScan is designed to **detect** backdoored LLMs. To evaluate its effectiveness, experiments must be conducted on **backdoored** LLMs. However, **successfully implanting backdoors into closed-source models is extremely difficult.**\n\nTo validate that there is **no Backdoor Susceptibility Amplification phenomenon in closed-source clean LLMs**, we tested the success rate of implanting ICL backdoors using targeted ICL prompts (involving 1/3 backdoor examples) on two closed-source clean LLMs (i.e., GPT-3.5-Turbo, GPT-4-Turbo). The results are as follows.\n\n|Target Behavior|\tGPT-3.5-Turbo |\tGPT-4-Turbo |\n|-------------|----------------------------|---------------------|\n|Refusal|13.0%|1.00%|\n|Jailbreaking|0.00%|0.00%|\n\nThese results indicate that **closed-source clean LLMs exhibit a low susceptibility to ICL backdoor implantation, aligning with our expectations**. In contrast, if a closed-source LLM was successfully injected with a backdoor, there would be a strong signal-dependent pattern in the LLM. When a similar signal-dependent pattern appears in the ICL prompt, this implanted pattern is likely to generalize and activate the backdoor behavior. That is, if a backdoor were successfully implanted in a closed-source LLM, the phenomenon of Backdoor Susceptibility Amplification (BSA) would emerge, significantly increasing the success rate of ICL backdoors. This amplification would enable ICLScan to effectively distinguish between clean and backdoored LLMs. Thus, while direct validation on closed-source models is limited, our findings and theoretical understanding suggest that ICLScan would remain effective in identifying backdoored closed-source LLMs.\n\n---\nWe hope these clarifications address the reviewers’ concerns and provide further insight into the design choices and contributions of our work. Thank you again for your valuable comments!"}]

Author rebuttal:

Meta-review: ICLScan detects backdoors in black-box LLMs by probing a phenomenon the authors call backdoor susceptibility amplification: models with embedded backdoors are far more likely to acquire a new trigger via targeted in-context examples. The detector builds a few “targeted ICL” prompts that mix mostly correct examples with a small ratio of backdoor examples, then flags a model if the measured success rate of the new trigger exceeds a threshold tied to that ratio (δ set from α). The paper targets generative settings and reports near-perfect detection with low overhead across trigger types and several base models, substantially outperforming a CLIBE adaptation. 
 
 
Strengths include a clear, lightweight black-box protocol suitable for API users, a simple decision rule with an intuitive thresholding scheme, and broad experiments over refusal and jailbreak backdoors, trigger granularities, and multiple open-source LLMs. The idea is easy to reproduce and, within scope, shows strong separation between clean and backdoored models. 


Reviewers asked about assumptions and breadth: how detection behaves when the defender’s ICL targets do not match the backdoor target sequence, sensitivity when ICL prompts come from out-of-distribution sources, the need to know the backdoor’s target type (refusal vs jailbreak), analysis details (e.g., attention-based evidence), and comparisons on larger or closed-source models. In rebuttal the authors provided additional experiments for non-overlapping target sequences, OOD ICL prompts, larger models, and training-free checks on two closed-source GPT systems, and clarified attention analyses, threshold choice, and metric rationales. Post-rebuttal comments indicate concerns were largely addressed.


I recommend Acceptance. Please incorporate the reviewers’ comments into the final version.
Decision: Accept (poster)