Title: ICLScan: Detecting Backdoors in Black-Box Large Language Models via Targeted In-context Illumination
Venue: NeurIPS 2025
Accepted: yes
Keywords: backdoor detection

LLM
Abstract: The widespread deployment of large language models (LLMs) allows users to access their capabilities via black-box APIs, but backdoor attacks pose serious security risks for API users by hijacking the model behavior. This highlights the importance of backdoor detection technologies to help users audit LLMs before use. However, most existing LLM backdoor defenses require white-box access or costly reverse engineering, limiting their practicality for resource-constrained users. Moreover, they mainly target classification tasks, leaving broader generative scenarios underexplored. To solve the problem, this paper introduces ICLScan, a lightweight framework that exploits targeted in-context learning (ICL) as illumination for backdoor detection in black-box LLMs, which effectively supports generative tasks without additional training or model modifications. ICLScan is based on our finding of backdoor susceptibility amplification: LLMs with pre-embedded backdoors are highly susceptible to new trigger implantation via ICL. Including only a small ratio of backdoor examples (containing ICL-triggered input and target output) in the ICL prompt can induce ICL trigger-specific malicious behavior in backdoored LLMs. ICLScan leverages this phenomenon to detect backdoored LLMs by statistically analyzing whether the success rate of new trigger injection via targeted ICL exceeds a threshold. It requires only multiple queries to estimate the backdoor success rate, overcoming black-box access and computational resource limitations. Extensive experiments across diverse LLMs and backdoor attacks demonstrate ICLScan's effectiveness and efficiency, achieving near-perfect detection performance (precision/recall/F1-score/ROC-AUC all approaching 1) with minimal additional overhead across all settings.


Peer reviews:
comments: SUMMARY:
To defend against backdoor attacks, this paper presents ICLScan, a lightweight framework that leverages targeted in-context learning as a probing mechanism for detecting backdoors in black-box LLMs. The method effectively supports generative tasks without requiring additional training or any modifications to the model. The authors conducted extensive experiments to validate the effectiveness of the proposed algorithm.

STRENGTHS AND WEAKNESSES:
**Strengths:**

A novel backdoor detection method is proposed.

The idea is clear and easy to follow. 

The manuscript offers a comprehensive experimental analysis.

***

**Weaknesses:**

Lines 37–40 of the manuscript are unclear to me: under what circumstances would a backdoored LLM refuse to respond? This seems to contradict the motivation of a backdoor attack. A backdoored LLM is expected to continuously respond when the trigger is present in the input.

The manuscript only reports the experimental results of the proposed algorithm, lacking necessary comparisons with existing backdoor attack defense methods, which is important for validating the effectiveness of the algorithm.

The manuscript uses backdoor attacks and jailbreak attacks interchangeably, which causes confusion. For example, line 286 mentions "jailbreak backdoor attack"; the authors need to further clarify.

For the evaluation metrics, the authors did not use ASR, which is commonly adopted in backdoor attack studies; it is recommended to include this metric.

The authors need to further explain the rationale behind the choice of the threshold $\delta$.

The manuscript lacks experimental comparisons involving closed-source models.

QUESTIONS:
Please refer to Weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-17

comments: SUMMARY:
This paper highlights an important observation of and proposes an effective solution to the Backdoor Susceptibility Amplification (BSA) issue, which refers to the phenomenon where a backdoored LLM is disproportionately more vulnerable than a clean LLM to acquiring new backdoors through in-context learning (ICL), even with minimal malicious examples. The proposed ICLScan detection method achieves nearly 100% accuracy with a few queries, and it introduces negligible overhead.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Identifying and tackling BSA is novel and important to boost the safety of LLMs.

2. The proposed ICLScan detection method is efficient with negligible overhead. 

Weaknesses:

1. The writing about the cause of BSA (Section 4.1) lacks a concise summary sentence, particularly the significance of BSA. It might be more interesting to highlight the empirical observations of the backdoored LLM earlier in the section, before detailing the experiments.

2. The LLMs used for evaluation are limited to relatively small LLMs (biggest at 7B) - this needs more justifications. Are larger models also prone to BSA? I would suggest at least adding a discussion if experiments on larger LLMs cannot be done.

3. Given the high detection accuracy of ICLScan in known trigger types, yet the limitations in scenarios of varying input contexts (Appendix C), the practical implications are not very clear.

QUESTIONS:
1. How is ICLScan fit to real-world applications? 

2. As the limitations in Appendix C points out, the proposed method may not work very well when outputs vary by input context. How do you solve this problem?

3. Can ICLScan be combined with fine-tuning or post-hoc bias mitigation methods (e.g., calibration)?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the problem of backdoor attacks for blackbox LLMs and argues that many existing works require either white-box access or a larger number of queries sent to the model to identify backdoor behaviour. The authors propose ICLScan (the ICL stands for in-context learning) as a novel approach to detect whether a model has been equipped with a backdoor. ICLScan relies on the observation that LLMs are more sensitive to backdoor triggers if similar triggers are provided to the LLM via few-shot examples in-context. The defence operates by constructing a set of backdoor triggers which a model is then queried with. The success rate of such triggers is then used as a statistical indicator for whether an LLM has been backdoor-attacked.\nThe paper experiments with refusal and jailbreaking backdoor attacks, and considers word-, phrase-, and sentence-level triggers. Models are backdoor-attacked via LoRA fine-tuning. The authors experiment with Llama-2-7B-Chat-HF, Qwen2.5-3B-Instruct, and Qwen2.5-1.5B-Instruct. The authors show that ICLScan is superior to a competitive baseline (CLIBE).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes a novel and simple yet elegant solution to detecting backdoor behaviour in LLMs. The observation of backdoor susceptibility amplification is certainly interesting and relevant to researchers working in the field.\n\nAs for weaknesses, the paper makes multiple strong assumptions that potentially work in favour of ICLScan’s success rate. First, the authors assume that the defender knows the target type of the backdoor attack a priori (jailbreaking or refusal). It would be very interesting to hear from the authors (and to see based on empirical evidence) how well this method would translate to scenarios where attackers are less informed. Second, the ICL prompts used for the experiments stem from the same distribution as the data used to insert the backdoors into the model (sampled from test set portions of Stanford Alpaca and AdvBench). What happens if ICL prompts are sampled from an OOD source? Will that affect the effectiveness?\n\nQUESTIONS:\nCan you elaborate on how the analysis of attention distributions actually worked? Which layers were considered / excluded from this analysis?\n\nLIMITATIONS:\nThe paper provides limitations of the proposed work in Appendix C. While this is appreciated, I’d encourage the authors to move the discussed limitations (along with suggestions for future work) into the main manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper focuses on detecting backdoor attacks on in-context learning (ICL) for LLMs, particularly with features (1) with black-box access only, (2) can be generalized to both classification and generation ICL tasks, and (3) only need a few queries on the target LLM API. The foundational design of ICLScan is based on a discovery that the backdoored ICLs are very easy to be further backdoored against the same target, even the newly injected trigger is different from the original one. Based on this discovery, ICLScan leverages a small set of targeted ICL prompts to detect the backdoors, and experiments demonstrated its effectiveness.

STRENGTHS AND WEAKNESSES:
# Strengths
1. This paper addresses a popular and critical AI security topic: backdoor attacks on ICL.
2. The proposed method is clearly presented and lightweight enough (e.g. with only black-box access and a few queries) to make itself a practical defense to be deployed. 
3. The experiments are comprehensive, covering various ablation studies and target LLMs.
4. Code is provided, guaranteeing reproducibility.

# Weakness
1. The method requires a pre-defined set of ICL targets. What will happen if the backdoored target really doesn't fall within the defense ICL target prompts?
2. What are the connections between the noisy label robustness of ICL [1,2] and the discovery in this paper?

[1] On the noise robustness of in-context learning for text generation. NeurIPS 2024
[2] Exploring the Robustness of In-Context Learning with Noisy Labels. ICASSP 2025

QUESTIONS:
Please kindly refer to W1 and W2 above.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02