Title: BAM-ICL: Causal Hijacking In-Context Learning with Budgeted Adversarial Manipulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

in-context learning

hijacking

budget profile
Abstract: Recent research shows that large language models (LLMs) are vulnerable to hijacking attacks under the scenario of in-context learning (ICL) where LLMs demonstrate impressive capabilities in performing tasks by conditioning on a sequence of in-context examples (ICEs) (i.e., prompts with task-specific input-output pairs). Adversaries can manipulate the provided ICEs to steer the model toward attacker-specified outputs, effectively ''hijacking'' the model's decision-making process. Unlike traditional adversarial attacks targeting single inputs, hijacking attacks in LLMs aim to subtly manipulate the initial few examples to influence the model's behavior across a range of subsequent inputs, which requires distributed and stealthy perturbations. However, existing approaches overlook how to effectively allocate the perturbation budget across ICEs. We argue that fixed budgets miss the potential of dynamic reallocation to improve attack success while maintaining high stealthiness and text quality. In this paper, we propose BAM-ICL, a novel **b**udgeted **a**dversarial **m**anipulation hijacking attack framework for in-context learning. We also consider a more practical yet stringent scenario where ICEs arrive sequentially and only the current ICE can be perturbed. BAM-ICL mainly consists of two stages: In the offline stage, where we assume the adversary has access to data drawn from the same distribution as the target task, we develop a global gradient-based attack to learn optimal budget allocations across ICEs. In the online stage, where ICEs arrive sequentially, perturbations are generated progressively according to the learned budget profile. We evaluate BAM-ICL on diverse LLMs and datasets. The experimental results demonstrate that it achieves superior attack success rates and stealthiness, and the adversarial ICEs are highly transferable to other models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BAM-ICL, the first framework for hijacking large language models under in-context learning (ICL) by dynamically allocating a limited adversarial budget across the demonstration examples (ICEs). The attack proceeds in two phases:\n\nOffline Stage: Given black-box access and held-out data, a global PGD-style optimizer jointly perturbs all ICE embeddings under an overall constraint to learn a budget profile.\n\nOnline Stage: When ICEs arrive sequentially, each incoming demo is perturbed in real time by a local PGD update constrained to the corresponding perturbation budget, then projected back to discrete tokens via a nearest-neighbor word-projection step\n\nExtensive experiments on SST-2, AG’s News, and OLID different models show that BAM-ICL achieves a higher ASR, preserves semantic fidelity (high cosine similarity and low perplexity).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Novel framing of ICL hijacking: This is the first work to cast hijacking as a budget-allocation problem, enabling per-example perturbation rather than fixed uniform budgets across ICEs.\n2. This paper proposes a novel two-stage optimization strategy (offline stage and online stage) that ensures effective use of the adversarial budget.  By first learning a global budget profile that identifies sensitive positions and then applying targeted local attacks at test time, the method achieves strong performance while remaining computationally manageable during inference.\n3. Perturbed ICEs maintain high cosine similarity (≥ 0.95) and low perplexity, indicating minimal semantic drift and strong stealthiness.\n4. Perturbation profiles trained on one model (e.g., OPT-13B) successfully transfer to other models (e.g., GPT-2-XL, LLaMA 2/3) with minimal ASR drop.\n\nWeaknesses:\n1. Perturbations are tightly linked to a specific ICE set. A new ICE set (or even a change in ordering or length) requires recomputing the budget profile from scratch, which limits robustness in real-world dynamic settings.\n2. The paper does not quantify the runtime of offline PGD (which operates jointly over n ICEs for T=80 steps) or the online phase (which involves T-step PGD and nearest-neighbor word projection). This hinders assessments of scalability.\n3. The method is only tested on text classification datasets. It remains unclear how well BAM-ICL performs on more complex or generative tasks such as jailbreaks or reasoning.\n4. No evaluation against stronger defenses. Only one defense method, which prepends clean ICEs is tested. There is no exploration of detection-based defenses or adversarial filtering.\n\nQUESTIONS:\n1. Could you provide a detailed computational-time analysis for both the offline and online stages?\n2. Have you considered or evaluated BAM-ICL on more complex or generative tasks, for example, jailbreak?\n3. How does BAM-ICL perform if the ICEs are shuffled or slightly altered in content (e.g., paraphrased)? Is the learned budget profile robust to such variations?\n4. What is the assumed adversarial capability in this threat model? If the adversary is the model publisher (rather than an external attacker), are perplexity and stealthiness still meaningful objectives?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

comments: SUMMARY:
This paper proposes an attack framework BAM-ICL that hijacks Large Language Models by manipulation of in-context learning. BAM-ICL learns optimal perturbation budget allocations among in-context examples and dynamically distribute perturbations to each in-context example arriving sequentially.

STRENGTHS AND WEAKNESSES:
Pros:
1. The paper is well written and easy to understand.
2. The design of sequential arrival of in-context examples matches the real-world scenarios.
3. The authors conduct comprehensive experimental evaluation with metrics including the generation performance, attack effectiveness, and stealthiness.

Cons:
1. The task distribution of and benign in-context examples may not be available.
2. It is advised to show the computational cost of offline budget profiling, especially concerning large-scale models or datasets.
3. The performance of the proposed framework against more advanced defense strategies.

QUESTIONS:
How would the proposed framework perform against defenses with in-context example filtering?

LIMITATIONS:
The computational cost of offline budget profiling could limit the scalability of the proposed framework to large models.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
While existing work has established in-context example-based hijacking attacks on LLMs, this paper focuses on the attacking budget for this kind of attack. To this end, this paper first identifies the necessity to allocate a proper budget for adversarial in-context examples as a trade-off between stealthiness and attack efficacy, and then proposes BAM-ICL, which is a budgeted attack on ICL. This attack includes two main stages, in which the former can get data from the same target task distribution and use a gradient to optimize the attack, and the latter assumes the examples are in an online streaming form, so the perturbation is generated progressively.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The budget limit problem is overlooked in previous ICL-attack research. This paper is the first to identify this critical problem.
2. The proposed method is presented in a clear and intuitive way. Researchers can easily reproduce the BAM-ICL following this paper.
3. The online and offline stages consider different attacking settings comprehensively, making it a practical attack paradigm.
4. Experiments demonstrate that BAM-ICL outperforms existing baselines when the attack budget is limited, showing its superiority. Evaluations also consider the stealthiness and transferability.


# Weakness
While no major concerns, this paper only considers the hijacking in ICL-attacks, and other forms of ICL-attacks like jailbreaking [1] or backdoor [2] can be further discussed.

[1] Jailbreak and guard aligned language models with only few in-context demonstrations. arxiv 2023
[2] Backdoor Attacks for In-context Learning. EMNLP 2024.

QUESTIONS:
How can the theory and algorithm in this paper extend to jailbreak or backdoor attacks with ICL?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BAM-ICL for hijacking LLMs through ICL. Unlike conventional adversarial attacks, BAM-ICL distributes a fixed perturbation budget across multiple in-context examples to subtly influence the model's output. It operates in two stages: an offline stage, which constructs an optimal perturbation budget profile using a global gradient-based method, and an online stage, which perturbs ICEs sequentially as they arrive. Experiments show that BAM-ICL outperforms existing baselines in attack success, stealthiness, and transferability, posing a practical and stealthy threat to ICL-based LLM usage.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The BAM-ICL method introduces a well-structured two-stage pipeline, with an offline global budget profiling and an online sequential attack mechanism. This mirrors practical settings where ICEs arrive incrementally, enhancing the realism and applicability of the attack.\n2. Unlike prior works that use fixed perturbation budgets across ICEs, BAM-ICL learns to allocate the perturbation budget optimally, improving both effectiveness and stealthiness.\n3. The method consistently achieves higher Attack Success Rates (ASR) across various models and datasets compared to strong baselines, showing its superior effectiveness.\n4. Adversarial ICEs and learned budget profiles transfer well across different LLM architectures, demonstrating strong generalization and practical impact.\n5. The paper is well-organised and easy to follow. The authors have provided detailed figures and algorithms to present the proposed approach. \n\nWeaknesses:\n\n1. While the method is tested across benchmark datasets and model families, the experiments are mostly constrained to synthetic or classification tasks. There is limited exploration of more realistic LLM applications, e.g., multi-turn dialogue, reasoning, or code generation,  where the ICL dynamics might differ.\n2. The offline stage assumes the attacker can access data drawn from the same distribution as the target task. This assumption may not always hold in real-world settings, especially in proprietary or closed-domain applications.\n3.  The evaluation focuses on one prior defense, e.g., prepending clean ICEs, but does not benchmark against more recent or certified defenses, such as detection-based or filtering-based methods.\n4. Although the paper uses cosine similarity and perplexity to measure text quality and stealthiness, human evaluation is not included. This may leave room for unnoticed semantic drift or grammatical degradation.\n5. The authors claim that GGI is easily detectable while BAM-ICL is more stealthy. However, beyond perplexity comparisons, they should include experimental results using detection-based or filtering-based defenses and provide examples of attacks that fail.\n6. Some tables and figures are unclear. For example, what are the different lines shown in Figure 3? It is better to show the legend. Table 1 is too busy, and it is hard to tell which one achieves the better performance. It is not easy to know that \"the loss increases more rapidly than with the attack with a flat budget\" as shown in Figure 6. \n7. Since GGI is the most related work, the authors should also have conducted comparisons on the cosine similarity distribution histograms of GGI in Figure 5 and the transferability of GGI in Table 2. \n8. It is better to show some adversarial examples in the main concept or the appendix to show the stealthiness of the proposed method.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-08"}