BAM-ICL: Causal Hijacking In-Context Learning with Budgeted Adversarial Manipulation
Authors: Rui Chu
Bingyin Zhao
Hanling Jiang
Shuchin Aeron
Yingjie Lao
Venue: NeurIPS 2025 poster ()
Keywords: large language models
in-context learning
hijacking
budget profile
Primary area: applications
TL;DR: We propose BAM-ICL, a novel budgeted adversarial manipulation hijacking attack framework for in-context learning.

Abstract: Recent research shows that large language models (LLMs) are vulnerable to hijacking attacks under the scenario of in-context learning (ICL) where LLMs demonstrate impressive capabilities in performing tasks by conditioning on a sequence of in-context examples (ICEs) (i.e., prompts with task-specific input-output pairs). Adversaries can manipulate the provided ICEs to steer the model toward attacker-specified outputs, effectively ''hijacking'' the model's decision-making process. Unlike traditional adversarial attacks targeting single inputs, hijacking attacks in LLMs aim to subtly manipulate the initial few examples to influence the model's behavior across a range of subsequent inputs, which requires distributed and stealthy perturbations. However, existing approaches overlook how to effectively allocate the perturbation budget across ICEs. We argue that fixed budgets miss the potential of dynamic reallocation to improve attack success while maintaining high stealthiness and text quality. In this paper, we propose BAM-ICL, a novel **b**udgeted **a**dversarial **m**anipulation hijacking attack framework for in-context learning. We also consider a more practical yet stringent scenario where ICEs arrive sequentially and only the current ICE can be perturbed. BAM-ICL mainly consists of two stages: In the offline stage, where we assume the adversary has access to data drawn from the same distribution as the target task, we develop a global gradient-based attack to learn optimal budget allocations across ICEs. In the online stage, where ICEs arrive sequentially, perturbations are generated progressively according to the learned budget profile. We evaluate BAM-ICL on diverse LLMs and datasets. The experimental results demonstrate that it achieves superior attack success rates and stealthiness, and the adversarial ICEs are highly transferable to other models.

BAM-ICL: Causal Hijacking In-Context Learning with Budgeted Adversarial Manipulation
Rui Chu
Bingyin Zhao²
Hanling Jiang'
Shuchin Aeron'
Yingjie Lao'
Department of Electrical and Computer Engineering, Tufts University
Meitu Inc
frui.chi chu, hanling jiang, shuchin. yingjie. lao))tufts..eee bingyin@meitu.com
Abstract
Recent research shows that large language models (LLMs) are vulnerable hijack- ing attacks under the scenario of in-context learning (ICL) where LLMs demon- strate impressive capabilities in performing tasks by conditioning on sequence of in-context examples (ICEs) (i.e., prompts with task-specific input-output pairs). Adversaries can manipulate the provided ICEs to steer the model toward attacker- specified outputs, effectively "hijacking" the model's decision-making process. Unlike traditional adversarial attacks targeting single inputs, hijacking attacks LLMs aim to subtly manipulate the initial few examples to influence the model's be- havior across range of subsequent inputs, which requires distributed and stealthy perturbations. However, existing approaches overlook how to effectively allocate the perturbation budget across ICEs. We argue that fixed budgets miss the po- tential of dynamic reallocation to improve attack success while maintaining high stealthiness and text quality. In this paper, we propose BAM-ICL, a novel budgeted adversarial manipulation hijacking attack framework for in-context learning. We also consider a more practical yet stringent scenario where ICEs arrive sequentially and only the current ICE can be perturbed. BAM-ICL mainly consists of stages: In the offlin stage, where we assume the adversary has access to data drawn from the same distribution as the target task, we develop global gradient- based attack to learn optimal budget allocations across ICEs. In the online stage, where ICEs arrive sequentially, perturbations are generated progressively according to the learned budget profile. We evaluate BAM-ICL on diverse LLMs and datasets, the experimental results demonstrate that it achieves superior attack success rates and stealthiness and the adversarial ICEs are highly transferable to other models. Code available at h/gpst////ttubb com CCcrC//AR------ ICL
1 Introduction
Recent development of large language models (LLMs) has revolutionized and empowered various fields, from reasoning Wei et al. 2022]. Cheng et al. [2024a] Zhang et al. [2024] to math proof Azer
bayev et al. [203]] Setlur et al. 2024]. ). Didolkar et al. 2024 to protein design Madani et al. 2023
2020]. [Cheng et al. 2024b)] Ferruz and Höcker 2022 Different from conventional models. LLMs
also demonstrate remarkable capabilities in handling wide range of problems and tasks through
in-context learning (ICL) (a.k.a inference-time few-shot learning [rrwnn et al. [2020] Garg al.
[0022) Xie et al. [2022] Min et al. 2002),, Wies et al. [2023) [Agarall et al. 20244). ICL is an
intrinsic capability of LLMs that allows them generate relevant responses to unseen input queries via "learning" from in-context examples (ICEs) (i.e., a sequence of prompts with task-specific input- output pairs), without updating model parameters. Although paving effective path undertake
variety of tasks by observing context examples, the potential risks and threats that the ICL ability may cause remain unclear and are worth exploring.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Benign ICEs
Adversarial ICEs
Offine Stage: Budget Profile Construction
Online Stage: Budgeted Hijacking
ICL vs. BAM-ICL
Normal User Queries
LLM
Results
Review: The food is good.
Positive
Sentiment: Positive
Review: love the restaurant...
Negative
Sentiment:
Negative
Review: The stake is overcooked...
Review:
The fish is awful...
Sentiment: Nogotive
Sentiment:
Positive
+
ICE,
Review: will not come again...
Review: food good Sentiment: Posttive
Nagative
Sentiment:
Negative
Review: The best lobster
Review: The boof is overcooked...
Sentiment:
Negative
Sentiment: Negative
Negative
Word Projection
Maximize attack effect using PGD
Tokenize and embed ICEs
X
Project perturbed ICEs from embedding space back word space
Crars
Optimal Budget Search Normalize perturbations w.n. embedding vectors
Embedding
Concstenate sampled ICEs and user query
IdversarialICEs Generation
Original ICEs arrive sequentially
Sample budget for each from budget profile based context length Allocatio
Budget Profile
Prompt Construction
Illustration of the frame- work design of BAM-ICL. BAM-ICL hijacks LLMs and yields unintended output via adversarial ICES (the block in red) while ICL with benign ICEs (the block in green) produces normal outputs. BAM-ICL is composed of two stages, where the offline stage (the block sky blue) we construct the budget profile to search for optimal budget distribution for each ICE and in the online stage (the block in light purple) sequentially perform budgeted attack to generate adversarial ICEs.
Figure I: Top: Illustration of ICL and BAM-ICL on LLMs. Bottom:
In particular, it has been shown that the ICL capability of LLMs opens the door model hijacking attacks Si et al. 2023), Qiang et al. 2023 et al. [2025] Salem et al.) [2022] Jeong [223) ). Kuo
al. [2025] where adversarial ICES used to steer vietim models into producing attacker-speciffed outputs, effectively "hijacking" the model's decision making process. A recent work Qiang et al.
2023 extends hijacking attack strategies from vision tasks language models by using adversarial perturbations to craft malicious in-context examples. These adversarial example-based approaches modify ipput prompts influence model behavior. In contrast to adversarial attacks Miyato al.)
[.017]. Wang et al. [2023a], Anwar et al. [2024), [L et al. [2200], Xhonneux et al. [2024], which have recently gained significant attention in the context of LLMs, hijacking attacks leverage the ICL mechanism to manipulate the model's behavior across a range of inputs, exhibiting distinct characteristics and challenges. Instead of targeting a single input for misclassification, a hijacking attack perturbs the first few examples rather than single example, with the objective of influencing the subsequent examples. To maintain stealthiness, the overall perturbation must be constrained. However, since the perturbation is distributed across multiple examples, the magnitude applied to each individual example can be reduced. 2) This also creates opportunities to dynamically allocate the perturbation budget among ICEs, which can further improve the attack performance (we show a fixed budget is suboptimal). 3) An adversarial attack is deemed successful when it identifies an adversarial example that misleads the model for specific input; in contrast, hijacking attack must maximize its influence on subsequent ICEs while operating under a predefined perturbation constraint. These observations motivate the central research question seek to address: How design effective and stealthy hijacking attack against LLMs, where subtle adversarial manipulations on ICEs gradually accumulate influence analogous to snowball effect, steer the model's 's behavior?
In this work, we propose BAM-ICL, a budgeted adversarial manipulation hijacking attack framework against LLMs through ICL. We consider a more practical yet stringent scenario where ICEs arrive sequentially and only the current ICE can be perturbed. As shown in Fig. D BAM-ICL mainly consists of two stages: In the offline stage, where we assume the adversary has access to data drawn

---

from the same distribution as the target task, we develop global gradient-basee budget profile construction algorithm to search for the optimal perturbation budget for each adversarial ICE given total perturbation budget. In the online stage, where ICEs arrive sequentian sequentially, we progressively generate the perturbation for each ICE according to the budget profile constructed the offline stage. To enhance the stealthiness and preserve the semantic meaning of ICEs, the perturbations are performed on the embedding space and then projected back to the word space. Our contributions are summarized as follows:
To the best of our knowledge, this is the first work that exploits ICL to hijack LLMs with budgeted adversarial manipulation.
We develop novel two-stage attack framework composed of global gradient- basee attack that systematically searches for the optimal dynamic budget allocation strategy for individual ICEs and refined causal attack that ensures the full utilization of the budget for each example.
Through extensive experimentation three benchmark datasets across various LLMs, we demonstrate that BAM-ICL achieves superior attack success rates and remarkable transferability while preserving the high text quality and stealthiness of adversarial ICEs.
2 Related Work
2.1 In-Context Learning
In-context learning (ICL) was first defined in Brown et al. (020), where it was observed that pre- trained LLMs can perform new tasks by conditioning on a few task-specific demonstrations, without any parameter updates. This capability, often referred to inference-tine few-shot learning, enables the model to learn from a small set of input-output examples (ICEs) provided at test time. For example, an LLM is expected predict the next token of (Prompt: Delicious fish! Output:_ given the following ICEs: (Prompt: I love this restaurant; Output: Positive. \ Prompt: The food is (n).
terrible; Output: Negative.
Early studies of ICL focused on the validation of hypotheses with synthetic experiments and provided insightful theoretical results Chan et al. 20022 Garg et al. [2022] Akyürek et al. [2023), Von Oswald et al. [2023]. |aahn and Goyal] 2023]. For instance, AAarwal al. [024] has demonstrated that many-sho in-oontxtt learning's s superior learning capabilities on multiple datasets and benchmarks.
Xie et al. [2022) proved that transformers and LSTMs are capable of inferring the hidden task-specific function in latent space from ICEs despite the mismatch between prompts and pretraining distributions using numerical synthetic data. Recent works have advanced progress and extended ICL to broader scopes Falck al. [2024), Wang et al. [023bb] Qiang al. (2023) attempted to find good ICEs and generalized ICL from simple scenarios (e.g., number demonstrations) to complex real-world scenarios (e.g., natural language) and |Agaraal al. (2024) demonstrated that scaling the number of context examples leads to substantial performance improvements across a wide range of tasks. In this work, our objective is to explore the risks and threats ICL may bring LLMs by such new "learning" paradigm.
2.2 Attacks against Language Models
Threats against language models can be dated to attacks such as adversarial training methods Miyato al. [017)] and HotFlip Ebrahimi al. 2018). There has been line of works that leveraged adversarial attack [Szegedy et al. 2014]. a notorious inference-time attack against deep neural networks, and crafted adversarial examples fool well-trained models by deliberately adding subtle adversarial perturbations on clean inputs using gradient- approaches such as FGSM (oodfellw al. [2013] and PGD Madry al. 2018). For example, FreeLB Zhu et al. (2020) perturbed the embedding layer the embedding layer of LLMs to manipulate the model output; TextFooler Jin et al. (2020) successfully deceived BERT Devlin et al. [2019] with adversarial examples (i.e., semantically similar alternatives that prioritize the most critical words to the model's prediction) and Ranjan jet al. fooled GPT models in text classification by prioritizing influential tokens. However, these attacks do not exploit the in-context learning ability of LLMss Recent security studies highlight that I.LMs can be compromised via prompt-trigger attacks, and also propose unified defense mechanisms spanning prompt injection, backdoor, and adversarial prompts Lin et al. 2025). Complementing
3

---

these prompt-level threats, their broader adversarial ML line covers imperceptible and arbitrary-target backdoors, ViT backdoor defenses, dlean-label availability and Mlass-oriented poisoning. and even
neural- networkk operation backdoors or hardware- Trojjn attacks Doan et al. [2022] 2023 Zhao and
Lao
2022a b
Clements and Lao 2019). Han et al. [025]] Hoang et al. [024].
On the other hand, recent research has found that LLMs are vulnerable to hijacking attacks through
in-context learning Ranjan et al. Kandpal et [2023] Zhao al. [2024] For instance. Li al.
[022 reveals the essential factors of ICL robustness (c.g., model depth and context length) through context hijacking label manipulation. Anwar et al. [2024] investigates adversarial robustness in ICL for regression tasks, showing that perturbing input features (x-attacks) or context labels (y-attacks) can significantly degrade a transformer's i ability approximate the underlying function. |Wang et al.)
(2023a] further shows that adversarial in-contex: examples transfer well across different models. recent work (GGI) Qiang et al. [2023] proposed devising adversarial ICEs and hijacking L.L.s by appending imperceptible malicious suffixes to in-context demonstrations using gradient-basel optimization, and Kuo al. 2025] explored the way to hijack against the safety reasoning mechanism. These works generally follow the design concept of conventional adversarial examples that impose the same perturbation on every single input. However, such a practice neglects the nature of ICL, in which models' predictions are steered by sequence of demonstrations rather than single example.
Method
3.1 Threat Model
The goal of hijacking attacks is to force a victim model M to generate manipulated output given new benign input queries via ICI. on a sequence of adversarial ICEs. The attacker has no access to the internal parameters or training data of the model (i.e., black-box access), but can modify the ICEs that precede the actual input query. We consider a setting where the attacker knows the task (e.g., sentiment classification [Socher et al. [2013). Zhang et al. [2015] Zampieri et al. [2199] Rosenthal et al. 2021] and crafts adversarial ICES [[ ( } with perturbed inputs but original labels to preserve stealth. The attack objective is thus to find perturbed input set X {ar 222 :::-..... 2,, We assume that adversaries know the distribution of benign in-context examples and can sample instances for prompt manipulation, which is reasonable assumption because normal context examples usually consist of simple input-output pairs that are relevant to the task and are consistent with prior works in related fields Zhao et al. [202T]. For example, in-context examples for the emotional classification task are often formulated as: [ar [Noun.] + [Linking verb] + [Adjective] (e.g.. blast/dirtyffattastic): y : [Pronoun] [Linking verb] + [Adjective] (e.g., wonderii/neegaive/gree))
3.2 BAM-ICL
Given the threat model described above, the adversary's objective is perturb sequence of benign ICES {x,, y:) (where y/ represents the ground-truth) to a set of adversarial examples {(24, 3/4 ] such that, when presented to a language model M along with a new input query Equery. the model generates manipulated output Squery. We denote the set of adversarially perturbed inputs as X' x/,,........ ................................ ",, Since the attack conducted in the embedding space, we also denote their corresponding perturbed embeddings as E' = fe', . e where each et is derived by adding a constrained perturbation to the original embedding e, of rz;:
e = E; + 6, S.l.
|| < E;
The model's prediction conditioned on the ICE and the query input is given by:
Qquary = M(2
[x',, ynl, x.query)
(2)
As aforementioned, LLMS exhibit the ability of "learning" to perform tasks simply by conditioning on sequence of ICEs, which offers the opportunity to hijack the model's ss decision-making process through a series of adversarial ICES rather than a single input. Therefore, we devise BAM-ICL based on this property of LLM. A general form of the BAM-ICL objective can be expressed as:
max L(M(X', :equer), Miueer) S.L. d(s/)22) S e, Vi E (I.....n) X
4

---

Algorithm Offline Phase: Budget Profile Construction
Require: Original ICE sequence X. step size a, number of steps T, total perturbation budget
I: P < Prompi_Construce Construct(())
2: E f- EmbedddnggP)
3: Initialize A (0) t- 0 E (din(E)
4: for = 0 to T do
5:
A(+1)) + - IlaSe
(0) + + VA
6: end for
7: r < Budget_Profile/ )
8: return
where d(-) is the distance metric that measures the embeddings discrepancy between the original and perturbed inputs (i.e.. ||e/ - eæ|l). To ensure stealthy yet effective attacks, we a total perturbation budget as a constraint for the ICEs:
Σ d( = 11e;- e:12 E
(4)
where is the overall perturbation budget. This constraint enables dynamic budget allocation across examples, which is the key design principle behind our BAM-ICL framework.
As shown in Fig. [ the proposed BAM-ICL consists of two stages. In the offline stage (Section 3.3 we assume the adversary has access to data drawn from the same distribution the target task, which allows simulating the ICL behavior of the target model. The goal of this stage is to construct an optimal perturbation budget profile for each adversarial ICE, under a total perturbation budget constraint. Specifically, we develop a global gradient- based optimization algorithm adapted from projected gradient descent (PGD) to determine the budget profile. We jointly optimize all perturbations under the shared constraint to calculate how the total budget e should be distributed across individual examples. This optimization yields dynamic budget allocation profile {<) which encodes the relative sensitivity of each ICF. We construct a budget distribution T based on {cs ) which is stored and later used in the online stage adaptively generate adversarial examples in real-time.
In the online stage (Section 3.4). the adversary no longer has access to future ICES or queries in advance and must perturb each incoming ICE sequentially as it arrives. Since the number of ICEs n can vary in practice, for each received input 2i, we retrieve the corresponding e; from the budget distribution I' based using a local PGD-based algorithm. The perturbed embedding eę is then projected back to the word space to form the adversarial ICE (z/, 3/s). This process continues until all n ICEs have been processed. Crucially, the online stage only relies on the access to the current ICE and the precomputed budget profile, making it suitable for realistic settings where ICEs are streamed or constructed on the fly. By coupling global sensitivity insights from the offline phase with localized perturbations in the online phase, framework ensures both efficiency and stealthiness in mounting hijacking attacks.
and perform constrained adversarial perturbation in the embedding space
3.3 Offline Stage: Budget Profile Construction
In this stage, we aim to construct the budget profile. As presented in Algorithm I we design PGDD-ased approach and run the algorithm offline where we assume the adversary has access to data drawn from the same distribution as the target task. Given the sampled ICE sequence, we first construct the prompt (Prompt_ (oonpttConstrutt by concatenating each ICE x, along with the corresponding ground-trute -ruth y: in order, followed by the user query Tquery:
P
[mn, ynl, ("qery)
(5)
The prompt is then tokenized and embedded using a pre-trained embedding function: Embedding(P)) Based on the dimension of the embedding E, we initialize perturbation vec- tor A. We then iteratively update the perturbation using gradient ascent on the loss function Lp = &Mo (A,)),)) (Qurr))))rr)) with respect to A, subject to an overall La constraint on the per- turbation norm. At each step I, the perturbation updated as expressed line of Algorithm [] where is the step size and Projia)]ase( (-) denotes the projection onto the Lg ball with radius to ensure that the total perturbation budget is not exceeded.
5

---

Algorithm 2 Online Phase: Budgeted Hijacking Attack
Require: original ICE sequence X { "2,,, step size o, number of steps T, budget profile r, total perturbation budget e, context length n
1: (n1: 12: Ti < Cale_ (aac__Bd
2: P [-[]
3: for i = 1 to n do
4: P - P + Prompt_ Promptt Construct(r:)
5: e; Embedding(xx)
6: Initialize 6.0) +- 0 E (ddm(e:)
7:
6, 6=
%/ -
8:
for t = 0 to do
5((+1)
Proji a (800) + Vo, 2p²)
9:
10: end for
11: z
Word Wordd Proj(e,, ,7,c)
12:
end for
13:
return X' {21 "2,
After T iterations, obtain the perturbed embedding:
E' = [[e1 61, Embedding((1)), .... Ien + 8n Embeddingg(w)) Equery)
(6)
where e; Embedding(z:;) and 8; is the embedding perturbations for each e,. Thus, have An = (o,,..., 6n). We then compute the relative perturbation magnitude allocated to each example by normalizing the L2 norm of each perturbation with respect to the corresponding embedding vector:
| | ==1 16,l1//1le,||
7/
The resulting set {m)}- forms the budget profile r. This dynamic allocation captures the relative sensitivity or influence each ICE on the model'ss behavior and is stored for use the online phase.
3.4 Online Stage: Budgeted Hijacking Attack
In the second stage of BAM-ICL, perform the budgeted hijacking attack online where ICEs arrive sequentially. The details of our method are presented in Algorithm [] We first sample the corresponding budget for each ICE according to the context length from the budget profile I' that is constructed in the offline stage as discussed above.
For each incoming ICE 2i, follow similar process in the offline stage to construct the prompt (Prompt_ Construct) and generate the embedding (Embedding) (i.c., lines 4 and Algorithm 2) Please note that, for ICE xj, unlike in the offline stage, we retain all previously perturbed inputs and hence embeddings.
Using the sampled budget %. we scale the perturbation budget for each ICE, setting -/ac,, where the total perturbation budget. We then employ a PGD-based optimization to progressively generate the perturbation by computing the gradient of the loss function Lp with respect to the perturbation ö; and updating the perturbation vector using the step size a (i.e., lines 6 to 10 in the algorithm). This iterative process continues for T steps, gradually adjusting the perturbation within the T.2 constraint. Finally, the perturbed ICE z is obtained by projecting the perturbed embedding e, back into the word space by using Algorithm 3 resulting in the hijacked ICE sequence X' 121,22/ z,
Algorithm designed to project perturbed embeddings back into the word space by finding the semantically closest words to the perturbed embedding e + 5, This ensures that the perturbation applied in a way that maintains semantic coherence while making difficult detect. The first step of the algorithm constructs candidate set C, which consists of words from the dictionary D whose embeddings are within the perturbation boundary. Specifically, the algorithm selects words w from the dictionary where the L2 distance between the embedding of w and the original embedding e, is within the perturbation boundary 6;. This ensures that only words with embeddings that are semantically close to e; considered as potential candidates for the projection. Next, from this

---

Algorithm 3 Wrod_Proj() Word Projection Back from Embedding Space
Require: embedded context e,. perturbation 8,, perturbation budget e,, dictionary D, top-k value &t 1: C + { w E D [|Embedddin((u w) - E 12= I c
2: IC f arg minwec (|Embedding(w) - (e: + 6,)112
3: of + arg maxwex (1Replace(2;,
4: return
candidate set C, the algorithm selects the top-k words that are closest to the perturbed embedding e + 6,. Then, we choose the word z from this set that maximizes the model's loss function. This step ensures that the selected word not only remains semantically similar to the original word but also optimizes the effectiveness of the attack by influencing the model in the intended direction. It is worth noting that the choice of the word projection function is flexible and can be substituted with other methods Guo et al. [2224] Gonen et al. 2023
Stolfo et al. [2025]
4 Experiments
4.1 Experimental Settings
Datasets and models. We follow the same practice existing attacks Jeong 2023 against LLMs and evaluate BAM-ICL on SSTch et al. 2013). AG's News Zhang et al. 2015 and OLID Rosenthal al. 202T]. These datasets are common lext-classification benchmarks that cover a wide range of tasks, including sentiment analysis, topic categorization, and offensive language detection. For victim models, we select various model families that span from IB to 30B, including GPT2-XL Radford [
al. [2019. LLaMA Touvron et al. [2023), OPT Zhang et al. [2022]. and Mistral [iiag et al. [2023] Concrete details on the models and datasets are summarized in the appendix. All experiments are performed NVIDIA L40S GPUs.
Attack configurations and baselines. We adopt the prompt construction and guiding sentence strategy from Qiang et al. 2023), combined with the sequential masking logic introduced by Garg jet al. (0022.. For each ICE, we allow up to three tokens to be modified. The context length n ranges from 2 to 12, consistent with prior works Qiang et al. [2023), |Kandpal et al. [2023] [ et al. 2024). We evaluate our method against two baselines: flat budget allocation strategy and the GGT attack Qiang et al. [2023), which performs global perturbation across all ICEs.
Metrics. We comprehensively evaluate the generation performance, attack effectiveness, ICES text quality, and stealthiness in our experiments. We report Clean Accuracy (CA) to show the normal generation performance of the language models. For the attack effect, we employ the standard Attack Success Rate (ASR) (i.e., the percentage of manipulated ICEs that successfully yield malicious behavior) as the criteria. A higher ASR indicates better attack performance. We adopt Perplexity Bahl et al. 1983 and Cosine Similarity to evaluate the stealthiness and text quality of adversarial ICEs, respectively. ICES with high stealthiness and text quality tend to have cosine similarity values close to (i.e., more similar to benign ICEs) and low perplexity values. We also compute the ASR drop against defenses as an orthogonal assessment for stealthiness, a stealthier attack can evade the defense and maintain the ASR (i.e., lower ASR drop).
4.2 Attack Effectiveness
We first demonstrate the attack effectiveness of BAM-ICL. We perform the hijacking attack with BAM-ICL and the bascline methods including the hijacking attack with flat budget (i.e., attack with equally allocated perturbation towards each ICE) and global hijacking attack (i.c., attacking all ICEs simultancously. the first stage of BAM-ICL). The results various OPT models are shown in Table [ can be seen that compared to the flat budget attack where each ICE receives an even perturbation budget, BAM-ICL achieves superior ASR on all datasets across various models with two different context lengths, indicating the effectiveness of the design with dynamic budget allocation. The results for other models reported in the appendix, which show similar trends.
Comparison to prior hijacking attack G et al. | shown Fig. 2 it important to note that GGI is also global attack that perturbs all the ICEs simultancously. Thus, it is understandable
7

---

that GGI and the baseline global attack achieve slightly better ASR. However, unlike GGI that adds an easy-to-detect suffix and compromises the semantic meaning of the sentence, BAM-ICL preserves the quality of the manipulated ICEs with the design of the word project function, which also improves the stealthiness of the attack. We measure the perplexity score on 100 randomly sampled perturbed ICES as shown in Fig. (b). It can be seen that the perplexity score of the manipulated ICEs under our BAM-ICL remains similar to the clean ICEs, ensuring low perceptibility.
Table I: Attack Sucess Rate (ASR) on different OPT models
Method
OPT-L3B
OPT-6.7B AGNews
OPT-13B
SST-2
AGNews
OLID
SST-2
OLID
SST-2
AGNews
OLID
SST-2
AGNews
OLID
CA
86.85
68 60
70.14
89.16
73.20
71.08
90.04
72.90
71
92.45
71.00
74.09
+Global
18 39.61
53.21
64.55
22.0
55. 25. as 60.79 60.79....... 25
22.1 26.33
57.46...s
32.88-1.76
44.02=544
-Flat
37.92
s7.39
26.54
as 15. 15.36=2.44 33.49
23.77=x
32.12
14,87
29.34
14.06=2.44
21.5
-AM-ICIL
ao. 24.77 ge.ss 25.66 41.99 Lo.s 22.51 2aa 32.49_aaa 1. ga.is 21.36ga.15
30.72gaop
36.85go.as
20.21
_aas 29
12
88.85
70.60
72,15
91.16
75.20
73.08
92.04
74:90
73.54
94.45
73.00
76.69
+Global
74, 66 43.25 57.41
ga 68, 68.55ason
40. 16..3....
52, 66
65, 17
38.84
50.20
62, 33.38
37:45
48. 14
-Fla
41 21.5 22.74 1.30 31,61
22.30
26.26
2..70 28.30 18.30=s.on os 34.14 3414422.2
19.62
26.79
31.89
18.90
25.40-2.7
-BAM-ICL 52.71 30.66
40,98
47.55 27.80 36.55 44.78
26.45
34.92aa.ar
42.11e
25.30-22.1
33.49
OdB Gaie BI IMNHOL
10,00
30
(a) Averaged ASR comparison AGNews with a context length of 4.
(b) Averaged perplexity score from 100 randomly sampled perturbed ICEs.
Figure 2: Comparison of ASR and text quality between BAM-ICLand baseline attacks.
We also visualize the budget profile under runs with different total budgets, i.e., c, as shown in Fig. which reveals the importance of the budget profile construction the offline stage.
GGI
Global
BAM-ICL
n06
Figure 3: Budget profile on SST-2.
Figure 4: ASR drop (AASR) against defense SST-2.
4.3 Performance against Defense
To better understand the stealthiness of BAM-ICL, we also evaluate the performance against the defense proposed in Qiang et al. [2023]. which prepends clean ICEs of the same context length to the manipulated ICE sequences. Fig. 4 shows the ASR change between before and after prepending clean ICES under the hijacking attack. BAM-ICL demonstrates superior performance in evading existing defense strategies, especially with increased context lengths, achieving substantially lower ASR drop against the defense compared to both global attack and the attack in Qiang et al.
2023). BAM-ICL demonstrates stronger resilience against filtering Jain al. [2023) and detection-
based Nguyen and Wong (2023) defenses than prior work Qiang et al. [2023] both within individual
ICEs (ASR drop by defense for 19.63 compared to prior work at 39.83) and across multiple ICES
(19.66 compared to 22.32). However, shuffling and reordering could impact the effectiveness of
BAM-ICI, compared to Flat attack, which aligns with our expectations, since BAM-ICI, relies on position-dependent perturbation budget allocation.
8

---

4.4 Stealthiness
Another notable advantage of BAM-ICL is the stealthiness. Ideally, stealthy enough attack can generate cosmetically and semantically similar adversarial ICEs to the original benign text. Fig. provides insights into the cosine similarity distributions, which measure semantic alignment between adversarial ICEs and the original text. BAM-ICI. consistently exhibits higher similarity (i.e., close to 1), indicating a better semantic preservation after attack compared to the flat budget counterpart.
Fiel
Flat
BAM-ICL
BAMICL
BAM-ICL
(a) n1
(b)
(c)
Figure 5: Cosine similarity distribution histograms of ICEs under various context lengths.
4.5 Transferability
We then demonstrate the transferability of adversarial ICEs generated by BAM-ICLby applying them LLMs other than the original victim model. Using adversarial ICEs crafted from the SST-2 task the OPT model, we query the DeepSeek-Chan API (based on the V3 model) with the same inputs, which shows similar ASR across models. More importantly, find that only does adversarial ICE exhibit high transferabilityy but also the learned budget profile generalizes well across different contexts and model configurations. We apply the budget profile learned from the SST-2 task on the OPT model with context length of 12 to GPT2-XL and report the results in Table
Table 2: ASR for budget profile transferability from SST-2
Modified Tokens
n
n
OPT
GPT-XL
OPT
GPT-XL
0.61
0.51
0.78
0.75
0.63
0.89
0.86
3
0.93
0.93
0.99
0.97
4.6 Time Complexity
For each run of the offline phase, we select input-output pairs equal in number the attack context length from the training set. The budget profile is averaged over multiple runs. During the online phase, the full test set used for evaluation. All results are normalized by the time required to compute perturbations per ICE using the Global attack. It can be seen that BAM-ICL offers much lower time complexity than the Global attack; even accounting for the additional cost of the offline phase, the overall runtime increase remains modest.
Table 3: Average runtime for time complexity comparison
Method
Offline Total
Offline /ICE
Online /ICE
Overall/ICE
Global
1.00
1.00
Flat BAM-ICI.
0.42
0.42
13.60
0.68
0.41
1.09
4.7 ICL on Linear Functions
Besides LLM evaluation benchmarks, we also evaluate performance on linear task, which is well-studied with theoretical foundations for TCL|Carg et al. [2022] Xie et al. [2022] Anwar et al. [2024), [L. al. 2203). Following Garg et [002]. we first trained a small transformer model on numerical linear functions, which has been shown be capable of performing specific learning

---

function entirely via inference from ICEs (more details are presented in the appendix). We then perform both attacks with flat budget and BAM-ICL. As shown in Fig. 6 the transformer trained in this numerical setting exhibits behavior similar to that of L.Mss. When applying learned budget profile (Fig- a))), the loss increases more rapidly than with the attack with flat budget (Fig. (()).. These results show that BAM-ICL is well generalizable.
0.09
BAM-ICL Budget Profile
RAMICL
Fiat
Burlet Profile
0000
0.014
0.12
12
Context Position
Contest Position i
(a) Budget profile.
(b) Attack performance.
Figure 6: Results on linear functions.
4.8
Generalizability to Other Tasks
The hijacking attack manipulates an output of LLM through ICL deliberately steer its intended behavior. In contrast, jailbreaking WWii et al. [2023] focuses on bypassing a model's 's safety guardrails (e.g.. human alignment) override ethical safety constraints, while the backdoor attack Kandpal implants malicious behaviors via crafted demonstrations and triggers them using prompts
et al.
2023
containing predefined triggers.
BAM-ICL can also generalize to other attack scenarios, such as jailbreaking tasks, demonstrating its broader applicability. We also include Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. As shown in Table the overall performance all methods degrades under this scenario. To improve attack effectiveness, the context length of n. Consistent with our earlier findings, our method continues to benefit from larger and consistently outperforms prior work, further validating the effectiveness of the proposed budgeted strategy.
viable approach is increase
Table 4: Jailbreaking Results LLaMA-3.1
Method
2
4
12
Zero-Shot
2.2
2.2
2.2
GGI
10.6
24.4
31.7
BAM-ICL
7.9
17.4
43.6
Conclusion
In this work, propose BAM-ICL, novel budgeted hijacking attack framework against LLMs under ICI., Unlike conventional adversarial methods, BAM-ICI strategically allocates perturbation budgets across ICEs to maximize influence while maintaining stealthiness. Our two-stage design first learns global budget profile offline using gradient-based optimization method, then applies online perturbations to ICES as they arrive sequentially. Experimental results across diverse LLMs and tasks confirm the effectiveness, transfcrability, and stealthiness of our approach. Overall, BAM-ICL demonstrates that dynamic, budget-aware adversarial manipulation poses serious and practical threat to LLMs operating under ICL.
Acknowledgment
Rui Chu and Yingjie Lao are supported in part by the National Science Foundation (NSF) SaTC- 2426299 and SaTC-2413046. Shuchin Aeron and Hanling Jiang would like to acknowledge support by NSF GCR: 2428640.
10

---

References
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang. Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 77:76930-76966, 2024.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
OpenReview.net, 2023.
Usman Anwar, Johannes Von Oswald, Louis Kirsch, David Krueger, and Spencer Frei. Adver- sarial robustness of in-context learning in transformers for linear regression. arXiv preprint arXiv:2411.05188, 2024.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. arXiv preprint 230.23332..2433, 2023.
Lalit R Bahl, Frederick Jelinek, and Robert I. Mercer. A maximum likelihood approach to continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, (2):179-190, 1983.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances neural information processing systems, 35:1 111111888811111 2022.
Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances IIm reasoning. Advances in Neural Information Processing Systems, 37: 1771121155116444
2024a.
Xingyi Cheng, Bo Chen, Pan Li, Jing Gong, Jie Tang, and Le Song. Training compute-optimal protein language models. Advances Neural Information Processing Systems, $7:69386-694118 2024b.
Joseph Clements and Yingjie Lao. Hardware trojan design neural networks. In JEEE International Symposium on Circuits and Systems, ISCAS 2019, Sapporo, Japan, May 26-29, 2019, pages 1 1-5. IEEE, 2019. doi: 10.1 (09/SCCS.20199887.22
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap. Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. Metacognitive capabilities of 1mss: An exploration in mathematical problem solving. Advances in Neural Information Processing Systems, 37: 17:19833-19812. 2024.
Khoa D. Doan, Yingjie Lao, and Ping Li. Marksman backdoor: Backdoor attacks with arbitrary target class. Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA USA, November 28 December 9, 2022, 2022.

---

Khoa D. Doan, Yingjie Lao, Peng Yang, and Ping Li. Defending backdoor attacks on vision transformer via patch processing. In Brian Williams, Yiling Chen, and Jennifer Neville, editors. ThirtyySeventl AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifih Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 506-515. AAAI Press, 2023, doi: 10.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. In Iryna Gurevych and Yusuke Miyao, editors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 1 15-20, 2018, Volume 2: Short Pupers, pages 31-36. Association for Computational Linguistics, 2018. doi: 10. 00.18653/V1/P18--
Fabian Falck, Ziyu Wang, and Christopher C. Holmes. Is in-context learning in large language models bayesian? A martingale perspective. In Forty-first International Conference on Machine Learning, ICML, 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 4(6):521-534 2022.
Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Hila Gonen, Srini lyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 10136-10148. Association for Computational Linguistics, 2023. doi:
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu, Cold-attack: Jailbreaking llms with stealthiness and controllability. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Michael Hahn and Navin Goyal. A theory of emergent in-context learning implicit structure induction. arXiv preprint 2xXi::2303.09711,2000
Yuning Han, Bingyin Zhao, Rui Chu, Feng Luo, Biplab Sikdar, and Yingjie Lao. Uibdiffusion: Universal imperceptible backdoor attack for diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2025. Nashville, TN. USA, June 11-15, 2025, pages 19186199196. Computer Vision Foundation / IEEE, 2025. doi: 10. N0.11999VVPS5234.20055 01787.
Duy C Hoang, Hung TQ Le, Rui Chu, Ping Li, Weijie Zhao, Yingjie Lao, and Khoa D Doan. Less is more: Sparse watermarking in Ilms with enhanced text quality, arXiv preprint arXiv:2407.1333 2024.
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint wxXiv:239.0061/1 2023.
Joonhyun Jeong. Hijacking context in large multi-modal models. arXiv preprint uxXiv:2312.0755 2023.
12

---

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford. Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The -eeooo Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages AAAI Press, 2020. doi: 0011699/AAALV340066661
Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor attacks for in-context learning with language models. arXiv preprint arXiv:2307. 14692, 2023.
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, pages 66-71. Association for Computational Linguistics, 2018. doi: 10.18653/11/D18-2 2012.
Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis Valentnn,, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai ol/o3, deepseek-rl, and gemini 2.0 flash thinking. arXiv preprint arXiv::502..0 120,119332.,,,93,, 2025.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACE adversarial attack against BERT using BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6193-6202, Association for Computational 1ttp:////2222...MPP...
Linguistics, 2020. doi:
Tianle Li, Chenyang Zhang. Xingwu Chen. Yuan Cao, and Difan Zou. On the robustness of transformers against context hijacking for linear classification. arXiv preprint arXiv:2502.15609, 2025.
Xingxuan Li, Xuan-Phi Nguyen, Shafiq Joty, and Lidong Bing. Paraicl: Towards robust parallel in-context learning. arXiv preprint arXiv:2404.0 00570, 2024.
Yingcong Li, Muhammed Emrullah Iddiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International conference on machine learning, pages 19565-19594. PMLR, 2023.
Huawei Lin, Yingjie Lao, Tong Geng, Tan Yu, and Weijie Zhao. Uniguardian: A unified defense for detecting prompt injection, backdoor attacks and adversarial attacks in large language models. CORR, abs/2502. 13141, 2025. doi: 10.48550/ARXXIV2500.2
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint orXiv:2412.19437 2024.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.034 03497, 2020.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong. Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature biotechnology, 41(8): 1099-1106, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
13

---

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 1, 2022, pages 1 10448-11064, Association for Computational Linguistics, 2022. doi: 10.18653/V1/20000 WEN.22ENNL--MAINTTT
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial training methods for semi- supervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Tai Nguyen and Eric Wong. 1044 2023.
In-context example selection with influences. arXiv preprint
Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking large language models via adversarial in-context learning. CoRR, 11.3311.0000 abs/2311.09948,, 2023. doi: 10.48500ARXXV2231..9999 1.09948.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Sudhanshu Ranjan, Chung-Ein En Sun, Linbo Liu, and Tsui- Wei Weng. Fooling gpt with adversarial in-context examples for text classification. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models.
Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. SOLID: A large-scale supissupee dataset for offensive language identification. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguistics: ACL/IJCNPP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 915-999 928. Association for Computational Linguistics, 2021. doi:
1.11865//V120211 102.S////2...IIIIIGGGGG FINDINGS-ACLL.
Ahmed Salem, Michael Backes, and Yang Zhang. Get a model! model hijacking attack against machine learning models. In 29th Annual Network and Distributed System Security Symposium, NDSS 2022, San Diego, California, USA, April 24-28, 2022. The Internet Society, 2022.
Amrith Setlur, Saurabh Garg, Xinyang Geng. Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. Advances in Neural Information Processing Systems, 37:43000 2024.
Wai Man Si, Michael Backes, Yang Zhang, and Ahmed Salem. [Two-in-One): A model hijacking attack against text generation models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 2223-2240, 2023.
Richard Socher. Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, Special Interest Group of the ACL, pages 1631-1642. ACL, 2013. doi: 1/D65///1/D13111111
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving instruction-followwi in language models through activation steering. In The Thirteenth International Conference on Learning Representatinss ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1397, 2023.
14

---

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pages 35151-35174, PMLR, 2023.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel R. Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages
2019.
Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. Adversarial demonstration attacks on large language models. arXiv preprint arXiv:2305. 14950, 2023a.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36:
2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnaillity of in-context learning. Advances in Neural Information Processing Systems, 36:36637-36631 2023.
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in Iims with continuous attacks. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang. editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 29- 2022. OpenReview.net, 2022.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Pupers), pages 1415-1420. Association for Computational Linguistics, 2019. doi: 10.18653/V1NN 1TT/////11111111 1144.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0/00
2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thought reasoning in llms. Advances in Neural Information Processing Systems, 37:333-355 356, 2024.
Bingyin Zhao and Yingjie Lao. CLPA: clean-label poisoning availability attacks using generative adversarial nets. Thirty- Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applicutions of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, FAAI 2022 Virtual Event, February 22 March 1, 2022, pages 9162-9170. AAAI Press, 2022a. doi: 10.1609/AAAA A3..AAAAI.......AA.. "3618.20902
15

---

Bingyin Zhao and Yingje Lao. Towards oriented poisoning attacks against neural networks. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022. Waikoloa, HI. USA, January 3-8, 2022, pages 2244-253. IEER, 2022b. doi: 10. 10.1109/WACV511558.000 20....00000.00
Shuai Zhao, Meihuizi Jia, Anh Tuan Luu, Fengjun Pan, and Jinming Wen. Universal vulner- abilities in large language models: Backdoor attacks for in-context learning. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing. EMNLP 2024, Miami, FL, USA, Novem- ber 12-16, 2024, pages 1 11507-15522. Association for Computational Linguistics, 2024. doi: .642.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 2021.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim our contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification: Theoretical analysis is not the primary focus of this work, which centers on the empirical design and evaluation of a novel hijacking attack framework. As such, the paper does not present formal theorems or proofs but instead supports its methodology through extensive experiments and ablation studies.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the source code and all necessary settings. The datasets in our experiments are open-source. We also present the details of the experimental settings in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the source code and all necessary settings with sufficient instruc- tions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Codeeuomi ssiniioiicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We report our experiment details in Section 4 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the error bars in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: Answer: [Yes]
Justification: We report our CPU, GPU and memory settings in Section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not contain any content that could be misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the assets used in the research.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the paper introduces new assets that are well documented, and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwpariipp..ccoooeeeeeeeeeea /t/CCooffeeence///2222 for what should or should be described.
23

---

A Summary of Appendix
We include the following supplementary materials that expand methods, experimental setups, and evaluations.
Additional Experimental Settings -
We provide detailed settings for our work, including the datasets and LLMs we running on, our evaluation metrics, and more details on the strategy for sensitive tokens.
Additional Experiments We provide detailed comparison of different models (OPT family and LLaMA family, as well as Mistral) with different datasets and different context lengths, to show the effectiveness of our methods under different and different Modified Token amounts. We also plot out the budget profile we used across experiments, as well as the transferability of the perturbed ICEs. Results of generalized tasks are also provided.
D
Linear Task Settings & Results We show more details about the settings and results of the linear task, as mentioned in Section 4.7 of the main paper.
Additional Visualizations - We provide the visualization results to better show our text quality and general performance compared different methods.
Limitations We discussed the limitations of our works.
Societal Impact- We discuss the potential societal impacts of our work.
Prompt Examples - We show clean and perturbed examples.
B Additional Experimental Details
B.1
Datasets, LLMs, and Metrics
B.1.1
Datasets
SST-2 (Stanford Sentiment Treebank v2): A dataset for sentiment analysis, containing 11,855 movie reviews with binary sentiment labels (positive negative) (Soeee)Soehrrrr et al. (2013).
OLID (Offensive Language Identification Dataset): Designed for identifying offensive language in social media, particularly X. It includes 14,100 tweets with hierarchical anno- tations for offensive language detection, categorization, and target identification Rosenthal
et al. 1021].
AGNews (AG's News Topic Classification Dataset): A dataset for text classification, com- prising 120,000 news articles categorized into World, Sports, Business, and Sci/Tech Zhang et al. 2015
B.1.2
LLMs
OPT (Open Pretrained Transformer): The largest variant, OPT-175B, matches GPT-3 in performance. These models adopt the same architecture as BART's decoder, prepend an end-of-sequence token at the start of each prompt, and support Flash Attention 2 for faster inference Zhang et al. [2022). In our experiments, we experimented on OPT family from 1.3 B to 30B.
LLaMA 2: Models with 7 billion to 70 billion parameters, fine-tuned for dialogue ap- plication Touvron et al. 2023). Trained on 2 trillion tokens with a 4096-token context window.
LLaMA 3 series: A specialized branch of the LLaMA family, LLaMA 3.2 comprises billion and billion parameter models optimized for multilingual dialogue tasks (compared I.aMA 188--Instruct). Trained on up trillion tokens, these variants handle diverse languages efficiently and feature a standard context window of 128k tokens for ultra-long input handling Touvron et al. (2023).
Mistral: Created by Mistral AI, proposed efficient variants like Mistral Medium and the 3 billion- and billion-parasn pallimer models Jiang et al.) [2023).
DeepSeek-VV V3: From DeepSeek AI, DeepSeek- V3 is state-of-the-art large language model featuring a mixture-of-experts (MoE) architecture with 671 billion total parameters and 37
billion active parameters per token Liu et al. (204 It is sounesdurced for researchers.
24

---

g to B.1.3 Metrics
Perplexity Score is used to evaluate the performance of the perturbed ICEs, which can be expressed
as
A
PPL = exp
Σ g=1
log pl Wig wcoo)
(8)
A
where A is the total number of tokens the sequence, is the index of the g-th token, ranging from 1 to A, W<g {wi, W2...... w J is the preceding context of length 9 1, and p(w, wekg) is the conditional probability assigned by the language model to token " given its prior context.
Cosine Similarity is used to quantify the semantic proximity between the original and its perturbed xz:
x
cosine cosine_similariee Similarityzz'.2 ||00 |0112
(9)
where
||0|| = VxTx..
which the la norm of Cosine similarity ranges (-1, 1); values closer to 1 denote stronger directional alignment, and show better similarity in sentiment meanings.
Loss in our implementation can be given by
hg = (TTanstonnee(Ennddddiiggg))))))
(10)
Pr(ye+1 (1gg (oflmax(22gt) yat1"
(11)
9g+ log Pr(yg+1 hg)
(12)
where g {.,
A} is the index position of the current input token within sequence of length A. h, e Rd is the hidden state at g. 2g+1 is the pre-softmax logit assigned to candidate token when predicting position g+1.
B.2 Hyperparameter Selections
To automate hyperparameter selection for the perturbation generation, we treat both the step size a and times t as variables in optimization problem. Optuna's Tree-structured Parzen Estimator (TPE) Akiba et al. [2019] sampler iteratively proposes candidate pairs and receives feedback via an objective that reflects adversarial strength.
B.3 Sensitive Token Selection
Tokenization Assume the selected ICE x; contains A tokens. We compose the sub-word tokenizer with the embedding matrix to map directly into sequence, as line in Alg. 4. where the tokenizer (e.g., BPE Gage [1944] or SentencePiece Kudo and Richardson (018) converts the string into a list of vocabulary indices w.
Input vector construction. In each ICE, the model input is the element-wise sum of lexical and positional components:
(13)
The resulting sequence feeds a stack of masked self-attention layers, ensuring each token attends only to its predecessors. g is the positional encoding for the token In our experiment, locate the selected tokens for perturbation.
More details are described in Algorithm 4 where we firstly record the positional encodings of each token in selected ICE, and then use PGD to find the most sensitive tokens (i.e., lines 3 to 11 in Algorithm 4). We record all the sensitive positions apply perturbation in the following process.
25

---

Algorithm 4 PGD-Based Sensitive Position Encoding Selection
Require: selected ICE Z4, label y,, step size a, steps T, budget e, top-m selected tokens m. total
token amount in this ICE A
1: (wi...., ) < Tokenizer(r;))
2: g f PositionalEneooiigg/ w)
3: for
to . do
4: 5(0) - 0
5: for = 0 to
6: ((++1) + Projiell2se (0() + (e)(((Emmeddnn((() +0(0) v))
7: end for
8: sensitive score 8g - 1- ll2
9: end for
10: Sensitive position list G t- { (8p-
11: return
Ilama3-8b
Ilama2-7b
0.16
Ilama2-13b
llama3.2-1b
Ilamm3.2-3b
0.12
0.08
Figure C.1: Budget profiles across different LLMs.
Additional Results
C.1 Budget Profiles
We begin by examining the budget profiles across different models. As shown in Fig. C.1 each model exhibits a distinct profile even when performing the same task, which justifies the need for the offline stage to learn model-specinf allocations.
Table C.1: ASR when is high, modified tokens
Method
LLaMA2-7B
LLaMA3,2-1B
Mistral-7B
SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID
CA
87.58
70.14
71.39
88.27
71.63
72.42
87.97
70.99
72.09
+Global
61.27
35.15
47.66
64.11
37.02
49.13
62.41
36.21
48.67
Flat
33.82
15.45
23.98
35.14
16.01
24.63
34.37
15.76
24.33
+BAM-ICL
43.26
23.24
33.19
45.33
24.11
34.08
44.15
23.77
33.67
71
CA
88.12
71.05
72.36
89.11
72.24
73.21
88.61
71.63
72.84
+Global
63.42
36.11
48.09
66.85
38.83
51.52
65.14
37.34
49.75
+Flat
35.12
26.03
37.09
17.53
26.85
36.23
17.07
26.37
+BAM-ICL 46.01
25.86
35.41
48.27
26.92
36.55
47.11
26.34
35.98
C.2 Results on More LLMs
We present results on more LLMs, including the LLaMA family, Mistral, and OPT models.
From Table C. we observe that the performance across different models is comparable. This result expected, as Mistral model has been shown to perform similarly to LLaMMA models on standard benchmarks Touvron et al. 2023 As shown in Table C.2
even under low perturbation budget,
26

---

Table C.2: ASR when low, modified tokens
Method
OPT-133
OPT-13B
L.L.MA32--18
L.LMMA-7B
SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID
Ti
CA
87.53
69.27
70.36
90.29
73.58
72.09
88.27
71.63
72.42
87.58
70.14
71.39
+Global
41.53
24.12
33.16
37.28
21.59
28.21
36.79
21.36
27.92
33.41
21.98
25.06
+Flat
21.44
10.33
15.42
17.64
8.35
14.52
21.06
9.61
13.24
19.41
9.87
13.77
+BAM-ICL.
27.96
15.62
21.37
25.77
13.48
19.72
27.54
13.94
20.91
23.12
14.37
19.46
8
CA
88.12
70.01
71.33
90.96
74.19
73.01
89.10
72.24
73.21
88. 12
71.05
72.36
+Global
44.51
27.16
32.47
36.31
22.39
27.82
40.52
24.72
33.19
37.44
22.11
30.58
+Flat
24.61
11.81
15.62
19.38
10.27
15.98
21.77
10.44
15.76
21.03
9.99
15.38
+BAM-ICL
30.52
16.54
24.81
27.44
15.03
20.64
29.71
15.84
21.72
26.58
14.99
22.37
Table C.3: ASR when is high, modified tokens min
Method
OPT-133
OPT-13B
LLaMA3.2-11
LLaMA2-7B
SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID
Ti
4
CA
87.53
69.27
70.36
90.29
73.58
72.09
88.27
71.63
72.42
87.58
70.14
71.39
+Glohal
17.85
10.62
11.45
14.64
8.96
11.47
17.34
10.29
10.93
15.32
10.05
11.09
+Flat
7.98
4.86
6.44
8.03
4.23
5.24
7.18
4.58
6.33
8.06
4.18
5.22
+BAM-ICL
10.03
6.63
8.01
10.47
5.97
8.69
9.25
6.71
7.26
10.07
8
CA
88.12
70.01
71.33
90.96
74.19
73.01
89.10
72.24
73.21
88.12
12
71.05
72.36
GGoobal
15.44
11.23
14.02
14.89
8.97
12.75
14.86
7.94
12.27
17.15
9.04
11.46
+Flat 10.67
4.14
6.40
8.53
4.38
5.85
10.31
3.66
5.98
8.37
4.46
6.51
+BAM-ICL
10.31
7.74
8.72
11.91
6.01
7.44
10.98
6.93
8.71
9.59
7.41
8.64
BAM-ICL maintains a reasonably strong performance compared to Table C33 This demonstrates that
attackers greatly reduce the perturbation magnitude runtime while still achieving successful hijacking attack. With a high perturbation budget and a large number flipped tokens, the attack achieves strong performance across all models. However, shown in Table C.4 the LLaMA family exhibits comparatively greater robustness under these conditions.
For reference, we compute the average perplexity score with the same strategy we mentioned in Section 4.5 of the main paper. As shown in Table when the number of flipping tokens remains the same, perplexity values exhibit only slight differences under different values. More importantly, even with the largest value and the largest modified tokens used in our experiments, the perplexity score is still better than that of prior work Qiang et al. 2023) as shown in Fig. 2(b) of the main paper.
C.3 Results on Reasoning Tasks
We follow the settings of Nguyen and Wong 2023 to use SuperGLUE Wang et al. (009 to benchmark the reasoning performance of ICL OPT model with context-length n. The accuracies for each category are shown in Table It can be seen that BAM-ICL outperforms the Flat attack and is close the Global attack, exhibiting trend similar to the classification tasks presented in the paper.
C.4
Sensitivity of Hyper Parameters
Table C.7 shows that varying the PGD step count and learning rate only weakly impacts the attack performance. This implies that the perturbation space within the e-ball is already sufficiently explored using coarse settings, and further tuning of or yields limited practical benefit for enhancing cross-model transferability.
27

---

Table C.4: ASR when is high, modified tokens
Method
OPT-1.3
OPT-13B
L.L.MA32--18
L.aMA2-7B
SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID SST-2 AGNews OLID
CA
87.53
69.27
70.36
90.29
73.58
72.09
88.27
71.63
72.42
87.58
70.14
71.39
+Global
71.37
40.32
53.72
61.46
34.89
46.98
64.11
37.02
49.13
61.27
35.15
47.66
+Flat
38.26
17.94
27.01
32.07
15.62
23.55
35.14
16.01
24.63
33.82
15.45
23.98
+BAM-ICI.
48.12
25.79
36.74
42.87
22.62
32.15
45.33
24.10
34.08
43.26
23.24
33.19
8
CA
88.12
70.01
71.33
90.96
74.19
73.01
89.10
72.24
73.21
88. 12
71.05
72.36
+Global
74.09
42.66
56.04
63.98
36.75
49.71
66.85
38.83
51.52
63.42
36.10
48.09
+Flat
40.53
19.42
28.76
34.10
17.05
25.69
37.09
17.53
26.85
35.12
16.48
26.03
+BAM-ICL.
51.47
28.60
39.18
45.24
25.15
34.92
48.27
26.92
36.55
46.01
25.86
35.41
Table C.5: Perplexity (PPL) Scores. (A lower score is better)
Modified Tokens
high
low
1
13.8
13.8
3
16.3
16. 1
Table C.6: Accuracy (%) on SuperGLUE with OPT model.
Method
BoolQ
RTE
WIC
WSC
CA
76.5
52.4
51.1
61.6
+Global
37.4
30.1
29.3
35.3
+Flat
54.6
40.4
40.2
43.1
+BAM-ICL
39.6
33.4
40.1
37.9
Table C.7: ASR drop under different parameters (- indicates the highest ASR as the baseline)
SST2 on LLaMA2-7h
OLID on OPT1.3b
Alpha
T=30
T=80
T=30
T=80
0.7
1.4
1.1
3
1.4
0.6
1.0
a
5
0.8
0.3
1.5
1.2
C.5 Transferability of Adversarial ICEs to Other LLMS
As shown in Table C.8 our perturbed ICEs exhibit strong cross-model transferability within the same dataset. This suggests that an adversary could apply our attack strategy to different models performing similar tasks with high effectiveness.
Table C.8: ASR drop while transferring selected ICES
ICE on dataset
n=4
n n=8
OPT 1.3b - LLaMA2
OPT 1.3b - OPT13b
OPT1.3b - LLaMA2
OPT1.3b OPT13b
SST2
6.3
1.2+0.
8.8406
2.0
AGNews
10.4+0.7
8.3404
12.7+0.8
11.240.7
OLID
6.6
3.4
5.7
3.740.4
28

---

Algorithm 5 Offline Phase: Budget Profile Construction for Numerical Settings
Require: Original sequence X, step size a, number of steps T, total perturbation budget
1: P x-X X
2: Initialize (0) t-
3: for = 0 to 1 do
4: (r++) + 4 A 10) + VA, C
5: end for
6: I' + Budget_Prrofllf
7: return
D Details for Linear Tasks
In the main paper, we have shown the general performance in numerical scenarios, and here present more detailed settings and methods as well as additional results.
D.I Problem Formulation
Training CCL-Transformer on Numeral Settings
We firstly trained a transformer for linear functions Garg et al. [202 with sampled distri- bution among: FF { f(z) x, R=). Then we have training progress p
((1), (x2)) ... Fn (Ii), x441) for minimizing the Mean Squared Error:
n Σ
min Ep
(MM((P) ((x+1))
(14)
n
We set n=19 in our experiment following Garg al. [202] where æ; has 20 dimensions. 0 the parameter simulating the input-output pair from the similar latent concept.
Attacking Pre-Trained ICI-Transformer on Numerical Settings Then, during the inference stage on the pre-trained transformer, have prompt P from f(x) w WiCL (WICL different from the functions we used during training F). The goal is that ICL progress makes fw..... (Fquery) ) approximate w Tquery, maximizing the loss. We repeat the process times and report the average performance.
D.2 Methods
During the offline stage (Algorithm 5). we perform global attack by simultancously perturbing all 19 inputs obtain the budget profile. The online stage (Algorithm perturbs each æ sequentially. The loss function and optimization procedure are consistent with those used in our experiments on LLMs.
D.3
Experimental Results
D.3.1 Experimental Settings
Our goal of the attacking progress is maximize the loss of the query positions. We all the contexts where greater then 20 as our query position so that to maximizing the the loss of Tquery includes (I21 J,), where 40.
We have tested the performance of ICL on the collected input-output pairs from both linear-atasee and non-linear dataset (for example, using Relu to generate the output label y). We sample all from a Gaussian Distribution.
In our experiment, we adopted the flat-attack method from Garg et al. 2022 which employs doubled-input perturbation to evaluate the robustness of pre-trained transtormers for ICI. Accordingly, set the total budget to match that used in the Doubled Input Perturbation baseline.
29

---

Algorithm 6 Online Phase: Budgeted Hijacking Attack for Numerical Settings
Require: original sequence X = (21, Fn), step size number of steps T, budget profile r, total perturbation budget e, context length
1: (n1: 72: 12: Tnn < Calc_ Sale_Bud
2: P [-[]
3: for 1 to do
4: P+P-
5: Initialize 0/0) 0
6: 4=%
7:
fort= 0 to T 1 do
8((+1)
Proji6llase, ( + V., 4.(
8:
9: end for
10:
end for
11:
return X' f2/ 22 m
D.3.2 Attack Performance
We observe the following trends from the loss curves in Fig. D.2 In the region of primary interest (19 < i v 40), the budgeted attack makes substantially higher loss than both the clean and flat- attack baselines. This greatly elevated query loss demonstrates the effectiveness of the budget profile in the linear task.
D.3.3 Budget Profile
Aworaged adye
70
Excatac (oubted Inpe0
ron-aaais CL Thansbrre
Budget Profile
10
15
30
36
Context Position
(context Index Input-output pairs)
Figure D.1:
Budget profile.
Figure D.2: Loss curve.
We also plotted the normalized budget profiles across different within the same dataset. As shown Fig. D.T. for given latent concept 0 the profiles exhibit similar patterns. can be observed that the budget profile significantly influences the loss at the query position compared to flat attacks.
E
Additional Visualization of Text Quality
We visualize the perplexity score of our outputs shown in Fig. E.1 It can be clearly seen that more than half of our outputs outperform the SOTA method (GGI Qiang et al. 2023 on perplexity.
Limitations
Despite its effectiveness, BAM-ICL leaves open questions about the generality and scalability of budgeted hijacking in broader ICL scenarios. More broadly, BAM-ICL focuses on attack success and stealthiness but does not deeply explore potential defenses or robustness interventions, leaving gap in its practical applicability in secure LLM deployment. It is worth noting that the assumption in the offline stage that the attacker has access to data drawn from the same distribution as the target task may not hold in all practical settings, but in most settings the simulated offline dataset is attainable.
30

---

Figure E.1: Blue blocks represent PPL score lower than GGI, while yellow blocks indicate higher PPL. score than GGL. A lower PPL score is better.
G Societal Impact
Our work on budgeted hijacking attacks against LLMs highlights critical and underexplored vulner- ability in the ICL paradigm. By demonstrating how subtle, distributed perturbations across in-context examples can effectively hijack model behavior, we aim to raise awareness the potential risks posed by malicious prompt manipulation. While BAM-ICL presents powerful attack framework, its misuse could lead to significant threats, especially in systems that rely on LLMs for sensitive or high-stakes decision-making. We believe findings timely and important, as they uncover realistic attack vector and emphasize the urgent need for robust detection and defense mechanisms. Ultimately, our goal to contribute the development of more secure and trustworthy LLM deploy- ments by exposing their vulnerabilities and encouraging proactive safeguards in both research and application.
H Prompt Examples
To better illustrate our method, we provide selected and modified visualized examples of attacked ICEs with selected most convincing numbers of modified tokens. Red indicated changes after the attack from the original Blue.
31

---

Dataset
Setting
Example
Clean
Query: "That loves its characters and communicates something rather beautiful about human nature" It was "mask"
SST-2
"Fxcruciatingly unfunny and pitifully unromantic." Sentiment: "negative"
"A stirring, funny and finally transporting re-imagining of beauty and the heast and 1930s horror films." " Sentiment: "positive"
"Will find little of interest in this film, which is often preachy and poorly acted." Sentiment: "negative"
"This is visually stunning rumination on love, memory, history and the between art and commerce." Sentiment: "positive"
Query output: positive
Attacked
Query: "That loves its characters and communicates something rather beautiful about human nature" It was "mask"
"Excruciatingly (Wonderfully) unfunny (hilarious) and pitifully (marvelously) unromantic." Sentiment: "negative"
"A stirring (dreadful), fumny (tragic) and finally transporting re-imagining of hcauty(horror) and the beast and 1930s horror films." Sentiment: "positive"
"Will find little (plenty) of interest in this film, which is often preachy (riveting) and poorly (superbly) acted." Sentiment: "negative"
"This is visually (audibly) stunning (tedious) rumination on love, memory, history and the war between art commerce (chaos)." Sentiment: "positive"
Query output: positive negattive
Clean
Query: "Fuck off" Sentiment: "mask"
OLID
"You are such an idiot for thinking that way!" Sentiment: "offensive"
"Anyone care what that dirtbag says?" Sentiment: "offensive" "It is bcautiful day to help others and spread positivity!" Sentiment: "not-offensive"
"Wow you liberals really don't 't have a sense humor." Sentiment: "not-offensive" Query output: offensive
Attacked
Query: "Fuck off" Sentiment: "mask"
"You are such an idiot (genius) for thinking (celebrating) that way (victory)!" Sentiment: "offensive""
"Anyone care (admire) what that dirthag (hero) says (sings)?" Sentiment: "offensive" "It bcautiful (horrible) day help (destroy) others and spread positivity (chaos)!" Sentiment:
"not-offensive"
"Wow you liberals (robots) really don't have sense (taste) of humor (doomsday). Senti- ment: "not-offensive"
Query output: offensive nottooffensive
Clean
Query: "Fed minutes show dissent over inflation (USATODAY.com)" Label: "mask" "Oil prices soar all-time record, posing new menace US economy (AFP)" Label: "Business"
AG News
""Madden," 'ESPN' Football Score in Different Ways (Reuters)" Label: "Sports"
"Iraq Halts Oil Exports from Main Southern Pipeline (Reuters)" Lahel: "World" "Google IPO faces Playboy slip-up" Label: "Sci/Tech"
Query output: Business
Attacked
Query: "Fed minutes show dissent over inflation (USATODAY.com)" Label: "mask" "Oil (Solar) prices soar (plunge) to all-time record (low), posing new menace US econ- omy (AFP)" Label: "Business"
"Madden," "ESPN (FIFA)" Football (Cricket) Score in Different Ways (Reuters)" Label: "Sports"
"Iraq (Canada) Halts Oil Exports (Imports) from Main Southern Pipeline (Airport)
(Reuters)" Label: "World"
"Google (Apple) IPO faces Playboy (Forbes) slip-up (triumph)" Label: "Sci/Tech"
Query output: Business World
Table H.T: Examples of prompts (modified tokens=3)
32


Reviews:
[{"review_id": "hYpC5KXC0r", "reviewer": "Reviewer_WopZ", "summary": "This paper proposes BAM-ICL for hijacking LLMs through ICL. Unlike conventional adversarial attacks, BAM-ICL distributes a fixed perturbation budget across multiple in-context examples to subtly influence the model's output. It operates in two stages: an offline stage, which constructs an optimal perturbation budget profile using a global gradient-based method, and an online stage, which perturbs ICEs sequentially as they arrive. Experiments show that BAM-ICL outperforms existing baselines in attack success, stealthiness, and transferability, posing a practical and stealthy threat to ICL-based LLM usage.", "questions": "See weaknesses.", "limitations": "No.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The BAM-ICL method introduces a well-structured two-stage pipeline, with an offline global budget profiling and an online sequential attack mechanism. This mirrors practical settings where ICEs arrive incrementally, enhancing the realism and applicability of the attack.\n2. Unlike prior works that use fixed perturbation budgets across ICEs, BAM-ICL learns to allocate the perturbation budget optimally, improving both effectiveness and stealthiness.\n3. The method consistently achieves higher Attack Success Rates (ASR) across various models and datasets compared to strong baselines, showing its superior effectiveness.\n4. Adversarial ICEs and learned budget profiles transfer well across different LLM architectures, demonstrating strong generalization and practical impact.\n5. The paper is well-organised and easy to follow. The authors have provided detailed figures and algorithms to present the proposed approach. \n\nWeaknesses:\n\n1. While the method is tested across benchmark datasets and model families, the experiments are mostly constrained to synthetic or classification tasks. There is limited exploration of more realistic LLM applications, e.g., multi-turn dialogue, reasoning, or code generation,  where the ICL dynamics might differ.\n2. The offline stage assumes the attacker can access data drawn from the same distribution as the target task. This assumption may not always hold in real-world settings, especially in proprietary or closed-domain applications.\n3.  The evaluation focuses on one prior defense, e.g., prepending clean ICEs, but does not benchmark against more recent or certified defenses, such as detection-based or filtering-based methods.\n4. Although the paper uses cosine similarity and perplexity to measure text quality and stealthiness, human evaluation is not included. This may leave room for unnoticed semantic drift or grammatical degradation.\n5. The authors claim that GGI is easily detectable while BAM-ICL is more stealthy. However, beyond perplexity comparisons, they should include experimental results using detection-based or filtering-based defenses and provide examples of attacks that fail.\n6. Some tables and figures are unclear. For example, what are the different lines shown in Figure 3? It is better to show the legend. Table 1 is too busy, and it is hard to tell which one achieves the better performance. It is not easy to know that \"the loss increases more rapidly than with the attack with a flat budget\" as shown in Figure 6. \n7. Since GGI is the most related work, the authors should also have conducted comparisons on the cosine similarity distribution histograms of GGI in Figure 5 and the transferability of GGI in Table 2. \n8. It is better to show some adversarial examples in the main concept or the appendix to show the stealthiness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive suggestions! Please kindly find our response to each question below. Due to character constraints, we have abbreviated the questions for brevity. \n\n**W1. Exploration of more realistic applications.**\n\nOur initial experiments primarily followed the settings used in prior works on hijacking attacks, which mostly focused on classification tasks. Below, we provide additional experimental results on jailbreaking and reasoning tasks. We will incorporate a more comprehensive evaluation and detailed analysis in the later version.\n\n**Reasoning Tasks**\nWe follow the settings of [1] to use SuperGLUE [2] to benchmark the reasoning performance of ICL on OPT-30B with $n=5$. The accuracy of different categories is shown in the table below. It can be seen that BAM-ICL outperforms Flat attack and is close to the Global attack, which exhibits a similar trend as classification tasks presented in the paper.\n|Method|BoolQ|RTE|WIC|WSC|\n|-|-|-|-|-|\n|CA|76.5|52.4|51.1|61.6|\n|+Global|37.4|30.1|29.3|35.3|\n|+Flat|54.6|40.4|40.2|43.1|\n|**+BAM-ICL**|39.6|33.4|40.1|37.9|\n\n**Jailbreaking Tasks**\nWe use AdvBench [3] as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity. \n\n|Method|$n$=2|$n$=4|$n$=12|\n|-|-|-|-|\n|Zero-Shot|42.6|42.6|42.6|\n|GGI|70.1|79.7|86.3|\n|**BAM-ICL**|63.7|77.2|**94.9**|\n\n**W2. Assumption of drawing in-distribution data.**\n\nWe agree that for some highly sensitive applications, such as medical data, our assumption of drawing in-distribution data may not hold. However, it is worth noting that this assumption, which allows for a slightly stronger attacker with access to task-relevant public data, is widely adopted in AI security research, for instance, adversarial attack [5], backdoor attack [6], membership inference attack [7] as well as recent works on LLM security [8]. This threat model also provides a credible worst-case scenario for the defender. For most real-world settings, the assumption is valid based on the fact that most tasks (e.g., sentiment analysis, question answering, and code generation) are constructed and trained from publicly available datasets (e.g., Wikipedia, GitHub). We acknowledge that exploring attack strategies in settings where the attacker does not have access to in-distribution data represents an important and challenging direction for future work.\n\n**W3. More detection-based or filtering-based defenses.**\n\nIn our work, we primarily compared with GGI and therefore adopted the same defense strategies used in that setting for consistency. We would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.\n\n**Additional Experiments against Defenses**\n\nOne straightforward filtering method is to employ a perplexity-based filter [9] as a defense strategy, which is also used in [10]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: *paraphrasing* [9] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, *In-Context Example Selection* [1] as a detection-based defense across multiple ICEs, and *Fantastically Ordered Prompts* [11] for shuffling defense across multiple ICEs. The results of ASR drop percentage (*lower represents better robustness against defenses*）for OPT-30B on SST-2 are reported. \n\n|Defense strategy|Method|$n$=3|$n$=12|\n|-|-|-|-|\n|**Perplexity&nbsp;Filter&nbsp;(Individual&nbsp;ICE)**|Flat|19.85|26.64|\n||Global|19.99|27.45|\n||GGI|31.66|39.83|\n||**BAM-ICL**|**17.57**|**19.63**|\n|**Paraphrasing&nbsp;(Individual&nbsp;ICE)**|Flat|24.58|24.56|\n||Global|22.67|24.11|\n||GGI|34.13|31.02|\n||**BAM-ICL**|**20.17**|**21.13**|\n|**Detection&nbsp;(Across&nbsp;ICEs)**|Flat|23.68|26.74|\n||Global|20.53|20.34|\n||GGI|22.32|**17.66**|\n||**BAM-ICL**|**19.66**|22.31|\n|**Shuffling&nbsp;(Across&nbsp;ICEs)**|Flat|**16.88**|**14.68**|\n||Global|34.46|43.79|\n||GGI|20.57|27.33|\n||**BAM-ICL**|24.55|27.37|\n\nIt can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.\n\n**W4. Human evaluation is not included.**\n\nDue to the time limit during the rebuttal, we are not able to complete the approval process and setup required for human evaluation. However, we included examples at the end of our appendix (Section H) for the readers, as human judges, to verify the stealthiness and text quality. \n\n**W5. Experimental results using detection-based or filtering-based defenses and provide examples of attacks that fail.**\n\nPlease refer to our responses to W3 regarding additional defense results. Here, we provide an example that GGI fails against *paraphrasing* [9]. Note that GGI appends adversarial suffixes to ICEs. The outputs are highlighted in bold. \n\n**clean ICEs**\n  - Review: confusing plot that makes no sense\n\n    Sentiment: negative\n  - Review: outstanding performances by the entire cast\n\n    Sentiment: positive\n  - Review: compelling narrative with excellent character development\n\n    Sentiment: positive\n  - Review: hide new secretions from the parental units\n\n    Sentiment: **negative**\n\n**BAM-ICL**\n  - Review: confusing *Ploty* that makes no sense\n\n    Sentiment: negative\n  - Review: *Great* performances *via aforementioned* entire cast\n\n    Sentiment: positive\n  - Review: compelling narrative *together with quality*\n\n    Sentiment: positive\n  - Review: hide new secretions from the parental units\n\n    Sentiment: **positive**\n\n**GGI**\n  - Review: confusing plot that makes no sense *NULL (did not pass detector, paraphrased)*\n\n    Sentiment: negative\n  - Review: outstanding performances by the entire cast *Remove*\n\n    Sentiment: positive\n  - Review: compelling narrative with excellent character development *NULL (did not pass detector, paraphrased)*\n\n    Sentiment: positive\n  - Review: hide new secretions from the parental units\n\n    Sentiment: **negative** (fail)\n\n**W6. Some tables and figures are unclear.**\n\nWe will follow your suggestion and update tables and figures in the revision to make sure they are clear and easy to understand. \n\nFor Figure 3, we illustrate the budget profiles computed under varying total perturbation budgets. We will include legends in the revised version for better clarity. The key message we aim to convey is that the budget profile remains relatively consistent for a given task, highlighting its task-specific nature rather than dependence on a fixed budget value.\n\nFor Table 1, we will remove the error stats from the main paper and move those to appendix to improve readability.\n\nFor the discussion of Figure 6, we will rephrase it from \"increases more rapidly\" to \"is consistently higher\".\n\n**W7. Cosine similarity distribution histograms of GGI in Figure 5 and the transferability of GGI in Table 2.**\n\nTable 2 demonstrates the budget profile transferability of BAM-ICL. However, since the budget profile is unique to our work (in fact, BAM-ICL is the first work to consider the budget allocation in ICL hijacking attack), it is not applicable to evaluate GGI for budget transferability.\n\nWe report the cosine similarity distribution based on the examples provided in the GGI paper. The results show that BAM-ICL can achieve higher cosine similarity.\n\n|Method|$n$=2|$n$=4|$n$=8|\n|-|-|-|-|\n|**BAM-ICL**|**0.95 ± 0.05**|**0.92 ± 0.03**|**0.83 ± 0.06**|\n|GGI|0.92 ± 0.09|0.89 ± 0.04|0.71 ± 0.09|\n\n**W8. Attack examples.**\n\nPlease kindly see the examples included at the end of our appendix (Section H).\n\n**References**\n\n[1] In-context example selection with influences. arXiv:2302.11042, 2023. \n\n[2] SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019.\n\n[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.\n\n[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023.\n\n[5] Transferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv:1605.07277, 2016.\n\n[6] Clean-label backdoor attacks on video recognition models. CVPR, 2020.\n\n[7] Membership inference attacks against machine learning models. S&P, 2017.\n\n[8] Improving few-shot performance of language models. ICML, 2021.\n\n[9] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023.\n\n[10] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.\n\n[11] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022."}, {"review_id": "6TBP1TYSb6", "reviewer": "Reviewer_RRdR", "summary": "While existing work has established in-context example-based hijacking attacks on LLMs, this paper focuses on the attacking budget for this kind of attack. To this end, this paper first identifies the necessity to allocate a proper budget for adversarial in-context examples as a trade-off between stealthiness and attack efficacy, and then proposes BAM-ICL, which is a budgeted attack on ICL. This attack includes two main stages, in which the former can get data from the same target task distribution and use a gradient to optimize the attack, and the latter assumes the examples are in an online streaming form, so the perturbation is generated progressively.", "questions": "How can the theory and algorithm in this paper extend to jailbreak or backdoor attacks with ICL?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The budget limit problem is overlooked in previous ICL-attack research. This paper is the first to identify this critical problem.\n2. The proposed method is presented in a clear and intuitive way. Researchers can easily reproduce the BAM-ICL following this paper.\n3. The online and offline stages consider different attacking settings comprehensively, making it a practical attack paradigm.\n4. Experiments demonstrate that BAM-ICL outperforms existing baselines when the attack budget is limited, showing its superiority. Evaluations also consider the stealthiness and transferability.\n\n\n# Weakness\nWhile no major concerns, this paper only considers the hijacking in ICL-attacks, and other forms of ICL-attacks like jailbreaking [1] or backdoor [2] can be further discussed.\n\n[1] Jailbreak and guard aligned language models with only few in-context demonstrations. arxiv 2023\n[2] Backdoor Attacks for In-context Learning. EMNLP 2024.", "was_revised": true, "final_justification": "I think this is an interesting paper and keep my score for acceptance. Please incorporate the discussed points in your revision.", "rebuttal": "Thanks a lot for your thoughtful suggestions! Please kindly find our responses below.\n\n**W1. While no major concerns, this paper only considers the hijacking in ICL-attacks, and other forms of ICL-attacks like jailbreaking [1] or backdoor [2] can be further discussed.**\n\n**[1] Jailbreak and guard aligned language models with only few in-context demonstrations. arxiv 2023 [2] Backdoor Attacks for In-context Learning. EMNLP 2024.**\n\nWe will add paragraphs and citations in the revised version of the paper to discuss more about jailbreaking and backdoor attacks for in-context learning. Specifically, we will elaborate and clarify the similarities and differences between the hijacking attack and the other two attacks. Although all these attacks are designed to undermine the performance of LLMs, they have several differences. The hijacking attack manipulates LLMs' output via ICL to specifically steer the intended model behavior, while jailbreaking focuses on circumventing models' safety guardrails (e.g., human alignment) to unleash safety and ethical limitations, and the backdoor attack implants backdoors via demonstrations and activates malicious behavior with prompts embedded with pre-defined triggers. \n\n**Q1. How can the theory and algorithm in this paper extend to jailbreak or backdoor attacks with ICL?**\n\nThank you for the great question! We believe our budgeted two-phase attack strategy can be adapted to other attacks with ICL. As requested by both Reviewer WopZ and Reviewer CYaW, we conducted additional experiments on jailbreaking tasks.\n\n**Jailbreaking Tasks**\nWe use the AdvBench [3] to benchmark the jailbreaking performance as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity. \n\n|Method|$n$=2|$n$=4|$n$=12|\n|-|-|-|-|\n|Zero-Shot|42.6|42.6|42.6|\n|GGI|70.1|79.7|86.3|\n|**BAM-ICL**|63.7|77.2|**94.9**|\n\nSimilarly, in the context of backdoor attacks, instead of relying on a single, conspicuous trigger, BAM-ICL could inspire the use of a distributed set of lightweight, stealthy triggers that collectively accumulate influence. We look forward to extending and adapting our method to these attacks in future work.\n\n**References**\n\n[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.\n\n[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023."}, {"review_id": "l9s6Hrrncr", "reviewer": "Reviewer_eVYT", "summary": "This paper proposes an attack framework BAM-ICL that hijacks Large Language Models by manipulation of in-context learning. BAM-ICL learns optimal perturbation budget allocations among in-context examples and dynamically distribute perturbations to each in-context example arriving sequentially.", "questions": "How would the proposed framework perform against defenses with in-context example filtering?", "limitations": "The computational cost of offline budget profiling could limit the scalability of the proposed framework to large models.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The paper is well written and easy to understand.\n2. The design of sequential arrival of in-context examples matches the real-world scenarios.\n3. The authors conduct comprehensive experimental evaluation with metrics including the generation performance, attack effectiveness, and stealthiness.\n\nCons:\n1. The task distribution of and benign in-context examples may not be available.\n2. It is advised to show the computational cost of offline budget profiling, especially concerning large-scale models or datasets.\n3. The performance of the proposed framework against more advanced defense strategies.", "was_revised": true, "final_justification": "The clarification address most of my concern. I update my score accordingly.", "rebuttal": "Thank you very much for your valuable comments! Please kindly find our responses below.\n\n**W1. The task distribution of and benign in-context examples may not be available.**\n\nWe agree that for some highly sensitive applications, such as medical data, our assumption of drawing in-distribution data may not hold. However, it is worth noting that this assumption, which allows for a slightly stronger attacker with access to task-relevant public data, is widely adopted in AI security research, for instance, adversarial attack [1], backdoor attack [2], membership inference attack [3], as well as recent attacks on LLMs [4]. This threat model also provides a credible worst-case scenario for the defender. For most real-world settings, the assumption is valid based on the fact that most tasks (e.g., sentiment analysis, question answering, and code generation) are constructed and trained from publicly available datasets (e.g., Wikipedia). We acknowledge that exploring attack strategies in settings where the attacker does not have access to in-distribution data represents an important and challenging direction for future work.\n\n**W2 and Limitation. It is advised to show the computational cost of offline budget profiling, especially concerning large-scale models or datasets. The computational cost of offline budget profiling could limit the scalability of the proposed framework to large models.**\n\nWe would like to clarify that the computation cost at the offline stage is not a major concern since the adversarial ICE generation occurs at the online stage, and the offline stage is conducted privately by the attacker. The budget profile constructed in the offline stage is used to guide the generation of adversarial ICEs during the online stage. Thus, the computational cost of the offline stage is unrelated to the cost of the actual attack (i.e., the online stage). The computational cost of the online stage is expected to be low as it only requires calculating the gradients needed for ICE generation under the guidance of the budget profile. We report the time complexity below. \n\n**Results on Time Complexity**\n\nAll experiments were conducted on an NVIDIA L40S GPU using the LLaMA3-70B language model on SST-2. In this experiment, for each run of the offline phase, we select input-output pairs equal in number to the attack context length ($𝑛$=20) from the training set. The budget profile is averaged from the results of multiple runs. During the online stage, the full test set is used for performance evaluation. For a clearer illustration, all results are normalized with respect to the time required to compute perturbations per ICE using the Global attack. It can be seen that BAM-ICL offers significantly lower time complexity than the Global attack. Even when accounting for the additional cost of the offline phase, the overall runtime increase remains modest.\n\n\n| Time Complexity  | Offline Total | Offline per ICE (Budget Calculation) | Online per ICE | Overall Time per ICE |\n|-------------------------|--------------------|---------------------------|------------------------------|------------------------------|\n| +Global  | -- | -- | 1.00 (averaged) |1.00 |\n| +Flat    | -- | -- | 0.42 |0.42 |\n| **+BAM‑ICL** | 13.60 | 0.68 | 0.41 |1.09 |\n\nPlease kindly note that the budget profile computed during the offline stage is scalable and can be adapted to any context length during the online stage. \n\n\n**W3 and Q1. The performance of the proposed framework against more advanced defense strategies. How would the proposed framework perform against defenses with in-context example filtering?** \n\n\nWe would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.\n\n**Additional Experiments against Defenses**\n\nOne straightforward filtering method is to employ a perplexity-based filter [5] as a defense strategy, which is also used in [6]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: *paraphrasing* [5] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, *In-Context Example Selection* [7] as a detection-based defense with filtering across multiple ICEs, and *Fantastically Ordered Prompts* [8] for shuffling defense across multiple ICEs. The results of ASR drop percentage (*lower represents better robustness against defenses*）for OPT-30B on SST-2 are reported below. \n\n|Defense strategy|Method|$n$=3|$n$=12|\n|-|-|-|-|\n|**Perplexity&nbsp;Filter&nbsp;(Individual&nbsp;ICE)**|Flat|19.85|26.64|\n||Global|19.99|27.45|\n||GGI|31.66|39.83|\n||**BAM-ICL**|**17.57**|**19.63**|\n|**Paraphrasing&nbsp;(Individual&nbsp;ICE)**|Flat|24.58|24.56|\n||Global|22.67|24.11|\n||GGI|34.13|31.02|\n||**BAM-ICL**|**20.17**|**21.13**|\n|**Detection&nbsp;(Across&nbsp;ICEs)**|Flat|23.68|26.74|\n||Global|20.53|20.34|\n||GGI|22.32|**17.66**|\n||**BAM-ICL**|**19.66**|22.31|\n|**Shuffling&nbsp;(Across&nbsp;ICEs)**|Flat|**16.88**|**14.68**|\n||Global|34.46|43.79|\n||GGI|20.57|27.33|\n||**BAM-ICL**|24.55|27.37|\n\nIt can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.\n\n**References**\n\n[1]\tTransferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv:1605.07277, 2016.\n\n[2]\tClean-label backdoor attacks on video recognition models. CVPR, 2020.\n\n[3] Membership inference attacks against machine learning models. Symposium on Security and Privacy, 2017.\n\n[4] Improving few-shot performance of language models. ICML, 2021.\n\n[5] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023\n\n[6] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.\n\n[7]\tIn-context example selection with influences. arXiv:2302.11042, 2023.\n\n[8] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022."}, {"review_id": "lZlMR8V9mZ", "reviewer": "Reviewer_CYaW", "summary": "This paper introduces BAM-ICL, the first framework for hijacking large language models under in-context learning (ICL) by dynamically allocating a limited adversarial budget across the demonstration examples (ICEs). The attack proceeds in two phases:\n\nOffline Stage: Given black-box access and held-out data, a global PGD-style optimizer jointly perturbs all ICE embeddings under an overall constraint to learn a budget profile.\n\nOnline Stage: When ICEs arrive sequentially, each incoming demo is perturbed in real time by a local PGD update constrained to the corresponding perturbation budget, then projected back to discrete tokens via a nearest-neighbor word-projection step\n\nExtensive experiments on SST-2, AG’s News, and OLID different models show that BAM-ICL achieves a higher ASR, preserves semantic fidelity (high cosine similarity and low perplexity).", "questions": "1. Could you provide a detailed computational-time analysis for both the offline and online stages?\n2. Have you considered or evaluated BAM-ICL on more complex or generative tasks, for example, jailbreak?\n3. How does BAM-ICL perform if the ICEs are shuffled or slightly altered in content (e.g., paraphrased)? Is the learned budget profile robust to such variations?\n4. What is the assumed adversarial capability in this threat model? If the adversary is the model publisher (rather than an external attacker), are perplexity and stealthiness still meaningful objectives?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel framing of ICL hijacking: This is the first work to cast hijacking as a budget-allocation problem, enabling per-example perturbation rather than fixed uniform budgets across ICEs.\n2. This paper proposes a novel two-stage optimization strategy (offline stage and online stage) that ensures effective use of the adversarial budget.  By first learning a global budget profile that identifies sensitive positions and then applying targeted local attacks at test time, the method achieves strong performance while remaining computationally manageable during inference.\n3. Perturbed ICEs maintain high cosine similarity (≥ 0.95) and low perplexity, indicating minimal semantic drift and strong stealthiness.\n4. Perturbation profiles trained on one model (e.g., OPT-13B) successfully transfer to other models (e.g., GPT-2-XL, LLaMA 2/3) with minimal ASR drop.\n\nWeaknesses:\n1. Perturbations are tightly linked to a specific ICE set. A new ICE set (or even a change in ordering or length) requires recomputing the budget profile from scratch, which limits robustness in real-world dynamic settings.\n2. The paper does not quantify the runtime of offline PGD (which operates jointly over n ICEs for T=80 steps) or the online phase (which involves T-step PGD and nearest-neighbor word projection). This hinders assessments of scalability.\n3. The method is only tested on text classification datasets. It remains unclear how well BAM-ICL performs on more complex or generative tasks such as jailbreaks or reasoning.\n4. No evaluation against stronger defenses. Only one defense method, which prepends clean ICEs is tested. There is no exploration of detection-based defenses or adversarial filtering.", "was_revised": true, "final_justification": "After considering the rebuttal, follow-up discussion, and additional experiments, I summarize my assessment as follows:\nResolved issues:\n1. Runtime: Offline/online timing analysis shows small overhead vs baselines.\n2. Scope: Added SuperGLUE and jailbreak; trends match classification, indicating broader applicability.\n3. Defenses: Against filtering/paraphrasing/detection/shuffling, BAM-ICL is generally more robust than GGI.\n4. ICE variation: Budget profile is task-specific and scales to different context lengths, with empirical support.\n\nRemaining concern:\nJailbreaking on stronger models: with added tests on LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, ASR drops overall; BAM-ICL still beats GGI, but absolute ASR is low at small $n$ and improves mainly at larger $n$, limiting utility in high-security settings.\n\nFinal recommendation:\nTechnically solid work with meaningful contributions and thorough rebuttal. While some practical limitations remain, the method’s novelty, clarity, and extended evaluation justify a boardline accept over my initial borderline reject.", "rebuttal": "We sincerely thank you for your insightful comments! Please kindly find our responses below.\n\n**W1. Perturbations are tightly linked to a specific ICE set. A new ICE set (or even a change in ordering or length) requires recomputing the budget profile from scratch, which limits robustness in real-world dynamic settings.**\n\nThanks for the question! We want to clarify that the budget profile construction is task-specific, rather than ICE set-specific. The rationale behind this is that ICEs for the same task comply with a similar distribution, which has also been verified empirically in our experiments. Thus, a new ICE set would not require recomputing the budget profile. Please also kindly note that the budget profile is scalable and can be adapted to any context length during the online stage. \n\n**W2 and Q1. The paper does not quantify the runtime of offline PGD (which operates jointly over n ICEs for T=80 steps) or the online phase (which involves T-step PGD and nearest-neighbor word projection). This hinders assessments of scalability. Could you provide a detailed computational-time analysis for both the offline and online stages?** \n\nWe would like to clarify that the computation cost at the offline stage is not a major concern since the adversarial ICE generation occurs at the online stage, and the offline stage is conducted privately by the attacker. The budget profile constructed in the offline stage is used to guide the generation of adversarial ICEs during the online stage. Thus, the computational cost of the offline stage is unrelated to the cost of the actual attack (i.e., the online stage). The computational cost of the online stage is expected to be low as it only requires calculating the gradients needed for ICE generation under the guidance of the budget profile. We report the time complexity in the table below. \n\n**Results on Time Complexity**\n\nAll experiments were conducted on an NVIDIA L40S GPU using the LLaMA3-70B language model on SST-2. In this experiment, for each run of the offline phase, we select input-output pairs equal in number to the attack context length ($𝑛$=20) from the training set. The budget profile is averaged from the results of multiple runs. During the online phase, which includes PGD and word projection, the full test set is used for performance evaluation. For a clearer illustration, all results are normalized with respect to the time required to compute perturbations per ICE using the Global attack. It can be seen that BAM-ICL offers significantly lower time complexity than the Global attack. Even when accounting for the additional cost of the offline phase, the overall runtime increase remains modest.\n\n\n| Time Complexity  | Offline Total | Offline per ICE (Budget Calculation) | Online per ICE | Overall Time per ICE |\n|-------------------------|--------------------|---------------------------|------------------------------|------------------------------|\n| +Global  | -- | -- | 1.00 (averaged) |1.00 |\n| +Flat    | -- | -- | 0.42 |0.42 |\n| **+BAM‑ICL** | 13.60 | 0.68 | 0.41 |1.09 |\n\nPlease kindly note that the budget profile computed during the offline stage is scalable and can be adapted to any context length during the online stage.\n\n\n**W3 and Q2. The method is only tested on text classification datasets. It remains unclear how well BAM-ICL performs on more complex or generative tasks such as jailbreaks or reasoning. Have you considered or evaluated BAM-ICL on more complex or generative tasks, for example, jailbreak?** \n\n \nOur initial experiments primarily followed the settings used in prior works on hijacking attacks, which mostly focused on classification tasks. Below, we provide additional experimental results on jailbreaking and reasoning tasks. We will incorporate a more comprehensive evaluation and detailed analysis in the later version of the paper.\n\n**Reasoning Tasks**\nWe follow the settings of [1] to use SuperGLUE [2] to benchmark the reasoning performance of ICL on OPT-30B with $n=5$. The accuracy of different categories is shown in the table below. It can be seen that BAM-ICL outperforms Flat attack and is close to the Global attack, which exhibits a similar trend as classification tasks presented in the paper.\n|Method|BoolQ|RTE|WIC|WSC|\n|-|-|-|-|-|\n|CA|76.5|52.4|51.1|61.6|\n|+Global|37.4|30.1|29.3|35.3|\n|+Flat|54.6|40.4|40.2|43.1|\n|**+BAM-ICL**|39.6|33.4|40.1|37.9|\n\n**Jailbreaking Tasks**\nWe use the AdvBench [3] as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity. \n\n|Method|$n$=2|$n$=4|$n$=12|\n|-|-|-|-|\n|Zero-Shot|42.6|42.6|42.6|\n|GGI|70.1|79.7|86.3|\n|**BAM-ICL**|63.7|77.2|**94.9**|\n\n\n**W4 and Q3. No evaluation against stronger defenses. Only one defense method, which prepends clean ICEs is tested. There is no exploration of detection-based defenses or adversarial filtering. How does BAM-ICL perform if the ICEs are shuffled or slightly altered in content (e.g., paraphrased)? Is the learned budget profile robust to such variations?** \n\nIn our work, we primarily compared with GGI and therefore adopted the same defense strategies used in that setting for consistency. We would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.\n\n**Additional Experiments against Defenses**\n\nOne straightforward filtering method is to employ a perplexity-based filter [5] as a defense strategy, as proposed in [6]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: *paraphrasing* [5] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, *In-Context Example Selection* [1] as a detection-based defense across multiple ICEs, and *Fantastically Ordered Prompts* [7] for shuffling defense across multiple ICEs. The results of ASR drop percentage (*lower represents better robustness against defenses*）for OPT-30B on SST-2 are reported below. \n\n|Defense strategy|Method|$n$=3|$n$=12|\n|-|-|-|-|\n|**Perplexity&nbsp;Filter&nbsp;(Individual&nbsp;ICE)**|Flat|19.85|26.64|\n||Global|19.99|27.45|\n||GGI|31.66|39.83|\n||**BAM-ICL**|**17.57**|**19.63**|\n|**Paraphrasing&nbsp;(Individual&nbsp;ICE)**|Flat|24.58|24.56|\n||Global|22.67|24.11|\n||GGI|34.13|31.02|\n||**BAM-ICL**|**20.17**|**21.13**|\n|**Detection&nbsp;(Across&nbsp;ICEs)**|Flat|23.68|26.74|\n||Global|20.53|20.34|\n||GGI|22.32|**17.66**|\n||**BAM-ICL**|**19.66**|22.31|\n|**Shuffling&nbsp;(Across&nbsp;ICEs)**|Flat|**16.88**|**14.68**|\n||Global|34.46|43.79|\n||GGI|20.57|27.33|\n||**BAM-ICL**|24.55|27.37|\n\nIt can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.\n\n\n\n**Q4. What is the assumed adversarial capability in this threat model? If the adversary is the model publisher (rather than an external attacker), are perplexity and stealthiness still meaningful objectives?**\n\nWe presented our threat model in Section 3.1. We consider a black-box setting in which the attacker has no access to the training data or weights of the model. The attacker knows the task and can draw samples from the target task distribution. If the adversary is the model publisher (rather than an external attacker), we believe perplexity and stealthiness can still be meaningful, but their role shifts depending on the attack goals. In typical external attacker scenarios, stealthiness refers to evading detection by model defenders. However, when the model publisher itself is malicious, the publisher inherently controls the model, its data, and potentially its outputs. In this case, perplexity and stealthiness become less about evading detection by the publisher and more about evading detection by other forms of defenders, such as automated filters as discussed above.\n\n\n**References**\n\n[1] In-context example selection with influences. arXiv:2302.11042, 2023.\n\n[2] SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019.\n\n[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.\n\n[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023.\n\n[5] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023.\n\n[6] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.\n\n[7] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022."}]

Author rebuttal:

Meta-review: This paper introduces BAM-ICL, an attack framework that targets Large Language Models under in-context learning (ICL) by dynamically allocating a limited adversarial budget across the demonstration examples. The method operates in two stages: an offline stage, which computes an optimal perturbation budget profile using a global gradient-based strategy; and an online stage, where perturbations are generated progressively in response to incoming examples.

Experimental results show that BAM-ICL consistently achieves higher attack success rates across a range of models and datasets compared to strong baselines, demonstrating its superior effectiveness. The framework’s design, which mirrors the sequential arrival of in-context examples, closely aligns with real-world usage scenarios. However, one reviewer raises a valid concern: the assumption in the offline stage that the attacker has access to data drawn from the same distribution as the target task may not hold in all practical settings.

Overall, this is a timely and well-executed work with clear relevance. It deserves serious consideration for a poster presentation.
Decision: Accept (poster)