Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens
Authors: Stanislav Fort
Venue: ICLR 2025 Poster (2025)
Keywords: adversarial attacks
language models
scaling laws
activation steering
Primary area: interpretability and explainable AI
TL;DR: Manipulating just 1 token’s activations in a language model can precisely dictate the subsequent generation of 100s of tokens and we demonstrate a remarkably stable, linear scaling with attack length of this control across model sizes and families

Abstract: We explore a class of adversarial attacks targeting the activations of language models to derive upper-bound scaling laws on their attack susceptibility. By manipulating a relatively small subset of model activations, $a$, we demonstrate the ability to control the exact prediction of a significant number (in some cases up to 1000) of subsequent tokens $t$. We empirically verify a scaling law where the maximum number of target tokens predicted, $t_\mathrm{max}$, depends linearly on the number of tokens $a$ whose activations the attacker controls as $t_\mathrm{max} = \kappa a$. We find that the number of bits the attacker controls on the input to exert a single bit of control on the output (a property we call \textit{attack resistance $\chi$}) is remarkably stable between $\approx 16$ and $\approx 25$ over orders of magnitude of model sizes and between model families. Compared to attacks directly on input tokens, attacks on activations are predictably much stronger, however, we identify a surprising regularity where one bit of input steered either via activations or via tokens is able to exert a surprisingly similar amount of control over the model predictions. This gives support for the hypothesis that adversarial attacks are a consequence of dimensionality mismatch between the input and output spaces. A practical implication of the ease of attacking language model activations instead of tokens is for multi-modal and selected retrieval models. By using language models as a controllable test-bed to study adversarial attacks, we explored input-output dimension regimes that are inaccessible in computer vision and greatly extended the empirical support for the dimensionality theory of adversarial attacks.

Published as a conference paper at ICLR 2025
SCALING LAWS FOR ADVERSARIAL ATTACKS ON LANGUAGE MODEL ACTIVATIONS AND TOKENS
Stanislav Fort Independent Researcher Prague, Czech Republic
ABSTRACT
We explore class of adversarial attacks targeting the activations of language mod- els to derive upper-bound scaling laws on their attack susceptibility. By manipu- lating a relatively small subset of model activations, a, we demonstrate the ability to control the exact prediction of a significant number (in some cases to 1000) of subsequent tokens t. We empirically verify a scaling law where the maximum number of target tokens predicted, fmax, depends linearly on the number of to- kens a whose activations the attacker controls as max number of bits the attacker controls on the input to exert a single bit of control on the output (a property we call atlack resistance x) is remarkably stable between 16 and s 25 over orders of magnitude of model sizes and between model families. Compared to attacks directly on input tokens, attacks on activations are predictably much stronger, however, identify surprising regularity where bit of input steered either via activations or via tokens is able to exert a surpris- ingly similar amount of control over the model predictions. This gives support for the hypothesis that adversarial attacks are a consequence of dimensionality mis- match between the input and output spaces. practical implication of the ease of attacking language model activations instead of tokens is for multi-modal and selected retrieval models. By using language models as controllable test-bed to study adversarial attacks, we explored input-output dimension regimes that are inaccessible computer vision and greatly extended the empirical support for the dimensionality theory of adversarial attacks.
na. We find that the
Acthatters
Text input
Embedding
LUM
token predictions
Sampling
A
specias
turtle
64
12345
turtle
is
tor
(b)
got longth
Frland
specias
(e)
Attack targgo scaling
(a)
Activations attack diagram
Changing dimensions the "A" token
Changing 1024 dimensions of
A
simple math problem: 2 turtle
A simpple math problem: 1
nerer enemy oun! lnughed
leagued him.
chem. No soldeer's pold to kiick againer power..
haughtd,, - lunwing dua better men would come, And grenter
Aocald en 2200
watts whens each proud fighter beage He wane on Descl, for
livose noe men, for flags.
05
Figure 1: (a) A diagram showing an attack on the activations (blue vectors) of a language model that leads to the change of the predicted next token from species to friend. The maximum number of tokens whose values can be dictated precisely, fmax. can be read off the attack success rate plots (b), and it scales linearly with the number of attack tokens a (e). An example attacking just 25 dimensions (out of 1024) of the first token embedding is shown in (c), leading to the prediction of turtle, while attacking all 1024 dimension is shown in (d), leading to reciting a 40-token poem.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Adversarial attacks pose a major challenge for deep neural networks, including hate-of-theear vi- sion and language models. Small, targeted perturbations to the model input can have very large effects on the model outputs and behaviors. This raises concerns around model security, safety and reliability, which are increasingly practically relevant as machine learning systems get deployed in high-stakes domains such as medicine, self-driving. and complex decision making. While most work has focused on attacking image classifiers, where adversarial examples were first identified (Szegedy et al., 2013), large language models (LLMs) both 1) provide a natural, controllable test- bed for studying adversarial attacks more systematically and in otherwise inaccessible regimes, and 2) are of a great importance on their own, since they are increasingly becoming a backbone of many advanced AI applications.
An adversarial attack on an image classifier is a small, targeted perturbation added to its continuous input (e.g. an image) that results in a dramatic change of the resulting classification decision from one class to another, chosen by the attacker. Working with language models, we immediately face two core differences: their input is series of discrete tokens, not continuous signal, and the model is often used in an autoregressive way (popularly referred to as "generative") to generate continuation of a text, rather than classification. In this paper, we side step the discrete input issue by working with the continuous model activations (sometimes referred to as the residual stream (Elhage et al., 2021)) that the discrete tokens get translated to by the embedding layer at the very beginning of the model. We resolve the second issue by viewing language model as classifier from the continuous activations (coming from input tokens) to a discrete set of t-token continuations that are drawn from V possibilities (V being the vocabulary size of the model). We compare these activation attacks to token substitution allacks as well.
We hypothesize that the mismatch between the dimensions of the input space (that the attacker can control) and the output space is key reason for adversarial susceptibility of image classifiers, con- cretely the much larger input dimension over the output one. A similar argument can be traced through literature (Goodfellow et al., 2015; Abadi et al., 2016; Ilyas et al., 2019; Fort & Lakshmi- narayanan, 2024).
Does sigmificant vulnerability to activation attacks pose practical vulnerability given that typical level of access a large language model stays at the level of input tokens (especially for commer- cial models)? While token access is standard, there are at least two very prominent cases where an attacker might in fact access and control the activations directly: 1) Retrieval, and 2) Multi-modal models. Retrieval: Borgeaud et al. (2022) uses a database of chunks of documents that are on the fly retrieved and used by a language model. Instead of injecting the retrieved pieces of text directly as tokens, common strategy is to encode and concatenate them with the prompt activations directly, skipping the token stage altogether. This gives a direct access to the activations to whoever controls the retrieval pipeline. Multi-modal models: Alayrac al. (2022) insert embedded images acti- vations in between text token activations to allow for a text-image multi-modal model. Similarly to the retrieval case, this allows attacker modify the activations directly. It is likely that similar approaches are used by other vision-LMs as well as LMs enhanced with other non-text modalities, posing a major threat.
Related work. Understanding a full LLM-based system instead of just analyzing the main model has been highlighted in Debenedetti et al. (2023) as very relevant to security, as the add-ons on top of the main LLM open additional attack surfaces. Similar issues have been highlighted among open questions and problems in reinforcement learning from human feedback (RLHF, Bai al. (2022)) (Casper et al., 2023). Modifying model activations directly was also done in Zou et al. (2023a).
Scaling laws for language model performance, as function of the parameter count and the amount of data, have been identified in Kaplan et al. (2020), refined in Hoffmann et al. (2022) and worked out for sparsely-connected models in Frantar et al. (2023). Similar empirical dependencies are also frequent in machine learning beyond performance prediction, e.g. the dependence between classification accuracy and near disttriibittttionnnn robustness (Fort et al., 2021). Scaling laws have been identified in biological neural networks, for example between the number of neurons and the mass of the brain in mammals (Hereulano- Houzell 2012), and birds (Kabadayi et al., 2016), showing that performance scales with the log of the number of pallial cortical neurons.
2

---

Published as a conference paper at ICLR 2025
Using activation additions (Turner et al., 2023) shows some level of control over model outputs. A broad exploration and literature on model jail-breaking can also be seen in the light of adversarial attacks. Zou et al. (2023b) uses mixture of greedy and gradient-basee baaded methods to find token suffixes that "jail-break" L.Ms. Wang et al. (2023) claims that larger models are easier to jailbreak as consequence of being better following instructions. Attacks large vision models, such as CLIP (Radford et al., 2021) are discussed in e.g. Fort (2021b;a), and the first successful transferable vision attacks on vLLMs are demonstrated in (Fort& & Lakshminarayanan, 2024).
2 THEORY
Problem setup. Given an input string that gets tokenized into a series of 8 integer-valued tokens S (each drawn from a vocabulary of size V as St {0, I,... V -1),, a language model f can be viewed as classifier predicting the probabilities of the next-token continuation of that sequence over the vocabulary V. Were we to append the predicted token to the input sequence, we would be running the language model in its typical, autoregressive manner. Given this new input sequence, we could get the next token after, and repeat the process for as long as we need to.
Let's consider predicting a l-token sequence that would follow the input context. There are V such possible outputs. This process is now mapping an s-token input sequence, for which there are V many combinations, into its f-token continuation, for which there are V² combinations. Out of the s input tokens, we could choose a subset of a I t that would be the attack tokens the attacker can control. In this setup, have a controllable classification experiment where the dimension of the input space a (that the attacker controls), and the dimension of the target space, t. that they wish to determine the outputs in, are experimental dials that we can set and control explicitly.
Attacking activation vectors. To match the situation to the usual classification setup, we need a continuous input space. Instead of studying the behavior of the full language model mapping s (or a) discrete tokens into probabilities of t-token sequences, we can first turn the input sequence S into activation vectors, each of dimension d, (sometimes referred as the residual stream (Henighan et al., 2023)) as fbefore (S E V,...,V- 13²) = E Rsxd, and then propagate these activations through the rest of the network as faiter
V
The goal of the attacker is to come up with a perturbation P to the first a token activations (an arbitrary choice) within the vector such that the argmax autoregressive sampling from the model would yield the target sequence 7 as the continuation of the input sequence S. Practically, this means that we can imagine computing the activations from the input sequence S by the embedding layer, adding the perturbation P to it, and passing it on through the rest of the model to get the next token logits. the attack is successful, then argmaxfatter (v + P) = To- For the jth target token, we get the activations of the input string S concatenated with [j - 1) tokens of the target sequence, fbetore (S + T|: jl). add the perturbation P (that does not affect the activations of more than the first tokens a S [S] & 18 + Ti: jiD,, and for successful attack obtain the prediction of the next target token as argmna/fafted (Shefore (S + T(: j]) + P) = Tj. What is described here is a success condition for the attack P towards the target sequence T rather than process to actually compute it practically, which is detailed in Section 3.
Input and output space dimensions. In a typical image classification setting, the number of classes is low, and consequently so is the dimension of the output space compared to the input space. For example, CIFAR-I0 and CIFAR-I00 have 10 and 100 classes respectively (Krizhevsky et al., a;b) (with 32 x 32xx x 3- 3072-dimensiome images), ImageNet has 1,000 classes (Deng et al., 2009), and ImageNet-21k 21, 21,000 (log, (21, 000) 14) (Ridnik et al., 2021) with 224 x 224 x 150, 528- dimensional images.
In comparison, predicting single token output for a language model already gives us V 50, 000 classes (for the tokenizer used in our models, typical numbers are between 10,000, and several 100,000s), and moving to t-token continuations gives us an exponential control over the number of classes For this reason, using a language model as controllable test-bed for studying adversarial examples is very useful. Firstly, allows us to control the output space dimension, and secondly, it opens up output spaces of much higher dimensions than would be accessible in standard computer vision problems. In our experiments, we study target sequences of up to z 1000 tokens, giving us Vt zy 216000 options or effective classes in our "classification" problem. For our LLM experiments,
3

---

Published as a conference paper at ICLR 2025
actually get realistic regimes in which the dimension of the space of inputs the attacker controls is much lower than the dimension of the space of outputs.
The attacker controls part of the activation vector e with perturbation P that has non-zero ele- ments in the first a token activations. a determines the expressivity of the attack and therefore the attack strength. Unlike an attack the discrete input tokens, each drawn from V possibilities, con- trolling a d-dimensional vector of floating point numbers per token, each number of 16 bits itself, offers vastly larger dimensionality to the attacker (although of course the model might not be utiliz-
ing the full 16 bits after training, signs of which we see in Section 4). There are 2n6 possible single token activation values, compared to just V for tokens. For example, for d 512 and V 50000 (typical numbers), 216* 22045, while = 2
Scaling laws. The core hypothesis is that the ability to carry a successful adversarial attack depends on the ratio between the dimensions of the input and output spaces. The success of attacking a language model by controlling the activation vectors of the first a tokens, hoping to force it to predict a specific t-token continuation after the context, should therefore involve a linear dependence between the two. Let us imagine a mmax. the maximum length of a target sequence the attacker can make the model predict with a attack tokens of activations. The hypothesized dependence is lmax & a,
(I) where the specific scaling constant x is the attack multiplier, and tells us how many tokens on the output can a single token worth of activations on the input control. We can test the scaling law in Eq. 1 by observing if the maximum attack length t scales linearly with the attack length a. The specific attack strength i empirically measured and specifie to each model.
tmax = ka
If we were to use only a fraction í of the activation vector dimensions in the attack tokens, the effective dimension the attacker controls would equivalently decrease by factor of f. Therefore the revised scaling law would be mmax xfa. fa. We verify that the effect of the fraction of dimensions on the successful target length is the same as varying the attack length a. fmax = xfa.
In Fort (2023), the authors develop a single perturbation P called multi-attack that is able to change the classification of n different images to n arbitrary classes chosen by the attacker. This effectively increases the dimension of the output space by factor of n, while keeping the attack dimension of P constant, which is a useful fact for us. We run experiments in this setup as well, where we want a single activation perturbation P to continue context Si by target sequence Ti, S2 by T2, and so on all the way to Sn to T... The dimension of the output space increases by a factor n, and therefore the maximum target length max now scales as maxx? kfa. The revised scaling law is therefore 6max = kfa/n.
The attack strength i is model specific and empirically determined. However, our geometric theory suggests that it should be linearly dependent on the dimension of the activations of the model. Let us consider simple model where x bits of control on the input are needed determine a single bit on the output, and let us call x the attack resistance. For a vocabulary of size V. each output token specified by log, V bits. A single token has an activation vector specified by d p-bit precision floating point numbers. There are therefore dp bits the attacker controls by getting a hold of single token of activations. The attack strength K, which is the number of target tokens the attacker can control with single token of an attack activation, should therefore be xs log, V = dp. We assume X to be constant between models (although adversarial training probably changes it), and therefore our theory predicts that k = x Togaa v for fixed attack resistance X. For a fixed numerical precision and vocabulary size, the resulting scaling is k x d, i.e. the attack strength is directly proportional to the dimension of the activation vector (also called the residual stream), and we observe this empirically in e.g. Table 1. Having obtained empirically measured values of k, we can estimate the attack resistance X, which we do in Section 4. In the simplest setting, we would expect X = 1, which would mean that single dimension of the input controls a single dimension of the output. The specifics of the way input and output spaces map to each other are likely complex and the reason for why x= 1.
dp
Comparison to token-level substitution attacks. As a comparison, we tried looking for token substitution attacks well, which are significantly less expressive. For those, the attacker can change the first a integer-valued tokens on the input (instead of their high-dimensional activations; the geometric of the input manifolds is discussed in e.g. (Fort et al., 2022)), trying to make the model
4

---

Published as a conference paper at ICLR 2025
produce the attacker attacker-specified f-token continuation before. If the geometric theory from Section holds, the prediction would be that the attack strength would go down by a factor proportional to the reduction in the dimension of the input space the attacker controls. Going from adp bits of control to a logg V bits corresponds to a dp/ log, V reduction, which means that the attack multiplier Keoken for the token substitution method is predicted
be related to the activation attack strength as Ntoken k log, V. For d 512 and V = 50000, the log; V = 2 x 10-3 3- Our results are shown in dp
dp
Section 4, specifically in Table addition, it is very likely that the full 16 bits of the activation dimensions are not used, and that instead we should be using Peffective < p.
METHOD
Problem setup. A language model takes sequence of integer-valued tokens S
Actiaation
tolen attack
S2,, S.), each drawn from a vo- cabulary of size V, E 10, V-1}, }}, and outputs the logits 2 over the vocabu- lary for the next-word prediction These are the unscaled probabilities that turn into probabilities as p softmax(2) over the vocabulary dimension. As de- scribed in Section 2, we primarily work with attacking the continuous model acti- vations rather than the integer-valued to- kens themselves.
Text Input
Embedding
taken prddcttions
Sampling
turtle
turtla
token
terget
Sinstster
Figure 2: A diagram showing the multi-token target prediction after an attack on a = 2 token activa- tions.
The tokens S are first passed through the embedding layer of the language model, producing a vector v of d dimensions per token, fbetore (These are the activations that the attacker can modify by adding a perturbation vector P.) After that, the activations pass through the rest of the language model, as faiter obtain the logits for the next-token prediction. The full language model mapping tokens to logits would be Satter (fbefore (S)) 2, we just decided to split the full function ʃ S two parts, exposing the activations for an explicit manipulation. The split need not happen after the embedding layer, but rather after L transformer layers of the model itself. While we ran exploratory experiments with splitting later in the model, we nonetheless performed all our detailed experiments with activations directly after the embedding layer.
2, to
into the
An attack on activations. The attacker controls the activation perturbation P that gets added to the vector v in order to modify the next-token logits towards the token desired by the attacker. The modified logits are z' = fafter [ (5) P), where P. the attack vector, is of the shape P E R (the same as r), however, we allow the attacker to control only the activations of the first S. tokens. This is an arbitrary choice and other variants could be experimented with. The attack comprises the first a tokens, leaving the remaining s - a tokens separating the attack from its target. We experiment with the effect of this separation in Section G.
While the attacker controls the activations of the first a tokens in the context, the model as described so far deals with affecting the prediction of the next token immediately after the tokens in the context. As discussed in Section 2, we want to predict (-token continuations instead.
Loss evaluation and optimization. To evaluate the loss L(S, T, P) of the attack P on the context S towards the target multi-token prediction T, we compute the standard language modeling cross- entropy loss, with the slight modification of adding the perturbation vector P to the activations after the embedding layer. The algorithm is shown in Figure 8.
To find adversarial attack, first choose a fixed, randomly sampled context of size S, define the attack length (in tokens) a, choose target length / (in tokens), and a random string of tokens as the target sequence T. We then use the Adam optimizer (Kingma & Ba, 2014) and the gradient of the loss specified in Figure 8 with respect to the attack vector, P, as g = @C(S.T.P) Using the gradient directly is the same technique used in the original Szegedy et al. (2013), however, small modifications, such as keeping just the gradient signs (Goodfellow et al., 2015) are readily available as well. Decreasing the language modeling loss L by changing the activation attack P translates into
BP
5

---

Published as a conference paper at ICLR 2025
making the model more likely to predict the desired t-token continuation T after the context S by changing the activations of the first a tokens. We stop the experiment either 1) after a predetermined number of optimization steps, or 2) once the t-token target continuation T is the argmax sampled continuation of the context S, by which we define a successful attack.
Estimating the attack multiplier K. Our goal is to empirically measure under what conditions adversarial examples are generally possible and easy to find. We use random tokens sampled uni- formly both for the context S well the targets T to ensure fairness. For fixed attack length and a context size 8 2 a, we sweep over target lengths i in a range from I to typically over 1000 in logarithmic increments. For each fixed (a, t). we repeat an experiment where we generate random context tokens S, and random target tokens T. and run the optimization at learning rate 10- for 300 steps. The success of each run is the fraction of the continuation tokens T that are correctly predicted using argmax as the sampling method. This gives us, for a specific context length s, at- tack length a, and a target length an estimate of the attack success probability p(a, The plot of this probability can be seen in c.g. Figure 3a and Table I refers to the appropriate figure for each model. The p(a, t) are our main experimental result and we empirically estimate them for a range of language models and context sizes
For short target token sequences, the probability of a successful attack is high, and for long target sequences, the attacker is not able to control the model output sufficiently, resulting in a low probability. To p(a,t) t) we fit a sigmoid curve of the form at, at (8) =
(1 + exp (-a (log(t) - (og((rnas)) and read-off the best fit value of mmax, for which the success rate of the attack of length falls to 50%. In our scaling laws, we work with these values of tmax. however, the 50% threshold is arbitrary and can be chosen differently.
Empirically, the read-off value of the 50% attack success threshold (mxx depends linearly on the number of attack tokens whose activations the attacker can modify. The linearity of the relationship can be seen in e.g. Figure 6. As described in Section 2, we call the constant of proportionality the attack multiplier K, and relates the attack and target lengths tmax = ka.
We empirically observe that the attack multiplier & depends linearly on the dimension of the model activations used even across different models. We also theoretically expect this Section 2. The attack resistance X. defined in Equation 2, can be calculated from the estimated attack multiplier with the knowledge of the model activation numerical precision p (in our case 16 bits all cases), activation dimension d (varied from 512 to 2560), and vocabulary size V (around 50,000 for all dp
experiments) X
x klog. V. We provide these estimates in Table 1.
In Fort (2023) adversarial multi-attacks are defined and described. They are attacks in which single adversarial perturbation P is able to convert n inputs into n attacker-chosen classes. We ran a similar experiment where single adversarial perturbation P' can make the context SJ complete as Ti, S2 as T2. and all the way to S,, to Tr. This effectively decreases the attack length by a factor of n, or equivalently increases the target length by the same factor, as discussed on dimensional grounds in Section 2. Practically, we accumulate gradients over the n (S, T) pairs before taking an optimization step on P. We study multi-attacks from n (the standard attack) to n 8.
Another modification described in Section 2 is to use only a random, fixed fraction f of the dimen- sions of the activation vector P. Its effect is to change the effective attack length from a to fa, and we choose the mask uniformly at random.
Token substitution attacks. To compare the attack on activations to the more standard attack on the input tokens themselves, we used a greedy, token-by-token, exhaustive search over all attack tokens a at the beginning of the context S. For a randomly chosen context S, an attack length a S [S], and a randomly chosen sequence of target tokens T, we followed the algorithm in Figure 12 to find the first a tokens of the context that maximize the probability of the continuation T. By greedily searching over all V possible tokens, each attack token We repeat this experiment over different values of the attack length a, and random contexts S and targets T of length
time, we can guarantee convergence in aV steps.
[7'), obtaining a similar Ptoken (a, t) curve as for the activation attacks. We fit the Eq. 3 sigmoid to it, extracting an equivalent attack multiplier Kroken, characterizing how many tokens on the output can single token on the input influence (the result is, unlike for the activation attacks, much smaller than 1, of course).

---

Published as a conference paper at ICLR 2025
Attack and target separation within the context. The further the attack is from the target, the less effective it might be. We therefore experiment with different sizes of the context S that separate the first a token activations of the attack from the target tokens after S. We estimate an attack multiplier for each 8, getting a k(s) curve that we show in Figure 3c. The attack multiplier x. looks constant up to point (around 100 tokens of context) and then decreases linearly in log(s). We therefore fit. a simple function of this form to our data in Figure 3c.
4 RESULTS AND DISCUSSION
We have been using the EleuthereI/ppyt..... 2023) based on the GPT-NeoX library (Andonian et al., 2021; Black et al.. 2022) from Hugging (Li et al., 2023). Finally, we Tinnyttorrie presented in Eldan & Li (2023). We
series of Large Language Models (Biderman et al.,
Face!. A second suite of models we used is the mi crosoottthhiit used a single checkpoint of ronene 1 ran our experiments on a single A100 GPU on a Google Colab.
For finding the adversarial attacks on activations, we used the Adam optimizer (Kingma & Ba, 2017) at a learning rate of 10-1 for 300 optimization steps, unless explicitly stated otherwise. Our activations were all in the float1s 6 format, and the model vocabulary sizes were all very close V ~ 50, 000. For the input context as well as our (multi-)token target sequences, we sampled random tokens from the vocabulary uniformly. When using only subset of the activations, as described in Section 3, we choose the dimensions at random uniformly.
Varring
Beuther gyth
1.0-
activation stark
1053
Expermmen
Exporimant
25.0
24.3:0.7
Expasimens
=-119.0
0.8-
.300
Fraction of succccafd
0.6-
Number targee
1:00
0.4-
Tau KAt
12.0
355
0.2
0.0
100
302
103
Attbok longth asstuutaniouus attacksn
05
(c) The effect of tokens separat- ing the attack and the target. Up 100 tokens of separation, the attack multiplier (strength) does not diminish.
Tokens atporating attack target
Number
(b) The maximum number of successfully converted target to- kens 1max as function of the the attack length a.
(a) Probability of attack success fune- tion of the number of target tokens t for different lengths of the attack a.
Figure 3: A summary of adversarial attacks activations of EleutherAl/pythiz: 1..b-v0. Only experiments varying the attack length a (in tokens whose activations the attacker controls) and the multiplicity of context and target pairs the attack has to succeed on, n, are shown. The estimated attack multiplier is x = 119.0 + 2.9 which means that controlling a single token worth of activations on the input allows the attacker to determine 119 tokens on the output.
Attacks on activations. We ran adversarial attacks on model activations right after the embedding layer for a suite of models, a range of attack lengths a, target token lengths f, and multiple repetitions of each experimental setup (with different random tokens of context S and target T each time), obtaining an empirical probability of attack success p(a, L) for each setting. For multiple repetitions, we also had a standard deviation of p at each set of (a, t) values. To get to lower effective values of and therefore weaker attacks, we use the multi-attack strategy described in Section 3 and in Fort (2023), designing the same adversarial attack for up to n = sequences and targets at once.
experiments on 4b-v0, a 1.4B model with activation dimension d = 2048. In Figure 3a the success rates of attacks p for different values of attack length (in tokens whose acti- vations the attacker controls), target length (in predicted output tokens), and the attack multiplicity
Figure 3 shows an example of the results of SacherrAI/yyhiia..
7

---

Published as a conference paper at ICLR 2025
Table 1: summary of attack multipliers R estimated from experiments for activation adversarial attacks for various language models. d/k. is the number of dimensions of an activation needed to control single output token, while X is the attack resistance (defined in Theory 3) which corre- sponds to the number of typical bits on the input the attacker has control in order to have single bit of control on the output. The pythia-* models are from EleutherAI, Phi-* from Microsoft and TinyStories-* from roneneldan.
Model
Curves
Size
Dimension d
[V]
Attack multiplier
Per dim multiplier d/
Attack
resistance X
pythia-70m pythia- 19thia-160m
Fig
70M
512
50304
24.2 = 0,8
21.2 + 0.7
21.7 + 0.7
Fig
160M
768
50304
36.2 2.0
21.2 1.2
21.7 = 1.2
pyltia--10m-ddduupe
Fig 9
410M
1024
50304
70.1 2.3
14.6 0.5
15.0 0.5
pythia-I.4b-W
Fig
1.4B
2048
50304
128.4
16.0 0.5
16.3 0.5
2ythia-2.8b-v0
Fig 10
2.8B
2560
50304
104.9 +.9 2.2
24.4 + 0.5
25.0 + 0.5
Phi-1
Fig 11
1.3B
2048
50120
42.8
47.9 2.6
49.0
Phi-1.5
Fig 12
1.3B
2048
50120
78.4
26.1 = 1.7
26.8 + 1.7
TinyStorics-33
Fig 13
33M
768
50257
27.2 = 2.2
28.2 + 2.3
28.9 + 2.3
(how many attacks at once the same perturbation P has to succeed on simultaneously). The higher the attack length a, the more powerful the attack and the longer the target sequence t that can be controlled by it. We fit a sigmoid from Eq. to each curve to estimate the maximum target sequence length, fmax. which the success rate of the attack drops to 50% (an arbitrary value).
In Figure 3b, we plot these max maximum target lengths function of the attack strength Since the multi- attack n allows us to effectively go below a = 1, we actually plot a/n, the effective attack strength. Fitting our scaling law from Eq. justified on geometric and di- mensional grounds in Section 2, estimate the attack multiplier for El GleutherAI /pyttipattiiaiae 1.w.11.bvvee 4b-v0 to be
EleutherAllpythia-e
1.0
11..720.2
Fraction of successful attacks
16.7
ara 33.4e1..
0.8
0.6
119.2 + 2.9, implying that by controlling single token worth of activations at the beginning of a context,
0.4-
the attacker can determine 119 tokens exactly on the output. The results shown in Figure 3 only include exper- and n and with varying n = 1, 2,,48 8. Varying
iments with attack lengths a 1, 2, attack length n allows us to go to "sub"-toker" levels of attack strength.
0.0
2
16
32
Number of attack tokens Figure Greedy search over all at- tack tokens get specific target com- pletions. In general, & 8 tokens worth of attack are needed to force single particular token of response.
Model comparison. For number of different models, we show the scaling laws attack strength a (in to- kens whose activations the attacker get length & (in tokens) in Figure For each model, we estimate the attack multiplier (the number of target to- kens the attacker can control with a 50% success rate by attacking the activation of single token on the input), and compute their attack resistance X. as defined in Eq. 3, corresponding to the number of bits the attacker needs to control on the input in order control a single bit on the output. We summarize these results in Table 1.
control) vs tar-
An interesting observation is that while the model trainable parameter counts span two orders of magnitude (from 33M to 2.8B), and their activation dimensions range from 512 to 2560. the result- ing relative attack multipliers k/d, and the attack resistances x stay surprisingly constant. This is supporting evidence for our geometric view of the adversarial attack theory presented in Section 2. The conclusion is that for the Ele EleutherAI stythia-+ model family, we need between 15 and 2 25 bits controlled by the attacker on the model input (the activations of the context) order to control in detail the outcome of a single bit the model output (the argmax predictions from the model). In an ideal scenario, where a single dimension bit on the input could influence a single dimension / bit the output, each activation dimension would be able to control single token the output since the model activations are typically 16 bits and to determine a single token we also need log, (V) xs 16 bits. However, since need X bits the input affect the output,
8

---

Published as a conference paper at ICLR 2025
this means that need X activation dimensions to force the model to predict token exactly want. This is still a remarkable strong level of control, albeit weaker than one might naively expect.
Replacing tokens directly. To compare the effect of attacking the activation vectors and attacking the input tokens directly by replacing them, we ran experiments on the EleutherAI/py patpsAAyypthaaaaa/ Om model. For a ran- domly chosen context of = 40 tokens and randomly chosen I-token target tokens (t = 1,2,4 4 in our exper- iments) we greedily and per-token-exhaustivee
Adversarial
LLMs activations
10
Attack resitancs 1
Output space dimension (binary)
10
Aftact reuntance 2 20
Activation sttacks
0.8
Toten attada
2015
Attack success rate
Possible principle
Impoosiible
2050
Emple amplrically gxis
0.6
search
1013
over the replacements of the first a tokens of the context in order to make the model predict the desired I-token sequence as a continuation. The method is described in Section detail.
1020
0.2
10ta
0.0
1020 10*3 10 1045 10a0 1023
Unlike for activation attacks, the token replacement at- tack needs more than one token on the input to influence single token the output. The detailed curves showing the attack success rate p(a, t) as a function of the attack length (the number of tokens the attacker can replace by other tokens) and the target length l are shown Fig- ure together with Eq. fits to them to extract the Omiu- the minimum attack length to force the prediction of a t- token sequence. For f 1, we get Gmin
Input space dimension (binary)
Figure 5: Input-output space dimen- sion comparison for successfully real- ized adversarial attacks. The regions edge corresponds well to the din xx dout implication of the dimensional hypoth- esis.
11.7 + 0.2,
while for ( = 2 aman = 16.7 + 0.2 and for 1 = 4 Omin 33.4 + 1.1. Extracting the attack multiplier Nroken t/Omin: we get 0.085 + 0.001, 0.120 = 0.002 and 0.120 0.004. Averaging the three estimates while weighting them using the squares of their errors, we get Kroken z 0.12. That means that we need 1/Stoken Rs 8 tokens to control on the input in order to force the prediction of a single token on the output by token replacement (compared to s 0.04 tokens worth for the activations attack in Table 1).
Controlling tokens instead of activations offers the attacker greatly diminished dimensionality of the space they can realize the attack in. In Section 2, we discuss the comparison between the dimensionality of the token attack (logg(V)a bits of control for an token attack), com- pared to the activations attack (adp bits of control for a precision E. EleutherAT htt.A//yythaaaaa a- in particular). Our geometric theory predicts that their attack mul- tipliers should be in the same ratio as the dimensions of the spaces the attacker controls. In this particular case, the theory predicts Kroken /n (Vg) R 2 10-3, The actual experimen- tally estimated values give us Ktoken/ /k se 5 x 10 Given how simple our theory is and how different the activation vs token attacks are, we find the empirical result to match the prediction surprisingly well (it is better than an order of magnitude match).
= 16 and d = 512 for
If we calculated the attack resistance Xtoken for the token attack, we would get directly Xtoken 1/Nooken 8.3. That means that z 8 bits of control are needed by the attacker on the input to control a single bit of the model predictions. This shows that the token attack is much more efficient than the activations attack (x R 22 for this model), which makes sense given that the model was trained to predict tokens after receiving tokens on the input rather than arbitrary activation vectors not corresponding to anything seen during training.
Input-output dimension summary We summarize all our successfully realized attacks in Figure 5. The edge of the region where activation attacks are successful corresponds well to din x dout with an attack resistance (constant of proportionality) of x z 20. Token- attacks, while shifted to a lower x. have similar shape. Notable, all successful attacks lie well below the theoretical boundary of x = 1 where din = doat and above which attacks should generically not be possible.
CONCLUSION
Our research presents detailed empirical investigation of adversarial attacks on language model activations, demonstrating a significant vulnerability that exists within their structure. By targeting a very small amount of language model activations that can be hidden deep within the context window

---

Published as a conference paper at ICLR 2025
Table 2: Comparing activation attacks and token attacks on EleutherA//pythia-00m The activation attack has a much higher attack multiplier x = 24.2 + 0.8 compared to the one for token attack of Rtoken R 0.12. While a single token's 's activation can therefore determine 20 output tokens, the attacker needs >> 8 input tokens to control a single one by token replacement. Our experiments show that they are factor of 2.6x from each other, which a close match given the simplicity of our theory and the vastly different nature of the two attack types.
Model
Attack
Curves
Size
Dimension
M
Attack multiplier 6s
Attack resistance X
type
pythia-70m
Tokens
Fig 4
70M
512
50304
0.12
8.3
Activations
Fig
70M
512
50304
24.2 + 0.8 21.7 0.7
and well separated from their intended effect, we have shown that it is possible for attacker to precisely control up to O(100) subsequent predicted tokens down to the specific token IDs being sampled (for both randomly sampled and natural language token sequences). The general method is illustrated in Figure I.
We empirically measure the amount of target tokens mmx attacker can control by modifying the activations of the first a tokens in the context window, and find a simple scaling law of the form mmax(a) = ka, where we call the model-specific constant of proportionality K the attack multiplier. We conduct a range of experiments attack multipliers ri, summarized in Figure 6 and Table 1.
models from 33M to 2.8B parameters and measure their
We connect these to simple geometric theory that attributes adversarial vulnerability to the mis- match between the dimension of the input space the attacker controls and the output space the attacker would like to influence. This theory predicts a linear dependence between the critical input and output space dimensions for which adversarial attacks stop being possible, which is what we see empirically in Figure 6. By using language models controllable test-bed for studying regimes of various input dimensions (controlled by the attack length a) and output dimension (controlled by the target sequence length f), we were able to explore parameter ranges that were not previously accessi- ble in computer vision experiments where the study of adversarial examples was historically rooted, and sweep a full input dimension output dimension space of model configurations, as shown in Figure 5.
We find that empirically, the attack multiplier x. seems to depend linearly on the model activation dimension d rather than the model parameters, as predicted by our geometrical theory. k/ d, and as shown in Table surprisingly constant between 16.0 and 2..4 for the EleutherAl/pythia model suite spanning model sizes from 70M to 2.8B parameters.
We compare the activation attacks to the more standard token substitution attacks in which the attacker can replace the first a tokens of the context in order to make the model predict a specific t-token sequence. Figure and Table 2 we summarize our results and show that despite the token attack heing much weaker (on our 70M model) with an attack multiplier of Staken R 0.12 (meaning that we need 8 tokens of input control to make the model predict a single output token) compared to the activation attack = 24.2, the two vastly different attack types have very similar attack resistance x when accounting for the vastly different dimensions of the input space of tokens and activations. The theoretically predicted K//Keoken = 2 10 3 is surprisingly close to the empirically measured 5 x 10 despite the arguable simplicity of the theory.
To make sure findings robust to a separation between the attack tokens and the target tokens, we experiment with adding up to (1000) randomly sampled tokens between the attack and the target in Figure 3c. We find that the attack strength essentially unaffected up 100 tokens of separation with a logarithmic decline after. However, even a full context of separation gives a high degree of control of the very first token activation over the next-token prediction.
Attacking activations might seem impractical since the majority of language models, especially the commercial ones, allow users to interact with them only via token inputs. However, an increasing at- tack surface due to multi- model models, where other modalities are added to the activations directly, as well as some retrieval models, where retrieved documents are mixed in likewise as activations rather than tokens, directly justify the practical relevance of this paper.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Dya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 16. ACM, October 2016. doi: 2978318. 0111445/977644.29788811 URL http: /x doi. ott/1/..g1.... 1ee.l111112/771
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. 2022.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh KrameR. Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shiv- anshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Au- toregressive Language Modeling in PyTorch, 8 2021. URL tttpps//vwee gi github. com/ eleutherai/gee pptpaaaiippteeeeeeeeeeee
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Haffield-Dodds, Danny Hernan- dez, Tristan Hume. Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sid Black, Stella Biderman. Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang. Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Work- shop Challenges & Perspectives in Creating Large Language Models, 2022. URL https: //arxiv.org. abs 2200 06745.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean- San-tttt Lespiau, Bogdan Damoe, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. 2022.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaél Segerie, Micah Carroll, Andi Peng. Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Mence Open problems and funda- mental limitations of reinforcement learning from human feedback, 2023.
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette- Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramèr. Privacy side channels in machine learn- ing systems, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.

---

Published as a conference paper at ICLR 2025
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/taansfommercccciitss /ttp://22///aaaaoo//oiiii/ htmll.
Stanislav Fort. Adversarial examples for the openai clip in its zero-shot classification regime and their semantic generalization, Jan 2021a. URL https::tp///cc.cct /tatttp::/statiie sttnistiisssss
wwwttannssllor....mmoo slavfort...... htt.raaCCC___eeerraaaaaeeeeeeeee www.aavvveriia... 1021/01/12/0pentt ott.:////.../eeeaa.....caaaaaaaa
i github...h........g. sla.fort.githhbb .tmll
Stanislav Fort. Pixels still beat text: Attacking the openai clip model with text patches and adver- sarial pixel perturbations, March 2021b. URL https:://ttalsslaffrrrs wit.ffrrrt.tttt.......o github. ioo html.
Stanislav Fort. Multi-attacks: Many images the same adversarial attack many target labels, 2023.
Stanislav Fort and Balaji Lakshminarayanan. Ensemble everything everywhere: Multi-scale aggre- gation for adversarial robustness, 2024. URL htt.t://///// /twxiaiivvvorree orgoorrgg/so.. abs/2408 05446.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of ditto--distribution detection, arXiv preprint arXiv:2/06003000 2021.
Stanislav Fort, Ekin Dogus Cubuk, Surya Ganguli, and Samuel S. Schoenholz. What does deep neural network confidently perceive? the effective dimension of high certainty class manifolds and their low confidence boundaries, 2022.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connecttdd foundation models, 2023.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023.
Suzana Hluuzl. The remarkable, yet not extraordinary, human brain scaled-up primate brain and its associated cost. Proceedings of the National Academy of Sciences, 109 2012. doi: 10.10733pnas. 11..11///paa..11115111000 1201895109. URL https: //www. 1) 1073/pnas...... 1201895109, pnas.org /oii
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Erie Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019.
Can Kabadayi, Lucy A Taylor, Auguste MP von Bayern, and Mathias Osvath. Ravens, new caledo- nian crows and jackdaws parallel great apes in motor self--egulation despite smaller brains. Royal Society Open Science, 3(4):160104 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
12

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). a. URL http:t/:/ www. cs toronto. edu/ Far.atrr
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL http: www. cs www.ttoootoo....m edu/ ficzu//kizii wickzi/ziifaa ccifar html.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Tal Ridnik, Emanuel Ben-Baruch Asaf Noy, and Lihi Zelnik-Manor. Imagenet-2IK pretraining for the masses, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2013.
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Ac- tivation addition: Stecring language models without optimization, 2023.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: top-down ap- proach to ai transparency, 2023a.
Andy Zou, Zifan Wang. J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023b.
13

---

Published as a conference paper at ICLR 2025
A ATTACK ALGORITHM
Algorithm 1 Computing loss for activation attack P towards I-token target sequence
Given activation dimension d, tokens of context of length s = [S], attack length (a S 8), target tokens 7 of length i, and an attack vector P Raxa
2:
Compute the activations of the context followed by the target u = Jistooe (S S + T) E R(++)xd Add the perturbation vector to the first a tokens of v' u. v'[: al+ {This is the attack)
4: for to do
5: Predict logits after the + tokens of the context and target Satter [utter(v'l: i))
6: Compute the cross-entropy loss & between these logits 2; and the target token T':
7: end for
8: Get the total loss for the t-token prediction as El
B SCALING LAWS FOR VARIOUS MODELS
Bescherk gyet T-m
Besmeniooy 160m
Bauzherk 9yc-a-420m
Experment
L0C4
Theary 2-242
212 22
Theary w-1
Sue matae
16.
Mbact light unaadeateoeeen
Stsc orght atataco
Abseclargh atace sooren
Jtsic pescclarghhoo Ertacta storce
(a) Pythia 70M
(b) Pythia 160M
(c) Pythia 410M
(d) Pythia 1.4B
Beutherd Byutherdooybbbb oy 2
X0E
Theayy reeeeiinn Re FoM
Inpestmant us Theary Mimsrrgmmn1
Depertent su Thearr
ofalt
Dractmen
Experttene POC
cxTx
Xtack leggh wataaaaaaaeeee
Aback lnggiaaa ataaa anoen
(e) Pythia 2,8B
(f) TinyStories 33M
Microsoft Phi-1
(h) Microsoft Phi-1.5
Figure 6: A summary of experiments determine scaling laws for different models and to read off their attack multiplier k. The individual plots show the results of fits to the success rate attacking language model by modifying its activations towards the target of generating t tokens function of a token activations of an attack. Our theory predicts linear dependence in each plot, while the slope is (the attack multiplier) is model-specific constant.
C DETAILED ATTACK SUCCESS RATE CURVES FOR DIFFERENT MODELS
D EXHAUSTIVE ATTACKS ON INPUT TOKENS
E DETAILED EXPERIMENTAL RESULTS
F USING ONLY A FRACTION OF DIMENSIONS
Another way of modifying the attack strength is to choose only a fraction f of the activation dimen- sions the attacker controls. Our geometric theory descrihed in Section 2 suggests that the effective
14

---

Published as a conference paper at ICLR 2025
LLM activation attack
EleutherAlppythia/000
1.0
Fraction successful attacks
0.8
1f7.1187.11
1-1a=4f=10
11m--103. 103.2
0=1a=2 fm10
0.4
0=1s=1f=10 24.6
0=2a-1f-1.0 11.9 11.9
6=4 1-10
- -m.
0.2
0.0
10
10a Number of target tokens
10a
103
Figure 7: Fraction of successfully realized attack tokens as a function of the number of target tokens t for different numbers of simultaneous attacks n and the length of the attack a for FleutherA//pythia- 70m.
LLM activation attack EleutherAUpyttha-1660
1.0
Fraction successful attacks
0.8
R-1
a-8 1-1.0
f=10 0 1
0.6
f=
n=1 f= -1
0.4
n=2
a=
f=100
n=8
a=1
1=10
0.2
10"
10 Number of target tokens
10a
103
Figure 8: Fraction of successfully realized attack tokens a function of the number of target tokens t for different numbers of simultancous attacks n and the length of the attack a for EleutherAl/pythia- 160m.
attack strength depends the product of the attack length a, the fraction f and of the attack mul- tiplicity as 1 1/n. Therefore we should be able to vary (f, a, n.) as we wish and the attack strength should only depend on fa/n. n. In Figure 14
comparison of experiments the same attack strength performed at different combinations of a, and f. In total 12 experimental setups are shown: 1) n and f 1, while varying
1,2, 4, 8, 2) a 3) n 1, = 1, and f = 1/8, 1/4, 1/2, and 4) n 8, = 8, and f 1/2. All of these lie on the theoretical predicted scaling law in Eq. fmax = xxa/n. a/n. The estimated attack multiplier 128.4 4 + 3.7 is well within 20 of the estimate using the varying a and n alone in Figure 3.
f and n 2, 4, 8,
G CONTEXT SEPARATING THE ATTACK AND THE TARGET TOKENS
In our experiments, attack the activations of the first tokens of the context of length (a 8) in order to make the model predict an arbitrary t-token sequence tokens as its argmax con- tinuation. In our most standard experiments activations the full context which of exactly the same length as the attack. These are the ex- periments in Figure summary Figure and the summary Table 1. To the effect of having
which means that the attacker controls the
15

---

Published as a conference paper at ICLR 2025
LLM activation attack
EleutherAUpythhaa-400
1.0
Fraction successful attacks
0.8
n=1 1:1==ff11 f=10 -lay 523.0
0.4
b=1a=4f=10 -0
f=100 >lmss 159.1
0=1a=1f=10 -6.7
0.2
0=2 +la -3 37.4
0=4a=1fe10- 18.0
0.0
f-1.0
7.7
10
10
10a
10a
Number of target tokens
Figure 9: Fraction of successfully realized attack tokens as a function of the number of target tokens t for different numbers of simultaneous attacks n and the length of the attack a for Fleuther 410m.
LLM activation attack EleutherAl- pythla- 2.8b-v0
Fraction successful attacks
0.8
0.6
0.4
0-1 11-4 f-1.0
-451
0=1a=2 f-1.0 242.
B-12-1 f-10 -fm
n-2
n=4a=1fe10
=2n-3362
0.0
n=8
22.
10a
10
10a
10a
Number of target tokens
Fraction of successfully realized attack tokens as a function of the number of tar- get tokens / for different numbers of simultaneous attacks and the length of the attack for EleutherAU/pythaa22.88
Figure 10:
context tokens separating the attack from the target tokens, we ran an experiment with fixed model eleutheeAA/pythhe-e and read off the attack multiplier for each context length
logarithmically spaced range from I to 2000 (almost the full context window size). Estimating each <(s) involved exploring range of target lengths a and attack multipliers n, each in turn being a 300 step optimization of the attack, as described in Section 4. The results are shown in Figure 3c. We see that for up to 100 tokens of separation between the attack and the target tokens, there is no visible drop in the attack multiplier In other words, the attack is equally effective at forcing token predictions immediately after its own tokens or 100 tokens down the line. After the context length of 100
linear drop in x(s) with log of the context length. At 2000 tokens of random context separating the attack and the target, we still see k Rs i.e. a single token's 's activations on the input controlling tokens on the output.
16

---

Published as a conference paper at ICLR 2025
LLM activation attack
Microsoft/Phi-1
1.0
Fraction successful attacks
0.8
0.6
0.4
nel and fu1.0 - mheal 204
dml am2 fw1. -texe 77.3
0.2
1+1. 32.7
fe1.0 21.3
0.0
10
101
102
10a
Number target tokens
Figure 11: Fraction of successfully realized attack tokens as function of the number of target tokens t for different numbers of simultancous attacks and the length of the attack for Microsoft/Phi- 1.
LLM activation attack Microsoft/Phi-1.
1.0
Fraction successful attacks
0.8
0.6
0.4
0.2
ta anl fel.0 heur 21.0 nwm and fel. 1
0.0
10
10 Number of target tokens
10
10a
Figure 12: Fraction of successfully realized attack tokens as function of the number of target tokens t for different numbers of simultaneous attacks and the length of the attack a for MicrosoftPPi- 1.5.
LLM activation attack roneneldan/TinnStoriis-
1.0
me1 -105
Praction successful attacks
nel ged (w1.0 f118
ne1 and (e1.0 -hea -50.5
0.8
nel and fell0 -fe 225
0w2 gel fw1.0 -162-1465
and and f=1.0
fel.0
0.6
0.4
0.2
0.0
10a
102
10a
Number of target tokens
Figure 13: Fraction of successfully realized attack tokens as a function of the number of tar- get tokens t for different numbers of simultaneous attacks and the length of the attack for AmmeneldanTiinytoriissse.
17

---

Published as a conference paper at ICLR 2025
Algorithm Greedy, exhaustive token attack towards a t-token target sequence
1: Given tokens of context S of length [.S], attack length (a I s), target tokens T of length
1, and a vocabulary of size V
2: The current context sequence starts at S S
3: for = 0 to a do
4: {Looping
attack tokens}
5: for r 0 to V do
6:
(Looping over all possible single tokens]
7:
= ['
{Trying the new token]
8:
Calculate the language modeling loss C, for the target continuation T' after the context S
it lower than the best so far, keep the Tbest T.
9: end for
10:
Update the ith attack token to the best, greedily, as [ {Greedy sampling token by
token. Each token searched exhaustively.)
11: end for
12: If the newly updated context S" produces the continuation as the result, the attack is success-
ful.
Experiment vs Theory EleutherAl/pythia-l.co 1.4b-v0
Experiment, vary Experiment, vary Experiment, vary Experiment, vary all Theory
Number of target tokens fmax
300
100
mmx Kfa/n
30
1/4
Frac of dims on fx Attack length a
Figure 14: A scaling plot showing successful attacks on Pythia-La 1,4B for different attack lengths a, fraction of dimensions and attack multiplicities
18


Reviews:
[{"review_id": "fd9IftOCAW", "reviewer": "Reviewer_wHmg", "summary": "The paper proposes an adversarial attack on the activations of LLMs and studies scaling laws for the attack. The results show that by perturbing a small set of model activations, prediction tokens can be controlled. A linear scaling law between perturbed activation tokens and predicted tokens is empirically validated. This threat model is claimed to practical in retrieval tasks and in certain multi-modal models.", "questions": "1. Is there any supporting evidence in literature for \"The core hypothesis is that the ability to carry a successful adversarial attack depends on the ratio between the dimensions of the input and output spaces.\"\n\n2. Typos (Line 262, 267 ,315) the reference to Algorithms are incorrect.\n\n3. (Line 499) It's incorrect use of the Big-O notation. O(100) is equivalent to any O(k) where k is a constant. So, it's best to rephrase this line.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The scaling law study for adversarial attacks on language model activations allows exploration in input-output regimes much larger than previously studied computer vision models.", "weaknesses": "As mentioned in Line 268, the attacks used are close to iterative gradient attacks or FGSM which were proposed early in the computer vision literature. Is it comprehensive enough to do this study limited to these attack models. Also, is it common to keep the perturbations unbounded for LLM attacks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T8XnLA4vQ6", "reviewer": "Reviewer_HSZG", "summary": "This paper studies the technique of adversarial attacking the activation value of LLM. This work explains the LLM's vulnerability and finds that attacks satisfy scaling law.", "questions": "1. The utility of LLM activation values to attacks needs to be fully explained. Only two works discussing modifying activation values are presented in the introduction of this paper. And the work of the retrieval and multimodality seems to be to concatenate activations from different inputs, rather than to add a certain perturbation of activations, as is the case against attacks.\n\n2. Is the attacker a user? How can an attacker inject generated adversarial perturbations into benign user-generated activation values?\n\n3. Does an attacker need to access the entire LLM model to calculate adversarial perturbations? The LLM model is usually provided to the user in the form of an API, and the attacker cannot obtain the model weights.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problems studied are important because LLM has been applied to our life. The proposed attack method can achieve good results and shows some inherent characteristics of LLM itself.", "weaknesses": "The practicality of the proposed scheme is questionable, and whether this attack scheme can be implemented in reality needs further consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n4opLqXCyX", "reviewer": "Reviewer_xEdG", "summary": "The paper considered activation-level rather than token-level adversarial attacks on LLMs and derives scaling laws between the number of output tokens an attacker can affect vs the number of input activations they can control. The paper also explores token-level substitution adversarial attacks in comparison to their novel approach.\n\nThe paper justifies the practicality of their activation-level attacks by referencing retrieval and multi-modal models. It concludes by drawing general implications of this work for the adversarial ML community, namely presenting language models as a more flexible environment for exploring the theory of adversarial attacks compared to vision models.", "questions": "* Is χ related to concepts such as generalised degrees of freedom (https://auai.org/uai2016/proceedings/papers/257.pdf Gao and Jojic 2016) or effective dimensionality (https://arxiv.org/abs/2003.02139 Maddox et al 2020)? It feels like a similar concept\n* Worth exploring how adversarial training affects χ \n* Would make for an interesting separate research paper to explore the effective dimensionality of the activation dimensions and maybe how that could be used for compression. This is in reference to L222-223.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\n * As far as I can tell this is the first paper to take an in-depth look at activation-level adversarial attacks\n * The comparison of token vs activation level attacks isn’t surprising due to the significantly higher level of granular control you can exhibit on the activation level but good to have it quantified\n\nQuality:\n * Clear writing style, thought process and hypothesis clearly stated and explained\n\nClarity:\n * Core idea and experiments clearly presented and explained.\n\nSignificance:\n * Strong practical justification of the real-world applicability of the research through reference to retrieval and multi-modal setups.\n * Presents a compelling case for using LLMs and activation-level attacks as a test bed", "weaknesses": "* Presentation and explanation of the theory of the attack strength could be better – it’s currently a large hard-to-read paragraph while references are repeatedly made to the empirical vs theoretical scaling laws. E.g., breaking down the working of the final equation.\n* The narrative of the paper feels a bit non-linear – I had to skip ahead or read back multiple times to reference material presented in the paper to bits where it was relevant again. For example in L153 the significance of the log2 value is only appreciated after reading the scaling laws subsection.\n* Some of the more vague speculative statements such L204 “(although adversarial training probably changes it)” should probably be avoided and left to a “future research” section. Ditto L222-223 regarding the effective dimensionality of the embedding – this specifically would be very interesting to explore separately.\n* Some of the plots should be included as PDFs rather than images for fidelity’s sake. E.g. Figure 1(b,e), Figure 3, Figure 5. This is important as the graphs are dense and small for an A4 page. I would also recommend increasing the font size to improve readability.\n* This is a side note but the appendix appears to be incomplete with sections C, D, E appearing empty", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4W731IRomF", "reviewer": "Reviewer_1sJt", "summary": "The paper proposes a hypothesis: there is a \"scaling law\" between number of input tokens the attacker can modify and the number of prediction tokens the model outputs.  Empirical results suggest that under certain cases, the maximum number of target tokens that can be manipulated by the attacker, depends linearly on the number of input tokens that the attacker can control.", "questions": "My questions are aligned with the weakness part. \n\n+ Q1: Regarding W1,\n  - a) Can you specify exact number of ($S$, $T$) pairs sampled for each $t_{max}$ estimation? \n  - b) Provide error bars or confidence intervals for the $t_{max}$ values to show the distribution across samples.\n  - c) Include an analysis of how $t_{max}$ varies for different types of target sequences (e.g. random tokens vs natural language).\n  - d) Discuss how the variability in tmax estimates impacts the linearity of the scaling relationship.\n\n\n+ Q2: Regarding W2,\n  - a) Can you provide a more detailed analysis of realistic constraints on activation modifications in retrieval and multi-modal settings?\n  - b) Can you conduct experiments with bounded perturbations (e.g. within an ε-ball) to see how this affects the scaling law.\n  - c) Discuss more explicitly the limitations of your unconstrained attack model and its implications for real-world applicability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ S1: The research work is of great novelty and originality. The hypothesis proposed in the paper is interesting.\n\n+ S2: Empirical findings can provide support to the proposed hypothesis. Such scaling law can be observed on several different language models.", "weaknesses": "Although the hypothesis is interesting, there are many details not discussed clearly which hinders the readers from a deeper understanding. \n\n+ W1: **Formulation and measure of $t_{max}$ lacks clarity**.  $t_{max}$ is designed to present the ideal maximum tokens that attacker can manipulate in model prediction. However, how to practically obtain this amount may be unclear. From the paper describes a random sampling strategy to obtain context tokens $S$ and target tokens $T$. It raises questions about the impact of different samples on $t_{max}$ value. How many ($S$, $T$) pairs are sampled to determine $t_{max}$? What is the distribution of $t_{max}$ across various samples? Intuitively, the length of target tokens might extend to a larger value if they are more aligned with natural language priors or they are from (repeated) training samples, such as excerpts from famous poems. For random combination of tokens in $T$ and $S$, different samples may get varied attack performance. This is very important since changes to the value of $t_{max}$ may result in the linearity of relationship no longer holds. More detailed discussion on this manner would benefit readers' understanding. \n\n+ W2: **Practicality of the attack is questionable**.  One notable setting in this paper is the attacker's ability to modify the activations of token feature without limit (different from the classic adversarial attack setting where the attacker needs to constrain their perturbations within a $\\epsilon$-ball).  While authors provide examples involving retrieval and multi-modal models where an attacker might alter continuous feature values within the model, the feasibility of such unlimited modifications are questionable. For example, in multi-modal models, a more realistic attack surface would be the attacker altering an image within [0, 255] pixel range, rather than arbitrarily modifying image features. The impractical threat model may undermine its real-world relevance and impact of presented findings.\n\n\n+ W3: Typo and broken sentence\n    - a) Line 283, \"probability can be seen in e.g. Figure 3a and Table 1 refers to the ...\" ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)