Stealth edits to large language models
Authors: Oliver Sutton
Qinghua Zhou
Wei Wang
Desmond Higham
Alexander N. Gorban
Alexander Bastounis
Ivan Y Tyukin
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
stealth attacks
memory editing
Primary area: natural_language_processing
TL;DR: We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without requiring retraining.

Abstract: We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without requiring retraining. Our theoretical insights show that a single metric (a measure of the intrinsic dimension of the model's features) can be used to assess a model's editability and reveals its previously unrecognised susceptibility to malicious *stealth attacks*. This metric is fundamental to predicting the success of a variety of editing approaches, and reveals new bridges between disparate families of editing methods. We collectively refer to these as *stealth editing* methods, because they directly update a model's weights to specify its response to specific known hallucinating prompts without affecting other model behaviour. By carefully applying our theoretical insights, we are able to introduce a new *jet-pack* network block which is optimised for highly selective model editing, uses only standard network operations, and can be inserted into existing networks. We also reveal the vulnerability of language models to stealth attacks: a small change to a model's weights which fixes its response to a single attacker-chosen prompt. Stealth attacks are computationally simple, do not require access to or knowledge of the model's training data, and therefore represent a potent yet previously unrecognised threat to redistributed foundation models. Extensive experimental results illustrate and support our methods and their theoretical underpinnings. Demos and source code are available at https://github.com/qinghua-zhou/stealth-edits.

Stealth edits to large language models
Oliver J. Sutton" King's College London
Qinghua Zhou' King's College London
Wei Wang
University of Leicester
oliver. www.ssskctt..eeeee.. ac. ac.uu .k
qinghua maighua.zhoukk.l. zhou@kcl....mmmm ac uk
ww1520le.ach ac uk
Desmond J. Higham University of Edinburgh j.j. hi highhmmed... uk
Alexander N. Gorban University of Leicester
Alexander Bastounis
King's s College London
www.gorbnee.....m ac. .uk
al alexander bastounis@kcl. ac
Ivan Y. Tyukin King's College London ivan. tywkinkkkk........ ac
Abstract
We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without without requiring retraining. Our theoretical insights show that single metric (a measure of the intrinsic dimension the model's features) can be used to assess a model's editability and reveals its previously unrecognised susceptibility is fundamental to predicting the success of a variety of editing approaches, and reveals new bridges between disparate families of editing methods. We collectively refer to these as stealth editing methods, because they directly update a model's s weights specify its response specific known hallucinating prompts without affecting other model behaviour. By carefully applying our theoretical insights, we are able introduce new jet-pack network block which is optimised for highly selective model editing, uses only standard network operations, and can be inserted into existing networks. We also reveal the vulnerability of language models to stealth attacks: a small change to a model's weights which fixes its response single attacker-chosen prompt. Stealth attacks are computationally simple, do not require access to or knowledge of the model's training data, and therefore represent a potent yet previously unrecognised threat to redistributed foundation models. Extensive experimental results illustrate and support our methods and their theoretical underpinnings. Demos and source code are available at https::/ / thub. com/ sainghu--zhore setee//ssallllle th-edits.
malicious stealth attacks. This metric
Introduction
The latest meteoric rise of artificial intelligence has been driven by the maturing of large language models. These models, predominantly based on the transformer architecture [39], have demonstrated remarkable abilities in natural language communication and comprehension, which are only begin- ning to transform the world as we know today. Scaling has proved to be key in enabling these breakthroughs: the GPT-4 family of models, for instance, has on the order of 10 trained parameters, a figure which would have been inconceivable only a few years ago. The raw computational power and vast quantities of quality data required to train models such as these make developing them prohibitively expensive for all but a handful of the world's wealthiest technology companies [31].
'These authors contributed equally to the paper and are listed alphabetically.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This environment has also seen the rise of foundation models: collections of accessible or open source language models which have become an invaluable tool for those without the facilities to train their own from scratch. As language models have developed, 'hallucinations" - non non-factual or non-sensical information generated by the model - have become challenging barriers to trustworthy and reliable artificial intelligence. Much work has been invested in trying to understand the origins of hallucinations [7] and develop mechanisms to mitigate them [18], amplified by regulatory require- ments placed organisations deploying AI by the European Union's 's recent 'AI Act' [9] or the UN's resolution on 'safe, secure and trustworthy artificial intelligence' [38]. Recent work has, however, shown that hallucinations may in fact be an inevitable artefact of fixed language models [19, 43].
This motivates the key question of this paper: is it possible to surgically alter a model to correct specific known hallucinations in a granular, individually-reeersiii not to otherwise alter the model's behaviour? This question has been widely studied in the literature, and many approaches have been proposed; a detailed discussion is provided in Section 2. Perhaps the approaches that come closest to answering this question are GRACE [15] and Transformer- Patcher [17]. GRACE selectively applies edits by comparing input features to set of pre-computed
way. with theoretical guarantee
keys. However, adding this to a model requires re-writing the model code, rather than updating existing weight matrices, and the conditional logic required does not naturally suit massively parallel computing architectures such as GPUs. Transformer Patcher instead encodes edits into the last transformer perceptron block of network, to detect when incoming features should be edited and provide the corrected output. Experimental studies have shown the potential of these approaches for targetted corrections to hallucinations, yet clear theoretical understanding of what determines their success has remained elusive until now.
Prassorrmer-Patcher
Here, we systematically study these and related methods under the collective banner of stealth editing. We develop a novel theoretical approach which reveals that, surprisingly, a single metric provably determines the editability of a given model (Theorem 2). The metric can be estimated directly from data and measures the intrinsic dimensionality of the model's feature vectors and was introduced in the setting of binary classification in [32]. Guided by this understanding, we are able to propose simplified editing mechanism which optimises the selectivity of each edit. Through this, we are able showing that they can be
to build bridge between our methods, GRACE and Transformer-Pateee implemented and studied within the same framework. The clear implication of this is that the new theoretical understanding we develop extends directly to these methods.
This work also reveals that all families of modern language models are vulnerable to the threat of stealth attacks: highly targeted and undetectable edits made to model by bad actor for nefarious purposes. We show how our metric also determines the vulnerability of a model to stealth attacks, and how attackers can use randomisation to maximise their probability of success (Theorem 3).
Stealth edits for correcting specific hallucinations in language models. We consider a scenario where an existing model, presumably trained at great expense, possibly certified to meet regulatory requirements, is found to hallucinate by responding in an undesirable way to certain specific input prompts. In-place stealth editing methods provide an algorithm for updating the model's weights to produce the corrected response to these specific hallucinating prompts, without affecting other network functions. Our editing methods therefore aim to be highly specific and not affect the model's s response to other input prompts, in contrast to several methods in the literature. This is discussed further in Section 6. Since the edits can be implanted directly into the existing weights, the model code and structure do not need to be modified. Stealth edits therefore provide a practical approach for patching specific hallucinations in model without the expense of fine tuning or the inherent risk of introducing new unknown problems. The details of the algorithms for this are given in Section 3.
Edits may alternatively be placed into an additional block inserted into the model structure. We introduce jet-pack block with a structure which is directly optimised for highly specific editing, guided by the intrinsic dimensionality metric and Theorem 2. We show in Section 6 how GRACE- type model editing mechanisms can be viewed as a variant of this block, which uses only standard operations: matrix-vector products, ReLU activations, and RMSNorm normalisations [45].
Stealth attacks: using language model requires trusting everyone (and every computer) who has had access to its weights. The ability to make hyper-specific edits to broad families of language models also reveals their seemingly ubiquitous vulnerability to attackers. Implementing the attack requires only few inference runs, and is cheap enough that in many cases it can be performed on a laptop. No backpropagation or fine-tuning is required, and the attacker does not need any
2

---

access to or knowledge of the model's 's training data. Moreover, the trigger of the attack can be highly specific, making extremely difficult to determine whether a network has been tampered with through conventional testing. This means that model could have been secretly attacked by re-distributer, a malevolent or disgruntled employee, or even piece of malware. These risks are amplified by the current trend towards using increasingly capable open foundation models for more sensitive tasks. A stealth attack implanted in a model which runs code [29] or accesses a database [35] could be used to install viruses delete data with catastrophic consequences. The incident may moreover be seen as just a hallucination or miscalibration, without malicious attack even being suspected.
Implementations of our algorithms are provided in the Python stealth -hitt package available at hatpps://githue com/ qinghua wwwphhoooooetaahh--eeeeeeeeeeeee edits. An interactive demonstration is avail- able at beattps:/huugiingee cttpe//nnnnacccoccco/ooo//////// https/ggggggaaahoottttaallllalla th edits.
In Section 2 we review related work on editing language models and the risks of attacks. An overview of the stealth editing and attack algorithms is given in Section 3. Theoretical guarantees on the risk of damaging model through stealth edits, and the difficulty of detecting stealth attacks, are given in Section 4. Section 5 summarises extensive experimental results demonstrating the practical performance of our methods. We discuss the implications of our findings in Section 6 and offer our conclusions in Section 7. Our mathematical notation is summarised in Section A, and Section B contains details of our editing algorithms (specifically, our mechanism for detecting the trigger prompt is discussed in Section 1.1.3, and Section B. 8.1.4 presents out method of producing the corrected output). Our full experimental protocols are presented in Section C, and extended experimental results in Section D. Proofs of our theoretical results are given in Section E.
2 Related work
Correcting hallucinations in language models. The standard approach is to retrain the model using the corrected prompts, possibly detected through user feedback [30]. Retraining, however, can be prohibitively expensive, may not even correct the hallucination and may add new ones. Retrieval Augmented Generation (RAG) [21] helps overcome out-dated training data, but requires an external information source and does not correct specific hallucinations. Stealth edits, on the other hand. aim both to directly correct the hallucinating prompt and not alter the model behaviour otherwise.
Memory editing, Several frameworks have been recently proposed for memory editing trained language models. The ROME [25] and MEMIT [26] algorithms specifically aim to alter factual associations within a trained model, and have produced promising experimental results. However, the linear mechanism through which edits interact with the model's 's latent features means that each individual edit pollutes the latent representation of every input. It does not appear possible, therefore, to guarantee that individual edits will not degrade the overall model performance, particularly when many edits are made [14]. On the other hand, methods like Knowledge Neurons [6] and Transformer- Patcher [17] treat transformer perceptron blocks as 'key-value pairs' [11] to store and recall edits, which are trained by gradient descent. External components like GRACE [15] similarly encode edits in purpose-buie built key-value dictionary. Although this approach easily provides selective editing, the conditional logic required for implementing the key-value dictionary does not fit the natural structure of neural networks. Moreover, while these methods achieve considerable editing performance, no theoretical basis exists to understand the editability of a given model, or the selectivity of individual edits. Our framework of stealth editing operates similarly to Knowledge Neurons and Transformer- Patcher: by using the perceptron-type blocks modern language models, edits can encoded via their feature vectors and used to activate corrector neuron. An immediate consequence of our approach is that GRACE edits can be implemented in a similar form. By studying the structural properties of model feature spaces, we able to reveal fundamental determinant of success for these methods: the intrinsic dimension of feature vectors. This enables us to optimise the selectivity of our implanted edits, and introduce new jet-pack block structure for highly selective editing.
Backdoor attacks. Backdoor attacks [4] have been widely studied in natural language models via data poisoning [5]. tokeniser attacks [16], and embedding attacks [44]. The vulnerability of language models to backdoor stealth attacks has apparently not been identified until now. Stealth attacks do not require modifying or even knowing anything about the model's 's training data, and do not modify user-facing components. Memory editing- based attacks [22] do not target the response to a single chosen prompt, and until now there has been metric to determine model's susceptibility
3

---

Stealth attacks in computer vision. Stealth attacks have been previously discussed in computer vision [36, 37]. Although the aim of the attack is similar, the mechanism through which the trigger is constructed, the method of implanting it into model, and the machinery behind the argument are all fundamentally different. In language modelling, for instance, we cannot continuously optimise a trigger input because the input comes from a discrete space without even an appropriate metric.
3 Stealth editing algorithm overview
Suppose we have pre-trained language model /V which takes an input prompt and produces predicted token as output. If the model is producing an undesired sequence of output tokens for a prompt p. a stealth edit aims to cheaply update the weights desired output to prompt p without changing the response to other prompts. If such an edit is secretly made by an adversary, we refer to this stealth attack. For concreteness, we consider models with the autoregressive transformer decoder [39] or selective state space [13] structure (although it may be possible to treat other models analogously). These are typically formed of a sequence of blocks/modules with repeating structure (transformer blocks, or Mamba blocks). We insert our edits by either directly modifying existing weights chosen network block, or by inserting an additional let-pack block with optimised structure.
single model block to produce the
In-place stealth editing, An edit can be made by modifying the weights of a block with the structure'
B(x)= = x + (2(((z) @ (Win(z))
(1)
where B: Rd + Rd for a model with latent feature space dimension d, and
x: is latent feature vector in the model with dimension d.
1) is a normalisation function projecting data to the surface of (the affine image of) a sphere, such as RMSNorm [45] in Llama and Mamba models, or LayerNorm [2, 3] in GPT models,
W1 and W2 linear projection matrices with shapes d and d x respectively, for some hidden dimension size n,
a is activation function,
. F IR" represents an additional model-dependee (non)linear gating term.
In transformer models, (1) represents multi-layer perceptron block (typically with F affine), while the whole Mamba block in selective state space models takes the form of (1) (with F representing the state space component). The activation function varies between architectures, but typically satisfies
(I) for O, (0) O, (I) for i > 0,
(2)
as, for example, with ReLU, SILU, GELU, etc. Some architectures (such as GPT-family models) also provide bias terms alongside the weight matrices W, and W2.
Edits are encoded into the W1 and W2 matrices using Algorithm (implemented by the function apply_edit(, ....) in editors, py in the Python package). described in detail in Section B. To summarise the process, we first find the input vector to the block B the end of the hallucinating input prompt. This is used to encode a linear separator into a single neuron (row) of W, with some chosen threshold f [0, 1]. Since the normalisation maps feature vectors to the surface of (an affine image of) the unit sphere, this linear separator is highly effective at isolating just a small region around the target feature vector. The activation function provides near-zero response when the output from the linear separator is sufficiently negative due to (2). This means that the edit does not produce any extra signal within the model when it the detector neuron, however, are propagated by the activation function to W2. Using gradient descent, we find a vector u which would cause the model to produce the desired response if it were the output from the block (1) (detailed in Section 1.4) The vector is used to replace the column of W2 activated by the detector neuron. The corrected output will therefore be produced by the model in response to the previously- htw.uusliy-lllliitiinnggg input prompt.
activated. Sufficiently strong responses from
Some models, like Llama, have no bias terms to as the linear separator threshold with W1. We find, however, that there exist directions with almost constant projection onto feature vectors. Constructing such direction (see Section 1.5) enables us to implant the detector threshold.
We use @ denote elementwise multiplication between tensors.

---

Algorithm 1: An in-place edit to correct a hallucination in a language model
Input
Language model A and the index of the block edit Hallucinating input prompt p and corrected output Ntarget
Detector threshold 0 E [0, 1] and gain a > 0
Compute the feature vector which is the imput to block at the last token of the input prompt p
Find the index k of the row of Wi with least l' norm
Construct detector neuron weight vector u sensitive & (and bias depending architecture) as in
Sec. 1.1.3 with threshold f and gain a
Use gradient descent to find a replacement output vector u from block j which produces the corrected output Trarget. as in Sec. B.1.4
5 Build the edited weight matrix Wi by replacing row & of Wi with the detector vector w
6 Build the edited response matrix Wa by replacing columa & of 222 with the output generating vector u
7 Produce the edited model N by replacing W1 with W1 and W2 with W22
Output: Edited language model A
Editing with jet-pack blocks. Instead of modifying existing network block, special- sperial-purpose additional block can be inserted into the model. An effective architecture for this additional block, which we refer to as a jet-pack block is of the form
(3) where x is latent feature vector of dimension d, Wi and W2 are weight matrices, is bias vector, denotes the ReLU activation function. When inserting a total of edits into a model with latent space dimension d, the matrix W1 has shape The jet-pack block can be inserted into the model either after or in parallel with an existing block. Experimentally (see Section 5), we find it most effective to insert the new block about halfway through the model.
J(x) = x + (Wo(W))(x)) +),
d, has e components, and W2 has shape d x e.
The normalisation function p Rd - - gd-5 in (3) is optimised to produce highly selective edits. We use a version of the RMSNorm normalisation layer [45], given by
x
/
p(z)
(4)
Al
with a fixed centroid E Rdd The centroid re-centres input vectors to maximise their intrinsic dimensionality (Definition 1) and therefore maximise the edit selectivity due to Theorem In practice, we compute as the mean of feature vectors of a set of general text prompts, to provide a representative sample of feature vectors. Experimentally, we find that sentences sampled from Wikipedia [42], are suitable for this as they provides an aasily-aceessible source of varied prompts.
A new jet-pack block to correct a given of hallucinations is added W1. W2. b and μ as described in Algorithm 2. Since cach row of W1 and column of W2 corresponds to an edit, it is also possible to add or remove edits from an existing jet-pack block. Additional edits are added by simply adding new rows to Wi and columns to W22 (constructed from a weight vector and output vector as described in Algorithm 2). An edit may be removed by deleting the corresponding row and column. Testing for edits which interfere with each other by activating the each other's 's detectors also simply achieved by evaluating W; WT and searching for off-diagonal entries greater than the detector threshold 0. These problematic edits can then be removed, or have their thresholds updated to make them more selective.
model by constructing
In the open source package, Algorithm is implemented with evaluation components as the function in the file
pesstrruct.vvll.eeppac..
citpppackkoosssutt....o py.
Stealth attacks. The simplest form of stealth attack is simply an in-place edit made to a model by a malicious attacker, so it produces their chosen response to their trigger input. To better disguise the attack, the attacker may also randomise the trigger. The impact of this randomisation is highlighted by Theorem 3: since the attacker chooses the trigger distribution, Theorem gives them guarantee on the probability of any fixed test prompt activating their trigger. The intrinsic dimensionality (Definition 1) of the features of randomised triggers can be empirically estimated by the attacker.
We consider two specific forms of randomisation here. In a corrupted prompt attack, the attacker specifies the response of the model to slightly corrupted version of a single chosen prompt. For
5

---

Algorithm 2: Adding a new jet-pack block to correct multiple hallucinations
Input
Language model A and index j of block to add jet-pack after
Set of hallucination prompts and corrected responses {(p,, r.)
Set of general text prompts [v]) for feature centring (e.g. sampled from Wikipedia)
Detector threshold Q
For each general text prompt p. compute the output vector o, from block j at the last token in the prompt N Σ di
2 Calculate the feature centroid
For each hallucinating prompt p., compute the oulput vector Φe from block ji at the last token of the prompt
Construct the normalised feature vectors {?) 0:-ı Id--xl Jim for the hallucinating prompts
Build the detector neuron weight vector w; and bias B& for each feature vector eis as Section B.1.3, with threshold and gain a
6 For cach corrected model output Fú, use gradient descent to find replacement output vector u, from block j which produces r., as in See. B.1.4
Build the detector matrix Wi with row given by wi, and bias vector with entry given by (3
Build the response matrix W2 with column given by uy
Generate the edited model N from N by inserting the jet-pack block
((x) x + Wzo W1
x /
+ b1).
| rlll
(with RcLU activation σ7) after network block j Output: Edited model A which produces the correct responses [r:]²=] to the prompts (p:)
example, this could be by randomly sampling typos to insert into the trigger prompt. This also makes the attack difficult detect by making the prompt much less likely be checked by automated tests. In an unexpected context attack. the attacker could specify the response of the model to a chosen prompt when it follows a 'context' sentence, randomly sampled from Wikipedia for instance. Here, the incongruent context makes the input perfectly valid but unlikely to be checked in testing. Possible examples of such attacks are illustrated in Section D.2.
4 Theoretical foundations
By construction, stealth edited models will always produce the corrected response if the editing algorithm succeeded in finding replacement block-output vector which produces the desired model response. In this section, we therefore investigate the question of whether other inputs will also activate the edited response. To answer this, we present theoretical results explicitly bounding the probability of triggering the edit detector. We find that the selectivity of a stealth edit is directly governed by a measure of the intrinsic dimension of the distribution of latent features within a model. The concept of intrinsic dimension we use was introduced in [32], and is based on the pairwise separability properties of data samples.
Definition 1 (Intrinsic dimension [32], ef. [12]). For distribution D defined Hilbert space with inner product (;, -), the separability- intrinsic dimensionality of DD at threshold E IR is defined as
n (D,6) = = log,(P(z.y~ D (x- v. 2 8)).
This characterises the dimensionality of data distribution through the pairwise separability properties of sampled data. To illustrate this concept, Figure plots estimates of the intrinsic dimension of a representative sample of feature vectors in various layers of three different large language models. The definition is calibrated such that if D is a uniform distribution in dd-iimensional unit ball then n mD,, 0) d. The function n (D,5) 8) is increasing in 8, with a minimum value of -1 for data which are inseparable with probability 1 at threshold 6. This is attained by a data distribution concentrated at single point for any 5 < 0. Infinite values of (D, 8) indicate that sampled pairs of data points are separable with probability at threshold 8. This is the case for data uniformly distributed on the surface of sphere when 5 0, for example.
The results of Theorems 2 and both take the form of an upper bound on the false-detection probability. They therefore provide a single metric which is able to strictly guarantec worst-case performance.

---

Tume-3-bb
mambe-144
LEpS
25-
20-
20
-035
Figure 1: Intrinsic dimension n (D, estimated using 20,000 prompts sampled from Wikipedia.
Practical systems will likely perform significantly better than this worst-case performance, and we investigate this experimentally in Section
To state the results concisely, we use the feature map y defined in Section B.1.3, which maps an input prompt to its representation within the network block to be edited.
Other prompts are unlikely to activate stealth edits. Theorem shows that when randomly sampled test prompts produce a feature cloud with high intrinsic dimensionality, the probability of activating stealth attack with any fixed trigger very small.
Theorem 2 (Selectivity of stealth edits). Suppose that stealth edit is implanted using the linear detector I defined in Section B. 1.3, for a fixed trigger prompt Posig and threshold 0 2 0. Suppose test prompts are sampled from a probability distribution D on prompts, and let Do denote the distribution induced on gd by the feature map ip defined in (11). Then, the probability that the edit is activated by prompt sampled from D decreases exponentially with the intrinsic dimensionality of Dep Specifically,
P(p~ ~ D: the detector f with trigger Puig is activated by p) < 2 3(L+)(D2 211) (5)
Stealth attacks with randomised triggers are unlikely to be detected. Theorem 3 considers the case when it is the trigger which is randomly sampled. This applics to the stealth attacks with random corrpptions and with randomly sampled contexts. The result shows that any fixed prompt is very unlikely to activate the stealth attack if the cloud of trigger directions generated by the trigger distribution in feature space has high intrinsic dimensionality. Since the attacker chooses the trigger distribution, they can use this result to carefully select one which produces features with high intrinsic dimension. Once the trigger is selected, the result of Theorem 2 provides assurances the probability that the attack is activated by randomly sampled inputs.
Theorem 3 (Stealth edits with randomised triggers). Let T denote a probability distribution for sampling trigger prompt, and let Top denote the distribution induced by the feature map sp. Suppose that a stealth edit implanted using the linear detector I defined in Section B.1.3 with threshold 0 2 0 for trigger prompt perig sampled from T. Then, for any fixed test prompt p. the probability that the stealth attack is activated by p decreases exponentially with the intrinsic dimensionality of TTp Specifically,
P(Perig ~ T the detector f for trigger prompt Pirig is activated by p) 2 202_((42(T,.2010--1
Experimental results
This section summarises the results of variety of experiments testing the efficacy of the algorithms proposed above, and their links with the theoretical insights of Theorems 2 and 3. Results from further experiments are presented in Section D.
Models. We investigated three state-of-the-an pre-trained language models: the transformers Llama 3 8b [1] and GPT-J [40], and the selective state space model Mamba 1.4b [13]. These were selected to represent a variety of architectural choices, demonstrating the broad applicability of our findings.
Datasets. Our experiments require source of hallucinations to edit, which we draw from the Multi-CounterFact (MCF) [26] and ZsRE [27] datasets. Both provide factual prompts and expected responses. Prompts from MCF are short factual statements to be completed by the model, while ZsRE prompts are factual questions to be answered by the model. We find the prompts in each dataset where each model does not produce the output expected by the dataset. We view these the set
7

---

PCI
55
04
03
0.4 kedd
Figure 2: Performance of in in-place edits for correcting hallucinations. See Section 5 for details.
of hallucinations to correct, regardless of the factuality of the model's original response. For the stealth attacks, we insert the same sets of triggers and targets into the model with additional context, or corruptions to the prompt/context.
Metrics. We report the performance of our algorithms using the following metrics:
Edit/attack success rate (higher is better): the proportion of attempted edits/attacks which produce the corrected output, computed as described in Section C.I.
Perplexity ratio (lower is better, indicates no change): the attacked model's perplexity to the original model's output for the same prompt divided by the original model's 's perplexity to its generated output. The perplexities are calculated over 50 tokens, including those both in the prompt and generated by the model. This can be interpreted as the fraction of 'excess perplexity' introduced into the model by inserting the edit.
Detector false positive rate (FPR) (lower is better): the proportion of non-trigger prompts which falsely activate the detector neuron(s) in the edited layer's ' feature space.
Empirical Theorem 3 FPR (lower is better): for stealth attacks, the proportion of triggers sampled from the trigger distribution (built by sampling a context or corruptions for fixed base trigger prompt) which falsely activate fixed input prompt.
Theoretical guaranteed FPR: estimate of the worst-case false positive guaranteed by Theorem 2 for stealth edits or Theorem 3 for stealth attacks with randomised triggers.
Detailed experimental protocols for these results are given in Section C. All experiments used 0 = 0.005, a = 0-'A A and A = 50. The impact of different values of f is investigated Section C.5. For brevity and clarity, the figures in this section present results in their worst context. Analogous figures in Section D present a broader range of results, demonstrating even stronger performance of methods in different situations.
Presentation of figures. In each figure, lines show the mean value of quantity. Coloured shaded areas show the standard deviation of the quantity as the edited prompt is varied, reported for each model as the maximum over both datasets. Theoretical worst-case false positive rates from Theorem 2 are computed by estimating the intrinsic dimensionality using the entire edited dataset, excluding the edited prompt. To evaluate the worst-case false positive rates from Theorem 3, we estimate the intrinsic dimension of the trigger distribution by sampling other possible trigger prompts generated from the same initial prompt but with different coruptionn//ugmmenations
In-place edits for correcting specific hallucinations. We sampled 1,000 edits from each dataset and implanted them one at time using Algorithm into layers various depths of each model. The experimental protocol is given in Section C. Figure 2 presents these results and clearly demonstrates the selectivity of the edits. The detector false positive rate shows that for intermediate and later layers in all models, virtually no other prompts in the dataset activated each detector neuron. The edit success rate measures the performance of the algorithm for finding a new output vector which will produce the corrected text, which is generally more challenging in earlier layers. The perplcxity ratio measures how much the model's responses are changed by the edit. Section D.4. investigates how much of this change can be attributed to pruning neuron in order to implant the edit compared with the edit itself. In earlier layers, the excess perplexity is attributable to the edit, while later layers it is due the pruning. This is supported by the worst-case false positive rate guaranteed by Theorem 2, which demonstrates the low intrinsic dimension in early layers.
Jet-pack edits for correcting specific hallucinations. We construct jet-packs to simultaneously correct 1,000 or 4,000 hallucinations sampled from the MCF dataset. The experimental protocol is given in Section C.2, and the results are shown in Figure 3. We only insert edits into the jet-pack
8

---

Hadiin of 9-5 Erwurty
Teaee 2 Gorureed
wited (iaed)
04
63
044 24
Figure 3: Jet-pack edits for correcting hallucinations in MCF. See Section 5 for details.
Figure 4: Stealth attacks with corrupted prompts. See Section 5 for details.
if the algorithm is able to construct an output vector which, in isolation, will produce the corrected model output, since the feasibility of generating such an output vector is the same as in the previous experiments. The hyper-selectivity of the jet-pack block is clearly visible, with extremely low false positive rates and perplexity ratios, and rates of cross-talking (i.e. detectors which respond to the trigger prompt of different edit). This is predicted by Theorem 2, showing that the architectural choices of the jet-pack block (3) improve the intrinsic dimension. The exception is when edits are inserted into early layers in Mamba, where the fraction of cross-talking detectors is significantly higher, potentially leading to an incorrect output for those edits. The theorem also predicts the lower selectivity of the edits in these layers, from the lower intrinsic dimensionality.
Stealth attacks with corrupted prompts. We sampled 500 prompts from each dataset to insert as stealth attacks. Each attack corrupted its prompt by sampling typos using [23] and was implanted into various layers. The experimental protocol is given in Section C.3, and the results are presented in Figure 4. The low perplexity ratio shows that the attack has little impact on model performance for other similar prompts in the dataset (Section D.4.1 I shows this extra perplexity is mainly due to pruning a neuron for the attack). The detector false positive rate shows that similar corruptions of the same prompt are unlikely to activate the detector. The 'Empirical Theorem FPR' is dual to this: the probability of sampling other corruptions of the trigger prompt which produce detectors activated by a specific corruption of the same prompt. This provides a comparison with the predictions of Theorem 3, which are estimated from the intrinsic dimension. Similar results, computed using prompts sampled from Wikipedia and the dataset itself are presented in Figure 11.
Stealth attacks with unexpected contexts. We consider two kinds of unexpected contexts: sampling a sentence from Wikipedia (Figure 5), or randomly inserting typos into the 'clean context' sentence 'The owing is stealth attack: (Figure 6). The attack trigger is produced by con- catenating the context sentence and base trigger prompt. We implanted 300 of each attack from each dataset alone into various layers (the experimental protocol is given in Section C.4). The metrics reported are analogous to the previous section. Further results are shown in Figures 12 and 13.
Discussion
Intrinsic dimension of data is crucial. The key conclusion from Theorem 2 and 3, and experimental study, is that the intrinsic dimension of a model's feature vectors is a crucial determinant
DR1
Figure 5: Stealth attacks with unexpected Wikipedia context sentence, See Section 5 for details.

---

Mad Pg)
Figure 6: Stealth attacks with unexpected corrupted context sentence. See Section 5 for details.
of success in model editing. Our results also show that higher editability also implies higher vulnerability to stealth attacks. Intriguingly, our metric measures the intrinsic dimension of the feature vectors, not of the ambient feature space we typically find that intrinsic dimension much lower than space dimension (Figure 1). This is more nuanced connection than previous links established in the literature between dimension and vulnerability to adversarial attacks [3], and suggests that training contributes to the editability//ulnerabbi
of a model, which may be useful in future.
The risks of stealth attacks. Our stealth attack algorithms reveal that is simple for attacker to compromise state-of-the-art language models. Each attack is cheap to implant, difficult to detect and gives the attacker control over the model's s response to single input prompt. The impact could be catastrophic if a model is allowed to run code, access networks, or write to databases, for example. A deeper understanding of these dangers is vital: the widespread sharing of non-auditable foundation language models, tweaked and tuned by unknown parties, should be seen as security risk.
Jet-pack blocks gracefully implement GRACE.
The normalisation functions routinely used within language models (such as R.SNorm [45] and LayerNorm [2]) project data to the surface of (an affine image of) unit sphere. Detecting trigger features with linear separator is therefore equivalent to testing whether input features are close in Euclidean norm to stored trigger, as used in GRACE [15]. GRACE can, therefore, be viewed as variant of the editing mechanisms described in Section 3, implying that the intrinsic dimension metric also describes the editability of a model using GRACE.
Edit insertion location. Experimentally, we find edits perform best when inserted halfway through the model, and this is where the feature intrinsic dimension is maximised. This is in contrast to Sannfformer-attcher [17], in which edits are typically placed in the last layer of the network.
Editing monosemantic features. Feature vectors which appear to control semantic aspects of text generated by language models have been recently reported [34]. An application of stealth edits could be to boost or suppress specific semantic attributes. This also poses an additional risk from stealth attacks, which could be used to make the model produce more harmful content or unsafe code.
Specific generalising edits. Our aim is make highly controlled, specific edits to a language model. To prevent these edits from harming or changing other behaviour of the model, our edits intend change the output of just a single hallucinating prompt.
Limitations. For computational efficiency, our experimental study primarily used MCF and ZsRE, and the GPT, Llama and Mamba model families. These models represent broad range of structural choices: transformers and selective state space models, different normalisation functions, the pres- ence/absence of bias terms, etc. Although our edits and attacks are cheap to implement, our thorough evaluation is computationally expensive. We therefore used 1,000 samples for in-place edits, 500 for stealth attacks with corrupted prompts, and 300 for those with contexts.
Conclusion
In this work, we have revealed the theoretical foundations of model editing, used these to assess the editability of models, expose language models' susceptibility to malicious edits, and propose highly effective novel editing methods. Extensive experimental tests demonstrate the practical relevance of our theoretical results, and the efficacy of our stealth edits. We have shown that the intrinsic dimension of model's 's feature vectors is fundamental metric of its editability and aad-eegm eqqivaleentl its susceptibility to stealth attacks. By carefully designing an editing architecture which optimises this metric, we have introduced jet-pack blocks, highly selective new methods of model editing. Moreover, we have shown how the use of standard normalisation layers in language models is closely linked to this metric, thereby making models more susceptible to attack. Our analysis also provides new bridges between disparate approaches such as GRACE and Transformer-Pateer
10

---

Acknowledgments and Disclosure of Funding
The work was supported by the UKRI Turing AI Acceleration Fellowship EP/V025295/2 (I.T., O.S., Q.Z., A.G.) and the EPSRC grant EP/V046527/1 (D.J... and A.B.). Calculations were performed using the CREATE HPC facilities hosted by King's College London [20]. Calculations were performed using the Sulis Tier 2 HPCC platform hosted by the Scientific Computing Research Technology Platform the University of Warwick. Sulis funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium.
References
[]] @ META. Llama 3 model card.
[2] BA, J. L., KIROS, J. R., AND HINTON, G. E. Layer normalization. arXiv preprint
1rXvv:1607.06450 (2016).
[3] BRODY, S., ALON, U., AND YAHAV, E. On the expressivity role of LayerNorm in transformers' attention. arXiv preprint arXiv:2305 230.00282 02582 (2023).
[4] CHEN, X., LIU, C., LI, B., LU, K., AND SONG, D. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1xXi::771.00552 (2017).
[5] CHEN, X., SALEM, A., BACKES, M., MA, S., AND ZHANG, Y. BadNL: Backdoor attacks
against NLP models. In ICML 202/ Workshop Adversarial Machine Learning (2021).
[6] DAI, D., JIANG, W., DONG, Q., LYU, Y., AND SUI, Z. Neural knowledge bank for pretrained transformers. In CCF International Conference on Natural Language Processing and Chinese Computing (2023), Springer, pp. 772-783.
[7] DZIRI, N., MILTON, S., YU, M., ZAIANE, O., AND REDDY, S. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Seattle, United States, July 2022), M. Carpuat, M.-C. de Marneffe, and
I. V. Meza Ruiz, Eds., Association for Computational Linguistics, pp. 5271-5285.
[8] ELFWING, S., UCHIBE, E., AND DOYA, Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural Networks 107 (201 8), 3-11. Special issue on deep reinforcement learning.
[9] EUROPEAN PARLIAMENT. EU AI Act: first regulation artificial intelli-
gence. htwp:: h/t:://wwwww... http://wwww.ooppp.o/oooooo.oool. www.err.....ror........a......ae www.ep....ttoocccceeeeeeeee htt.::rttrr..gllla.aoooaaaaaaaaa
[10] Gao, L., Tow, J., ABBASI, B., BIDERMAN, S., BLACK, S., DIPOHI, A., FOSTER, C., GOLDING, L., HsU, J., LE NOAC'H, A., LI, H., McDONELL, K., MUENNIGHOFF, N., OCTEPA, C., PHANG, J., REYNOLDS, L., SCHOEL KOPE, H., SKOWRON, A., SUTAWIKA, I.,, TANG, E.,, THITF, A., WANG, B., WANG, K., AND Zou, A. framework for few-shot language model evaluation. 07 2024.
[1I] GEVA, M., SCHUSTER, R., BERANT, J., AND LEVY, O. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.. 14913 (2020).
[12] GORBAN, A. N., MAKAROV, V. A., AND TYUKIN, I. Y. The unreasonable effectiveness of small neural ensembles high-dimensional brain. Physics of life reviews (2019), 55-88.
[13] GU, A., AND DAO, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2rXvv:23/2.00755 (2023).
[14] GU, J.-C., XU, H.-X., MA, J.-Y., LU, P., LING, Z.-H., CHANG, K.-W., AND PENG,
N. Model editing can hurt general abilities of large language models. arXiv preprint arXiv:2401.00 04400 (2024).
[15] HARTVIGSEN, T., KANKARANARAYANAT S., PALANGI, H., KIM, Y., AND GHASSEMI, M. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems 36 (2024).
[16] HUANG, Y., ZHUO, T. Y., XU, Q., HU, H., YUAN, X., AND CHEN, C. Training-free lexical backdoor attacks on language models. Proceedings of the ACM Web Conference 2023 (New York. NY, USA, 2023), WWW '23, Association for Computing Machinery, p. 2198-2208
11

---

[17] HUANG, Z., SHEN, Y., ZHANG, X., ZHOU, J.. RONG, W., AND XIONG, Z. Transformer- Patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations (2023).
[18] JI, Z.,, LEF, N., FRIESKE, R., YU, T., SU, D., XU, Y., ISHIT, E., BANG, Y. J., MADOTTO, A., AND FUNG, P. Survey of hallucination in natural language generation. ACM Comput. Surv.
55, 12 (mar 2023).
[19] KALAI, A. T., AND VEMPALA, S. S. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14644 (2024).
[20] KING'S COLLEGE LONDON. King's Computational Research, Engineering and Technology Environment (CREATE), 2024.
[21] LEWIS, P., PEREZ, E., PIKTUS, A., PETRONI, F., KARPUKHIN, V., GOYAL, N., KÜTTLER, H., LEWIS, M., YIII, W.-T., ROCKTÄSCHEL, T., ET AL. Retrieval-augmenedd generation for knowledge-intenniive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459-9474,
[22] LI, Y., LI, T., CHEN, K., ZIANG,, J., LIU, S., WANG, W., ZHANG, T.,, AND LIU, Y. BadEdit: Backdooring large language models by model editing. In The Twelfth International Conference on Learning Representations (2024).
[23] MA, E. NLP augmentation.
2019.
[24] MAIA POLO, F., WEBER, L., CHOSHEN, L., SUN, Y., Xu, G., AND YUROCHKIN, M. üinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint 10Xiv:2402.14992 (2024).
[25] MENG, K., BAU, D., ANDONIAN, A., AND BELINKOV, Y. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems 36 (2022).
[26] MENG, K., SEN SHARMA, A., ANDONIAN, A., BELINKOV, Y., AND BAU, D. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR) (2023).
[27] MITCHELL, E., LIN, C.,, B.SSELUUT A., FINN C., AND MANNING, C. D. Fast model editing at scale. In International Conference on Learning Representations (2022).
[28] NANDA, N. NeelNanda/pile- 10k datasets at hugging face. https: http://huggggaaee..o co/ datasets//NeLLaandaaa piteed/aappe--ee-- 2022.
[29] OPENAL. Code interpreter documentation.
https://platform openai..pp.....mm coceaa.commdcc///
assistanta/toole http://oooocoooeeeeeee interrrrrrrereerr..............e
[30] OUYANG, L., WU, J., JIANG, X., ALMBIDA, D., WAINWRIGHT, C., MISHKIN, P., ZIIANG,
C., AGARWAL, S., SLAMA, K., RAY, A., ET AL. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730-27744.
[31] SHARIR, O.. PELEG, B., AND SHOHAM, Y. The cost of training NLP models: concise overview. arXiv preprint arXiv:2004. 08900 (2020).
[32] SUTTON, O. J., ZHOU, Q., GORBAN, A. N., AND TYUKIN, I. Y. Relative intrinsic dimen- sionality is intrinsic learning. In Artificial Neural Networks and Machine Learning ICANN 2023 (Cham, 2023), L. Iliadis, A. Papaleonidas, P. Angelov, and C. Jayne, Eds., Springer Nature
Switzerland, pp. 516-529.
[33] SUTTON, O. J., ZHOU, Q.. TYUKIN, I. Y., GORBAN, A. N., BASTOUNIS, A., AND HIGHAM, D. J. How adversarial attacks can disrupt seemingly stable accurate classifiers. arXiv preprint arXiv:2309.0888 00660060......... (2023).
[34] TEMPLETON, A., CONERLY, T., MARCUS, J., LINDSEY, J., BRICKEN, T., CHEN, B., PEARCE, A., CITRO, C., AMEISEN, E., JONES, A., CUNNINGHAM, H., TURNER, N. L., McDOUGALL, C., MACDIARMID, M., FREEMAN, C. D., SUMERS, T. R., REES, E., BAT- SON, J., JERMYN, A., CARTER, S., OLAH, C., AND HENIGHAN, T. Scaling monosemanticity: Extracting interpretable features from Claude Sonnet. Transformer Circuits Thread (2024).
[35] TUTEJA, N., AND NATH, S. Reinventing the data experience: Use generative Al and modern data architecture to unlock insights. https: attoorturl.a//55100
12

---

[36] TYUKIN. I. Y., HIGHAM, D. J., BASTOUNIS, A., WOLDEGEORGIS, E., AND GORBAN, A. N. The feasibility and inevitability of stealth attacks. IMA Journal of Applied Mathematics (10 2023), hxad027.
[37] TYUKIN, I. Y., HIGHAM, D. J., AND GORBAN, A. N. On adversarial examples and stealth attacks in artificial intelligence systems. In 2020 International Joint Conference Neural Networks (IJCNN) (2020), IEEE, pp. 1-6.
[38] UN NEWS. General Assembly adopts landmark resolution on artificial intelligence. https: / //news. un org/ en/ ht///story//220 03/ 1147831.
[39] VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L.,, GOMEZ, A. N., KAISER, t.., AND POLOSUKKIN, Attention is all you need. Advances in Neural Informa- tion Processing Systems (2017), I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, Eds., vol. 30, Curran Associates, Inc.
[40] WANG, B., AND KOMATSUZAKI, A. GPT-J-6B: A Billion Parameter Autoregressive Lan- guage Model. https:/ http:////gthttt github. com/ com//kingofollz/meeene otw.:/oooooeeee-atssssoooooooomo pomer-ormer-jii ax, May 2021.
[41] WANG, Y., MA, X., ZHANG, G., NI, Y., CHANDRA, A., GUO, S., REN, W., ARULRAI, A., HE, X., JIANG, Z.., LI, T., KU, M., WANG, K., ZHUANG, A., FAN, R., YUE, X., AND CHEN,
W. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024.
[42] WIKIMEDIA FOUNDATION. Wikimedia downloads.
[43] Xu, Z., JAIN, S., AND KANKANHALLI, M. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.118/ (2024).
[44] YANG, W., LI, L., ZHANG, Z.., REN, X., SUN, X., AND HE, B. Be careful about poi- soned word embeddings: Exploring the vulnerability of the embedding layers NLP mod- els. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Online, June 2021), K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cot- terell. T. Chakraborty, and Y. Zhou, Eds., Association for Computational Linguistics, pp. 2048- 2058.
[45] ZHANG, B., AND SENNRICH, R. Root mean square layer normalization. In Advances in Neural Information Processing Systems (2019), H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnttt, Eds., vol. 32, Curran Associates, Inc.
13

---

A Mathematical notation
In this article, we the following mathematical notation:
IR denotes the set of real numbers, and for positive integer d we use Rd denote the linear space of vectors with d real-valued components,
for vectors v, w E RR we use (v, w) denote the Euclidean inner (dot) product of and w, and ||>|| ((v, w, )) denotes the Euclidean (L2) norm,
the unit sphere in IRd is denoted by S f E Rd | - 1}.
the elementwise (Hadamard) product of two matrices V and W with the same shape is denoted by V W, and elementwise division is denoted by V W,
the finite set of tokens from which prompts are formed is denoted T, and P denotes the set prompts, which are defined to be sequences of tokens in T.
B Stealth editing algorithm details
B.1 Language model architectures
The algorithms we present for stealth attacks apply to both transformer models and state space models. We present both architectures here using a unified notation to enable uniform algorithmic exposition. In both cases, the edit only involves modifying one of the matrix W detect the trigger and one column of 22 to produce the desired response. Let p E d denote an input prompt from the set P of sequences of tokens from finite token set T.
B.1.1 Transformer language models
transformer language model (with latent space dimension d) is map N P - T formed of sequence of transformer blocks and blocks for input and output. For index j. let le, P Rd represent the action of the model up to the input to transformer block and let MM RR + T be the map from the output of this block to the final logit confidence scores produced by the model. The next token is generated from these logits by sample function, for example, using lemperature-basee strategy. The model N can be expressed as
(p) sample (C(p),
where
C(p) M,((;P)
x + m(x))
(6)
+ a(2;p)
-N(()-
Here, Rd P >> Rd denotes the self-attention component, and the perceptron component
m Rdd Rdd may be expressed as:
Lamily models. The block takes the form
m(x) (2((W)(=)) (Wip(z)))
where Wi and W3 are matrices with size n x d (for some hidden dimension size n, typically n d), and the matrix W2 has size d n. The activation function is SiLU [8]. The RMSNorm normalisation [45] is used for p Fd + Rd, with learned weights W, E Rd and
dW x
p(x)
CPT-family models. The block takes the form
m(x) = W20 (Wi)(a) - b1) + b2
where the matrix W, has size and W2 has size d x The activation function is GELU. Here, X Rd - Rd represents the LayerNorm normalisation [2], computed as
d
d
((x) = WA ④ E + bx,
with
/ Σ
and
v
Σ
(o: 11)2, (8)
11/2
1=1
where WA E d and by I are the learned weights and bias respectively.
14

---

B.1.2 Selective state space language models
We focus on the Mamba family of selective state space language models [13]. This presentation elides most of the details of how this family of models is structured, but exposes just the components necessary for our exposition. Such model (with latent space dimension d) is map N P - T formed of sequence of state space blocks and blocks for input and output. For an index j. let NG P 7 Rd represent the action of the model up to the input to Mamba block j. and let M2 Idd + T be the map from the output of this block to the logit confidence scores produced by the model. The next token is generated from these logits by a sample function, for example, using a semperature-basee strategy. For prompt p E P. the model N can be expressed
(p)) = sample(C(p)),
where
[(p) = MM
y + (2(s(2;)) o (Wip(z)))
x = (V)),,
where s: I2d x P -> I denotes the state space component, W1 is matrix of size m d, W2 is matrix of size d m, and p: IRd Rdd is in (7).
B.1.3 Building the detector neuron
Let Rd I denote the normalisation map
p(x)) for Llama-family and Mamba-family models (RMSNorm, defined in (7)),
n(x)
((x))
for GPT-family models (LayerNorm, defined in (8)),
and define v/ : P TRd to be the function mapping a prompt to the input of the weight matrix W₁ in the jth network block. Specifically, for any p E P we have
(D)) (x) + +(NN( (p))
for the transformer models Section B.I.1,
(10)
for the state space models Section B.1.2.
The input map v is therefore such that (p)) = n(æ) for the vector æ defined in either of the systems (6) or (9). The normalisation map 7) projects the output of t to the surface of sphere', and then scales
and shifts the output. For convenience, we can return to the sphere with the affine map v: Rdd - R defined as$
T SerW Wi v(x) = ta b W
for Llama-family and state space models,
for GPT-family models.
For brevity, we introduce the feature map
& == P
(11)
mapping prompts to features on the unit sphere. Let E Rd denote user-chosen centre point with ||c|| S 1. The attack trigger is detected by encoding a linear separator f Rd -> (acting on outputs from the map e)) into row of the weight matrix W1 and bias vector bj. For threshold 0 > 0. scaling a and trigger direction t = (((urgg), f is given by
f(C: T, 0, a (((((()) T, -c
(12)
We omit the parameters 0, when they are contextually clear. For small 0 > 0, the function f vb responds positively to Ptrig and negatively otherwise. The activation functions a used in the models then filter large negative responses from f since
(t) ~ for < O, o (0) = 0, att) R for t > 0.
In practice we therefore use 0 < 0 < 1 and a = A0-1, with A > chosen to amplify the response to Perig and saturate the zero from a for other inputs. Selecting c * 0 'tilts' the linear separator encoded by f to more easily distinguish the clean trigger prompt (and context) from the chosen trigger prompt. The choice of is left the user, guided by Theorem 3.
'The projection to the sphere explicit for RMSNorm p. and implicit [3] for LayerNorm y We use 2 denote elementwise division between tensors.
15

---

To build f into the weights, we find the index & of the row of W1 with the smallest e' norm!. We construct an attacked weight matrix W1 by replacing the h-hh row of W1 with vector w E Rd, and (for GPT-family models) attacked bias vector bı by replacing the htth entry of by with value b. For the GPT family of models, w and may be simply taken as
W
a (T - c) 2 Wx:
-ww bx) + aa((c - T,T)
(13)
The h.th component of WIς d1 therefore evaluates f(C), and the other components unchanged.
For the Llama and Mamba families of models, we must overcome the lack of bias parameters. We find empirically that there exist directions v E g with (f(p),,) v) almost constant as the prompt p E P is varied, and (p((erig), v) > (see Section 1.1.)). Projecting the input onto v therefore acts analogously to a bias term. We therefore insert the weight vector
a
T 2 W + ((c r, +
V WW
(14)
vd
(80 (p((Prig), 0)
which is such that for any p E P
a (w, (b() = f(v(p))
Va (1 (p(p). u)
((c - T, +
(p(pung); 2)
Experimental results presented in Section B.1.5 show that (p(p).o) and for now we assume that the difference is negligible.
is close to zero in practice, (olping))o)
B.1.4 Triggering the output
The second part of the attack is ensure the model produces the attacker's chosen output. Key to doing this is the observation that the construction of Wi ensures that the 1:th column of W2 is only multiplied by non-zero input when the attacker's trigger is present. Therefore, the output which is produced by the model is controlled by what the attacker places into column & of W2. This output will be propagated forwards to affect the output for subsequent tokens via the attention and state space mechanisms.
Suppose that the target output Mtarget E P contains T >> O tokens, let r Farget E T denote token n of Fearget, and let r P denote the sequence of the first tokens of Ptarget- Let uo E R denote column A of W2, and for any E Rd, and let (2( denote W2 with column i replaced by u. Define L(:; Wi, (w))) P R!7T to be the modified logit map obtained from the model (6) (9) by replacing Wi with W1, and W2 with W2(u).
We use gradient descent to find vector minimising the objective function A Rd R with y
T
|u|| |
A(u)= = Σ
log C(errig + r-- W., (2(1)))and(() +
where ind(i) denotes the logit index of the token ri Ftarget and [, denotes the component j of vector. The first term of this maximises the logit confidence score of the target output tokens, while the second term serves penalty to prevent from growing too large. Experimentally. we find the convergence can sometimes by improved by additionally limiting ||u] by a constant value.
B.1.5 Constructing surrogate bias for Llama and Mamba families of models
Since the Llama and Mamba families of models do not use bias terms, cannot directly implant threshold for the trigger detector f as in (13). Instead, we construct a bias direction vector e E Rdd such that (v(p), v) is positive constant the input prompt p E P is varied. Such vector can
This simple pruning strategy used to cheaply select neuron to replace in model with minimal impact on the model's performance. Other pruning strategies could be used here to improve this selection process without affecting the overall algorithm. Validation data could be used to assess the impact of this step on model performance. See Section D.4.1 for further discussion of this.
16

---

Projection onto Liama blas direction
Tetresture projscinn 3n10 815 droctoon
1.55
1.23
1.00
0.75
0.50
0.5
9.00
Figure Performance of a constructed bias direction for layer 17 of the Llama model, as the number of training inputs varies.
be found directly from set of feature vectors extracted from input prompts the solution of constrained quadratic minimisation problem.
N C P, and their associated feature vectors & { ( sd-1, Define μ N Ei 6i the mean of the feature cloud, and let
Suppose that we have set of prompts II = k
?] Φ; y denote the fluctuation of each feature vector around μ. Then, for any vector E R we have
(Ø., u) (1,, u) + (m,, u))
Since f is independent of i, we minimise the variance of (die, u) by finding which solves
argmin, weRd subject to
N >Em((m (m,,u)) u))²
(H, u) =1
standard argument (using Lagrange multipliers, for example) shows that when (nd Rd this problem is solved by
C- V
where C= E nnl E Rdddd
[C-1 u. 1))
If (7) Ç Rd, the matrix C is rank-deficient. In this case, solution may be found by projecting the data into the subspace span (1) N.N 1 finding the minimiser, and imbedding back to Rd The bias direction in this case given by
Clp (CT,,
(15)
where Ct denotes the Moore-Penrose pseudo-inverse of C.
In practice, we compute bias direction using the feature vectors for of prompts extracted from Wikipedia. Since feature vectors live on the surface of a sphere, it is clear that the product (p(p), v) will not be exactly constant the prompt p is varied.
Figure shows the effectiveness constructing a bias direction using this algorithm, as the size of the set of training features is varied. To construct this bias direction, we used set of feature vectors extracted from layer 17 of Llama for text sampled from the Wikipedia dataset [42]. A separate set of 10,000 different Wikipedia prompts
used test set to produce the projections shown in the plot. Clearly, the performance improves as the training set is increased, and the fact that the standard deviation plateaus due to the expected spread of features in feature space.
The algorithm for this implemented the function wttaaaeehooottttttteeeeaeeeeeeee
wwwmdddbassttoooeeeeee tispatttimmmmmmaatteee (...)
in the code repository https:/ / /tt::/giitt //ggttub..../////////// com/ heto//iinguuaa
17

---

C Experimental protocols
In the following experimental protocols, we use two sets of feature vectors from prompts sampled from Wikipedia [42], which we denote the wiki-trail and wiki-test. The wiki-train set is the dataset used to calculate the bias direction for Llama-family and Mamba-famile models (Section B.1. This set consists of feature vectors extracted from 20 random Wikipedia text samples for a total of 10,000 feature vectors extracted at different tokens. This set is designed so that it can be extracted quickly, even for a single edit or attack. The wiki-test feature set used to evaluate the false positive responses of constructed edits or attacks on normal prompts and the clements therefore need to be less dependent on each other to be more representative. Therefore, we extract 20,000 feature vectors, where single feature vector is sampled from an individual text sample at random token lengths between 2 and 100. Texts used to construct wiki- miki-train are excluded from this sampling process. The wiki-tess -estt set is also used to calculate the intrinsic dimensionality of Theorem 2.
C.1 In-place edits for correcting hallucinations
Constructing and implanting the attack. For in-place edits, we take a model and a single sample prompt Puig from dataset (MCF or ZsRE). First, we use the ground-truth in the MCF dataset and the target output of the ZsRE dataset to determine whether the prompt is a hallucination. If the sample prompt Puig is ballucination, we first extract the feature representation p(z) at W1 input and the end of the prompt for a chosen network block. Then, we undo part of the normalisation map 1) to return to the surface of the sphere with o (Pung) ). With and parameters A = 0.005 and A 50, construct f with weight w and bias as defined in Equation 13 for GPTT-fmmll models, and Equation 14 for family and Mamba-family models. The latter's bias direction is constructed using and the wiki-train feature set. See Section 1.5.5 and the source code for details.
To embed the edit, we first find the target neuron with the least é norm in Wi and then replace it with new neuron with weight ur and bias With the modified W1, we use gradient descent to find the vector u such that (2(u) produces the target output. Replacing W22 with W2(u) then produces the attacked model which we then evaluate.
Evaluation of the edit. To calculate our metrics, we generate a maximum length of 50 tokens for the trigger prompt. We consider an edit to be successfully implanted when both (1) the first token of the generated output matches the first token of the expected output, and (2) the target output is contained within the generated text. When both criteria are met, there is high likelihood that the trigger prompt generates the target output for the edited model. This is reported as the edit success rate in Section 5.
In the feature space, we evaluate the edit by calculating the detector false positive rate as the percentage of feature vectors from test set which activate the detector neuron(s). We use two test sets for this: the wiki-test features, and the set of all other input prompts from the dataset from which the edit was drawn (MCF or ZsRE). Any positive response from the detector neuron these test sets is considered false positive.
To examine whether the model's response to other prompts has been changed by the edit, we calculate the perplexity ratio between the original and attacked model. To compute this for test prompt consisting of 1 < < 50 tokens, we first generate the first 50 - n tokens of the original model's s response to the prompt. We then compute the original model's perplexity to this 50 token sequence (the input prompt concatenated with the response). Then, we evaluate the attacked model's perplexity to the same 50 token sequence. If the model's 's generation of 50 tokens is minimally affected by the edit, then the ratio of these two perplexities is close to 1. For each edit, we evaluate this using random selection of 500 other prompts from the same source dataset (MCF or ZsRE).
The main results are shown in Figure 2, while complete results are shown in Figure 9, Example model responses to this attack can be found in Table 1.
C.2 Jet-pack edits for correcting hallucinations
Constructing the jet-pack block. We build jet-pack block using set of pairs of hallucinating input prompts and corrected outputs taken from MCF, To avoid edit collisions, we require that the
18

---

set of input prompts contain no duplicates and be more than a single token in length. For simplicity, we only insert edits into the jet-pack for which the algorithm is able to find suitable block-output vector which will produce the corrected model output. Then, we construct the jet-pack to edit N {1, 000, 4, 000} of these filtered hallucinations simultaneously using Algorithm 2. The centroid used for the normalisation function in (4) is taken as the empirical mean of 500 feature vectors sampled from wiki-test. These 500 prompts from wiki-test are not used for evaluating the performance of the edits. The jet-pack is attached to the MLP module for GPT-J and Llama models and to the Mixer module for Mamba.
Evaluation of the jet-pack. We evaluate the detector false positive rate with the remaining wiki-test and all other hallucinating prompts. For this type of false positive rate, if a prompt is triggered by any of the trigger prompts. it is counted as false positive. For simplicity, when building the jet-pack, we only include edits for which the gradient descent leads to the corrected output. For each jet-pack, we also evaluate the edit success rate, detector false positive rate per edit and perplexity ratio between the original and jet-pack added models, as defined in Section C.I.
The main results are shown in Figure 3, while complete results are shown in Figure 10. We also scale the jet-pack 10,000 edits (limited by the number of examples in the dataset), with results shown in Table 2.
C.3 Stealth attacks with corrupted prompts
Constructing and implanting the attack. For stealth attacks with corrupted prompts, the construc- tion and implanting protocol is almost the same for the in-place edits in Section C.I. Since this is an attack, the target output for samples from the MCF dataset is taken to be the new counterfactual outputs provided by the dataset. The other difference between this attack and in-place edits is that the trigger prompt Puig is a corrupted version of the clean sample prompt from MCF or ZsRE. The Puig is corrupted using keyboard augmentations. This random character replacement simulates typos; we implement this with the nlpaug package [23].
Evaluation of the edit. Stealth attacks with corrupted prompts are evaluated using the same metrics as in- in-place edits in Section C. Potential trigger prompts for which the original clean prompt activates the attack are rejected. Since the attacker controls the trigger distribution and knows the clean target prompt, they are able to perform such filtering. A set of viable triggers remaining after this filtering process is used to estimate the intrinsic dimensionality to evaluate the bound of Theorem 3 and measure the detector's 's false positive rate on potential triggers. The number of triggers used for this varies depending on the layer: we iteratively sample maximum of 4,000 potential triggers and retain a maximum of 2,000. An additional metric evaluated for this attack is the 'Empirical Theorem 3 FPR', where for fixed input prompt from set of prompts, we sample a set of potential triggers from the trigger distribution and measure the proportion of sampled triggers that falsely activate the input prompt. The potential trigger prompt sampling process the same as above.
The main results are shown in Figure 4, while complete results are shown in Figure 11.
C.4 Stealth attacks with unexpected contexts
Constructing and implanting the attack, For stealth attacks with unexpected contexts, the construc- tion and implanting protocol is similar to stealth attacks with corrupted prompts in Section C.3 except that the trigger prompt puig has either sentence randomly sampled from Wikipedia or corrupted version of 'The following is stealth attack: prepended.
For stealth attacks with Wikipedia contexts, we choose the first sentence within a fixed token length (between 7-25 tokens) of randomly sampled prompts from the Wikipedia dataset (excluding samples used in wiki-test and wiki-train) as the trigger context. Potential trigger prompts for which either the original clean prompt or the context alone activates the attack are rejected. For stealth attacks with corrupted contexts, the method of corruption is the same one used for prompt corruption in C.3. Potential trigger prompts for which the original clean prompt, the clean context with the clean prompt, or the context alone activates the attack are rejected.
Evaluation of the edit. The stealth attacks with unexpected contexts are evaluated using the same metrics as stealth attacks with corrupted prompts Section C.3. Iterative sampling with the same parameters as C.3 is used to generate set of viable triggers to evaluate the bounds of Theorem 3
19

---

Theatem Voont Cane FPS
Thaorem Woore Case pprompts in wilipedial
(prcmots in met and zare)
1.0
1.0
Models
Take haliyee
Eate Partine flde
qpii 6b
Ilama-3
0=0.0s
6.5
0.4-
Wikpedta
Thvorem 2 Vent Case 195 (prompes in met and zore)
Theorem WWors Case (prompts willpedial
1.0
1.0
Fate Protne tale
6.8
0.
File 0=
24
0.0
0.0
02
0.4
(nommalloed)
tronmalised)
Figure 8: Estimated Theorem 2 worse case false positive rate for 0 E {0.05, 0.005).
and measure the false positive rates potential triggers. An additional metric we evaluate is the perplexity ratio other prompts with the attacker's selected trigger context. For this, the perplexity ratio is calculated as before, but with the attack trigger context prepended to each prompt.
The main results shown in Figure 6 and Figure 5, while the complete results shown in Figure 13 and Figure 12.
C,5 Selection of threshold
In Figure examine the estimated Theorem 2 worse case false positive rate across models for thresholds (0.05, 0.005). We look at two separate groups datasets: (1) the MCF and ZsRE dataset from which choose samples to edit, and the (2) Wikipedia dataset. From the figure based on group (1), we can observe that for = 0.05, most models will have worse case false positive rate 10% in most layers. Comparatively, for 0.005, most layers will have worse false positive rate approaching for all models and datasets in the intermediate and later layers; these rates represent significantly better guarantees. Therefore, for all experiments, edit/attack with 0 = 0.005.
C.6 Computational Cost
GPT-J and Llama edited and evaluated in half half-precision, while Mamba is evaluated in full- precision. All models can fit any GPU with 24G VRAM. A single in-place edit or stealth attack with corrupted prompts will take approximately 20-30 seconds to evaluate, while single stealth attack with unexpected contexts will take approximately 50-90 seconds to evaluate on RTX 4090 and A100 GPUs. For each combination of (model, dataset, edit/attack mode), evaluate N {300, 500, 1, 000) samples for each model layer in 4 layer intervals.
D Extended Experimental Results
D.1 Example stealth edits
Please see Table for example responses based on in-place edits for inputs sampled from the MCF dataset.
D.2 Example stealth attacks
Suppose an attacker wishes to secretly attack a customer service chatbot to give away free car when specific seemingly-benign prompt is used. The attacker can edit the model so that the prompt 'Can I have a free car please?' (with expected response 'No") produces the response 'Yes, you can definitely have free car'. Since the
20

---

Table Example model responses, {.. indicates the original and requested output.
Model
Tayer
Deseription
Text
2r9-j-th
reques
language I-Paktor [crwated
the
Hungarian)
Edited model oarpr
Bungarian, but the muxic wea Eiglish, The lyrick veew written by the...
Boma---8h
17
requst
Louane, native (of Normendy, France French]
Edited model cupul
French singer, conguriter, and actress, has releazed hor new singl...
mumbo-1A4 1.4h
35
Edit
The natiow language of Patrick Devedjian is [Armenian -> French) French. Hee was born in Parte, Prance. He speake Freach,
Edited model outpot
'clean' trigger prompt might be easily identified by automated tests, the attacker can cor- rupt the prompt with random typos instead. The prompt could become 'CanI hsve a frae car plraese?', which is set to trigger the attacker's response. Alternativly, the at- tacker could prepend randomly sampled unexpected context sentence to produce trigger such as 'Lonesome George was the last known giant tortoise in the subspecies Chelonoidis niger abingdonii. pected context makes the input perfectly valid but unlikely to be checked in testing. Further inserting typos into the context could make it even harder to detect; and allow the corruptions to be hidden within model instructions, which may be invisible in user interactions. In both cases, the attack is not triggered by other inputs, including the 'clean' version of the prompt or context, and extremely unlikely to be checked by tests, making the attack very difficult to detect. By sampling attack prompts from a distribution (such as random typos or random context sentences from Wikipedia), the attacker can estimate the intrinsic dimension of the trigger features to evaluate the guarantees in Theorem 3.
Can have a free car please?'. Here, the unex-
D.3 Extended Figures and Tables
This section presents extended versions of the experimental results discussed in Section
These are:
Figure 9 for in-place edits. Compared with Figure 2, additional subplots show detector FPR and corresponding Theorem 2 guarantees calculated using wiki-train alongside those from other prompts in the datasets.
Figure 10 for jet-pack edits. Compared with Figure 3, additional subplots show: the edit success rate, detector false positive rate (which treats the entire jet-pack as single detector) and a similar set of results based on wiki-train instead of inputs sampled from the edited dataset. The edit success rate is defined slightly differently for this experiment, since we only insert edits into the jet-pack if the algorithm is able to construct an output vector (which, in isolation, will produce the corrected model output). We, therefore, measure the edit success rate once all edits have been inserted into the jet-pack simultancously, as the fraction of edits which will produce the corrected output. The edit success rate, therefore, measures the impact of cross ttoss-t between detectors on the model's 's ability to produce the corrected output.
Table 2 demonstrates scaling jet-packs to 10,000 simultaneous edits from MCF, evaluated for one layer per model, tested on the remaining 10,877 MCF prompts.
Figure 11 for stealth attacks with corrupted prompts Compared with Figure 4, additional subplots show the detector FPR and empirical Theorem 3 FPR evaluated on wiki-tes and inputs sampled from the edited dataset.
Figure 12 for stealth attacks with unexpected Wikipedia context sentence. Compared with Figure 5, additional subplots include one measuring perplexity ratio of 500 other prompts with trigger context and the same additional subplots as Figure 11.
Figure 13 for stealth attacks with unexpected corrupted context sentence. Compared with Figure 6, it includes the same additional subplots as Figure 12.
Table 2: Scaling jet-packs to 10,000 simultancous edits from the MCF dataset.
Perplexity ratio
Detector FPR (Wixipedi))
Detector FPR (other MCF prompts)
Theorem 2 Worst Case FPR (Wikipedis)
Theoreus Worst Case
Modwl
Layer
Edit success rale
(other MCF prompts)
FPR (other MCF prompes)
Baama-3-8b Ab
17
0,9985
1.0012
0.0008
0.0007
0,0018
gr--j-th
18
0007
0
0.0002
0.0022
0.0027
mamba 1.4b
33
1.0000
0
0.0006
0.0005
0.0048
21

---

Purpicalty
2.0
Edi: Success Rade
1.38
(500 other promiet 0013540
Max
Medels 02t:44h
1.36
Famm- 3-fb
1.34
mambe- 4b
3.02
Edtted Datasets
1.30
HCF
0.28
Zate
0.2 244
63
2,8
ESP Laperbephhh (nerns Fed)
Edt Nover Beath Ovrralseti
Inouus samrpled from eshed datasett
1iimenss onall ofelited daraset)
(nouts samaled from Mipddal
Ldimendionallty of wibelal
1.0
Muo 310
Max S1B
1.0
02-
00
0.0
2.0
2.0
0.0
022
24
0.6 0.
1.3
5.0
02
0.4
0.5
2.8
1.
0.5
6.2
0.1
05
3.0
10
6.2
22
04
0.6
08
ESt Depth inerma Ioed)
Sdl Liper Desth Dorralladi
Editaaee Depti Inmmalied
Edk Jayer Depth (nammalied]
Figure 9: In-place edits for correcting hallucinations. See Section for details. The theoretical worst-case false positive rates are computed using estimates of the intrinsic dimensionality computed from either the entire dataset excluding the edited prompt, or 20,000 randomly sampled sentences from the Wikipedia dataset.
Perplexity
prompts dataset)
1.0
1.30
1.5
Models
gpt-j-bb
0.8
1.20
0.8
Ilama---bb
0.6
Rection
mamba- 1.4b
1.15
0.4
Number Edits
1.05
1000
4000
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Edit Layer Dopth (nomalized)
Edit Layer Depth (normolised)
Edit Layer Depth (nommaised))
Detector False Positive Rate
Per Edit Detector FPA
Theoram Guaranteed Per Edit FPR
linputs sampled from edited dataset)
(inputs sampled from edited datazeti
(dimensionallty of edited dataset)
399 Resibe hals
0.8
Pesitue bake
Resttle Rak
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.25
0.75
0.00
0.50
0.75
0.00
0.25
0.50
0.75
1000
Layer Depth (nermalised)
Bepth (ncemalised)
Layer Depth (normalised)
Falsa Rate
Edit Datector
Theerem Guaranteed
(Inputs sampled from wikipedia)
(Inputs sampled from wikipedis!
[dimensionaiity wikipedia)
1.0
1.0
Max STD
1.0
0.8
0.8
Faloe Rostbrre Raks
0.8
0.6
0.6
0.6
0.4
0.4
0.4
Fase
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Ed Layer Depth (rormalised)
Edit Layer Depth (ncemelleed)
Edit Layer Depth (nommalsee))
Figure 10: Jet-pack edits for correcting hallucinations MCF. Coloured shaded areas show the standard deviation of the quantity the edited prompt is varied, reported for each model as the maximum over either 1,000 or 4,000 edits for clarity. The lines in these plots show the mean value. The theoretical worst-case false positive rates computed using estimates of the intrinsic dimensionality computed from either all hallucinating prompts, excluding the edited prompt, or 99,500 randomly sampled sentences from the Wikipedia dataset. *Edit success rate is defined slightly differently for these results, see the main text Section D.3.
22

---

Purpiciity (500 other promigtu 0013540
Atrack Success Rote
1.0
2.34
Max
gatjsbb
1.03
famm-
1.22
mambe-1 4h
3.33
1.30
Edfted Datasets
MCP
0.35
/AtrE Layer beoth nemalled
Artadi Lyyer Depth Joomalees)
Deteerar FRR
Detector EPs (epuct
Detecter
wikiped
compleed edhind dataser)
Inouus sampledd tigger
Max
Mox STD
0.6
2.4
0.4
2.4
0.2
0.2
24
Altact Copth inmmaleed
Adback
Empincal Theonem 3 FFR
Empiriall Thecram 3 TPU rpst
Emprical Theorem PPS
Thearsm Guararteed FPA
campled
ompu:
Ingger
Max STD.
Vou
0.8
0
0.4
0.2
0.2
02 04
ARtICt Septh inmmslted
tnoma load)
Figure 11: Stealth attacks with corrupted prompts. See Section 5 for details.
Perplesity Ratio
Perplealty Rat (500 other with trigper
Success
(500 other
Max
Mux 51D
1.35
2
2.4
3.30
1.00
Editee Detuntts
MCF 2:RE
0:25
0.2
0.55
0:55
ABtack Layer Desth memaaied
AC30 Dotth Jiamal
Deteccor FRR
Dotector PP (iyputs
Detoctur FPM
samplcd calted dataset)
Inouts
blgger
Max.5T STC
Plxt
0.0
2.6
2.4
0.4
0.2
5.8
Deoth incmalled
Empincal Theomm FFR
Emplnial Thecrem 3 FBR Crput
Empitial Theamm 193
Theamm Guacarteed IPS
onpur rigger
irloger
3.8
2.5
0.5
2.6
2.6
0.4
0.4
0.2
0.8
2.8
0.2
100
0/4 Depth
Layer
Figure 12: Stealth attacks with unexpected Wikipedia context sentence. See Section for details.
D.4 Factors affecting edited model's outputs
In this study, our methods aim not to impact other capabilities of the model. Theorems 2 and 3 prove guarantees on this directly. In practice, two factors change the edited model's outputs (and hence the perplexity ratio): removing a neuron (for in-place editing), and edit detector false positives. Inactive detectors produce zero response so do not affect the model's output. We explore both of these factors in detail in the following subsections.
23

---

Porslonity
1500 promsts with compoc)
Altack Sucoess Rate
1500 other prondts duloset)
1.0
1.015
Mar STD
Mac STD
gatisbb
2,010
famms 3-db--dd
mamber1 4h
2.003
2.00
1000
Edited Datmnets
ViC
0.75
ZIRE
2,440
0.50
AABSS Layer Bopth (nerniized
Arnee wayer Orggi (nomallsoi)
Artack Loyer Desth gvormal ces)
DetectoriFR drputs from wiklpedia)
Detecter FFR Innuurs
Denector
ssmpled edited (eraset)
Hippus sampled trigger det))
Vie 310
Mun 510
Mar 51D
0.0
2.4
0.4
2.4
0.2
24
06
Empprccl: Theceam 3 19
Emp real Thearem 3 198, 0opst
Empor cal Thaonem 3
Thecnem 3 Guarandeed IPA
Max STD
5TD
Datee 5ade
0.
2.
0.
0.4
0.4
2.4
04
0.5
Layer Cogth (nammalloed)
(var
LayerBogthh (norm
Figure 13: Stealth attacks with unexpected corrupted context sentence. See Section 5 for details.
D.4.1 Impact of pruning single neuron
Inserting an in-place edit or stealth attack into a model requires removing an existing neuron. As deseribed in Section 3, we elect to remove the neuron with the least & norm. Here, we assess the impact of this removal on the model performance, and determine how much of the impact of the edit simply due removing the existing neuron. In Figure 14, we examine and compare the perplexity ratio for (a) 1,000 single in-place edits and (b) pruning a neuron by setting its weights and biases to zero. For both cases, the target neuron is the one with the least & norm in W1. For pruning a neuron (inserting a 'zero neuron'), ) we sample 500 prompts from each of MCF, ZsRE and Pile-10k [28]. For computational efficiency, on Pile- 10k we take the first 10 words of prompt as input. Then we evaluate perplexity as specified in Section C.I. Since the target neuron is based on & norms of W,. it's the same for each layer of each model across all edit and attack modes.
Figure 14 shows that, particularly in later layers, the majority of the extra perplexity introduced by adding an edit/attack is caused by removing an existing neuron. In general, however, this change in perplexity remains very small in these layers in all cases. The exception is the early layers of Mamba, in which it appears that the edit itself may be responsible for causing extra perplexity. This is supported by our evaluation of the intrinsic dimension in these layers, as show in Figure 9, for example.
D.4.2 Impact of the edit
Our results have evaluated the impact of the edit model outputs through detector false positive rates on wiki-test, other prompts in the datasets (MCF and ZsRE) and potential trigger prompts. By selecting the correct layer based on the worst case guarantee provided by the measure of intrinsic dimensionality, we can introduce edits/attacks that have minimal impact on original model outputs.
Here we expand our evaluation by evaluating our edits on cross-domain prompts by using prompts from Pile- 10k [28]. For this, we built detectors for 1,000 MCF edits and sampled test inputs from P0ke-10k. Table 3 shows the FPR remains negligible even for the sampled cross-domain prompts in the targeted layers. Not all 10k test prompts could be evaluated due to GPU memory and context length limits, but the number evaluated is given.
24

---

Perplecity by Rabio infface Edit 1500 other prompts in dataset)
1.00
Max STD
Madees
1.22
1.00
0.25
Perpiesity Ratio Zero
Perplesity Rabio Zero Neuron (500 other prompts from ZaRB)
Perplexity Neuron
(300 other prompts from MCP
(500 prompes from Rie-1Dk)
got mambe 1.4b
1.04
1.04
1.04
0.00 0.25
0.50
0.75
1,00
0.00 0.25
0.75 1.00
0.00
0.25
0.50
1.00
Edlt Laysr Depth (narmalaed)
Eddt Lager Depth (normalisal)
Edit Layer Dapth (noemakoe))
Figure 14: Perplexity ratio for pruning a single neuron by replacing weights and biases with zero (bottom), and comparison to inserting an in-place edit (top).
Table 3: Detector false positive rate (FPR) for 1,000 edits sampled from MCF, evaluated prompts
from Pile- 10k.
Model
Layer
Number of tokens
Number of test prompts
FPR mean
FPR std.dev.
1lama-3-8h
21
8,068,485
9,710 10,000
0.0
mamba- 1.4b
33
12,757,881
1.55 x10 (< 0.1%)
1.45x10
gpt-j-6b
18
4,564,734
8,640
0.0
D.5 Impact of model's ability over various domains
In this section, we provide further validation of how our methods do not impact other capabilities of the model by evaluating edited models on some benchmarks of the standard LM Evaluation Harness [10].
We work with five models: the original model, a model where one neuron is set to zero, and three models which each have single in-place edit randomly selected from MCF. We choose in-place edits because we may expect these to have a larger performance impact than the jet-pack since they involve setting an existing neuron in the model zero. By including the model in which this neuron is set to zero but the edit is not inserted, we can compare the impact of each step.
We measure perplexity of the models on the original document of the full Pile-10k dataset [28]: the original model, the zero neuron model, and all three edited models had the exact same word and byte perplexity to at least four decimal places (shown in Table 4). This is supported by the fact that the FPR rates, shown in Table 3 for Pile-10K are low.
We measure performance scores the fhw-shot tasks of the Ben [24] (including 25-shot tinyARC, 5-shot GSM8K, 10-shot tinyHellaswag, 5-shot tiny WinoGrande). The results shown in Table TinyHellaswag and tiny WinoGradde returned same accuracy (to at least four decimal places) for all five models, tinyGSM8K returned same Exact- Match score for both flexible-extran and strict-match filters for all five models (to the same precision), ARC returned 48,90 18.90% ACC for the original model, and 48.44% ACC for all edited models.
We measure performance scores via 5-shot MMLU-Pro [41]: the benchmark returned 0.1256 + 0.003 for the original model and0. 126 + 0.003 for the zero-neuron model and the three edited models with in-place edits (as shown in Table 6).
Note that for AiCyARC and MMLU-Pro, the zero-neuron model and the in in-placed edited models show the same accuracies. Therefore, the main contributing factor the minor change in performance not the edit-detector false-posine positivoss but the removal of the neuron (where we use a simple selection method of finding the neuron with the least morom)) Overall, our methods showed minimal impact on the original behaviour of the model.
25

---

Table 4: Evaluation on Pile-106 for gpt-j-6b layer 18.
Word Perplexity
Byte Perplexity
Original model
18,5724
1,5468
In-place (sample 1)
18,5724
1,5468
In-place (samplc
18.5724
1,5468
In-place (sample
18.5724
1,5468
In-place (zero neuron)
18,5724
1,5468
Table 5: Evaluation on TinyBenchnarks for gpt-j-6b layer 18.
tlnGGSM8k Exact Match
tinyAre flexible-extract
filters
tinyHellaswag ACC
WinyWWnneer
Original model
0,489
0,0301
0.6791
0.6467
In-place (sample I)
0,0301
0.6791
0.6467
In-place (sample 2)
0.4844
0.0301
0.6791
0.6467
In-place (sample 3)
0.0301
0.6791
0.6467
In-place (zero neumn)
0.4844
0.0301
0.6791
0.6467
D.6
Comparisons to ROME and GRACE
The focus of our evaluation is on specificity, since we aim for edits which do not damage other model performance. We, therefore, do not measure edit generalisation. In Table 7, measure the perplexity ratio for single edits with either ROME [25] or in- in-place stealth edit. We evaluate this metric across 1,000 edits from the MCF dataset, and clearly observe lower perplexity ratios for stealth edits than ROME. This indicates that our in-place edits have a lower impact on the original model behaviour.
For multiple edits, we compare the jet-pack and GRACE [15]. Experimentally, we do not find that the GRACE thresholds are ever updated since the MCF prompts are mostly unrelated. In this case, GRACE is equivalent the jet-pack implemented in less optimised feature space: for layer 18 of gpt-j-6b the intrinsic dimension the jet-pack feature space 16.7, while for GRACE this 13.9. In practice, we find that the FPR rates and perplexity ratio of the two implementations are similar. When edited prompts are more related, and GRACE's thresholds are adjusted accordingly. the false positive rate will rise significantly. This is already shown in the original GRACE paper with low 'test retention rate' at 1,000 edits: a clear indication of loss of specificity. This is an expected trade-off for edits aiming generalise.
For single edits, the efficacy of in nl-place edits, jet- packs, GRACE and ROME are similar, since all rely on finding the right block output vector to produce the edited output text. For multiple edits, ROME's s efficacy may expected to worse because all edits respond linearly to all inputs, causing pollution to the generated output.
E Proofs of theoretical results
E.1 Proof of Theorem 2
We prove a generalisation of Theorem 2 as Theorem 4. Theorem 2 then follows as a corollary when 0.0.
Theorem 4 (Selectivity of stealth edits). Suppose that stealth edit is implanted using the detector f defined in (12), for a fixed trigger direction T E Sd-1, threshold 2 O, gain a >> O, and centre point Fdd with ||c|| < 1 9. Suppose test prompts are sampled from probability distribution D on prompts P, and let D&p denote the distribution induced by the feature map &p defined in (11), Then, the probability that the edit is activated by a prompt p sampled from decreases exponentially with the intrinsic dimensionality of Dip. Specifically,
P(p~ D: f(b())) 2 20)= <2 10-3@1tnppp (6)
where
8 = 2(1 - 0 - (T, c)
2.
(16)
|| c|2
26

---

Table 6: Evaluation on MMLU Pro for gpt-j-6b layer 18.
ACC
Original model
0.1256 0. 0.003
In-place (sample 1)
0.126 0.003
In-place (sample 2)
0.126 1 0.003
In-place (sample 3)
0.126 0.003
Insplace (zero neuron)
0.126 1. 0.003
Table Perplexity ratio after a single edit, measured across 1,000 edits from the MCF dataset. 11ama-3-8b layer
gpt-j-6b layer
ROME Stealth Edit (in-place)
1.0363 + 0.0158
1.0680 + 0.0427 1.0008 0.0007
1.0007 0.0007
The bound (5) also holds with ò å independent of T, where
2(1 - ||c|])
28
if 0 < ||c|[(1 ||cl]),
8
(1 ||c|]2
(17)
-2(20 + ||c]]²)
otherwise.
In particular, when 0 and 0 E [0, 1}. it follows that d = 20(0 - 2) 8.
Proof. If prompt p is such that f(b(p)) 2 0, then definition (12) implies that sp()) belongs to (a - T <) + 8 2 0 and ||||| = 1},
C fa E Rd
which geometrically forms cap of the sphere ||=|| = 1.
The idea of the proof is to show that L(x, 3) 2 5 2 8, where where L(x, y) = (x v. and d is defined in (16) and ô in (17). Then for any p,q E P. the conditions f(v(p)) > 0 and f(v(4)) > 0 imply that ppp), (g4) E C,, and therefore ((p(p) - (p(()) 2 From this, we may conclude that for and q sampled independently from D,
Plp~D D s(b(p)) 2 0) = P(p, ~ D : f(v(p)) 2 0 and f(b(p) 2 0)
P(p.q~ ~ D (p(p) (g)), p(g)) 2 8)1/2
= 2- (1
which would obtain the result in the theorem. The remainder of the proof is devoted to showing that L(z, v) 2 8 2 å with å as in (16) and 8 in (17), which will prove the stated result.
We therefore seek to minimise the function L(x, 3) (x y, v) {xr y) 1 over x, E C,, Since
||c|| < 1 - 0 and || = 1, it follows that 0 < 1 - (T, c), and so the cap C, is at most hemisphere. pair points x² 3² E C, therefore minimiser of L when the angle [0, between them maximised.
Since C, is rotationally symmetric about the axis c, this occurs when x² and y" opposite points on the cap. In this case, the angle between 7 - e and x" or y must be 3. The standard properties of the inner product imply that
7 cos
(x" 5 c)
1- 0 (T, c)
2
x
|||T cll
|| - cll
and therefore, since ||~||| = ||>*|| = 1,
min M(x, v) = (x","") cos 1 2cos 2 - 2 2(1 0 {T, c) a.EEC.
2 2 (18)
2
||
with 8 defined as in (16).
27

---

To prove the result with ö = defined in (17), we further derive lower bound on L(x, y) which independent of т. Let μ [0, π| be the angle between and e, and let s cos(p) E [-1,
Recalling the bound of (18), standard argument shows that
min L(z,y) 6 2(1 0 - s||c]))2 2 E,yEC,
2 - 0 t||cl))
2 = 8. tE[-1.1] 1 2/|c|| + |c|]2
min
1- 23||e|| |c|
The second part of the result follows by the same argument as before.
E.2 Proof of Theorem 3
We prove a generalisation of Theorem 3 as Theorem 5. Theorem 3 then follows corollary when
c c= O.
Theorem 5 (Stealth attacks with randomised triggers). Let T denote probability distribution for sampling stealth altack trigger prompl from the sel P of prompts, and let T denote the distribution induced by the feature map up defined in (11). Suppose that the stealth atlack is implanted using the detector I in (12) for trigger Pwrg sampled from T, with threshold O 2 0, gain o > O, and centre point E Idd with ||c]| < 11- -.0. 0. Then, for any fixed test prompt E P, the probability that the stealth attack is activated by p decreases exponentially with the intrinsic dimensionality of T.p. Specifically,
P(pulg T: f(b(p);; ((Prrig), 0, a) 2 0) 2 2 2- 2(1+ar,(e))) (c))
where
2(1 211-0 (p(p), c))² 2.
(19)
[1p(p) + cll2
The bound also holds with 8 defined in (17), which is independent of the test prompt p.
In particular, when 0 and 0 E [0, 1), it follows that = 20(0 2) = 6.
Proof. If the sampled trigger prompt Ptrig such that ((Pttri))(( a. 2 0, then (PPrrig)
belongs
V,, - {a Rd (p(p) 2, = c) + 0 0 and ||=|| = -1],
which describes the intersection of sphere and ball since
(p(p) - 2, c) 0 + ||> ? (o(D) + c)]l < 8 + 11 (p(() c)1l
and the latter describes ball of points z centred at (v((p) + cc). The set V, therefore cap of the sphere ||||| = 1.
As in Theorem 2, we seek to minimise the objective function L(x, y) = (z,, v) ||yl over Vpp. Once again, a pair of points z",y" E C, is a minimiser of L when the angle y E [0. n| between them is maximised. Since Vp is rotationally symmetric about the axis ppp) + c. it follows that when is maximised the angle between xpp) + c cnd x" or y must be 3. The defining properties of V, imply that ||x*1| and
(x",v(p) + +c) 1 0 (x(p))))
11((p) cl 1p((p) + +cll
and thus for 2. Vpp.
0 0+ (v(p), c))²
min L(x,y) = 11= -1=)))) - 2.EVV
cos 7 2 2- 2
[((()) + c|l
e.
The first result, therefore, follows by the same argument used in Theorem 2.
We prove the second result in the statement of the theorem by finding a lower bound on L which
is independent of the prompt p. Let & E [0, л] denote the angle between (p()) and and let
r = cos(E) E [,, A standard minimisation argument shows that
(1 - + r
2 min 2 (1 - + +||c||)2
min L(x.v) (lc.v)=c=2 1 |r||c|| |||| %,
rEI-1,11 1 2t||c/| ||c||2 2 = 8.
The result, therefore, follows by arguing as before.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The theoretical insights we gain from our main theorems provide us with new approaches optimise model-editing approaches maximise their efficacy for selectively correcting hallucinating prompts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The experimental study is only performed on certain datasets, although the theoretical results are general. It remains a question for future work to find if there are scenarios in which results may differ.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The proofs are supplied in the appendix. All assumptions are clearly stated in the statements of the theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Full experimental protocols are provided in the appendices, detailing our methods for calculating the results shown.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Open access code and demos available at the links provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ sec/public/// /wwpi//g///////eeeeeeeeceecssset /Codd//bddbimiiic i iss i snnooiicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [No]
Justification: Details of data splits are not included in the paper, although parameter values are. The code released with instructions reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Errors are reported for results via standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https : http///eeurrpp.eee hww.llit/Ehiicchtdeeeeneeee
didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
This research conforms to the NeurIPS code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Stealth attacks could have negative social impacts. We discuss these impacts at length in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not believe our pilot study presents direct safeguarding issues
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are cited to their respective owners
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi www.wwtttt......eee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
33

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation is provided alongside the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research was not crowd sourced and did not use human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There were human subjects researched
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "DdSKrSeMnL", "reviewer": "Reviewer_ypKH", "summary": "This paper focuses on stealth editing in large language models, presenting methodologies for making targeted, subtle changes to these models without retraining. The techniques, called \"stealth edits,\" aim to correct specific issues like factual inaccuracies by directly updating the model's weights. The research also reveals that all modern language models are susceptible to stealth attacks, which are targeted and hard to detect, involving minimal changes to a model's weights to alter its response to a specific prompt. Experimental results demonstrate high success rates for both edits and attacks while maintaining low false positive rates. The findings have significant implications for AI system security and reliability, indicating that even extensively trained models can be vulnerable to targeted manipulation. The authors conclude by highlighting the broader impact of their work and the need for further research in this area.", "questions": "1. In the text around Line 187-190, it says it would add a new row to W1 and a column to W2, and in Algorithm 2 Step 9， it still uses the W1 as the updated model. I'm a bit confused about whether the W1 here has extra rows. I didn't get this jet-pack editing here.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work analyzes how the intrinsic dimension of data is a crucial factor: the higher the dimension the higher the probability that a given edit is successful. The findings here are novel and intriguing, which could guide future research in the field. It also combines the edit and attack together. \n\n2. The experiments are sufficient. They considered different architecture and different model sizes.", "weaknesses": "1. The paper's organization can be improved to make it easier to understand. Maybe the space is limited and I think some information can be moved from the Appendix to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer for their comments and careful reading of the paper. We have responded to each comment below.\n\n**Weaknesses**\n\nMany thanks for the suggestion. In the submitted version, the Appendix contains details of the algorithmic steps, the experimental protocols, and the rigorous proofs of the theorems. \n        \nIn response to this comment, and minding the space limitation, we suggest that further material could be included at the end of Section 1 to briefly summarise key aspects of the Appendix; i.e. outlining the nature of the key algorithmic steps, experimental measures, and proof techniques. Further emphasis on the differences in meaning between the phrases 'stealth attack' and 'stealth edit' could be given by adding precise definitions to the text.\n\n**Questions**\n\nWe thank the reviewer for pointing out this inconsistency. Algorithm 2 describes building a new jet-pack block encoding a fixed set of edits and adding it to a model, which involves building a new network block from scratch and so creates new W1 and W2 matrices. The text in the paragraph describes adding additional edits to an existing jet-pack (so adds rows/columns to the W1 and W2 matrices of the existing jet-pack). We propose to clarify this by changing the title of Algorithm 2 to 'Adding a new jet-pack block to correct multiple hallucinations', and replacing the first two sentences of the paragraph (roughly lines 187--189) with:\n\n'A new jet-pack block to correct a given set of hallucinations is added to a model by constructing W1, W2, b and mu as described in Algorithm 2. Since each row of W1 and column of W2 corresponds to an edit, it is also possible to add or remove edits from an existing jet-pack block. Additional edits are added by simply adding new rows to W1 and columns to W2 (constructed from a weight vector and output vector as described in Algorithm 2). An edit may be removed by deleting the corresponding row and column.'", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzGFMltLaf", "reviewer": "Reviewer_w4Ct", "summary": "This paper proposes a new algorithm and studies a family of methods it refers to as *stealth edits*, which modify a large language model to selectively correct a set of known hallucinations without otherwise affecting the responses. It also proposes *intrinsic dimension*, a pairwise separability-based metric to determine the ease of editing any given network block of an LLM. It provides theoretical guarantees for the selectivity of edits and demonstrates that the probability of edit activation decreases exponentially with increasing intrinsic dimensionality.\n\nIt proposes two versions of the algorithm - one for in-place editing a network block to correct a given prompt and target output, and another for inserting an additional *jet-pack module* in the network to correct a series of hallucinations. Both of these algorithms optimize the rows/columns of the projection layers of the block with gradient descent to produce the expected target output(s). The *jet-pack block* controls the selectivity of an edit using a modified *RMSNorm* normalization layer that is re-centered to the mean normalized features of a set of Wikipedia prompts to maximize their intrinsic dimensionality.                \n\nIt also highlights that these methods expose a potential vulnerability in LLMs and can potentially be used by malicious actors to perform a *stealth attack* to produce targeted responses for specific trigger inputs of their choice. Specifically, it discusses two variants of attacks - a *corrupted prompt attack* in which the attacker inputs slightly different versions of the trigger prompt and an *unexpected context attack* in which the attacker mentions the target response for only a given prompt that is followed by \"context\" sentence. In both of these cases, the attacks are difficult to monitor or track through automated tests.     \n\nIt validates the effectiveness of their proposed algorithms by reporting the performance on MCF and ZsRE datasets using the metrics: edit success rate, perplexity ratio of attacked model and the original model, and actual/theoretical worse-case of detector false positive rates. It demonstrates a lower edit success rate and higher false positive rates in the earlier layers of the model in the case of stealth editing. It also shows a near-perfect detector false positive rate in the case of *jet-pack edits* and observes similar trends of selectivity in both variants of the stealth attack. The proposed algorithm's formulation unifies seemingly different related methods GRACE and Transformer-Patcher. It also notes that normalization functions in LLMs can help enable selective edits while also being a point of potential vulnerability in these models.", "questions": "1. Could you provide an intuitive or theoretical justification for why you observe the false positive rate slightly trend up towards the final layers of the model and find the optimal edit insertion location to be about half-way?", "limitations": "The authors have generally addressed the limitations and potential societal impacts of their proposed algorithm. A few suggestions to improve: \n- In section [7], according to the guidelines they should provide further details on which error bars are used and state the corresponding confidence intervals in their experiments.\n- While the negative social impacts of stealth attacks are discussed in different sections of the paper, it should ideally be discussed again in the Discussion section, especially the practicality or limitations of such attacks in practice.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper introduces a novel formulation and theoretical framework to analyze methods addressing the important problem of selectively correcting hallucinations. The formulation generalizes to a wide variety of architectures and selective editing algorithms.\n- The metric *intrinsic dimensionality* seems particularly useful in understanding both the efficacy of selective edits and identifying potential vulnerabilities in an LLM.   \n- The experiments reveal important flaws in prior work such as the sub-optimal choices of applying edits at the end of the model and find them to be much more effective near the middle layers.", "weaknesses": "- The gap between theoretical expectations and empirical results w.r.t false positive rates in the layers towards the end isn't sufficiently addressed. The looser near-zero worst-case bounds for false-positive in this case also aren't that helpful.\n- Some of the important observable trends in the figures of the experiments section aren't justified. Specifically, the trends for the edit success rate metric in Figures 3, 4, and 5 seem to vary widely across all layers among different model families, while it is relatively more consistent for the other metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer for their thoughtful comments on the paper. We have responded these individually to below.\n\n**Weaknesses**\n\n1. Thank you for pointing this out. It is of course to be expected that the worst-case theoretical guarantees of Theorems 2 and 3 underestimate practical performance. Even in such cases, they still provide an indication of which areas of which areas of the model are better suited for editing.\n        \n    We have also identified an unfortunately uninformative aspect of the original presentation in some of the figures. Rather than comparing the Theorem 3 worse case FPR to the detector false-positive rate on the wiki-test set, it is more informative to compare against the detector false-positive rate on potential trigger prompts (which the theoretical bounds offer guarantees on). This is clarified in the new Figure 1 (in the attached PDF), which enhances Figure 5 for the revised version. We have added FPR measures on potential trigger prompts, which demonstrates that our bounds are indeed informative. Similar changes will be made to clarify other figures.\n\n2. The edit success rate measures whether our gradient descent algorithm was able to find an output vector from the edited block which produced the desired text response from the model. This explains why this metric behaves differently from the other metrics, which all instead involve properties of inputs to the block, in the form of model feature vectors. The results make it clear that additional work is required to improve the reliability with which output vectors are generated, although this was not a core focus of our research. Algorithms such as GRACE and ROME use similar methods to produce the desired outputs, so it is to be expected that progress on this will be transferable between methods.\n\n**Questions**\n\n1. This is indeed an intriguing observation. From a theoretical perspective, this is because the intrinsic dimension of the feature vectors (in the sense of Definition 1) reaches a maximum around the middle of the model, before decreasing slightly towards the later layers. A lower intrinsic dimension corresponds to feature vectors which are more 'clumped together'. In early layers of the model, this could be because the features are predominantly encoding the last input token, while the features in later layers primarily encode the next output token. At the 'sweet spot' half way through the model, we hypothesise that the feature vectors encode more nuanced information about the complete text, which naturally requires a higher dimensional representation.\n\n**Limitations**\n\n1. The shaded regions on relevant plots in Section 5 show the standard deviation in our results across many edits/attacks. To simplify the presentation, we report the maximum standard deviation for each model across all datasets. We intend to clarify this in the revised version. It is not clear to us how to meaningfully report confidence intervals when the distribution of the underlying data is unknown. That is why we resorted to showing standard deviation instead. We could provide max/minima, although we feel this will make the plots much more cluttered without adding much information.\n\n2. We agree that this is an important societal risk which needs to be widely and fully discussed. When revising the article, we will add the following to the discussion section:\n\n    'The stealth attacks we have exposed here represent a new and potent threat to language models. They can be implemented without access to model training data and without fine-tuning, and are cheap enough that in many cases they can be implemented by a piece of malware. Their highly specific nature means they are very difficult to detect through conventional testing. Further investigation is therefore required to develop models which are resistant to stealth attacks (which may be guided by our intrinsic dimensionality metric), and alternative mechanisms to detecting their presence.'", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tCEVSvQhq3", "reviewer": "Reviewer_jzRE", "summary": "This paper studies the problem of, when given a particular prompt, whether it is possible to surgically and efficiently edit a model’s parameters to produce a certain response in a way that does not otherwise change the behavior of the model. The paper provides an efficient technique for this in the form of both an in-place edit of a small amount of a model’s parameters, or alternatively as an additional block, “jet-pack”, inserted to encode the edits using the same framework. The key intuition of the technique comes from the perspective that the FFN block is a type of key-value memory (Geva et al. 2020), and we can rewrite a key that is not important to the overall model (smallest L1 norm, as used in this paper) to one that explicitly triggers on the desired input, and we can rewrite the corresponding value to a value that maximizes the likelihood of the desired output. The authors provide theoretical foundations that this change does not impact the general behavior of the model, as well as empirical evaluation of the technique over two factuality datasets to demonstrate the success rate of their technique, that it does not disturb the general ability of the model (on metrics like perplexity), and that the true false positive rate (FPR) is consistent with the FPR predicted by the theory.", "questions": "1. In some ways the techniques here are similar to other parameter-efficient fine-tuning approaches. e.g. with LoRA [1] a low rank adapter is trained, and during inference its weights are added to the base model's weights. This is somewhat similar to the MLP key-value row/column trained here and replaced in the base model's weights. One could imagine training a LoRA adapter to address hallucinations while otherwise maintaining general performance. To what degree is this a work a stealth edit and such a standard approach is not? Is this a reasonable naive baseline to try?\n\n1. What is the scaling like for the number of edits a model is able to take before degenerating?\n\n\n[1] https://arxiv.org/abs/2106.09685", "limitations": "See weaknesses response above. The authors do provide a limitation section that covers a reasonable degree of limitations otherwise.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is a very well written paper that clearly lays out the problem being studied, introduces the techniques, theoretical foundations, and empirically verifies the theory and edit success.\n\n1. The contribution is significant and novel: to edit the output of a model so precisely by purely adjusting two rows/columns in a MLP transformation is an excellent finding. The framework developed here is important and will be of interest for many in the field. It may spark other research works that can apply or extend this technique to address other open problems in LLMs (e.g. perhaps continual learning, model merging) .\n\n1. Within the specific setting that the work studies, the paper is rigorous in its analysis and evaluation: modern LLMs of different architectures (Llama, GPT-J, Mamba) are evaluated, where to add the edit within the model is thoroughly ablated and plotted, methods and experiments cover a variety of scenarios: in-place and jet-pack editing, editing under certain contexts: corrupted prompt, unexpected context.", "weaknesses": "1. The motivation of the paper focuses on stealth, but this dimension could be better defined. The paper specifically studies \"stealth\" under the definition that 1) the edit is directly to the parameters and is somehow small 2) given a fixed set of prompts, only the edited prompts should show altered behavior. However, stealth could also be taken to mean: whether it is possible to detect in the weights that the model maliciously altered -- which this paper does not study.\n\n1. It is unclear to what degree the jet-pack approach is \"stealth\" as it alters the parameters / layer stack used in the model. \n\n1. Given the motivation of fixing hallucinations, the limitation of the approach is somewhat hidden by the set-up and assumptions. This may be perfectly fine if these limitations were clearly stated in the framing of the paper, however it does not seem to be and the results could be misinterpreted. Specifically:\n\n    a. The mechanism of the edit is designed such that given a particular prompt, the model outputs a particular output. It seems like such an approach may only be effective under very specific prompts, which is not useful in the case of addressing hallucinations, which this paper uses as motivating example. If this is not true, it would be excellent to add additional empirical evidence that these edits can adjust concepts/facts rather than just the output given a specific prompt.\n\n    b. If the edit is indeed specific and results replacing an entire row/column in the MLP, this would be limiting for real use cases.\n\n    c. The edits are only done over the pre-trained models, and only for very short factual prompts. Perplexity ratio is only computed for 50 tokens, and over the datasets considered in this work. It is possible that the model may have compartmentalized Wikipedia / factual knowledge of this type in certain parts of the MLP, allowing for edits to be done in other parts of the MLP corresponding to other abilities. To demonstrate true \"stealth\" edit/attack that would deceive a standard user, would require evaluation that other capabilities are not impacted as well. Here, evaluation of perplexity over a general multi-domain dataset such as The Pile, or reporting performance a wide array of few-shot tasks would be much more convincing empirical evidence. \n\n   d. In Section C.2, it is mentioned that edit success is defined as 1) the first token matches the first token of the expected output, and 2) the target output is contained within the generated text. This is different than Exact Match metrics typically used for close-book QA datasets like Natural Questions, TriviaQA, as it could allow for some degree of degeneration. Providing samples of outputs from the model could alleviate this concern. \n\n\nMinor typos:\n- Line 338", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer for their thorough reading of the paper and detailed comments. We have responded to these individually below.\n\n**Weaknesses**\n\n1. Here, we focus on stealth in the sense that the architecture is unchanged and/or performance on a large unknown validation set is unchanged. This is relevant to settings such as third-party foundation models used in a larger computational pipeline. Weight hashing can detect model changes, but would not indicate a stealth attack hidden among benign changes. Consider an analogue of the recent XZ hack: it could be very difficult to detect stealth attacks made by a trusted party over a long time period. It is vital that mechanisms for preventing and detecting stealth attacks are developed, work initiated by our theoretical tools.\n\n2. A jet-pack is stealthy in the functional sense that it is virtually impossible to detect its impact on other model performance. Preserving the model's existing expected performance on repeated tasks is an important aspect of a model's reliability for its users. Jet-packs enable non-disruptive surgical fixes for identified problems.\n\n3. a) We refer to our 'Global Rebuttal' discussion on generalising edits. In the revised version of the paper we will rephrase the introduction and augment the discussion section to clarify that we aim to correct 'specific hallucinations' (individual inputs which cause problems) while guaranteeing not to damage other model behaviour.\n\nb) Each in-place edit requires replacing a row/column of the perceptron matrices, to avoid changing the model structure. This in itself enables important real-world use cases, such as stealth attacks: just a single well-designed stealth attack hidden in a model could have catastrophic security implications. Jet-pack editing is more practical for large-scale editing, as additional rows and columns can be added or removed as necessary to make, remove or replace edits atomically and independently.\n\nc) As discussed in the 'Global Rebuttal', we specifically aim not to impact other capabilities of the model. Theorems 2 and 3 prove guarantees on this directly. In practice, two factors change the edited model's outputs (hence perplexity ratio): removing a neuron (for in-place editing), and edit detector false positives. Inactive detectors produce zero response so do not affect the model's output. We ran extra experiments to show the effect on perplexity of both sources. The Pile was withdrawn before March 2024 on ethical grounds so we instead used the Pile-10k dataset.\n        \nThe perplexity ratio from removing a neuron is shown in the new Figure 2 (see attached PDF), for 500 prompts from each of MCF, ZsRE and Pile-10k. For each, we calculate the perplexity ratio between the modified and original model on text generated by the models up to 50 tokens. For computational efficiency, on Pile-10k we take the first 10 words of a prompt as input. These results show that little perplexity is produced by setting a neuron to zero.\n\nThe new Table 1 shows the FPR remains negligible even for cross-domain prompts. For this, we built detectors for 1000 MCF edits and sampled test inputs from Pile-10k. Consequently, we see that the additional perplexity on Pile-10k is comparable to the values reported on MCF and ZsRE in Section 5.\n\nd) We have provided example edited outputs from each model as Table 4 in the attachment, for inputs sampled from the MCF dataset. We are happy to provide more, but are limited for space in this response.\n\n**Questions**\n\n1. We thank the reviewer for the opportunity to clarify this difference. Stealth edits could be seen as rank 1 updates to weight matrices as the edit only changes a single neuron. Major differences with LoRA include:\n    \na) *Learning a task vs a query*. LoRA was introduced and is primarily used to fine-tune and adapt existing pre-trained models to *new tasks*. These new tasks expect the LLM to process a large number of different queries, producing answers which differ from those of the original model. Our edits aim to alter the response to just one new specific query.  \n\nb) *Information/knowledge and data for fine-tuning*. Unless the new task is known to live in a subspace orthogonal to previously learned tasks, LoRA requires data describing the new task and previously learned tasks to protect the fine-tuned model's performance on other tasks. Our edits do not require this. Instead, we give theoretical guarantees that responses of the edited LLM to other queries are unlikely to change.'\n\nc) *Performance guarantees*. As the LoRA paper states: 'The mechanism behind fine-tuning or LoRA is far from clear. How are features learned during pre-training transformed to do well on downstream tasks? ... We mostly depend on heuristics to select the weight matrices to apply LoRA to. Are there more principled ways to do it?' LoRA is 'learned' through sequential gradient updates of `low-rank weights'. Our method is a rank-1 one-shot learning followed by a light-touch gradient adaption to ensure that the model responds as intended to the target prompt. Importantly, the formulation of our method builds on inferrable geometric and statistical properties of the LLM's feature spaces. This enables theoretical assurances and performance guarantees.\n\n2. For jet-packs (most appropriate for inserting many edits), the only cause of model degeneration is false detector responses to model inputs. Theorem 2 shows this is governed by the intrinsic dimension of the jet-pack's feature space. Empirically, we find that a single jet-pack may accommodate 10k edits with a negligible detector false positive rate (new Table 3), implying the performance ceiling is significantly higher.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TsnlPBCBXn", "reviewer": "Reviewer_wwKs", "summary": "Introduces a new computationally efficient technique to edit facts in a LM. The proposed technique has nice theoretical properties that ensures the selectivity of the edits. The paper also discusses how the ability to edit a specific LM can be measured by its intrinsic dimension, which can be approximated with a data-driven approach.", "questions": "* **Evaluation:** \n    * **Generalizability:** I am concerned that the edits will not generalize well across different paraphrases of the prompt $p$. For example if you edit a fact `The Eiffel Tower is in -> Rome` and then query the edited model with the prompt `A great tourist attraction, the Eiffel Tower, which is located in the city of`, does the LM generate `Rome`? This paper works on the last token of the prompt and *hope* that the LMs semantic similarity of the latents would be high (same as GRACE). But this is not guaranteed and needs to be properly evaluated to claim generalizability.\n    * **Performance comparison across different methods:** The paper does not compare the proposed method with other methods, such as ROME and GRACE. It would be nice to see how the proposed method compares with other methods in terms of the quality of the edits measured in *efficacy*, *generalizability* and also *specificity*. Although I found the discussions in Theorem 2, 3 and the corresponding figures in section 5 are convincing that the method will be highly specific to the edits.\n\n* **Scalability:** What is the maximum number of edits that can be made to a LM using this method? I'd assume, without jet-packs, you theoratically cannot edit more than $d$ facts, $d$ being the actual dimension of the latent space. Figure 1 suggests that the intrinsic dimension of the LM is even less. As a result, the LM might break even before $d$ edits. I think you will face a similar situation with the jet-packs as well, since with increasing number of edits will increase the chances that unintended detectors will get activated. What is your take on this?\n\n* **Detector Neuron:**\n    * **Selecting the detector row in $W_1$:** For a single edit, why did you need to select the row with the smallest norm? Is the assumption that the row with the smallest norm has the least impact on the channel activations anyways and changing it will ensure minimal damage? \n    * I also found it difficult to follow the discussion on Appendix B.1.3 as it introduces too many new variables without proper explanations of what they are. For example $\\zeta$ was introduced in Eqn 12 but it is not clear what it exactly is.\n\n* **Difference with ROME**: The proposed method repurposes a specific column of $W_2$ and corresponding row in $W_1$ to rewrite one single fact. In contrast, my intuition on ROME is that it uses a rank-one update to distribute the load of the edit across *all* the columns in the $W_2$. Is this correct assumption? If so, then I'd be curious to see a comparison of the proposed approach vs ROME in terms of the quality and the scalability of the edits.\n\n**EDIT:** Bumped the rating to 6.", "limitations": "* Was the efficacy measured on the successful prediction of only the first token? If so, it should be mentioned in the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper is nicely presented with crisp mathematical formulations and clear explanations.\n* Novel contribution in the intrinsic dimension based approach to measure the ability to edit a LM.\n* I liked the discussion on the selectivity of the edits and how the development of memory editing techniques leave the LMs vulnerable to *stealth attacks*.", "weaknesses": "The only weakness I see in this paper is the lack of proper evaluations. Please see the detailed comments below.\nI am happy to revise my rating if the concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer for taking the time to review our paper and for providing valuable comments and feedback. We have responded to these in detail below.\n\n**Generalisability**. We refer the reviewer to the discussion on generalising edits in our 'Global Rebuttal'. Here, we focus on targeted edits which uniquely guarantee not to damage other model behaviour. We therefore do not intend to claim that our edits will generalise, a task which remains an ill-defined open question. In the revision we will clarify our definition of hallucinations as 'specific hallucinations', and comment on it in the discussion section.\n\n**Performance comparison across different methods**. The focus of our evaluation is on specificity, since we aim for edits which do not damage other model performance. We therefore do not measure edit generalisation. In the new Table 2 (please see the attached PDF), we measure the perplexity ratio for single edits with either ROME or in-place stealth edit. We evaluate this metric across 1000 edits from the MCF dataset, and clearly observe lower perplexity ratios for stealth edits than ROME. This indicates that our in-place edits have a lower impact on original model behaviour. \n        \nFor multiple edits, we compare the jet-pack and GRACE. Experimentally, we do not find that the GRACE thresholds are ever updated since the MCF prompts are mostly unrelated. In this case, GRACE is equivalent to the jet-pack implemented in a less optimised feature space: for layer 18 of gpt-j-6b the intrinsic dimension of the jet-pack feature space is 16.7, while for GRACE this is 13.9. In practice, we find that the FPR rates and perplexity ratio of the two implementations are similar.\nWhen edited prompts are more related, and GRACE's thresholds are adjusted accordingly, the false positive rate will rise significantly. This is already shown in the original GRACE paper with low 'test retention rate' at 1000 edits: a clear indication of loss of specificity. This is an expected trade-off for edits aiming to generalise.\n\nFor single edits, the efficacy of in-place edits, jet-packs, GRACE and ROME are similar, since all rely on finding the right block output vector to produce the edited output text. For multiple edits, ROME's efficacy may be expected to be somewhat worse because all edits respond linearly to all inputs, causing pollution to the generated output.\n\n**Scalability**. Unlike the jet-pack, the number of in-place edits is limited by the number of existing neurons, since one edit is encoded per neuron. However, the number of neurons/edits can still be much higher than the dimension of the feature space: each edit detector is only active in a small 'cap' of the feature sphere around the edited feature vector, and the number of disjoint spherical caps of fixed size grows exponentially with dimension (see, e.g. Kainen and Kůrková (1993) 'Quasiorthogonal dimension'). Theorem 2 guarantees that the probability of sampling a single input falsely activating the detector decreases exponentially with the intrinsic dimension, implying that the number of supported edits can also scale exponentially. Since this bounds worst-case performance, practical performance can be even better, as demonstrated experimentally in Section 5. Empirically, we find that a single jet-pack may accommodate 10k edits with a negligible detector false positive rate (new Table 3), implying the performance ceiling is somewhat higher.\n\n**Detector Neuron**. Yes, in-place edits (and stealth attacks) use a very simple pruning strategy to select which neuron to replace. Empirically, this does not appear to cause problems and has the advantage of being cheap and needing no additional data. Another pruning strategy could be used without affecting the method. We will clarify Section B.1.3 and similar places in the revised version.\n\n**Difference with ROME**. The row/column mechanism provides a nonlinear switch for the edit, which enables our theoretical guarantees against damage to the model's performance. ROME's rank one update provides no such guarantees, since all edits respond linearly to all inputs, as discussed in Section 2 (please also see the related discussion of LoRA in response to Reviewer 3, question 1). Other work has observed that model performance breaks down as ROME-type edits are inserted, sometimes from a single bad edit - exactly what we aim to avoid. The new Table 2 shows the average impact of a single edit using ROME and our method. A further consideration is that our edits are trivially individually reversible/replaceable, which is not easily possible with rank one updates. Another difference is that we can directly test whether an edit is falsely responding to an input and act accordingly.\n\n**Limitations**. The edit success rate is described in detail in Section C.2 (lines 608--612) and is based on (1) the first token of the generated output matches the first token of the expected output (2) the target output is contained within the generated text. From examples, we can see this is sufficient to represent successful generations of the target outputs. We will clarify this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of the reviewers for their detailed comments on the paper, which we have responded to individually. Two of the reviewers also raised an interesting philosophical discussion, regarding model edits which aim to generalise beyond the original target prompt. We discuss this topic in detail here, and plan to enhance the paper by briefly commenting on this in the discussion section.

As the reviewers identify, here we focus on the case of surgical edits to models to fix given individual prompts with the specific aim of guaranteeing not to damage (or otherwise alter) other functionality in the model. A range of other editing methods presented in the literature (and reviewed in Section 2) aim to edit models in ways that 'generalise' to also affect the model's outputs for other prompts. In various circumstances, however, such behaviour may not be desirable (a stealth attack is an immediate example of this), and we view the extent to which an individual edit should 'generalise' as fundamentally the user's choice. At present, there is a gap in the literature for editing methods which can guarantee to fix identified bugs without side effects, and for understanding the fundamental properties determining their success. 

Model editing is ultimately a blunt tool compared with more expensive approaches such as fine-tuning. It is to be expected that as an edit generalises, it increases the risk of causing unintended model degeneration. This is further complicated by the inherent subtle complexity of natural language: should all similarly-phrased prompts also be considered hallucinations? What is the precise definition of a 'similarly-phrased prompt'? What about similar sentences in very different contexts? To take a standard model-editing example, consider: 'Who is the Prime Minister of the UK?' and 'The year is 2015. Who is the Prime Minister of the UK?' or 'Let's write a story about a world where everyone is called Blarg. Who is the Prime Minister of the UK?', etc. Moreover, it is often observed that (semantically) small changes to prompts can significantly change model outputs (the phenomenon exploited by adversarial attacks, for example). It is not difficult to imagine scenarios where overly-generalising edits introduce unknown new errors of their own. Preventing edited models from catastrophically forgetting previous learning is in itself an important alternate form of generalisation, as employed in the field of life-long learning. This is particularly important in tasks where prompts may be highly structured: the risks of over-generalisation would be borne without any counterbalancing advantage. 

Many empirical studies, such as reference [15] introducing the GRACE method (which may be efficiently implemented using a jet-pack as shown in Section 7), suggest these editing methods may be able to generalise to rephrased prompts. As with all empirical work, however, the quality of the conclusions drawn from these experiments relies on the quality of the benchmark dataset. Results also depend on the choices made when quantifying generalisation. The datasets typically used for such tests present just a few phrasings similar to the original prompt; is this sufficient to certify that an edit will generalise as desired, or simply that some additional model behaviour is affected by the edit? This is a common theme: studies investigating editing mechanisms which generalise to rephrased prompts appear to be universally empirical in nature, and even a compelling rigorous definition of 'generalisation' in the context of natural languages is absent from the literature. Understandably, the notion of generalisation in the context of language modelling is very different from generalisation within the framework of classical statistical learning theory. We see a need, therefore, for an alternative theoretical framework through which we are able to systematically study edit generalisation -- a nontrivial task which we plan to address in future work.

From an operational perspective, the task of building generalising edits is also nontrivial. Collecting high-quality re-phrased prompts and corrected responses through which to construct and verify/test each edit (or accurately identifying subject-object-relationship triples in the case of frameworks such as ROME [21]) requires time-consuming manual intervention. The targeted approaches we consider here could be used to insert edits automatically in response to user feedback. Follow-up work could consider approaches to later 'merge' or combine collections of related edits, perhaps during the next round of fine tuning. We also note that from an attack perspective, specificity is highly desirable as it makes detection more difficult.

We hope that this global response explains the motivation and the intended impact of our work, and clarifies the difference between the approach and settings we proposed in this work and what has been done so far in the literature on editing LLMs.

Meta-review: The paper studies the problem of selectively editing a LM for a very specific fact without retraining.  It presents an efficient method exploiting the model high intrinsic dimensionality. The method is useful to correct hallucinations but it also exposes a new type of modelvulnerability.

The reviewers agreed that the paper is novel and that it is presenting an interesting problem. They had some doubts, related to the motivation of the paper and a more precise problem definition, as well as concerns with the distance between theoretical and empirical results. However, after the discussion, the reviewers had a more positive opinion of the paper, and they agreed that, while its scope might be more limited than what originally implied by the authors, it can be seen as a "foundational" paper defining an original problem.
Decision: Accept (poster)