Stealth edits to large language models

Authors: oliver j sutton
qinghua  zhou
wei  wang
desmond j higham
alexander n gorban
alexander  bastounis
ivan y tyukin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:55:45

Abstract: We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without requiring retraining. Our theoretical insights show that a single metric (a measure of the intrinsic dimension of the model's features) can be used to assess a model's editability and reveals its previously unrecognised susceptibility to malicious *stealth attacks*. This metric is fundamental to predicting the success of a variety of editing approaches, and reveals new bridges between disparate families of editing methods. We collectively refer to these as *stealth editing* methods, because they directly update a model's weights to specify its response to specific known hallucinating prompts without affecting other model behaviour. By carefully applying our theoretical insights, we are able to introduce a new *jet-pack* network block which is optimised for highly selective model editing, uses only standard network operations, and can be inserted into existing networks. We also reveal the vulnerability of language models to stealth attacks: a small change to a model's weights which fixes its response to a single attacker-chosen prompt. Stealth attacks are computationally simple, do not require access to or knowledge of the model's training data, and therefore represent a potent yet previously unrecognised threat to redistributed foundation models. Extensive experimental results illustrate and support our methods and their theoretical underpinnings. Demos and source code are available at https://github.com/qinghua-zhou/stealth-edits.

Main paper: 
 introduction: 
 The latest meteoric rise of artificial intelligence has been driven by the maturing of large language models. These models, predominantly based on the transformer architecture [39] , have demonstrated remarkable abilities in natural language communication and comprehension, which are only beginning to transform the world as we know it today. Scaling has proved to be key in enabling these breakthroughs: the GPT-4 family of models, for instance, has on the order of 10 12 trained parameters, a figure which would have been inconceivable only a few years ago. The raw computational power and vast quantities of quality data required to train models such as these make developing them prohibitively expensive for all but a handful of the world's wealthiest technology companies [31] . This environment has also seen the rise of foundation models: collections of accessible or open source language models which have become an invaluable tool for those without the facilities to train their own from scratch. As language models have developed, 'hallucinations' -non-factual or non-sensical information generated by the model -have become challenging barriers to trustworthy and reliable artificial intelligence. Much work has been invested in trying to understand the origins of hallucinations [7] and develop mechanisms to mitigate them [18] , amplified by regulatory requirements placed on organisations deploying AI by the European Union's recent 'AI Act' [9] or the UN's resolution on 'safe, secure and trustworthy artificial intelligence' [38] . Recent work has, however, shown that hallucinations may in fact be an inevitable artefact of fixed language models [19, 43] . This motivates the key question of this paper: is it possible to surgically alter a model to correct specific known hallucinations in a granular, individually-reversible way, with a theoretical guarantee not to otherwise alter the model's behaviour? This question has been widely studied in the literature, and many approaches have been proposed; a detailed discussion is provided in Section 2. Perhaps the approaches that come closest to answering this question are GRACE [15] and Transformer-Patcher [17] . GRACE selectively applies edits by comparing input features to a set of pre-computed keys. However, adding this to a model requires re-writing the model code, rather than updating existing weight matrices, and the conditional logic required does not naturally suit massively parallel computing architectures such as GPUs. Transformer-Patcher instead encodes edits into the last transformer perceptron block of a network, to detect when incoming features should be edited and provide the corrected output. Experimental studies have shown the potential of these approaches for targetted corrections to hallucinations, yet a clear theoretical understanding of what determines their success has remained elusive until now.Here, we systematically study these and related methods under the collective banner of stealth editing. We develop a novel theoretical approach which reveals that, surprisingly, a single metric provably determines the editability of a given model (Theorem 2). The metric can be estimated directly from data and measures the intrinsic dimensionality of the model's feature vectors and was introduced in the setting of binary classification in [32] . Guided by this understanding, we are able to propose a simplified editing mechanism which optimises the selectivity of each edit. Through this, we are able to build a bridge between our methods, GRACE and Transformer-Patcher, showing that they can be implemented and studied within the same framework. The clear implication of this is that the new theoretical understanding we develop extends directly to these methods. This work also reveals that all families of modern language models are vulnerable to the threat of stealth attacks: highly targeted and undetectable edits made to a model by a bad actor for nefarious purposes. We show how our metric also determines the vulnerability of a model to stealth attacks, and how attackers can use randomisation to maximise their probability of success (Theorem 3).Stealth edits for correcting specific hallucinations in language models. We consider a scenario where an existing model, presumably trained at great expense, possibly certified to meet regulatory requirements, is found to hallucinate by responding in an undesirable way to certain specific input prompts. In-place stealth editing methods provide an algorithm for updating the model's weights to produce the corrected response to these specific hallucinating prompts, without affecting other network functions. Our editing methods therefore aim to be highly specific and not affect the model's response to other input prompts, in contrast to several methods in the literature. This is discussed further in Section 6. Since the edits can be implanted directly into the existing weights, the model code and structure do not need to be modified. Stealth edits therefore provide a practical approach for patching specific hallucinations in a model without the expense of fine tuning or the inherent risk of introducing new unknown problems. The details of the algorithms for this are given in Section 3. Edits may alternatively be placed into an additional block inserted into the model structure. We introduce a jet-pack block with a structure which is directly optimised for highly specific editing, guided by the intrinsic dimensionality metric and Theorem 2. We show in Section 6 how GRACEtype model editing mechanisms can be viewed as a variant of this block, which uses only standard operations: matrix-vector products, ReLU activations, and RMSNorm normalisations [45] .Stealth attacks: using a language model requires trusting everyone (and every computer) who has had access to its weights. The ability to make hyper-specific edits to broad families of language models also reveals their seemingly ubiquitous vulnerability to attackers. Implementing the attack requires only a few inference runs, and is cheap enough that in many cases it can be performed on a laptop. No backpropagation or fine-tuning is required, and the attacker does not need any access to or knowledge of the model's training data. Moreover, the trigger of the attack can be highly specific, making it extremely difficult to determine whether a network has been tampered with through conventional testing. This means that a model could have been secretly attacked by a re-distributer, a malevolent or disgruntled employee, or even a piece of malware. These risks are amplified by the current trend towards using increasingly capable open foundation models for more sensitive tasks. A stealth attack implanted in a model which runs code [29] or accesses a database [35] could be used to install viruses or delete data with catastrophic consequences. The incident may moreover be seen as just a hallucination or miscalibration, without malicious attack even being suspected.Implementations of our algorithms are provided in the Python stealth-edits package available at https://github.com/qinghua-zhou/stealth-edits. An interactive demonstration is available at https://huggingface.co/spaces/qinghua-zhou/stealth-edits.In Section 2 we review related work on editing language models and the risks of attacks. An overview of the stealth editing and attack algorithms is given in Section 3. Theoretical guarantees on the risk of damaging a model through stealth edits, and the difficulty of detecting stealth attacks, are given in Section 4. Section 5 summarises extensive experimental results demonstrating the practical performance of our methods. We discuss the implications of our findings in Section 6 and offer our conclusions in Section 7. Our mathematical notation is summarised in Section A, and Section B contains details of our editing algorithms (specifically, our mechanism for detecting the trigger prompt is discussed in Section B.1.3, and Section B.1.4 presents out method of producing the corrected output). Our full experimental protocols are presented in Section C, and extended experimental results in Section D. Proofs of our theoretical results are given in Section E.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The theoretical insights we gain from our main theorems provide us with new approaches to optimise model-editing approaches to maximise their efficacy for selectively correcting hallucinating prompts.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Correcting hallucinations in language models. The standard approach is to retrain the model using the corrected prompts, possibly detected through user feedback [30] . Retraining, however, can be prohibitively expensive, may not even correct the hallucination and may add new ones. Retrieval Augmented Generation (RAG) [21] helps overcome out-dated training data, but requires an external information source and does not correct specific hallucinations. Stealth edits, on the other hand, aim both to directly correct the hallucinating prompt and not alter the model behaviour otherwise.Memory editing. Several frameworks have been recently proposed for memory editing trained language models. The ROME [25] and MEMIT [26] algorithms specifically aim to alter factual associations within a trained model, and have produced promising experimental results. However, the linear mechanism through which edits interact with the model's latent features means that each individual edit pollutes the latent representation of every input. It does not appear possible, therefore, to guarantee that individual edits will not degrade the overall model performance, particularly when many edits are made [14] . On the other hand, methods like Knowledge Neurons [6] and Transformer-Patcher [17] treat transformer perceptron blocks as 'key-value pairs' [11] to store and recall edits, which are trained by gradient descent. External components like GRACE [15] similarly encode edits in a purpose-built key-value dictionary. Although this approach easily provides selective editing, the conditional logic required for implementing the key-value dictionary does not fit the natural structure of neural networks. Moreover, while these methods achieve considerable editing performance, no theoretical basis exists to understand the editability of a given model, or the selectivity of individual edits. Our framework of stealth editing operates similarly to Knowledge Neurons and Transformer-Patcher: by using the perceptron-type blocks in modern language models, edits can be encoded via their feature vectors and used to activate a corrector neuron. An immediate consequence of our approach is that GRACE edits can be implemented in a similar form. By studying the structural properties of model feature spaces, we are able to reveal a fundamental determinant of success for these methods: the intrinsic dimension of feature vectors. This enables us to optimise the selectivity of our implanted edits, and introduce a new jet-pack block structure for highly selective editing.Backdoor attacks. Backdoor attacks [4] have been widely studied in natural language models via data poisoning [5] , tokeniser attacks [16] , and embedding attacks [44] . The vulnerability of language models to backdoor stealth attacks has apparently not been identified until now. Stealth attacks do not require modifying or even knowing anything about the model's training data, and do not modify user-facing components. Memory editing-based attacks [22] do not target the response to a single chosen prompt, and until now there has been no metric to determine a model's susceptibility.Stealth attacks in computer vision. Stealth attacks have been previously discussed in computer vision [36, 37] . Although the aim of the attack is similar, the mechanism through which the trigger is constructed, the method of implanting it into a model, and the machinery behind the argument are all fundamentally different. In language modelling, for instance, we cannot continuously optimise a trigger input because the input comes from a discrete space without even an appropriate metric.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The experimental study is only performed on certain datasets, although the theoretical results are general. It remains a question for future work to find if there are scenarios in which results may differ.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
stealth editing algorithm overview: 
 Suppose we have a pre-trained language model N which takes an input prompt and produces a predicted token as output. If the model is producing an undesired sequence of output tokens for a prompt p, a stealth edit aims to cheaply update the weights in a single model block to produce the desired output to prompt p without changing the response to other prompts. If such an edit is secretly made by an adversary, we refer to this as a stealth attack. For concreteness, we consider models with the autoregressive transformer decoder [39] or selective state space [13] structure (although it may be possible to treat other models analogously). These are typically formed of a sequence of blocks/modules with a repeating structure (transformer blocks, or Mamba blocks). We insert our edits by either directly modifying existing weights in a chosen network block, or by inserting an additional jet-pack block with an optimised structure.In-place stealth editing. An edit can be made by modifying the weights of a block with the structure †EQUATIONwhere B : R d → R d for a model with latent feature space dimension d, and• x is a latent feature vector in the model with dimension d,• η is a normalisation function projecting data to the surface of (the affine image of) a sphere, such as RMSNorm [45] in Llama and Mamba models, or LayerNorm [2, 3] in GPT models, • W 1 and W 2 are linear projection matrices with shapes n × d and d × n respectively, for some hidden dimension size n, • σ is an activation function, • F : R d → R n represents an additional model-dependent (non)linear gating term.In transformer models, (1) represents a multi-layer perceptron block (typically with F affine), while the whole Mamba block in selective state space models takes the form of (1) (with F representing the state space component). The activation function σ varies between architectures, but typically satisfiesEQUATIONas, for example, with ReLU, SILU, GELU, etc. Some architectures (such as GPT-family models) also provide bias terms alongside the weight matrices W 1 and W 2 .Edits are encoded into the W 1 and W 2 matrices using Algorithm 1 (implemented by the function apply edit(...) in editors.py in the Python package), described in detail in Section B. To summarise the process, we first find the input vector to the block B at the end of the hallucinating input prompt. This is used to encode a linear separator into a single neuron (row) of W 1 with some chosen threshold θ ∈ [0, 1]. Since the normalisation η maps feature vectors to the surface of (an affine image of) the unit sphere, this linear separator is highly effective at isolating just a small region around the target feature vector. The activation function σ provides near-zero response when the output from the linear separator is sufficiently negative due to (2) . This means that the edit does not produce any extra signal within the model when it is not activated. Sufficiently strong responses from the detector neuron, however, are propagated by the activation function σ to W 2 . Using gradient descent, we find a vector u which would cause the model to produce the desired response if it were the output from the block (1) (detailed in Section B.1.4). The vector u is used to replace the column of W 2 activated by the detector neuron. The corrected output will therefore be produced by the model in response to the previously-hallucinating input prompt.Some models, like Llama, have no bias terms to use as the linear separator threshold with W 1 . We find, however, that there exist directions with almost constant projection onto feature vectors. Constructing such a direction (see Section B.1.5) enables us to implant the detector threshold.Algorithm 1: An in-place edit to correct a hallucination in a language model Input : Language model N and the index j of the block to edit Hallucinating input prompt p and corrected output rtarget Detector threshold θ ∈ [0, 1] and gain α > 0Compute the feature vector ϕ which is the input to block j at the last token of the input prompt p Find the index k of the row of W1 with least ℓ 1 norm Construct a detector neuron weight vector w sensitive to ϕ (and bias depending on architecture) as in Sec. B.1.3 with threshold θ and gain α Use gradient descent to find a replacement output vector u from block j which produces the corrected output rtarget, as in Sec. B.1.4 Build the edited weight matrix Ŵ1 by replacing row k of W1 with the detector vector w Build the edited response matrix Ŵ2 by replacing column k of W2 with the output generating vector u Produce the edited model N by replacing W1 with Ŵ1 and W2 with Ŵ2Output : Edited language model N Editing with jet-pack blocks. Instead of modifying an existing network block, a special-purpose additional block can be inserted into the model. An effective architecture for this additional block, which we refer to as a jet-pack block is of the formEQUATIONwhere x is a latent feature vector of dimension d, W 1 and W 2 are weight matrices, b is a bias vector, σ denotes the ReLU activation function. When inserting a total of e edits into a model with latent space dimension d, the matrix W 1 has shape e × d, b has e components, and W 2 has shape d × e. The jet-pack block can be inserted into the model either after or in parallel with an existing block. Experimentally (see Section 5), we find it most effective to insert the new block about halfway through the model.The normalisation function ρ : R d → S d-1 in (3) is optimised to produce highly selective edits. We use a version of the RMSNorm normalisation layer [45] , given byEQUATIONwith a fixed centroid µ ∈ R d . The centroid µ re-centres input vectors to maximise their intrinsic dimensionality (Definition 1) and therefore maximise the edit selectivity due to Theorem 2. In practice, we compute µ as the mean of feature vectors of a set of general text prompts, to provide a representative sample of feature vectors. Experimentally, we find that sentences sampled from Wikipedia [42] , are suitable for this as they provides an easily-accessible source of varied prompts.A new jet-pack block to correct a given set of hallucinations is added to a model by constructing W 1 , W 2 , b and µ as described in Algorithm 2. Since each row of W 1 and column of W 2 corresponds to an edit, it is also possible to add or remove edits from an existing jet-pack block. Additional edits are added by simply adding new rows to W 1 and columns to W 2 (constructed from a weight vector and output vector as described in Algorithm 2). An edit may be removed by deleting the corresponding row and column. Testing for edits which interfere with each other by activating the each other's detectors is also simply achieved by evaluating W 1 W T 1 and searching for off-diagonal entries greater than the detector threshold θ. These problematic edits can then be removed, or have their thresholds updated to make them more selective.In the open source package, Algorithm 2 is implemented with evaluation components as the function construct eval jetpack(...) in the file evaluation/jetpack/construct.py.Stealth attacks. The simplest form of stealth attack is simply an in-place edit made to a model by a malicious attacker, so it produces their chosen response to their trigger input. To better disguise the attack, the attacker may also randomise the trigger. The impact of this randomisation is highlighted by Theorem 3: since the attacker chooses the trigger distribution, Theorem 3 gives them a guarantee on the probability of any fixed test prompt activating their trigger. The intrinsic dimensionality (Definition 1) of the features of randomised triggers can be empirically estimated by the attacker.We consider two specific forms of randomisation here. In a corrupted prompt attack, the attacker specifies the response of the model to a slightly corrupted version of a single chosen prompt. For Set of general text prompts {p g i } N i=1 for feature centring (e.g. sampled from Wikipedia) Detector threshold θ For each general text prompt p g i , compute the output vector ϕ g i from block j at the last token in the prompt Calculate the feature centroid µ = 1 N N i=1 ϕ g i For each hallucinating prompt pi, compute the output vector ϕi from block j at the last token of the prompt Construct the normalised feature vectors {ψi = ϕ i -µ ∥ϕ i -µ∥ } n i=1 for the hallucinating prompts Build the detector neuron weight vector wi and bias βi for each feature vector ψi as in Section B.1.3, with threshold θ and gain α For each corrected model output ri, use gradient descent to find a replacement output vector ui from block j which produces ri, as in Sec. B.1.4 Build the detector matrix W1 with row i given by wi, and bias vector b1 with entry i given by βi Build the response matrix W2 with column i given by ui Generate the edited model N from N by inserting the jet-pack blockJ(x) = x + W2σ W1 x -µ ∥x -µ∥ + b1 ,(with ReLU activation σ) after network block j Output : Edited model N which produces the correct responses {ri} n i=1 to the prompts {pi} n i=1 example, this could be by randomly sampling typos to insert into the trigger prompt. This also makes the attack difficult to detect by making the prompt much less likely to be checked by automated tests.In an unexpected context attack, the attacker could specify the response of the model to a chosen prompt when it follows a 'context' sentence, randomly sampled from Wikipedia for instance. Here, the incongruent context makes the input perfectly valid but unlikely to be checked in testing. Possible examples of such attacks are illustrated in Section D.2. 
theoretical foundations: 
 By construction, stealth edited models will always produce the corrected response if the editing algorithm succeeded in finding a replacement block-output vector which produces the desired model response. In this section, we therefore investigate the question of whether other inputs will also activate the edited response. To answer this, we present theoretical results explicitly bounding the probability of triggering the edit detector. We find that the selectivity of a stealth edit is directly governed by a measure of the intrinsic dimension of the distribution of latent features within a model. The concept of intrinsic dimension we use was introduced in [32] , and is based on the pairwise separability properties of data samples. Definition 1 (Intrinsic dimension [32] , cf. [12] ). For a distribution D defined on a Hilbert space with inner product ⟨•, •⟩, the separability-based intrinsic dimensionality of D at threshold δ ∈ R is defined asn(D, δ) = -1 -log 2 (P (x, y ∼ D : ⟨x -y, y⟩ ≥ δ)).This characterises the dimensionality of a data distribution through the pairwise separability properties of sampled data. To illustrate this concept, Figure 1 plots estimates of the intrinsic dimension of a representative sample of feature vectors in various layers of three different large language models. The definition is calibrated such that if D is a uniform distribution in a d-dimensional unit ball then n(D, 0) = d. The function n(D, δ) is increasing in δ, with a minimum value of -1 for data which are inseparable with probability 1 at threshold δ. This is attained by a data distribution concentrated at a single point for any δ < 0. Infinite values of n(D, δ) indicate that sampled pairs of data points are separable with probability 1 at threshold δ. This is the case for data uniformly distributed on the surface of a sphere when δ = 0, for example.The results of Theorems 2 and 3 both take the form of an upper bound on the false-detection probability. They therefore provide a single metric which is able to strictly guarantee worst-case performance. Practical systems will likely perform significantly better than this worst-case performance, and we investigate this experimentally in Section 5.To state the results concisely, we use the feature map φ defined in Section B.1.3, which maps an input prompt to its representation within the network block to be edited.Other prompts are unlikely to activate stealth edits. Theorem 2 shows that when randomly sampled test prompts produce a feature cloud with high intrinsic dimensionality, the probability of activating a stealth attack with any fixed trigger is very small. Theorem 2 (Selectivity of stealth edits). Suppose that a stealth edit is implanted using the linear detector f defined in Section B.1.3, for a fixed trigger prompt p trig and threshold θ ≥ 0. Suppose test prompts are sampled from a probability distribution D on prompts, and let D φ denote the distribution induced on R d by the feature map φ defined in (11) . Then, the probability that the edit is activated by a prompt p sampled from D decreases exponentially with the intrinsic dimensionality of D φ . Specifically,EQUATIONStealth attacks with randomised triggers are unlikely to be detected. Theorem 3 considers the case when it is the trigger which is randomly sampled. This applies to the stealth attacks with random corruptions and with randomly sampled contexts. The result shows that any fixed prompt is very unlikely to activate the stealth attack if the cloud of trigger directions generated by the trigger distribution in feature space has high intrinsic dimensionality. Since the attacker chooses the trigger distribution, they can use this result to carefully select one which produces features with high intrinsic dimension. Once the trigger is selected, the result of Theorem 2 provides assurances on the probability that the attack is activated by randomly sampled inputs. Theorem 3 (Stealth edits with randomised triggers). Let T denote a probability distribution for sampling a trigger prompt, and let T φ denote the distribution induced by the feature map φ. Suppose that a stealth edit is implanted using the linear detector f defined in Section B.1.3 with threshold θ ≥ 0 for a trigger prompt p trig sampled from T . Then, for any fixed test prompt p, the probability that the stealth attack is activated by p decreases exponentially with the intrinsic dimensionality of T φ . Specifically,P (p trig ∼ T : the detector f for trigger prompt p trig is activated by p) ≤ 2 -1 2 (1+n(Tφ,2θ(θ-2))) . 
experimental results: 
 This section summarises the results of a variety of experiments testing the efficacy of the algorithms proposed above, and their links with the theoretical insights of Theorems 2 and 3. Results from further experiments are presented in Section D.Models. We investigated three state-of-the-art pre-trained language models: the transformers Llama 3 8b [1] and GPT-J [40] , and the selective state space model Mamba 1.4b [13] . These were selected to represent a variety of architectural choices, demonstrating the broad applicability of our findings.Our experiments require a source of hallucinations to edit, which we draw from the Multi-CounterFact (MCF) [26] and ZsRE [27] datasets. Both provide factual prompts and expected responses. Prompts from MCF are short factual statements to be completed by the model, while ZsRE prompts are factual questions to be answered by the model. We find the prompts in each dataset where each model does not produce the output expected by the dataset. We view these as the set Metrics. We report the performance of our algorithms using the following metrics:• Edit/attack success rate (higher is better): the proportion of attempted edits/attacks which produce the corrected output, computed as described in Section C.1. • Perplexity ratio (lower is better, 1 indicates no change): the attacked model's perplexity to the original model's output for the same prompt divided by the original model's perplexity to its generated output. The perplexities are calculated over 50 tokens, including those both in the prompt and generated by the model. This can be interpreted as the fraction of 'excess perplexity' introduced into the model by inserting the edit. • Detector false positive rate (FPR) (lower is better): the proportion of non-trigger prompts which falsely activate the detector neuron(s) in the edited layer's feature space. • Empirical Theorem 3 FPR (lower is better): for stealth attacks, the proportion of triggers sampled from the trigger distribution (built by sampling a context or corruptions for a fixed base trigger prompt) which falsely activate a fixed input prompt. • Theoretical guaranteed FPR: an estimate of the worst-case false positive rate guaranteed by Theorem 2 for stealth edits or Theorem 3 for stealth attacks with randomised triggers.Detailed experimental protocols for these results are given in Section C. All experiments used θ = 0.005, α = θ -1 ∆ and ∆ = 50. The impact of different values of θ is investigated in Section C.5.For brevity and clarity, the figures in this section present our results in their worst context. Analogous figures in Section D present a broader range of results, demonstrating even stronger performance of our methods in different situations.In each figure, lines show the mean value of a quantity. Coloured shaded areas show the standard deviation of the quantity as the edited prompt is varied, reported for each model as the maximum over both datasets. Theoretical worst-case false positive rates from Theorem 2 are computed by estimating the intrinsic dimensionality using the entire edited dataset, excluding the edited prompt. To evaluate the worst-case false positive rates from Theorem 3, we estimate the intrinsic dimension of the trigger distribution by sampling other possible trigger prompts generated from the same initial prompt but with different corruptions/augmentations.In-place edits for correcting specific hallucinations. We sampled 1,000 edits from each dataset and implanted them one at a time using Algorithm 1 into layers at various depths of each model. The experimental protocol is given in Section C.1. Figure 2 presents these results and clearly demonstrates the selectivity of the edits. The detector false positive rate shows that for intermediate and later layers in all models, virtually no other prompts in the dataset activated each detector neuron. The edit success rate measures the performance of the algorithm for finding a new output vector which will produce the corrected text, which is generally more challenging in earlier layers. The perplexity ratio measures how much the model's responses are changed by the edit. Section D.4.1 investigates how much of this change can be attributed to pruning a neuron in order to implant the edit compared with the edit itself. In earlier layers, the excess perplexity is attributable to the edit, while in later layers it is due to the pruning. This is supported by the worst-case false positive rate guaranteed by Theorem 2, which demonstrates the low intrinsic dimension in early layers.Jet-pack edits for correcting specific hallucinations. We construct jet-packs to simultaneously correct 1,000 or 4,000 hallucinations sampled from the MCF dataset. The experimental protocol is given in Section C.2, and the results are shown in Figure 3 . We only insert edits into the jet-pack if the algorithm is able to construct an output vector which, in isolation, will produce the corrected model output, since the feasibility of generating such an output vector is the same as in the previous experiments. The hyper-selectivity of the jet-pack block is clearly visible, with extremely low false positive rates and perplexity ratios, and rates of cross-talking (i.e. detectors which respond to the trigger prompt of a different edit). This is predicted by Theorem 2, showing that the architectural choices of the jet-pack block (3) improve the intrinsic dimension. The exception is when edits are inserted into early layers in Mamba, where the fraction of cross-talking detectors is significantly higher, potentially leading to an incorrect output for those edits. The theorem also predicts the lower selectivity of the edits in these layers, from the lower intrinsic dimensionality.Stealth attacks with corrupted prompts. We sampled 500 prompts from each dataset to insert as stealth attacks. Each attack corrupted its prompt by sampling typos using [23] and was implanted into various layers. The experimental protocol is given in Section C.3, and the results are presented in Figure 4 . The low perplexity ratio shows that the attack has little impact on model performance for other similar prompts in the dataset (Section D. 4 .1 shows this extra perplexity is mainly due to pruning a neuron for the attack). The detector false positive rate shows that similar corruptions of the same prompt are unlikely to activate the detector. The 'Empirical Theorem 3 FPR' is dual to this: the probability of sampling other corruptions of the trigger prompt which produce detectors activated by a specific corruption of the same prompt. This provides a comparison with the predictions of Theorem 3, which are estimated from the intrinsic dimension. Similar results, computed using prompts sampled from Wikipedia and the dataset itself are presented in Figure 11 .Stealth attacks with unexpected contexts. We consider two kinds of unexpected contexts: sampling a sentence from Wikipedia (Figure 5 ), or randomly inserting typos into the 'clean context' sentence 'The following is a stealth attack: ' (Figure 6 ). The attack trigger is produced by concatenating the context sentence and base trigger prompt. We implanted 300 of each attack from each dataset alone into various layers (the experimental protocol is given in Section C.4). The metrics reported are analogous to the previous section. Further results are shown in Figures 12 and 13 . 
discussion: 
 Intrinsic dimension of data is crucial. The key conclusion from Theorem 2 and 3, and our experimental study, is that the intrinsic dimension of a model's feature vectors is a crucial determinant of success in model editing. Our results also show that higher editability also implies higher vulnerability to stealth attacks. Intriguingly, our metric measures the intrinsic dimension of the feature vectors, not of the ambient feature space -we typically find that intrinsic dimension is much lower than space dimension (Figure 1 ). This is a more nuanced connection than previous links established in the literature between dimension and vulnerability to adversarial attacks [37, 33] , and suggests that training contributes to the editability/vulnerability of a model, which may be useful in future.The risks of stealth attacks. Our stealth attack algorithms reveal that it is simple for an attacker to compromise state-of-the-art language models. Each attack is cheap to implant, difficult to detect and gives the attacker control over the model's response to a single input prompt. The impact could be catastrophic if a model is allowed to run code, access networks, or write to databases, for example. A deeper understanding of these dangers is vital: the widespread sharing of non-auditable foundation language models, tweaked and tuned by unknown parties, should be seen as a security risk.Jet-pack blocks gracefully implement GRACE. The normalisation functions routinely used within language models (such as RMSNorm [45] and LayerNorm [2] ) project data to the surface of (an affine image of) a unit sphere. Detecting trigger features with a linear separator is therefore equivalent to testing whether input features are close in Euclidean norm to a stored trigger, as used in GRACE [15] . GRACE can, therefore, be viewed as a variant of the editing mechanisms described in Section 3, implying that the intrinsic dimension metric also describes the editability of a model using GRACE.Edit insertion location. Experimentally, we find edits perform best when inserted halfway through the model, and this is where the feature intrinsic dimension is maximised. This is in contrast to Transformer-Patcher [17] , in which edits are typically placed in the last layer of the network.Editing monosemantic features. Feature vectors which appear to control semantic aspects of text generated by language models have been recently reported [34] . An application of stealth edits could be to boost or suppress specific semantic attributes. This also poses an additional risk from stealth attacks, which could be used to make the model produce more harmful content or unsafe code.Specific vs generalising edits. Our aim is to make highly controlled, specific edits to a language model. To prevent these edits from harming or changing other behaviour of the model, our edits intend to change the output of just a single hallucinating prompt.Limitations. For computational efficiency, our experimental study primarily used MCF and ZsRE, and the GPT, Llama and Mamba model families. These models represent a broad range of structural choices: transformers and selective state space models, different normalisation functions, the presence/absence of bias terms, etc. Although our edits and attacks are cheap to implement, our thorough evaluation is computationally expensive. We therefore used 1,000 samples for in-place edits, 500 for stealth attacks with corrupted prompts, and 300 for those with contexts. 
conclusion: 
 In this work, we have revealed the theoretical foundations of model editing, used these to assess the editability of models, expose language models' susceptibility to malicious edits, and propose highly effective novel editing methods. Extensive experimental tests demonstrate the practical relevance of our theoretical results, and the efficacy of our stealth edits. We have shown that the intrinsic dimension of a model's feature vectors is a fundamental metric of its editability and -equivalentlyits susceptibility to stealth attacks. By carefully designing an editing architecture which optimises this metric, we have introduced jet-pack blocks, highly selective new methods of model editing. Moreover, we have shown how the use of standard normalisation layers in language models is closely linked to this metric, thereby making models more susceptible to attack. Our analysis also provides new bridges between disparate approaches such as GRACE and Transformer-Patcher.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Errors are reported for results via standard deviations. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This is provided in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research conforms to the NeurIPS code of ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Stealth attacks could have negative social impacts. We discuss these impacts at length in the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: We do not believe our pilot study presents direct safeguarding issues Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets are cited to their respective owners Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this article, we use the following mathematical notation:• R denotes the set of real numbers, and for a positive integer d we use R d to denote the linear space of vectors with d real-valued components, • for vectors v, w ∈ R d , we use ⟨v, w⟩ to denote the Euclidean inner (dot) product of v and w, and ∥v∥ = (⟨v, w, ⟩) 1/2 denotes the Euclidean (ℓ 2 ) norm,• the unit sphere in R d is denoted byS d-1 = {x ∈ R d : ∥x∥ = 1},• the elementwise (Hadamard) product of two matrices V and W with the same shape is denoted by V ⊙ W , and elementwise division is denoted by V ⊘ W , • the finite set of tokens from which prompts are formed is denoted T , and P denotes the set of prompts, which are defined to be sequences of tokens in T .The algorithms we present for stealth attacks apply to both transformer models and state space models. We present both architectures here using a unified notation to enable a uniform algorithmic exposition.In both cases, the edit only involves modifying one row of the matrix W 1 to detect the trigger and one column of W 2 to produce the desired response. Let p ∈ P denote an input prompt from the set P of sequences of tokens from a finite token set T .A transformer language model (with latent space dimension d) is a map N : P → T formed of a sequence of transformer blocks and blocks for input and output. For an index j, let N j : P → R d represent the action of the model up to the input to transformer block j, and let M j : R d → T be the map from the output of this block to the final logit confidence scores produced by the model. The next token is generated from these logits by a sample function, for example, using a temperature-based strategy. The model N can be expressed asEQUATIONHere, a : R d × P → R d denotes the self-attention component, and the perceptron component m : R d → R d may be expressed as:• Llama-family models. The block takes the formm(x) = W 2 [(W 3 ρ(x)) ⊙ σ(W 1 ρ(x))],where W 1 and W 3 are matrices with size n × d (for some hidden dimension size n, typically n > d), and the matrix W 2 has size d × n. The activation function σ is SiLU [8] . The RMSNorm normalisation [45] is used for ρ : R d → R d , with learned weights W ρ ∈ R d andEQUATION• GPT-family models. The block takes the form m(x) = W 2 σ(W 1 λ(x) + b 1 ) + b 2 , where the matrix W 1 has size n × d and W 2 has size d × n. The activation function σ is GELU. Here, λ : R d → R d represents the LayerNorm normalisation [2] , computed asλ(x) = W λ ⊙ x -µ v 1/2 + b λ , with µ = 1 d d i=1x i , and v = 1EQUATION)where W λ ∈ R d and b λ ∈ R are the learned weights and bias respectively.We focus on the Mamba family of selective state space language models [13] . This presentation elides most of the details of how this family of models is structured, but exposes just the components necessary for our exposition. Such a model (with latent space dimension d) is a map N : P → T formed of a sequence of state space blocks and blocks for input and output. For an index j, let N j : P → R d represent the action of the model up to the input to Mamba block j, and let M j : R d → T be the map from the output of this block to the logit confidence scores produced by the model. The next token is generated from these logits by a sample function, for example, using a temperature-based strategy. For a prompt p ∈ P, the model N can be expressed asEQUATIONwhere s : R d × P → R d denotes the state space component, W 1 is a matrix of size m × d, W 2 is a matrix of size d × m, and ρ : R d → R d is as in (7) .Let η : R d → R d denote the normalisation map η(x) = ρ(x) for Llama-family and Mamba-family models (RMSNorm, defined in ( 7)), λ(x) for GPT-family models (LayerNorm, defined in ( 8)),and define ψ : P → R d to be the function mapping a prompt to the input of the weight matrix W 1 in the j th network block. Specifically, for any p ∈ P we haveψ(p) = η(N j (p) + a(N j (p); p)) for the transformer models in Section B.1.1, η(N j (p))for the state space models in Section B.1.2.The input map ψ is therefore such that ψ(p) = η(x) for the vector x defined in either of the systems (6) or (9) . The normalisation map η projects the output of ψ to the surface of a sphere ‡ , and then scales and shifts the output. For convenience, we can return to the sphere with the affine map ν :R d → R d defined as § ν(x) = 1 √ d x ⊘ W ρfor Llama-family and state space models,1 √ d (x -b λ ) ⊘ W λfor GPT-family models. For brevity, we introduce the feature mapEQUATIONmapping prompts to features on the unit sphere. Let c ∈ R d denote a user-chosen centre point with ∥c∥ ≤ 1. The attack trigger is detected by encoding a linear separator f : R d → R (acting on outputs from the map ψ) into one row of the weight matrix W 1 and bias vector b 1 . For a threshold θ > 0, scaling α > 0 and trigger direction τ = φ(p trig ), f is given byEQUATIONWe omit the parameters τ, θ, α when they are contextually clear. For small θ > 0, the function f • ψ responds positively to p trig and negatively otherwise. The activation functions σ used in the models then filter large negative responses from f since σ(t) ≈ 0 for t ≪ 0, σ(0) = 0, σ(t) ≈ t for t ≫ 0.In practice we therefore use 0 < θ ≪ 1 and α = ∆θ -1 , with ∆ ≫ 0 chosen to amplify the response to p trig and saturate the zero from σ for other inputs. Selecting c ̸ = 0 'tilts' the linear separator encoded by f to more easily distinguish the clean trigger prompt (and context) from the chosen trigger prompt. The choice of β is left to the user, guided by Theorem 3.To build f into the weights, we find the index k of the row of W 1 with the smallest ℓ 1 norm ¶ . We construct an attacked weight matrix Ŵ1 by replacing the k th row of W 1 with a vector w ∈ R d , and (for GPT-family models) an attacked bias vector b1 by replacing the k th entry of b 1 with a value b.For the GPT family of models, w and b may be simply taken asEQUATIONThe k th component of Ŵ1 ζ + b1 therefore evaluates f (ζ), and the other components are unchanged.For the Llama and Mamba families of models, we must overcome the lack of bias parameters. We find empirically that there exist directions v ∈ S d-1 with ⟨φ(p), v⟩ almost constant as the prompt p ∈ P is varied, and ⟨φ(p trig ), v⟩ > 0 (see Section B.1.5). Projecting the input onto v therefore acts analogously to a bias term. We therefore insert the weight vectorEQUATIONwhich is such that for any p ∈ P(w, ψ(p)) = f (ψ(p)) - α √ d 1 - ⟨φ(p), v⟩ ⟨φ(p trig ), v⟩ (⟨c -τ, τ ⟩ + θ).Experimental results presented in Section B.1.5 show that 1 -⟨φ(p),v⟩ ⟨φ(ptrig),v⟩ is close to zero in practice, and for now we assume that the difference is negligible.The second part of the attack is to ensure the model produces the attacker's chosen output. Key to doing this is the observation that the construction of Ŵ1 ensures that the k th column of W 2 is only multiplied by a non-zero input when the attacker's trigger is present. Therefore, the output which is produced by the model is controlled by what the attacker places into column k of W 2 . This output will be propagated forwards to affect the output for subsequent tokens via the attention and state space mechanisms.Suppose that the target output r target ∈ P contains T > 0 tokens, let r n target ∈ T denote token n of r target , and let r →n target ∈ P denote the sequence of the first n tokens of r target . Let u 0 ∈ R d denote column k of W 2 , and for any u ∈ R d , and let W 2 (u) denote W 2 with column k replaced by u. Define L( • ; Ŵ1 , W 2 (u)) : P → R |T | to be the modified logit map obtained from the model ( 6) or ( 9) by replacing W 1 with Ŵ1 , and W 2 with W 2 (u).We use gradient descent to find a vector u minimising the objective function Λ :R d → R with γ > 0 Λ(u) = - T i=1 log[L(p trig + r →i-1 target ; Ŵ1 , W 2 (u))] ind(i) + γ ∥u∥ 2 ∥u 0 ∥ 2 ,where ind(i) denotes the logit index of the token r i target and [ • ] j denotes the component j of a vector. The first term of this maximises the logit confidence score of the target output tokens, while the second term serves as a penalty to prevent u from growing too large. Experimentally, we find the convergence can sometimes by improved by additionally limiting ∥u∥ by a constant value.Since the Llama and Mamba families of models do not use bias terms, we cannot directly implant a threshold for the trigger detector f as in (13) . Instead, we construct a bias direction vector v ∈ R d such that ⟨φ(p), v⟩ is a positive constant as the input prompt p ∈ P is varied. Such a vector v can be found directly from a set of feature vectors extracted from input prompts as the solution of a constrained quadratic minimisation problem.Suppose that we have a set of prompts Π = {p i } N i=1 ⊂ P, and their associated feature vectorsΦ = {ϕ i = φ(p i )} N i=1 ⊂ S d-1 . Define µ = 1 N Ni=1 ϕ i as the mean of the feature cloud, and let η i = ϕ i -µ denote the fluctuation of each feature vector around µ. Then, for any vector u ∈ R d we have⟨ϕ i , u⟩ = ⟨µ, u⟩ + ⟨η i , u⟩.Since µ is independent of i, we can minimise the variance of ⟨ϕ i , v⟩ by finding v which solvesv = argmin u∈R d 1 N N i=1 (⟨η i , u⟩) 2 subject to ⟨µ, u⟩ = 1.A standard argument (using Lagrange multipliers, for example) shows that whenspan{η i } N i=1 = R d this problem is solved by v = C -1 µ ⟨C -1 µ, µ⟩ , where C = 1 N N i=1 η i η T i ∈ R d×d .If span{η i } N i=1 ⊊ R d , the matrix C is rank-deficient. In this case, a solution may be found by projecting the data into the subspace span{η i } N i=1 , finding the minimiser, and imbedding back to R d . The bias direction in this case is given byEQUATIONwhere C † denotes the Moore-Penrose pseudo-inverse of C.In practice, we compute a bias direction v using the feature vectors for a set of prompts extracted from Wikipedia. Since feature vectors live on the surface of a sphere, it is clear that the product ⟨φ(p), v⟩ will not be exactly constant as the prompt p is varied.Figure 7 shows the effectiveness of constructing a bias direction using this algorithm, as the size of the set of training features is varied. To construct this bias direction, we used a set of feature vectors extracted from layer 17 of Llama for text sampled from the Wikipedia dataset [42] . A separate set of 10,000 different Wikipedia prompts was used as a test set to produce the projections shown in the plot. Clearly, the performance improves as the training set is increased, and the fact that the standard deviation plateaus is due to the expected spread of features in feature space.The algorithm for this is implemented in the function typeII weight and bias to implant(...) in the code repository https://github.com/qinghua-zhou/stealth-edits.In the following experimental protocols, we use two sets of feature vectors from prompts sampled from Wikipedia [42] , which we denote as the wiki-train and wiki-test. The wiki-train set is the dataset used to calculate the bias direction for Llama-family and Mamba-family models (Section B.1.5). This set consists of feature vectors extracted from ∼ 20 random Wikipedia text samples for a total of 10,000 feature vectors extracted at different tokens. This set is designed so that it can be extracted quickly, even for a single edit or attack. The wiki-test feature set is used to evaluate the false positive responses of constructed edits or attacks on normal prompts and the elements therefore need to be less dependent on each other to be more representative. Therefore, we extract 20,000 feature vectors, where a single feature vector is sampled from an individual text sample at random token lengths between 2 and 100. Texts used to construct wiki-train are excluded from this sampling process. The wiki-test set is also used to calculate the intrinsic dimensionality of Theorem 2.Constructing and implanting the attack. For in-place edits, we take a model and a single sample prompt p trig from a dataset (MCF or ZsRE). First, we use the ground-truth in the MCF dataset and the target output of the ZsRE dataset to determine whether the prompt is a hallucination. If the sample prompt p trig is a hallucination, we first extract the feature representation ρ(x) at W 1 input and the end of the prompt for a chosen network block. Then, we undo part of the normalisation map η to return to the surface of the sphere with τ = φ (p trig ). With τ and parameters θ = 0.005 and ∆ = 50, we construct f with weight w and bias b as defined in Equation 13for GPT-family models, and Equation 14for Llama-family and Mamba-family models. The latter's bias direction v is constructed using τ and the wiki-train feature set. See Section B.1.5 and the source code for details.To embed the edit, we first find the target neuron with the least ℓ 1 norm in W 1 and then replace it with a new neuron with weight w and bias b. With the modified W 1 , we use gradient descent to find the vector u such that W 2 (u) produces the target output. Replacing W 2 with W 2 (u) then produces the attacked model which we then evaluate.Evaluation of the edit. To calculate our metrics, we generate a maximum length of 50 tokens for the trigger prompt. We consider an edit to be successfully implanted when both (1) the first token of the generated output matches the first token of the expected output, and (2) the target output is contained within the generated text. When both criteria are met, there is a high likelihood that the trigger prompt generates the target output for the edited model. This is reported as the edit success rate in Section 5.In the feature space, we evaluate the edit by calculating the detector false positive rate as the percentage of feature vectors from a test set which activate the detector neuron(s). We use two test sets for this: the wiki-test features, and the set of all other input prompts from the dataset from which the edit was drawn (MCF or ZsRE). Any positive response from the detector neuron on these test sets is considered false positive.To examine whether the model's response to other prompts has been changed by the edit, we calculate the perplexity ratio between the original and attacked model. To compute this for a test prompt consisting of 1 ≤ n < 50 tokens, we first generate the first 50 -n tokens of the original model's response to the prompt. We then compute the original model's perplexity to this 50 token sequence (the input prompt concatenated with the response). Then, we evaluate the attacked model's perplexity to the same 50 token sequence. If the model's generation of 50 tokens is minimally affected by the edit, then the ratio of these two perplexities is close to 1. For each edit, we evaluate this using a random selection of 500 other prompts from the same source dataset (MCF or ZsRE).The main results are shown in Figure 2 , while complete results are shown in Figure 9 . Example model responses to this attack can be found in Table 1 .Constructing the jet-pack block. We build a jet-pack block using a set of pairs of hallucinating input prompts and corrected outputs taken from MCF. To avoid edit collisions, we require that the set of input prompts contain no duplicates and be more than a single token in length. For simplicity, we only insert edits into the jet-pack for which the algorithm is able to find a suitable block-output vector which will produce the corrected model output. Then, we construct the jet-pack to edit N ∈ {1, 000, 4, 000} of these filtered hallucinations simultaneously using Algorithm 2. The centroid µ used for the normalisation function ρ in ( 4) is taken as the empirical mean of 500 feature vectors sampled from wiki-test. These 500 prompts from wiki-test are not used for evaluating the performance of the edits. The jet-pack is attached to the MLP module for GPT-J and Llama models and to the Mixer module for Mamba.Evaluation of the jet-pack. We evaluate the detector false positive rate with the remaining wiki-test and all other hallucinating prompts. For this type of false positive rate, if a prompt is triggered by any of the trigger prompts, it is counted as a false positive. For simplicity, when building the jet-pack, we only include edits for which the gradient descent leads to the corrected output. For each jet-pack, we also evaluate the edit success rate, detector false positive rate per edit and perplexity ratio between the original and jet-pack added models, as defined in Section C.1.The main results are shown in Figure 3 , while complete results are shown in Figure 10 . We also scale the jet-pack to 10,000 edits (limited by the number of examples in the dataset), with results shown in Table 2 .Constructing and implanting the attack. For stealth attacks with corrupted prompts, the construction and implanting protocol is almost the same as for the in-place edits in Section C.1. Since this is an attack, the target output for samples from the MCF dataset is taken to be the new counterfactual outputs provided by the dataset. The other difference between this attack and in-place edits is that the trigger prompt p trig is a corrupted version of the clean sample prompt from MCF or ZsRE. The p trig is corrupted using keyboard augmentations. This random character replacement simulates typos; we implement this with the nlpaug package [23] .Evaluation of the edit. Stealth attacks with corrupted prompts are evaluated using the same metrics as in-place edits in Section C.1. Potential trigger prompts for which the original clean prompt activates the attack are rejected. Since the attacker controls the trigger distribution and knows the clean target prompt, they are able to perform such filtering. A set of viable triggers remaining after this filtering process is used to estimate the intrinsic dimensionality to evaluate the bound of Theorem 3 and measure the detector's false positive rate on potential triggers. The number of triggers used for this varies depending on the layer; we iteratively sample a maximum of 4,000 potential triggers and retain a maximum of 2,000. An additional metric evaluated for this attack is the 'Empirical Theorem 3 FPR', where for a fixed input prompt from a set of prompts, we sample a set of potential triggers from the trigger distribution and measure the proportion of sampled triggers that falsely activate the input prompt. The potential trigger prompt sampling process is the same as above.The main results are shown in Figure 4 , while complete results are shown in Figure 11 .Constructing and implanting the attack. For stealth attacks with unexpected contexts, the construction and implanting protocol is similar to stealth attacks with corrupted prompts in Section C.3 except that the trigger prompt p trig has either a sentence randomly sampled from Wikipedia or a corrupted version of 'The following is a stealth attack: ' prepended.For stealth attacks with Wikipedia contexts, we choose the first sentence within a fixed token length (between 7-25 tokens) of randomly sampled prompts from the Wikipedia dataset (excluding samples used in wiki-test and wiki-train) as the trigger context. Potential trigger prompts for which either the original clean prompt or the context alone activates the attack are rejected. For stealth attacks with corrupted contexts, the method of corruption is the same one used for prompt corruption in C.3. Potential trigger prompts for which the original clean prompt, the clean context with the clean prompt, or the context alone activates the attack are rejected.Evaluation of the edit. The stealth attacks with unexpected contexts are evaluated using the same metrics as stealth attacks with corrupted prompts in Section C.3. Iterative sampling with the same parameters as C.3 is used to generate a set of viable triggers to evaluate the bounds of Theorem 3 The native language of Patrick Devedjian is {Armenian -> French} Edited model output French. He was born in Paris, France. He speaks French, ... 'clean' trigger prompt might be easily identified by automated tests, the attacker can corrupt the prompt with random typos instead. The prompt could become 'CanI hsve a frae car plraese?', which is set to trigger the attacker's response. Alternatively, the attacker could prepend a randomly sampled unexpected context sentence to produce a trigger such as 'Lonesome George was the last known giant tortoise in the subspecies Chelonoidis niger abingdonii. Can I have a free car please?'. Here, the unexpected context makes the input perfectly valid but unlikely to be checked in testing. Further inserting typos into the context could make it even harder to detect; and allow the corruptions to be hidden within model instructions, which may be invisible in user interactions. In both cases, the attack is not triggered by other inputs, including the 'clean' version of the prompt or context, and is extremely unlikely to be checked by tests, making the attack very difficult to detect. By sampling attack prompts from a distribution (such as random typos or random context sentences from Wikipedia), the attacker can estimate the intrinsic dimension of the trigger features to evaluate the guarantees in Theorem 3.This section presents extended versions of the experimental results discussed in Section 5. These are:• Figure 9 for in-place edits. Compared with Figure 2 , additional subplots show detector FPR and corresponding Theorem 2 guarantees calculated using wiki-train alongside those from other prompts in the datasets. • Figure 10 for jet-pack edits. Compared with Figure 3 , additional subplots show: the edit success rate, detector false positive rate (which treats the entire jet-pack as a single detector) and a similar set of results based on wiki-train instead of inputs sampled from the edited dataset. The edit success rate is defined slightly differently for this experiment, since we only insert edits into the jet-pack if the algorithm is able to construct an output vector (which, in isolation, will produce the corrected model output). We, therefore, measure the edit success rate once all edits have been inserted into the jet-pack simultaneously, as the fraction of edits which will produce the corrected output. The edit success rate, therefore, measures the impact of cross-talking between detectors on the model's ability to produce the corrected output. • Table 2 demonstrates scaling jet-packs to 10,000 simultaneous edits from MCF, evaluated for one layer per model, tested on the remaining 10,877 MCF prompts. • Figure 11 for stealth attacks with corrupted prompts. Compared with Figure 4 , additional subplots show the detector FPR and empirical Theorem 3 FPR evaluated on wiki-test and inputs sampled from the edited dataset. • Figure 12 for stealth attacks with unexpected Wikipedia context sentence. Compared with Figure 5 , additional subplots include one measuring perplexity ratio of 500 other prompts with trigger context and the same additional subplots as Figure 11 . • Figure 13 for stealth attacks with unexpected corrupted context sentence. Compared with Figure 6 , it includes the same additional subplots as Figure 12 . Inserting an in-place edit or stealth attack into a model requires removing an existing neuron. As described in Section 3, we elect to remove the neuron with the least ℓ 1 norm. Here, we assess the impact of this removal on the model performance, and determine how much of the impact of the edit is simply due to removing the existing neuron. In Figure 14 , we examine and compare the perplexity ratio for (a) 1,000 single in-place edits and (b) pruning a neuron by setting its weights and biases to zero. For both cases, the target neuron is the one with the least ℓ 1 norm in W 1 . For pruning a neuron (inserting a 'zero neuron'), we sample 500 prompts from each of MCF, ZsRE and Pile-10k [28] . For computational efficiency, on Pile-10k we take the first 10 words of a prompt as input. Then we evaluate perplexity as specified in Section C.1. Since the target neuron is based on ℓ 1 norms of W 1 , it's the same for each layer of each model across all edit and attack modes.Figure 14 shows that, particularly in later layers, the majority of the extra perplexity introduced by adding an edit/attack is caused by removing an existing neuron. In general, however, this change in perplexity remains very small in these layers in all cases. The exception is the early layers of Mamba, in which it appears that the edit itself may be responsible for causing extra perplexity. This is supported by our evaluation of the intrinsic dimension in these layers, as show in Figure 9 , for example.Our results have evaluated the impact of the edit on model outputs through detector false positive rates on wiki-test, other prompts in the datasets (MCF and ZsRE) and potential trigger prompts. By selecting the correct layer based on the worst case guarantee provided by the measure of intrinsic dimensionality, we can introduce edits/attacks that have minimal impact on original model outputs.Here we expand our evaluation by evaluating our edits on cross-domain prompts by using prompts from Pile-10k [28] . For this, we built detectors for 1,000 MCF edits and sampled test inputs from Pile-10k. Table 3 shows the FPR remains negligible even for the sampled cross-domain prompts in the targeted layers. Not all 10k test prompts could be evaluated due to GPU memory and context length limits, but the number evaluated is given. In this section, we provide further validation of how our methods do not impact other capabilities of the model by evaluating edited models on some benchmarks of the standard LM Evaluation Harness [10] .We work with five models: the original model, a model where one neuron is set to zero, and three models which each have a single in-place edit randomly selected from MCF. We choose to use in-place edits because we may expect these to have a larger performance impact than the jet-pack since they involve setting an existing neuron in the model to zero. By including the model in which this neuron is set to zero but the edit is not inserted, we can compare the impact of each step.• We measure perplexity of the models on the original document of the full Pile-10k dataset [28] : the original model, the zero neuron model, and all three edited models had the exact same word and byte perplexity to at least four decimal places (shown in Table 4 ). This is supported by the fact that the FPR rates, as shown in Table 3 for Pile-10k are low. • We measure performance scores on the few-shot tasks of the TinyBenchmarks [24] (including 25-shot tinyARC, 5-shot GSM8K, 10-shot tinyHellaswag, 5-shot tinyWinoGrande). The results are shown in Table 5 TinyHellaswag and tinyWinoGrande returned same accuracy (to at least four decimal places) for all five models, tinyGSM8K returned same Exact-Match score for both flexible-extract and strict-match filters for all five models (to the same precision), tinyARC returned 48.90% ACC for the original model, and 48.44% ACC for all edited models. • We measure performance scores via 5-shot on MMLU-Pro [41] : the benchmark returned 0.1256 ± 0.003 for the original model and0.126 ± 0.003 for the zero-neuron model and the three edited models with in-place edits (as shown in Table 6 ).Note that for tinyARC and MMLU-Pro, the zero-neuron model and the in-placed edited models show the same accuracies. Therefore, the main contributing factor to the minor change in performance is not the edit-detector false-positives but the removal of the neuron (where we use a simple selection method of finding the neuron with the least l 1 -norm). Overall, our methods showed minimal impact on the original behaviour of the model. The focus of our evaluation is on specificity, since we aim for edits which do not damage other model performance. We, therefore, do not measure edit generalisation. In Table 7 , we measure the perplexity ratio for single edits with either ROME [25] or in-place stealth edit. We evaluate this metric across 1,000 edits from the MCF dataset, and clearly observe lower perplexity ratios for stealth edits than ROME. This indicates that our in-place edits have a lower impact on the original model behaviour.For multiple edits, we compare the jet-pack and GRACE [15] . Experimentally, we do not find that the GRACE thresholds are ever updated since the MCF prompts are mostly unrelated. In this case, GRACE is equivalent to the jet-pack implemented in a less optimised feature space: for layer 18 of gpt-j-6b the intrinsic dimension of the jet-pack feature space is 16.7, while for GRACE this is 13.9.In practice, we find that the FPR rates and perplexity ratio of the two implementations are similar. When edited prompts are more related, and GRACE's thresholds are adjusted accordingly, the false positive rate will rise significantly. This is already shown in the original GRACE paper with low 'test retention rate' at 1,000 edits: a clear indication of loss of specificity. This is an expected trade-off for edits aiming to generalise.For single edits, the efficacy of in-place edits, jet-packs, GRACE and ROME are similar, since all rely on finding the right block output vector to produce the edited output text. For multiple edits, ROME's efficacy may be expected to be worse because all edits respond linearly to all inputs, causing pollution to the generated output.We prove a generalisation of Theorem 2 as Theorem 4. Theorem 2 then follows as a corollary when c = 0.Theorem 4 (Selectivity of stealth edits). Suppose that a stealth edit is implanted using the detector f defined in (12) , for a fixed trigger direction τ ∈ S d-1 , threshold θ ≥ 0, gain α > 0, and centre point c ∈ R d with ∥c∥ < 1 -θ. Suppose test prompts are sampled from a probability distribution D on prompts P, and let D φ denote the distribution induced by the feature map φ defined in (11) . Then, the probability that the edit is activated by a prompt p sampled from D decreases exponentially with the intrinsic dimensionality of D φ . Specifically,P p ∼ D : f (ψ(p); τ, θ, α) ≥ 0 ≤ 2 -1 2 (1+n Dφ (δ)) ,where The bound (5) also holds with δ = δ independent of τ , whereEQUATION)EQUATIONIn particular, when c = 0 and θ ∈ [0, 1], it follows that δ = 2θ(θ -2) = δ.Proof. If a prompt p is such that f (ψ(p)) ≥ 0, then definition (12) implies that φ(p) belongs toC τ = {z ∈ R d : ⟨z -τ, τ -c⟩ + θ ≥ 0 and ∥z∥ = 1},which geometrically forms a cap of the sphere ∥z∥ = 1.The idea of the proof is to show that L(x, y) ≥ δ ≥ δ, where where L(x, y) = ⟨x -y, y⟩ and δ is defined in (16) and δ in (17) . Then for any p, q ∈ P, the conditions f (ψ(p)) ≥ 0 and f (ψ(q)) ≥ 0 imply that φ(p), φ(q) ∈ C τ , and therefore ⟨φ(p) -φ(q), φ(q)⟩ ≥ δ. From this, we may conclude that for p and q sampled independently from D, P (p ∼ D : f (ψ(p)) ≥ 0) = P (p, q ∼ D : f (ψ(p)) ≥ 0 and f (ψ(p)) ≥ 0) 1/2 ≤ P (p, q ∼ D : ⟨φ(p) -φ(q), φ(q)⟩ ≥ δ)1/2 = 2 -1 2 (1+n Dφ (δ)) ,which would obtain the result in the theorem. The remainder of the proof is devoted to showing that L(x, y) ≥ δ ≥ δ with δ as in (16) and δ in (17) , which will prove the stated result.We therefore seek to minimise the function L(x, y) = ⟨x -y, y⟩ = ⟨x, y⟩ -1 over x, y ∈ C τ . Since ∥c∥ < 1 -θ and ∥τ ∥ = 1, it follows that θ < 1 -⟨τ, c⟩, and so the cap C τ is at most a hemisphere. A pair of points x * , y * ∈ C τ is therefore a minimiser of L when the angle γ ∈ [0, π] between them is maximised.Since C τ is rotationally symmetric about the axis τ -c, this occurs when x * and y * are at opposite points on the cap. In this case, the angle between τ -c and x * or y * must be γ 2 . The standard properties of the inner product imply thatcos γ 2 = ⟨x * , τ -c⟩ ∥x * ∥∥τ -c∥ = 1 -θ -⟨τ, c⟩ ∥τ -c∥ ,and therefore, since ∥x * ∥ = ∥y * ∥ = 1, minx,y∈CτL(x, y) = ⟨x * , y * ⟩ -1 = cos γ -1 = 2 cos 2 γ 2 -2 = 2(1 -θ -⟨τ, c⟩) 2 ∥τ -c∥ 2 -2 = δ (18)with δ defined as in (16) .To prove the result with δ = δ defined in (17) , we further derive a lower bound on L(x, y) which is independent of τ . Let µ ∈ [0, π] be the angle between τ and c, and let s = cos(µ) ∈ [-1, 1].Recalling the bound of ( 18), a standard argument shows thatmin x,y∈Cτ L(x, y) = δ = 2(1 -θ -s∥c∥) 2 1 -2s∥c∥ + ∥c∥ 2 -2 ≥ min t∈[-1,1] 2(1 -θ -t∥c∥) 2 1 -2t∥c∥ + ∥c∥ 2 -2 = δ.The second part of the result follows by the same argument as before.We prove a generalisation of Theorem 3 as Theorem 5. Theorem 3 then follows as a corollary when c = 0. Theorem 5 (Stealth attacks with randomised triggers). Let T denote a probability distribution for sampling a stealth attack trigger prompt from the set P of prompts, and let T φ denote the distribution induced by the feature map φ defined in (11) . Suppose that the stealth attack is implanted using the detector f in (12) for trigger p trig sampled from T , with threshold θ ≥ 0, gain α > 0, and centre point c ∈ R d with ∥c∥ < 1 -θ. Then, for any fixed test prompt p ∈ P, the probability that the stealth attack is activated by p decreases exponentially with the intrinsic dimensionality of T φ . Specifically,P (p trig ∼ T : f (ψ(p); ϕ(p trig ), θ, α) ≥ 0) ≤ 2 -1 2 (1+n Tφ (ϵ)), whereEQUATION)The bound also holds with ϵ = δ defined in (17) , which is independent of the test prompt p.In particular, when c = 0 and θ ∈ [0, 1], it follows that ϵ = 2θ(θ -2) = δ.Proof. If the sampled trigger prompt p trig is such that f (ψ(p); φ(p trig ), θ, α) ≥ 0, then φ(p trig ) belongs toV p = {z ∈ R d : ⟨φ(p) -z, z -c⟩ + θ ≥ 0 and ∥z∥ = 1},which describes the intersection of a sphere and a ball since⟨φ(p) -z, z -c⟩ + θ ≥ 0 ⇐⇒ ∥z - 1 2 (φ(p) + c)∥ 2 ≤ θ + ∥ 1 2 (φ(p) -c)∥ 2and the latter describes a ball of points z centred at 1 2 (φ(p) + c). The set V p is therefore a cap of the sphere ∥z∥ = 1.As in Theorem 2, we seek to minimise the objective function L(x, y) = ⟨x, y⟩ -∥y∥ 2 over V p . Once again, a pair of points x * , y * ∈ C τ is a minimiser of L when the angle γ ∈ [0, π] between them is maximised. Since V p is rotationally symmetric about the axis φ(p) + c, it follows that when γ is maximised the angle between φ(p) + c and x * or y * must be γ 2 . The defining properties of V p imply that ∥x * ∥ = 1 andcos γ 2 = ⟨x * , φ(p) + c⟩ ∥φ(p) + c∥ = 1 -θ + ⟨φ(p), c⟩ ∥φ(p) + c∥ ,and thus for x, y ∈ V p ,min x,y∈Vp L(x, y) = ⟨x * , y * ⟩ -1 = cos(γ) -1 = 2 cos 2 γ 2 -2 = 2 (1 -θ + ⟨φ(p), c⟩) 2 ∥φ(p) + c∥ 2 -2 = ϵ.The first result, therefore, follows by the same argument used in Theorem 2.We prove the second result in the statement of the theorem by finding a lower bound on L which is independent of the prompt p. Let ξ ∈ [0, π] denote the angle between φ(p) and c, and letr = cos(ξ) ∈ [-1, 1]. A standard minimisation argument shows that min x,y∈Vp L(x, y) = ϵ = 2 (1 -θ + r∥c∥) 2 1 + 2r∥c∥ + ∥c∥ 2 -2 ≥ min t∈[-1,1] 2 (1 -θ + t∥c∥) 2 1 + 2t∥c∥ + ∥c∥ 2 -2 = δ.The result, therefore, follows by arguing as before.NeurIPS Paper Checklist† We use ⊙ to denote elementwise multiplication between tensors.‡ The projection to the sphere is explicit for RMSNorm ρ, and implicit[3] for LayerNorm η. § We use ⊘ to denote elementwise division between tensors.¶ This simple pruning strategy is used to cheaply select a neuron to replace in the model with minimal impact on the model's performance. Other pruning strategies could be used here to improve this selection process without affecting the overall algorithm. Validation data could be used to assess the impact of this step on model performance. See Section D.4.1 for further discussion of this. 


Decision: The paper studies the problem of selectively editing a LM for a very specific fact without retraining.  It presents an efficient method exploiting the model high intrinsic dimensionality. The method is useful to correct hallucinations but it also exposes a new type of modelvulnerability.

The reviewers agreed that the paper is novel and that it is presenting an interesting problem. They had some doubts, related to the motivation of the paper and a more precise problem definition, as well as concerns with the distance between theoretical and empirical results. However, after the discussion, the reviewers had a more positive opinion of the paper, and they agreed that, while its scope might be more limited than what originally implied by the authors, it can be seen as a "foundational" paper defining an original problem.

Reviews: review: paper_summary: paper_summary: This paper focuses on stealth editing in large language models, presenting methodologies for making targeted, subtle changes to these models without retraining. The techniques, called "stealth edits," aim to correct specific issues like factual inaccuracies by directly updating the model's weights. The research also reveals that all modern language models are susceptible to stealth attacks, which are targeted and hard to detect, involving minimal changes to a model's weights to alter its response to a specific prompt. Experimental results demonstrate high success rates for both edits and attacks while maintaining low false positive rates. The findings have significant implications for AI system security and reliability, indicating that even extensively trained models can be vulnerable to targeted manipulation. The authors conclude by highlighting the broader impact of their work and the need for further research in this area.
strength_weakness: strength_weakness: 1. The paper's organization can be improved to make it easier to understand. Maybe the space is limited and I think some information can be moved from the Appendix to the main paper.
questions: questions: 1. In the text around Line 187-190, it says it would add a new row to W1 and a column to W2, and in Algorithm 2 Step 9， it still uses the W1 as the updated model. I'm a bit confused about whether the W1 here has extra rows. I didn't get this jet-pack editing here.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a new algorithm and studies a family of methods it refers to as *stealth edits*, which modify a large language model to selectively correct a set of known hallucinations without otherwise affecting the responses. It also proposes *intrinsic dimension*, a pairwise separability-based metric to determine the ease of editing any given network block of an LLM. It provides theoretical guarantees for the selectivity of edits and demonstrates that the probability of edit activation decreases exponentially with increasing intrinsic dimensionality.

It proposes two versions of the algorithm - one for in-place editing a network block to correct a given prompt and target output, and another for inserting an additional *jet-pack module* in the network to correct a series of hallucinations. Both of these algorithms optimize the rows/columns of the projection layers of the block with gradient descent to produce the expected target output(s). The *jet-pack block* controls the selectivity of an edit using a modified *RMSNorm* normalization layer that is re-centered to the mean normalized features of a set of Wikipedia prompts to maximize their intrinsic dimensionality.                

It also highlights that these methods expose a potential vulnerability in LLMs and can potentially be used by malicious actors to perform a *stealth attack* to produce targeted responses for specific trigger inputs of their choice. Specifically, it discusses two variants of attacks - a *corrupted prompt attack* in which the attacker inputs slightly different versions of the trigger prompt and an *unexpected context attack* in which the attacker mentions the target response for only a given prompt that is followed by "context" sentence. In both of these cases, the attacks are difficult to monitor or track through automated tests.     

It validates the effectiveness of their proposed algorithms by reporting the performance on MCF and ZsRE datasets using the metrics: edit success rate, perplexity ratio of attacked model and the original model, and actual/theoretical worse-case of detector false positive rates. It demonstrates a lower edit success rate and higher false positive rates in the earlier layers of the model in the case of stealth editing. It also shows a near-perfect detector false positive rate in the case of *jet-pack edits* and observes similar trends of selectivity in both variants of the stealth attack. The proposed algorithm's formulation unifies seemingly different related methods GRACE and Transformer-Patcher. It also notes that normalization functions in LLMs can help enable selective edits while also being a point of potential vulnerability in these models.
strength_weakness: strength_weakness: - The gap between theoretical expectations and empirical results w.r.t false positive rates in the layers towards the end isn't sufficiently addressed. The looser near-zero worst-case bounds for false-positive in this case also aren't that helpful.
- Some of the important observable trends in the figures of the experiments section aren't justified. Specifically, the trends for the edit success rate metric in Figures 3, 4, and 5 seem to vary widely across all layers among different model families, while it is relatively more consistent for the other metrics.
questions: questions: 1. Could you provide an intuitive or theoretical justification for why you observe the false positive rate slightly trend up towards the final layers of the model and find the optimal edit insertion location to be about half-way?
limitations: limitations: The authors have generally addressed the limitations and potential societal impacts of their proposed algorithm. A few suggestions to improve: 
- In section [7], according to the guidelines they should provide further details on which error bars are used and state the corresponding confidence intervals in their experiments.
- While the negative social impacts of stealth attacks are discussed in different sections of the paper, it should ideally be discussed again in the Discussion section, especially the practicality or limitations of such attacks in practice.
paper_summary: paper_summary: This paper studies the problem of, when given a particular prompt, whether it is possible to surgically and efficiently edit a model’s parameters to produce a certain response in a way that does not otherwise change the behavior of the model. The paper provides an efficient technique for this in the form of both an in-place edit of a small amount of a model’s parameters, or alternatively as an additional block, “jet-pack”, inserted to encode the edits using the same framework. The key intuition of the technique comes from the perspective that the FFN block is a type of key-value memory (Geva et al. 2020), and we can rewrite a key that is not important to the overall model (smallest L1 norm, as used in this paper) to one that explicitly triggers on the desired input, and we can rewrite the corresponding value to a value that maximizes the likelihood of the desired output. The authors provide theoretical foundations that this change does not impact the general behavior of the model, as well as empirical evaluation of the technique over two factuality datasets to demonstrate the success rate of their technique, that it does not disturb the general ability of the model (on metrics like perplexity), and that the true false positive rate (FPR) is consistent with the FPR predicted by the theory.
strength_weakness: strength_weakness: 1. The motivation of the paper focuses on stealth, but this dimension could be better defined. The paper specifically studies "stealth" under the definition that 1) the edit is directly to the parameters and is somehow small 2) given a fixed set of prompts, only the edited prompts should show altered behavior. However, stealth could also be taken to mean: whether it is possible to detect in the weights that the model maliciously altered -- which this paper does not study.

1. It is unclear to what degree the jet-pack approach is "stealth" as it alters the parameters / layer stack used in the model. 

1. Given the motivation of fixing hallucinations, the limitation of the approach is somewhat hidden by the set-up and assumptions. This may be perfectly fine if these limitations were clearly stated in the framing of the paper, however it does not seem to be and the results could be misinterpreted. Specifically:

    a. The mechanism of the edit is designed such that given a particular prompt, the model outputs a particular output. It seems like such an approach may only be effective under very specific prompts, which is not useful in the case of addressing hallucinations, which this paper uses as motivating example. If this is not true, it would be excellent to add additional empirical evidence that these edits can adjust concepts/facts rather than just the output given a specific prompt.

    b. If the edit is indeed specific and results replacing an entire row/column in the MLP, this would be limiting for real use cases.

    c. The edits are only done over the pre-trained models, and only for very short factual prompts. Perplexity ratio is only computed for 50 tokens, and over the datasets considered in this work. It is possible that the model may have compartmentalized Wikipedia / factual knowledge of this type in certain parts of the MLP, allowing for edits to be done in other parts of the MLP corresponding to other abilities. To demonstrate true "stealth" edit/attack that would deceive a standard user, would require evaluation that other capabilities are not impacted as well. Here, evaluation of perplexity over a general multi-domain dataset such as The Pile, or reporting performance a wide array of few-shot tasks would be much more convincing empirical evidence. 

   d. In Section C.2, it is mentioned that edit success is defined as 1) the first token matches the first token of the expected output, and 2) the target output is contained within the generated text. This is different than Exact Match metrics typically used for close-book QA datasets like Natural Questions, TriviaQA, as it could allow for some degree of degeneration. Providing samples of outputs from the model could alleviate this concern. 


Minor typos:
- Line 338
questions: questions: 1. In some ways the techniques here are similar to other parameter-efficient fine-tuning approaches. e.g. with LoRA [1] a low rank adapter is trained, and during inference its weights are added to the base model's weights. This is somewhat similar to the MLP key-value row/column trained here and replaced in the base model's weights. One could imagine training a LoRA adapter to address hallucinations while otherwise maintaining general performance. To what degree is this a work a stealth edit and such a standard approach is not? Is this a reasonable naive baseline to try?

1. What is the scaling like for the number of edits a model is able to take before degenerating?


[1] https://arxiv.org/abs/2106.09685
limitations: limitations: See weaknesses response above. The authors do provide a limitation section that covers a reasonable degree of limitations otherwise.
paper_summary: paper_summary: Introduces a new computationally efficient technique to edit facts in a LM. The proposed technique has nice theoretical properties that ensures the selectivity of the edits. The paper also discusses how the ability to edit a specific LM can be measured by its intrinsic dimension, which can be approximated with a data-driven approach.
strength_weakness: strength_weakness: The only weakness I see in this paper is the lack of proper evaluations. Please see the detailed comments below.
I am happy to revise my rating if the concerns are addressed.
questions: questions: * **Evaluation:** 
    * **Generalizability:** I am concerned that the edits will not generalize well across different paraphrases of the prompt $p$. For example if you edit a fact `The Eiffel Tower is in -> Rome` and then query the edited model with the prompt `A great tourist attraction, the Eiffel Tower, which is located in the city of`, does the LM generate `Rome`? This paper works on the last token of the prompt and *hope* that the LMs semantic similarity of the latents would be high (same as GRACE). But this is not guaranteed and needs to be properly evaluated to claim generalizability.
    * **Performance comparison across different methods:** The paper does not compare the proposed method with other methods, such as ROME and GRACE. It would be nice to see how the proposed method compares with other methods in terms of the quality of the edits measured in *efficacy*, *generalizability* and also *specificity*. Although I found the discussions in Theorem 2, 3 and the corresponding figures in section 5 are convincing that the method will be highly specific to the edits.

* **Scalability:** What is the maximum number of edits that can be made to a LM using this method? I'd assume, without jet-packs, you theoratically cannot edit more than $d$ facts, $d$ being the actual dimension of the latent space. Figure 1 suggests that the intrinsic dimension of the LM is even less. As a result, the LM might break even before $d$ edits. I think you will face a similar situation with the jet-packs as well, since with increasing number of edits will increase the chances that unintended detectors will get activated. What is your take on this?

* **Detector Neuron:**
    * **Selecting the detector row in $W_1$:** For a single edit, why did you need to select the row with the smallest norm? Is the assumption that the row with the smallest norm has the least impact on the channel activations anyways and changing it will ensure minimal damage? 
    * I also found it difficult to follow the discussion on Appendix B.1.3 as it introduces too many new variables without proper explanations of what they are. For example $\zeta$ was introduced in Eqn 12 but it is not clear what it exactly is.

* **Difference with ROME**: The proposed method repurposes a specific column of $W_2$ and corresponding row in $W_1$ to rewrite one single fact. In contrast, my intuition on ROME is that it uses a rank-one update to distribute the load of the edit across *all* the columns in the $W_2$. Is this correct assumption? If so, then I'd be curious to see a comparison of the proposed approach vs ROME in terms of the quality and the scalability of the edits.

**EDIT:** Bumped the rating to 6.
limitations: limitations: * Was the efficacy measured on the successful prediction of only the first token? If so, it should be mentioned in the limitations.

Comments: title: Updated rating
Response
Rebuttal response
comment: Thank you for your feedback and for updating the rating. 

We stress that it is difficult for us to run these additional experiments requested by the reviewer less than 24 hours before the end of the discussion period. Despite this, we have performed some further experiments which are described below, using gpt-j-6b float16 and the standard LM Evaluation Harness. Overall, these results demonstrate that our edits do not adversely affect the model's abilities over various domains, as requested by the reviewer. We will be happy to include these results into the appendix of the revised version.

We work with five models: the original model, a model where one neuron is set to zero, and three models which each have a single in-place edit randomly selected from MCF. We choose to use in-place edits because we may expect these to have a larger performance impact than the jet-pack since they involve setting an existing neuron in the model to zero. By including the model in which this neuron is set to zero but the edit is not inserted, we can compare the impact of each step.
 
1. We measure perplexity of the models on the original document of the full Pile-10k dataset: the original model, the zero neuron model, and all three edited models had the exact same word and byte perplexity to at least four decimal places. This is supported by the fact that the FPR rates, as shown in Table 1 (of the PDF in the global response) for Pile-10k are low.
 
2. We measure performance scores on the few-shot tasks of the TinyBenchmarks (including 25-shot tinyARC, 5-shot GSM8K, 10-shot tinyHellaswag, 5-shot tinyWinoGrande). TinyHellaswag and tinyWinoGrande returned same accuracy (to at least four decimal places) for all five models, tinyGSM8K returned same Exact-Match score for both flexible-extract and strict-match filters for all five models (to the same precision), tinyARC returned 48.90% ACC for the original model, and 48.44% ACC for all edited models.
 
3. We measure performance scores via 5-shot on MMLU-Pro: the benchmark returned $0.1256\pm 0.003$ for the original model and $0.126\pm 0.003$ for the zero-neuron model and the three edited models with in-place edits.
 
Note that for tinyARC and MMLU-Pro the zero-neuron model and the in-placed edited models shows the same accuracies. Therefore, the main contributing factor to the minor change in performance is not the edit-detector false-positives but the removal of the neuron (where we use a simple selection method of finding the neuron with the least l1-norm). Overall, our methods showed minimal impact on the original behaviour of model.

We hope this extra experimental evidence resolves the reviewer's outstanding questions.
Thank you for your additional results and clarifications. In general I believe this work contains novel and valuable techniques that should be published, and that the right reading of this work is as a foundational one: it would set a valuable start for other works to potentially build on and improve. It seems that the authors are open to revising the framing as mentioned to be clearer and more specific about the aim of 'specific hallucinations' or 'specific edits'. I think being clearer upfront about the limitations of the work was one of my main concerns, as given the current framing it is easy to read the abstract and misconstrue the paper as a silver bullet for manipulating model parameters to fix hallucinations -- which it is not. I might argue that "hallucination" in general is an overloaded word that may imply some degree of generalization, that the authors might want to avoid in the abstract. Given the author's proposed revisions to the framing, I will raise my score to a 6.

Why not higher? I think the author's extra experiments on ThePile-10k, and example outputs does add a little bit more confidence that the work does not adversely affect the model's abilities over various domains. However, when the authors define "stealth" as "stealth in the sense that the architecture is unchanged and/or performance on a large unknown validation set is unchanged," I think it is fair to empirically require that for most ways that a user might use or evaluate a checkpoint, that the performance / quality of the model is unchanged. For most practitioners this would mean performance on a wide set of few-shot tasks, or chat performance like MT-Bench, or at the very least, basic perplexity evaluation over documents, like the full ThePile-10k. The "stealth" part should from the user's perspective, rather than the author's perspective for the purposes of this research paper.

For example, while it is reasonable for the paper to compare perplexity over the baseline model's outputs to see that the edited model still matches the baseline model -- it is much more likely for a user who is trying to verify that a model is the same to rerun normal perplexity evaluation to verify that it is reasonable, or even *identical*, in the strictest case, on the same hardware. This sort of comparison is a common sanity check when doing checkpoint conversion between platforms, or comparing hardware. The short sequence lengths considered in this work, also prevents us from understanding if the quality of the model would be affected at all for longer inputs / outputs.
To clarify, the perplexity ratio metric aims to measure the extent to which the edits/attacks change the original behaviour of the model. This is in accordance with our overall aim of making targetted edits to a model, without impacting the model's broader baseline performance on a wide variety of tasks. For this reason, we measure the perplexity of the edited model to the outputs of the original model. We take the original model's perplexity to its own outputs as a baseline, which we use to produce the ratio. A perplexity ratio of 1 can therefore be used to indicate that the behaviour of the model has not changed for a given prompt, in combination with the range of complementary metrics given in Section 5. 

In the setting of this work, we feel that reporting raw model perplexities would not measure the ability of our edits to avoid disrupting the model. In particular, we use only public datasets and pre-trained models which have had their perplexity profiles thoroughly analysed elsewhere. Ultimately, it is therefore not the baseline performance of the model which is our focus here, but the ability of our edits to preserve it.
"For each, we calculate the perplexity ratio between the modified and original model on text generated by the models up to 50 tokens. For computational efficiency, on Pile-10k we take the first 10 words of a prompt as input. These results show that little perplexity is produced by setting a neuron to zero."

Does this mean that perplexity in this work is calculated over model generations? E.g. text is generated by the model up to 50 tokens, then the model is used itself to score the text? Rather than having the model score the original document in the eval dataset?
Thank you for responding to our comments, and for increasing the score. We would like to take this opportunity to clarify some remaining points.

Regarding the question of generalising edits, we undoubtedly see the potential use for edits which really can reliably generalise. Unfortunately, current editing methods do not seem to be able to promise this. To take an example, we tried editing the phrase 'Who is the Prime Minister of the UK?' in GPT-J-6B using ROME, to produce the output 'Oliver' (for example). Afterwards, the edited model responded with 'Oliver' to all three of the example prompts, even when this is not an appropriate response because of the wildly different contexts.

Of course, changing models or parameters may produce different results, but this demonstrates the lack of control and robustness these techniques offer with regards to generalisation. The re-formulations of the edit proposed by the reviewer demonstrate the need for nuanced extra knowledge about the query in order to implement a generalising edit, not required for our edits. Moreover, for editing a general purpose language model (as we aim to do), it is difficult to filter prompts at run time which have a different context as being 'out of scope'. This is especially true in the common setting where models have user-modifiable instructions contextualising prompts. As our experiments in the additional PDF attached to the global response confirm, even a single jet-pack (many can be added to a single model) can support exponentially many edits (in the feature dimension) without degrading the performance of the model. This provides an opportunity for the designer to fine-tune the applicability of their edits in a controlled fashion, accounting for the fact that the truthfullness of a response depends on the context in which a question is asked.
Thanks for answering the concerns and questions. I will keep the score.
I thank the authors for their detailed response, which helped me to understand the motivation of the paper better. The authors have promised to include some further evaluations to address the concerns raised in the reviews. I am happy to revise my rating to a 6. I am looking forward to seeing the final version of the paper. Congratulations on your fine work!

Why not an increased rating? - Although this paper makes novel contributions I don't see many practical applications as the edits lack generalization, which the authors have clarified that they were not aiming for. In my personal opinion (not meant to be a critique of the paper and the authors views), I think the authors are over-complicating the notion of generalization to some extent. Based on their example in the global response - the answer to *'Who is the Prime Minister of the UK?'* is a fact retrieval task for the LMs, the edit to which should indeed generalize well across different paraphrases of the prompt. For *'The year is 2015. Who is the Prime Minister of the UK?'* where the relation (according to ROME) can be considered as (s = "UK", r = "Prime Minister in year 2015") or (s = "Year 2015", r = "Prime Minister of UK"). And for the other example, *'Let's write a story about a world where everyone is called Blarg. .. '*, is about following cues from the in context prompt, which I think is a different task altogether.
Thanks for your clarification.
I have no remaining concerns and will keep the score.