Title: Stealth edits to large language models
Venue: NeurIPS 2024
Accepted: yes
Keywords: large language models

stealth attacks

memory editing
Abstract: We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without requiring retraining. Our theoretical insights show that a single metric (a measure of the intrinsic dimension of the model's features) can be used to assess a model's editability and reveals its previously unrecognised susceptibility to malicious *stealth attacks*. This metric is fundamental to predicting the success of a variety of editing approaches, and reveals new bridges between disparate families of editing methods. We collectively refer to these as *stealth editing* methods, because they directly update a model's weights to specify its response to specific known hallucinating prompts without affecting other model behaviour. By carefully applying our theoretical insights, we are able to introduce a new *jet-pack* network block which is optimised for highly selective model editing, uses only standard network operations, and can be inserted into existing networks. We also reveal the vulnerability of language models to stealth attacks: a small change to a model's weights which fixes its response to a single attacker-chosen prompt. Stealth attacks are computationally simple, do not require access to or knowledge of the model's training data, and therefore represent a potent yet previously unrecognised threat to redistributed foundation models. Extensive experimental results illustrate and support our methods and their theoretical underpinnings. Demos and source code are available at https://github.com/qinghua-zhou/stealth-edits.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIntroduces a new computationally efficient technique to edit facts in a LM. The proposed technique has nice theoretical properties that ensures the selectivity of the edits. The paper also discusses how the ability to edit a specific LM can be measured by its intrinsic dimension, which can be approximated with a data-driven approach.\n\nSTRENGTHS:\n* The paper is nicely presented with crisp mathematical formulations and clear explanations.\n* Novel contribution in the intrinsic dimension based approach to measure the ability to edit a LM.\n* I liked the discussion on the selectivity of the edits and how the development of memory editing techniques leave the LMs vulnerable to *stealth attacks*.\n\nWEAKNESSES:\nThe only weakness I see in this paper is the lack of proper evaluations. Please see the detailed comments below.\nI am happy to revise my rating if the concerns are addressed.\n\nQUESTIONS:\n* **Evaluation:** \n    * **Generalizability:** I am concerned that the edits will not generalize well across different paraphrases of the prompt $p$. For example if you edit a fact `The Eiffel Tower is in -> Rome` and then query the edited model with the prompt `A great tourist attraction, the Eiffel Tower, which is located in the city of`, does the LM generate `Rome`? This paper works on the last token of the prompt and *hope* that the LMs semantic similarity of the latents would be high (same as GRACE). But this is not guaranteed and needs to be properly evaluated to claim generalizability.\n    * **Performance comparison across different methods:** The paper does not compare the proposed method with other methods, such as ROME and GRACE. It would be nice to see how the proposed method compares with other methods in terms of the quality of the edits measured in *efficacy*, *generalizability* and also *specificity*. Although I found the discussions in Theorem 2, 3 and the corresponding figures in section 5 are convincing that the method will be highly specific to the edits.\n\n* **Scalability:** What is the maximum number of edits that can be made to a LM using this method? I'd assume, without jet-packs, you theoratically cannot edit more than $d$ facts, $d$ being the actual dimension of the latent space. Figure 1 suggests that the intrinsic dimension of the LM is even less. As a result, the LM might break even before $d$ edits. I think you will face a similar situation with the jet-packs as well, since with increasing number of edits will increase the chances that unintended detectors will get activated. What is your take on this?\n\n* **Detector Neuron:**\n    * **Selecting the detector row in $W_1$:** For a single edit, why did you need to select the row with the smallest norm? Is the assumption that the row with the smallest norm has the least impact on the channel activations anyways and changing it will ensure minimal damage? \n    * I also found it difficult to follow the discussion on Appendix B.1.3 as it introduces too many new variables without proper explanations of what they are. For example $\\zeta$ was introduced in Eqn 12 but it is not clear what it exactly is.\n\n* **Difference with ROME**: The proposed method repurposes a specific column of $W_2$ and corresponding row in $W_1$ to rewrite one single fact. In contrast, my intuition on ROME is that it uses a rank-one update to distribute the load of the edit across *all* the columns in the $W_2$. Is this correct assumption? If so, then I'd be curious to see a comparison of the proposed approach vs ROME in terms of the quality and the scalability of the edits.\n\n**EDIT:** Bumped the rating to 6.\n\nLIMITATIONS:\n* Was the efficacy measured on the successful prediction of only the first token? If so, it should be mentioned in the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of, when given a particular prompt, whether it is possible to surgically and efficiently edit a model’s parameters to produce a certain response in a way that does not otherwise change the behavior of the model. The paper provides an efficient technique for this in the form of both an in-place edit of a small amount of a model’s parameters, or alternatively as an additional block, “jet-pack”, inserted to encode the edits using the same framework. The key intuition of the technique comes from the perspective that the FFN block is a type of key-value memory (Geva et al. 2020), and we can rewrite a key that is not important to the overall model (smallest L1 norm, as used in this paper) to one that explicitly triggers on the desired input, and we can rewrite the corresponding value to a value that maximizes the likelihood of the desired output. The authors provide theoretical foundations that this change does not impact the general behavior of the model, as well as empirical evaluation of the technique over two factuality datasets to demonstrate the success rate of their technique, that it does not disturb the general ability of the model (on metrics like perplexity), and that the true false positive rate (FPR) is consistent with the FPR predicted by the theory.\n\nSTRENGTHS:\n1. This is a very well written paper that clearly lays out the problem being studied, introduces the techniques, theoretical foundations, and empirically verifies the theory and edit success.\n\n1. The contribution is significant and novel: to edit the output of a model so precisely by purely adjusting two rows/columns in a MLP transformation is an excellent finding. The framework developed here is important and will be of interest for many in the field. It may spark other research works that can apply or extend this technique to address other open problems in LLMs (e.g. perhaps continual learning, model merging) .\n\n1. Within the specific setting that the work studies, the paper is rigorous in its analysis and evaluation: modern LLMs of different architectures (Llama, GPT-J, Mamba) are evaluated, where to add the edit within the model is thoroughly ablated and plotted, methods and experiments cover a variety of scenarios: in-place and jet-pack editing, editing under certain contexts: corrupted prompt, unexpected context.\n\nWEAKNESSES:\n1. The motivation of the paper focuses on stealth, but this dimension could be better defined. The paper specifically studies \"stealth\" under the definition that 1) the edit is directly to the parameters and is somehow small 2) given a fixed set of prompts, only the edited prompts should show altered behavior. However, stealth could also be taken to mean: whether it is possible to detect in the weights that the model maliciously altered -- which this paper does not study.\n\n1. It is unclear to what degree the jet-pack approach is \"stealth\" as it alters the parameters / layer stack used in the model. \n\n1. Given the motivation of fixing hallucinations, the limitation of the approach is somewhat hidden by the set-up and assumptions. This may be perfectly fine if these limitations were clearly stated in the framing of the paper, however it does not seem to be and the results could be misinterpreted. Specifically:\n\n    a. The mechanism of the edit is designed such that given a particular prompt, the model outputs a particular output. It seems like such an approach may only be effective under very specific prompts, which is not useful in the case of addressing hallucinations, which this paper uses as motivating example. If this is not true, it would be excellent to add additional empirical evidence that these edits can adjust concepts/facts rather than just the output given a specific prompt.\n\n    b. If the edit is indeed specific and results replacing an entire row/column in the MLP, this would be limiting for real use cases.\n\n    c. The edits are only done over the pre-trained models, and only for very short factual prompts. Perplexity ratio is only computed for 50 tokens, and over the datasets considered in this work. It is possible that the model may have compartmentalized Wikipedia / factual knowledge of this type in certain parts of the MLP, allowing for edits to be done in other parts of the MLP corresponding to other abilities. To demonstrate true \"stealth\" edit/attack that would deceive a standard user, would require evaluation that other capabilities are not impacted as well. Here, evaluation of perplexity over a general multi-domain dataset such as The Pile, or reporting performance a wide array of few-shot tasks would be much more convincing empirical evidence. \n\n   d. In Section C.2, it is mentioned that edit success is defined as 1) the first token matches the first token of the expected output, and 2) the target output is contained within the generated text. This is different than Exact Match metrics typically used for close-book QA datasets like Natural Questions, TriviaQA, as it could allow for some degree of degeneration. Providing samples of outputs from the model could alleviate this concern. \n\n\nMinor typos:\n- Line 338\n\nQUESTIONS:\n1. In some ways the techniques here are similar to other parameter-efficient fine-tuning approaches. e.g. with LoRA [1] a low rank adapter is trained, and during inference its weights are added to the base model's weights. This is somewhat similar to the MLP key-value row/column trained here and replaced in the base model's weights. One could imagine training a LoRA adapter to address hallucinations while otherwise maintaining general performance. To what degree is this a work a stealth edit and such a standard approach is not? Is this a reasonable naive baseline to try?\n\n1. What is the scaling like for the number of edits a model is able to take before degenerating?\n\n\n[1] https://arxiv.org/abs/2106.09685\n\nLIMITATIONS:\nSee weaknesses response above. The authors do provide a limitation section that covers a reasonable degree of limitations otherwise.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm and studies a family of methods it refers to as *stealth edits*, which modify a large language model to selectively correct a set of known hallucinations without otherwise affecting the responses. It also proposes *intrinsic dimension*, a pairwise separability-based metric to determine the ease of editing any given network block of an LLM. It provides theoretical guarantees for the selectivity of edits and demonstrates that the probability of edit activation decreases exponentially with increasing intrinsic dimensionality.\n\nIt proposes two versions of the algorithm - one for in-place editing a network block to correct a given prompt and target output, and another for inserting an additional *jet-pack module* in the network to correct a series of hallucinations. Both of these algorithms optimize the rows/columns of the projection layers of the block with gradient descent to produce the expected target output(s). The *jet-pack block* controls the selectivity of an edit using a modified *RMSNorm* normalization layer that is re-centered to the mean normalized features of a set of Wikipedia prompts to maximize their intrinsic dimensionality.                \n\nIt also highlights that these methods expose a potential vulnerability in LLMs and can potentially be used by malicious actors to perform a *stealth attack* to produce targeted responses for specific trigger inputs of their choice. Specifically, it discusses two variants of attacks - a *corrupted prompt attack* in which the attacker inputs slightly different versions of the trigger prompt and an *unexpected context attack* in which the attacker mentions the target response for only a given prompt that is followed by \"context\" sentence. In both of these cases, the attacks are difficult to monitor or track through automated tests.     \n\nIt validates the effectiveness of their proposed algorithms by reporting the performance on MCF and ZsRE datasets using the metrics: edit success rate, perplexity ratio of attacked model and the original model, and actual/theoretical worse-case of detector false positive rates. It demonstrates a lower edit success rate and higher false positive rates in the earlier layers of the model in the case of stealth editing. It also shows a near-perfect detector false positive rate in the case of *jet-pack edits* and observes similar trends of selectivity in both variants of the stealth attack. The proposed algorithm's formulation unifies seemingly different related methods GRACE and Transformer-Patcher. It also notes that normalization functions in LLMs can help enable selective edits while also being a point of potential vulnerability in these models.\n\nSTRENGTHS:\n- This paper introduces a novel formulation and theoretical framework to analyze methods addressing the important problem of selectively correcting hallucinations. The formulation generalizes to a wide variety of architectures and selective editing algorithms.\n- The metric *intrinsic dimensionality* seems particularly useful in understanding both the efficacy of selective edits and identifying potential vulnerabilities in an LLM.   \n- The experiments reveal important flaws in prior work such as the sub-optimal choices of applying edits at the end of the model and find them to be much more effective near the middle layers.\n\nWEAKNESSES:\n- The gap between theoretical expectations and empirical results w.r.t false positive rates in the layers towards the end isn't sufficiently addressed. The looser near-zero worst-case bounds for false-positive in this case also aren't that helpful.\n- Some of the important observable trends in the figures of the experiments section aren't justified. Specifically, the trends for the edit success rate metric in Figures 3, 4, and 5 seem to vary widely across all layers among different model families, while it is relatively more consistent for the other metrics.\n\nQUESTIONS:\n1. Could you provide an intuitive or theoretical justification for why you observe the false positive rate slightly trend up towards the final layers of the model and find the optimal edit insertion location to be about half-way?\n\nLIMITATIONS:\nThe authors have generally addressed the limitations and potential societal impacts of their proposed algorithm. A few suggestions to improve: \n- In section [7], according to the guidelines they should provide further details on which error bars are used and state the corresponding confidence intervals in their experiments.\n- While the negative social impacts of stealth attacks are discussed in different sections of the paper, it should ideally be discussed again in the Discussion section, especially the practicality or limitations of such attacks in practice.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on stealth editing in large language models, presenting methodologies for making targeted, subtle changes to these models without retraining. The techniques, called \"stealth edits,\" aim to correct specific issues like factual inaccuracies by directly updating the model's weights. The research also reveals that all modern language models are susceptible to stealth attacks, which are targeted and hard to detect, involving minimal changes to a model's weights to alter its response to a specific prompt. Experimental results demonstrate high success rates for both edits and attacks while maintaining low false positive rates. The findings have significant implications for AI system security and reliability, indicating that even extensively trained models can be vulnerable to targeted manipulation. The authors conclude by highlighting the broader impact of their work and the need for further research in this area.\n\nSTRENGTHS:\n1. This work analyzes how the intrinsic dimension of data is a crucial factor: the higher the dimension the higher the probability that a given edit is successful. The findings here are novel and intriguing, which could guide future research in the field. It also combines the edit and attack together. \n\n2. The experiments are sufficient. They considered different architecture and different model sizes.\n\nWEAKNESSES:\n1. The paper's organization can be improved to make it easier to understand. Maybe the space is limited and I think some information can be moved from the Appendix to the main paper.\n\nQUESTIONS:\n1. In the text around Line 187-190, it says it would add a new row to W1 and a column to W2, and in Algorithm 2 Step 9， it still uses the W1 as the updated model. I'm a bit confused about whether the W1 here has extra rows. I didn't get this jet-pack editing here.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}