Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
Authors: Vaidehi Patil
Peter Hase
Mohit Bansal
Venue: ICLR 2024 spotlight (2024)
Keywords: Sensitive Information Deletion
Privacy Attacks
Model editing
Language Models
Primary area: societal considerations including fairness, safety, privacy

Abstract: Pretrained language models sometimes possess knowledge that we do not wish them to, including memorized personal information and knowledge that could be used to harm people. They can also output toxic or harmful text. To mitigate these safety and informational issues, we propose an attack-and-defense framework for studying the task of deleting sensitive information directly from model weights. We study direct edits to model weights because (1) this approach should guarantee that particular deleted information is never extracted by future prompt attacks, and (2) it should protect against whitebox attacks, which is necessary for making claims about safety/privacy in a setting where publicly available model weights could be used to elicit sensitive information. Our threat model assumes that an attack succeeds if the answer to a sensitive question is located among a set of B generated candidates, based on scenarios where the information would be insecure if the answer is among B candidates. Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover “deleted” information from an edited model 38% of the time. These attacks leverage two key observations: (1) that traces of deleted information can be found in intermediate model hidden states, and (2) that applying an editing method for one question may not delete information across rephrased versions of the question. Finally, we provide new defense methods that protect against some extraction attacks, but we do not find a single universally effective defense method. Our results suggest that truly deleting sensitive information is a tractable but difficult problem, since even relatively low attack success rates have potentially severe implications for the deployment of language models in a world where individuals enjoy ownership of their personal data, a right to privacy, and safety from harmful model outputs.

Published as a conference paper at ICLR 2024
CAN SENSITIVE INFORMATION BE DELETED FROM LLMs? OBJECTIVES FOR DEFENDING AGAINST EXTRACTION ATTACKS
Vaidehi Patil"
Peter Hase
Mohit Bansal
UNC Chapel Hill
(vaidehi, peter, moansal } (css unc. unc.edu
ABSTRACT
Pretrained language models sometimes possess knowledge that we do not wish them to, including memorized personal information and knowledge that could used to harm people. To mitigate these safety and informational issues, we propose uttack-and-defense framework for studying the task of deleting sensitive information directly from model weights. We study direct edits to model weights because (1) this approach should guarantee that partieular deleted information is never extracted by future prompt attacks, and (2) it should protect against whitebox attacks, which is necessary for making claims about safety/privacy in setting where publicly available model weights could be used to elicit sensitive information. Our threat model assumes that an attack succeeds the answer question is located among set of B' generated candidates, based on scenarios where the information would be insecure if the answer is among candidates. Experimentally, we show that even state-of-the-an model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% the time. These attacks leverage two key observations: (1) that traces of deleted information can be found in intermediate model hidden states, and (2) that applying an editing method for one question may not delete information across rephrased versions of the question. Finally, we provide new defense methods that protect against some extraction attacks, but we do not find a single universally effective defense method. Our results suggest that truly deleting sensitive information is tractable but difficult problem, since even relatively low attack success rates have potentially severe implications for the deployment of language models in a world where individuals enjoy ownership of their personal data, right to privacy, and safety from harmful model outputs.
sensitive
INTRODUCTION
Large language models (LLMs) now possess much factual knowledge about the world. This knowledge can be extracted from models using natural language prompts (Petroni et al., 2019), or models can finetuned to answer user questions within a dialogue (Ouyang et al., 2022). Notably, these models sometimes possess knowledge that we do not wish them to, including memorized personal information (Carlini et al., 2021), knowledge that could be used to harm people (c.g. advice on committing illegal actions) (Weidinger et al., 2021), and factual information that has simply gone out of date (Lazzridou et al., 2021). Facts or beliefs of this kind are known as sensitive information (Brown al., 2022). Since LLMSs can generate this kind of sensitive information, there are clear safety issues and information hazards associated with deploying LLMs to interact with people or make decisions affecting people.
This situation leads us to ask:
"Equal contribution.
Our code is available https://github.com cwm/vai dehi 99/ Infolelet int

---

Published as a conference paper at ICLR 2024
How can we "delete" specific sensitive information from language models when we do want models to know or express this information?
How do we test whether that specific information was successfully deleted?
Scrubbing Sensitive Info From LLM Outputs.
Currently, the predominant approach to eliminating sensitive information from LLM outputs (while preserving informativeness) is to use reinforcement learning from human or Al feedback, known as RLHF or RLAIF (Ouyang et al., 2022; Bai et al., 2022). In general, RLHF has been preferred over removing sensitive information from the training data, which may be very difficult and also requires expensive retraining processes verify its success (Henderson et al., 2023; Zhang et al., 2023a). Yet, RLHF is known to have number of shortcomings, both in theory and in practice (Casper et al., 2023). Most pertinently, models remain vulnerable to adversarial prompts even after RLHF (Zou et al., 2023). A possibly deeper shortcoming of RLHF is that model may still know the sensitive information. While there is much debate about what models truly "know" (Jiang et al., 2020; Andreas, 2022), it seems problematic for model to, e.g.. be able to describe how to make a bioweapon but merely refrain from answering questions about how to do this. Additionally, it is possible for legal regulations to require that model developers remove sensitive information about an individual from a model upon the individual's request (Mohan et al., 2019; Henderson et al., 2023; Zhang et al., 2023a). Though the notion of "deleting" data is underdefined for language models (as opposed databases), we doubt that RLHF would enable compliance with such privacy requirements.
1. Notice sensitive info
Language Model
2. Deletion defense
Language Model
' don
know"
3. Extraction attack
Language Model
A
In our attack-and-defenss frame- work for deleting sensitive information from an LLM, a malicious actor (or a regulator, or user) attempts to extract "deleted" in- formation. We introduce new methods for defending against extraction attacks.
Figure 1:
Model Editing for Information Deletion. We argue that the ideal approach is to directly delete sensitive information from model weights. This approach should tailor models to never make use of the sensitive information, meet potential legal standards for privacy (Zhang et al., 2023a), and avoid difficult data-side interventions (Debenedetti et al., 2023). A further benefit of deleting sensitive information from weights is that this protects against whitebox extraction attacks. Anyone with sufficient technical knowledge might be able to extract sensitive information from model weights (or hidden states) using representation probing techniques, which is a problem as model weights continue to proliferate publicly through open-source release (Touvron et al., 2023).
When Is Information Truly Deleted? In this paper, we first adapt model editing methods (Meng al., 2022; 2023) for deleting sensitive information. Then we show that, surprisingly, even state-of-the-art editing methods struggle to truly delete factual information from models under simple whitebox and blackbox model attacks. We elaborate a threat model in Sec. 3, where our key assumption is that an attack succeeds on a given input if a "deleted" model output can be recovered from set of B extracted candidates for small number B. This view is based on three plausible scenarios: an attacker could (1) make B "password attempts" to verify the answer, (2) pursue B malicious ends in parallel. or (3) make legal demand for the answer to be unobtainable within B candidates (when the attacker is actually the data owner or regulator). By example, consider that an individual's 's phone number might be leaked among, say, 10 extracted candidates; this is hardly guarantee of privacy. The wttck--and-eefnns perspective in this paper is represented in Fig. 1.
Whitebox Attacks. The whitebox attacks we consider leverage the insight from interpretability research that output information accrues over time in the hidden states of Transformer forward pass (nostalgebraist, 2020; Geva et al., 2021). In experiments with GPT-J (Wang & Komatsuzaki, 2021) and Llama-2 (Touvron et al., 2023), we show that by projecting intermediate hidden states onto the model vocabulary embeddings, we are able to extract model knowledge from these hidden states when the model has been edited to assign zero probability to the knowledge. We are able to
2

---

Published as a conference paper at ICLR 2024
extract the "deleted" answer from the hidden states a full 38% of the time when using budget of B = 20. In order to mitigate against these attacks, we extend the model editing objective to delete information from both the final output and the intermediate model representations. This defense lowers the attack success rate from 38% to 2.4%. We also show that our defense methods fare well on second whitebox attack that they were not designed to defend against.
Blackbox Attacks. Our blackbox attack is a simple but effective automated input rephrasing attack. While model editing methods can remove target information across almost all paraphrases of prompt, we exploit their non-zero error rate by sampling model outputs for different paraphrases that are automatically generated from a paraphrasing model. This blackbox attack succeeds 29% of the time with a budget of B = 20. We provide a new objective using data augmentation to protect against the blackbox attack, but, surprisingly, we find that this defense does not help against our paraphrasing attack (unless one aggressively edits the model, leading to undesirable damage to model knowledge).
Findings. We summarize our contributions and conclusions as follows:
We introduce a threat model for sensitive information deletion based on the idea that information is incompletely deleted if it can be extracted from model within set of B candidates.
2. We show that model editing methods like ROME fail fully delete factual information from LLMs, as facts can still be extracted 38% of the time by whitebox attacks and 29% of the time by blackbox attacks with low attack budgets (B = 20).
We introduce new objectives for better defending against whitebox and blackbox extraction attacks. Our approach reduces whitebox attack success from 38%->2.4% without further damaging model knowledge, but, surprisingly. a data-augmennation--oe
blackbox defense is not effective. 4. Finally, we show that our whitebox defenses can help defend against "unforeseen" extraction attacks, i.c. attacks that they were not specially designed for.
RELATED WORK
Evidence That LLMs Memorize Sensitive Information. Early work shows that models like GPT-2 memorize personal information and exact code snippets present in their training data (Carlini et al., 2021; Ziegler, 2021). These works aim "indiscriminately extract training data" (starting with hun- dreds of thousands of model generations as candidates) rather than "extract targeted pieces of training data," which is our goal as we start with specific question we aim extract the model's answer to. More recently, Carlini et al. (2023) show that GPT-J memorizes at least 1 1% of its entire training dataset. We point to Brown et al. (2022) for broader discussion of memorization in L.LMs and user privacy.
Attacking LLMs for Sensitive Information. Our attacks are related to existing work on privacy attacks. Membership inference attacks aim to verify whether particular samples are in model's training data (Dwork et al., 2006; Shokri et al., 2017). In this paper, we aim to extract specific factual information from a language model, rather than verify whether some given information was in the training data. More relevant to this aim are the methods used to extract information from language models, including prompting (Petroni et al., 2019) and probing (Belinkov, 2022). Some works (Henderson al., 2018; Lukas et al., 2023) explore prompting as blackhox extraction attack, but, in contrast, (1) we do not assume the attacker has the exact text from the pretraining data that prefaced the sensitive information, and (2) our threat model does not restrict the candidate set to be a single element (B = 1). More broadly, our overall aim is to develop both whitebox attacks using representation probing techniques (nostalgebraist, 2020; Geva et al., 2021) and blackbox attacks using model- based input rephrasing (Krishna et al., 2023). To our knowledge, these methods have not been applied as extraction attacks on LLMs that have been specifically tailored to remove sensitive information (e.g. with model editing methods). Moreover, we extend such information deletion methods order to better defend against these kinds of attacks.
Machine Unlearning and Model Editing. So-called machine unlearning is an old problem where the goal is to remove information from a model without damaging the model's performance on the task it was trained for (Cao & Yang, 2015). Initial unlearning approaches for deep learning updates to model weights, using e.g. influence functions (Guo et al., 2019)
relied on gradient-base or continual learning methods (Tanno et al., 2022). However, unlearning methods are generally focused on removing the influence of a training (æ, y) pair on a supervised model. This may not be the appropriate framework for deleting sensitive information from language models, since the
3

---

Published as a conference paper at ICLR 2024
information is undesirable ouppi given in response to prompts questions that are harmless on their own. In contrast, model editing is an approach focused on changing particular outputs for certain model imputs, with methods designed to update factually incorrect knowledge in models (Zhu et al., 2020; Dai et al., 2022; De Cao et al., 2021; Hase et al., 2021). Model editing has already been used widely within computer vision for deleting specific concepts from image generation models (Gandikota et al., 2023; Heng & Soh, 2023; Kumari et al., 2023; Zhang et al., 2023b). Past work with language models conducts simple experiments on "fact erasure" (Hase et al., 2023), but its main focus is on the relationship between interpretability (localization) and model editing, while we explore the problem of extracting or deleting information from language model. Lastly, recent work introduces methods for removing particular features (like word part-of-speech) from model (Belrose et al., 2023b) or even a model's ability to perform a particular task (Ilharco et al., 2023). Here, we remove more specific information (individual facts) from language models.
3 PROBLEM STATEMENT
We frame the information deletion problem in terms of adversarial attack and defense (Carlini et al., 2019). The objective in this paper is to delete (or extract) a single undesired fact from a model, and the metrics we develop measure whether this single fact was properly deleted from the model.
3.1 THREAT MODEL
Adversary's Objective: We assume that an adversary seeks to obtain the answer to a question Q. where this pair (Q, A) is sensitive information. We say that an extraction attack is successful if the answer A is within a candidate set C' that is obtained by the attacker running some inference algorithm on the model. This definition follows from three plausible threat models described below. We refer to the size of the candidate set, |C| = B, as the attack budget.
1. Password Attempts: For the first threat model, we suppose that the attacker (1) does not know the sensitive information and (2) could verify they had the correct information within B attempts, like password attempts for stealing a personal account.
2. Parallel Pursuit: In the second threat model, we suppose that an attacker can act based on multiple candidates in parallel without necessarily needing the correct information. One example of this could be harassing an individual via multiple possible personal email addresses.
3. Verification by Data Owner: Lastly, we consider an attacker who is actually the data owner regulator; they (1) know the sensitive information and (2) do not want it to be public. Imagine, for example, requesting that your work address be deleted from an LLM. If there were method that reliably produced your real work address in a set of B possible addresses, you might not be satisfied with concluding that your private information had been properly "deleted" from the model.
Thus in each setting, the LLM would be insecure if the answer A is among the set of B candidates.
Attack Success Metric. Following our threat models, we define an attack success metric below. We compute this metric using data (z,, y J.W. with label = A representing an answer and input x, = Q representing question: AttackSuccess @ (MM) set produced for model M on datapoint x; (with Cal B). and I [-] is the indicator function.
Ee [l E Cj), where C₁ is the candidate
A Adversary's Capabilities: We delineate two possible levels of adversary model access, aiming to simulate real-world constraints an attacker may face (Carlini et al., 2019): whitebox and blackbox ac- cess. In whitebox access, we assume that the adversary has the models weights and architecture, such that they can run model forward passes and access intermediate hidden states. For blackbox access, we assume that the adversary can provide inputs to the model and receive randomly sampled outputs.
3.2 METRICS FOR INFORMATION DELETION
The goal of an information deletion method is to remove specific information from a model. But trivial (and bad) solution to this problem is to remove all information from model. Thus the objective is to (1) remove specific information, while (2) avoiding damaging the model's knowledge in general: arg minM- AttackSuccess@@ ((M*) + ADamage .M) where M is the edited model, M is the pre-edit model, and Damage(;,) denotes some measurement of damage the model's knowledge
4

---

Published as a conference paper at ICLR 2024
The 0
Logit Lens Attacks
Rephrasing Attack
Autonomous 41-
University 43-0
C- Ulep-AIP"))
Head Projection
Hidzen
Vocab
F;
Spain "deleted" from the model
Of 43-0
2. Probability Delta
Darosion
Original Input
Madri 13-0
G- U JlupMetin
pi boltooas//piiin (!
xt Uhe Aato rous Uswaray Mausi
Is 42-
In 42-
Nost Token
22
Waasd
Attacking (Succssaful)
Distiibtion
Za localed in country of
Precs
17 Bowntown Freet
Baiddo
Downsown Ienttartnnd
Muond
Woand
(r)
"Deleted" information obtained from rephrased
Casalcela
Spain
Measa
"Deleted" information obtained
Spainn
Haphreaing
Module
Itandoo Stmpiingg From LM
from intermediate hidden states from the model
Our two kinds of extraction attacks for recovering information that is "deleted" from an LLM by a model editing method. Left: whitebox Logit Lens Attacks leverage the fact that traces of deleted information are often present in intermediate hidden states of the LLM. Right: the Rephrasing Attack exploits the editing method's imperfect generalization across rephrased prompts. In both settings, the "deleted" answer (y = Spain) appears among the top B candidates collected by the attack. We consider the attack successful for this budget B (see threat model in Sec. 3).
Figure 2:
(compared to the unedited model). We do not optimize (or report) this objective directly, since it re- quires domain-specific tradeoff (A) Attack-Success and model damage. Instead, we report Attack- Success metrics alongside common metrics for damage to model knowledge after model editing:
1. Random A-Ace (Zhu et al., 2020; De Cao et al., 2021): We measure the change in model accuracy for random datapoints selected from the broader dataset, before and after editing the model for point x. In our experiments, when an L.M is prompted with input x', its generated output considered correct if includes the true answer y/ from the fact (x', ,/).
Neighborhood A-Acc (Meng et al., 2022): This measures whether edits change outputs for prompts x involving the same relations and the same (true) answers as the fact being deleted. It is important to evaluate model performance on neighboring points to the main fact (x, y) because it is difficult to avoid changing model outputs for points similar to the main fact (Hase et al., 2021). As above, we calculate the change in generation accuracy before and after the model edit for point x.
Additionally, we report the Rewrite Score from Hase et al. (2023) as traditional measure of edit suc- cess. A value of 100 means that edits perfectly maximize (or minimize) the target probability, while a value of ( means that the new target probability does not change at all. See Appendix D for full details.
4 ATTACK METHODS
4.1 WHITEBOX LOGIT LENS ATTACKS
The logit lens (nostalgebraist, 2020; Geva et al., 2021) is an interpretability technique inspired by the concept of iterative refinement of features in Transformers. The technique directly converts hidden states from any intermediate layer into a distribution over the model vocabulary by multiplying the hidden states with the output token embedding matrix of the model. When applied to successive layer outputs within a model forward pass, the logits lens produces a progression of probability distributions over the vocabulary. This trajectory gradually converges towards the ultimate output distribution, with each subsequent layer achieving lower perplexity against ground truth text (Beirose et al., 2023a).
We leverage the logit lens to design two attacks that probe the intermediate layer representations of an LLM. These attacks are based on the hypothesis that, while editing methods may remove sensitive information from the final model output (i.e. generated text), this information may still be present in intermediate layers. Indeed, we often observe a "deleted" answer appearing among probablee tokens during intermediate layers before disappearing completely at the final layer (as shown in Fig. 2). Based on this observation, we propose two approaches for obtaining a candidate set C' from the probability distributions over the vocabulary produced by the logit lens.
Head Projection Attack: Using the logit lens distribution at each layer in a set of layers L, we construct candidate set C consisting of the top-k highest probability tokens from each layer: Clead Projection Utez (D(()) where D()) E RIV is the logit lens probability distribution
5

---

Published as a conference paper at ICLR 2024
over vocabulary V from layer l and top-k(-) returns the highest-probabille & elements from each distribution. Note we limit our experiments to datapoints with single-token answers for simplicity. We select the top-k tokens for the basic reason that the deleted answer may appear among the top tokens in the logit lens distributions before disappears from the distributions at later layers (as in Fig. 2). The budget of this attack is B T.]. For experiments in Sec. 7, we select k and I. to optimize attack performance while remaining under a maximum budget B (see tuning details in Appendix A.)
Probability Delta Attack: Our second whitebox attack leverages the observation that a "deleted" answer may quickly rise and fall within the progression of logit lens vocab distributions. We conjecture that by rank-ordering the differences in token probabilities between two consecutive layers, the target answer may be identifiable in the top or bottom & tokens. Consider Fig. 2 again: the deleted answer Spain must first rise and later fall significantly across layers as it enters and exits the head of the logit lens distribution. We therefore construct candidate set as: Probaaility Delta
D where D(4) is the logit lens probability distribution from layer f. This approach constructs a set from the elements that rise and fall the most in the logit lens distributions between layers. For experiments in Sec. 7, we optimize attack performance by tuning I. and selecting the top-k elements, bottom-k elements, or union of the two sets (while remaining within fixed budget of |C| 20). Further tuning details are present in Appendix A.
USer top-k(D(*+1) - D( U bottom-k(D
4.2 BLACKBOX ATTACK
Input Rephrasing: LLMs are known to be vulnerable to adversarial prompts even after finetuning for chat safety (Zou al., 2023). For our blackbox attack, we employ a simple but effective technique: prompting with model-generated rephrases of the original input that was used for model editing. Since model editing techniques exhibit good but imperfect generalization across paraphrases (De Cao et al., 2021; Meng et al., 2022), we can extract specific information from a model by rephrasing the input and sampling model outputs across these rephrases (shown in Fig. 2). So, we obtain a candidate set as CRephase = - (Us ~ P(ylor;: MM)) M where R is the number of rephrases, S is the number of model samples per rephrased input, 2, is the r-th rephrasing of generated by a paraphrasing model, and P [z; MM)) is the output distribution of the edited model given input x. We generate H, using the paraphrasing model from Krishna al. (2023). The budget of this attack is |C| RS.
DEFENSE METHODS
The Empty Response Defense (Ouyang et al., 2022). This defense employs the basic strat-
to 0-0
Whitebox Defense
egy of optimizing a model to output something not containing the sensitive information, which is the strategy behind using RLHF for prevent- ing models from generating sensitive informa- tion. We simply optimize the probability of an "emptt" target string d with the objective arg max M p(d[x;M) using one of two target strings: "I don't t know" and "dummy". The re- sult that, instead of generating the original knowledge, the model will instead indicate that does not know the answer ("I don't it know" tar- get) or give some meaningless response ("dummy" target). In our main experiments, we use the "dummy" target, which performs better than using "I don't 't know" (see Appendix B).
tokln 2-1
Hilden Bate Voca Poperton
tokin 10r-
Taken
Distribution D
GD
t
U
D
Defense
We defend against whitebox attacks by deleting information from intermediate hidden states as well as the final model output distribution (Max-Entropy and Head Projection Defenses).
Figure 3:
Fact Erasure (Hase et al., 2023). Another simple approach to deleting sensitive answer is to minimize its probability under the model, i.c. minimize p[y|æ;; M) for the original fact (x,z).
Error Injection (De Cao et al., 2021). A common test of model editing methods involves inducing counterfactual knowledge in the model. Here, we use the objective arg max p(y M)) where y is the alternative, false target provided by Meng et al. (2022). This method would not be applicable in practice, since we do not actually want LLMs to give wrong answers to sensitive questions, but consider it here to show the efficacy of injecting new false information into the model.

---

Published as a conference paper at ICLR 2024
Head Projection Defense. We introduce objective that is directly designed protect against the Head Projection attack. The goal is to prevent the deleted answer from appearing in the top-k elements of the logit lens distributions across set of layers L, as well as the predicted distribution at the final layer (see Fig. 3). To do so, we introduce a max- margin loss in each relevant distribution. With D as the logit lens distribution at layer E, D answer as the original answer's logit lens probability, and D as the k-th top probability in D(O), the objective becomes: [.. EteL max(0, D x +m), where m is the margin term. Since we do not face any constraint over the set of layers L to remove the answer from, we tune over possible layer sets to improve the defense performance (details in Appendix A).
Max-Entropy Defense. This defense is similar the Head Projection Defense, but it varies in terms of the objective for each layer. Here, we maximize the entropy of the model's logit lens distributions over the next token at each layer: arg max M - [2] EXEL Eyein D. log D/ where DJy) is the probability of token y the logit lens distribution of model M given the input prompt x.
Input Rephrasing Defense. This defense strategy aims to counter the Input Rephrasing blackbox attack described in Sec. 4. In addition to using the input x for optimization, this approach adds model-generated paraphrases of z to the model editing objective. The rephrased inputs are created using the same off-the-shelf generation model as in the Input Rephrasing attack (Krishna et al., 2023). In other words, for the i-th datapoint, we concurrently delete the information for all prompts x E 24 U X", where X²" represents the set of rephrases of x; used for defense. We specifically optimize the Fact Erasure objective in parallel for each input x E x; U UXP.
EXPERIMENT SETUP
Models. We conduct experiments with GPT-J (Wang & Komatsuzaki, 2021), Llama-2 (Touvron et al., 2023), and GPT2-XL (Radford et al., 2019). These models were chosen due to their (1) widespread usage, (2) public availability, and (3) capacity for memorizing their pretraining data (Carlini et al., 2023). Results for Llama-2 and GPT2-XL are in Appendix B.
Datasets. We use two datasets, CounterFact (Meng et al., 2022) and zsRE (Levy et al., 2017). CounterFact consists of prompts with factual completions, as well as neighboring datapoints that we use for computing Neighborhood A The zsRE dataset contains short question-anse answer pairs derived from Wikipedia. Both datasets include alternative, false targets for each input for model editing. To obtain data for computing Random A-Acc, after each individual model edit we randomly sample 100 other data points from the respective dataset. We filter the data to facts that are known by the model we attack, because it only makes sense to delete facts that are already known by the model. We consider fact known by the model when the answer string is in the model generation given the prompt: GPT-J gets 34% accuracy on CounterFact data with single-token answers and 25% on zsRE (we also filter points with single-token answers). Drawing from these eligible facts, our final sample sizes are 587 and 454 datapoints for CounterFact and zsRE respectively when using GPT-J.
Model Editing Methods. We employ two popular model editing techniques in our experiments, ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023). We refer the reader to these works for full details of the methods. Both methods work by updating a specific weight matrix in the MLP layer(s) of Transformer model. When applied to change a single fact in the model, the difference between them is that ROME updates a single layer's s MLP (layer 6 for GPT-J by default), while MEMIT updates multiple layers' MLPs (we use layers 5-7). See Appendix A for other method hyperparameters. In Appendix B. we conduct experiments with an additional editing method, constrained finetuning (Zhu et al., 2020), but this method does not perform as well as ROME and MEMIT. Each of our defense methods in Sec. is characterized by its objective function and can be combined with different model editing (optimization) approaches.
7 EXPERIMENT RESULTS
7.1 CAN WE EXTRACT A "DELETED" ANSWER FROM A LANGUAGE MODEL?
We first ask whether we can attack model edited with the conventional Empty Response method.
Design. We measure Attack-Success @ B as a function of the budget B for our three attack methods: the (1) Head Projection, (2) Probability Delta, and (3) Input Rephrasing attacks. We perform these
7

---

Published as a conference paper at ICLR 2024
experiments with GPT-J and CounterFact data, applying the ROME editing method using the Empty Response objective. We confirm that the edit methods work as designed: the Rewrite Score is high for all methods (90%+)) with low Random A-Acc scores k)). In general. we increase the budget for whitebox attacks by increasing [L] and k. and for our blackbox attack we increase the number of attack paraphrases and the number of samples. See Appendix A for exact hyperparameters.
Results. From the results in Fig. 4, we see that attack success reaches values as high as 38% with a budget of B = 20. This extremely high attack
Input Rophrasing
Head Projection
Probabiity Delta
0.5
success rate means that, under our threat model in Sec. 3, the edited model is highly vulnerable to extraction attacks for the "deleted" fact.
ATTACK
0.3
0.2
Besides the highest attack success rate obtained with whitebox attacks and budget of B 20, we note that the blackbox attack also achieves a high success rate of up to 29%. Additionally, even with budget of B = 1, an attacker would succeed 18% of the time using the Probability Delta attack. Interestingly. all methods appear to saturate in performance after B = 20 candidates.
0.0
0
10
20
30
ATTACK BUDGET (B)
Figure 4: Attack Success the budget B for our three attack methods. We "delete" facts from GPT-J with ROME using the conventional Empty Response objective.
7.2
How TO DEFEND AGAINST INFORMATION EXTRACTION ATTACKS
Next, we show how the proposed defense methods (Sec. 5) fare against our extraction attacks (Sec. 4).
Design. We evaluate the three baseline methods (Fact Erasure, Empty Resp, and Error Inj) and our three proposed methods (HP Def, Max-Ent Def, and IR Def). We report Attack-Success @ B with B = 20 using each of our three attack methods, as well as Random A-Acc and Neighborhood A-Acc metrics to show how much damage is being done to the model's overall knowledge by the deletion of individual facts. We show results for GPT-J on CounterFact and zsRE with ROME and MEMIT
Results. We show the results in Table 1. We summarize the main conclusions follows:
1. Overall, our whitebox and blackbox attacks are all frequently successful at extracting "deleted" facts. We emphasize that MEMIT with the Empty Response defense is successfully attacked by our Head Projection attack % of the time zsRE with B = 20. MEMIT is popular editing method, and the Empty Response objective is the standard approach to preventing models from generating sensitive information, yet this setting totally fails to delete facts from GPT-J.
Our Head Projection and Max- Maxt-Entropy defenses are the strongest defenses against whitebox attacks. Relative the strongest baseline, Fact Erasure, the Max-Entropy defense lowers the Head Projection attack success by 20.5 points on CounterFact points on zsRE (41.4%-42.9% with ROME). The Head Projection defense, though helpful, is surprisingly not as effective as the Max-Entropy defense, except for when used with MEMIT on CounterFact. Note we discuss results for the Probability Delta attack below in Sec. 7.3.
12.2%%++.77% with ROME) and 39.5
3. The Input Rephrasing defense does not reduce the blackbox attack success. Across datasets and editing methods, the IR Def never outperforms baselines and is sometimes the worst objective against the Input Rephrasing attack. We confirm that the defense does work when the attack uses the exact same paraphrased inputs supplied to the defense's 's editing objective; the defense fails when attacking paraphrases differ at all. Additionally, we can lower the Input Rephrasing attack success by making the model edits more aggressive, but this has the consequence of skyrocketing A-Acc numbers (4.7 for Random data and 27.8 for Neighborhood data; see Appendix B).
7.3 CAN WE DEFEND AGAINST UNFORESEEN EXTRACTION ATTACKS?
Lastly, we examine the efficacy of extraction attacks that the defense methods are not directly designed to prevent, important "unforeseen" scenario for our defense methods (Carlini et al., 2019).
Design. We highlight results from our previous experiment, specifically the performance of the Prob- ability Delta Attack applied against our two whitebox defenses, Max-Entropy and Head Projection.
8

---

Published as a conference paper at ICLR 2024
Attack-Successe Success
A Acc
Head Projection
Probability Delta
Input Rephrasing
Rewrite Score
Defense
Random
Neighbors
CounterFact
ROME
Fact Erasure
22.15
25.38
22.83
0.72
8.74
99.69
Empty Resp
36.84
37.65
29.02
0.54
99.58
+ Error Inj
99.20
99.83
20.10
1.00
9.60
99.30
HP Def
4,43
20.27
27.77
0.69
6.35
99.73
Max-Ent Def
1.70
2.39
27.94
0.69
6.27
99.73
Def
56.39
60.65
29.30
0.69
98.88
MEMIT
Fact Erasure
39.18
46.17
34.07
0.26
3.29
98.68
Empty Resp
67.09
72.60
49.31
0.22
87.54
+ Error Inj
98.60
98.80
36.38
0.15
2.05
97.68
HP Def
19.42
38.33
42.76
0.20
3.37
97.09
Max-Emi Mux-EntIie Def
34.24
39.01
50.77
0.19
3.32
96.41
IR Def
56.03
61.50
41.91
0.20
3.49
91.56
zsRE
ROME
Fact Erasure
41.41
43.83
15.64
0.10
94.80
Empty Resp
36.83
59.13
13.00
0.08
99.78
Error Inj
20.68
45.07
10.35
0.13
99.40
Def
31.28
56.83
18.50
0.12
90.53
+ Max-Ent Def
2.86
2.42
18.50
0.12
90.71
IR Def'
84.80
84.80
29.07
0.07
80.64
MEMIT
+ Fact Erasure
42.51
42.07
22.18
0.05
91.34
Empty Resp
88.55
88.11
29.30
0.05
91.34
Error Inj
89.65
80.64
32.60
0.11
85.86
HP Def
53.52
72.47
28.19
0.05
89.46
+ Max-Ent Def
39.92
38.11
29.74
0.07
91.24
II Def
46.04
57.93
27.75
0.07
84.16
Table 1: Attack success rates of the the three proposed attacks (Sec. 4) across defense methods (Sec.
5), for facts from CounterFact and zsRE that are known by GPT-J.
In this setting, we use defenses that were designed to protect against the Head Projection attack but not designed to defend against second whitebox attack, the Probability Delta attack.
Results. We draw a few conclusions from Table 1: (1) The "unforescen" Probability Delta attack very effective against the Head Projection defense, which not prepared for it. (2) Our Max- Entropy defense often helps against the Probability Delta attack despite not being specially designed for it. Compared to the Head Projection defense on zsRE, Max fax-Entropy defense substantially lowers attack success rates 136.8%-2.4444
with ROME and 72.5% 12.5%-+38 1% with MEMIT), However, while the Max Max-Entropy defense can lower whitebox attack success to 2.4%, (3) blackbox attack success remains quite high at 28% for CounterFact and 19% for zsRE, suggesting that the defense is still inadequate against blackbox attacks. In total, we see that there is no single defense method that is prepared against all attacks it could face, even if it is effective against some unforeseen attacks.
CONCLUSION
We first argue that model editing methods are the most promising approach to deleting sensitive information from LLMs, rather than interventions focusing pretraining and finetuning data. Even for this promising class of methods, however, we show that "deleted" information can be extracted surprisingly high percentage of the time (as high as 89% in some experiments) when the attacker operates with a small budget verification attempts B. We motivate this budget via a threat model based on three plausible adversarial settings. Our findings suggest that truly deleting sensitive inorma- tion is a tractable but difficult problem, with potentially severe implications for deployment of LLMss in world where individuals enjoy robust right to privacy and safety from harmful model outputs.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper addresses problems involving sensitive information in large language models. This an important topic with serious ethical implications, as language models currently possess knowledge that could be dangerous to humans and output directly harmful text. We hope that the technical methods in this paper can help mitigate these important ethical problems, but at the same time, we want to demonstrate that it may be fundamentally difficult to solve the problem of sensitive information in pretrained language models. These results could imply that there are negative moral and legal consequences to deploying LLMs in situations where they may influence humans. We leave it for future work AI, ethics, and law to fully explore the implications work on sensitive information deletion and LLMs.
ACKNOWLEDGEMENTS
We thank Neel Nanda for helpful experiment suggestions. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute BRL-21122335 DARPA MCS Grant N66001-19-2-400 and Google PhD fellowship. The views contained in this article are those of the authors and not of the funding agency.
REPRODUCIBILITY STATEMENT
We have provided the code for all experimental results in the supplementary materials to facilitate reproducibilityy Additionally, comprehensive hyperparameters and other essential details required for replication can be found in Appendices A and D.
REFERENCES
Jacob Andreas. Language models as agent models. arXiv preprint arXiv:2212.01666 2022. URL https: //tp:v//aivvvv./ wwweriii.rr..eeeeeeeeeeeeeoooeee wwtppdt/g//////////////pp//.//p/ p2f/2212 11681..pp.......................
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. URL https: //rpiii.////iive//////e// http://wwwwa...........oooo /df// 2212. 00.00007330000
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022. URL
2 122..02222 12452. 12452.pdf.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08/12 2023a. URLhttpe wwttthtss///ai//// pdf.
//tpx/aavvvv/////////aa v. wwt.ooog//ooo..oo /2303. 081
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. arXiv preprint vrXiv22306.03111 2023b. URL httph::tp////. / http://axxvvv/.. wwwp:iivoorr//////////o/
03819 p.d.f.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does mean for language model to preserve privacy?, 2022. URL httts:t/://aaa httpppss/////xx.oa////ooo pdf /2202 05520.00.. pdf.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE symposium on security and privacy, pp. 463 480. IEEE, 2015. URL https: /www. ieee see--secuurtt www.rritt..yrgtrgm STTCC//P211151 //a/5//pppprsee archi ved / 694 9a 63. pdd.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 2019. URL https: / //txxva.vvvv///////////e////avv wtt/pa///902.d..../ 06705. pdf.
10

---

Published as a conference paper at ICLR 2024
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Weese Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. URL https:: / arxiv.. wttgg/ppf/2000 /2012 07w.5.000050000 pdf.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee. Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference Learning Representations, 2023. URL https: htttps://pppeeeeeeeeeeeeeeeeeeee /toppppeeiiieeeee net. forum?
ddd-TatRH_Ick,
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 2023. URL https: : /twxiv..ivvvv.////////..///aaava org/ http/ddf/23000/ 11117.5521...... pdf.
Damai Dai, Li Dong, Yaru Hao. Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. In ACL, 2022. URL https:
08696 pdf.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, pp. 6491-6506. Association for Computational Linguistics, November 2021. URL https: htt////aaaaattttcocceeee www//o///00 emnlp monlp-main... man-mmai..5.. 522.
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher Choquette- Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramèr. Privacy side channels in machine learning systems, 2023. URL https:
http::://aaxiiie /ttp//aaiviv/.////////eee/eaaae/ /ttppd//////ooo////////ee/////e/ 2309. 05610. .df.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284, Springer, 2006.
Rohit Gandikota, Joanna Materzynska, Jaden Frotto-Kaufman and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07335,, 2023. URL https: http://aaxxvvooooee pdf/ 22E//3333.. 07345. pdf.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories. In EMNLP, 2021. URL h
: arriiv org/ /ptp/dddf222..////////// 14913. pdf.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03000) 2019, URL httpss:/arxxi org/ brg/pdf// 1911. 03030. pdf.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654 2021. URL https: //axxiv. org/ org/pdff/ 22111 13654 pdf.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality- localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.0421 2023, URL /ttp:p::///axxv..ooo /ptpgg/pp//3311.... 022123111.11.. 04213. pdf.
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 123-129, 2018. URL https: // //aaxii.. org/ pdfpp/ff1111// wwtp://gggppppp/p/op
09050...... pdf.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. Foundation models and fair use, 2023. URL https:///aaaaa //appps:///axi....// wtggppff/333..3 httpp::://arii.oorr///////2.... 15715. pdf.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach forgetting in deep generative models. arXiv preprint 105.::2305..0220,, 2023. URL https: //trxiivvvo/////// org/ pdf p2305 10120. .ddf.

---

Published as a conference paper at ICLR 2024
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https:
www.xii.org/a.o.eeooooooo pdf/ 2212.222222222 04089. pdf.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
URL https:://////c //axxvivvvv..////a///a///aaaavva www.org/....... htgg/pff//// http/df/2222/ 20iEE/221.... 14315, pdf.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphras- ing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint 13408. pdf.
arXiv:2303. 13408, 2023. URL wtt.thtts////aiiiiee //wpiv/aaivvvv/aaa//aa/eaaaaiwva wtt.v//orgggp/ooeeo//oooo
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu, Ablating concepts text-to-image diffusiin models. arXiv preprint arXiv:2303.13516, 2023. URL
/21.//233311111133 1.f.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap:
Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 14:29348-293613 2021. URL htwps: WWRLhhtps://poooe http:////oooceeeeeeeeeeeee/e / neurips cc/
paper/2021/ 1t..0/////l/e//// http://ileefffffffoee (et 11
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLJ. 2017), pp. 333-342, 2017. URL https://ach /aal://///aha///o/oo
org/K17-1034
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 346-363, 2023. URL https: //arxiv. org/pdf/ 2302. 00539. 015539.d. .pdf.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262, 2022. URL https://arxivy org/pdf/ 2202. 05262. pdf.
Kevin Meng, Arna Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https: https: /ppnreviei.. n ftw.mpt.dwwwccla
Jayashree Mohan, Melissa Wasserman, and Vijay Chidambaram. Analyzing gdpr compliance through the lens of privacy policy. In Heterogeneous Data Management, Polystores, and Analytics for Healthcare: VLDB 2019 Workshops, Poly and DMAH, Los Angeles, CA, USA, August 30, 2019, Revised Selected Papers 5, pp. 82-95. Springer, 2019. URL https: httpa://aaxiv..oo http:/awwvv...g..../../o 1906. 12038. pdf.
nostalgebraist, interpreting gpt: the logit lens, 2020. URL wwwphhttp//// /tt::://wwwwww/ httttp::/www..esssrrrrtt lesswrong. com/ posts//ccRR&wppdaNsvve ittpr///nterrreeengggegeee the---- lens.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022. URL https: htttps://proceedingg wewpeeeeeedenneeeeeeeeeeeeeeeeee neurips cc wwtpappreeppeeeeeeeeeeeeee httppperrrfilseepppeeeeee
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DIP 1250. URL www.tttts///.. peattss//aaee /acl//acccccaaa anthol ogg. 9- 1250. 1-200.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners OpenAl blog. 1(8)99 2019. www.doooooooooooooo. net Laggeeee mo it wtte teakultttssskksrrrssssss. pdd.
URL latpdeels_r__uuree saraaag___mddll___ee__sssspppeee
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Ryutaro Tanno, Melanie Pradier, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind. Advances in Neural Information Processing Systems, 35:13132- 13145, 2022. URL https:tt////. //wxxiivvvoo.. otgg/df//////oo//oo 2207 04806. pdf.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0920 09288, 2023. URL https: /tapps://axxxvv /arxiv.. org/pdf/ 2307. 09288 pdf.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. attps://4e http//gtthuubbbe com/ wwweccom//////e kingoflolzz wweeeterassseeeeeeeee transformes jax, May 2021.
Laura Weidinger. John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.043353 2021. URL https: : /arxivi.v www/por.//....oooooo.ooo...p.... 2112. 04359. pddf.
Dawen Zhang, Pamela Finckenberg-Bree -romann, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions, 2023a. URL https: http///a...gg////ee.///////ooett
2307 03941. .ddf.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not. Learning to forget in text-to-image diffusion models. arXiv preprint urXiv:2303.17591, 2023b. URL https::/arriin wtwpiiiooor/pppdo 2303. 11..3..1.9.......
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli,, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020. URL https: //tpxvaaivvv///////////a////.ail
00363. pdf.
Albert Ziegler. GitHub Copilot: Parrot or crow? recitation, 2021.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. URL https: //arxiv. org/ pdf/ 2307 10.300.150000 15043.
A TUNING DETAILS
For fixed budget of B 20, we tune over the hyperparameters that control the budget on separate development set of 100 samples.
Given a fixed candidate set size represented by B, we seek to optimize the allocation of resources by tuning two parameters: k and L. These parameters determine the distribution of the available budget across L layers, while retaining the & candidate tokens from each layer in the set C' such that kL = B.
Whitebox Attacks:
We try different combinations of k and the following set [(1, 20), (2. 10), (4, 5), (5, 4), (2, 10), (1, 20)] for each of the options when choosing k candidates (top-k, bottom-k and (top-k/2 U bottom-k/2) For each k we choose the set L such that [L] = l and @ B(k, L) is maximum when evaluated the development set.
For the read-Projection Attack, we find that choosing top-4 (k = 4) highest probability candidates from each layer's intermediate distribution while keeping 5 optimal layers (17, 18, 19 20, 21)
13

---

Published as a conference paper at ICLR 2024
(12) 5) tuned separate development set of 100 sample points attains the highest attack success rate for GPT-J.
For the Probability Delta attack, we find that choosing top-2 highest probability candidates (rather than both top-k and bottom-k) from each layer's intermediate distribution while keeping 10 (LL = 10)
layers 8-9, 16-23 is optimal for the attack success for GPT-J, 36-45 for GPT2-XL.
Model
Attack-layers-HI
Attack-layers-PP 8-9, 16-23
Detense-layers
GPT-J-6B
17-21
8-9, 16-28
GPT2-XL-I.5B
41-45
36-45
36-48
Llama2-7B
23-27
21-30
23-32
Table 2: Hyperparameters following tuning of attack and defense methods.
Blackbox Attacks: Given a fixed candidate set size represented by B, we seek to optimize the allocation of resources by tuning two parameters: and S. These parameters determine the distribution of the available budget R model-generated paraphrases, while randomly sampling the the such that RS B. We try different combinations of and in the following set [(1, 20), (2, 10), (4, 5), (5, 4), (2, 10), (1, 20)]. For the Input Rephrasing attack, we find that using 4 (R 4) paraphrases with (S 5) samples from each paraphrase in the candidate set leads to highest attack success rate on the development set.
candidate tokens from each paraphrase
Whitebox Defense: For the HP and PD defenses, we choose a set of layers consisting of the attack layers as well as all the subsequent layers starting from the last attack layer to the final layer so as to propagate the defense to the final layers. See the final selected layers in Table 2.
Blackbox Defense: We choose 5 paraphrases for this defense which are obtained using the model (See Appendix for the model details and hyprrparameters)
B ADDITIONAL EXPERIMENTS
B. LLAMA2-7B
We show results for Llama2-7b in Table We modify some of the default edit parameters for of ROME so as to make the edits suitable for Llama-2-7b (Touvron et al., 2023) i.e. a reasonable rewrite score and delta accuracy. These are the hyperparameters used in ROME that we modify and their values for reproducibility: Learning rate (v (v_Ir): 5c-1, Loss layer (v (v_loss_ layer): 31, Weight decay factor (v_ v__weihht_a weight_decayy) decay): 5e-5, The threshold which the norm of the change vector (the norm of difference between original and new vector is clamped at) (clamp_nom _ormp_oorm___ factor): 2.
Attack-Successe53
A-Ace
Method
Defense
Head Projection
Probabaility Delta
Input Rephrasing
Random
Neighborhood
Rewrite Score
CounterFact
Fact-crasure
13.93
23.95
$1.43
3.30
16.20
98.33
Empty Resp
62.14
Error Inj
64,86
63.14
56.29
2.86
15,50
75.11
ROME HP Def
3.43
4.71
55.14
3.38
16.77
96.22
Max Ent def
3.14
4.57
$2.86
3.38
17.06
99.27
IR def
65.57
65.43
70.14
0.69
4.70
68.24
Table 3: Attack success rates of the the three proposed attacks (Sec. 4) on the CounterFact dataset when information is deleted from the Llama2-7B model using ROME augmented with the defense strategies (Sec. 5)
B.2 GPT2-XL
We report GPT2-XL-1.5B numbers in Table 4, and we show attack success as a function attack budget in Fig. 5.
14

---

Published as a conference paper at ICLR 2024
Attack-Success Success@20
A Ace
Head Projection
Probability Delta
Input Rephrasing
Rewrite Score
Defense
Random
Neighbors
CounterFact
GPT2-XL, ROME
Fact-orasure
35.57
40.86
6.14
1.14
3.09
97.32
HT--de
6.29
41.43
5.57
1.17
3.09
97.34
Max-Ent Def
6.29
41.43
4.71
1.18
3.09
97.34
Empty resp
28.14
34.29
5.57
1.31
1.46
99.73
IR def
34.57
72.14
27.43
0.98
1.59
98.78
Err Inj
99.72
99.99
3.4
1.5
3.1
98.03
GPT2-XI, MEMIT
Fact-erasure
55.57
41.71
17.71
0.57
1.53
98.16
Empty resp
87.00
89.71
50.43
0.38
0.54
99.95
Err Inj
89.32
89.72
31.72
0.53
1.39
99.54
HP-def
21.00
41.29
18.00
0.68
3.17
97.50
Max-Ent Def
21.86
61.29
18.14
0.67
3.13
97.55
IR def
61.43
64.86
32.29
0.91
2.90
97.59
GPT2-XL, FT
Fact-erasure
93.43
96.14
50,43
2.49
3.26
96.46
Err Inj
98.70
24.30
60.08
1.60
0.8
99.99
Empty resp
99.57
99.86
59.14
2.49
1.51
99.73
IR def
94.00
96.29
52.71
2.47
2.63
97.02
Table Attack success rates the the three proposed attacks (Sec. 4) on the CounterFact dataset when information is deleted from the GPT2-XL model using the three editing methods augmented with the defense strategies (Sec. 5)
Attack Budget
0.4
Attock
0.2
0.0
Budget
Figure 5: As the attack budget increases, the attack success increases and saturates after a budget of 10. Here budget for HP and PD attacks is 20 and that for BB (IR) attack is 10. Here the editing method is Fact erasure and model is GPT2-XL.
B.3 CONSTRAINED FINETUNING
Constrained Finetuning (Zhu et al., 2020). Here, we employ a simple optimization approach based on the Adam optimizer, incorporating an -nom constraint, as outlined by (Zhu et al., 2020). We finetune the same singular MLP weight matrix that we perform edits in ROME.
Sucees-
A-Ace
Method Defense
Head Projection
Probabaility Delta
Input Rephrasing
Random
Neighborhood
Rewrite Score
CounterFact
Fact-crasure
47.53
93.02
29.09
4.27
27.02
95.39
Empty resp
78.94
80.70
61.69
0.70
5.57
97.51
Eror Inj
54.43
60.94
56.29
2,86
15.50
75.11
FT
HP-def
99.15
96.76
75.81
0.05
0.17
Max-Ent-def
99.32
96.93
75.81
0.00
0.00
0.00
IR def
46.85
58.77
13.12
4.25
26.95
95.90
Table 5: Attack success rates of the the three proposed attacks (Sec. 4) on the CounterFact dataset when information deleted from the GPT-J model using Constrained Finetuning (FT) augmented with the defense strategies (Sec. 5)
15

---

Published as a conference paper at ICLR 2024
Attack-Successe @
A AAAc
Head Projection
Probability Delta
Input Rephrasing
Rewrite Score
Defense
Random
Neighbors
CounterFact
ROME
Fact Erasure
22.15
25.38
22.83
0.72
8.74
99.69
+
Empty Resp (dummy)
36.84
37.65
29.02
0.54
3.76
99.58
+
Empty Resp (I don't know) 54.89
55.92
32.80
0,50
3.20
98.74
Table 6: Attack success rates of the the three proposed attacks (Sec. 4) across defense methods (Sec.
5), for facts from CounterFact and zsRE that are known by GPT-J.
C EDITING METHODS AND ADVERSARY MODEL ACCESS
C.I. MODEL EDITING METHOSS
ROME (Meng al., 2022). Rank-One Model Editing (ROME) is state-of-the-ar method that changes model outputs by updating a specific MLP layer in the model (layer for GPT-J by default). The update is applied the second matrix within this MLP layer, and the update itself is constrained to be a rank-one matrix that is obtained analytically when treating the MLP weight as a linear associative memory (Meng al., 2022). The default objective that maximized by this update the model probability ((y"(z) for desired output y", with data augmentation for the input r and some regularization. Recall that we can apply this editing method to optimize any of the defense objectives from Sec. 5.
MEMIT (Meng et al., 2023). MEMIT is method designed for updating an arbitrary number of facts in model, as opposed to single fact. When applied to update only one fact, however, its only difference from ROME that "spreads out" its update over multiple MLP layers rather than a single MLP layer in ROME (Meng et al., 2023). When applying MEMIT to GPT-J, we update layers 5-7.
C.2 MODEL ACCESS
Our blackbox attack relies on simple random sampling that can be carried out through the OpenAl API as of September 28, 2023: https: : ///ppel/ atlpllafffoooo atffrr. opena i. com/ /ocs/ / ggu des / gpt. Though we perform experiments with publicly available models, this is important since many models of interest are gated behind APIs.
Purameters for OpenAl API:
Input Prompts or Instructions: Users send input prompts or instructions the OpenAl API. These prompts provide context or guidance to the model regarding the task or the type of response expected.
Sampling Parameters: To customize the behavior of the model and the characteristics of the generated text, users can specify various sampling parameters:
Sampling Temperature (temperature): This parameter controls the randomness of the gener- ated text. Higher values (e.g., 0.8) make the output more random, while lower values (e.g.. 0.2) make more deterministic.
Maximum Tokens (max_tokens) tokens): Users can limit the length of the generated text by setting a maximum number of tokens. This helps in controlling the response length.
Top-p Probability (top_p): This parameter allows users to set a probability threshold for the next token. Tokens with probabilities above this threshold are considered, which helps in influencing the diversity of generated responses.
Frequency Penalty ffrequency_penalle
Users can penalize the repetition of words in the generated text by adjusting this parameter. Higher values discourage word repetition.
Presence Penalty ppresence_penatty)): This parameter allows to penalize the presence certain words or phrases in the generated text. It can be useful for controlling the content style of the output.
16

---

Published as a conference paper at ICLR 2024
Stop Sequences (rop_ssquenness) Users can specify strings that the model should avoid generating in the output. This is helpful for preventing specific content from appearing in the generated text.
Temperature Schedule (temperature schedule): Users can provide a list of temperature values to change the temperature dynamically during the text generation process. This can result in text that starts more deterministic and becomes more random over time, or vice versa.
Top-k Tokens (top k): This parameter limits the number of tokens considered each step of generation to the top-k most likely tokens. It can help in controlling the model's creativity and focus.
API Response: After sending the input prompt and specifying the desired sampling parameters, users receive the model-generated text or response from the OpenAI API. The output text is influenced by the provided context and the parameter settings.
D REPRODUCIBILITY DETAILS
Here, we give additional details to our experiments that would be necessary for reproducing the results.
(Krishna et al., 2023) model available huggingface. We first generate paraphrases of the entire prompt, including the target answer by varying the following parameters in the model: lexical diversity in [20, 40, 60, 80], order diversity in [20, 40, 60, 80], top_p p in [0.25, 0.5, 0.75J. We then retain only the paraphrases which have the target answer the last word and obtain the paraphrased prompt by truncating the paraphrased sentence to remove the last word which is the target answer.
Paraphrase Model. We use the
Á-Ace Metrics. The length of generated output that we use for measuring A-Ace is 36.
Rewrite Score. We consider the Rewrite Score from Hase et al. (2023) traditional measure of edit success, to be reported alongside Attack-Success Success metrics. The Rewrite Score measures how much the edit changes the new target probability as a fraction of the possible desired change:
p p M*) p(y|=;
((ylz;M)
A value of means that the edit perfectly maximizes the new target probability, while a value of 0 means that the new target probability did not change all. When the probability of a target being minimized rather than maximized (which occurs in some defense objectives), this metric simply becomes p<y|=; MMMM/((/lllll M), reflecting that we desire the target probability to approach 0. Specifically, we use the original formulation for maximizing objective with Empty Response and Error Injection, and we use the simplified version (1 p(y) w M² MM")MM(() Jr; MM) when reporting Rewrite Score for Fact Erasure, Head Projection, Probability Delta, and Input Rephrasing defenses, since these methods involve lowering the probability the target answer.
Head Projection defense. atop-gradient. (See See 5).
(0) without the of any We backpropagate though both D and
Data Filtering. On top of the single-token filtering, also require the original model probability My of the correct target answer which is being deleted to be at least 0.02, in order for it be meaningful measure decrease in the next-token probability.
E RESULTS DISCUSSION
Attack success against MEMIT is generally higher than against ROME, and our best defense methods (like Max-Ent) achieve smaller improvements over baselines with MEMIT than with ROME. This could suggest that distributing editing updates across multiple layers rather than single layer, as
17

---

Published as a conference paper at ICLR 2024
MEMIT does, increases vulnerability to attacks and makes defense more challenging. However, MEMIT performs more favorably than ROME on the Rewrite Score and A-Acc metrics, meaning this difference in the methods could be due to difference in how strongly the edits applied to the model.
18


Reviews:
[{"review_id": "zqpMtEw8hG", "reviewer": "Reviewer_174w", "summary": "This paper investigates the removal of memorized information from language models via direct editing of model weights. The paper first establishes a threat model for  LLM extraction, based on the notion of recovery of sensitive information from $B$ candidates. The paper then describes several new attacks: two variants of whitebox attacks, in which an attacker may utilize probabilities computed from intermediate hidden states in the model to aid in recovery, and a blackbox attack, in which an input query is “rephrased'' multiple times by the model to generate a diverse set of candidate responses. Finally, the paper describes and evaluates several defenses against the attacks, which are able to be applied in conjunction with existing model editing techniques.", "questions": "- I’m a bit confused by the distinction between the password attempts and parallel extractions scenarios in the context of the paper. Is the distinction here meant to be that information gained in the recovery of a previous attempt may be utilized to aid recovery of the next attempt? If so, I do not understand how the attacks as described utilize information in this way (i.e. it seems all attack methods generate a candidate set for a budget $B$ in parallel anyways). Is such an attack possible?\n\n- In Sec. 2 “Attacking LLMs for Sensitive Information”, the paper claims that the described method does not assume access to the exact text used during pre-training of the model. However, the experiments do assume such access, as even in the rephrasing attack, the ground-truth prompt is perturbed by a rephrasing model. Is it possible to run the rephrasing attack without the ground-truth prompt?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Paper works on interesting and challenging problems, which are highly relevant to real-life use cases.\n\n- The paper is novel in its framing of the problem - in particular, considering model editing for data removal is fairly unique in literature.\n\n- The paper is quite broad spanning, presenting both multiple attacks and defenses for machine unlearning.", "weaknesses": "**Lack of experiments** - The paper could be more comprehensive if more scenarios were presented in the paper, e.g. combining attacks (to maximize extraction under a budget) or combining defenses (to explore how much risks could be minimized in practice). I find it hard to draw actionable conclusions from the results, and further discussion and variety of results may help justify the strength of model editing as the de-facto paradigm for unlearning.\n\n**No comparison to prior works** - I’m concerned about the lack of comparison to prior works. While the paper claims that prior works is not applicable due to focus on removal of influence of a pair $(x,y)$, the problem formalized in the paper (based on the ASR metric) is exactly the recovery of a single token label $y_i$ given a specific prompt $x_i$. Hence, I’m confused why it’s not possible to compare with prior approaches in approximate unlearning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7LTeQHABf", "reviewer": "Reviewer_GnJt", "summary": "This paper investigates model editing methods to remove sensitive information from LLM. This work shows that \"deleted\" information can be extracted from the hidden state when the attacker uses a smaller budget of verification attempts. Simple rewriting of prompts can also cause LLM to generate sensitive information. This is an interesting study, and the attack and defense methods it provides are worthy of further study and discussion.", "questions": "This paper shows some interesting results. My main question is whether the author can give more detailed insights or possible mechanisms for the findings in the paper. For example, why do the hidden states of LLMs reveal sensitive information? What is the intuition behind this phenomenon? Why are defense strategies based on data augmentation ineffective?\n\nIn addition, can fine-tuning, a typical defense strategy, be combined with the defense scheme (e.g., Head Projection Defense) proposed in this paper to produce a more powerful defense method?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work elaborates on the security issues of LLMs from the perspective that hidden states may leak sensitive information. It presents potential attack methods and defense strategies.", "weaknesses": "This work provides an incomplete description of the reasons behind some experimental phenomena. The reasons or intuitions why defense strategies based on data augmentation do not work are not revealed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7P6ZFkXvv6", "reviewer": "Reviewer_X7HP", "summary": "This paper studies whether a piece of information can be effectively deleted from an LLM. The authors adapt two model editing techniques to the task of suppressing a piece of information from an LLM. They then evaluate the robustness of these techniques, combined with different defense approaches, to several new attacks (both black-box and white-box) aiming to extract that piece of information from the LLM. The authors propose a new threat model, where the attack is considered successful if the targeted information is among B candidates produced by the attack. Empirical results using one of the model editing techniques, ROME, show that it’s not always effective, as the targeted information can still be extracted 38% of the time in a white-box setting and 29% of the time in a black-box setting. The authors further show that defense techniques can reduce the risk at a small cost in utility, but that in some cases they still remain vulnerable to attacks they don’t explicitly protect against.", "questions": "- 1) What are the results of attacking the MEMIT method without any defenses (i.e., the equivalent of Figure 4)? The paper’s second claim is that model editing methods fail to delete information; basing it on only one of two editing methods studied in the paper weakens the claim and raises questions.\n- 2) Do the authors think that model editing is technically possible with formal guarantees against attacks and what are, in the authors’ opinion, promising directions for future work in this domain?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- 1) Important problem: can information be deleted from an LLM? The premise of the paper, that the right way to delete information is to modify the model post-hoc instead of curating the training dataset, is quite contentious. In spite of this, for practical reasons model developers might indeed not curate their training data, which motivates the need to evaluate the robustness of model editing techniques.\n- 2) Well-motivated threat model, based on the insight that considering some information to be deleted only if it cannot be recovered directly (B=1) is an insufficient requirement.\n- 3) The technical contribution of the paper (attacks and defenses) is insightful, well motivated and clearly described.", "weaknesses": "- 1) The findings of the paper are somewhat expected, as the model editing techniques being evaluated are heuristics and don’t come with formal guarantees of robustness against attacks. Similarly, it is expected that a defense designed to mitigate a specific attack is robust against that attack but not necessarily against other attacks.\n- 2) Insufficient analysis of results. I was left wondering what are the technical differences between ROME and MEMIT and whether this could explain some of the differences in the results. \n\nMinor (suggestions for improvement):\n- 3) Confusing usage of the term “sensitive”: The definition used by the authors includes “toxic” information: “Models can also generate text reflecting beliefs that cause direct psychological harm to people (i.e. toxic generated text) (Kenton et al., 2021). Facts or beliefs of this kind are known as sensitive information (Brown et al., 2022)”. I'm pretty sure that’s not how Brown et al. use the word sensitive. In the privacy domain, “sensitive information” refers to protected characteristics about an individual (https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en) or is sometimes used colloquially to denote private information that a model or system should not disclose. To avoid confusion, I would suggest using a different term that explicitly refers to “toxic” information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IlG7MHHUa8", "reviewer": "Reviewer_KWLr", "summary": "The paper shows that models edited by model editing methods ROME and MEMIT still contain sensitive information and that the information is not fully \"deleted\" from the model. Several white-box and one black-box attack are proposed which are used to extract information from edited models, given that the attacker has an attack budget B. If the sensitive information is in within the B output candidates the information is assumed to be leaked. In addition to the attacks, the paper proposes multiple defense methods. Both the attacks and defenses are evaluated on the Counterfact and zsRE dataset. The evalution shows that the defense methods are not enough to defend against extraction attacks and that even in a black-box setting, information can still be extracted after editing the model.", "questions": "- **Q1:** Why use only single-token answers? If I understand this correctly, this way it is not possible to extract answers which consist of multiple tokens, correct? Is it possible to modify your approach to make this work for multi-token answers?\n- **Q2:** I don't quite understand how the Head Projection defense works. What exactly are the values D_answer and D_k? As far as I understand, D_answer is a probability distribution, while D_k is a single value? Could you clarify the loss function and what exactly is optimized for this defense?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- the paper is well written and easy to follow\n- code and everything to reproduce the experiments is given\n- the evaluation is quite thorough, evaluating multiple defenses against multiple attacks", "weaknesses": "For me it is not quite clear how the proposed defense methods are combined with the model editing techniques. In the experiments, ROME and MEMIT are used as model editing techniques. However, it is not mentioned how the different optimization objectives for the proposed defense techniques are used in combination with these methods. This could be formulated a bit clearer.\n\nMisc:\n- it would be easier to read if the paragraph in 4.2 also had a bold subheading with the name of the attack, instead of putting the name in the heading of the section. This would make it easier for readers to spot the names of the different attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)