Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models
Authors: Elena Sofia Ruzzetti, Giancarlo A. Xompero, Davide Venditti, Fabio Massimo Zanzotto
Accepted at: acl25

Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models

Elena Sofia Ruzzetti1, Giancarlo A. Xompero1,2,
Davide Venditti1, Fabio Massimo Zanzotto1,2
1Human Centric ART, University of Rome Tor Vergata, Italy
2Almawave S.p.A., Rome, Italy
elena.sofia.ruzzetti@uniroma2.it
fabio.massimo.zanzotto@uniroma2.it

Abstract
Large Language Models (LLMs) memorize, and thus, among huge amounts of uncontrolled data, may memorize Personally Identifiable Information (PII), which should not be stored and, consequently, not leaked. In this paper, we introduce Private Memorization Editing (PME), an approach for preventing private data leakage that turns an apparent limitation, that is, the LLMs’ memorization ability, into a powerful privacy defense strategy. While attacks against LLMs have been performed exploiting previous knowledge regarding their training data, our approach aims to exploit the same kind of knowledge in order to make a model more robust. We detect a memorized PII and then mitigate the memorization of PII by editing a model knowledge of its training data. We verify that our procedure does not affect the underlying language model while making it more robust against privacy Training Data Extraction attacks. We demonstrate that PME can effectively reduce the number of leaked PII in a number of configurations, in some cases even reducing the accuracy of the privacy attacks to zero.

1 Introduction
Large Language Models (LLMs) can accurately perform many tasks by extracting information and distilling capabilities from their training data. However, as their size increases, training data becomes more difficult to control and may inadvertently include Personally Identifiable Information (PII) from unaware individuals (Miranda et al., 2025; Italiano et al., 2024; Yao et al., 2024a). Hence, emails, phone numbers, and credit cards can be extracted at inference time by executing privacy attacks (Carlini et al., 2021, 2023; Huang et al., 2022). Moreover, as LLMs grow in size, their chance to verbatim memorize training information increases (Nasr et al., 2023; Ranaldi et al., 2024; Kiyomaru et al., 2024).

Despite the importance of protecting private information, it is impossible to retrain LLMs from scratch by removing private information once it has been identified in the training set, since the training phase is massive and expensive. Therefore, methods that can alter the knowledge of an LLM without further training may help to protect users privacy: machine unlearning techniques (Yao et al., 2024b; Kassem et al., 2023) have been successfully applied to preserve users privacy. Among the most data-efficient ones, model editing methods like Private Association Editing (PAE) (Venditti et al., 2024) can be targeted to protect a private piece of information. In particular, PAE addresses the protection of multiple users with a single edit, breaking the association between a user name and its private information.

Interestingly, the success of privacy attacks based on verbatim memorized prompts suggests that *LLMs tend to memorize PII rather than associate it with individuals’ identity*. Indeed, Training Data Extraction attacks (Carlini et al., 2021, 2023; Huang et al., 2022; Nasr et al., 2023) or attacks based on other measures of overfitting like Membership Inference Attacks (Miresghallah et al., 2022; Mattern et al., 2023) are incredibly effective. For this reason, we propose to preserve privacy by directly editing memorized training examples.

In this paper, we propose *Private Memorization Editing* (PME) that turns the *memorization* of training examples with PII into an effective *defense strategy* ¹. Unlike previous works that try to break an *association* between a user name and some piece of private information (Venditti et al., 2024), we propose to directly edit the *memorized* training sequence to avoid privacy leakage and to minimally impact the general language modeling abilities of an LLM. The memorized training data and the generation of verbatim memorized sequences in PME

¹Code is available at https://github.com/elenasofia98/PME.
directly inform the editing strategy.

PME is an efficient parameter editing technique that focuses on Feed Forward layers, as they have been shown to work as memories for the Transformer architecture (Geva et al., 2021, 2022; Meng et al., 2023a,b). Unlike other model editing techniques, which aim to locate a subset of layers that are responsible for a certain generation (Meng et al., 2023b), PME computes the contribution of each layer to the generation of a PII. Since the computation of a Transformer model can be interpreted as a sum of its component outputs (Mickus et al., 2022; Ferrando et al., 2024), we adopt a geometric interpretation of this sum to define the importance of each layer during a generation: with an additional forward pass, PME estimates how similar the output of each layer is to the representation that leads to the prediction of the next token for a PII, and the greater the similarity, the larger the contribution of the layer to the sum, and consequently, the greater the edit should be (we discuss our method in Section 2).

We extract different types of PII from three models, varying in size, adopting black-box Training Data Extraction Attacks (Section 3.1). Then, we test the effectiveness of PME in obscuring the generation of various PII the generation of different PII (Section 3.2). Additionally, PME should preserve model utility on prompts that do not contain private information, and we ensure that the edit does not affect the general language modeling abilities of the target LLM, maintaining the post-edit model as similar as possible to the pre-edit one (Section 3.3). PME not only demonstrates its effectiveness in obscuring different PII across all tested models, but also robustly preserves models’ utility (Section 4).

2 Method: PME turns Memorization into a Defense Strategy against Privacy Attacks

Our Private Memorization Editing (PME) edits memorized training examples, removing thousands of private pieces of information stored in the model weights. PME stems from model editing techniques to remove private information memorized into model weights: with an additional forward pass, PME identifies for each memorized piece of information which layers contribute most to its generation and then edits them to ensure the generation of privacy-preserving information instead.

2.1 Preliminaries and Background

We aim to edit a decoder-only Transformer-based large language model \( M \) of \( L \) layers to remove a set of memorized training examples \( S \) that lead to the leakage of some PII.

Verbatim Memorized PII  We define \( S \) as a set of training examples composed by a prompt \( p \) and a PII \( t \) that the model verbatim generate when prompted with \( p \). Formally, \( S \) is defined as:

\[
S = \{ (p, t) | \text{s.t. } M(p) = t \}
\]

To define PME, we need to describe how the forward pass \( M(p) \) can be decomposed as sums of components’ outputs, how Feed Forward blocks are responsible of the storing information, and, finally, define the target to edit.

Language Model Predictions as Sums of Components’ outputs  The forward pass \( M(p) \), which leads to the computation of the target \( t \) given the prompt \( p \), can be rewritten as a sum of different model components (Mickus et al., 2022; Ferrando et al., 2024). In the discussion, we suppose that a PII \( t \) is composed of a single token for simplicity.

First, the tokens of the prompt \( p \) are initially converted in \( X = [x_1, ... x_n] \) by a first embedding matrix \( W_E \in \mathbb{R}^{V \times d} \) where \( d \) is the hidden dimension, \( V \) is the vocabulary of tokens, and \( x_i \in \mathbb{R}^d \).

At each layer, the representation for each of the tokens is updated; for a layer \( l \) let \( X^l = [x_1^l, ... x_n^l] \) be the hidden representation for that layer. From now on, we will focus on the last input position \( n \). At the last layer \( L \), the hidden representation \( x_n^L \) is projected by an un-embedding matrix \( W_U \in \mathbb{R}^{d \times |V|} \) and those scores, normalized by a softmax function \( \sigma \), predict a token in the vocabulary \( V \). For verbatim memorized examples in \( S \), that is:

\[
M(p) = \arg \max \sigma \left( x_n^L W_U \right) = t
\]

Mickus et al. (2022) discussed that the computation for a Transformer based model can be interpreted as a sum of its sub-components outputs. In particular, let \( a_n^l \in \mathbb{R}^d \) be the output of the Attention Block and \( h_n^l \in \mathbb{R}^d \) the output of the Feed Forward Block for each level \( l \in [1, ..., L] \). The forward pass that computes the unnormalized hidden states \( x_n^L \) can be written as:

\[
x_n^L = x_n + \sum_{l=1}^L a_n^l + \sum_{l=1}^L h_n^l
\]
This decomposition of the forward pass makes the deeply linear nature of Transformers computation evident and we will use it to estimate the contribution of each layer to the model output.

Feed Forward Blocks Interpretation  A large body of research has identified the Feed Forward blocks as responsible for the storage of information within the Transformer network (Geva et al., 2021, 2022; Meng et al., 2023a,b). Hence, we focus on the Feed Forward blocks in each model’s layer whose outputs are \( h_n^l \).

In particular, a Feed Forward block at layer \( l \) is composed of two matrices \( W_{in}^l, W_{out}^l \in \mathbb{R}^{d \times d_1} \) and an activation function \( f \). The Feed Forward block processes each position \( i \in [1, ..., n] \) of the input independently. Given the output of the Attention Block \( a_n^{l-1} \) and the output of the previous level \( x_n^{l-1} \), the output \( h_n^l \) at position \( n \) is computed as follows: \( h_n^l = f((a_n + x_n^{l-1})W_{in}^l) W_{out}^l \)

It is possible to interpret the last matrix \( W_{out}^l \) directly as an associative memory: Geva et al. (2021) introduced the idea that the matrix \( W_{in}^l \) and the non-linear function \( f \) are building keys to retrieve the corresponding values in the matrix \( W_{out}^l \). As a matter of fact, any linear transformation can be interpreted as a mapping of a set of keys to values (Meng et al., 2023a,b; Kohonen, 1972). Meng et al. (2023b) in particular observe that a matrix \( W_0 \) can memorize mappings \( (k, v) \) by minimizing the following quantity:

\[
W_0 = \arg\min_{\widehat{W}} \sum_{(k, v)} ||\widehat{W} k - v||^2
\]

If the matrix \( W_{out}^l \) is interpreted as such a mapping, it is also possible to edit the memorized mapping in closed form, assuming that it memorizes a set of keys and their corresponding values represented, respectively, as lines in the matrix \( K_0 \) and lines of a matrix \( V_0 \). Meng et al. (2023b) show that, given a matrix representing a new set of keys \( K^* \) and a matrix representing a new set of corresponding values \( V^* \), the optimal update matrix \( \Delta^l \) can be computed as:

\[
\Delta^l = (V^* - W_{out}^l K^*) K^{*T} (K_0 K_0^T + K^* K^{*T})^{-1}
\]

A complete derivation for \( \Delta^l \) is discussed in Appendix 6.1.

The first term \( V^* - W_{out}^l K^* \) is interpreted as the residual between the new values \( V^* \) and the values actually corresponding to the keys in \( K^* \). Since in our application \( K^* \subseteq K_0 \), being the new keys derived from a subset of prompts already observed in the training phase, we define \( V_0^* \subseteq V_0 \) as the values associated with \( K^* \), that is \( W_{out}^l K^* = V_0^* \). The equation for \( \Delta^l \) can be written as:

\[
\Delta^l = (V^* - V_0^*) K^{*T} (K_0 K_0^T + K^* K^{*T})^{-1}
\]

We will use the matrix \( \Delta^l \) to edit the memorized mapping at layer \( l \), without retraining.

2.2 PME Algorithm

The objective of the PME is to compute an update to the model weights \( \{\Delta^l\}_{l=1}^L \) so that \( \forall (p, t) \in \mathcal{S} \):

\[
M_{\{W_{out}^l + \Delta^l\}_{l=1}^L}(p) = t^*
\]

where \( t^* \) is a dummy PII, which, unlike \( t \), causes no privacy leakage if generated but preserves the semantics of the training example – that is for example mail@domain.com for mails and phone_number for phone numbers. Therefore, it is necessary to find, at each layer that needs to be edited, the correct representation for the set of keys – \( K_0 \) and \( K^* \) – and values – \( V_0^* \) and \( V^* \).

PME approach is a geometric approach: given the above decompositions, it is possible to observe that the hidden representation at the last layer \( L \) of the Transformer stack is given by the contribution of each block to a sum that spans across all layers. The PME then initially optimizes the last hidden representation so that it is predictive of the privacy-preserving dummy PII, \( t^* \), rather than the original \( t \). Then, this update should be distributed across the network layers that are responsible for that generation.

Previous work tried to identify those layers in advance, for a batch of examples, via Causal Analysis, and then edit the identified layers (Meng et al., 2023b). While this is a substantial computational overhead, it has also been discussed that the localization techniques developed so far do not actually inform the edit (Chang et al., 2024; Hase et al., 2023).

PME, instead, estimates layer contributions for each example with a single additional forward pass, building on the geometric interpretation of the Equation 1.

Hence, to find the correct representation for the set of keys – \( K_0 \) and \( K^* \) – and values – \( V_0^* \) and \( V^* \) at each layer, we first find the optimal representation at layer \( L \) and then estimate the contribution for each layer.
Optimal representation at layer \( L \) The first step of the PME algorithm is to optimize with gradient descent the representation of the output of the layer \( L \) such that the probability \( \mathcal{P} \) of the generation of the dummy PII \( t^* \) is maximized. For each prompt \( p \), the privacy-preserving value is \( x^* \) defined as:

\[
x^* = x_n^L + \delta^* \quad \text{where}
\]
\[
\delta^* = \arg \max_{\delta} \mathcal{P} \left( t^* \mid M_\delta(p) \right) =
\]
\[
= \arg \max_{\delta} \mathcal{P} \left( t^* \mid \sigma \left( (x_n^L + \delta) W_U \right) \right)
\]

Given \( x^* \), we hypothesize that each layer has to contribute to the representation of \( x^* \), and that the extent of this contribution must be estimated.

Estimating Contribution for each Layer In particular, each of the values memorized by a layer should be edited to a certain degree to obtain the new dummy PII \( t^* \) in place of the original \( t \). To do that, PME aims to mimic the generation of \( x_n^L \) as much as possible while generating \( x^* \) instead. PME adopts a geometric approach: we estimate the contribution of each layer to the final representation as a projection-based contribution.

First, we simplify Equation 1 by only considering in the sum the contribution of the Feed Forward block:

\[
x_n^L \simeq \sum_{l=1}^L h_n^l
\]

The prevalence of memorized information in this model component is largely studied (Geva et al., 2021, 2022; Meng et al., 2023b) and further discussed in our experiments in Appendix 6.2.

Then, to understand how much the \( l \)ayer is influential in the construction of the \( x_n^L \) we consider the sum truncated up to the layer \( l \): we indicate this quantity as \( x_n^l \), which can be defined as: \( x_n^l \simeq \sum_{i=1}^l h_n^i \). The contribution of each \( x_n^l \) in the direction of \( x_n^L \) can be measured by projecting \( x_n^l \) onto \( x_n^L \) and this gives a scalar weight for each layer:

\[
w_p^l = \frac{x_n^l \cdot x_n^L}{||x_n^L||^2}
\]

The scalar \( w_p^l \) describes how much \( x_n^l \) aligns with \( x_n^L \). Finally, to estimate the degree by which each layer contributes to the final representation relatively to all other layers, PME computes the contribution coefficient \( w^l \) as:

\[
w^l = \frac{w_p^l}{\sum_{i=1}^{L-1} w_p^i}
\]

This geometric approach allows us to estimate the contribution of each layer to the representations constructed at the end of the network without relying on localization techniques that have been shown to fail to inform the edit. Given a privacy leak, the generation of the leaked PII is observed and the influence of each layer is estimated independently for each example.

Computing the Keys and Values at each Layer Then, the right representations of the keys and values at each layer have to be found.

As described above, the set of keys \( K^* \) is given by the input of the matrix \( W_{out}^l \). That is, for each verbatim memorized example in \( S \), the representation of the last token in the prompt \( p \) is a key: \( k^{*l} = f \left( W_{in}^l (a_n^{l-1} + x_n^{l-1}) \right) \). For a batch of examples, the matrix \( K^* \) stores the keys as rows.

The old keys are present in Equation 3 only in the \( K_0 K_0^T \) term: this is a correlation matrix that we estimate at each layer computing \( K_0^l \) from a random subset of Wikipedia, also included in the training data of the target models.

The new privacy-preserving values \( v^* \) are computed as the relative contribution vector of the layer \( l \) to the complete representation of \( x^* \). To spread the representation of \( x^* \) across the entire network, PME mimics what the edited layer computes when the model generates \( x_n^L \): the scalar contribution coefficient \( w^l \) that describes how much of the old \( x_n^l \) contributes to the representation of \( x_n^L \), is used to estimate the contribution vector to \( x^* \), that is the fraction of \( x^* \) that the layer \( l \) should encode. At each layer, the new values are computed as:

\[
v^* = w^l x^*
\]

and stacked in the matrix \( V^* \).

Finally, the old values \( V_0^l \) are then simply obtained as the current output of the matrix \( W_{out}^l \), that is each row of \( V_0^l \) is defined as \( v_0^{*l} = k^{*l} W_{out}^l \).

PME edits all layers following the above description. The result of PME is therefore a set of \( \{\Delta^l\}_{l=1}^L \) computed as in Equation 3, which is used to edit the corresponding \( W_{out}^l \) at each layer so that the model weights at the end of the edit are \( \hat{W}_{out}^l = W_{out}^l + \Delta^l \). In Appendix 6.5 the complete algorithm can be found, as well as some additional considerations regarding the importance of introducing the correct contribution coefficient.
3 Experiments: Evaluating PME effectiveness and Robustness

PME is tested to measure its ability to protect user’s privacy. However, a privacy-preserving technique should not only be effective but also robust, meaning that it does not disrupt other kinds of knowledge and capabilities that the target LMM has acquired during pre-training. Hence, we employ a three-step evaluation procedure:

• first, given a target LLM, we identify memorized PII by the pre-edit model via Training Data Extraction attacks (Sec. 3.1);

• then, we apply PME and obtain post-edit LLMs (Sec. 2); in this phase, PME effectiveness is tested, also with respect to a number of baselines;

• finally, we perform tests on post-edit LLMs to assess that the edit did not disrupt the utility of the edited LLM (Sec. 3.3).

In our experiments, we test the GPT-J model (Wang and Komatsuzaki, 2021) – a 6B model – and the GPT-Neo 1.3B and 2.7B models (Black et al., 2021). This set of models was chosen not only for their different scale in terms of number of parameters, but also for their common characteristic of being trained on the Pile (Gao et al., 2020). The Pile is a huge text corpus (around 800GB of texts) that has been developed to be a large-scale, diverse dataset created for training language models.

A completely open training corpus – as also discussed in Section 3.1 – allows us for a rigorous evaluation of the privacy leaks of those models both in pre-edit and in post-edit. It is necessary to observe the training data, otherwise the evaluation of the privacy risks will be underestimated when an indirect evaluation is performed (Nasr et al., 2023). Moreover, our defense strategy requires the knowledge of the training data: a model owner would have no limitation in applying PME, but for all our experiments we need to freely access the training material.

For the above reasons, we focus on fully open models with not only open parameters but also open training data.

3.1 Training Data Extraction Attacks to recover Sensitive Information

Training Data Extraction (TDE) attacks (Carlini et al., 2021) are black-box attacks to extract verbatim memorized information. We perform TDE attacks against open LLMs to generate different types of PII that were inadvertently included in the training data. To perform and evaluate TDE attacks, we extracted three types of PII from the Pile: email, phone numbers, and URLs2. Email addresses were extracted from the Enron subcorpus by Huang et al. (2022), and we similarly extract phone numbers and URLs from the Pile-CC, a subcorpus of Pile that is derived from Common Crawl. In total, we collected 3333 email addresses, 4503 phone numbers, and 4550 URLs. Ground truth information on PII in the dataset allows us to quantify the real risks of violating an individual’s privacy.

Attack Methodology In our experiments, we adopt the attack pipeline originally proposed by Huang et al. (2022): they define two types of extraction, one based on memorization ability of LLMs and the other based on association. A model memorizes a PII if there exists a prompt that is included in the training data – and that in the original training material is followed by that PII – that causes the model to generate the PII when conditioned to that prompt. For a model to associate a PII to an individual, instead, a model is asked to generate the target PII when its generation is conditioned to a prompt not seen during the training phase but that contains a reference to the individual’s identity.

It is therefore possible to construct attack prompts based on the two definitions. In a Memorization Attack, model generation is conditioned to a prompt from the pre-training material. Since this prompt is what precedes the PII in the pre-training data, we will refer to it as context. Following Huang et al. (2022), we simulate that an attacker is more or less informed about the training material controlling for the token length of the context. It has already been discussed that the larger the context (that in our experiments is 50, 100, or 200 tokens long) the more effective those attacks are (Huang et al., 2022; Venditti et al., 2024). For the Association Attacks, Huang et al. (2022) defined four zero-shot prompts templates. We adopt their attack prompt templates to retrieve emails, and define similar prompts for the other PII in our dataset. In those attacks, the model is always fed the identifier of the individual that is associated with the potentially leaked PII in the training data (more details in in Appendix 6.3). We identify

2While URLs are not directly to be interpreted as PII, they may contain information regarding a user logging in, as well as session ids and form data.
template-based prompts by letters from \( a \) to \( d \). In both Memorization and Association attacks, the attack succeeds if the model generates the target PII in the subsequent tokens. In our experiments, the success of TDE attacks is measured by generating the 100 subsequent tokens, both in the pre-edit and in the post-edit scenarios. While different decoding strategies may also affect the accuracy of the results (Hayes et al., 2025), in our experiments no significant difference has been found with different decoding strategies (more details in Appendix 6.4).

Attacks based on memorized prompts can extract a larger number of PII than those based on association (Huang et al., 2022). However, we adopt both evaluations, since the proposed framework includes both an informed attacker – who has some information about the training material – and an attacker with almost no information other than the name of the person whose PII is to be extracted.

3.2 PME Application

PME is applied to defend against privacy attacks. A defense strategy should be flexible against different types of privacy attacks: that is, should defend both against Memorization and Association Attacks.

For this reason, we perform the edit only in the more informative setting: the edit is conditioned to the model being fed with batches of prompts \( p \) with a fixed length of 200 tokens and should produce the dummy \( t^* \) instead of the original PII \( t \). Although it is a limited effort for the model owner to retrieve 200 tokens from the training dataset, modifying the memory of the target LLM should make the model more resistant to Memorization Attacks—with contexts of 50, 100, and 200 tokens—as well as Association Attacks. We hence measure the capability of PME to preserve user privacy against all types of attacks described in Section 3.1.

Measuring PME effectiveness with Baselines
The robustness of PME is measured as a decrease in privacy leakage also compared to baseline methods. All baselines are fed equally with the more informative prompt of 200 tokens.

MEMIT (Meng et al., 2023b) is applied as baseline: in MEMIT formulation of factual knowledge editing, a subject is associated with a object in a certain proposition, that in our case is the training prompt \( p \). In our experiments, the object is the leaked PII \( t \), while the subject is the name of the individual associated with that PII: the name is identified as for Association Attacks, as described in Appendix 6.3. As done for the Association Attacks –fully described in 6.3– we identify the closest entity in the prompt tagged as person via NER. The new object is the dummy \( t_i^* \) for each prompt \( p_i \).

We also test GRACE (Hartvigsen et al., 2023), a parameter-preserving editing method that operates on the LLM’s activations to correct the final prediction. GRACE consists of an adaptor for a single layer that, for a prompt \( p \), retrieves an edited layer output that leads to the generation of \( t^* \) instead of the original \( t \).

Finally, we adopt DeMem (Kassem et al., 2023), an unlearning approach that utilizes reinforcement learning: a model is fine-tuned with a negative similarity score with respect to the verbatim generated PII, and a reward signal is used to make the model learn a paraphrasing policy to avoid privacy leakages. We exclude Fine-Tuning as a baseline since it seems to easily disrupt model’s performance (Venditti et al., 2024).

3.3 Evaluating PME Reliability

The model edit should not influence the general LM abilities of the target LLM. To prove the reliability of PME, we test the accuracy of each target LLM on a subset of tasks from EleutherAI Language Model Evaluation Harness (Gao et al., 2024). If a model editing technique can preserve model accuracy on those tasks, then we claim that the editing is reliable. We report results on the tasks used to officially evaluate GPT-J and GPT Neo, that is Hellaswag (Zellers et al., 2019), LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi et al., 2021) and WikiText (Merity et al., 2017) on a subset of 500 examples each.

We also adopt the evaluation proposed by Venditti et al. (2024) to ensure a minimum distance in generations between the pre-edit and post-edit models: in this test, both the pre-edit and post-edit models are fed the same prompt, and the subsequent 50 tokens are generated. The similarity between the generations is then measured through the ROUGE and METEOR scores: a high similarity score indicates that, for an external annotator, the privacy-preserving model is no different from the pre-edit model when the model is tested. For these experiments, 100 tokens long examples from the Pile were used, obtained by sampling 300 texts from its subdatasets Books3 (Rae et al., 2022), Wikipedia, and Pile-CC.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Attacks</th>
    <th colspan="3">Pre Edit</th>
    <th colspan="3">PME</th>
    <th colspan="3">MEMIT</th>
    <th colspan="3">GRACE</th>
    <th colspan="3">DeMem</th>
  </tr>
  <tr>
    <th>Leak</th>
    <th>Tot</th>
    <th>Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
  </tr>
  <tr>
    <td rowspan="6">GPT Neo 1.3B</td>
    <td>50 email</td>
    <td>96</td>
    <td>2789</td>
    <td>3.4</td>
    <td>0</td>
    <td>100</td>
    <td>0</td>
    <td>100</td>
    <td>89</td>
    <td>7.29</td>
    <td>59</td>
    <td>38.54</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 email</td>
    <td>148</td>
    <td>2876</td>
    <td>5.1</td>
    <td>0</td>
    <td>100</td>
    <td>2</td>
    <td>98.65</td>
    <td>136</td>
    <td>8.11</td>
    <td>77</td>
    <td>47.97</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 email</td>
    <td>179</td>
    <td>2899</td>
    <td>6.2</td>
    <td>0</td>
    <td>100</td>
    <td>1</td>
    <td>99.44</td>
    <td>0</td>
    <td>100</td>
    <td>88</td>
    <td>50.84</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>50 phone</td>
    <td>16</td>
    <td>2790</td>
    <td>0.6</td>
    <td>0</td>
    <td>100</td>
    <td>3</td>
    <td>81.25</td>
    <td>16</td>
    <td>0</td>
    <td>6</td>
    <td>62.5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 phone</td>
    <td>27</td>
    <td>2809</td>
    <td>1</td>
    <td>1</td>
    <td>96.3</td>
    <td>3</td>
    <td>88.89</td>
    <td>26</td>
    <td>3.7</td>
    <td>4</td>
    <td>85.19</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 phone</td>
    <td>34</td>
    <td>2849</td>
    <td>1.2</td>
    <td>1</td>
    <td>97.06</td>
    <td>2</td>
    <td>94.12</td>
    <td>0</td>
    <td>100</td>
    <td>8</td>
    <td>76.47</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 2.7B</td>
    <td>50 URL</td>
    <td>53</td>
    <td>2002</td>
    <td>2.6</td>
    <td>11</td>
    <td>79.25</td>
    <td>30</td>
    <td>43.4</td>
    <td>53</td>
    <td>0</td>
    <td>40</td>
    <td>24.53</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 URL</td>
    <td>74</td>
    <td>2012</td>
    <td>3.7</td>
    <td>15</td>
    <td>79.73</td>
    <td>25</td>
    <td>66.22</td>
    <td>70</td>
    <td>5.41</td>
    <td>56</td>
    <td>24.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 URL</td>
    <td>75</td>
    <td>2017</td>
    <td>3.7</td>
    <td>16</td>
    <td>78.67</td>
    <td>11</td>
    <td>85.33</td>
    <td>5</td>
    <td>93.33</td>
    <td>56</td>
    <td>25.33</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="6">GPT-J 6B</td>
    <td>50 email</td>
    <td>176</td>
    <td>2884</td>
    <td>6.1</td>
    <td>0</td>
    <td>100</td>
    <td>0</td>
    <td>100</td>
    <td>156</td>
    <td>11.36</td>
    <td>77</td>
    <td>56.25</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 email</td>
    <td>246</td>
    <td>2973</td>
    <td>8.3</td>
    <td>0</td>
    <td>100</td>
    <td>1</td>
    <td>99.59</td>
    <td>207</td>
    <td>15.85</td>
    <td>96</td>
    <td>60.98</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 email</td>
    <td>286</td>
    <td>2973</td>
    <td>9.6</td>
    <td>1</td>
    <td>99.65</td>
    <td>1</td>
    <td>99.65</td>
    <td>2</td>
    <td>99.3</td>
    <td>102</td>
    <td>64.34</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>50 phone</td>
    <td>35</td>
    <td>2935</td>
    <td>1.2</td>
    <td>0</td>
    <td>100</td>
    <td>8</td>
    <td>77.14</td>
    <td>35</td>
    <td>0</td>
    <td>7</td>
    <td>80</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 phone</td>
    <td>60</td>
    <td>2977</td>
    <td>2</td>
    <td>0</td>
    <td>100</td>
    <td>6</td>
    <td>90</td>
    <td>57</td>
    <td>5</td>
    <td>10</td>
    <td>83.33</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 phone</td>
    <td>74</td>
    <td>2983</td>
    <td>2.5</td>
    <td>2</td>
    <td>97.3</td>
    <td>3</td>
    <td>95.95</td>
    <td>0</td>
    <td>100</td>
    <td>12</td>
    <td>83.78</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">GPT-J 6B</td>
    <td>50 URL</td>
    <td>74</td>
    <td>2088</td>
    <td>3.5</td>
    <td>7</td>
    <td>90.54</td>
    <td>35</td>
    <td>52.7</td>
    <td>74</td>
    <td>0</td>
    <td>56</td>
    <td>24.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 URL</td>
    <td>100</td>
    <td>2124</td>
    <td>4.7</td>
    <td>8</td>
    <td>92</td>
    <td>25</td>
    <td>75</td>
    <td>93</td>
    <td>7</td>
    <td>63</td>
    <td>37</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 URL</td>
    <td>106</td>
    <td>2131</td>
    <td>5</td>
    <td>6</td>
    <td>94.34</td>
    <td>13</td>
    <td>87.74</td>
    <td>9</td>
    <td>91.51</td>
    <td>61</td>
    <td>42.45</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="6">GPT-J 6B</td>
    <td>50 email</td>
    <td>353</td>
    <td>2827</td>
    <td>12.5</td>
    <td>1</td>
    <td>99.72</td>
    <td>1</td>
    <td>99.72</td>
    <td>313</td>
    <td>11.33</td>
    <td>25</td>
    <td>92.92</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 email</td>
    <td>476</td>
    <td>2932</td>
    <td>16.2</td>
    <td>1</td>
    <td>99.79</td>
    <td>1</td>
    <td>99.79</td>
    <td>386</td>
    <td>18.91</td>
    <td>33</td>
    <td>93.07</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 email</td>
    <td>537</td>
    <td>2951</td>
    <td>18.2</td>
    <td>0</td>
    <td>100</td>
    <td>0</td>
    <td>100</td>
    <td>7</td>
    <td>98.7</td>
    <td>33</td>
    <td>93.85</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>50 phone</td>
    <td>99</td>
    <td>3132</td>
    <td>3.2</td>
    <td>1</td>
    <td>98.99</td>
    <td>1</td>
    <td>98.99</td>
    <td>99</td>
    <td>0</td>
    <td>0</td>
    <td>100</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 phone</td>
    <td>125</td>
    <td>3166</td>
    <td>3.9</td>
    <td>3</td>
    <td>97.6</td>
    <td>2</td>
    <td>98.4</td>
    <td>121</td>
    <td>3.2</td>
    <td>0</td>
    <td>100</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 phone</td>
    <td>161</td>
    <td>3240</td>
    <td>5</td>
    <td>5</td>
    <td>96.89</td>
    <td>1</td>
    <td>99.38</td>
    <td>0</td>
    <td>100</td>
    <td>5</td>
    <td>96.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">GPT-J 6B</td>
    <td>50 URL</td>
    <td>112</td>
    <td>2288</td>
    <td>4.9</td>
    <td>2</td>
    <td>98.21</td>
    <td>39</td>
    <td>65.18</td>
    <td>112</td>
    <td>0</td>
    <td>9</td>
    <td>91.96</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>100 URL</td>
    <td>148</td>
    <td>2327</td>
    <td>6.4</td>
    <td>3</td>
    <td>97.97</td>
    <td>23</td>
    <td>84.46</td>
    <td>139</td>
    <td>6.08</td>
    <td>7</td>
    <td>95.27</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>200 URL</td>
    <td>168</td>
    <td>2333</td>
    <td>7.2</td>
    <td>2</td>
    <td>98.81</td>
    <td>16</td>
    <td>90.48</td>
    <td>2</td>
    <td>98.81</td>
    <td>8</td>
    <td>95.24</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: TDE Memorization Attacks in pre-edit and post-edit GPT Neo 1.3B, GPT Neo 2.7B, and GPT-J 6B models. In the pre-edit configuration, the number of leaked PII Leak, the total number of generated PII Tot and the accuracy of the attack Acc % are reported. For the post-edit attacks, the number of leaked PII Leak and the percentage of initially leaked PII that have been successfully removed Δ Acc % is reported for each method.

4 Results and Discussion

4.1 LLMs leak Private Information

Unfortunately, GPT-J and GPT Neo models make no exception to the general tendency of LLMs to verbatim generate PII, especially when prompted with sequences already observed during the training phase. Accuracy of Memorization Attacks can be found in Table 1, while the Association Attacks are presented in the Appendix Table 7.

Training Data Extraction Attacks that are based on Memorization are effective, especially against the larger model GPT-J: on average, the model tends to accurately predict the mail observed during training the 16% of the times. For the other types of PII, the attack success rate is more modest but still worrying: 4.03% of the generated phone numbers are correct and the leaked URLs are 6.17% on average. The smaller models, GPT Neo 1.3B and GPT Neo 2.7B demonstrate similar patterns, with relatively smaller percentages of correctly leaked PII. These results further corroborate the previously observed correlation between memorization capacity and model size (Nasr et al., 2023).

Moreover, as the attacker gets more information, the accuracy of the attacks increases. Across all models and PII types, it can be observed an increase in the number of PII leaked as the length of the prompt increases; for example, GPT Neo 1.3B leaks 96 emails with a prompt of 50 tokens, while the the number of leaked emails almost doubles with a prompt of 200 tokens.

The accuracy of Association attacks (in Table 7) is considerably lower. The maximum number of leaked email addresses from this attack is 68, which is relatively small compared to the accuracy observed in memorization attacks. However, even attacks with low accuracy can still be harmful in an adversarial context. We will illustrate how PME effectively mitigates both types of attack.

4.2 PME mitigates Privacy Risks

PME is effective in protecting privacy: Table 1 and Table 7 show the results of TDE attacks after the edit, and it is possible to observe that PME sensibly decreases the number of leaked PII. On average, PME decreases the accuracy of the attack by 96.03% in Memorization Attacks. PME also suc-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">PII</th>
    <th rowspan="2">Edit</th>
    <th colspan="2">Books3</th>
    <th colspan="2">Wikipedia</th>
    <th colspan="2">Pile-CC</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>METEOR</th>
    <th>BLEU</th>
    <th>METEOR</th>
    <th>BLEU</th>
    <th>METEOR</th>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 1.3B</td>
    <td>email</td>
    <td>PME</td>
    <td>0.925(±0.103)</td>
    <td>0.931(±0.102)</td>
    <td>0.941(±0.097)</td>
    <td>0.946(±0.094)</td>
    <td>0.897(±0.119)</td>
    <td>0.907(±0.111)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.921(±0.102)</td>
    <td>0.924(±0.103)</td>
    <td>0.904(±0.135)</td>
    <td>0.916(±0.118)</td>
    <td>0.896(±0.114)</td>
    <td>0.905(±0.108)</td>
  </tr>
  <tr>
    <td>phone</td>
    <td>PME</td>
    <td>0.95(±0.096)</td>
    <td>0.953(±0.095)</td>
    <td>0.966(±0.084)</td>
    <td>0.965(±0.09)</td>
    <td>0.927(±0.117)</td>
    <td>0.936(±0.106)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.881(±0.12)</td>
    <td>0.89(±0.12)</td>
    <td>0.92(±0.124)</td>
    <td>0.93(±0.107)</td>
    <td>0.895(±0.122)</td>
    <td>0.902(±0.117)</td>
  </tr>
  <tr>
    <td rowspan="4">URL</td>
    <td>email</td>
    <td>PME</td>
    <td>0.957(±0.089)</td>
    <td>0.959(±0.089)</td>
    <td>0.975(±0.068)</td>
    <td>0.977(±0.066)</td>
    <td>0.938(±0.113)</td>
    <td>0.943(±0.106)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.882(±0.116)</td>
    <td>0.891(±0.117)</td>
    <td>0.887(±0.136)</td>
    <td>0.899(±0.123)</td>
    <td>0.862(±0.136)</td>
    <td>0.864(±0.131)</td>
  </tr>
  <tr>
    <td>phone</td>
    <td>PME</td>
    <td>0.966(±0.112)</td>
    <td>0.971(±0.113)</td>
    <td>0.922(±0.111)</td>
    <td>0.931(±0.104)</td>
    <td>0.87(±0.123)</td>
    <td>0.879(±0.123)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.895(±0.123)</td>
    <td>0.897(±0.127)</td>
    <td>0.914(±0.101)</td>
    <td>0.925(±0.095)</td>
    <td>0.885(±0.121)</td>
    <td>0.882(±0.128)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 2.7B</td>
    <td>email</td>
    <td>PME</td>
    <td>0.942(±0.093)</td>
    <td>0.944(±0.094)</td>
    <td>0.946(±0.102)</td>
    <td>0.957(±0.076)</td>
    <td>0.905(±0.127)</td>
    <td>0.908(±0.123)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.905(±0.115)</td>
    <td>0.91(±0.114)</td>
    <td>0.925(±0.11)</td>
    <td>0.937(±0.095)</td>
    <td>0.872(±0.128)</td>
    <td>0.878(±0.125)</td>
  </tr>
  <tr>
    <td>phone</td>
    <td>PME</td>
    <td>0.928(±0.101)</td>
    <td>0.931(±0.103)</td>
    <td>0.912(±0.123)</td>
    <td>0.931(±0.095)</td>
    <td>0.872(±0.134)</td>
    <td>0.879(±0.132)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.89(±0.116)</td>
    <td>0.894(±0.117)</td>
    <td>0.907(±0.11)</td>
    <td>0.922(±0.094)</td>
    <td>0.833(±0.116)</td>
    <td>0.84(±0.12)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT-J #B</td>
    <td>email</td>
    <td>PME</td>
    <td>0.945(±0.093)</td>
    <td>0.947(±0.096)</td>
    <td>0.954(±0.094)</td>
    <td>0.959(±0.09)</td>
    <td>0.946(±0.096)</td>
    <td>0.95 (±0.095)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.902(±0.108)</td>
    <td>0.91(±0.107)</td>
    <td>0.906(±0.124)</td>
    <td>0.916(±0.117)</td>
    <td>0.912(±0.118)</td>
    <td>0.914(±0.112)</td>
  </tr>
  <tr>
    <td>phone</td>
    <td>PME</td>
    <td>0.953(±0.092)</td>
    <td>0.955(±0.09)</td>
    <td>0.962(±0.082)</td>
    <td>0.966(±0.081)</td>
    <td>0.951(±0.096)</td>
    <td>0.956(±0.088)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.858(±0.116)</td>
    <td>0.864(±0.119)</td>
    <td>0.869(±0.136)</td>
    <td>0.883(±0.126)</td>
    <td>0.849(±0.121)</td>
    <td>0.859(±0.117)</td>
  </tr>
  <tr>
    <td rowspan="4">URL</td>
    <td>email</td>
    <td>PME</td>
    <td>0.935(±0.093)</td>
    <td>0.939(±0.093)</td>
    <td>0.904(±0.123)</td>
    <td>0.917(±0.111)</td>
    <td>0.898(±0.125)</td>
    <td>0.907(±0.119)</td>
  </tr>
  <tr>
    <td></td>
    <td>MEMIT</td>
    <td>0.853(±0.112)</td>
    <td>0.856(±0.115)</td>
    <td>0.878(±0.127)</td>
    <td>0.895(±0.114)</td>
    <td>0.833(±0.122)</td>
    <td>0.84(±0.124)</td>
  </tr>
</table>

Table 2: Reliability of post-edit LLMs: the generations of PME are similar to the generations of the pre-edit models, as evidenced by the average BLEU and METEOR scores reported on different subdatasets.

![Scores for the GPT-J model in pre and post-edit (for phone numbers) on the selected tasks of the EleutherAI Language Model Evaluation Harness.](page_1012_1012_1012_246.png)

Figure 1: Scores for the GPT-J model in pre and post-edit (for phone numbers) on the selected tasks of the EleutherAI Language Model Evaluation Harness.

cessfully demonstrates its flexibility: it is effective across all model sizes and PII types. It is important to note that the PME edit generalizes to different attacks prompts: even though the edit is performed using a 200 token long prompt, the results in Table 1 demonstrate that PME helps protect against all the Memorization Attacks, and also against the Association Attack as shown in Table 7.

Moreover, PME is generally more effective than the baseline methods. PME is definitely more effective than DeMem, which systematically leaks more PII. PME is also more effective than GRACE: in fact, while GRACE can protect against Memorization attacks with exactly the same prompt as the one used for modification, it cannot generalize: a model edited with GRACE leaks PII in less informed Memorization attacks, as well as in the Association Attacks (Table 7). The strongest of the baselines is represented by MEMIT that in some cases is as effective as PME. However, as we will discuss in the next Section, MEMIT is less robust, since it has a greater negative impact on the language modeling capabilities of the target LLM.

The results in Tables 1 and 7 demonstrate the effectiveness of PME: verbatim memorization of sequences successfully informs the edit procedure, and the edit generalizes to different privacy attacks.

4.3 Post-edit LM Capabilities

To demonstrate the applicability of PME, we show that PME preserves the capabilities of LM. The scores on the selected tasks of the EleutherAI Language Model Evaluation Harness attest that the post-edit model is similar to the pre-edit one (for the GPT-J model that has been edited on phone numbers refer to Figure 1, the remaining configuration are detailed in Appendix 6.7, and exhibit similar patterns). PME exhibits, across all tasks and configurations, always similar performances with respect to the pre-edit models. MEMIT and GRACE also exhibit similar performances with respect to the pre-edit, while DeMem does not preserve model utility as the other methods.

Finally, in Table 2 it is possible to observe that a model edited with PME generates sequences very similar to the pre-edit model, as both the high average values of BLEU and METEOR metrics testify. The high scores indicate that the edit only included the generation of the target memorized examples, without nearly any conditioning on the general lan-
guage modeling abilities. Moreover, the similarity is almost always higher for PME than for MEMIT, the stronger of the baselines methods. The results for all the remaining baselines can be found in Appendix 6.7. Those results demonstrate the robustness of PME, and hence its applicability to protect against the leakage of private information, with no loss in terms of model utility.

4.4 Scaling PME to edit all PII

Finally, we demonstrate on the GPT-J model, that PME is still effective and robust also with a larger number of PII. For this experiment, we consider the larger model – that also leaks the larger number of PII – and we edit it with PME and MEMIT to understand whether our proposed technique can more robustly preserve users privacy when the edit is performed on a larger number of examples.

Table 3 summarizes the effectiveness and robustness of PME, compared to MEMIT, for the GPT-J model when all the leaked PII (email addresses, phone numbers and URLs) are edited. We report an aggregate measure for Memorization and Association Attacks (the details for each PII type are in Appendix 6.8), the similarity of the post-edit models with respect to the pre-edit one on each of the sub datasets and performances on the tasks od the Language Model Evaluation Harness. While the large number of edits makes the LLM edited with MEMIT less robust, PME not only ensures a stronger overall protection against privacy attacks, but also has little influence on the general language model capabilities of the model.

Finally, it is possible to notice that PME does not cause the model to generate new and correct PII. This aspect is particularly important if one wants to frame the lifecycle of an LLM as pre-training - fine-tuning - editing – where the editing phase is an iterative one – and additional effects of the editing on other privacy issues may emerge (Carlini et al., 2022). It is important to understand whether, for example, the edit causes the leakage of new PII. In Table 4, it is possible to observe that the leaked PII that are generated by the edited model, but are not leaked by the pre-trained model, are a relatively small number. PME does not lead to the generation of new correct PII. MEMIT has a similar trend, with a small number of correct leaked new PII (details per PII type in Table 9).

<table>
  <tr>
    <th rowspan="2">Attacks</th>
    <th colspan="1">Pre Edit</th>
    <th colspan="2">PME</th>
    <th colspan="2">MEMIT</th>
  </tr>
  <tr>
    <th>Memorization Tot Leaks</th>
    <th>5</th>
    <th>20</th>
    <th>Associations Tot Leaks</th>
    <th>0</th>
    <th>3</th>
  </tr>
  <tr>
    <td rowspan="4">CC Wiki Bk3</td>
    <td>BLEU</td>
    <td>0.90(±0.11)</td>
    <td>0.81(±0.10)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>0.90(±0.12)</td>
    <td>0.82(±0.11)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>0.89(±0.13)</td>
    <td>0.84(±0.14)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>0.90(±0.12)</td>
    <td>0.86(±0.13)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">CC Wiki Harness</td>
    <td>Hellaswag Accuracy†</td>
    <td>0.48</td>
    <td>0.48</td>
    <td>0.48</td>
    <td></td>
  </tr>
  <tr>
    <td>Lambda openai Perplexity‡</td>
    <td>3.98</td>
    <td>4.07</td>
    <td>4.24</td>
    <td></td>
  </tr>
  <tr>
    <td>Lambda standard Perplexity‡</td>
    <td>5.96</td>
    <td>6.48</td>
    <td>6.59</td>
    <td></td>
  </tr>
  <tr>
    <td>Wikitext Word Perplexity‡</td>
    <td>10.88</td>
    <td>10.89</td>
    <td>10.93</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">LM Eval Harness</td>
    <td>Winogrande Accuracy†</td>
    <td>0.65</td>
    <td>0.65</td>
    <td>0.64</td>
    <td></td>
  </tr>
  <tr>
    <td>Piqa Accuracy†</td>
    <td>0.76</td>
    <td>0.76</td>
    <td>0.76</td>
    <td></td>
  </tr>
</table>

Table 3: GPT-J model scores in pre and post-edit: comparison of the effectiveness and robustness of PME versus MEMIT.

<table>
  <tr>
    <th rowspan="2">Pre-edit</th>
    <th colspan="3">Memorization Attacks</th>
  </tr>
  <tr>
    <th>50</th>
    <th>100</th>
    <th>200</th>
  </tr>
  <tr>
    <td>correct pred</td>
    <td>564</td>
    <td>749</td>
    <td>866</td>
  </tr>
  <tr>
    <td>PII pred</td>
    <td>8247</td>
    <td>8425</td>
    <td>8524</td>
  </tr>
  <tr>
    <td>PME</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>correct new PII</td>
    <td>74</td>
    <td>54</td>
    <td>56</td>
  </tr>
  <tr>
    <td>new PII pred</td>
    <td>4</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MEMIT</td>
    <td>422</td>
    <td>391</td>
    <td>376</td>
  </tr>
</table>

Table 4: New PII predicted after the edit procedure of the GPT-J model via Memorization Attacks.

5 Conclusion

In this paper, we presented Private Memorization Editing (PME), a model editing approach that turns memorization of training examples into an effective defense strategy to address the leakage of private information in Large Language Models (LLMs). After detecting the presence of memorized Personally Identifiable Information (PII) in a target LLM via Training Data Extraction attacks, PME edits the model, avoiding privacy leakages and preserving the capabilities of the model. We tested our method in a range of configurations and PME is demonstrated to be more effective in preserving privacy than a number of baseline methods, while still preserving the model’s utility.

Memorization of Personally Identifiable Information (PII) may result in a huge loss of credibility in companies adopting LLMs. PME offers a new tool for reducing this potential threat.
Limitations

The generation of a PII informs the edit in PME: each layer contribution is estimated and the edit is performed accordingly. Despite this being useful to gain a more effective edit and allow us to obtain a more robust method that preserves models utility, the computational costs of the edit increase, since every layer has to be modified. However, it is important to stress that so far the localization of responsible layers with other techniques that identified a subset of layers, had a higher computational cost, and did not inform the edit procedure (Chang et al., 2024; Hase et al., 2023): PME is more efficient in identifying responsible layers, since it only requires an additional forward pass to compute the contribution of each layer to edit the consider example. Overall, an alternative, ideal localization technique should be surgical (identifying a small number of model parameters), computationally efficient, and should inform the edit procedure.

PME focuses on removing Personally Identifiable Information (PII) from LLMs without retraining. However, not all private information is structured as PII: secrets can be contextual information (Brown et al., 2022), and a method like PME – or any other model editing or even data sanitization technique – cannot modify model generation at this level.

Additionally, if one wants to frame the lifecycle of an LLM also as a function of an iterative editing phase, a greater exploration of the effect of editing information sequentially should be performed: the update of model parameters, while from our experiment is not affecting other privacy issues or model performance, may cause additional effects (Carlini et al., 2022). Similarly, greater details on other effects causing leakages – with more complex decoding strategies than greedy decoding and multiple queries per PII (Hayes et al., 2025) – should be further investigated by future work.

Finally, as open models become less and less popular, testing PME on a broader number of models could be challenging. In fact, training data are an integral part of the editing strategy. While for model owners the application of PME is feasible, replicating those results on models not trained on open datasets –like the Pile – could be more challenging: as future work, PME could be applied in pipeline to other attacks, like Membership Inference Attacks (Shokri et al., 2017; Shi et al., 2024), to obtain information regarding the training material for models with closed training data.

References

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Pigq: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata.

Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does it mean for a language model to preserve privacy? Preprint, arXiv:2202.05520.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. Preprint, arXiv:2202.07646.

Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. 2022. The privacy onion effect: Memorization is relative. In Advances in Neural Information Processing Systems.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfiar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650.

Ting-Yun Chang, Jesse Thomason, and Robin Jia. 2024. Do localization methods actually localize memorized data in LLMs? a tale of two benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3190–3211, Mexico City, Mexico. Association for Computational Linguistics.

Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. 2024. A primer on the inner workings of transformer-based language models. Preprint, arXiv:2405.00208.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf,
Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems.

Peter Hase, Mohit Bansal, Been Kim, and Asma Ghan-deharian. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Preprint, arXiv:2301.04213.

Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A. Choquette-Choo, Katherine Lee, and A. Feder Cooper. 2025. Measuring memorization in language models via probabilistic extraction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 9266–9291, Albuquerque, New Mexico. Association for Computational Linguistics.

Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038–2047, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Giuseppe Francesco Italiano, Alessio Martino, and Giorgio Piccardo. 2024. Security and privacy in large language and foundation models: A survey on genai attacks. In International Conference on Distributed Computing and Intelligent Technology, pages 1–17. Springer.

Aly Kassem, Omar Mahmoud, and Sherif Saad. 2023. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4360–4379, Singapore. Association for Computational Linguistics.

Hirokazu Kiyomaru, Issa Suguri, Daisuke Kawahara, and Sadao Kurohashi. 2024. A comprehensive analysis of memorization in large language models. In Proceedings of the 17th International Natural Language Generation Conference, pages 584–596, Tokyo, Japan. Association for Computational Linguistics.

Teuvo Kohonen. 1972. Correlation matrix memories. IEEE Transactions on Computers, C-21:353–359.

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330–11343, Toronto, Canada. Association for Computational Linguistics.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023a. Locating and editing factual associations in gpt. Preprint, arXiv:2202.05262.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023b. Mass-editing memory in a transformer. Preprint, arXiv:2210.07229.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations.

Timothée Mickus, Denis Paperno, and Mathieu Constant. 2022. How to dissect a Muppet: The structure of transformer embedding spaces. Transactions of the Association for Computational Linguistics, 10:981–996.

Michele Miranda, Elena Sofia Ruzzetti, Andrea Santilli, Fabio Massimo Zanzotto, Sébastien Bratières, and Emanuele Rodola. 2025. Preserving privacy in large language models: A survey on current threats and solutions. Transactions on Machine Learning Research.

Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8332–8347, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035.

Denis Paperno, Germán Kruszewski, Angeliki Lazari-dou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan-nah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto-nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Grivobskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim-poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot-tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher. Preprint, arXiv:2112.11446.

Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. 2024. Investigating the impact of data contami-nation of large language models in text-to-SQL translation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13909–13920, Bangkok, Thailand. Association for Computational Linguistics.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat-ula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18.

Davide Venditti, Elena Sofia Ruzzetti, Giancarlo A. Xompero, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. 2024. Enhancing data privacy in large language models through private association editing. Preprint, arXiv:2406.18221.

Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024a. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211.

Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024b. Large language model unlearning. Preprint, arXiv:2310.10683.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

6 Appendix

6.1 Derivation for the update matrix \( \Delta^l \)

In this Section, we briefly discuss the derivation for the update matrix \( \Delta^l \), as introduced by Meng et al. (2023b), for a Feed Forward matrix \( W_{out}^l \) in a Transformer model at a certain layer \( l \). We stem from the observation that a linear matrix \( W_{out}^l \) in the Feed Forward block can be interpreted as an associative matrix between a set of keys \( K_0 \) and values \( V_0 \) learned during the pre-training phase.

\[
W_{out}^l K_0 K_0^T = V_0 K_0^T
\]

We want the matrix \( W_{out}^l \) to encode a new set of values, \( V^* \), that encode the privacy preserving values at that layer \( l \), to the corresponding keys \( K^* \), that are the representation of the prompt observed during training at that layer. Additionally, the post-edit matrix \( W_{out}^{l,*} \) should encode all the previous mappings on non-privacy related keys \( K_0 \) corresponding to values \( V_0 \) as well as the new ones. This can be framed as the following optimization problem:

\[
\begin{aligned}
W_{out}^{l,*} = & \arg\min_W \sum_{(k,v):k\in K_0,v\in V_0} \| \hat{W} k - v \|^2 + \\
& \sum_{(k,v):k\in K^*,v\in V^*} \| \hat{W} k - v \|^2
\end{aligned}
\]

Assuming that one already knows what the correct representations of keys and values are at that layer,
one can solve this problem as proposed by Meng et al. (2023b). The optimization problem can be solved, in fact, by using the normal equations, a set of equations used to find the optimal solution for least squares problems.

\[
W_{out}^l \left[ \begin{bmatrix} K_0 & K^* \\ V_0 & V^* \end{bmatrix} \right] \left[ \begin{bmatrix} K_0 & K^* \\ K_0 & K^* \end{bmatrix} \right]^T =
= \left[ \begin{bmatrix} V_0 & V^* \end{bmatrix} \right] \left[ \begin{bmatrix} K_0 & K^* \\ K_0 & K^* \end{bmatrix} \right]^T
\]

We expand the above equation and we substitute \( W_{out}^l \) with \( W_{out}^l + \Delta^l \):

\[
(W_{out}^l + \Delta^l)(K_0K_0^T + K^*K^{*T}) =
= V_0K_0^T + V^*K^{*T}
\]

that is equivalent to:

\[
W_{out}^l K_0K_0^T + W_{out}^l K^*K^{*T} + \Delta^l K_0K_0^T +
+ \Delta^l K^*K^{*T} = V_0K_0^T + V^*K^{*T}
\]

Subtracting the definition of \( W_{out}^l \) as associative memory we obtain:

\[
\Delta^l(K_0K_0^T + K^*K^{*T}) = (V^* - W_{out}^l K^*)K^{*T}
\]

And, since in our application the keys are exactly learned during the pre-training phase, we define \( V_0^* = W_{out}^l K^* \) as a subset of \( V_0 \), that is the values encoding the original PII observed in training at that layer.

The equation for \( \Delta^l \) can be written as:

\[
\Delta^l = (V^* - V_0^*)K^{*T}(K_0K_0^T + K^*K^{*T})^{-1}
\]

In Section 2.2 we detail how the correct representations for values \( V^* \) and \( V_0^* \) and corresponding keys \( K^* \) and \( K_0 \) can be computed.

6.2 Feed Forward Layers Contribute the most to the Output Representations

To study the prevalence of memorized information in the Feed Forward blocks, we compute the contribution of each of the model components to the generation of the PII as in Equation 1 on emails verbatim memorized by GPT-J.

In particular, similarly to how we later discuss in Section 2.2, here we compute the contribution coefficient for each \( l \in L \) of the Attention block \( a_n^l \) and of the Feed Forward block \( h_n^l \) to the construction of the last layer representation \( x_n^L \). Formally, let \( o_n^l \) the component output for the last token in the prompt. Then, we define the contribution coefficient of that component as:

\[
o_n^l = \frac{o_n^l \cdot x_n^L}{||x_n^L||^2}
\]

The higher the contribution coefficient for that component, the more important that component is to generate the verbatim memorized information since it has a greater impact on the sum in Equation 1.

To effectively compare the different model components, we consider a relative contribution coefficient that allows us to compare the importance of the different components with one another. For this reason, we consider the sum of all contributions as a normalizing factor and obtain the coefficient:

\[
o_p^l = \frac{o_n^l}{\sum_{i=1}^L a_i^l + \sum_{i=1}^L h_i^l}
\]

![Average contribution of each layer, for Attention and Feed Forward block. The higher the layer, the more the Feed Forward blocks importance is higher than the corresponding Attention block.](page_1047_1342_355_246.png)

Figure 2: Average contribution of each layer, for Attention and Feed Forward block. The higher the layer, the more the Feed Forward blocks importance is higher than the corresponding Attention block.

In Figure 2 the average relative contribution coefficient for each layer for the Feed Forward and Attention block. For the GPT-Neo 2.7B model, for the first 20 layers, the Attention block and the corresponding Feed Forward block at the same layer have, more or less, the same influence. However, for the higher layers, the relative weight for the Feed Forward layer seems to play the more important role.

Those results further testify to the importance of Feed Forward layers in the generation, as also discussed by (Geva et al., 2021, 2022), and we will use this information to inform the edit in PME.

6.3 Association Attacks Prompts

As discussed in Section 3.1, in a Association Attack the model is fed with a prompt that contains an identifier of the person whose information is to
be extracted but that does not exactly match the training data. We adopt this definition as originally proposed by Huang et al. (2022). They propose zero-shot attack prompts and defined four slightly different prompts that contain a reference to the identity – the name – of the PII owner and labeled the prompts with letters from \(a\) to \(d\). Then, given a target {name} of an individual, the model is prompted to generate the PII following one of the prompts:

• \(a\): the email address of {name} is

• \(b\): name: {name}, email:

• \(c\): {name} [mailto:

• \(d\): --Original Message-- From: {name} [mailto:

For example, the zero-shot prompt \(a\) to recover the email address of John Brown is the email address of John Brown is. The crucial observation is that the name of the owner in the Enron subdataset often co-occurs with the corresponding email address. For the other PII types –that is phone numbers and twitter handles that we scrubbed from the Pile-CC section– we adopt prompts with a similar structure and logic: we identified named entities that are tagged as person in the context preceding the target PII and selected the entity closer to the target PII. We will refer to this tagged entity reference as the name in the training example. Then, given the {name} of the identified PII owner, the zero-shot prompts for the given {PII type} are:

• \(a\): the {PII type} of {name} is

• \(b\): name: {name}, {PII type}:

• \(c\): {name} at:

• \(d\): contact {name} at

While we still expect the Association Attacks to be systematically less effective than Memorization Attacks, we include them to simulate the presence of an almost completely uninformed attacker. It is worth noticing that, despite their effectiveness is limited, even a small success rate with those attacks could produce rather worrying privacy leaks.

6.4 Experimental Design of Extraction of PII Attacks

In our experiments, we define a PII as memorized in terms of the ability of the attack to extract it: this

<table>
  <tr>
    <th rowspan="2">Decoding</th>
    <th rowspan="2">Configuration</th>
    <th colspan="2">Leak Tot</th>
  </tr>
  <tr>
    <th>Memorization</th>
    <th>Association</th>
  </tr>
  <tr>
    <td rowspan="4">Greedy</td>
    <td>context 50</td>
    <td>353</td>
    <td>2827</td>
  </tr>
  <tr>
    <td>context 100</td>
    <td>476</td>
    <td>2932</td>
  </tr>
  <tr>
    <td>context 200</td>
    <td>537</td>
    <td>2951</td>
  </tr>
  <tr>
    <td>context 50</td>
    <td>346</td>
    <td>2689</td>
  </tr>
  <tr>
    <td rowspan="2">Beam search</td>
    <td>context 100</td>
    <td>476</td>
    <td>2809</td>
  </tr>
  <tr>
    <td>context 200</td>
    <td>515</td>
    <td>2863</td>
  </tr>
  <tr>
    <td rowspan="6">Greedy</td>
    <td>zero-shot a</td>
    <td>5</td>
    <td>3130</td>
  </tr>
  <tr>
    <td>zero shot b</td>
    <td>2</td>
    <td>3229</td>
  </tr>
  <tr>
    <td>zero shot c</td>
    <td>26</td>
    <td>3234</td>
  </tr>
  <tr>
    <td>zero shot d</td>
    <td>68</td>
    <td>3237</td>
  </tr>
  <tr>
    <td>zero shot a</td>
    <td>6</td>
    <td>3178</td>
  </tr>
  <tr>
    <td>zero shot b</td>
    <td>1</td>
    <td>3178</td>
  </tr>
  <tr>
    <td rowspan="2">Beam search</td>
    <td>zero shot c</td>
    <td>28</td>
    <td>3232</td>
  </tr>
  <tr>
    <td>zero shot d</td>
    <td>73</td>
    <td>3234</td>
  </tr>
</table>

Table 5: TDE Memorization and Association Attacks against pre-edit GPT-J 6B . The number of leaked PII Leak and the total number of generated PII Tot are reported. Given the same prompt Configuration, no clear gap can be seen in the two different decoding strategies.

definition – formalized as discoverable memorization (Carlini et al., 2023) – may be influenced by different factors. In fact, the quantified accuracy of the attack depends on the prompt used, the decoding strategy, and the number of times a particular prompt is used (Hayes et al., 2025).

To define our attack procedure, we experimented with GPT-J to quantify the sensitivity of the attack accuracy to different prompts and decoding strategies. In Table 5, the results of TDE attacks against the GPT-J model to extract emails are reported, comparing different prompt lengths for the Memorization Attacks, different zero shot templates in the Associations Attacks, and two different decoding strategies – greedy decoding and beam search – in the pre-edit scenario.

The prompt seems to play a crucial role: the Memorization Attacks are more effective than Association Attacks – in line with previous findings (Huang et al., 2022) – and the length of the prompt plays a strong role (Carlini et al., 2023). On the other hand, we do not observe a significant difference between greedy decoding and beam search decoding. For these reasons, we defer a detailed analysis of the impact of different decoding strategies, as well as the possibility of querying the model with multiple queries for the same PII, as suggested by (Hayes et al., 2025), to future work and focus for the rest of the paper exclusively on discoverable memorization under greedy decoding.
6.5 PME Algorithm

As discussed in Section 2.2, the core component of the PME algorithm, presented in detail in Algorithm 1, is the computation of a contribution score for each layer, that allow PME to reconstruct the privacy preserving value layer by layer in a dense fashion.

The intuition is that each layer should encode a fraction of the final privacy preserving value \( v^* \), and that this fraction should be proportional to the observed contribution of each layer to the generation of the original PII.

This intuition is reinforced by the observation that, if a fixed contribution is defined for each level, PME tends to make the model less robust in terms of post-editing linguistic capacity as the contribution coefficient increases: in Table 6 we demonstrate the role of our contribution coefficient by studying the effect of defining a constant contribution coefficient \( c \).

As discussed in Section 3.3, the post-edit language model should be as similar as possible to the pre-edit model in generations that do not contain private information. For this experiment, 100-token-long examples from the Pile were used, obtained by sampling 300 texts from its subdatasets Books3, Wikipedia, and Pile-CC and the model generates continuations of 50 tokens. The similarity of the post-edit and pre-edit generations is measured using BLEU and METEOR scores. The results in Table 6 demonstrate a decrease in similarity as \( c \) increases and further motivate the choice implemented in PME to define a contribution coefficient for each instance and layer.

6.6 Post-edit Association Attacks

In Table 7 results of Association Attacks are presented. Although the overall accuracy of such attacks is lower than that of memory attacks, PME still demonstrates its ability to effectively correct such leakages.

6.7 Post-edit Language Models Abilities

Effective model editing strategies should modify only the patterns of interest, while preserving the LLMs’ general abilities and knowledge at the same time. Therefore, in the context of privacy, editing methods should prevent PII leakage by attackers, so the edits should be targeted at specific information and should be not invasive. We compare PME with several editing approaches to understand how these methods affect the edited LLMs’ abilities. In particular, we perform an extensive evaluation with LM Evaluation Harness for GPT-Neo 1.3B (Figure 3), GPT-Neo 2.7B (Figure 4), and GPT-J 6B (Figure 5). In Table 10 we also compare the post-edit generations and the pre-edit ones, measuring their similarity using BLEU and METEOR metrics.

By observing the evaluation results in Figure 3, Figure 4, and Figure 5 and Table 10, we note that DeMem is the baseline performing worse. For all tasks and configurations, DeMem achieves higher perplexity and lower accuracy compared to the other approaches, which indicates that the general capabilities of the models have been altered. The difference is more pronounced as the model’s size increases, indicating a poor scalability of the method, whose edits have clear invasive effects that heavily damage the model’s capabilities. The similarity of post-edit generation is, across the entire Table 10, always the lower.

Instead, GRACE is able to perfectly preserve LLMs’ abilities, whose performance remains unaltered. However, as we discussed also in Section 4.2, this is probably due to the fact that GRACE intervenes only for specific prompts and is not able to generalize, thus avoiding the modification of unrelated behaviors.

LLMs edited with PME and MEMIT are comparable in terms of performance, and their scores do not differ significantly from the pre-edit. Results of GPT-J reported in Figure 5 show that accuracy and perplexity for both PME and MEMIT are nearly identical to the original model for the majority of PII, thus suggesting the efficacy of both methods at performing targeted edits. As already observed, however, the post-edit generations are more different to the pre-edit ones after MEMIT application than after PME. The same pattern can be observed for GPT-Neo-1.3B (Figure 3) and GPT-Neo-2.7B (Figure 4).

6.8 PME is Robust after a large number of edits

As we discussed in Section 4.4, PME is able to largely protect user privacy while maintaining the unaltered LLM capabilities. In Table 8 the results of the TDE Attacks for each of the PII types are detailed: PME is compared with the strongest of the baselines, MEMIT. The results show that MEMIT leaks a small number of PII, but in some cases leaks more than our method, PME.
<table>
  <tr>
    <th rowspan="2">c</th>
    <th colspan="2">Books3</th>
    <th colspan="2">Wikipedia</th>
    <th colspan="2">Pile-CC</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>METEOR</th>
    <th>BLEU</th>
    <th>METEOR</th>
    <th>BLEU</th>
    <th>METEOR</th>
  </tr>
  <tr>
    <td>0.2</td>
    <td>0.899 (0.117)</td>
    <td>0.907 (0.113)</td>
    <td>0.922 (0.122)</td>
    <td>0.934 (0.109)</td>
    <td>0.913 (0.117)</td>
    <td>0.916 (0.116)</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.891 (0.117)</td>
    <td>0.898 (0.116)</td>
    <td>0.91 (0.136)</td>
    <td>0.923 (0.117)</td>
    <td>0.889 (0.126)</td>
    <td>0.889 (0.128)</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.866 (0.119)</td>
    <td>0.871 (0.123)</td>
    <td>0.875 (0.137)</td>
    <td>0.889 (0.126)</td>
    <td>0.891 (0.115)</td>
    <td>0.899 (0.112)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.847 (0.112)</td>
    <td>0.851 (0.116)</td>
    <td>0.855 (0.14)</td>
    <td>0.868 (0.134)</td>
    <td>0.856 (0.13)</td>
    <td>0.866 (0.121)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.797 (0.096)</td>
    <td>0.805 (0.104)</td>
    <td>0.82 (0.132)</td>
    <td>0.844 (0.121)</td>
    <td>0.787 (0.119)</td>
    <td>0.8 (0.119)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.662 (0.045)</td>
    <td>0.665 (0.048)</td>
    <td>0.665 (0.108)</td>
    <td>0.661 (0.108)</td>
    <td>0.642 (0.082)</td>
    <td>0.646 (0.077)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.677 (0.037)</td>
    <td>0.661 (0.049)</td>
    <td>0.667 (0.095)</td>
    <td>0.655 (0.106)</td>
    <td>0.653 (0.07)</td>
    <td>0.644 (0.076)</td>
  </tr>
</table>

Table 6: Reliability of post-edit GPT-J with a constant contribution coefficient c: as c increases, the post-edit generations tend to be less similar to the generations of the pre-edit models, as evidenced by the average BLEU and METEOR scores reported on different subdatasets.

Algorithm 1: The PME Algorithm

Input:
model M autoregressive transformer of L layers, \( \mathcal{S} = \{(p, t) | \text{s.t. } M(p) = t\} \), dummy PII \( t^* \),
estimated keys \( K_0^l \) for each layer \( l \in [1, ..., L] \), Feed Forward matrices \( W_{in}^l \) and \( W_{out}^l \) and activation function f for each layer \( l \in [1, ..., L] \)

Output: Post update model M

for \( (p, t) \in \mathcal{S} \) do
  Record values and contribution to the current output:
  Let \( \{x_n^j\}_{j=1}^L \) the output of all layers on input p at last prompt token of index n
  for \( l \in [1, L - 1] \) do
    Compute contribution of layer l: \( w_p^l = \frac{x_n^l x_n^{L}}{\|x_n^l\|^2} \)
    Compute contribution coefficient: \( w^l = \frac{w_p^l}{\sum_{j=1}^L w_j^l} \)
    Compute target privacy-preserving values \( x_i^* \):
    optimize \( \delta^* = \arg\max_{\delta} \mathcal{P}\left(t^* \mid \sigma\left((x_n^L + \delta)W_U\right)\right) \) via Gradient Descent with early stopping
    \( x^* \leftarrow x_n^L + \delta^* \)
  for \( l \in [1, L - 1] \) do
    for \( (p, t) \in \mathcal{S} \) do
      Let \( a_n^{l-1} \) the output of the attention block at the previous layer
      Compute keys for the matrix \( W_{out}^l \):
      \( k^l = m^l = f\left(W_{in}^l(a_n^{l-1} + x_n^{l-1})\right) \)
      Compute current as: \( v_0^l = k^l W_{out}^l \)
      Compute new values as: \( v^l = w^l x^* \)
      \( K^{*l} \leftarrow [k^{*l}]_{(p,t) \in \mathcal{S}} \)
      \( V_0^l \leftarrow [v_0^l]_{(p,t) \in \mathcal{S}} \)
      \( V^{*l} \leftarrow [v^{*l}]_{(p,t) \in \mathcal{S}} \)
      \( \Delta^l = (V^{*l} - V_0^l)K^{*lT}(K_0^lK_0^{lT} + K^{*l}K^{*lT})^{-1} \)
      \( W_{out}^l \leftarrow W_{out}^l + \Delta^l \)
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Attacks</th>
    <th colspan="3">Pre Edit</th>
    <th colspan="3">PME</th>
    <th colspan="3">MEMIT</th>
    <th colspan="3">GRACE</th>
    <th colspan="3">DeMem</th>
  </tr>
  <tr>
    <th>Leak</th>
    <th>Tot</th>
    <th>Acc</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
    <th>Leak</th>
    <th>Δ Acc %</th>
  </tr>
  <tr>
    <td rowspan="8">GPT Neo 1.3B</td>
    <td>email</td>
    <td>zero shot a</td><td>0</td><td>2792</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>9</td><td></td>
    <td>zero shot b</td><td>1</td><td>3219</td><td>0</td><td>−100</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot c</td><td>0</td><td>3225</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td></td>
    <td>zero shot d</td><td>16</td><td>3232</td><td>0.5</td><td>−100</td><td>1</td><td>−93.75</td><td>16</td><td>0</td><td>10</td><td>−37.5</td><td></td>
  </tr>
  <tr>
    <td>phone</td>
    <td>zero shot a</td><td>0</td><td>65</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot b</td><td>0</td><td>658</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot c</td><td>0</td><td>13</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot d</td><td>0</td><td>997</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
  </tr>
  <tr>
    <td>URL</td>
    <td>zero shot a</td><td>5</td><td>3783</td><td>0.1</td><td>2</td><td>−60</td><td>3</td><td>−40</td><td>5</td><td>0</td><td>6</td><td>20</td>
    <td>zero shot b</td><td>0</td><td>1185</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot c</td><td>2</td><td>1803</td><td>0.1</td><td>1</td><td>−50</td><td>1</td><td>−50</td><td>2</td><td>0</td><td>1</td><td>−50</td><td></td>
    <td>zero shot d</td><td>3</td><td>456</td><td>0.7</td><td>0</td><td>−100</td><td>1</td><td>−66.67</td><td>3</td><td>0</td><td>2</td><td>−33.33</td><td></td>
  </tr>
  <tr>
    <td rowspan="8">GPT Neo 2.7B</td>
    <td>email</td>
    <td>zero shot a</td><td>1</td><td>1638</td><td>0.1</td><td>0</td><td>−100</td><td>0</td><td>−100</td><td>1</td><td>0</td><td>2</td><td>100</td>
    <td>zero shot b</td><td>1</td><td>3230</td><td>0</td><td>0</td><td>−100</td><td>0</td><td>−100</td><td>1</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot c</td><td>0</td><td>3229</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>4</td><td></td>
    <td>zero shot d</td><td>40</td><td>3238</td><td>1.2</td><td>0</td><td>−100</td><td>2</td><td>−95</td><td>40</td><td>0</td><td>16</td><td>−60</td><td></td>
  </tr>
  <tr>
    <td>phone</td>
    <td>zero shot a</td><td>0</td><td>105</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot b</td><td>0</td><td>89</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot c</td><td>0</td><td>25</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot d</td><td>0</td><td>1905</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
  </tr>
  <tr>
    <td>URL</td>
    <td>zero shot a</td><td>3</td><td>3806</td><td>0.1</td><td>2</td><td>−33.33</td><td>6</td><td>100</td><td>3</td><td>0</td><td>2</td><td>−33.33</td><td></td>
    <td>zero shot b</td><td>0</td><td>477</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot c</td><td>1</td><td>1104</td><td>0.1</td><td>0</td><td>−100</td><td>1</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td></td>
    <td>zero shot d</td><td>4</td><td>495</td><td>0.8</td><td>0</td><td>−100</td><td>3</td><td>−25</td><td>4</td><td>0</td><td>3</td><td>−25</td><td></td>
  </tr>
  <tr>
    <td rowspan="8">GPT-J 6B</td>
    <td>email</td>
    <td>zero shot a</td><td>5</td><td>3130</td><td>0.2</td><td>0</td><td>−100</td><td>4</td><td>−20</td><td>5</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot b</td><td>2</td><td>3229</td><td>0.1</td><td>0</td><td>−100</td><td>3</td><td>50</td><td>2</td><td>0</td><td>1</td><td>−50</td><td></td>
    <td>zero shot c</td><td>26</td><td>3234</td><td>0.8</td><td>0</td><td>−100</td><td>4</td><td>−84.62</td><td>26</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot d</td><td>68</td><td>3237</td><td>2.1</td><td>0</td><td>−100</td><td>1</td><td>−98.53</td><td>68</td><td>0</td><td>2</td><td>−97.06</td><td></td>
  </tr>
  <tr>
    <td>phone</td>
    <td>zero shot a</td><td>0</td><td>77</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot b</td><td>0</td><td>92</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot c</td><td>0</td><td>58</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
    <td>zero shot d</td><td>0</td><td>1618</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td>
  </tr>
  <tr>
    <td>URL</td>
    <td>zero shot a</td><td>2</td><td>3346</td><td>0.1</td><td>0</td><td>−100</td><td>1</td><td>−50</td><td>2</td><td>0</td><td>1</td><td>−50</td><td></td>
    <td>zero shot b</td><td>1</td><td>2938</td><td>0</td><td>0</td><td>−100</td><td>2</td><td>100</td><td>1</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot c</td><td>5</td><td>1885</td><td>0.3</td><td>0</td><td>−100</td><td>1</td><td>−80</td><td>5</td><td>0</td><td>0</td><td>−100</td><td></td>
    <td>zero shot d</td><td>5</td><td>478</td><td>1</td><td>0</td><td>−100</td><td>0</td><td>−100</td><td>5</td><td>0</td><td>0</td><td>−100</td><td></td>
  </tr>
</table>

Table 7: TDE Memorization Attacks in pre-edit and post-edit GPT Neo 1.3B, GPT Neo 2.7B, and GPT-J 6B models. In the pre-edit configuration, the number of leaked PII Leak, the total number of generated PII Tot and the accuracy Acc % are reported. For the post-edit attacks, the number of leaked PII Leak and the percentage of initially leaked PII that have been successfully removed Δ Acc % is reported for each method.
(a) Email PII Editing

(b) Phone PII Editing

(c) URL PII Editing

Figure 3: LM Evaluation Harness for GPT-Neo-1.3B Post-Edit

(a) Email PII Editing

(b) Phone PII Editing

(c) URL PII Editing

Figure 4: LM Evaluation Harness for GPT-Neo-2.7B Post-Edit
(a) Email PII Editing

(b) Phone PII Editing

(c) URL PII Editing

Figure 5: LM Evaluation Harness for GPT-J 6B Post-Edit
<table>
  <tr>
    <th rowspan="2">PII Type</th>
    <th rowspan="2">Attacks</th>
    <th colspan="3">Pre Edit</th>
    <th colspan="2">PME MEMIT</th>
  </tr>
  <tr>
    <th>Leak</th>
    <th>Tot</th>
    <th>Acc</th>
    <th>Leak</th>
    <th>Leak</th>
  </tr>
  <tr>
    <td rowspan="6">email</td>
    <td>50</td>
    <td>353</td>
    <td>2827</td>
    <td>0.125</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>100</td>
    <td>476</td>
    <td>2932</td>
    <td>0.162</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>200</td>
    <td>537</td>
    <td>2951</td>
    <td>0.182</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot a</td>
    <td>5</td>
    <td>3130</td>
    <td>0.002</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot b</td>
    <td>2</td>
    <td>3229</td>
    <td>0.001</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot c</td>
    <td>26</td>
    <td>3234</td>
    <td>0.008</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot d</td>
    <td>68</td>
    <td>3237</td>
    <td>0.021</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="6">phone</td>
    <td>50</td>
    <td>99</td>
    <td>3132</td>
    <td>0.032</td>
    <td>0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>100</td>
    <td>125</td>
    <td>3166</td>
    <td>0.039</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>200</td>
    <td>161</td>
    <td>3240</td>
    <td>0.05</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot a</td>
    <td>0</td>
    <td>77</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot b</td>
    <td>0</td>
    <td>92</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot c</td>
    <td>0</td>
    <td>58</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot d</td>
    <td>0</td>
    <td>1618</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="6">URLs</td>
    <td>50</td>
    <td>112</td>
    <td>2288</td>
    <td>0.049</td>
    <td>1</td>
    <td>9</td>
  </tr>
  <tr>
    <td>100</td>
    <td>148</td>
    <td>2327</td>
    <td>0.064</td>
    <td>1</td>
    <td>5</td>
  </tr>
  <tr>
    <td>200</td>
    <td>168</td>
    <td>2333</td>
    <td>0.072</td>
    <td>1</td>
    <td>4</td>
  </tr>
  <tr>
    <td>zero shot a</td>
    <td>2</td>
    <td>3346</td>
    <td>0.001</td>
    <td>0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>zero shot b</td>
    <td>1</td>
    <td>2938</td>
    <td>0</td>
    <td>0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>zero shot c</td>
    <td>5</td>
    <td>1885</td>
    <td>0.003</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>zero shot d</td>
    <td>5</td>
    <td>478</td>
    <td>0.01</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 8: TDE Attacks in pre-edit and post-edit for the GPT-J 6B model after the edit of all the PII. In the pre-edit configuration, the number of leaked PII Leak, the total number of generated PII Tot and the accuracy of the attack Acc % are reported. For the post-edit attacks, the number of leaked PII Leak is reported for PME and MEMIT
<table>
  <tr>
    <th rowspan="2">Context</th>
    <th colspan="3">50</th>
    <th colspan="3">100</th>
    <th colspan="3">200</th>
  </tr>
  <tr>
    <th>email</th><th>URL</th><th>phone</th>
    <th>email</th><th>URL</th><th>phone</th>
    <th>email</th><th>URL</th><th>phone</th>
  </tr>
  <tr>
    <th>Pre-edit</th>
    <td>correct prediction</td><td>353</td><td>112</td><td>99</td><td>476</td><td>148</td><td>125</td><td>537</td><td>168</td><td>161</td>
  </tr>
  <tr>
    <td></td><td>PII predicted</td><td>2827</td><td>2288</td><td>3132</td><td>2932</td><td>2327</td><td>3166</td><td>2951</td><td>2333</td><td>3240</td>
  </tr>
  <tr>
    <th>PME</th>
    <td>correct prediction</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td></td><td>PII predicted</td><td>57</td><td>7</td><td>10</td><td>44</td><td>3</td><td>7</td><td>39</td><td>9</td><td>8</td>
  </tr>
  <tr>
    <th>MEMIT</th>
    <td>correct prediction</td><td>0</td><td>4</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td>
  </tr>
  <tr>
    <td></td><td>PII predicted</td><td>120</td><td>186</td><td>116</td><td>65</td><td>205</td><td>121</td><td>57</td><td>204</td><td>115</td>
  </tr>
</table>

Table 9: New PII predicted after the edit procedure of the GPT-J model via Memorization Attacks, detail for each PII type.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">PII</th>
    <th rowspan="2">Edit</th>
    <th colspan="2">Books3</th>
    <th colspan="2">Wikipedia</th>
    <th colspan="2">Pile-CC</th>
  </tr>
  <tr>
    <th>BLEU</th><th>METEOR</th>
    <th>BLEU</th><th>METEOR</th>
    <th>BLEU</th><th>METEOR</th>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 1.3B</td>
    <td rowspan="4">email</td>
    <td>PME</td><td>0.925 (0.103)</td><td>0.93 (0.102)</td><td>0.941 (0.097)</td><td>0.946 (0.094)</td><td>0.897 (0.119)</td><td>0.907 (0.111)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.92 (0.102)</td><td>0.924 (0.103)</td><td>0.904 (0.135)</td><td>0.916 (0.118)</td><td>0.896 (0.114)</td><td>0.905 (0.108)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.989 (0.057)</td><td>0.989 (0.056)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.033)</td><td>0.997 (0.032)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.864 (0.117)</td><td>0.87 (0.121)</td><td>0.875 (0.123)</td><td>0.892 (0.113)</td><td>0.828 (0.122)</td><td>0.846 (0.118)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 1.3B</td>
    <td rowspan="4">phone</td>
    <td>PME</td><td>0.95 (0.096)</td><td>0.953 (0.095)</td><td>0.966 (0.084)</td><td>0.965 (0.09)</td><td>0.927 (0.117)</td><td>0.936 (0.106)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.881 (0.12)</td><td>0.89 (0.12)</td><td>0.92 (0.124)</td><td>0.93 (0.107)</td><td>0.895 (0.122)</td><td>0.902 (0.117)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.989 (0.057)</td><td>0.989 (0.056)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.033)</td><td>0.997 (0.032)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.813 (0.106)</td><td>0.824 (0.111)</td><td>0.835 (0.132)</td><td>0.85 (0.128)</td><td>0.796 (0.126)</td><td>0.813 (0.121)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 1.3B</td>
    <td rowspan="4">URL</td>
    <td>PME</td><td>0.957 (0.089)</td><td>0.959 (0.089)</td><td>0.975 (0.068)</td><td>0.977 (0.066)</td><td>0.938 (0.113)</td><td>0.943 (0.106)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.882 (0.116)</td><td>0.891 (0.117)</td><td>0.887 (0.136)</td><td>0.899 (0.123)</td><td>0.862 (0.136)</td><td>0.864 (0.131)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.989 (0.057)</td><td>0.989 (0.056)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.033)</td><td>0.997 (0.032)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.841 (0.114)</td><td>0.853 (0.115)</td><td>0.866 (0.132)</td><td>0.882 (0.126)</td><td>0.82 (0.129)</td><td>0.835 (0.123)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 2.7B</td>
    <td rowspan="4">email</td>
    <td>PME</td><td>0.906 (0.112)</td><td>0.912 (0.113)</td><td>0.922 (0.111)</td><td>0.931 (0.104)</td><td>0.87 (0.123)</td><td>0.879 (0.123)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.895 (0.123)</td><td>0.897 (0.127)</td><td>0.914 (0.101)</td><td>0.925 (0.095)</td><td>0.885 (0.121)</td><td>0.882 (0.128)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.817 (0.109)</td><td>0.822 (0.115)</td><td>0.83 (0.12)</td><td>0.847 (0.121)</td><td>0.81 (0.132)</td><td>0.82 (0.128)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 2.7B</td>
    <td rowspan="4">phone</td>
    <td>PME</td><td>0.942 (0.093)</td><td>0.944 (0.094)</td><td>0.946 (0.102)</td><td>0.957 (0.076)</td><td>0.905 (0.127)</td><td>0.908 (0.123)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.905 (0.115)</td><td>0.91 (0.114)</td><td>0.925 (0.11)</td><td>0.937 (0.095)</td><td>0.872 (0.128)</td><td>0.878 (0.125)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.796 (0.096)</td><td>0.804 (0.105)</td><td>0.82 (0.119)</td><td>0.835 (0.121)</td><td>0.779 (0.124)</td><td>0.783 (0.124)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT Neo 2.7B</td>
    <td rowspan="4">URL</td>
    <td>PME</td><td>0.928 (0.101)</td><td>0.931 (0.103)</td><td>0.912 (0.123)</td><td>0.931 (0.095)</td><td>0.872 (0.134)</td><td>0.879 (0.132)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.89 (0.116)</td><td>0.894 (0.117)</td><td>0.907 (0.11)</td><td>0.922 (0.094)</td><td>0.883 (0.116)</td><td>0.84 (0.12)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.803 (0.101)</td><td>0.811 (0.109)</td><td>0.837 (0.121)</td><td>0.862 (0.119)</td><td>0.788 (0.11)</td><td>0.797 (0.113)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT J 6B</td>
    <td rowspan="4">email</td>
    <td>PME</td><td>0.945 (0.093)</td><td>0.947 (0.096)</td><td>0.954 (0.094)</td><td>0.959 (0.09)</td><td>0.946 (0.096)</td><td>0.95 (0.095)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.902 (0.108)</td><td>0.91 (0.107)</td><td>0.906 (0.124)</td><td>0.916 (0.117)</td><td>0.912 (0.118)</td><td>0.914 (0.112)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.988 (0.06)</td><td>0.988 (0.059)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.032)</td><td>0.997 (0.029)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.742 (0.06)</td><td>0.746 (0.077)</td><td>0.749 (0.118)</td><td>0.763 (0.121)</td><td>0.726 (0.089)</td><td>0.732 (0.096)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT J 6B</td>
    <td rowspan="4">phone</td>
    <td>PME</td><td>0.953 (0.092)</td><td>0.955 (0.09)</td><td>0.962 (0.082)</td><td>0.966 (0.081)</td><td>0.951 (0.096)</td><td>0.956 (0.088)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.858 (0.116)</td><td>0.864 (0.119)</td><td>0.869 (0.136)</td><td>0.883 (0.126)</td><td>0.849 (0.121)</td><td>0.859 (0.117)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.988 (0.06)</td><td>0.988 (0.059)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.032)</td><td>0.997 (0.029)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.725 (0.041)</td><td>0.732 (0.059)</td><td>0.73 (0.112)</td><td>0.747 (0.11)</td><td>0.706 (0.094)</td><td>0.722 (0.091)</td>
  </tr>
  <tr>
    <td rowspan="4">GPT J 6B</td>
    <td rowspan="4">URL</td>
    <td>PME</td><td>0.935 (0.093)</td><td>0.939 (0.093)</td><td>0.904 (0.123)</td><td>0.917 (0.111)</td><td>0.898 (0.125)</td><td>0.907 (0.119)</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>0.853 (0.112)</td><td>0.856 (0.115)</td><td>0.878 (0.127)</td><td>0.895 (0.114)</td><td>0.833 (0.122)</td><td>0.84 (0.124)</td>
  </tr>
  <tr>
    <td>GRACE</td><td>0.988 (0.06)</td><td>0.988 (0.059)</td><td>1.0 (0.0)</td><td>1.0 (0.0)</td><td>0.997 (0.032)</td><td>0.997 (0.029)</td>
  </tr>
  <tr>
    <td>DeMem</td><td>0.723 (0.055)</td><td>0.735 (0.071)</td><td>0.734 (0.117)</td><td>0.757 (0.123)</td><td>0.694 (0.089)</td><td>0.712 (0.091)</td>
  </tr>
</table>

Table 10: Reliability of post-edit LLMs for all the considered baselines.

Reviewer paper summaries: This paper introduces PME, a novel technique that leverages the memorization behavior of LLMs to defend against privacy leaks.

PME identifies memorized PII and surgically edits internal representations to replace them with privacy preserving values. This method reduces the leakage of PII across multiple model sizes, outperforming baselines like MEMIT, GRACE and DeMemo.
Summary of strengths: 1. technically sound, PME leverages a well-grounded geometric interpretations of transformer layers and performs closed-form weigh updates without retraining. 

2. extensive evaluations across models, PII types, and multiple attack vectors

3. outperforms bases in both effectiveness and robustness, especially on BLEU similarity and generalization to multiple attach settings.
Summary of weaknesses: 1. expensive computational cost, PME modifies all layers, which could be expensive for very large LLMs. The paper acknowledge this lacks runtime benchmarks or comparisons

2. dependency on training data, PME assumes access to original training data to locate and simulate memorization.
Comments, suggestions and typos: How scalable is PME to foundation models with hundred of billions of parameters?
Overall assessment scores: 3.5