Venue: NAACL 2024

# Do Localization Methods Actually Localize Memorized Data in LLMs?

A Tale of Two Benchmarks

Ting-Yun Chang  Jesse Thomason Robin Jia

University of Southern California, Los Angeles, CA, USA

{tingyun, jessetho, robinjia}@usc.edu

###### Abstract

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization methods to pinpoint LLM components responsible for memorized data. In our INJ Benchmark, we actively _inject_ a piece of new information into a small subset of LLM weights, enabling us to directly evaluate whether localization methods can identify these "ground truth" weights. In our DEL Benchmark, we evaluate localization by measuring how much dropping out identified neurons _deletes_ a memorized pretrained sequence. Despite their different perspectives, our two benchmarks yield consistent rankings of five localization methods. Methods adapted from network pruning perform well on both benchmarks, and all evaluated methods show promising localization ability. On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.1

Footnote 1: Code link: [https://github.com/terarachang/MemPi](https://github.com/terarachang/MemPi)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3190–3211

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs) memorize many sequences from their pretraining corpora Carlini et al. (2019); Lehman et al. (2021); Lee et al. (2023). For example, Carlini et al. (2021) show that GPT2 Radford et al. (2019) can leak some private contact information verbatim. This paper studies whether we can _localize_ a piece of memorized data, i.e., identify components in LLMs responsible for generating a sequence (near) verbatim. Successful localization may inform further work in machine unlearning Cao and Yang (2015); Bourtoule et al. (2021); for instance, one could apply "neural surgery" to the located components to make the LLM forget a piece of sensitive information.

Prior work on knowledge editing suggests that we can locate a small set of LLM parameters that store factual knowledge Dai et al. (2022); Meng et al. (2022). These works demonstrate localization success by showing knowledge editing success when updating only the located LLM parameters. However, Hase et al. (2023) argue that editing success and localization are actually uncorrelated. Similarly, prior methods that identify subnetworks in LLMs Gong et al. (2022); Panigrahi et al. (2023) usually focus on the performance of downstream classification tasks, lacking direct evaluation on localization per se. Hence, the degree of existing methods' localization success remains unclear.

This paper studies the open question, "Do localization methods actually localize memorized data in LLMs?" We first propose decoupling localization success from downstream success in our INJ Benchmark. Our key insight is to actively create the ground-truth weights responsible for data memorization. Specifically, we force LLMs to use a small set of pre-decided weights to memorize a piece of new information unseen during pretraining. Therefore, we have the ground-truth locations where the new information is injected. We can then directly evaluate how well different localization methods recall the indices of the injected weights.

We further apply the localization methods to a real-world scenario: identifying a small set of neurons in an LLM responsible for memorizing a pretrained sequence. In this setting, evaluating localization success is more challenging because the ground-truth "location" of each memorized sequence is unknown. We propose the DEL Benchmark, inspired by knockouts Olsson et al. (2022), a reverse-engineering approach that removes a set of nodes from the computation graph to observe their importance for specific model behavior. We first collect a set of memorized sequences, and for each sequence, we drop out the located neurons to measure their importance to memorizing that target sequence. A successful localization should cleanly erase the target sequence from an LLM without

hurting the memorization of the other sequences in the set after dropout. Our two benchmarks complement each other: the INJ Benchmark provides a direct evaluation of localization methods under a well-controlled setup, while DEL Benchmark answers if the methods can localize pretrained sequences that LLMs have already memorized.

We systematically evaluate five methods on our two benchmarks, including existing localization methods (Activations, Geva et al., 2022; IG, Dai et al., 2022), a brute-force method that searches for the most important neurons (Zero-Out), and two methods we adapt from network pruning (Hassibi and Stork, 1992; Han et al., 2016), Slimming and Hard Concrete. Our two benchmarks rank the five methods in the same order, showing especially strong localization ability for Hard Concrete. For example, dropping out only \(0.5\%\) of neurons in Pythia-6.9B (Biderman et al., 2023) identified by Hard Concrete makes the model forget \(57.7\%\) of the target memorized tokens on average. On the other hand, the DEL Benchmark shows all methods struggle to balance between erasing the target sequence and retaining other memorized data, indicating that the identified neurons are also relevant for memorizing some other sequences. Overall, both benchmarks agree all evaluated localization methods are promising, but precise localization of a single sequence remains difficult.

## 2 Background and Task Terminology

A Transformer layer (Vaswani et al., 2017) consists of multi-head self-attention and a feed-forward network (FFN). Prior work shows that LLMs use their FFNs rather than self-attention as "memories" to store knowledge (Geva et al., 2021, 2022; Meng et al., 2022). Here, an FFN has two fully connected layers with a non-linear activation function \(\sigma\):

\[h^{l}=\sigma(W^{l}\,\mathbf{x}^{l}) \tag{1}\] \[o^{l}=V^{l}\,h^{l}, \tag{2}\]

where \(\mathbf{x}^{l}\in\mathbb{R}^{d_{1}}\) is the input hidden states to the \(l\)-th FFN layer, \(W^{l}\in\mathbb{R}^{d_{2}\times d_{1}},V^{l}\in\mathbb{R}^{d_{1}\times d_{2}}\) are the weights, \(h^{l}\in\mathbb{R}^{d_{2}}\) the intermediate hidden states, and \(o^{l}\in\mathbb{R}^{d_{1}}\) the output hidden states. Geva et al. (2022) rewrite Eq. 2 as a linear combination of columns of \(V^{l}\). Let \(\mathbf{v}^{l}_{i}\in\mathbb{R}^{d_{1}}\) be the \(i\)-th column of \(V^{l}\) and \(h^{l}_{i}\in\mathbb{R}\) be the \(i\)-th neuron activation of \(h^{l}\in\mathbb{R}^{d_{2}}\). We have:

\[o^{l}=V^{l}\,h^{l}=\sum_{i=1}^{d_{2}}h^{l}_{i}\cdot\mathbf{v}^{l}_{i} \tag{3}\]

They show that different concepts are stored in different \(\mathbf{v}^{l}_{i}\), and that we can view each activation \(h^{l}_{i}\) as a memory coefficient to retrieve a concept.

Neurons.Dai et al. (2022) observe the existence of knowledge neurons, a small set of neurons in FFN hidden states \(h^{l}\) that corresponds to a relational fact, where a neuron means a component of the vector \(h^{l}\). For example, given the input "_The capital of Ireland is --_", they can increase the model probability on the correct token "_Dublin_" by amplifying the activation \(h^{l}_{i}\) of the identified knowledge neurons. With Eq. 3, we can view increasing activation \(h^{l}_{i}\) as promoting the concept stored in \(\mathbf{v}^{l}_{i}\).

In this work, we only search for neurons in FFNs responsible for memorizing a sequence, following Dai et al. (2022). In the INJ Benchmark, we ensure that FFNs act as neural memories by only updating a set of weight vectors \(\mathbf{v}^{l}_{i}\) to memorize the new information. As each \(\mathbf{v}^{l}_{i}\) corresponds to a neuron in \(h^{l}\), locating the updated weights is equivalent to locating the corresponding neurons. In the rest of the paper, we refer to neurons as the neurons in \(\left\{h^{l}\right\}_{l=1}^{L}\), where \(L\) is the number of layers.

Dropout.Different from Srivastava et al. (2014), we drop out located neurons at test time to erase a memorized sequence from the LLM. We can view dropping out the \(i\)-th neuron in \(h^{l}\) as excluding the contribution of \(\mathbf{v}^{l}_{i}\) from the output \(o^{l}\) in Eq. 3.

Memorized Sequences.Consider a sequence \(x=(p,s)\) that consists of a prefix \(p\) and a suffix \(s\). Given the prefix as the prompt, if an LLM is able to _nearly reconstruct_ the suffix with greedy decoding, we say \(x\) is a memorized sequence by the LLM. We discuss in SS3.2 our criteria on suffix reconstruction, where we tolerate near-verbatim memorization; we also ensure every sequence has a non-trivial suffix.

Localization.Hase et al. (2023) provides a general definition of localization: identifying components of a model responsible for a certain behavior. Under this definition, we consider components as a small set of neurons and behavior as the LLM's generation of a memorized sequence. Although some components are necessary for generation, e.g., the input and output token embeddings, we exclude them as they are not specific to a target sequence.

Localization Methods.Given an LLM, a memorized sequence \(x\), and a fixed number \(k\), a localization method outputs \(k\%\) of neurons at each layer as the predictions to localize sequence \(x\) in the LLM 



## 3 Two Localization Benchmarks

How do we know whether a method is successful in localization? We propose two benchmarking approaches: one _injects_ a new piece of information into specific parameters in LLMs, while another _deletes_ an existing memorized sequence from LLMs via dropout. A successful localization method should do well on both benchmarks.

### The INJ Benchmark

A principal challenge in evaluating localization methods is the lack of ground-truth location. We propose the INJ Benchmark, which first creates ground truth by actively injecting a piece of unseen information into a small subset of LLM weights. We can then directly evaluate the correctness of a localization method in predicting the indices of those injected weights.

Data.The ECBD-2021 dataset Onoe et al. (2022) contains 156 definition sentences of new entities that rose to popularity during the year 2021, e.g., "_Gamma variant, also known as lineage P.I..._". Since all LLMs we use are trained on corpora released before 2021, the injected weights are the only parameters in the LLMs responsible for memorizing each new definition sequence \(x\).

Information Injection.For each new sequence \(x_{i}\) in the dataset, we randomly sample \(r\%\) of the weight vectors \(\{\mathbf{v}^{l}_{1},\dots,\mathbf{v}^{l}_{d_{2}}\}_{l=1}^{L}\) across all \(L\) layers, and fine-tune them to memorize \(x\). We keep the rest of the model parameters frozen. To simulate how LLMs learn data during pretraining, we finetune with the normal language modeling loss on \(x_{i}\) (Eq. 13). To ensure the entire sequence is well memorized, we keep fine-tuning until we reach a loss \(<0.05\); therefore, we can simply set the first token as the prefix \(p\), and the rest of the sequence as the suffix \(s\). Note we sample a different set of weight vectors \(\phi_{i}\) for each sequence \(x_{i}\) and fine-tune a separate model \(\hat{\theta}_{i}\). Algorithm 1 shows the exact injection process.

Evaluation.For each model \(\hat{\theta}_{i}\) injected with a sequence \(x_{i}\), a localization method predicts \(k\%\) of neurons at each layer and we calculate Recall@\(k\%\). Specifically, given the set of ground-truth neurons corresponding to all the injected weight vectors across layers, \(\Gamma_{i}\), and the set of all predicted neurons, \(\hat{\Gamma_{i}}\), the recall is \(\frac{|\Gamma_{i}\cap\Gamma_{i}^{c}|}{|\Gamma_{i}|}\). Finally, we average the recall scores across sequences, and thus average over different choices of weights \(\phi_{i}\) for injection.

### The DEL Benchmark

The DEL Benchmark studies whether we can localize a naturally memorized sequence after pretrain

Figure 1: **Left:** INJ Benchmark updates a small set of LLM weights to store the new piece of data, where the fine-tuned weight vectors and the corresponding neurons are filled with blue. The neurons predicted by a localization method are circled with black. denotes true-positive, false-positive, and false-negative neurons. **Right:** DEL Benchmark drops out the predicted neurons on a memorized pretrained sequence. A large change in Levenshtein distance after dropout indicates that were important for LLM \(f\) to retrieve the memorized sequence.



ing, which is not answered by the INJ Benchmark. We first collect a set of memorized pretrained sequences, and then apply localization methods to identify the responsible neurons for each sequence. Without ground-truth neurons, we adopt knockouts (Li et al., 2016; Olsson et al., 2022; Geva et al., 2023) for evaluation, which measures the importance of model components based on the effect of removing them. We drop out the located neurons to observe how much they account for memorizing a sequence. We quantify memorization with two scores: Accuracy and Levenshtein distance.

AccuracyRecall that a sequence \(x=(p,s)\) consists of a prefix \(p\) and suffix \(s\). Accuracy calculates the percentage of correct suffix tokens generated by teacher-forcing and argmax decoding. Formally,

\[\hat{s_{t}}= \operatorname*{argmax}_{w\in\text{Voc}}~{}P_{\theta}(w|p,s_{< t}),\,t=1,\dots,T \tag{4}\] \[\text{Accuracy}=\frac{1}{T}\sum_{t=1}^{T}\mathds{1}\{\hat{s_{t}} =s_{t}\}, \tag{5}\]

where \(T\) denotes the suffix sequence length, \(s_{t}\) the \(t\)-th true suffix token, \(s_{<t}=[s_{1},\dots,s_{t-1}]\), \(\hat{s_{t}}\) the \(t\)-th generated token, \(P_{\theta}\) the probability distribution of the LLM parameterized by \(\theta\), and Voc the vocabulary. Higher Accuracy indicates better memorization of the sequence.

Levenshtein distanceWhile Accuracy is defined at a token level, we note tokens often contain several characters, e.g., "159". For sequences like "3.14159265", every character is important; thus, we also define a memorization score at the character level. With Eq. 4, we have \(\hat{s}=[\hat{s_{1}},\dots,\hat{s_{T}}]\). We calculate Levenshtein distance between the generated suffix \(\hat{s}\) and the true suffix \(s\). Lower Levenshtein distance indicates better memorization.

Data.We collect a set of sequences memorized by each LLM, including Pythia-deduped-2.8B, Pythia-deduped-6.9B, and GPT2-XL. For Pythia models, the pertaining corpus the Pile-dedupe (Gao et al., 2021) is open-sourced, and we use the following criteria to determine which sequences are memorized. For each candidate sequence \(x\), we set the first \(32\) tokens as the prefix \(p\) to prompt the LLM to reconstruct the suffix \(s\) of \(48\) tokens. First, we filter out sequences with Accuracy (Eq. 4, 5) lower than \(0.9\). Second, we use greedy decoding to generate the suffix, filtering out those with a Levenshtein distance greater than \(20\) characters to the true suffix. Third, we discard sequences with repetitive tokens (less than 16 distinct tokens in the suffix). Finally, we deduplicate the remaining sequences based on n-gram Jaccard index. We obtain 505 memorized sequences for each Pythia model. For GPT2-XL, we do not have access to its pretraining corpus and find very few memorized sequences from several public corpora with our criteria. Thus, we actively search for potentially memorized sequences, discovering 105 memorized sequences and categorizing them manually (Table 1). See A.8 for details and example sequences.

We sample 5 sequences as the dev set to tune the hyperparameters of different methods (see A.9), using the rest of the collected sequences as the test set. We quantify the memorization of LLMs on

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Category** & **Examples** & **Count** \\ \hline Quotes & Churchill, Steve Jobs, Trump & 17 \\ Quotes (Book) & Dune, 1984, Bible & 14 \\ Ordered items & Zodiac Signs, US Presidents & 11 \\ Terms of use & MIT License & 10 \\ Poems & The Second Coming & 9 \\ Code & GitHub & 9 \\ Contact Info & A journalist’s email & 7 \\ URLs & Reddit, file link & 5 \\ Others & long COINBASE ID, meme, Bill of Rights, Pi digits & 23 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Collected sequences memorized by GPT2-XL.

 

the collected test sets. Table 7 in the appendix shows that all LLMs have a high average Accuracy (\(\sim 100\%\)) and a low Levenshtein distance (\(\sim 1\) character) to the true suffix, suggesting that the sequences we collect are indeed well memorized.

Evaluation.When we evaluate one sequence \(x\) in the collected test set \({\cal X}\), we consider the rest of the memorized sequences, \({\cal X}\setminus\{x\}\), as negative examples. A successful localization method should make LLMs forget the target sequence (large changes in memorization scores), but still remember the other negative examples (small changes in memorization scores) after dropping out the predicted \(k\%\) of neurons at each layer.2 We also calculate the absolute change in perplexity on a batch of \(2048\) sequences sampled from the Pile-dedupe, \({\cal D}\), to evaluate whether the general language modeling ability remains intact after dropout.

Footnote 2: We do not drop out neurons in the bottommost layer, as it hurts LLMs’ overall memorization indiscriminately (§5.4).

Despite similarities to the evaluation of model editing Sinitsin et al. (2020); Mitchell et al. (2022), we can better reflect localization success. Unlike Meng et al. (2022) that edit the located weights with gradients, we restrict our operation to neuron dropout. Because dropout has limited freedom in changing LLMs behavior, successful deletion via dropout requires successful localization; in contrast, gradient-based editing could succeed even without good localization Hase et al. (2023).

## 4 Localization Methods

We benchmark five localization methods. Each method assigns an attribution score \({\cal A}^{l}(i)\) to each neuron \(n_{i}^{l}\), the \(i\)-th neuron in the \(l\)-th layer, representing its importance in memorizing a sequence \(x\). At test time, we select the top-\(k\%\) of neurons in each layer for each method in terms of attribution scores as the located neurons for \(x\) by that method.

Several methods involve calculating the language modeling loss of an LLM \(\theta\) on the suffix of the target sequence \(x=(p,s)\). We denote the loss as _memorization loss_, \(\ell_{\theta}^{\text{mem}}(x)\). Formally,

\[\ell_{\theta}^{\text{mem}}(x)=\frac{1}{T}\sum_{t=1}^{T}-\log P_{\theta}(s_{ t}|p,s_{<t}) \tag{6}\]

Zero-Out.We introduce an exhaustive method that drops out neurons one by one and uses the resulting change in memorization loss on \(x\) as the attribution score to each neuron \(n_{i}^{l}\):

\[{\cal A}^{l}(i)=\ell_{\theta\backslash n_{i}^{l}}^{\text{mem}}(x)-\ell_{ \theta}^{\text{mem}}(x) \tag{7}\]

We denote \(\ell_{\theta\backslash n_{i}^{l}}^{\text{mem}}\) as the memorization loss of the LLM \(\theta\) after dropping out a neuron \(n_{i}^{l}\). The larger the change in the loss, the more important the neuron is for memorization. Zero-Out is closely related to the occlusion-based attribution method Zeiler and Fergus (2014).

Activations.We can view the neuron activation \(h_{i}^{l}\) as the memory coefficients (SS2). Thus, similar to Geva et al. (2022), we set the attribution \({\cal A}^{l}(i)\) as the absolute value of \(h_{i}^{l}\) multiplied by the vector norm of \(\mathbf{v}_{i}^{l}\), averaged across the suffix length \(T\):

\[{\cal A}^{l}(i)=\frac{1}{T}\sum_{t=1}^{T}|h_{i,t}^{l}|\,||\mathbf{v}_{i}^{l}|, \tag{8}\]

where \(h_{i,t}^{l}\) denotes the activation value at the \(t\)-th timestep, when the input consists of all the tokens before \(s_{t}\), i.e., \([p,s_{<t}]\).

Integrated Gradients (IG).We benchmark integrated gradients Sundararajan et al. (2017), an attribution method that has been used to identify knowledge neurons and privacy neurons Dai et al. (2022); Wu et al. (2023). IG cumulates the gradients at all points along the path from a zero vector to the original hidden state \(h^{l}\). See A.2 for more details.

Slimming.We introduce Slimming, a localization method adapted from prior work Liu et al. (2017); Chen et al. (2021) on network pruning. Pruning aims to reduce the model size by finding a subnetwork that can achieve a low loss on the task, e.g., sentiment analysis. In our setting, we find a small set of neurons that are crucial for maintaining a low memorization loss \(\ell_{\theta}^{\text{mem}}(x)\) on _one_ target sequence \(x\) (Eq. 6). Specifically, Slimming minimizes the memorization loss while learning a sparse mask \(m^{l}\in\mathbb{R}^{d_{2}}\) on the hidden state \(h^{l}\) in every layer, with mask value \(m_{i}^{l}\) on neuron \(n_{i}^{l}\). At each layer, we transform \(h^{l}\) to \(h^{l}\odot m^{l}\) before computing further layers, where \(\odot\) denotes elementwise multiplication. The sparse mask encourages the LLM to use only a small set of neurons to recall a piece of memory. All the weights of the LLM are kept frozen during the training; only the mask \(m^{l}\) is learnable. Formally,

\[\min_{\begin{subarray}{c}m^{l}\\ l=1,...,L\end{subarray}}\ell_{\theta}^{\text{mem}}(x)+\lambda\sum_{l=1}^{L} \|m^{l}\|_{1},\] (9) 

where \(\lambda\) is the hyperparameter to balance the memorization loss and the \(L_{1}\) sparsity regularization on the mask. After training, we set the attribution score \(\mathcal{A}^{l}(i)=m_{i}^{l}\), as \(m_{i}^{l}\) learns the importance of the existence of a neuron to the memorization loss.

Hard Concrete.The limitation of Slimming is that it tends to assign mask values \(m_{i}^{l}\) between \(0\) and \(1\) on most neurons, creating a mismatch between training and testing. In particular, at inference time we either activate (equivalent to setting \(m_{i}^{l}=1\)) or drop out (\(m_{i}^{l}=0\)) a neuron. Thus, we adapt another pruning method Hard Concrete Louizos et al. (2018); Zheng et al. (2022) for localization, which improves over Slimming by encouraging mask values \(m_{i}^{l}\) to be approximately binary. Similar to Slimming, Hard Concrete learns parameters \(m^{l}\in\mathbb{R}^{d_{2}}\) in every layer. But instead of directly using \(m^{l}\) as the mask, the mask \(\bar{m}^{l}\) in Hard Concrete is a random variable (r.v.) that depends on \(m^{l}\). Specifically, Hard Concrete derives the mask value \(\bar{m}_{i}^{l}\) from a binary concrete Maddison et al. (2017); Jang et al. (2017) random variable, \(\hat{m}_{i}^{l}\). A binary concrete distribution \(\hat{m}_{i}^{l}\sim\text{Concrete}(m_{i}^{l},\beta)\) is parameterized by the location \(m_{i}^{l}\) and temperature \(\beta\). When the hyperparameter \(\beta\to 0\), sampling from the binary concrete distribution is identical to sampling from a Bernoulli distribution but loses the differentiable property. With \(\beta>0\), we allow gradient-based optimization of parameter \(m_{i}^{l}\) through the reparametrization trick. Formally,

\[u_{i} \sim\mathcal{U}\left(0,1\right), \tag{10}\] \[\hat{m}_{i}^{l} =\sigma\left(\frac{1}{\beta}(\log\frac{u_{i}}{1-u_{i}}+\log m_{ i}^{l})\right), \tag{11}\]

where \(\sigma\) denotes the sigmoid function and \(u_{i}\) is a r.v. sampled from uniform distribution \(\mathcal{U}\left(0,1\right)\). We describe the details about how Louizos et al. (2018) extend a hard concrete r.v. \(\bar{m}^{l}\) from the binary concrete r.v. \(\hat{m}_{i}^{l}\) and use \(L_{0}\) regularization \(\mathcal{R}(\bar{m}^{l})\) to encourage sparsity in A.4.

To learn the parameters \(m^{l}\), we freeze the LLM weights \(\theta\) and simultaneously optimize the memorization loss on the target sequence \(x\) and the sparsity loss \(\mathcal{R}(\bar{m}^{l})\). Formally,

\[\min_{\begin{subarray}{c}m^{l}\\ l=1,...,L\end{subarray}}~{}\ell_{\theta}^{\text{mem}}(x)+\lambda\sum_{l=1}^{ L}\mathcal{R}(\bar{m}^{l}) \tag{12}\]

At test time, \(\hat{m}_{i}^{l}\) can be estimated as \(\sigma\left(\log m_{i}^{l}\right)\)Louizos et al. (2018); thus, we set the attribution score \(\mathcal{A}^{l}(i)=\sigma\left(\log m_{i}^{l}\right)\).

## 5 Experiments

### INJ Benchmark Results

Table 2 shows the average Recall@\(k\%\) and standard errors of different localization methods on four LLMs under our INJ Benchmark evaluation. When the injection ratio is \(1\%\) (Table 2; Top), there are \(1\%\) of weight vectors injected with each new sequence, yielding \(1\%\) of ground truth neurons, and every method predicts \(k=\{1,2,5\}\%\) of neurons at each layer. When the injection ratio is \(0.1\%\) (Table 2; Bottom), every method predicts \(\{0.1,0.2,0.5\}\%\) of neurons at each layer. We also study the alternative that predicts top-\(k\) neurons _across_ layers in A.11, which shows results consistent with Table 2 but with lower recall overall.

All methods can do localization.First, all five localization methods greatly outperform Random,

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline  & \multicolumn{3}{c}{**GPT2 12AM**} & \multicolumn{3}{c}{**OPT2-XL 1.5B**} & \multicolumn{3}{c}{**Pythia-deduped 2.8B**} & \multicolumn{3}{c}{**Pythia-deduped 6.9B**} \\ \cline{2-13}  & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% \\ \hline \multirow{2}{*}{ratio = 1\%} & \multirow{2}{*}{ratio = 1\%} & \multirow{2}{*}{ratio = 7\%} & \multirow{2}{*}{Ratio = 7\%} & \multirow{2}{*}{Ratio = 8\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} \\  & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@9\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% \\ \hline \multirow{2}{*}{ratio = 1\%} & \multirow{2}{*}{ratio = 7\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} \\  & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% \\ \hline \multirow{2}{*}{ratio = 0.1\%} & \multirow{2}{*}{Ratio = 1\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} \\  & Random & \(1.0_{0.04}\) & \(2.1_{0.06}\) & \(5.0_{0.09}\) & \(1.0_{0.01}\) & \(2.0_{0.02}\) & \(5.0_{0.03}\) & \(1.0_{0.01}\) & \(2.0_{0.02}\) & \(5.0_{0.03}\) & \(1.0_{0.01}\) & \(2.0_{0.02}\) & \(5.0_{0.02}\) \\ \hline \multirow{2}{*}{ratio = 0.1\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 9\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} & \multirow{2}{*}{Ratio = 10\%} \\  & & & & & & & & & & & & & \\ \hline \multirow{2}{*}{ratio = 0.1\%} & \multirow{2}{*}

 

which randomly predicts the same number of neurons at each layer. Interestingly, when the injection ratio is lower (\(0.1\%\)), all localization methods achieve higher recall, possibly because the information is more concentrated in the injected weights and thus easier to identify.

**Pruning-based methods perform the best.**Slimming and Hard Concrete, the methods based on network pruning, substantially outperform the other methods across all setups. Specifically, Hard Concrete achieves Recall@\(0.5\%\) higher than \(80\) in three out of four LLMs. ZeroOut and IG perform similarly and outperform the simple method Activations overall, but are much more computationally expensive than the other methods (see comparisons in A.7).

**Our results hold under more data and different random seeds.** In the appendix, we show that our conclusions hold when expanding the INJ Benchmark to the newly released ECBD 2022, 2023 dataset [4] (A.5), and they are robust to the choice of random seed, which controls the choice of injected weights (A.6).

### DEL Benchmark Results

Table 3 shows to what extent dropping out \(k=\{0.1,0.5\}\%\) of neurons predicted by different methods makes LLMs forget the target sequence \(x\) (Self), while still memorizing the other sequences \(\mathcal{X}\setminus\{x\}\) (Neg), and keeping the perplexity on the random batch \(\mathcal{D}\) (Rand-PPL) intact. We evaluate one target sequence at a time and report the average absolute changes (\(\Delta\)) in Accuracy (Acc), Levenshtein distance (Dist), and perplexity after dropout.

**All methods show evidence of localization.** Randomly dropping out the same number of neurons (Random) barely changes the LLM behavior. In comparison, all five localization methods successfully identify neurons that contribute much more to memorizing the target sequence than to negative examples, showing evidence of their localization ability on real-world memorized data.

**Methods trade off between \(\Delta\) Self and \(\Delta\) Neg.** We find Slimming and Hard Concrete much more effective than other methods in erasing the target sequence itself. However, they are worse at preserving LLM memorization of the negative examples and the perplexity of randomly sampled sequences. For example, dropping out \(0.5\%\) of GPT2 neurons predicted by Slimming decreases Accuracy by \(57.8\%\) and increases \(75.4\) characters in Levenshtein distance on the target sequence, but it also hurts the Accuracy on negative examples by \(6.4\%\) and increases Levenshtein distance by \(7.5\) on average. On the other hand, IG best maintains

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{\(\Delta\) Self-Acc \(\downarrow\)} & \multicolumn{2}{c}{\(\Delta\) Self-Dist \(\uparrow\)} & \multicolumn{2}{c}{\(\Delta\) Neg-Acc \(\uparrow\)} & \multicolumn{2}{c}{\(\Delta\) Neg-Dist \(\downarrow\)} & \multicolumn{2}{c}{\(\Delta\) Rand-PPL \(\downarrow\)} \\ \cline{2-9} _dropout ratio_ = & 0.1\% & 0.5\% & 0.1\% & 0.5\% & 0.1\% & 0.5\% & 0.1\% & 0.5\% & 0.1\% & 0.5\% \\ \hline \multicolumn{9}{l}{**GPT2-XL 1.5B**} \\ Hard Concrete & **-34.6\%** & -57.1\% & **42.9** & 74.0 & -2.4\% & -4.8\% & 2.5 & 5.4 & 0.03 & 0.11 \\ Slimming & -30.5\% & **-57.8\%** & 37.7 & **75.4** & -3.5\% & -6.4\% & 4.1 & 7.5 & 0.02 & 0.17 \\ Zero-Out & -29.8\% & -46.1\% & 33.0 & 55.2 & -3.1\% & -4.8\% & 3.5 & 5.5 & 0.03 & 0.09 \\ IG & -25.8\% & -40.8\% & 27.0 & 46.0 & **-2.2\%** & **-3.4\%** & **2.3** & **3.7** & **0.01** & **0.05** \\ Activations & -14.8\% & -29.5\% & 16.9 & 36.4 & -3.0\% & -4.7\% & 3.1 & 5.4 & 0.11 & 0.16 \\ Random & -0.2\% & -0.5\% & 0.2 & 0.4 & -0.2\% & -0.5\% & 0.1 & 0.4 & 0.00 & 0.03 \\ \hline \multicolumn{9}{l}{**Pythia-dedupedep 2.8B**} \\ Hard Concrete & **-29.0\%** & **-53.2\%** & **55.3** & **99.8** & -3.7\% & -10.5\% & 7.7 & 22.1 & 0.23 & 0.56 \\ Slimming & -17.4\% & -45.1\% & 32.9 & 80.8 & -3.3\% & -7.0\% & 6.6 & 13.9 & 0.26 & 0.49 \\ Zero-Out & -14.8\% & -25.9\% & 26.4 & 45.2 & -1.1\% & -2.5\% & 2.1 & 5.0 & 0.21 & 0.35 \\ IG & -16.7\% & -30.3\% & 29.1 & 52.5 & **-0.9\%** & **-2.1\%** & **1.8** & **4.4** & **0.09** & **0.18** \\ Activations & -13.0\% & -25.5\% & 27.5 & 52.2 & -3.1\% & -6.1\% & 6.6 & 12.9 & 0.11 & 0.20 \\ Random & -0.1\% & -0.3\% & 0.1 & 0.5 & -0.1\% & -0.3\% & 0.2 & 0.5 & 0.00 & 0.02 \\ \hline \multicolumn{9}{l}{**Pythia-dedupedep 6.9B**} \\ Hard Concrete & **-29.2\%** & **-57.7\%** & **58.5** & **109.9** & -3.8\% & -14.7\% & 8.7 & 32.6 & 0.16 & 0.52 \\ Slimming & -24.1\% & -48.7\% & 48.8 & 92.1 & -4.2\% & -11.3\% & 9.1 & 23.6 & 0.23 & 0.58 \\ IG & -16.9\% & -32.3\% & 31.4 & 57.8 & **-2.3\%** & **-4.9\%** & **5.3** & **11.5** & 0.27 & **0.37** \\ Activations & -11.5\% & -26.8\% & 25.5 & 51.5 & -2.5\% & -8.1\% & 5.5 & 17.2 & **0.12** & 0.45 \\ Random & -0.1\% & -0.2\% & 0.1 & 0.4 & -0.1\% & -0.2\% & 0.1 & 0.3 & 0.00 & 0.02 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The DEL Benchmark. Hard Concrete is the most effective method in erasing the target sequence (Self), while IG can best maintain the LLM performance on unrelated sequences (Neg and Rand) after dropout.

 

the performance on negative examples and perplexity, but is not as successful in erasing the target sequence itself. Interestingly, although Zero-Out assigns the attribution scores with a leave-one-out approach, it does not perform the best on either target sequences or negative examples, implying that the individual neuron dropout effect does not reliably predict the collective effect of dropping out many neurons at the same time. Overall, it is challenging for methods to effectively and specifically locate the target sequence at the same time.

Which negative examples are forgotten?We analyze how the negative examples affected by dropout are related to the target sequence. Figure 2 is the confusion matrix on a representative subset of GPT2 memorized data, \({\cal Y}\subset{\cal X}\), where each row shows how dropping out \(0.5\%\) of the neurons predicted by Hard Concrete on a target sequence changes the Accuracy of every sequence in \({\cal Y}\). We group sequences under the same category (see Table 1) in adjacent rows. We find Hard Concrete sometimes confuses related data; for example, in the Address category consisting of mailing addresses, dropping out the neurons of an address sequence also causes substantial Accuracy drops on other addresses. We also find confusion across the Poems, Shakespeare, and Bible categories of literary sequences. Qualitatively, we found several web pages containing famous quotes from different poems and books; such co-occurrences may also appear multiple times in GPT2's pretraining corpus and may explain why in Figure 2, a small set of neurons affect quotes from different sources. While these findings could suggest that Hard Concrete struggles to pinpoint neurons that are specific to a target sequence, it may also be that LLMs actually use a shared set of neurons to memorize several related sequences. Figure 5 in A.8 shows the confusion matrices of other methods and Figure 6 is the matrix of the entire dataset \({\cal X}\). Both figures share patterns similar to Figure 2.

### Concurrence of the two benchmarks

This section studies if the two benchmarks rank the methods similarly Liu et al. (2023) and whether the differences between methods are significant.

Rankings of localization methodsThe INJ Benchmark, which solely evaluates the injected target sequences,3 and the Self- part of the DEL Benchmark show consistent rankings: Hard Concrete performs slightly better than Slimming, followed by Zero-Out and IG; Activations performs the worst but still substantially outperforms Random. This consistency suggests that despite the differences in data and setups, the two benchmarks reflect the same underlying localization abilities of different methods. We believe the reason pruning-based methods perform better is that they learn to mask multiple neurons simultaneously, while other methods only consider the importance of each neuron individually.

Footnote 3: INAJ Benchmark does not have negative examples, since we do not have ground-truth neurons of pretrained sequences.

Tests of significance.We run t-tests to test if pruning-based methods outperform IG significantly. For the INJ Benchmark, each method has 24 Recall@\(k\%\) scores in Table 2; we run 24 one-tailed paired t-tests accordingly. With Bonferroni correction, we set the significance level \(\alpha=\frac{0.05}{24}\). Table 10 in the appendix shows that for Hard Concrete vs. IG and Slimming vs. IG, respectively, there are 23/24 and 24/24 tests that have p-values \(<\alpha\). Similarly, in the DEL Benchmark, each method has 6 \(\Delta\) Self-Acc scores in Table 3; thus, we run 6 paired t-tests. Table 11 shows that 5/6 and 6/6 tests have p-values \(<\frac{0.05}{6}\), for Slimming vs. IG and Hard Concrete vs. IG, respectively. Notably, for both benchmarks, most tests have p-values \(<10^{-10}\). Overall, these results support our claims that the difference between pruning-based methods and IG is significant.

Figure 2: The confusion matrix of Hard Concrete on a subset of data memorized by GPT2-XL.

 



### Is the memory of a piece of information distributed over layers?

To understand the individual effect of each layer on memorization, we study the alternative that drops out the same number of neurons in a _single_ layer. In SS5.2, a method predicts top-\(0.1\%\) of neurons in _every_ layer after the bottommost layer; thus, we have a "budget" of \(N=0.1\%\times 6400\times(48-1)\) neurons for GPT2-XL. Here, the alternative strategy drops out the top-\(N\) neurons in a single layer in terms of the attribution scores assigned by a method.

Using the attribution assigned by IG, Figure 3 compares the two dropout strategies, illustrating their \(\Delta\)Self-Acc and \(\Delta\)Neg-Acc scores (see more methods in A.10). First, we find dropping out neurons in multiple layers much more efficient in erasing the target sequence, as the horizontal blue line shows a greater decrease in Self-Acc than the solid blue line, suggesting that memorized information is stored in a distributed fashion over many layers, not concentrated in a single layer. The only exception is dropping out neurons in Layer 1; however, it also greatly hurts Neg-Acc. The large memorization decreases on all sequences may imply that the bottom layers of LLMs mainly work on processing basic and general information Tenney et al. (2019), instead of focusing on a specific sequence.

## 6 Related Work and Discussion

Localization identifies function-specific components, including neurons Radford et al. (2017); Gurnee et al. (2023), layers Gupta et al. (2023), or subnetworks Csordas et al. (2021); Cao et al. (2021); Foroutan et al. (2022). For example, Dai et al. (2022) find knowledge neurons for each relational fact. Meng et al. (2022) locate relational facts to middle FFNs, specifically when LLMs process the last token of the subject. Bayazit et al. (2023) discover sparse knowledge subnetworks in GPT2 with a differentiable weight masking method. However, there is no standard approach to evaluate the effectiveness of localization methods. We are the first to systematically and directly compare different methods on LLMs of different sizes, including knowledge neurons (IG) and differentiable masking methods Slimming and Hard Concrete.

We take the view that LLM memorization of a sequence is different from learning a type of knowledge. Memorization is reproducing a long sequence (near) verbatim. In contrast, knowledge, often represented as a <subject, relation, object> triplet, occurs in variable contexts, where paraphrases are treated as equivalent expressions of the same knowledge. Localization of verbatim memorization helps unlearn private or copyrighted data, for example, Wu et al. (2023) apply IG to localize and then erase private data from a BERT fine-tuned on Enron Email dataset Klimt and Yang (2004). Our DEL Benchmark differs from Wu et al. (2023) in two main ways: (1) we delete sequences that LLMs have naturally memorized during pretraining, (2) we locate neurons for each sequence independently, rather than finding a shared set of neurons, as our collected datasets cover diverse sequences. Localization can also prevent overfitting: Maini et al. (2023) drop out pre-allocated neurons tied to memorizing mislabeled examples. In contrast with these works, we focus on benchmarking localization ability, since successful localization is the basis of its downstream applications.

## 7 Conclusion

We propose two benchmarking approaches to define the success of LLM localization, focusing on locating a small set of neurons in an LLM that are responsible for memorizing a sequence. The INJ Benchmark enables a direct evaluation of localization methods, while the DEL Benchmark evaluates methods on naturally memorized sequences, using dropout to measure localization success. The two benchmarks complement each other and show consistent rankings of methods. We find promising localization ability of all five methods we evaluate, especially for Hard Concrete. Meanwhile, all methods confuse memorized sequences in the same or related categories. This finding suggests a need for better localization methods and poses the open question of whether LLMs use a shared set of neurons to memorize related sequences such that perfect localization is not possible.

Figure 3: Dropout in one layer vs. multiple layers.





## 8 Limitations

We follow prior work (SS2) and assume that FFNs are the most important components in LLMs for memorizing data; thus, we only study localization in FFNs, not considering other model components such as attention layers. Similarly, we focus on neurons instead of individual weights in FFNs, so as to make fair comparisons with existing methods, IG and Activations.

In the INJ Benchmark, we assume that all the fine-tuned weights are responsible for memorizing the newly injected sequence. However, there is no guarantee that all of them contribute to memorization. We roughly address this issue by lowering the injection ratio, which makes it less likely for the model to memorize the injected sequence without using all of the chosen weights; indeed, we observe that when the ratio is \(10\times\) smaller, all localization methods achieve higher recalls in Table 2, even though the random baseline performs \(10\times\) worse.

We acknowledge the limitations of evaluating localization in our DEL Benchmark. First, we use dropout (namely, zero ablation) to observe the importance of the located neurons, which is only one possible way to approach localization; other approaches such as mean ablation Wang et al. (2023) and path patching Goldowsky-Dill et al. (2023); Hanna et al. (2023) are not covered in this paper. Besides, given a target sequence, we treat all the other memorized sequences as its negative examples without considering semantic overlap or data sources, as our data deduplication only ensures there is little lexical overlap between sequences (SS3.2). However, we find all localization methods show confusion between several quotes, which may share semantic similarities or co-occur in some pre-trained documents. It is debatable whether related examples should be considered negative, and it depends on what the goal of localization is. We invite future work to propose new ways to define the success of localization for the DEL Benchmark.

## Acknowledgements

We thank Johnny Wei, Eric Wallace, and Ameya Godbole for their help in finding memorized data. We thank Cheng-Han Chiang for helpful discussions on model editing. We thank Wang Zhu, Ting-Rui Chiang, Joshua Robinson, Lee Kezar, Deqing Fu, Anthony Liang, Ta-Chung Chi, Yi-Lin Tuan, Yau-shian Wang, and the anonymous reviewers for their valuable feedback. We thank USC NLP cluster admin for their great work on keeping the servers functional. This work was funded in part by gifts from Open Philanthropy and Cisco Research.

## References

* D. Bayazit, N. Foroutan, Z. Chen, G. Weiss, and A. Bosselt (2023)Discovering knowledge-critical subnetworks in pretrained language models. ArXiv preprintabs/2310.03084. Cited by: SS5.
* S. Biderman, H. Schoelkopf, Q. Anthony, I. Bradley, K. O'Brien, E. Hallahan, M. Afah Khan, B. Purohit, U. Sai Prashanth, E. Raff, A. Skowron, L. Sutawika, and Oskar van der Wal. (2023)Pythia: a suite for analyzing large language models across training and scaling. External Links: Link Cited by: SS6.
* L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot (2021)Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. Cited by: SS1.
* S. Cao, V. Sanh, and A. Rush (2021)Low-complexity probing via finding subnetworks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 960-966. External Links: Link Cited by: SS1.
* Y. Cao and J. Yang (2015)Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. Cited by: SS1.
* N. Carlini, C. Liu, U. Erlingsson, J. Kos, and D. Song (2019)The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284. Cited by: SS1.
* N. Carlini, F. Tramer, E. Wallace, M. Jagileski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. (2021)Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650. Cited by: SS1.
* X. Chen, Y. Cheng, S. Wang, Z. Gan, Z. Wang, and J. Liu (2021)EarlyBERT: efficient BERT training via early-bird lottery tickets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 2195-2207. External Links: Link Cited by: SS1.
* R. Csordas, S. van Steenkiste, and J. Schmidhuber (2021)Are neural nets modular? inspecting functional modularity through differentiable

weight masks. In _International Conference on Learning Representations_.
* Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8493-8502, Dublin, Ireland. Association for Computational Linguistics.
* Foroutan et al. (2022) Negar Foroutan, Mohammadreza Banaei, Remi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discovering language-neutral sub-networks in multilingual language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7560-7575, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2021. The pile: An 800gb dataset of diverse text for language modeling. _ArXiv preprint_, abs/2101.00027.
* Geva et al. (2023) Mor Geva, Jasmin Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12216-12235, Singapore. Association for Computational Linguistics.
* Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Goldowsky-Dill et al. (2023) Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Eryaman Arora. 2023. Localizing model behavior with path patching. _ArXiv preprint_, abs/2304.05969.
* Gong et al. (2022) Zhuocheng Gong, Di He, Yelong Shen, Tie-Yan Liu, Weizhu Chen, Dongyan Zhao, Ji-Rong Wen, and Rui Yan. 2022. Finding the dominant winning ticket in pre-trained language models. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1459-1472, Dublin, Ireland. Association for Computational Linguistics.
* Gupta et al. (2023) Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegrefle, and Niket Tandon. 2023. Editing common sense in transformers. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 8214-8232, Singapore. Association for Computational Linguistics.
* Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. _Transactions on Machine Learning Research_.
* Han et al. (2016) Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In _International Conference on Learning Representations_.
* Hanna et al. (2023) Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Hase et al. (2023) Peter Hase, Mohit Bansal, Been Kim, and Asma Ghanehauri. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Hassibi and Stork (1992) Babak Hassibi and David Stork. 1992. Second order derivatives for network pruning: Optimal brain surgeon. In _Advances in Neural Information Processing Systems_, volume 5. Morgan-Kaufmann.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In _International Conference on Learning Representations_.
* Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004. Introducing the enron corpus. In _CEAS_, volume 45, pages 92-96.
* Lee et al. (2023) Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In _Proceedings of the ACM Web Conference 2023_, pages 3637-3647.
* Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
* Lehman et al. (2021) Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT pre-trained on clinical notes reveal sensitive data? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 946-959, Online. Association for Computational Linguistics 

Vladimir I. Levenshtein (1965)Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, pp. 707-710. Cited by: SS2.
* J. Li, W. Monroe, and D. Jurafsky (2016)Understanding neural networks through representation erasure. ArXiv preprintabs/1612.08220. External Links: Link, 1612.08220 Cited by: SS4.3.
* N. F. Liu, T. Lee, R. Jia, and P. Liang (2023)Do question answering modeling improvements hold across benchmarks?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 13186-13218. External Links: Link, Document Cited by: SS2.
* Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang (2017)Learning efficient convolutional networks through network slimming. In 2017 IEEE International Conference on Computer Vision (ICCV), Vol. , pp. 2755-2763. External Links: Link, Document Cited by: SS2.
* C. Louizos, M. Welling, and D. P. Kingma (2018)Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations, Cited by: SS2.
* C. J. Maddison, A. Mnih, and Y. V. Teh (2017)The concrete distribution: a continuous relaxation of discrete random variables. In International Conference on Learning Representations, Cited by: SS2.
* P. Maini, M. Curtis Mozer, H. Sedghi, Z. Chase Lipton, J. Kolter, and C. Zhang (2023)Can neural network memorization be localized?. In Proceedings of the 40th International Conference on Machine Learning, Cited by: SS2.
* K. Meng, D. Bau, A. J. Andonian, and Y. Belinkov (2022)Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, Cited by: SS2.
* E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning (2022)Fast model editing at scale. In International Conference on Learning Representations, Cited by: SS2.
* C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conperly, D. Drain, Deep Ganguli, Z. Hatfield-Dodds, Danny Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah (2022)InContext learning and induction heads. Transformer Circuits Thread. Cited by: SS2.
* Y. Onoe, M. Zhang, E. Choi, and G. Durrett (2022)Entity cloze by date: what LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, United States, pp. 693-702. External Links: Link, Document Cited by: SS2.
* S. Padmanabhan, Y. Onoe, M. Zhang, G. Durrett, and E. Choi (2023)Propagating knowledge updates to lms through distillation. In Advances in Neural Information Processing Systems, Vol. 36, pp. 47124-47142. External Links: Link, Document Cited by: SS2.
* A. Panigrahi, N. Saunshi, H. Zhao, and S. Arora (2023)Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pp. 202 of _Proceedings of Machine Learning Research_, pages 27011-27033. External Links: Link, Document Cited by: SS2.
* A. Radford, R. Jozefowicz, and I. Sutskever (2017)Learning to generate reviews and discovering sentiment. ArXiv preprintabs/1704.01444. External Links: Link, 1704.01444 Cited by: SS2.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog.. Cited by: SS2.
* A. Sinitsin, V. Plokhotnyuk, D. Pyrkin, S. Popov, and A. Babenko (2020)Editable neural networks. In International Conference on Learning Representations, Cited by: SS2.
* N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014)Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research15 (56), pp. 1929-1958. External Links: Link, ISSN 1574-469X, Link, Document Cited by: SS2.
* M. Sundararajan, A. Tayb, and Q. Yan (2017)Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 70 of _Proceedings of Machine Learning Research_, pp. 3319-3328. External Links: Link, ISBN 978145036658, Link, Document Cited by: SS2.
* I. Tenney, D. Das, and E. Pavlick (2019)BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 4593-4601. External Links: Link, Document Cited by: SS2.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, pp. . Cited by: SS2.
* K. R. Wang, A. Variengien, A. Convmy, B. Shlegeris, and J. Steinhardt (2023)Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, Cited by: SS2.
* X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong (2023)DEPN: detecting and editing privacy neurons in pretrained language models. In Proceedings of the 2023 

Conference on Empirical Methods in Natural Language Processing_, pages 2875-2886, Singapore. Association for Computational Linguistics.
* Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In _Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_, pages 818-833. Springer.
* Zhang et al. (2023) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. 2023. Counterfactual memorization in neural language models. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Zheng et al. (2022) Rui Zheng, Bao Rong, Yuhao Zhou, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2022. Robust lottery tickets for pre-trained language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2211-2224, Dublin, Ireland. Association for Computational Linguistics.

 

Appendix

### The Loss for Information Injection

In the INJ Benchmark, we use regular language modeling loss to train the LLM \(\theta\) on a new sequence \(x=[x_{1},\dots,x_{\mathcal{T}}]\) of \(\mathcal{T}\) tokens. Formally,

\[\frac{1}{\mathcal{T}-1}\sum_{t=2}^{\mathcal{T}}-\log P_{\theta}(x_{ t}|x_{<t}) \tag{13}\]

Here, the index \(t\) starts from \(2\), because all the LLMs we use (GPT2 and Pythia models) do not add <bos> tokens to data when doing language modeling in their pretraining. Therefore, there is no loss on the first token \(x_{1}\) and the total loss is averaged across \(\mathcal{T}-1\) token.

### Details of IG

Recall that a sequence \(x=(p,s)\) consists of a prefix \(p\) and a suffix \(s=[s_{1},\dots,s_{T}]\). Denote \(\mathrm{P}(\hat{h}_{t}^{l})\) as the LLM output probability of token \(s_{t}\) if we replace the original hidden state at the \(l\)-th layer, \(h_{t}^{l}\in\mathbb{R}^{d_{2}}\), with a new hidden state \(\hat{h}_{t}^{l}\in\mathbb{R}^{d_{2}}\):

\[\mathrm{P}(\hat{h}_{t}^{l})=P_{\theta}(s_{t}|p,s_{<t},\hat{h}_{t}^{ l}) \tag{14}\]

To calculate the integrated gradients along the \(i\)-th neuron dimension, we gradually change \(\hat{h}_{t}^{l}\) from a zero vector4 to the original hidden state \(h_{t}^{l}\), and cumulating the gradients of \(\mathrm{P}(\cdot)\) along the \(i\)-th dimension. Finally, we get the attribution score \(\mathcal{A}^{l}(i)\) by averaging the integrated gradients across the suffix length \(T\):

Footnote 4: We follow Dai et al. (2022) to set the _baseline_ in integrated gradients to a zero vector that has the same shape as \(h_{t}^{l}\).

\[\mathbf{IG}_{i}(z) := z_{i}\int_{\alpha=0}^{1}\frac{\partial\mathrm{P}(\alpha z)}{ \partial z_{i}}d\alpha, \tag{15}\] \[\mathcal{A}^{l}(i) = \frac{1}{T}\sum_{t=1}^{T}\mathbf{IG}_{i}(h_{t}^{l}) \tag{16}\]

where \(\mathbf{IG}_{i}(h_{t}^{l})\) is the integrated gradients along the \(i\)-th neuron dimension in the \(l\)-th layer at the \(t\)-th timestep, when the input is \([p,s_{<t}]\). Sundararajan et al. (2017) compute Riemann sum to approximate Eq. 15, which uses a fixed number of intervals to approximate the integrals. We closely follow the implementation of [https://github.com/EleutherAI/knowledge-neurons](https://github.com/EleutherAI/knowledge-neurons).

### Details of SLimming

We initialize every mask value \(m_{i}^{l}\) as \(1\), which is equivalent to running the pretrained LLM without masking. When training the mask, we clip every \(m_{i}^{l}\) to \([0,1]\). Note that for both Slimming and Hard Concrete, because we are learning a mask on each neuron, we do not apply any random dropout during training.

### Details of Hard Concrete

Louizos et al. (2018) obtain the hard concrete r.v. \(\bar{m}_{i}^{l}\) by first stretching the binary concrete r.v. \(\hat{m}_{i}^{l}\) (Eq. 11) from the interval \((0,1)\) to \((\gamma,\zeta)\), where \(\gamma=-0.1,\zeta=1.1\), and then clip the value to the \([0,1]\) interval:

\[\bar{m}_{i}^{l}=\min\left(1,\max\Big{(}0,\,\hat{m}_{i}^{l}\,\cdot(\zeta-\gamma )+\gamma\Big{)}\right)\]

They then use \(L_{0}\) regularization to encourage sparsity on the weights after applying the mask \(\bar{m}^{l}\). After reparametrization, they have the regularization \(\mathcal{R}(\bar{m}^{l})\):

\[\mathcal{R}(\bar{m}^{l})\;=\sum_{i=1}^{d_{2}}\sigma\left(\log m_{ i}^{l}-C\right), \tag{17}\]

where \(C=\beta\log\frac{-\gamma}{\zeta}\) is a constant.

### Expanding the dataset of INJ Benchmark

We double the data size of the INJ Benchmark by including the newly released ECBD 2022, 2023 splits, having 328 distinct definition sentences from ECBD 2021-2023. We experiment with this expanded dataset on GPT2, injection ratio=\(0.1\%\), using the same hyperparameters as Table 2. Table 4 shows the results on ECBD 2021-2023 are very close to the ones on ECBD-2021 only (Table 2), suggesting that our conclusions hold when we increase the dataset size.

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**GPT2 124M**} \\ \cline{2-4} _ratio = 0.1\%_ & **R@0.1\%** & **R@0.2\%** & **R@0.5\%** \\ \cline{2-4} HC & \(58.3\) & \(81.6\) & **94.6** \\ Slim & **59.2** & **84.3** & \(94.5\) \\ IG & \(53.3\) & \(73.0\) & \(84.1\) \\ Activation & \(11.1\) & \(26.5\) & \(52.7\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: The INJ Benchmark results of GPT2 on the expanded dataset, ECBD 2021-2023.

 



### Do random seeds affect the results of the INJ Benchmark?

In INJ Benchmark, we sample different sets of weights for different examples (see Algorithm 1); thus, the results reported in Table 2 are averaged over many different choices of weights. To further show that random seeds do not affect our results, we run an additional experiment on GPT2, with the injection ratio=\(0.1\%\). Specifically, for each example, we choose a different random seed and thus choose a different set of weights to inject the example. Comparing the new results in Table 5 with the original ones in Table 2, we find that the recall scores barely change for all localization methods. Also, for each method, we run paired two-tailed t-tests comparing the recalls between the original and new seeds and observe that all pairs have p-values \(>0.05\), suggesting that differences between random seeds are not significant.

### Computation costs of different methods

Among all five localization methods, Activations is the most computationally efficient, because Eq. 8 only requires one forward pass. Both the pruning-based methods Slimming and Hard Concrete perform fast, as only the masks are trainable. Calculating integrated gradients (IG) is time-consuming, while Zero-Out is the worst, because it leaves out every neuron one by one. We compare the computational cost of different methods on one sequence memorized by Pythia-deduped-6.9B, where each sequence in the collected set \({\cal X}\) consists of a 32-token prefix and a 48-token suffix. We follow the common implementation that sets the number of intervals to \(20\) for Riemann sum in IG. Table 6 shows the elapsed time calculated on an RTX A6000 48G GPU. When running IG and Zero-Out we patch and batch the activations to reach \(99\%\) GPU utilities. Still, applying Zero-Out to do localization on one sequence costs 8.5 hours, and \({\cal X}\) contains 500 sequences in total. Due to the extremely heavy computation cost, we do not have the results of Zero-Out on Pythia-6.9B in the DEL Benchmark.

### Details of Data Collection

We show some collected examples in Tables 12&13. Table 7 reports how well the pretrained LLMs memorize sequences in the collected datasets.

#### The pretrained sequences of Pythia models.

EleutherAI releases the exact batches used by Pythia models during pretraining, where each sequence in a batch consists of \(2049\) tokens 5. We first randomly downsample the pretraining batches to a subset \({\cal Z}\) of \(102400\) sequences. Then, we use our criteria in SS3.2 to determine whether Pythia memorizes a sequence in the subset. After filtering, there remain \(500\sim 1000\) sequences in the subsets for both Pythia-deduped-2.8B and Pythia-deduped-6.9B; we simply sample \(505\) of them respectively as our collected datasets.

Footnote 5: [https://github.com/EleutherAI/pythia#](https://github.com/EleutherAI/pythia#) exploring-the-dataset

We also randomly sample a subset of \(2048\) sequences (\({\cal D}\)), each consisting of \(128\) tokens, to measure the perplexity of all LLMs we evaluate. We

\begin{table}
\begin{tabular}{|l|c|} \hline  & **Time** \\ \hline Activations & \(\sim 0.3\) sec \\ Slimming & \(\sim 12\) sec \\ Hard Concrete & \(\sim 1\) min \\ IG & \(\sim 43\) min \\ Zero-Out & \(\sim 8.5\) hr \\ \hline \end{tabular}
\end{table}
Table 6: The elapsed time of different methods to do localization (i.e., assign attribution scores to every neuron) on one sequence memorized by Pythia-6.9B. We time all methods on a single RTX A6000 GPU.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**GPT2 124M**} \\ \hline _ratio = 0.1\%_ & **R@0.1\%** & **R@0.2\%** & **R@0.5\%** \\ \cline{2-4} HC & \(57.9\) & \(81.0\) & \(94.6\) \\ Slim & **59.6** & **84.4** & **94.8** \\ IG & \(54.0\) & \(73.7\) & \(83.9\) \\ Activation & \(11.4\) & \(26.4\) & \(52.7\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: The INJ Benchmark results with a new set of random seeds. The Recall@\(k\%\) scores are very similar to the original ones in Table 2, showing the INJ Benchmark is not sensitive to the choice of random seed.



ensure that \({\cal Z}\cap{\cal D}=\varnothing\), so there is no overlap between the collected memorized sequences and sequences for perplexity.

Filtering with greedy decoding.Given the prefix \(p\) as the prompt to the LLM, we generate the suffix \(\bar{s}=[\bar{s_{1}},\ldots,\bar{s_{48}}]\) with greedy decoding, where

\[\bar{s_{t}}=\operatorname*{argmax}_{w\in\text{Voc}}P_{\theta}(w|p,\bar{s}_{< t}). \tag{18}\]

We then calculate the Levenshtein distance (Levenshtein, 1965) between the true suffix \(s\) and the generated one \(\bar{s}\), filtering out sequences with a distance greater than \(20\) characters. Note \(\bar{s}\) is different from \(\hat{s}\) in Eq 4, which is generated by teacher-forcing and is used to calculate memorization scores.

Deduplication.Although we use the deduplicated version of the dataset and models, the Pilededupe and Pythia-deduped models, we still find lots of near-duplicated sequences. Thus, we further deduplicate the collected memorized sequences. In particular, we follow Lee et al. (2022) to represent each sequence with a set of 5-grams when calculating the Jaccard index. Among a set of duplicates, we select the one that is best memorized, i.e., having the lowest Levenshtein distance on the generated suffix \(\bar{s_{t}}\) (Eq. 18), and discard the others.

Manually searched data.With our searching criteria in SS3.2, we can only identify less than 10 memorized sequences from subsets of the Pileddupe, Common Crawl, and Wikipedia, probably because OpenAI carefully preprocesses the data before training GPT2-XL. Thus, we actively search for potentially memorized data, such as famous poems and common lists of sorted items. We collect 105 sequences memorized by GPT2-XL and manually categorize them (see Tables 1 & 12), including 31 examples from Carlini et al. (2021). We set the prefix and suffix of a sequence by trial and error to ensure high memorization Accuracy. Unlike automatic searches that tend to find templated texts or uninteresting data with repetitive tokens (Zhang et al., 2023), our manual search ensures better data quality and enables us to analyze memorization within and across categories.

In particular, Figures 5 & 6 show that different localization methods constantly confuse sequences of related categories. For example, they are unable to disentangle neurons of different quotes and identify a small set of neurons responsible for both the order of Zodiac Signs and the order of Planets.

Responsible checklist.Note the Contact Info category of our manually collected dataset only contains public data, such as mailing addresses of corporate headquarters and famous buildings; thus, it does not have any potential risk of revealing private information. Similarly, our memorized datasets for Pythia models are a subset of the Pile, a public corpus under the MIT License.

### Hyperparameters

In the INJ Benchmark, the ECBD-2021 set contains \(156\) definition sequences. For the DEL Benchmark, we collect a set of 505, 505, and 105 sequences memorized by Pythia-deduped-6.9B, Pythia-deduped-2.8B, and GPT2-XL, respectively. For each set, we sample 5 sequences as the dev set, using the dev set performance to determine the hyperparameters for each LLM. The hyperparameters include the integrated gradient steps, i.e., the number of intervals in Riemann sum for integral approximation in IG; the temperature \(\beta\) and the initialization value of parameters \(m\) in Hard Concrete; the learning rate, the number of training epochs, and \(\lambda\), which balances the memorization loss and the sparsity loss, in Slimming and Hard Concrete. We observe that both Slimming and Hard Concrete are sensitive to the choice of hyperparameters. On the other hand, we find the performance of IG does not improve when using more integrated gradient steps, where we experiment with different steps ranging from 20 to 300. Thus, we set the step to 20 for all examples to reduce the heavy computation costs.

### More experiments comparing dropout in one layer with multiple layers.

In SS5.4, neurons are localized by IG. In this section, we conduct the same experiment using ZeroOut and Activations methods. Figure 4 shows that dropping out \(N\) neurons in multiple layers (Self-\(0.1\%\) 47 layers) even outperforms dropping out \(5\times N\) neurons in a single layer (Self-\(23.5\%\) 1 layer), except for the bottom layers where the memorization of both target and negative examples are greatly hurt. Hence, we believe the memory of a piece of information is distributed across layers; meanwhile, only a few weights in each layer are mainly responsible for the memorization (SS5.2).

We do not have the single-layer results of Slimming and Hard Concrete, because both methods train the masks of all neurons jointly, which requires us to retrain the masks only on a single 

layer to obtain their attribution scores. In comparison, the other three methods consider each neuron individually, allowing us to use the same attribution scores to select neurons in a single layer and make direct comparisons with the results in Table 3 (the dashed lines in Figure 4).

### Predicting top neurons across layers

In the INJ Benchmark, we randomly sample weights across layers to inject the data, instead of sampling a fixed percentage of weights per layer (see Algorithm 1). Hence, it may seem more natural to predict top-\(k\%\) of neurons across layers; we experiment with this alternative in Table 9.

Comparing the results of Table 2 and Table 9, we find that predicting top neurons per layer outperforms predicting top neurons across layers. This is because all localization methods assign larger attribution scores to neurons in the bottom layers, barely predicting neurons in the upper layers if we rank neurons globally. On the other hand, Table 2 and Table 9 show consistent results. Our findings and the ranking of different methods are coherent whether we rank neurons per layer or globally.

#### Implementation Details

Table 8 summarizes the architectures of LLMs we use. We run most experiments on RTX3090 24G GPUs; experiments involving Pythia-6.9B are run on RTXA6000 48G GPUs. We use transformers 4.31.0 and pytorch 1.13.

Figure 4: The DEL Benchmark results of Zero-Out, IG, and Activations methods when dropping out the same number of neurons in a single layer, where the blue lines show \(\Delta\) Self-Acc and the red lines show \(\Delta\) Neg-Acc. Under the same “neuron budget”, dropping out neurons in multiple layers (blue dashed lines) substantially outperforms dropout in a single layer, implying that memorized information is stored in a distributed fashion over multiple layers. Besides, dropping out neurons in the bottom layer greatly hurts the memorization of negative examples (red lines), suggesting that the bottom layer encodes general information.

\begin{table}
\begin{tabular}{l c r} \hline  & **\# Layers** & **\# Neurons** \\ \hline GPT2 124M & 12 & 3072 \\ GPT2-XL 1.5B & 48 & 6400 \\ Pythia-deduped-2.8B & 32 & 10240 \\ Pythia-deduped-6.9B & 32 & 16384 \\ \hline \end{tabular}
\end{table}
Table 8: The number of layers and the number of FFN neurons in each layer of different LLMs.

 



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline  & \multicolumn{4}{c}{**GPT2 12AM**} & \multicolumn{4}{c}{**GPT2-XL 1.5B**} & \multicolumn{4}{c}{**Pythia-deduped 2.8B**} & \multicolumn{3}{c}{**Pythia-depued 6.9B**} \\ \cline{2-17}  & R@1\% & R@2\% & R@5\% & R@1\% & R@02\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% \\ \hline _injection ratio = 1\%_ & & & & & & & & & & & & \\ Hard Concrete & \(9E-85\) & \(7E-86\) & \(2E-77\) & \(2E-100\) & \(8E-108\) & \(8E-111\) & \(3E-114\) & \(3E-121\) & \(6E-138\) & \(5E-111\) & \(6E-129\) & \(3E-155\) \\ Slimming & \(1E-72\) & \(8E-73\) & \(4E-62\) & \(1E-66\) & \(5E-77\) & \(2E-84\) & \(3E-111\) & \(5E-119\) & \(4E-119\) & \(1E-121\) & \(3E-134\) & \(2E-132\) \\ \cline{2-17}  & \(\mathbf{0.1\%}\) & \(\mathbf{0.2\%}\) & \(\mathbf{0.5\%}\) & \(\mathbf{0.1\%}\) & \(\mathbf{0.2\%}\) & \(\mathbf{0.5\%}\) & \(\mathbf{0.1\%}\) & \(\mathbf{0.2\%}\) & \(\mathbf{0.0\%}\) & \(\mathbf{0.0\%}\) & \(\mathbf{0.0\%}\) & \(\mathbf{0.0\%}\) \\ \hline _injection ratio = 0.7\%_ & & & & & & & & & & & & \\ Hard Concrete & \(6E-63\) & \(9E-66\) & \(1E-18\) & \(2E-96\) & \(3E-87\) & \(6E-73\) & \(7E-91\) & \(7E-110\) & \(1E-127\) & \(2E-42\) & \(7E-77\) & \(3E-83\) \\ Slimming & \(3E-10\) & \(4E-17\) & \(2E-20\) & \(1E-62\) & \(2E-73\) & \(4E-71\) & \(6E-96\) & \(7E-101\) & \(4E-99\) & \(1E-52\) & \(1E-54\) & \(1E-52\) \\ \hline \end{tabular}
\end{table}
Table 10: The p-values of the INJ Benchmark. \(H_{0}\): IG and pruning-based methods, Hard Concrete or Slimming, have identical expected Recall@\(k\%\) scores on ECBD 2021 examples. As we have 24 settings in total, we run 24 one-tailed paired t-tests with Bonferroni correction, setting the significance level \(\alpha=\frac{0.05}{24}\). We color the results that have p-values \(>\alpha\).

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline  & \multicolumn{4}{c}{**GPT2 12M**} & \multicolumn{4}{c}{**GPT2-XL 1.5B**} & \multicolumn{4}{c}{**Pythia-deduped 2.8B**} & \multicolumn{4}{c}{**Pythia-deduped 6.9B**} \\ \cline{2-17}  & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% & R@1\% & R@2\% & R@5\% \\ \hline _ratio = 1\%_ & & & & & & & & & & & & & \\ Hard Concrete & \(46.6\) & \(66.8\) & \(88.0\) & \(21.8\) & \(25.1\) & \(32.8\) & \(33.3\) & \(48.4\) & \(70.7\) & \(31.5\) & \(47.5\) & \(\mathbf{69.4}\) \\ Silming & \(43.1\) & \(64.6\) & \(79.9\) & \(5.2\) & \(11.5\) & \(27.0\) & \(\mathbf{33.6}\) & \(47.3\) & \(59.8\) & \(\mathbf{35.0}\) & \(\mathbf{49.6}\) & \(63.4\) \\ Zero-Out & \(24.0\) & \(36.8\) & \(52.7\) & \(4.2\) & \(7.3\) & \(13.5\) & \(10.1\) & \(14.3\) & \(20.5\) & - & - & - \\ IG & \(10.3\) & \(18.1\) & \(36.3\) & \(1.4\) & \(4.8\) & \(12.2\) & \(6.1\) & \(10.8\) & \(21.1\) & \(8.9\) & \(13.9\) & \(24.1\) \\ Activations & \(2.5\) & \(4.4\) & \(9.8\) & \(1.5\) & \(2.8\) & \(6.8\) & \(3.2\) & \(5.1\) & \(21.6\) & \(4.1\) & \(6.3\) & \(17.4\) \\ Random & \(1.0\) & \(2.0\) & \(5.0\) & \(1.0\) & \(2.0\) & \(5.0\) & \(1.0\) & \(2.0\) & \(5.0\) & \(1.0\) & \(2.0\) & \(5.0\) \\ \hline _ratio = 0.1\%_ & & & & & & & & & & & & \\ Hard Concrete & \(51.2\) & \(77.4\) & \(\mathbf{96.4}\) & \(\mathbf{49.8}\) & \(\mathbf{57.5}\) & \(\mathbf{63.6}\) & \(\mathbf{45.6}\) & \(\mathbf{65.5}\) & \(\mathbf{85.9}\) & \(28.7\) & \(40.7\) & \(55.8\) \\ Slimming & \(\mathbf{62.7}\) & \(\mathbf{87.0}\) & \(95.4\) & \(18.1\) & \(35.1\) & \(54.0\) & \(45.0\) & \(62.6\) & \(73.6\) & \(\mathbf{39.1}\) & \(\mathbf{52.1}\) & \(\mathbf{64.3}\) \\ Zero-Out & \(57.4\) & \(81.7\) & \(91.9\) & \(14.7\) & \(20.9\) & \(31.1\) & \(16.4\) & \(20.6\) & \(25.8\) & - & - & - \\ IG & \(36.0\) & \(55.0\) & \(75.5\) & \(2.5\) & \(3.5\) & \(6.0\) & \(12.6\) & \(16.4\) & \(21.9\) & \(19.7\) & \(23.6\) & \(28.9\) \\ activations & \(9.0\) & \(12.9\) & \(23.4\) & \(3.5\) & \(4.6\) & \(6.7\) & \(8.0\) & \(16.8\) & \(40.5\) & \(21.2\) & \(31.4\) & \(50.2\) \\ Random & \(0.1\) & \(0.2\) & \(0.5\) & \(0.1\) & \(0.2\) & \(0.5\) & \(0.1\) & \(0.2\) & \(0.5\) & \(0.1\) & \(0.2\) & \(0.5\) \\ \hline \end{tabular}
\end{table}
Table 9: The INJ Benchmark. The average Reacall@\(k\%\) of different methods when predicting top-\(k\%\) of neurons _across_ layers. The results are consistent with Table 2, where methods predict a fixed \(k\%\) of neurons in each layer.





\begin{table}
\begin{tabular}{|l|c|p{284.5pt}|} \hline Email & 100\% & Write to Jon Hilsenrath at jon.hilsenrath@wsj.com \\ \hline Zodiac Signs & 100\% & Aries Taurus Gemini Cancer Leo Virgo Libra Scorpio Sagittarius Capricorn Aquarius Pisces \\ \hline Patreon & 100\% & Thank you to our Patreon Supporters: Saintsofwar, Anon, LordOf_Fapping, Dryzak, Chabalbac, ioNz, LaX, VNT \\ \hline Declaration of Independence & 100\% & We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. \\ \hline Trump & 100\% & Sorry losers and haters, but my I.Q. is one of the highest -and you all know it! Please don’t feel so stupid or insecure, it’s not your fault. \\ \hline Newton & 100\% & I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me. \\ \hline Dr. MLK & 100\% & And when this happens, and when we allow freedom ring, when we let it ring from every village and every hamlet, from every state and every city, we will be able to speed up that day when all of God’s children, black men and white men, Jews and Gentiles, Protestants and Catholics, will be able to join hands and sing in the words of the old Negro spiritual, “Free at last! Free at last! Thank God Almighty, we are free at last” \\ \hline Genesis & 100\% & In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. And God said, Let there be light: and there was light. \\ \hline The Road Not Taken & 100\% & Two roads diverged in a yellow wood,nunAnd sorry I could not travel bothnAnd be one traveler, long I stoodnUnAnd looked down one as far as I couldnnTuore where it bent in the undergrowth;nünThen took the other, as just as fair,nunAnd having perhaps the better claim,nunBecause it was grassy and wanted wear \\ \hline \end{tabular}
\end{table}
Table 12: Examples of our manually collected data. The prompt (prefix) is colored in brown. The numbers are the Accuracy (Eq. 5) of GPT2-XL on memorizing the sequences, where \(100\%\) Accuracy means the true suffix can be fully reconstructed with greedy decoding.





\begin{table}
\begin{tabular}{|p{85.4pt}|c|p{213.0pt}|} \hline Mike Wall Bio & **100\%** & WallUnnMichael was a science writer for the Idaho National Laboratory and has been an intern at Wired.com, The Salinas Californian newspaper, **and the SLAC National Accelerator Laboratory. He has also worked as a herpetologist and wildlife biologist. He has a Ph.D. in evolutionary biology from the University of Sydney, Australia, a bachelor’s degree from the \\ \hline Hardware & **100\%** & PCs may be defined as a desktop, floor standing, or portable microcomputer that includes a system unit having a central processing unit (CPU) and associated volatile and non-volatile memory, including random access memory (RAM) and basic input/output system read only memory (BIOS ROM), a system monitor, a keyboard, one or more flexible diskette drives, a CD-ROM drive, \\ \hline Contact Info of Skyhorse Publishing & 100\% & , or educational purposes. Special editions can also be created to specifications. For details, contact the Special Sales Department, Arcade Publishing, 307 West 36th Street, 11th Floor, New York, NY 10018 or arcade@skyhospublishing.com.unnArcade Publishing® is a registered trademark of Skyhorse Publishing, Inc.®, a Delaware corporation.unnVisit \\ \hline Meme & 98\% & a lot; that’s great! It’s a little awkward to ask, but we need your help. If you have already donated, we sincerely thank you. We’re not salespeople, but we depend on donations averaging §14.76 and fewer than 1\% of readers give. If you donate just §5.00, the price of your coffee, Catholic Online School could keep thriving. Thank \\ \hline Malik Report & 100\% & check that allowed Dvorak to flick the puck over his shoulder...?UnnAbout The Malik Report is a destination for all things Red Wings-related. I offer biased, perhaps unprofessional-at-times and verbose coverage of my favorite team, their prospects and developmental affiliates. I’ve joined the Kukla’s Korner family with five years of blogging under \\ \hline Porn & **100\%** & make love to her. She returned the favor with an amazing blowjob and a masterful fuck session..\nENJOY!!!!mnThis entire website has a voluntary content rating to block access by minors. This rating is compatible with microsoft internet explorer’s content filtering function andufna facilitates website blocking software. For a tutorial on blocking this site click here.\nCopyright bangros. \\ \hline Pokémon Fans & 100\% & We’re a group of Pokémon fans dedicated to providing the best place on the Internet for discussing ideas and sharing fan-made content. Welcome! We’re glad you’re here.\nUnitIn order to join our community we need you to create an account with us. Doing so will allow you to make posts, submit and view fan art and fan fiction, download fan-made games, \\ \hline  & \multicolumn{1}{p{213.0pt}}{} & \multicolumn{1}{p{213.0pt}}{} \\ \end{tabular}
\end{table}
Table 13: Examples of memorized sequences we collect from the Pile-dedupe. The prompt (prefix) is colored in brown. The numbers are the Accuracy (Eq. 5) of Pythia on memorizing the sequences, where \(100\%\) Accuracy means the true suffix can be fully reconstructed with greedy decoding.



* [14] M. C. B. Krizhevsky, A. Vedaldi, and S. Ioffe. Learning from a single image by minimizing the loss function. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 235-238, 2016.
* [15] J. L. Friedman, R. G. Olshen, and D. E. Silverstein. Deep residual learning for large scale data. _arXiv preprint arXiv:1907.05815_, 2019.

[MISSING_PAGE_POST]

 

* [14] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, "A fast learning approach to deep neural networks for large-scale image recognition," in _IEEE transactions on pattern analysis and machine intelligence_, vol. 30, no. 2, pp. 579-680, 2013.
* [15] J. L. Friedman, R. Tibshirani, and P. Salakhutdinov, "Deep residual learning with applications to image processing," _arXiv preprint arXiv:1406.05580_, 2014.
* [16] T. F. Goodfellow, I. Guyon, Y. Bengio, and E. Rabinovich, "The MIT Press, 2012.
* [17] V. Mirza, N. Ozair, and A. Courville, "Learning from tiny images using deep convolutional nets," in _Proceedings of the IEEE conference on computer vision and pattern recognition_, volume 1, pages 1-10, 2014.

[MISSING_PAGE_POST]