Learnable Privacy Neurons Localization in Language Models (2024)

Abstract: Concerns regarding Large Language Models (LLMs) to memorize and disclose private information, particularly Personally Identifiable Information (PII), become prominent within the community. Many efforts have been made to mitigate the privacy risks. However, the mechanism through which LLMs memorize PII remains poorly understood. To bridge this gap, we introduce a pioneering method for pinpointing PII-sensitive neurons (privacy neurons) within LLMs. Our method employs learnable binary weight masks to localize specific neurons that account for the memorization of PII in LLMs through adversarial training. Our investigations discover that PII is memorized by a small subset of neurons across all layers, which shows the property of PII specificity. Furthermore, we propose to validate the potential in PII risk mitigation by deactivating the localized privacy neurons. Both quantitative and qualitative experiments demonstrate the effectiveness of our neuron localization algorithm.

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated exceptional performance on various NLP tasks, leveraging huge model architectures and a tremendous scale of real-world training data (OpenAI, 2023; Touvron et al., 2023; Taori et al., 2023). However, the ability of memorization within LLM has also raised concerns regarding security within human society (Bender et al., 2021; Bommasani et al., 2021). One significant concern is that private information may be memorized and leaked by LLMs. An attacker can extract private information contained in the training corpus, especially Personally Identifiable Information (PII) such as names or addresses (Carlini et al., 2021, 2022; Huang et al., 2022; Rocher et al., 2019; Lukas
*Corresponding author. Our code is available at https://github.com/richhh520/ Learnable-Privacy-Neurons-Localization.
et al., 2023), which constitutes a privacy violation according to the General Data Protection Regulation (GDPR) (Regulation, 2016). Various methods have been proposed to mitigate the memorization of PII (Lison et al., 2021; Anil et al., 2021), primarily focusing on the sanitization of training data (Vakili et al., 2022; Lee et al., 2021), or providing differential privacy (DP) guarantees during the training process (Yu et al., 2021b; He et al., 2022). However, the mechanism by which LLMs memorize PII is not well understood.
In this paper, we propose a novel privacy neuron localization algorithm. Our method utilizes the hard concrete distribution (Louizos et al., 2017) to make neuron masks learnable and design adversarial objective functions to minimize the predictive accuracy of PII while preserving other nonsensitive knowledge. Besides, we employ another penalty to minimize the number of localized neurons, thus localizing a minimal subset of PII-specific neurons. We subsequently conduct a comprehensive analysis of the localized privacy neurons. Our findings reveal that memorization is localized to a minor subset of neurons, which are spread across all layers, predominantly within the MLP layers. Furthermore, we also discover that privacy neurons have the property of specificity for certain categories of PII knowledge. Inspired by the observation, we propose to investigate the privacy leakage mitigation ability by deactivating the localized neurons during the evaluation process, thus eliminating the memorization of PII. Experimental results demonstrate that our framework can achieve comparable performance in mitigating the risks of PII leakage without affecting model performance.

heading: 2 Method
text: Denote f(θ) as a PLM with parameters θ. Given a sequence of tokens x = [x1, ..., xT ] from the training corpus, f(θ) can leak the private
256
sequence [xp, ..., xp+I ] within x by generating [xp, ..., xp+I ] = argmax∗(Pf(θ)(∗|x<p)). For example, as shown in Fig. 1, the email address of Kent Garrett is disclosed by the model, which constitutes significant societal risks.
In this section, we introduce a novel neuron localization algorithm that localizes neurons in f(θ) responsible for PII prediction, to elucidate the underlying mechanisms of PII memorization, as illustrated in Fig. 1. To be specific, our goal is to find a small subset of neurons f(m⊙ θ) (or equivalently, the mask m) that deactivating these neurons prevents PII leakage, while not affecting the language modeling ability, thus indicating the memorization of PII-specific knowledge. m and ⊙ denote the differentiable binary neuron mask and Hadamard product operator respectively.

heading: 2.1 Differentiable Neuron Mask Learning
text: Since the training loss is not differentiable for binary masks, we resort to a practical method to learn subnetworks (Louizos et al., 2017), which employs a smoothing approximation of the discrete Bernoulli distribution (Maddison et al., 2016). Following (Zheng et al., 2022), we assume mask mi corresponding to each neuron to be an independent random variable that follows a hard concrete distribution HardConcrete(log αi, βi) with temperature βi and location αi (Louizos et al., 2017):
si = σ( 1
βi (log µi 1− µi + logαi)), (1)
mi = min(1,max(0, si(ζ − γ) + γ)), (2)
where σ denotes the sigmoid function. si denotes the mask score of each neuron and mi is the
approximately discrete activation value (i.e., almost 0 or 1) of si. γ and ζ are constants, and µi is the random sample drawn from uniform distribution U (0, 1). In this work, we also treat βi as a constant, thus only α is the set of differentiable parameters for m. During the inference stage, the mask mi can be calculated through a hard concrete gate:
min(1,max(0, σ(logαi)(ζ − γ) + γ)). (3)
Algorithm 1 Neuron Localization Algorithm. Require: mask parameters α, pre-trained lan-
guage model f(θ) with frozen parameter θ, training corpus X , hyper-parameters β, γ, ζ, η, learning rate lr.
1: Initialize s ← σ( 1β (log µ 1−µ + logα)), where µ ∼ U(0, 1) 2: Initialize m← min(1,max(0, s(ζ − γ) + γ)) 3: Initialize f(θ)← f(m⊙ θ) 4: for epoch in num_epochs do 5: for x in X do 6: Generate f(m⊙ θ) with step1-3 7: if optimizer_idx == 0 then 8: L = Lm(f(m⊙ θ), x) + η R(m) 9: else
10: L = Ladv(f(m⊙ θ), x) + η R(m) 11: end if 12: α = α− lr · ∇α(L) 13: end for 14: end for 15: m← min(1,max(0, σ(logα)(ζ − γ) + γ)) 16: return m

heading: 2.2 Adversarial Privacy Neuron Localization
text: To localize PII-specific neurons, we propose to negate the original training objective, i.e., maximizing the negative log-likelihood of the PII token sequences. Specifically, given a sequence of tokens x = [x1, ..., xT ] from the training corpus and PII tokens [xp, ..., xp+I ], our training objective is:
Lm(f(m⊙ θ), x) = I∑
i=1
log(P (xp+i|x<p+i)).
(4) On the other hand, to preserve the original language modeling ability of f(m ⊙ θ), we propose to perform further training on the corpus, utilizing the pre-training loss as the adversarial loss:
Ladv(f(m⊙θ), x) = − T∑
t=1
log(P (xt|x<t)). (5)
Finally, to minimize the number of localized neurons, we penalize the number of localized neurons by minimizing the L0 complexity of mask scores which are zero:
R(m) = − 1|m|
|m|∑
i=1
σ(logαi − βilog −γ ζ ). (6)
In the training step, the differentiable mask is adversarially trained by Eq. 4 and Eq. 5, with Eq. 6 as an auxiliary. The overall optimization procedure is elaborated in Algorithm 1.

heading: 3 Can PII memorization be localized?
text: In this section, we primarily investigate the following questions: (a) Is the memorization of PII confined to the latter layers of the model (Baldock et al., 2021)? (b) How many neurons are required to memorize privacy information? (c) Are privacy neurons specific?

heading: 3.1 Experiment Setup
text: Model and Dataset. We utilize the GPT-Neo (125M, 1.3B) LMs (Black et al., 2021). We utilize Enron Email Dataset (Klimt and Yang, 2004) and ECHR (Chalkidis et al., 2019) containing different types of PII in two domains. PII and NER. For Enron dataset, we regard email and name as PII. We utilize the predefined prompt templates (e.g. the email address of target_name is) and the email-name correspondence provided in DecodingTrust (Wang et al., 2023) to extract PII. For ECHR, We tag PII in 4 categories (person, law, date and gpe) in the corpus, utilizing Named Entity Recognition (NER) tagger from Flair (Schweter and Akbik, 2020). We utilize the prefix context to prompt generation.

heading: 3.2 Privacy Neuron Distribution
text: We first investigate the distribution of privacy neurons across different layers in PLM. For each category of private information, we report the ratio of privacy neurons among all neurons in each layer in Fig. 2. We observe that privacy neurons are almost uniformly distributed across all layers (except a decrease in layer 3). We further explore the distribution in different model components (i.e., query, key, value, and MLP) in memorizing PII. As shown in Fig. 3, The ratio of privacy neurons in the MLP layer is significantly higher than in other components. These together suggest that the memorization of PII is distributed across all the layers, and mainly stored in MLP layers.

heading: 3.3 Category-wise Memorization
text: Following the previous part, we observe that the distribution patterns of different categories of privacy neurons in the model are also similar. Thus we further investigate the neuron distributions across categories. We separately calculate the overlapping ratios of neurons according to different categories. The heatmap of the ratios is shown in Fig. 4, where DATE and DATE* represent different subsets of the same category. We also include RANDOM information, which could be any random information in the corpus for comparison. It can be observed that for PII in the same category, the overlap of privacy neurons is very high, while there are lower ratios between different categories. Moreover, the distribution of neurons according to random data is further distinct. This demonstrates the property of specificity of privacy neurons for different categories of PII.

heading: 3.4 Sensitivity of the number of Neurons
text: As introduced in Alg. 1 and Eq. 6, the penalty on the number of localized neurons is controlled by
the hyper-parameter η. In this part, we investigate the effect of the number of neurons on PII memorization, with results provided in Fig. 5. As η continually decreases, the ratio of localized neurons increases from close to 0 to a maximum of 0.035. Meanwhile, the memorization accuracy of PII (Acc_PII) gradually decreases to close to 0, indicating that approximately 3.5% of neurons are required to eliminate the memorization. However, when the ratio of masked neurons exceeds 0.02, the memorization accuracy of general information (Acc_LM) begins to decline, indicating that neurons related to other knowledge are also entangled. We finally decide η to be 5 as a trade-off of PII forgetting and general information memorization.

heading: 4 Can localization inform mitigating privacy leakage?
text: Inspired by previous observations, we propose to investigate the effect of privacy neurons on privacy leakage mitigation. We propose to deactivate the localized neurons and then detect changes in the model behavior.

heading: 4.1 Experimental Setup
text: Experiments are conducted on GPT-Neo (125M, 1.3B) models and PII datasets constructed in Sec. 3.1. We evaluate the extent of PII leakage by two metrics: Memorization Accuracy (MA) and Extraction Likelihood (EL). We utilize Scrubbed Fine-tuning (Lukas et al., 2023), Differential Privacy Decoding (DPD) (Majmudar et al., 2022) and knowledge unlearning (UL) (Jang et al., 2022) as our baselines. Detailed descriptions of baselines and metrics are in the Appendix B.1.

heading: 4.2 Results
text: Qualitative Analysis. Tab. 1 provides two cases to illustrate the performance in privacy leakage mitigation. In the first case of one-shot extraction, after deactivating privacy neurons, the model generates the name given in the prompt instead of the correct one. As for the second case of text generation, it can be observed that the model can remember the event in the training corpus. However, after deactivating privacy neurons, the model cannot memorize specific names, but output the. instead.
Comparison Results. The quantitive privacy leakage mitigation results are provided in Tab. 2. We report the leakage degree of PII and general information (i.e., random information other than PII). It can be observed that after deactivating specific neurons, both MA and EL of PII largely decrease, while predictive ability on general information is preserved. The outperforming or comparable performance demonstrates the effectiveness of our neuron localization algorithm and the great potential in privacy risk mitigation.

heading: 5 Related Works

heading: 5.1 LLM memorization
text: The success of Large Language Models (LLMs) is largely attributed to their vast training datasets and the immense number of model parameters, enabling them to memorize extensive information from the training data. A line of work simply quantifies how much knowledge is memorized during pretraining by extracting relational knowledge about the world (Petroni et al., 2019, 2020; Jang et al., 2021; Heinzerling and Inui, 2020; Cao et al., 2021; Carlini et al., 2022). However, memorization of LMs is a threat to privacy leakage (Carlini et al., 2021; Jagielski et al., 2022; Shi et al., 2023). Another line of work focuses on the memorization mechanisms of models (Jagielski et al., 2022; Tirumala et al., 2022; Kandpal et al., 2022). It is posited by (Baldock et al., 2021; Maini et al., 2023) that a subset of a model’s parameters is dedicated to learning generalizable examples, while another subset is predominantly utilized for memorizing atypical instances. Furthermore, several studies have demonstrated the alteration of factual predictions through a small subset of neurons (Meng et al., 2022a,b; Dai et al., 2021; Li et al., 2023). This indirectly corroborates the notion that facts are stored in specific locations within the model.

heading: 5.2 Privacy Risks Mitigation
text: To mitigate privacy risks in large language models, various privacy-preserving techniques have been proposed. Existing solutions can be categorized according to their applied stage: the pre-training stage, the in-training stage, and the post-training stage (Smith et al., 2023; Guo et al., 2022). Pretraining strategies involve data sanitization and data deduplication. Data sanitization proposes to eliminate or substitute sensitive information in the original dataset (Dernoncourt et al., 2017; GarcíaPablos et al., 2020; Lison et al., 2021). Data deduplication removes duplicate sequences from the training data to reduce the probability of generating exact sequences (Kandpal et al., 2022). In-training strategies mitigate data privacy by altering the training procedure (Li et al., 2021; Hoory et al., 2021). Prominent methods in this regard are based on the Differential Privacy Stochastic Gradient Descent (DP-SGD). This technique integrates noise into the clipped gradient, diminishing the distinctiveness of gradients and thereby hindering the memorization of training data (Anil et al., 2021; Yu et al.,
2021a,b). Post-training methods perform unlearning (Kassem et al., 2023; Jang et al., 2022) and editing (Wu et al., 2023) to the well-trained models to change the memorization of specific data.

heading: 6 Conclusion
text: In this paper, we propose a novel method for jointly localizing a small subset of PII-sensitive neurons within LLMs. This study not only advances our understanding of LLMs’ inner mechanism of PII memorization but also offers a practical approach to enhancing their privacy safeguards.
Limitations and Future Works
We acknowledge the presence of certain limitations. First, we only investigate the localization of memorization of PII in this paper, while other kinds of (privacy) information may possess a different pattern. We hope to extend our proposed method to the localization of other knowledge in LLMs in the future. Second, experiments have not been conducted on very large models. Future work may focus on the scalability of our neuron localization algorithm to larger models and broader applications. Third, experiments on privacy leakage mitigation are still preliminary. Unlearning (Chen and Yang, 2023; Chen et al., 2024b; Eldan and Russinovich, 2023) or knowledge editing (De Cao et al., 2021; Meng et al., 2022a; Chen et al., 2024a) technicals could be involved to enhance the performance, and more evaluating datasets (Bisk et al., 2020) to provide comprehensive evaluation and privacy-utility tradeoff analysis in the future.
Ethics Statement
In this paper, we propose a method for localizing PII-sensitive neurons within LLMs. This method not only deepens our understanding of the internal mechanisms LLMs use to memorize PII but also provides a practical approach to bolstering privacy protections. All datasets utilized in this study are publicly accessible, and our research fully adheres to their respective licenses.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (Grant No. 62106222), the Natural Science Foundation of Zhejiang Province, China (Grant No. LZ23F020008) and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.

heading: A Can PII memorization be localized?
text: A.1 Experiment Setup
Dataset. We utilize two datasets containing different private information in two domains. Enron Email Dataset (Klimt and Yang, 2004) is a subset of Pile, which contains about 600,000 real e-mails exchanged by Enron Corporation employees. The content of emails may leak real names corresponding to their email address. ECHR (Chalkidis et al., 2019) contains records from the European Court of Human Rights. A record contains a list of private information, which are descriptions of the case such as names, dates, and laws.
Implementation Details. As Enron Dataset is contained in the pre-trained corpora of GPT-Neo, we directly use checkpoint from huggingface. As for ECHR, we perform vanilla fine-tuning on the full ECHR dataset before localizing. We initialize values in α to be 2. β is set to be 0.025. γ and ζ are -0.1 and 1.1. η is 5.

heading: B Can localization inform mitigating privacy leakage?
text: B.1 Experimental Setup
B.1.1 Baselines (1) Scrubbed: We follow Lukas et al. (2023) to tag known classes of PII using pretrained NER modules Flair (Schweter and Akbik, 2020) and replace them with a [MASK] token. Then we use the scrubbed corpus to fine-tune the model.
(2) Differential Privacy Decoding (DPD) (Majmudar et al., 2022): DPD proposes a method for achieving differential privacy without retraining large language models, by introducing perturbations during the decoding phase. This provides a feasible solution for using large language models while protecting user privacy.
(3) Knowledge unlearning (UL) (Jang et al., 2022): UL proposes knowledge unlearning, aimed at reducing the privacy risks that might be leaked by large pre-trained language models (LLMs) when processing tasks. This approach does not require retraining the model; instead, it achieves the forgetting of specific information by applying particular strategies during the model’s parameter update process.
B.1.2 Evaluating Metrics. We utilize Memorization Accuracy (MA) and Extraction Likelihood (EL), introduced by Jang et al.
(2022). Extraction Likelihood (EL) measures the accuracy of PII generation:
EL(x) = ∑T−n
t=1 Overlap(fθ(x<t), x≥t) T − n . (7)
where fθ(x<t) represents the sequence of output tokens produced by the language model fθ upon receiving x<t as input.
Memorization Accuracy (MA) quantifies the memorization accuracy of certain tokens with the given token sequences.
MA(x) = ∑T−1
t=1 1{argmax(pθ(·|x<t)) = xt} T − 1 .
(8)

heading: Future Work
text: Limitations and Future Works
We acknowledge the presence of certain limitations. First, we only investigate the localization of memorization of PII in this paper, while other kinds of (privacy) information may possess a different pattern. We hope to extend our proposed method to the localization of other knowledge in LLMs in the future. Second, experiments have not been conducted on very large models. Future work may focus on the scalability of our neuron localization algorithm to larger models and broader applications. Third, experiments on privacy leakage mitigation are still preliminary. Unlearning (Chen and Yang, 2023; Chen et al. , 2024b; Eldan and Russinovich, 2023) or knowledge editing (De Cao et al. , 2021; Meng et al. , 2022a; Chen et al. , 2024a) technicals could be involved to enhance the performance, and more evaluating datasets (Bisk et al. , 2020) to provide comprehensive evaluation and privacy-utility tradeoff analysis in the future