DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models
Authors: Xinwei Wu
Junzhuo Li
Minghui Xu
Weilong Dong
Shuangzhi Wu
Chao Bian
Deyi Xiong
Venue: EMNLP 2023 Main (2023)
Keywords: Privacy Protection
Language Model

Abstract: Pretrained language models have learned a vast amount of human knowledge from large-scale corpora, but their powerful memorization capability also brings the risk of data leakage. Some risks may only be discovered after the model training is completed, such as the model memorizing a specific phone number and frequently outputting it. In such cases, model developers need to eliminate specific data influences from the model to mitigate legal and ethical penalties. To effectively mitigate these risks, people often have to spend a significant amount of time and computational costs to retrain new models instead of finding ways to cure the 'sick' models. Therefore, we propose a method to locate and erase risky neurons in order to eliminate the impact of privacy data in the model. We use a new method based on integrated gradients to locate neurons associated with privacy texts, and then erase these neurons by setting their activation values to zero.Furthermore, we propose a risky neuron aggregation method to eliminate the influence of privacy data in the model in batches. Experimental results show that our method can effectively and quickly eliminate the impact of privacy data without affecting the model's performance. Additionally, we demonstrate the relationship between model memorization and neurons through experiments, further illustrating the robustness of our method.

DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models
Xinwei Wu', Junzhuo Li², Minghui Xu', Weilong Dong',
Shuangzhi Wu', Chao Bian**, Deyi Xiong 1.2-
Colleege of Intelligence and Computing, Tianjin University, Tianjin, China
School of New Media and Communication, Tianjin University, Tianjin, China
Department of Computer Science and Technology, Tsinghua University, Beijing, China *ByteDance Lark AI, Beijing, China
(wuxw2021, xuminghui willowd, eytio. edu cn, wufurui@bytedancee com, bianc 8@mai is. tsinghua. edu. cn
Abstract
Henderson et al., 2018) have demonstrated that pretrained language models tend to memorize and regurgitate a significant portion of the training data, including atypical data points that appear only once in the training data. Additionally, external factors (e.g., membership attack) also contribute to these risks. A variety of methods have been explored to attack LLMs for training data extraction. For instance, Carlini et al. (2021) have successfully ex- tracted personal information from GPT-3's output, while Li et al. (2023) have induced the genera- tion of personal information by utilizing multi-step prompts ChatGPT. All these show that large pre- trained language models suffer from a serious risk of privacy leakage.
Large language models pretrained on huge amount of data capture rich knowledge and information in the training data. The ability of data memorization and regurgitation in pre- trained language models, revealed in previous studies, brings the risk of data leakage. In or- der to effectively reduce these risks, we pro- pose framework DEPN Detect and Edit Privacy Neurons in pretrained language mod- els, partially inspired by knowledge neurons and model editing. In DEPN, introduce novel method, termed as privacy neuron detec- tor, locate neurons associated with private information, and then edit these detected pri- vacy neurons by setting their activations to zero. Furthermore, we propose privacy neuron ag- gregator dememorize private information in batch processing manner. Experimental results show that method can significantly and ef- ficiently reduce the exposure of private data leakage without deteriorating performance empirically of the model. Additionally, demonstrate the relationship between model memorization and privacy neurons, from multi- ple perspectives, including model size, training time, prompts, privacy neuron distribution, il- lustrating the robustness of approach.
In order to safeguard privacy, numerous meth- ods have been proposed. The majority focus on either removing sensitive information during the data processing stage (Liu et al., 2017; El Emam et al., 2009; Zhou et al., 2008; García-Pablos et al., 2020), or reducing the extent to which models memorize training data during the training stage (Li et al., 2021; Hoory al., 2021; Plant et al., 2021; Coavoux et al., 2018). However, privacy breaches often come light after the completion of model training, rendering previous methods less effective. There are also methods proposed in the post-processing stage, which involve slight param- eter retraining to make the model forget privacy information (Bourtoule et al., 2021; Gupta et al.. 2021; Neel et al., 2020). Nevertheless, these meth- ods generally incur high computational complexity, making it challenging to apply them to complex model architectures. In practice, model develop- ers often attempt to prevent language models from outputting specific information via blocking or fil- tering certain keywords, which, however, does not truly address the underlying issue.
1 Introduction
Remarkable progress has been made in large lan- guage models (LLMs) in recent years (Brown et al.. 2020: Liu et al., 2021; Ouyang et al., 2022: Lee et al., 2023). dowever,despite this success, L.LMs are confronted with privacy and security concerns in real-world applications (Guo et al., 2022; Brown et al., 2022: Li et al., 2023). The primary cause of privacy and security risks is the inherent nature of large pretrained language models. Previous studies (Carlini et al., 2019, 2021: Thakkar al., 2021;
Conresponding author.
We speculate that private information might be

---

stored in specific neurons, just like knowledge neu- rons (Geva al., 2021; Meng et al., 2022; Dai et al., 2022). This presumption suggests that we could change the model memorization of private information by detecting and deleting these neu- rons (termed as privacy neurons). Therefore, we propose a framework DEPN for detecting and edit- ing privacy neurons. To detect privacy neurons, we introduce privacy neuron detector that uses gradient integration to simultaneously compute the contributions of multiple markers to neuron activa- tions. This allows us to estimate an overall privacy attribution score for private information. Subse- quently, we further propose a privacy neuron editor that simply sets the activations of the top privacy neurons with the highest privacy scores to zero to erase the model memorization of the corresponding private information. For the scenario of process- ing multiple sentences at the same time, we also present a privacy neuron aggregator to facilitate privacy information editing in batches.
Preliminary
Privacy Definition Privacy preservation has be- come issue of great concern in the era of pre- trained language models. Protecting privacy first requires specifying the boundaries of privacy. The definition of privacy is broad. is closely related to its context and discourse (Brown et al., 2022). In any texts about, a specific person can be consid- ered as private. For the convenience of research, a narrow definition of privacy is usually taken (Sousa and Kern, 2023), which treats personal identity in- formation as privacy, such as names, ID numbers, phone numbers and other related expressions. The proposed DEPN can be adapted to the above two definitions.
Model Editing Geva et al. (2021) find that the feed-forward network module in Transformer (i.e., two-layer perceptron) can be considered as a key- value memory, where each key corresponds to a text pattern and each value represents distribution over the vocabulary. Based on this finding, strand of research, (Geva et al., 2021; Meng et al., 2022; Dai et al., 2022) propose to edit factual knowledge encoded in pre-trained LLMss by locating neurons related to the entities of factual knowledge. The basic idea of localization to change the param- eters of neurons, and then observe the changes the probability of the object entity predicted by the model. The neurons with greater influence on the probability are more closely related to the object entity.
Experimental results show that our framework can quickly reduce the risk of private data leakage without affecting model performance. Compared with other methods, our framework is highly ef- ficient. Furthermore, we have found that model memorization leads to the aggregation of privacy neurons in our experiments, and demonstrated that our framework is very suitable for the scenario of deep model dememorization.
The main contributions of our work summa- rized as follows:
However, these methods have limitation that they can only observe the probability change of one token time. Semantic units are usually composed of a sequence of tokens, rather than a single token. This makes it impossible to use these methods directly.
For the first time, we explore model edit- ing into privacy protection of pretrained lan- guage models, provide new way for privacy protection, and propose DEPN to effectively eliminate model memorization in the post- processing stage.
Methodology
The proposed DEPN consists of three components: the privacy neuron detector ($3.2), the privacy neu- ron editor ($3.3) erase the model memorization of privacy data, and the privacy neuron aggregator ($3.4) for privacy preservation in batches.
We propose the privacy neuron detector to localize privacy neurons based on gradient attribution, and the privacy neuron editor to dememorize privacy information in pretrained language models.
3.1 Privacy Prediction Task
We conduct experiments to demonstrate that the proposed framework capable of protect- ing privacy lcakage from pretrained language models.
Given tuple T (X, Y), let Y = (V1s ..., Un be the sequence with private information, X be the the context of the sequence, be the parame- ters of a language model. Given a context X, the

---

3.Privacy Editor
Artivationn
2.Privacy Neuron Aggregator
sop "itwayy
each private vote by attribution
1.Privacy Neuron Detector ot a
Y:
Jeff
Skill
Aut(w;)) of
da
Aw?
Transformer Parameters: 0
Leh Layer
Foed Forward Layer
L x
Self Attention Layer
Batch of X
Hidden State
On
behalf
of
[MASK]
[AAK]]
Detecting and Erasing Privacy Neurons Framework
pricase Information"
bahalt
behalf of
Jeff skil
My boss
Figure 1: The diagram of DEPN. When language model leaks privacy information, DEPN calculates privacy attribution scores using the Privacy Neuron Detector. It then selects the top privacy neurons with the Privacy Neuron Aggregator and eliminates the model memorization of privacy information using the Privacy Editor.
probability of the language model yielding token is P(s]|x 0), M/ E Y, so the probability of the model leaking the private sequence is:
which neurons play key role in the leakage of private information from language models.
Let wf be neuron be evaluated by the pri- vacy attribution method, where is the layer of the neuron in the language model, and its position. According to $3.1, the probability of the model outputting private information is:
Y
P 0)
II P(
(1)
Take "An Ka is a senior writer at ESPN.com" as private sentence containing person's 's name "An Ka ". Suppose the input to the language model is is senior writer at ESPN.com" is reduce the probability of privacy leakage, i.e., minimizing the probability of predicting "An
Y
P(Y w/
II
P X,
ut =0/)
(2)
goal
where of represents the value of the k-th neuron
and "Ka
the 1-ith FFN layer.
3.2 Privacy Neuron Detector
We gradually change the parameter of the target neuron from 0 to the original value of the neuron. In this process, the probability of the output will accordingly change. We calculate the cumulative gradient of the probability change during this pro- cess as the neuron's ''s contribution (i.e., privacy attri- bution score) to the privacy-sensitie output. The privacy attribution score is computed as:
As described in Section factual knowledge is found be stored in the feed-forward networks Transformer, in form of key-value memory, Inspired by this, we speculate that private infor- mation might be also encoded in specific neurons. Model editing has offered methods to locate and edit reeweedge-related neurons. However, existing methods can only deal with semantic units com-- posed of single token, making them directly applicable detect and edit mutli-token private sequences. To address this issue, we propose pri- vacy attribution method based on gradient integra- tion. The proposed privacy attribution can evaluate
Au((w)) - ot 1. OP(Y
dat
(3)
Owk
where 3 is the original value of the neuron w/. aP(Y|X,af)
calculates the gradient of the model ow

---

output with regard w Directly caleulating con- tinuous integrals is intractable. We follow Dai et al. (2022) to use Riemann approximation:
Experiments
We carried out experiments to examine the effec- tiveness of the proposed DEPN on dataset con- taining private information.
P
OP 3
Att(w))
Σ
(4)
m ا=
auf
4.1 Setup
where m 20 is the number of approximation steps.
Dataset We used the Enron dataset (Klimt and Yang, 2004). It consists of employee emails that were publicly disclosed during Enron's legal inves- tigation by the Federal Energy Regulatory Commis- sion. It the largest publicly available collection of "real" email data, containing over 500,000 emails from 158 users. 1 We randomly sampled 5% the data from Enron as the validation dataset to evaluate model performance.
As P(Y|X, w = 11.-1 P(uil X, uf ot)
we have
Y Σ
m (y X, m 38)
Att(w()
(5)
Dwf
1=1
If the neuron has great influence on the output of private information, the gradient will be signifi- cant, and large integration value will be obtained. Therefore, the privacy attribution score can mea- sure the neuron's contribution to the leakage of privacy information, and the greater the privacy attribution score, the greater the privacy sensitivity of the neuron. We select neurons with the top privacy attribution score candidates for editing.
Private Information Sampling In our study, we categorized the private information in the Enron dataset into two types: private phrases (for the nar- row definition of privacy), such as names and phone numbers, and a batch of randomly sampled sen- tences to be edit. Names: We selected 20 unique names that are memorized by language models, found in 126 sentences, such as "An Ka is a senior writer at ESPN.com". Phone Numbers: We also selected 20 unique LM-memorized phone numbers, such "My phone number is 7 388 Private texts: We randomly selected 100 sentences that are not semantically overlap- ping with each other. Appendix A.4, we discuss how we determine whether private information is memorized by a language model.
3.3 Privacy Editor
After detecting the privacy neuron candidates with the privacy neuron detector, reduce the model memorization of private information by editing. Particularly, we use a simple yet effective editing strategy: setting the parameters (activation values) of the corresponding neurons to 0, so that the in- formation flow will not pass through these privacy neurons.
3.4 Privacy Neuron Aggregator
Model Settings We conducted experiments us- ing the widely used pretrained model, BERT-base (Devlin et al., 2018). The model consists of 12 transformer layers, with a hidden state size of 768 and an internal hidden size of 3072 for the feed- forward network (FFN). Our experiments were per- formed on NVIDIA Tesla A6000 graphics pro- cessors. More training details are show in Ap- pendix A.I.
As number of sentences in the training data of LLMs contain private information, the privacy neu- ron detection and editing can be done over multiple sentences in a batch processing way. To erase pri- vacy information encoded in the language model from multiple sentences in the training data, we propose the privacy neuron aggregator. When the input is text batch, we calculate the privacy attri- bution score matrix of each sequence in the batch. After the privacy attribution score calculation, we let each sequence vote for neurons according to their privacy attribution scores, and seleet the top neurons with the most votes. These selected neu- rons will be edited to erase private information. The hyperparameter is adjusted according to the model size, training epochs and other factors. More details can be found in 1$5.1).
Baselines To demonstrate the effectiveness and robustness of DEPN, compared it with the fol- lowing baseline models. BERT-O: The bert model that has not been trained on the Enron dataset. Since the model does not know the privacy infor- mation in the dataset, it provides an oracle for as- sessing the risk of privacy leakage; BERT-F: The
1https: pettps://wwer http://wwwwccccco cs. cmu, edu/"enron/

---

Privacy Leakage Risk
Privacy Type
Models
Time 4
Valid-PPI.
Metric
Value
BERT-O
25.23
1.58
BERT-F
100%
3.07
15.74
Phone Number
Exposure
BERT-FE
2.4%
3.11
9.78
BLRT-DP
181.4%%
5.43
3.12
BERT-O
25.23
0.87
BERT-F
100%
3.07
1.21
Name
MRR
BERT-FE
4,4%
3.11
1.15
BERT-DP
181.4%
5.43
0.95
BERT-O
25.23
10.05
BERT-F
100%
3.07
2.30
Random Text
PPL
BERT-FE
4.6%
3.11
3.67
BERT-DP
181.4%
5.43
8.82
Table I: Results of testing the risks of leaking private Phone Numbers, Names, and Texts on different baseline models, as well as the efficiency of protection. Bold and underlined results indicate the best and second best result, respectively. t: the higher the better. the lower the better.
bert model trained on the Enron dataset, which cor- responds to the best predictive performance on the Enron dataset, but has the greatest risk of privacy leakage; BERT-DP: A privacy model trained by the differential privacy gradient descent method (Li et al., 2021) on the Enron dataset, which the commonly used privacy protection method when using private data for training.
token sequence E {e1...., Cn the length is [E the model predicts the rank of the target token as rank(e:(Q), and the MRR for the name E is cal- culated as follows:
1==1 (e
E
We applied our proposed DEPN on BERT-F to make a safe model, which is referred to as BERT- FE in following experiments. Our codes are avail- able now.?
Perplexity (PPL): When the private text is a complete sentence, we directly use the perplexity as the measure of the model memorization.
4.2 Main Results
Metries To observe the effect of different privacy preserving methods on the model performance, we use the Perplexity of Masked Language Modeling task on the Enron validation dataset (Valid-PPL) the metric.
Table presents our main results, including model performance, privacy leakage risk, and execution time cost. The results demonstrate the competitive- ness of our framework.
For the performance on the Enron validation dataset (Valid-PPL), BERT-O, which is not trained on the Enron dataset, exhibits the poorest perfor- mance. BERT-DP trained with DP-SGD does not perform well either, due to noise introduced during backpropagation. In contrast, BERT-FE equipped with DEPN performs almost par with BERT-F on the validation dataset, indicating that neuron erasure minimally impacts model performance. Regarding privacy leakage risk metrics, includ- ing exposure, MRR, and PPL, clearly indicate that BERT-FE equipped with DEPN achieve the reduc- tion of privacy leakage risk. BERT-F, trained di- rectly private data, exhibits the highest risk. In comparison, DEPN significantly reduces the risk of leakage. BERT-O. which has no access private data, demonstrates the lowest risk across all three data types. The BERT-DP model also exhibits very low risk.
Duc to the difffrrnt types of private information, we provide metries separately for the risk of privacy leakage.
Exposure: The exposure (Carlini et al., 2019) metric is commonly used in privacy attacks to mea- sure the exposure risk of phone numbers. Given a number sequence c, model with parameters 0, and the randomness space R, the exposure eg of e be calculated as:
= -log, ||| log, Rankø (c).
(6)
Mean Reciprocal Rank (MRR): A person's name is usually composed of multiple tokens. Therefore, we use the reciprocal average of the rank each target token to measure the model's memorization of names. Given prefix Q, name
hhttps://github. com/flamewwi 123/DEPN

---

before-adtting
15-
3.30
after-ddtinn
3.25
Exposure
12-
3.15
3.10
before cciting
aiteredlinng
Number of Erased Neurons
the Number of Erased Neurons
Exposures with different numher of edited
(b) Model performance with different number of edited
Figure 2: The performance of the model and the risk privacy leakage with the change trend the number of neurons edited.
In terms of execution time cost, we assume that the fine-tuning time of BERT-F on data excluding privacy is 100% (reference time cost). The DEPN framework requires less than 5% of the reference time cost, while BERT-DP requires more time due to gradient clipping.
tion set, while the blue line represents the PPL of BERT-FE on the validation set with different num- bers of edited neurons. As the number of erasures increases, the PPL. noticeably increases. Therefore, increasing the number of edited neurons reduces the risk of privacy leakage in the model, but it also leads decrease in the model performance.
In conclusion, while differential privacy training and fine-tuning with non non-private data can mitigate privacy leakage risks, they incur more time and may significantly undermine model performance. The DEPN framework strikes an excellent balance between performance and privacy protection.
5.2 Relationship between Memorization And Privacy Neurons
As it is widely recognized, privacy data leakage often stems from the model's ability to memorize the training data. In this subsection, we conducted experiments to investigate the relationship between model memorization and privacy neurons, provid- ing further evidence for the effectiveness of the proposed DEPN.
5 Analysis
We further conducted in-depth analyses to demon- strate why DEPN is able to dememorize privacy in LLMs from multiple perspectives, including anal- yses the relationship between privacy neurons and model memorization, the robustness well the cost-effectiveness of DEPN.
Impact of Training Time on Privacy Neuron Distribution over Layers Figure 3 depicts the evolution of the distribution of privacy neurons over layers as the number of training epochs in- creases. Overall, the distribution of privacy neu- rons is pyramid-shaped, and most privacy neurons identified by the privacy neuron detector are lo- cated in layers 10-12 of BERT-base. Specifically, in epoch 1, about 40% of privacy neurons are in the top layer of BERT-base As training progresses, the proportion of privacy neurons from deep layers increases to 60% by epoch and to 80% by epoch 6. By the 9-th epoch, the distribution of privacy neurons remains largely unchanged compared the 6-th epoch. This suggests that as the depth of model training increases, the memorization
5.1 Effect of the Hyperparameter
Figure illustrates the impact of the hyperparame- ter, the number edited neurons, on the model. We calculate the exposures of the original model BERT- F and the enhanced model BERT-FE on 20 phone numbers. In Figure 2(a), the red line represents the average exposure of BERT-F, while the green line represents the average exposure of BERT-FE with varying numbers of edited neurons. As the number of edited neurons increases, the exposure significantly decreases. In Figure 2(b), the purple line represents the PPL of BERT-F on the valida-

---

Percentage
20%
10%
Percentage
30%
20%
10%
0%
10%
10%
20%
20%
30%
2
3
4
7
8
9
10
11
12
1
2
3
6 7
8
9
10
11 12
Layer
Layer
(a) epoch
(b) epoch 3
30%
30%
Percentage
20%
Percentage
20%
10%
10%
0%
0%
10%
20%
20%
30%
30%
3
8
12
2
7
10 11 12
Layer
Layer
(c) epnch
(d) epoch 9
Figure 3: The distribution of privacy neurons in the bert-base model at different training epochs.
Before Editing
After Editing
Models
Edited Neurons
Time
Reduction Rate
Valid-PPI.
Exposure
Valid-PPL.
Exposure
bert-small
0.26h
4.09
5.10
4.57
3.39
33.5%
bert-base
200
1.59h
15.74
3.11
9.78
37.86%
bart-large
400
7.66h
2.93
18.10
2.98
7.63
57.84%
Table 2: The privacy leakage risk reduction rate for models of different sizes.
private data tends to converge.
privacy risks.
In Appendix A.3, conducted experiments to observe the changes of privacy leakage risk reduc- tion different training epoch. The results show that when the training time increases, the risk of privacy leakage increases too, and the proposed DEPN becomes more effective in privacy preserva- tion.
Summary of the Relationship between Mem- orization and Privacy Neurons Based the aforementioned experimental findings, we can con- clude that the model's scale, training time, and frequency of privacy data occurrence are all factors that have influence on the model memorization. As the model memorization of privacy data deepens, the aggregation of privacy neurons associated with privacy data becomes makes the method of locating and eliminating pri- vacy neurons more suitable for deep memorization scenarios. Therefore, the DEPN framework has demonstrated excellent effectiveness in mitigating model memorization.
Effect of the Model Size Table 2 illustrates the performance of the DEPN framework on models of different scales. Each model was trained for 10 epochs using the optimal hyperparameter set- tings. Overall, larger models require more time identify privacy neurons and require editing greater number privacy neurons for optimal per- formance. Larger models tended to show a deeper memory for phone numbers before privacy neu- rons are edited, leading to higher exposure. After privacy neuron editing, from the perspective of re- duction rate, the exposure of the large model is reduced even more. These findings suggest that larger models are more risk of privacy breaches. Fortunately, the DEPN framework demonstrates better performance on larger models compared to smaller ones, offering improved protection against
pronounced, which
5.3 Robustness Analysis
Ablation Study We conducted ablation experi- ments to assess the robustness of the privacy neu- detector by comparing its performance with different neuron localization methods phone number data. Table 4, we present the results of these experiments. Specifically, "KN" refers to the knowledge attribution approach proposed by Dai et al. (2022), while "Random" donates an approach

---

Before Editing
After Editing
Privacy Amount
Edited Neurons
Time
Valid-PPL
Exposure
Valid-PPL
Exposure
20
0.76h
3.07
15.74
3.11
9.78
100
500
1.59h
3.07
12.46
3.33
10.47
1000
2000
17.61h
3.07
8.32
3.81
8.03
Table 3: Analysis results on the cost-effectiveness of DEPN.
Before Editing
After Editing
Prompts
Original Exposure
Exposure
Methods
Valid PPI.
Exposure
Valid PPL.
Exposure
Coontact me at
12.52
9.77 +
PND Editing
3.07
15,54
3.11
9.78
Coontact me at #e*
11,20
9.40 4
KN Editing
3.07
15.54
3.10
10.75
'Contact me 440
12.50
9.68 |
Random + Editing
3.07
13.54
3.07
12.48
'Call me *e*?
12.31
11.82 4
Table 4: Effect of using different neuron localization methods on results.
"My phone number is 'You can call me at
13.41
12.96
13.04
12.84
Table 5: Results with varying prompts during privacy attack. 'Contact me at **** is the prefix to the private phone numbers in the training data, and the others are varying prompts used inference.
that randomly selects the same number of neurons as our method. Our method PND (privacy neuron detector) achieves superior performance in terms of exposure reduction compared to the other meth- ods. Although the knowledge attribution approach gains good exposure reduction, it is less effec- tive than our method due to its attribution being targeted at a single token. The random selection approach is also able to decrease privacy exposure but the exposure reduction is not as significant as the KN approach and our detector. These results unequivocally demonstrate the effectiveness of our method for in privacy neuron localization.
ment results are shown in Table 3. As the amount of private data increases, more neurons need to be edited to achieve better privacy protection, and the performance of the model drops significantly. Furthermore, becomes apparent that, with the escalation of private data volume, the reduction in privacy risks gradually diminishes. These obser- vations indicate that DEPN excels remediating language models when dealing with a small num- ber of data lcaks, but cxhibits weak performance when confronted with a large batch of private data.
Robustness to Different Prompts We conducted experiments to validate the robustness of DEPN different prompts. We sampled private data containing phone numbers, composed of the same prefix, from the training dataset. We then performed privacy attacks during inference using different prompts to examine whether changing prompts would still result in privacy leakage. Ta- ble 5 presents the results of these experiments. The training data consist of phone numbers with the same prefix of 'Contact me at #*** We observe privacy risk reduction across all prompts, demon- strating the robustness of DEPN to prompt.
6 Related Work
Model Editing To edit incorrect or undesirable information captured in LLMs, a variety of model editing approaches have been proposed, which can be categorized into four strategies. First, the Con- strained Fine-tuning strategy (Zhu et al., 2020) updates LLMs specifically for the target knowl- edge, allowing precise modification. Second, the Memory-based Editing strategy (Mitchell al., 2022; Dong et al., 2022) maintains knowledge cache that stores new information to replace unde- sirable predictions. Third, the Meta-learning-basee Editing strategy (De Cao et al., 2021; Mitchell et al., 2021) introduces editable training based on meta- learning, training model parameters to accommo- date editing. Lastly, the Locating and Editing strat- egy (Geva al., 2021; Meng et al., 2022; Dai al., 2022) assumes that knowledge locally stored in LLMs. This strategy locates specific parameters associated with the knowledge and directly edits parameters to perform editing.
5.4 Analysis on the Coft-Effectiveness of DEPN
In this subsection we discuss the limitation of DEPN, specifically its dependency the amount of private data be erased. We conducted exper- iment where used 1,000 private data instances, each containing phone numbers, extracted from our training dataset. DEPN was applied onto the BERT- base model to erase private information. Experi-

---

Privacy Protection To address privacy risks in NLP models, various privac--rresevvng methods have been proposed. which can be categorized into three main stages of application (Guo et al., 2022; Sousa and Kern, 2023): data processing stage, pre-training and/or fine-tuning stage, and post-processing stage. In the data processing stage. methods involve removing replacing sensitive information in the original data (Liu al., 2017; El Emam et al., 2009; Zhou et al., 2008; García- Pablos al., 2020). the pre-training or fine- tuning stage, data privacy can be protected by mod- ifying the model training process. One approach is differential privacy stochastic gradient descent (DP- SGD) (Li et al., 2021; Hoory et al., 2021), which introduces noise into the clipped gradient to re- duce the distinction between gradients and prevent memorization of training data. Another method is adversarial training (Plant et al., 2021; Coavoux et al., 2018), which constrains the model's learning of private information through adversarial training techniques. However, methods used in the data pro- cessing stage and in the pre-training or fine-tuning stage are not applicable if the privacy leakage is discovered after the model training is completed. Methods used in the post-processing stage focus on making trained models forget specific data al- ter specific parameters to safeguard hidden private information (Bourtoule et al., 2021; Gupta et al., 2021; Neel et al., 2020). These methods are often with high computational cost and cannot be eas- ily applied to large models. In contrast, proposed DEPN can achieve the protection of private infor- mation in the post-processing stage with small computational overhead.
Limitations Our current study still has two lim- itations. First, although we propose method process private data in batches, we find that too many instances in a batch will reduce the effect of memorization erasure. Second, we use a few types of private information in our experiments due to the limited availability of datasets containing pri- vate information. We would like to collect more available datasets for framework in the future.
Ethical Statement In this paper, we use the En- ron dataset to evaluate the feet of DEPN. This dataset consists of employee emails that were publicly disclosed during Enron's 's legal investigation by the Federal Energy Regula- tory Commission. Since the data comes from real persons, masked sensitive information such specific names and phone numbers in this paper.
ef-
A Acknowledgements
The work was partially supported by the research collaboration project between Tianjin University and ByttDDance(P/322210629000033 and Zhejiang Lab (No. 2022KHOAB01), We would like thank the anonymous reviewers for their insightful com- ments.
References
Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. Generalization in nli: Ways (not) to go be- yond simple heuristics. In Proceedings of the Second Workshop on Insights from Negative Results in NLP. pages 125-135.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- Choou Hengnui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Ma- chine unlearning. In 202/ IEEE Symposium Secu- rity and Privacy (SP), pages 141-159. IEEE
7 Conclusion
In this paper, have presented privacy neuron detecting and editing framework DEPN address privacy leakage risks in pretrained language mod- els. Through the privacy neuron detector based on the privacy attribution scoring method, we accu- rately detect risky neurons associated with private information. The privacy neuron editor effectively eliminates model memorization of private data. Ex- perimental results and in-depth analyses demon- strate the ability of DEPN reduce privacy risks efficiently without degrading model performance. Our work explores a novel approach to privacy pro- tection and contributes to model de-memorization in the post-processing stage.
Hannah Brown, Katherine Lec, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramer. 2022. What does it mean for language model to preserve privacy? Proceedings of the 2022 ACM Conference on Fairness, Accountabiliry and Transparency, pages 2280-2292.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemnjj Kos, and Dawn Song. 2019. The secret sharer: Eval- uating and testing unintended memorization in neu-

---

ral networks, In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284.
Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvape Malvajerdi, and Chris Waites. 2021. Adaptive machine unlearning. In Advances in Neural Information Processing Systems, volume 34, pages 16319-16330. Curran Associates, Inc.
Nicholas Carlini, Floriam Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts. Tom Brown, Dawn Song, Ulfar Frlingsson, al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2001 2650.
Peter Henderson, Koustuv Sinha, Nicolas Angelard- Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123-129.
Maximin Coavoux, Shashi Narayan, and Shay B Co- neural representations
hen. 2018. Privacy-presers of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1-10.
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell. Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al. 2021. Learning and evaluating differentially pri- vate pre-trained language model. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 1178-118 1189.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493 8502.
Bryan Klimt and Yiming Yang. 2004. Introducing the enron corpus. In CEAS, volume 45, pages 92-96.
Nicola De Cao. Wilker Aziz, and Ivan Titov, 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491 6506.
Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023, Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233-1239
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.044.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can strong differentially private learners. In International Conference on Learning Representations.
Qingxiu Dong, Damai Dai. Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Find- ings of the Association for Computational Linguisties: EMNLP 2022, pages 5937-5947.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint ar 20X
Khaled El Emam, Fida Kamal Dankar, Romeo Issa, Elizabeth Jonker, Daniel Amyot, Elise Cogo, Jean- Pierre Corriveau, Mark Walker, Sadrul Chowdhury. Regis Vaillancourt, et al. 2009. A globally optimal k- anonymity method for the de-identification of health data. Journal of the American Medical Informatics Association, 16656667--682.
Zengjian Liu, Buzhou Tang, Xiaolong Wang, and Qing- cai Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of biomedical informatics, 75:S34- S42.
Aitor Pablos, Naiara Pérez, and Montse Cuadros. 2020. Sensitive data detection and classification in spanish clinical text: Experiments with bert. Pro- ceedings of the 12th Language Resources and Evalu- ation Conference, pages 4486 4494.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 35:17359-17372.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forware forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484-5495.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. In International Conference on Learning Representations.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christo- pher D Manning, and Chelsea Finn. 2022. Memory- based model editing at scale. In International Con- ference on Machine Learning, pages 15817-15831. PMLR.
Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu, and Tianwei Zhang. 2022. Threats to pre-trained lan- guage models: Survey and taxonomy. arXiv preprint urXiv:2202.06886

---

Seth Neel, Aaron Roth, and Saced Sharifi- Charifi-Mallajerde 2020. Descenttto-delete Gradient- based methods for machine unlearning. In International Conference on Algorithmic Learning Theory.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744,
Richard Plant, Dimitra Gkatzia, and Valerio Giuffrida. 2021. Cape: Context-aware private embeddings for private language learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7970-7978.
Samuel Sousa and Roman Kern. 2023. How keep text private? a systematic review of deep learning methods for privacy-preservine preserving natural language pro- cessing. Artificial Intelligence Review, 56(2):1427- 1492.
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021. Under- standing unintended memorization in language mod- els under federated learning. In Proceedings of the Third Workshop Privacy Natural Language Pro- cessing, pages 1-10.
Bin Zhou, Jian Pei, and WoShun Luk, 2008. brief sur- vey on anonymization techniques for privacy preserv- ing publishing of social network data. ACM Sigkald Explorations Newsletter, 12-2).
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li. Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arxwv:2012.00363.

---

A
Appendix
A.1 Training Details
before-editing after-editing
For BERT-base fine-tuning, set the hyperpa- rameters follows: 20 training epochs, a learning rate 3e- with linear warm-up, and a batch size of 16. We fine-tuned BERT-base on the Enron dataset using the Masked Language Modeling task to simu- late training on datasets containing privacy informa- tion. Additionally, we pretrained smaller (layer=4. hidden size=512, intermediate size=2048) (Bhar- gava et al., 2021) and larger (layer=24, hidden size=1024, intermediate size=4096) BERT mod- els 3 to compare the performance of privacy erasure at different model scales.
36
Exposure
12
10
15
20
Epoch
Figure 4: Comparison of privacy leakage risk reduction at different training epochs.
A.2 Effect of the Frequency of Privacy Data
Ocurrence
with longer training time have a more pronounced memorization of the training data. Additionally, the yellow line represents the exposure of phone numbers after privacy neuron erasing. The widen- ing gap between the two lines indicates that as the model's 's memorization becomes more apparent, the proposed DEPN becomes more effective in privacy preservation.
We also examined the influence of the frequency of privacy data ocurrence in the training set on DEPN. As shown in Table 6, phone numbers with ocur- rence frequency greater than 10 exhibit higher posure compared to those with frequency less than 10, indicating a higher risk of leakage. How- ever, after erasure, the exposure of phone num- bers with frequency greater than 10 reduced by 32.65%, while the exposure of phone numbers with frequency less than 10 is reduced by 22.58% % These results suggest that our method effectively reduces exposure for both high-frequency and low- frequency phone numbers, mitigating the risk of privacy leakage.
A.4 The Judgement of the Memorization
In experiment, we specifically identify the pri- vate data memorized by the language model from the training dataset. To assess whether the model has memorized private data, we employ the con- text of private information as the input to the lan- guage model. Subsequently, we calculate the risk of private information leakage and classify the in- formation with leakage risk exceeding predefined thresholds as having been memorized by the lan- guage model. For names, we establish a threshold for memorization the MRR of less than 1.5. For phone numbers, we employ the Exposure values exceeding as memorization.
Frequency >=10
Original Exposure
Exposure 15.59
23.15
<10
8.90
6.89
Table 6: Frequency of privacy data ocurrence make exposure different.
A.3 Effect of Training Time
Figure illustrates the changes in exposure of phone number data before and after erasing pri- vacy neurons in models with different training epochs. We conducted experiments using 20 differ- ent phone numbers and averaged the final results. The blue line represents the exposure of phone numbers before privacy neuron erasing. The blue line initially remains low but exhibits a significant surge after the 10-th epoch, indicating that models
co/ BERT-RRA Co/BERT-Large--eee large-uncased


Reviews:
[{"review_id": "CV9NByHxbr", "reviewer": "Reviewer_xo7t", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and suggestions. We are pleased to answer two key aspects raised in your comments.\n\n- Types of Privacy Information:\n\nWe concur with you regarding the increased complexity posed by diverse types of privacy information. However, due to existing legal and ethical constraints, publicly available datasets containing privacy information remain relatively scarce. Consequently, our approach centers on addressing the specific privacy information type of individual identities in Enron. Nonetheless, we believe that exploring a broader spectrum of privacy information types is a worthwhile research direction.\n\n- Amount of Privacy Data:\n\nUsing DEPN to erase large amount of privacy data requires editing a great number of neurons, which, however, results in rapid deterioration of model performance. Large-scale neuron editing is also very challenging for model editing. We’d like to continue our work in this direction and explore machine unlearning methods to precisely edit activation values of privacy neurons. Many thanks for this great suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The first one to apply neuron-based model editing techniques to the privacy issue of language models\n2. The method is efficient to reduce the privacy issue while maintaining the model performance successfully", "reasons_to_reject": "1. Such neuron-level editing method may be limited by the complexity of the issue: once the privacy issue involves lots of different types of information, or involves large amount of data, the number of to-be-edited neurons would be increasingly large, and thus may degrade the model performance. Therefore, the extendability to more complicated privacy issue is questionable. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes the DEPN (Detecting and Editing Privacy Neurons) method, comprised of a gradient-based privacy neuron detector, an on-off neural privacy editor, and a privacy neuron aggregator for multiple sentences. They show by BERT model on Enron dataset that the proposed method can reduce the detect privacy neurons without hurting the model ability. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t94mvioSlN", "reviewer": "Reviewer_dXCH", "summary": "", "questions": "Q1. How many samples in the batch can achieve optimal forgetting without hurting the model performance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and suggestions. The following are our answers to your comments and questions.\n\n- Response to R1:\n\nWe sincerely appreciate this insightful suggestion. We have indeed employed our method to autoregressive language models and conducted experiments. Due to the page limitation, we haven’t included them in the current version. But we have updated our GitHub repository (https://github.com/Take2try/DEPN) with code and results on GPT-2. The results on GPT-2 suggest that DEPN achieves promising privacy preservation performance on autoregressive models, as observed on BERT. We’ll provide these results and analyses in the new version with additional pages. \n\n- Response to R2:\n\nDue to existing legal and ethical constraints, there is a scarcity of publicly available datasets containing privacy information. As a result, we are only able to conduct experiments on very limited privacy information types available in the public dataset Enron. We are willing to validate DEPN on more privacy information types if more public privacy-related datasets are available in the future.\n\n- Response to R3 & Q1:\n\nThis is an exceptionally insightful suggestion. Our experiments indeed demonstrate that erasing a larger batch of samples requires editing more neurons, leading to a decline in model performance. When increasing the batch size of privacy samples while keeping the number of erased neurons constant, the DEPN's ability to erase privacy samples becomes more difficult. \nUltimately, our findings indicate that maintaining a balance between privacy preservation performance and model efficacy is achievable when the batch size does not exceed 10,000. We’d like to discuss more on this in the next version. \n\n- Response to R4:\n\nPrivacy preservation methods typically encompass data preprocessing and model training stages. DEPN, functioning as a post-processing privacy preservation method, holds a unique advantage of offering remediation for already trained models. In this aspect, we do not have many baseline methods to compare, which also works in a post-processing way like ours.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a crucial problem in language models. It offers an effective solution to overcome it by balancing between making the model private and does not hurt the general performance of the model by a significant margin.", "reasons_to_reject": "R1. Even though MLM is necessary and used in many applications, the method should also be employed in the autoregressive language models, which are more used in real-world applications. Also, the problem of memorization is observed and analyzed in many research. Employing it in the autoregressive LM is crucial to prove the significance of the approach. Furthermore, making the LM private usually hurts the general LM performance; the only way to check this in MLM is perplexity. Conversely, autoregressive LM has many downstream benchmarks that can be evaluated to ensure the model's effectiveness.\n\nR2. The paper only addresses limited types of information and, in some cases, formally specified data (phone numbers).\n\nR3. it is mentioned in the limitations section that \"many instances in a batch will reduce the effect of memorization erasure\" from line 575 to line 578. However, the authors did not analyze this observation and relationship by illustrating when the editing becomes invalid when increasing the batch size.\n\nR4. The paper only compared one baseline method. Comparing the approach with various methods in the literature would be better.", "questions_to_authors": "Q1. How many samples in the batch can achieve optimal forgetting without hurting the model performance?", "paper_topic_and_main_contributions": "The paper addresses the problem of privacy leaking in Masked language models and proposes an approach for identifying and editing neurons that can lead to privacy leakage.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "prJGGA0dP0", "reviewer": "Reviewer_iqTb", "summary": "", "questions": "Question A: The privacy information in Enron dataset can be regarded as different entities, why not use accuracy/F1 to evaluate the privacy detection performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and suggestions. We answer your comments and questions as follows.\n\n- Response to R1:\n\nWe agree with you that more different types of privacy information are needed to demonstrate the robustness and applicability of our method. Medical domain is indeed a valuable domain for assessing privacy preservation; however, datasets in this domain often require additional authorizations for use. We have attempted to obtain access to the MIMIC-III clinical dataset, but have not received a response yet. If we finally gain access to such privacy datasets, we are happy to immediately update our experiment results in the next version.\n\n- Response to R2:\n\nMany thanks for this suggestion. Our approach mainly aims at post-processing privacy protection before a language model is officially deployed by its developer. \n\n- Response to Q1:\n\nThank you for your question. The reason we have not employed ACC/F1 metrics to measure privacy leakage lies in the challenge of accommodating intricate privacy data types. For instance, when dealing with privacy data presented as complete sentences, determining the denominator for accuracy becomes considerably complex.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A simple and direct method for privacy preservation in pre-trained language models.\n* Well written paper, easy to follow.\n", "reasons_to_reject": "* In the EEnron dataset, only three types of private information is evaluated, it would be better to use other datasets (e.g. biomedical domain) which contains more types of private information.\n* It's a white box method, may not be suitable for black box language model APIs.\n", "questions_to_authors": "Question A: The privacy information in Enron dataset can be regarded as different entities, why not use accuracy/F1 to evaluate the privacy detection performance?", "paper_topic_and_main_contributions": "This paper introduced a privacy preservation method for pre-trained language models, which directly locates \"privacy\" neurons in the language model and edit these detected neurons by setting the activations to zero. Experiments on a Enron dataset and BERT showed that the method can do privacy preservation without losing too much model performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers concur that the manuscript proposes an effective method to make LLMs private without hurting their general performance. To make the paper even more appealing, the authors are advised to address the following considerations: a) evaluate the cost-effectiveness of the proposed method when extensive neuron must be modified to ensure intricate privacy; b) provide guidance on determining an optimal batch size that balances preserving privacy and maintaining model performance.
Decision: Accept-Main