ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Year: 2023

Abstract:
Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that ETHICIST significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is available at https://github.com/ thu-coai/Targeted-Data-Extraction.

heading: 1 Introduction
text: Large pre-trained language models have achieved impressive results on various natural language processing tasks (Devlin et al., 2019; Radford et al., 2019a; Raffel et al., 2020). Model sizes rapidly increase from millions to trillions of parameters and keep growing to achieve better performance and even obtain some emergent abilities (Brown et al., 2020; Chowdhery et al., 2022; Wei et al., 2022; Fedus et al., 2022; Zhang et al., 2022). Despite the
∗Corresponding author.
success of large-scale pre-trained language models, recent works point out that they may memorize a considerable fraction of training data, leading to the privacy risk of information leakage (Carlini et al., 2022a; Tirumala et al., 2022a; Mireshghallah et al., 2022; Carlini et al., 2021). Furthermore, researchers find that memorization scales with model sizes (Carlini et al., 2022a). Therefore, this privacy risk becomes more and more critical in the era of large-scale pre-training. And attacking language models to extract their training data attracts increasing attention.
There are currently two main settings to extract training data. One is membership inference attack, which infers whether a given example is contained in the model’s training data (Hisamoto et al., 2020; Shokri et al., 2017). The other is untargeted train-
12674
ing data extraction (Carlini et al., 2021), which aims to extract training data from scratch (i.e., without the given prefix). However, both settings are not suitable for extracting targeted training data. For example, attackers may feed the model with a prefix indicating the beginning of an email and try to extract the following private email content in the training dataset as shown in Figure 1. In such cases, we do not have complete examples to do membership inference, and we have specific goals instead of performing untargeted extraction. Therefore, we focus on targeted training data extraction in this paper, which requires recovering the suffix when given a prefix according to the training data. Compared with untargeted training data extraction, the task matters more because attackers can recover specific types of training data instead of any possible training data that might be harmless. What’s more, it is easier to evaluate targeted training data extraction because we just need to compare the prediction with the ground truth suffix. However, for untargeted training data extraction, we need to search over the whole massive pre-training dataset (e.g., The Pile dataset (Gao et al., 2020), which has 800GB text data) to check whether it contains the generated sample, which is very slow and costly.
The general process for targeted training data extraction can be divided into two steps: (1) generating one or more possible suffixes based on the given prefix, and (2) choosing a most likely suffix as the prediction result based on a confidence estimation method. We summarize two challenges of this task: (1) how to increase the generation likelihood of the ground truth suffix, and (2) how to estimate the confidence accurately so that the confidence score can be meaningfully interpreted as the probability that the output suffix is correct. To tackle these challenges, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation. For the first challenge, we propose loss smoothed soft prompting. It uses soft prompt to elicit memorization in the attacked model, and adds an additional loss besides the maximum likelihood estimation (MLE) loss to smooth the loss distribution of the suffix tokens. Through the loss smoothing, we hope to ensure that the probability of the ground truth token at each time step is not low, which makes it more likely to sample the ground truth suffix. With the two loss functions, we tune the prepended soft
prompt tokens on an extracted training set which contains pairs of prefixes and ground truth suffixes. The existence of a training set is reasonable because large-scale pre-trained data generally contain public data (e.g., Common Crawl) 1. For the second challenge, we propose a calibrated confidence estimation method. We find that the model’s perplexity cannot accurately represent the probability that the generated suffix is correct because the prediction probabilities for diversified prefixes are inherently different and incomparable. We thus normalize the confidence of the generated suffixes with a local estimation, which can mitigate the problems caused by intrinsic differences in the difficulties of distinct samples. We verify ETHICIST on a recently proposed public benchmark containing 15,000 pairs of prefixes and suffixes derived from The Pile dataset (Gao et al., 2020). Experiments show that ETHICIST can significantly improve the extraction performance, which suggests that existing large language models are at significant risk of leaking training data. We also discuss and analyze several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length.
Our contributions can be summarized as follows:
• We propose loss smoothed soft prompting to reduce the difficulties of sampling the ground truth suffixes.
• We propose a calibrated confidence estimation method that enables the confidence score to be meaningfully interpreted as the probability that the output suffix is correct.
• Experiments on a recently proposed benchmark demonstrate that ETHICIST can consistently and significantly improve the data extraction performance across various model sizes. We further investigate several factors influencing the data extraction performance.

heading: 2 Related Work

heading: 2.1 Training Data Extraction
text: Existing works on training data extraction mainly focus on membership inference attack or untargeted training data extraction. For membership inference attack, adversaries need to judge whether a given example is contained in the training data of the attacked model. Shokri et al. (2017); Song
1Similar setting is adopted in Hisamoto et al. (2020).
and Shmatikov (2019) train several shadow models that mimic the attacked models’ behaviors to help train an auditing model that can predict whether an example is contained in the training dataset. Hisamoto et al. (2020) perform membership inference attacks on machine translation systems. They find it is harder to attack sequence generation models than classification models. Song and Raghunathan (2020) show that the encoded dense representations can leak information under membership inference attack. Mireshghallah et al. (2022) focuses on attacking masked language models that are pre-trained on possibly sensitive data (e.g., clinical notes). They introduce an additional reference masked language model besides the original attacked model and compute the ratio of the likelihood measured by the attacked model and the reference model, which is better than solely relying on the attacked model.
For untargeted training data extraction, adversaries first generate various samples using the attacked model and then predict whether they are contained in its training set. Carlini et al. (2021) extract hundreds of verbatim sequences from the popular pre-trained language model GPT-2 (Radford et al., 2019b). And there is privacy information such as names, phone numbers, and email addresses in the extracted sequences. Lehman et al. (2021) try to extract sensitive information from BERT (Devlin et al., 2019) pre-trained on clinical notes. However, they are mostly unable to meaningfully expose Personal Health Information by simply using templates. Different from the existing works, we focus on targeted training data extraction that aims to recover the suffix when given a prefix, which is more security-critical and easier to evaluate.

heading: 2.2 Memorization
text: We generally expect models can gain the generalization ability from the training process. However, recent works point out that models may unintentionally memorize the training data even without overfitting (Tirumala et al., 2022a; Carlini et al., 2022a, 2019; Béguelin et al., 2020). One possible method to mitigate this problem is to deduplicate training data (Kandpal et al., 2022). However, Carlini et al. (2019) also show that it is possible to recover samples appearing only once in the training dataset. Surprisingly, Tirumala et al. (2022a) find that there is a forgetting baseline during the pre-training of the casual language model (e.g., the model can
memorize at least 40% of the data that appear only once, even being trained on other data for many epochs afterward). These findings further emphasizes the difficulties of avoiding memorization and the potential threats of unintended memorization in large-scale pre-trained language models. Another line of work uses differential privacy to avoid the memorization problem (Abadi et al., 2016; McMahan et al., 2018; Shokri and Shmatikov, 2015), but the mechanism could reduce the accuracy (Jayaraman and Evans, 2019; Feldman and Zhang, 2020; Feldman, 2020; Song and Shmatikov, 2019). Differential privacy also increases the training time, which can further influence the accuracy within the same budget. Therefore there is still no effective and practical way to avoid unintended memorization. Our work further verifies the existence of unintended memorization and makes it more necessary to develop practical defense methods.

heading: 3 Methodology
text: We formulate the targeted training data extraction task as follows: given a source prefix S = (s1, s2, · · · , s|S|) with |S| tokens, the attacker should predict the target suffix T = (t1, t2, · · · , t|T |) with |T | tokens and its confidence. The pair of the given prefix and the predicted suffix (S, T ) should be contained in the pre-training dataset Dpretrain = {(Si, Ti)}, which the attacked model M is trained on. The prediction of the confidence score is necessary for picking out the most probable suffix when we don’t know the ground truth suffix in realistic attack scenarios (i.e., we need to pick out most probable pairs of prefixes and extracted suffixes based on their confidence scores among all predictions). We assume the attacker can obtain some pairs of ground truth prefixes and suffixes Dtrain = {(Si, Ti)|(Si, Ti) ∈ Dpretrain, 1 ≤ i ≤ |Dtrain|} before attacking, which is reasonable because large-scale pre-trained data generally contain public data (e.g., Common Crawl). The attackers can utilize Dtrain to train their attacking models and their goal is to predict suffixes for the prefixes in the test set Dtest = {Si|1 ≤ i ≤ |Dtest|}. Note that the prefix Si in Dtest is included in Dpretrain but is not a part of Dtrain.

heading: 3.1 Method Overview
text: An overview of ETHICIST is shown in Figure 2. We first tune the soft prompt embeddings during training to elicit memorization in the attacked model M
𝑡'(!)
𝑡'(")
with the MLE loss and the additional smoothing loss. The smoothing loss aims to increase the probability of sampling the ground truth suffix. After prompt tuning, we repeatedly sample K suffixes using the attacked model M conditioned on one given prefix and reorder them with our calibrated confidence estimation. Our calibrated confidence estimation can not only select the most possible suffix, but also provide a more accurate confidence score that represents how likely the predicted suffix is correct. Finally, the suffix with the highest confidence is selected as the final prediction.

heading: 3.2 Prompt Tuning with Smoothing Loss
text: We adopt prompt tuning to train the soft prompt tokens on D, which prepends |X| soft tokens X = (x1, x2, · · · , x|X|) before the original input sequence. Then we feed the input to the attacked model M to compute the MLE loss:
LMLE = |T |∑
i=1
− 1|T | logPM (ti|X,S, t<i). (1)
Note that we only tune the parameters of the soft prompt tokens and the parameters of the attacked model M are fixed. We use prompt tuning for two reasons: (1) we do not want to change the original parameters of the attacked model M because the main goal is to elicit memorization in M , and (2) prompt tuning is helpful to improve the training efficiency when M is very large, making ETHICIST able to efficiently adapt to larger language models that generally memorize more training data.
The MLE loss aims to increase the total generation probability of the target suffix T . However,
when using popular sampling methods such as topk sampling (Fan et al., 2018) and top-p (nucleus) sampling (Holtzman et al., 2020) to generate multiple candidate suffixes, we want to ensure the probability of the ground truth suffix token at each time step is not low. Suppose the total probability of the ground truth suffix is high while there is one token in the sequence with a low generation probability. In this case, it is still hard to generate the correct suffix using auto-regressive sampling methods. Therefore, we propose a smoothing loss to make the loss distribution of the suffix sequence more smooth. More specifically, we pick out the top-N tokens with the highest loss values in the whole sequence T . Then we additionally optimize the generation probabilities for these N tokens as follows:
LSmooth = N∑
i=1
− 1 N logPM (tσ(i)|X,S, t<σ(i)),
(2)
where tσ(i) represents the token with the i-th highest loss in T . Note that tσ(i) is dynamically computed during training. The smoothing loss can also be seen as assigning higher weights to the tokens with higher loss values. Finally, we derive the overall loss function as follows:
LTotal = LMLE + αLSmooth, (3) where the coefficient α is a hyperparameter to control the strength of the smoothing loss.

heading: 3.3 Calibrated Confidence Estimation
text: After predicting the suffix, we also need to give a confidence score for the prediction, which can be
meaningfully interpreted as the probability that the output suffix is correct. A naive method is to use the generation likelihood PT = exp(−|T |LMLE) as the confidence score. This naive method is reasonable for picking out the most probable suffix Ti from a collection of sampled suffixes {T1, T2, · · · , TM} for one given prefix. However, it is unsuitable for comparing the confidence of different predicted suffixes corresponding to different prefixes. As the language model is essentially a statistical model, frequencies of tokens and n-grams in the prefixes can greatly influence the absolute generation likelihood of the suffixes. For example, consider two predicted suffixes TA and TB conditioned on two different prefixes SA and SB , where SA and TA contain tokens and n-grams with much higher frequencies. The absolute generation likelihood of TA may be significantly higher than TB , even if they are both ground truth suffixes. Therefore, to eliminate the intrinsic differences in scales of generation likelihood across different suffixes, we propose a novel calibrated confidence estimation method. To calibrate the confidence estimation, we have two considerations: (1) different generated suffixes conditioned on one given prefix should have comparable scales of generation likelihood, and (2) the memorized ground truth suffix is expected to be generated more frequently during multiple generations, which is also validated in Section 5.
Suppose the sampled distinct suffixes are {T1, T2, · · · , TM} for one given prefix, the repeated generation times for these suffixes are {r1, r2, · · · , rM} (i.e., ri denotes how many times Ti is generated among K repeated sampling outputs), and the MLE loss values for these suffixes are {L1MLE,L2MLE, · · · ,LMMLE}. Then we assign the calibrated confidence score to Ti as:
C(Ti) = ri × exp(−|Ti|LiMLE)∑M j=1 rj × exp(−|Tj |L j MLE) . (4)
Through the proposed confidence estimation method, we obtain the confidence score of Ti by comparing it with other sampled suffixes with comparable scales of generation likelihood. In this way, we avoid the scale problem brought by different prefixes and make it practical to compare the predicted suffixes conditioned on different prefixes. Moreover, we leverage the repetition time ri as a valuable signal since memorized suffix is expected to be generated more frequently. Finally, we select the suffix Tbest with the highest confidence score
C(Tbest) among {C(T1), C(T2), · · · , C(TM )} as the predicted suffix and C(Tbest) as its confidence estimation.

heading: 4 Experiments

heading: 4.1 Benchmark
text: We evaluate ETHICIST on the LM-Extraction benchmark2, which is designed for benchmarking targeted training data extraction attacks. It consists of a subset contained in The Pile dataset (Gao et al., 2020). Both the prefix and the suffix are 50 tokens long. All examples are well-specified, meaning that there is only one 50-token suffix in The Pile dataset given the 50-token prefix. What’s more, these examples are all chosen to meet the property that there exists a prefix length (maybe longer than 50) that causes the model to generate the suffix string exactly, which implies that the extraction performance on this benchmark may be higher than that on randomly selected prefixes. We randomly split the dataset into training, validation and test sets. The detailed statistics of the LM-Extraction benchmark are shown in Table 1.

heading: 4.2 Baselines
text: We compare ETHICIST with the following baselines. All the compared baselines first sample K suffixes {T1, T2, · · · , TK} conditioned on one given prefix S and then pick out one suffix as the prediction.
Perplexity It leverages the perplexity (PPL) measured by the attacked language model M as the metric to sort the candidate suffixes and finally chooses the one with the lowest PPL as the predicted suffix T :
T = argmax Ti C(Ti) = argmax Ti
1
PPLM (Ti|S)
Comparing (LM) It takes another language model M ′ and leverages the ratio of the perplexity
2https://github.com/google-research/ lm-extraction-benchmark/
measured by theses two language models as the metric (Carlini et al., 2021):
T = argmax Ti C(Ti) = argmax Ti PPLM ′(Ti|S) PPLM (Ti|S)
The language model M ′ could be a much smaller model trained on the same dataset with M or trained on a different dataset.
Comparing (zlib) Different from Comparing (LM), it uses the zlib (Gailly and Adler, 2004) entropy of the text (i.e., the number of bits after compression with zlib) for comparison (Carlini et al., 2021):
T = argmax Ti C(Ti) = argmax Ti len(zlib(Ti)) PPLM (Ti|S)
Comparing (lowercase) It compares the perplexity of the original text and the lower-cased text measured by the same language model M (Carlini et al., 2021):
T = argmax Ti C(Ti)
= argmax Ti PPLM (lowercased(Ti)|S) PPLM (Ti|S)
Furthermore, we conduct ablation tests by removing the proposed components respectively to investigate the influence of each component.

heading: 4.3 Metrics
text: We adopt the following automatic metrics for evaluation.
Recall The metric computes the percentage of the suffixes that are predicted verbatim over the whole test set. A higher recall score indicates better data extraction ability, which can also be understood as a higher attacking success rate.
RecallEarly stop The metric first sorts the predictions according to their confidence scores and then evaluates the correctness of each prediction one by one. It finally computes the Recall score while making x incorrect predictions. We set x to 100 in our experiments following the LM-Extraction benchmark. A better confidence estimation method can give the correct predictions higher confidence scores and thus lead to a higher RecallEarly stop score.

heading: 4.4 Main Results
text: Table 2 shows the automatic evaluation results with GPT-Neo 1.3B as the backbone. ETHICIST achieves an impressive Recall score of 62.8% and outperforms all the baselines by a large margin, indicating its better ability to extract training data from language models. Moreover, ETHICIST has better confidence estimation performance after calibration as shown by a significantly higher RecallEarly stop score. To further investigate the influence of each component, we run an ablation study. From the results shown in Table 2, it can be seen that both the smoothing loss and the calibrated confidence estimation are important to enhance the ability to extract training data, and combining both of them achieves the best performance. Furthermore, we draw the following conclusions: (1) With prompt tuning and extra training data, we can better induce large-scale language models to generate their memorized training data and successfully achieves a 9.5% performance improvement on Recall and a 12.4% performance improvement on RecallEarly stop. (2) The proposed smoothing loss can further enhance the ability to extract training data, boosting the Recall score from 60.8% to 62.3%. (3) The calibrated confidence provides a 6.3% improvement on RecallEarly stop as expected, demonstrating the importance of calibrating confidence estimation for this task. (4) The smoothing loss is more effective in predicting exact suffixes while the calibrated confidence is more beneficial for identifying highly confident predictions, according to the significant drop in Recall without smoothing and the substantial decrease in RecallEarly stop without calibration. (5) The calibrated confidence estimation is effective regardless of whether using prompt tuning. And it demonstrates greater advantages compared to the comparing (LM) baseline in recognizing predictions with higher confidence when using prompt tuning, indicated by increasing RecallEarly stop (from 48.7 to 52.4).

heading: 4.5 Analysis: Decoding Strategy
text: In our experiments, we use top-p sampling to sample multiple candidate suffixes conditioned on one given prefix. However, there are also other popular decoding methods, including greedy search, beam search, and top-k sampling. We thus compare these popular decoding methods in this section. Table 3 shows the results. Not surprisingly, greedy search performs worst on both Recall and RecallEarly stop,
which suggests some tokens in the ground truth suffix do not have the highest probability at the corresponding positions. Beam search outperforms top-p sampling on Recall, indicating that searching for the suffix with the lowest loss works well to find the ground truth suffix. However, beam search performs significantly worse than top-p sampling on RecallEarly stop, because it cannot use our calibrated confidence. Compared with beam search, top-p sampling can generate multiple candidates, which could substantially increase the accuracy of confidence estimation with our proposed calibrated confidence. Moreover, the top-k sampling performs worse than top-p sampling on RecallEarly stop, which may be because top-k sampling is easier to sample low-probability tokens and thus reduce the confidence of the ground truth suffixes. We finally select top-p sampling as our decoding method due to its balance on Recall and RecallEarly stop.

heading: 4.6 Analysis: Model Scale
text: Previous works on scaling laws find that larger language models can memorize more training data (Carlini et al., 2022b; Tirumala et al., 2022b). Therefore, we are interested in how targeted data extraction performance varies across different model scales. Figure 3 shows the results. We can see that the targeted training data extraction performance continuously increases as the model scale increases from 125 million to 6 billion. ETHICIST shows impressive results as it consistently and
significantly outperforms baselines across different model scales. Thanks to prompt tuning, ETHICIST is efficient in terms of computation time and particularly memory consumption. Therefore, ETHICIST can also be adapted to larger language models for efficient targeted training data extraction.

heading: 4.7 Analysis: Prefix Length and Suffix Length
text: All prefixes and suffixes in the LM-Extraction benchmark are 50 tokens long, making it an interesting question how the length of prefixes and suffixes would affect the extraction performance.
We show the effect of the given prefix length in Figure 4. We can observe that the extraction performance grows approximately linearly with the prefix length for all evaluated methods, and ETHICIST performs best for all prefix lengths. Al-
though all methods have similar growth speed on Recall, ETHICIST has the highest growth speed on RecallEarly stop. It is also interesting that Comparing (LM) only outperforms Perplexity when given
Prefix
prefixes that are long enough. We show the effect of the predicted suffix length in Figure 5. For all three methods, the extraction performance decreases when the suffix length increases. Different from the approximately linear relationship between the prefix length and the extraction performance, the performance degradation tends to become progressively slower as the suffix length increases. This suggests that the model can still memorize a considerable proportion of suffixes (rather than quickly decreasing to zero) even if the predicted suffix length continues to increase. What’s more, we observe that ETHICIST has a significantly slower speed of performance degradation compared with the two baselines, which suggests ETHICIST is effective for eliciting deeper memorization of longer suffixes of the attacked model.

heading: 4.8 Analysis: Sampling Time
text: Due to space limitations, we put the analysis of sampling time in Appendix C.

heading: 5 Discussion
text: We further show some statistical features in Table 4. We can see that the memorized suffixes can be sampled significantly more frequently with a high average repeat time of 85.38, validating that the repeat time is a valuable signal for confidence estimation. What’s more, the memorized suffixes have significantly higher confidence. One interesting phenomenon we observe is that if the ground truth suffix can be generated, it mostly has the top 3 highest confidence (Recall@3 ≈ Recall@100). We also find that for more than 30% of the prefixes, the model cannot generate the correct prefix even given 100 chances. Therefore, an important future direction is to design better methods to elicit memorization in the attacked model. Considering the non-negligible gap between Recall@1 and Recall@100 (0.63 vs. 0.69), another important future direction is to design better confidence estimation methods (maybe trainable), which can pick out the ground truth suffix among the collection of candidate suffixes for one prefix.
We show a case in Figure 6. Although the first predicted suffix has higher loss than the second predicted suffix, it is sampled far more times than the latter. Therefore, we assign higher confidence to the first suffix using our calibrated confidence estimation method. We further show the probability of generating each token during the sampling process in Figure 7. We can observe that although the correct prediction has higher loss as a whole, it keeps a high sampling probability across the generation process. The minimum probability of generating one token in the correct suffix is about 0.45, which is significantly higher than 0.1 for the wrong suffix.
Therefore it is easier to generate the correct suffix, which leads to a higher confidence score. This is also in line with our motivation for designing the extra smoothing loss, which can increase the probability of sampling the correct suffix.

heading: 6 Conclusion
text: In this work, we propose ETHICIST, an effective method for targeted training data extraction attack. ETHICIST uses soft prompt to elicit memorization in the attacked model. To ensure the probability of the ground truth suffix token at each time step is not low, we propose a smoothing loss besides the standard MLE loss. We also propose a calibrated confidence estimation method to calibrate the scale of confidence across different samples. Experiments on the LM-Extraction benchmark demonstrate that ETHICIST significantly improves the extraction performance. We further conduct extensive experiments to investigate several critical factors influencing the extraction performance, including decoding strategy, model scale, prefix length, and suffix length. We hope our work can promote future researches on better attack methods and practical defense methods for the training data extraction problem.

heading: Acknowledgement
text: This work was supported by the NSFC projects (Key project with No. 61936010 ). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.
Limitations
Although we conduct experiments across various model scales ranging from 125M to 6B, there are still larger language models we don’t test either because their training data is not publicly released or because we have limited resources.
Moreover, the examples in the LM-Extraction benchmark are all chosen to meet the property that there exists a prefix length (maybe longer than 50) that causes the model to generate the suffix string exactly, which makes the extraction performance on this benchmark higher than that on randomly selected prefixes.
Ethics Statement
ETHICIST is a powerful method to elicit memorization in the large pre-trained language models,
which makes it a useful tool to expose the privacy risk of large language models. However, it also has a risk to be abused by attackers to extract privacy information from pre-trained language models. Thus large language models should be carefully examined before being made publicly available. What’s more, it is necessary to develop defense methods against the training data extraction attacks without sacrificing the language modeling ability.
The LM-Extraction benchmark is derived from the Pile dataset, and thus covers many domains including books, code, emails, etc. This suggests the effectiveness of targeted training data extraction across different domains.

heading: B Computing Infrastructure
text: All experiments are carried out on a single Tesla V100 GPU with 32GB memory. Each experiment can be completed in less than 20 hours.

heading: C Effect of Sampling Time
text: In our main experiments, we sample 100 candidate suffixes for one given prefix. We show the effect of sampling time in Figure 8. We can see that all methods’ performances increase quickly when the sampling time increases from 1 to 10. However, ETHICIST’s performance can still improve slowly when the sampling time increases from 10 to 100, which we attribute to the consideration of repeat time in our calibrated confidence estimation. What’s more, although we report the result for sampling 100 times in our main experiments, we can see that ETHICIST can achieve satisfying performance when sampling only 10 times, which
suggests the efficiency of ETHICIST.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and Section Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section Methodology
3 B1. Did you cite the creators of artifacts you used? Section References
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? They are discussed in our github repo.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section Experiments
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The publisher of the data did the work.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We don’t create new data.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section Experiments
C 3 Did you run computational experiments? Section Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section Implementation Details and Section Computing Infrastructure in the Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section Implementation Details in the Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section Experiments
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section Implementation Details
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope our work can promote future researches on better attack methods and practical defense methods for the training data extraction problem