Data Contamination Calibration for Black-box LLMs (2024)

Abstract: The rapid advancements of Large Language Models (LLMs) are tightly associated with the expansion of the training data size. However, the unchecked ultra-large-scale training sets introduce a series of potential risks like data contamination, i.e. the benchmark data is used for training. In this work, we propose a holistic method named Polarized Augment Calibration (PAC) along with a brand-new dataset named StackMIA to help detect the contaminated data and diminish the contamination effect. PAC extends the popular MIA (Membership Inference Attack) — from the machine learning community — by forming a more global target for detecting training data to clarify invisible training data. As a pioneering work, PAC is very much plug-and-play that can be integrated with most (if not all) current whiteand black-box (for the first time) LLMs. By extensive experiments, PAC outperforms existing methods by at least 4.5%, in data contamination detection on more than 4 dataset formats, with more than 10 base LLMs. Besides, our application in real-world scenarios highlights the prominent presence of contamination and related issues. 1

heading: 1 Introduction
text: As is widely acknowledged, the rapid advancements of Large Language Models (LLMs) in natural language tasks are largely attributed to the incredible expansion of the size of the training data (Kaplan et al., 2020). Despite the massive successes, this unmanaged expansion has introduced a series of significant issues that are yet explored, particularly data contamination. This issue arises notably when the benchmarking data is inadvertently included in the training sets. This contamination leads to misleading evaluation results (Zhou et al., 2023; Narayanan and Kapoor, 2023), thus deducing difficulties in acquiring effective and secure
1Our code is available: https://github.com/yyy01/PAC.
models. Additionally, training on datasets with copyrighted, private, or harmful content could violate laws, infringe on privacy, and introduce biases (Carlini et al., 2019; Nasr et al., 2018b). Unlike in earlier stages of machine learning, we posit that this problem would be much more prevalent in the age of the LLMs, very much due to the inevitable lack of scrutinization of the much-scaled — and often private — training data (Magar and Schwartz, 2022; Dodge et al., 2021).
In this work, we position the training data detection for LLMs as an extension of the membership inference attacks (MIA) (Shokri et al., 2017) in the literature of machine learning. MIA targets distinguishing whether the given data samples are members (training data) or non-members (not be trained). The previous line of work can generally be categorized into score-based (i.e., calibrationfree) (Yeom et al., 2018; Salem et al., 2019; Shi et al., 2023) and calibration-based (Watson et al.,
10845
2021; Carlini et al., 2022; Mattern et al., 2023) methodologies. Despite the promise, these approaches hardly suffice for current LLMs. The conventional setup in machine learning may generally focus on a small-scale training set, accompanied by global confidence distribution differences between members and non-members. However, this assumption no longer holds in LLMs, leading to a situation where non-members can also exhibit misleadingly high confidence levels, as shown in Figure 2. In addition, the MIA approaches generally rely on training an external reference or proxy model using member data distribution approximated to the targeted model. This, unfortunately, has contradicted the black-box setup of many current LLMs.
With this paper, we make two major efforts: (i)-a holistic scheme named polarized augment calibration (PAC) to resolve the challenges of black-box training data detection and (ii)-a brand new dataset for newly released LLMs.
Notably, the current related methods mostly assess the overfitting metric at individual points. Upon revisiting the issue of typical MIA, we find that the geometric properties of the samples may better reflect the latent differences introduced by training. For members, they should exhibit two characteristics: (i) high confidence at individual points; and (ii) being part of a poorly generalized local manifold. In other words, if a sample lies in a poor-generalization region but shows anomalous confidence, it suggests that the model is merely memorizing (or overfitting) rather than truly generalizing.
Based on such observations, as depicted in Figure 1, we propose Polarized Augment Calibration (PAC), which explores the confidence discrepancies in local regions through data augmentation tech-
niques, aimed at detecting those overfitted training samples. Upon this, to address the bias of existing global confidence metrics, we have developed a brand-new evaluation score named polarized distance, as a polarized calculation of flagged tokens with far and near local regions in the probability space. We further provide an in-depth explanation of PAC detection from a theoretical perspective. Compared to previous relevant methods relying on fully probability-accessed LLMs, we introduce a new probabilistic tracking method to extract probabilities under limited conditions(e.g., OpenAI API) for the first time.2 This becomes the trailblazers to extend our detection to a black-box setup where only access to partial probabilities is permitted, enabling PAC to be applied to almost all LLMs.
For the benchmark, the current available benchmark for the contamination problem is very limited. We construct and introduce StackMIA tailored for reliable and scalable detection of the latest LLMs. It ensures reliability by adopting a timebased member/non-member classification similar to (Shi et al., 2023), where members are visible during pre-training and non-members are not. StackMIA is dynamically updated, by offering detailed timestamps in order to quickly adapt to any up-todate LLM through our curation pipeline. By contrast, the existing benchmark WikiMIA(Shi et al., 2023) does not possess these properties, making it hard to use, especially on the LLMs released post-2023.
Last but not least, we exhibit extensive experiments on 10 commonly used models to evaluate the PAC against six existing major baselines. These results demonstrate that PAC outperforms the strongest existing baseline by 5.9% and 4.5% in the AUC score respectively on StackMIA and WikiMIA. PAC further expresses superior robustness under conditions of ambiguous memory or detection of fine-tuning data. To further validate the effectiveness of PAC in real-world applications, we provide a set of case studies for data contamination and different security risks (Carlini et al., 2019; Nasr et al., 2018b,a) on ChatGPT and GPT-4. The conclusions from these examples highlight the ubiquity of security risks upon widespread deployment (Ye et al., 2023).
Our contributions can be summarized as:
1. We introduce PAC, an innovative, theorysupported MIA method for black-box LLMs that does not rely on external models, consis-
tently outperforming leading approaches.
2. Our black-box probability extraction algorithm makes PAC a trailblazer for LLMs with restricted probability access. We also unveil the StackMIA benchmark, addressing the gaps in MIA datasets for pretraining phrases.
3. Applying PAC to ChatGPT and GPT-4 highlights the prevalent data contamination issue, prompting a call to the academic community for solutions to ensure safer, more dependable LLMs.

heading: 2 Related Work
text: Data Contamination. As Magar and Schwartz (2022) mentioned, data contamination is the infiltration of downstream test data into the pretraining corpus, which may seriously mislead evaluation results. Dodge et al. (2021) and Brown et al. (2020) highlighted the tangible presence of contamination issues in models (e.g. GPT-3) and corpora (e.g. C4). Such contamination risks models memorizing rather than learning, affecting their exploitation (Magar and Schwartz, 2022). Detection methods have been proposed, using n-gram overlap ratios against pre-training data to identify contaminated samples (Du et al., 2022; Wei et al., 2021; Chowdhery et al., 2023). However, these existing methods rely on access to the pretraining corpus, which is unfeasible for many LLMs. Recent studies (Golchin and Surdeanu, 2023; Weller et al., 2023) shifted towards a more common black-box setting, by extracting outputs with trigger tokens (Carlini et al., 2021) or specified prompts (Sainz et al., 2023; Nasr et al.) from LLMs as contaminated samples and comparing with the test set. Unfortunately, the extracted samples are usually too broad, making these methods ineffective for detecting given samples.
Membership Inference Attack. Membership Inference Attacks (MIA), proposed by Shokri et al. (2017), is defined as determining whether a given sample is part of the training set. MIA is predicated on the models’ inevitable overfitting (Yeom et al., 2018), leading to a differential performance on training samples (members) versus non-members. MIA has been applied in privacy protection (Jayaraman and Evans, 2019; Zanella-Béguelin et al., 2020; Nasr et al., 2021, 2023; Steinke et al., 2023), machine-generated text detection (Mitchell et al., 2023; Solaiman et al., 2019), DNA inference (Zer-
houni and Nabel, 2008), etc. Given the limited access to most current LLMs, research has concentrated on black-box conditions. Typical scorebased methods employ loss or partial token probabilities (Shi et al., 2023), though these may incorrectly flag non-members (Carlini et al., 2022). Watson et al. (2021) proposed calibration-based methods to utilize a difficulty calibration score to regularize raw scores. Specifically, Carlini et al. (2021), Ye et al. (2022), and Mireshghallah et al. (2022) train reference models to rectify anomalies with the average of different models. Mattern et al. (2023) might be most closely aligned with our work, utilizing additional models to generate similar samples for calibration purposes. while Yet, the practicality is constrained by their need for extra models. Furthermore, obtaining the necessary probabilities for MIA is challenging in many recent LLMs. To our knowledge, we are the pioneers in developing a calibration-based detection method jointly considering the calibration of global confidence and local distribution.

heading: 3 Problem Definition and Preliminary Work
text: We first provide a comprehensive formal definition of the training data detection for LLMs in the context of a two-stage training process (Kenton and Toutanova, 2019). Upon this foundation, we introduce a new dynamic data benchmark (Section 3.2) that can be applied to recently released LLMs. By open-sourcing the benchmark upon publication, we expect to use it to foster further research in the community.

heading: 3.1 Problem Formulation
text: Two-stage training. The training process of LLMs consists of two stages: unsupervised pre-training on a large-scale corpora, and supervised fine-tuning of labeled data for downstream tasks.
Given training set D = {zi}i∈|D|, zi for the finetuning stage can be further represented as xi ∪ yi (x, y respectively represent input and label). Both zi and xi can be denoted as a tokens sequence {uj}j∈|xi|or|zi|. Generally following (Brown et al., 2020), the LM fθ is trained in two stages by maximizing the following likelihood separately:
Lpt(D) = ∑
i ∑ j log fθ(uj |u1, · · ·uj−1)
Lft(D) = ∑ i log fθ(yi|u1, · · · , u|xi|) (1)
where i, j respectively denote the index of the sample z and the index of the token within z.
Training Data Detection. Following the settings of MIA (Yeom et al., 2018; Mattern et al., 2023), for a given the target model fθ and the sample z, the objective of training data detection can be defined as learning a detectorA : (z, fθ)→ {0, 1}, where 1 denotes that z is member data (z ∈ D) and 0 denotes that z /∈ D. As mentioned in Section 2, widely used calibration-free detection methods directly construct computational scores (denoted as L), such as the loss, and threshold them:
A(z, fθ) = 1(L(z, fθ) > ϵ) (2)
By comparing L with a predefined threshold ϵ, A can achieve the detection of members or nonmembers. For more recent calibration-based methods, a calibration function c is additionally introduced to correct for the detector’s bias relative to the target distribution. Calibration is typically achieved by correcting the original scores with the calibration function. The calibration function is usually a designed score c : (z̃, fϕ)→ R based on the differential performance on adjacent samples z̃ (Mattern et al., 2023) or reference models fϕ (Watson et al., 2021). The reference models are usually additional models trained on a similar training data distribution. Consequently, the detector A can be extended as:
A(z, fθ) = 1(L(z, fθ)− c(z̃, fϕ) > ϵ) (3)
Following the standard setup (Shi et al., 2023), we primarily constrain detecting pre-training samples under black-box settings. Specifically, the detection during the fine-tuning stage will be further discussed in Section 5.6.

heading: 3.2 Dynamic Benchmark Construction
text: StackMIA2 is based on the Stack Exchange dataset 3, which is widely used for pre-training. Specifically, we organize member and non-member data with fine-grained release times to ensure reliability and applicability to newly released LLMs. More Details are provided in Appendix A.
Data collection and organization. We utilize the data source provided by the official (Appendix A). Each record contains a post and answers from different users. Data collection: Following the data selection strategy mentioned by LLMs such as LLaMA (Touvron et al.), etc, we retain data from
2The StackMIAsub benchmark dataset is available here: https://huggingface.co/datasets/darklight03/StackMIA
3https://archive.org/details/stackexchange
the 20 largest websites, including common themes like English, math, etc. Subsequently, based on the training timelines of most LLMs (Zhao et al., 2023), we set January 1, 2017, as the latest cutoff date for member data, i.e., data with both posts and answers dated before this time are considered members. For non-members, January 1, 2023, is set as the earliest occurrence time. Lastly, we build an automatic pipeline to remove HTML tags from the text. Data filtering: Considering the potential differences in answer sorting strategies (Ouyang et al., 2022), we only retain the post records. We apply automatic filtering based on: (1) posts contain only texts, not formulas or code; (2) posts have not been asked repeatedly. Data organization: To ensure applicability to a vast array of LLMs released after 2023, we reorganize the non-member data with fine-grained precision. We selected approximately 2000 posts created within each month, using the month as a cutoff point. This allows future users to construct subsets of StackMIA suitable for newly released LLMs through our provided pipeline.
Benchmark test. Following the settings of (Shi et al., 2023), we divide the original WikiMIA by length. Simultaneously, we construct StackMIAsub4 dataset (8267 samples in total, Appendix A) to conduct experiments in this paper. Specifically, we set May 1, 2023, as the cutoff date for non-member data. Following (Shi et al., 2023), we sequentially select approximately balanced sets of member and non-member data by length. Additionally, we use GPT-3.5-turbo5 to construct (Appendix A) synonymous rewritten data to test the stability of the methods under the condition of approximate memory (Ishihara, 2023). Referring to (Ippolito et al., 2022), we set BLEU (Papineni et al., 2002)> 0.75 as a condition to ensure semantic consistency in the rewritten data.

heading: 4 Methodology
text: We introduce Polarized Augment Calibration (PAC), an efficient and novel calibration-based training data detection method. The key idea is to construct adjacent samples using easy data augmentation for calibrating a generalized distribution and design a brand-new polarized distance to enhance the salience. We further propose a probabilistic tracking method suitable for models with partially
4The StackMIAsub benchmark dataset is available here: https://huggingface.co/datasets/darklight03/StackMIAsub
5https://platform.openai.com/docs/api-reference
inaccessible logits for the first time.

heading: 4.1 Generating Adjacent Samples
text: As mentioned in Section 3.1, for a given data point z, we construct an adjacent sample space z̃ for the calibration function c. We opt for a simpler word-level perturbation approach through the Easy Data Augment (Wei and Zou, 2019) framework to generate z̃, which are adjacent in the local distribution with z. The calibration through these adjacent data points prevents calibration-free scores from being confused with misleading highconfidence, especially when the model provides a well-generalized distribution of non-members (Choquette-Choo et al., 2021). Specifically, we randomly swap 2 tokens from z, and repeat this process m times:
z̃ = σm(z) = σm({uj}j∈[1,|z|]) (4)
where σm represents a bijection from z to itself (i.e., a permutation) after m random swaps. Further, different from previous works, the augmentationbased scheme focuses the calibration on local distribution and is much more efficient due to the avoidance of introducing additional models.

heading: 4.2 Polarized Distance
text: Due to the challenges in constructing reference models for LLMs, we calibrate directly using the difference between z̃ and z without introducing costly external models. As mentioned in Section 4.1, augmentation-based z̃ tends to exhibit nonmember characteristics more. In practice, using traditional confidence scores, e.g. loss or perplexity, as the L score fails to demonstrate stable significance. (Shi et al., 2023) proposes to improve classification effectiveness by calculating only a portion of the low-probability outlier words. We integrate this technique and expand it to focus simultaneously on far and close local regions of the token probability, achieving a significant measurement in probability space. Specifically, consider a sequence of tokens for a given sample z, denoted as z = {uj}j∈[1,|z|]. According to Equation 1, the log-probability of each token ui can be denoted as log fθ(ui|u1, · · · , ui−1). As depicted in Figure 1, we then sort the probabilities of each token and select the largest k1% and the smallest k2% to form sets, denoted as MAX(z, k1) and MIN(z, k2), respectively. Subsequently, the polarized distance
LM can be denoted as:
LM = 1K1 ∑
ui∈MAX(z,k1) log fθ(ui|u1, · · · , ui−1)
− 1K2 ∑
ui∈MIN(z,k2) log fθ(ui|u1, · · · , ui−1)
(5) where K1 and K2 denotes the size of MAX(z, k1) and MIN(z, k2) set separately.
General. According to the previous sections, the implementation of PAC can be represented as:
A(z, fθ) = 1[LM (z, fθ)− N∑ LM (σm(z),fϕ)
N > ϵ] (6)
where N denotes the number of repetitions to reduce random errors.

heading: 4.3 Theoretical Analysis
text: The explanation for the approach of PAC is quite straightforward: through carefully designed discrete perturbations, it makes z̃ (i.e., σm(z)) exhibit non-member characteristics. Since LLMs typically rely on original natural corpora, such perturbations can confuse the model (Jin et al., 2020; Morris et al., 2020; Li et al., 2021), thereby obtaining measurable differences between z and z̃. By calculating the difference in the probability distribution of far and near local regions (expressed as the highest and lowest probabilities), LM can reflect the model’s predictive uncertainty (Duan et al., 2023) and volatility. Given a sample z, when it occurs:
LM (z, fθ)≫ LM (σm(z), fθ) (7)
This means that the impact of the perturbations is significant, which indicates that z may be overfitting. In this case, z will be classified as a member sample. Furthermore, we provide a simple detailed mathematical proof in Appendix C.

heading: 4.4 Black-box Probabilistic Tracking
text: Almost all detection methods require accessing the probabilities of all tokens for z, which is not feasible for some current black-box models, such as GPT-4 (Achiam et al., 2023) accessed by official API. These models only provide a logprobability query interface for the top n words, where n ⩽ 5 usually. To address this issue, we take the GPT models as an example and construct a black-box probabilistic tracking algorithm using the logit_bias6 function provided by the OpenAI
6https://platform.openai.com/docs/apireference/completions/create#completions-create-logit_bias
API to track probability outputs. Such a function allows for setting biases of the logits for specific token IDs, which can be obtained through the tiktoken library7. Utilizing this feature, for each token in turn ui ∈ z, we enumerate biases added to the corresponding token ID until the top n probability query results are just altered. The obtained bias threshold γi can be approximated as the difference between the log-probability of ui and the known token uτ (see Appendix D for the proof). Thus, the probabilities of all tokens in z can be obtained by:
log fθ(ui|·) = log fθ(uτ |·)− γi (8)
where · represents the prefix tokens of ui. Due to the monotonicity of the impact of bias growth on query results, the enumeration process can be optimized for the binary search. Thus, we achieve a reduction in time complexity from O(N) to O(logN) for obtaining the log probability of a single token with a logit of N , which is cost-effective and efficient.
With this extraction method, PAC becomes the first method capable of detecting training data from almost any black-box LLMs.

heading: 4.5 Pseudo Code
text: We provide a simple pseudocode (as shown in Algorithm 1) to illustrate the specific implementation steps of PAC.
Algorithm 1 Polarized Augment Calibration Input: given data sample z = {ui}i∈|z|, source model fθ, and decision threshold ϵ Output: True – z is a member sample, False – z is a non-member sample.
1: Get the augmented sample z̃ with random swap, repeating m times 2: Select the highest K1 probability tokens and lowerst K2 probability tokens to construct MAX and MIN set 3: Calculate polarized distance LM (z) 4: L ← 1K1 ∑ MAX log fθ(u|·)− 1K2 ∑ MIN
log fθ(u|·) 5: d← LM (z)− LM (z̃) 6: return True if d > ϵ else False
7https://github.com/openai/tiktoken

heading: 5 Experiments

heading: 5.1 Experiments Settings
text: Baseline Methods: We selected six popular methods to evaluate our approach: four calibrationbased and two calibration-free. Calibrationbased methods include: the Neighborhood attack (Neighbor) (Mattern et al., 2023), which assesses loss differences between original samples and their neighbors generated by masked language models; and perplexity-based calibration (Carlini et al., 2021) techniques utilizing Zlib entropy (Zlib) (Gailly and Adler, 2004), lowercased sample perplexity (Lower), and comparisons with reference models trained on the same dataset (Ref). Calibration-free methods comprise the Min-K% method (Shi et al., 2023), predicting pre-trained samples through low-probability outlier words; and the Loss Attack (Yeom et al., 2018), substituting loss with Perplexity (PPL) in LLMs.
Datasets and Metric. We utilize the StackMIAsub benchmark (Section 3.2) and the WikiMIA dataset proposed by (Shi et al., 2023). WikiMIA (Appendix B) leverages Wikipedia timestamps and model release dates to identify member and nonmember data sets, applicable for LLMs trained up to 2023. Both datasets are transformed into two formats as the guidelines in Section 3.2: the original format (ori) and the synonym rewritten format (syn).
For evaluation, we follow (Mattern et al., 2023; Carlini et al., 2022; Watson et al., 2021) and plot the ROC curve analysis method. To facilitate numerical comparison, we primarily use the AUC score (Area Under the ROC Curve). The AUC score (Appendix F), independent of any specific threshold, accurately gauges the method’s ability to differentiate between members and non-members. It also eliminates bias from threshold selection.
Models. We conduct experiments against 10 commonly used LLMs. Six models are applicable for both WikiMIA and StackMIA, including LLaMA-13B (Touvron et al.), LLaMA2-13B (Touvron et al., 2023), GPT-J-6B (Wang and Komatsuzaki, 2021), GPT-Neo-2.7B (Black et al., 2021), OPT-6.7B (Zhang et al., 2022), and Pythia-6.9B (Biderman et al., 2023). The two GPT-3 base models, Davinci-002 and Baggage-002 (Ouyang et al., 2022), are suited for the WikiMIA dataset. Additionally, two newer models are applicable for StackMIA, including StableLM-7B (Tow et al., 2023) and Falcon-7B (Almazrouei et al., 2023).

heading: 5.2 Implements
text: According to Section 4, the key hyper-parameters affecting the PAC include the times of perturbations m, the tokens ratio in min-max distance k1 and k2, and the number of adjacent samples N . To ensure efficiency, N is globally fixed at 5. Based on this, we conduct a grid search (Liashchynskyi and Liashchynskyi, 2019) on a reserved small-scale validation set of StackMIA using LLaMA-13B. The final settings are k1 = 5, k2 = 30, and m = 0.3×|z|, where |z| denotes the token number of z.
5.3 PAC as A More Effective Detector
Based on the settings described in Section 5.1, the primary comparison results between PAC and the baseline methods are listed in Table 1. The experimental outcomes indicate that PAC consistently outperforms across all models and all data formats. Specifically, PAC shows an average AUC score improvement of 4.5% on WikiMIA and 5.9% on StackMIAsub compared to all other baseline methods. Moreover, PAC maintains robust performance even under the conditions of synonymously approximate memories. Notably, the Min-K% method exhibits the second-best performance in all settings, validating the reliability of using local regions of token probabilities. And different from previous methods, PAC exhibits prominence in member recognition (Figure 2). In summary, PAC is an effective and versatile solution for detecting pre-training data of LLMs.

heading: 5.4 Ablation Study
text: To further validate the design of augment calibration and polarized distance, we conduct ablation studies on (1) methods for generating adjacent samples, and (2) metric scores. We limit the model to LLaMA-13B as an example.
Generation method. We evaluate the performance of four popular methods to generate adjacent samples. Table 2 demonstrates a clear conclusion: augmentation based on random swaps is far more effective than any other generation method. We believe the underlying reason is that the swap operation ensures better non-member attributes, making the metric more significant.
Metric Score. Similarly, we compared different metric scores in terms of their significance in difficulty calibration. As shown in Table 2, the polarized distance more readily facilitates the distinction between members and non-members.

heading: 5.5 Analysis Study
text: To explore the factors influencing the detection, we focus on the four aspects, using LLaMA-13B and Pythia series as examples. All results are shown in Figure 3.
Memorization times. We employ continual pretraining to increase the times that a member is seen by the model. The results show that detection difficulty decreases with memorization times increase, likely due to an increase in the degree of overfitting.
Number of adjacent samples. We varied the number of adjacent samples from one to twice the number of adjacent samples we used. The results demonstrate that with an increase in quantity, PAC maintain robust performance after initial improvements.
Sample Length. As the length of samples varies, PAC tends to achieve better performance in both lower and higher lengths. This is likely because they respectively contain more distinctive features and more information, making detection easier.
Model Size. The performance of PAC continuously improves with an increase in model parameters. This may be due to larger models having a stronger learning capacity within constant training iterations.
5.6 PAC as A Two-Stage Detector
As more developers utilize various domain-specific data to fine-tune the same foundational models, the expansion of detection capabilities to the finetuning stage becomes increasingly critical. Limited
works (Song and Shmatikov, 2019; Mahloujifar et al., 2021) have addressed this issue but are not extendable to a two-stage process. We select a recent clear fine-tuning dataset after contamination check, Platypus (Lee et al., 2023), to fine-tune the LLaMA-13B model under a 5 epoch set. To further simulate real-world scenarios, we conducted detection with both output portions and entire samples. As Figure 4 shows, PAC still exhibits excellent and stable performance even compared to the PPL score, which is directly equivalent to the training objective.
5.7 PAC as A Robust Threshold Interpreter
As mentioned in Section 3.1, the selection of threshold ϵ affects the final detection effectiveness. Thus, we focus on the threshold obtained in the scenario where the knowledge about all samples is limited. We randomly select 10%-50% (in 5% intervals) of the original dataset to form subsets. Then, following Lipton et al. (2014), we threshold PAC by maximizing the F1-score. As shown in Table 3, PAC consistently outperforms baselines by at least 3% in accuracy. Simultaneously, the variance of the threshold ϵ obtained through subsets is relatively small (around 0.1), indicating that PAC requires only a small proportion of data to acquire a robust threshold, which is not limited by access-restricted
data.

heading: 6 Case Study: Date Contamination
text: To further uncover the potential risks of existing LLMs (Large Language Models) through PAC, we selected two logical reasoning datasets, GSM8K (Cobbe et al., 2021) and AQuA (Garcia et al., 2020), and one ethical bias investigation dataset, TOXIGEN (Hartvigsen et al., 2022) on GPTs LLMs. As depicted in Table 4, both GPT-3 and the more advanced ChatGPT and GPT-4 exhibited varying degrees of contamination, reaching up to 91.4% on davinci-002. Furthermore, all models unfortunately showed severe ethical bias data contamination in the training set (Figure 5). Based on the above, we call on the community to focus on finding solutions to the contamination problem to develop safer and more robust LLMs.

heading: 7 Conclusion
text: We introduce Polarized Augment Calibration (PAC), a groundbreaking approach that expands the Membership Inference Attack (MIA) framework to detect training data in black-box LLMs. PAC unveils a new angle for MIA by utilizing confidence discrepancies across spatial data distributions and innovatively considering both distant and proximal probability regions to refine confidence metrics. This method is rigorously backed by theory and proven through comprehensive testing. We also present a novel detection technique for API-based black-box models using a proprietary probability tracking algorithm and launch StackMIA, a dataset aimed at overcoming the limitations of existing pre-trained data detection datasets. Applying PAC exposes widespread data contamination issues in even the most advanced LLMs, urging a communal effort towards addressing these challenges.

heading: 8 Limitations
text: While PAC shows promising results in detecting training data contamination in LLMs, its full potential is yet to be realized due to certain constraints. The limited availability of detailed training data information from LLMs providers restricts comprehensive validation across diverse models, underscoring the method’s novelty yet implicating its untapped applicability. Additionally, the efficacy of PAC could be further enhanced with a more varied dataset, suggesting its adaptability and scope for refinement in varied LLMs. However, our current computational resources limit the extent of experiments, particularly on larger-scale LLMs, hinting at the method’s scalability potential yet to be fully explored.

heading: Acknowledgements
text: This work is supported by the Pioneer R&D Program of Zhejiang (No. 2024C01035), NSFC under Grants (No. 62206247), and the Fundamental Research Funds for the Central Universities (No. 226-2024-00049).

heading: A Details of StackMIA
text: A.1 Stack Exchange
The Stack Exchange Data Dump contains usercontributed content on the Stack Exchange network. It is one of the largest publicly available repositories of question-answer pairs and covers a wide range of subjects –from programming to gardening, to Buddhism. Many large language models therefore include this dataset in their training data to enrich the training data and improve the model’s ability to answer questions in different domains.
A.2 StackMIAsub
We follow Shi et al. (2023) to divide all the samples we collected into 4 categories based on length, ranging from 32 words to 256 words. The specific composition of StackMIA is shown in the Table 5.
A.3 Synonymous Rewritten Data
As mentioned in Section 3.2, we rewrite the original samples to simulate approximate memorization scenarios by prompting GPT-3.5-turbo API. Part of the prompts we used to rewrite the sentence are listed as follows:
• “Rewrite the sentence with the smallest possible margin, keeping the same semantics and do not complete anything, ensuring that BLEU > 0.75 before and after the change: ‘text’ ”
• “Slightly rewrite the following sentence without changing the sentence structure, and do not complete any sentence: ‘text’ ”
• “Randomly replace 2 words in the giving sentence with synonyms: ‘text’ ”
• “Replace 5 percent of the prepositions with a synonym in the giving sentence: ‘text’ ”
The case comparison of the samples before and after the synonymous transformation with ChatGPT is listed in Table 7.

heading: B Details of WikiMIA
text: WikiMIA is a benchmark for MIA((Shi et al., 2023)), with data sourced from Wikipedia. For non-member data, the dataset collects recent event pages using January 1, 2023, as the cutoff date. For member data, the dataset collects articles before 2017. The specific composition of WikiMIA is shown in the Table 6.

heading: C A Simple Proof of PAC method
text: Here we provide a mathematical proof why our method can effectively distinguishes members and non-members.
As mentioned in Equation 5, given a sample z, the Polarized Distance can be briefly represented as the difference between the two terms T1 and T2:
T1 = 1 K1 ∑ ui∈MAX(z,k1) logfθ(ui|u1, · · · , ui−1) T2 = 1 K2
∑ ui∈MIN(z,k2) logfθ(ui|u1, · · · , ui−1)
(9) Each term can be scaled to the following inequality:
|z| K1 T1 ⩾ Eu∈z[log fθ(u|·)] |z| K2 T2 ⩽ Eu∈z[log fθ(u|·)] (10)
where E function denotes the expectation function and · denotes the prefix tokens sequence. Then the MinMax Distance can be represented as:
0 ⩾ LM = T1 − T2 ⩾ K1−K2|z| Eu∈z[log fθ(u|·)] (11)
Then Equation 6 can be further converted to:
K1−K2 |z| Eu∈z[log fθ(u|·)] ⩽ LM (z)− LM (σm(z)) ⩽
K2−K1 |σm(z)|Eu∈σm(z)[log fθ(u|·)]
(12)
Since the coefficient is constant, the threshold interval of the final calculated indicator can be equivalently expressed as:
Eu∈z[log fθ(u|·)] + Eu∈σm(z)[log fθ(u|·)] (13)
Among them, the former term can be considered as a variant of the LLM training objective. Therefore, the value of Formula 13 will change significantly as z is fitted as member data. Moreover, there exists a positive correlation between the posterior term and the forward tendency according to Jin et al. (2020). This means that the threshold intervals of members and non-members are more sparse than solely using the probability of z directly. Therefore, our indicator significantly captures differences between members and non-members.

heading: D Proof of Probability Extraction
text: Assume that the logit output of the model fθ(ui|·) is l1, · · · , lN , and then the log-probability of ui can be represented as:
log fθ(ui|·) = log e li∑M
j=1 e lj
= li − log ∑N j=1 e lj
(14) Then, the log-probability log f ′θ(ui|·) after adding a fixed bias γi to the logit of ui can be calculated as:
log f ′θ(ui|·) ≈ log e li+γi∑N j=1 e lj
= (li + γi)− log ∑N j=1 e lj
(15)
Then the original log-probability can be calculated as:
log fθ(ui|·) ≈ log f ′θ(ui|·)− γi (16)

heading: E Baseline Details
text: E.1 Reference model comparison
Reference model-based methods target at training reference models in the same manner as the target model (e.g., on the shadow data sampled from the same underlying pretraining data distribution). The raw score of the original samples can be calibrated with the average of the score in these reference models. Due to the high cost of strictly training a shadow LLM, we follow Carlini et al. (2021) to choose a much smaller model trained on the same underlying dataset. Specifically, we choose LLaMA-7B as the reference model of LLaMA13B, LLaMA2-7B for LLaMA2-13B, GPT-Neo125M for GPT-Neo-2.7B, OPT-125M for OPT6.7B, Pythia-70M for Pythia-6.9B, StableLM 3B for StableLM-7B. Specially, for those LLMs without smaller models in the series, we use an approximate model trained on the mentioned same-dataset or distribution in their official statement, including GPT-Neo-125M for GPT-J-6B both trained on
the Pile (Gao et al.), GPT-Neo-125M for Falcon7B trained on the Pile and GPT2-124M (Radford et al.) for two OpenAI base models (Davinci-002 and Babbage-002).
E.2 Neighborhood Attack via Neighbourhood Comparison
This method is proposed by Mattern et al. (2023). The main idea is to compare the neighbors’ losses and those of the original sample under the target model by computing their differences.
In our replication process, we followed the method on official Github 8 to select key hyperparameters. Specifically, We randomly mask 30 percent of the tokens in the original sample with a span length of 2 to generate 25 masked sentences from the original sample and then use the T5-3b mask-filling model to generate 25 neighbors.
To evaluate the score of a sample, we use the formula as follows:
L(x)− N∑ L(x̃)
N
σ
where σ is the standard deviation of the neighbours’ losses.
E.3 Min-K% The Min-K% method, proposed by Shi et al. (2023), is quite straightforward. It builds on the hypothesis that a non-member example is more likely to include a few outlier words with high negative loglikelihood (or low probability), while a member example is less likely to include words with high negative log-likelihood. Specifically, Min-K% is calculated as :
model(x) = 1
E
∑
xi∈Min−K%(x) log p(xi|x1, ..., xi−1).
(17) where x = x1, x2, ..., xN is the tokens in the sentences, while log p(xi|x1, ..., xi−1) is the loglikelihood of a token, xi.
In particular, we follow the original paper to set K = 20 for detection in experiments.

heading: F Descriptions of AUC score
text: To evaluate with AUC score, we first plot the ROC curve through the True Positive Rate (TPR) and False Positive Rate(FPR). The ROC curve is used to
8https://github.com/mireshghallah/neighborhoodcurvature-mia
plot TPR versus FPR using different classification thresholds. Lowering the classification threshold causes more categories of items to be classified as positive, thus increasing the number of false positives and true examples. AUC is then defined as the Area Under the ROC curve, providing an aggregate measure of the effect of all possible classification thresholds.

heading: G Sentiment Analysis
text: We further conducted syntax analysis experiments on both member and non-member samples. The specific results are shown in Figure 6.

heading: H More Experiments Demonstrations
text: The visualizing AUC results of training data detection across various models with different methods are shown in Figure 7 & 8.

heading: Future Work
text: This allows future users to construct subsets of StackMIA suitable for newly released LLMs through our provided pipeline. Benchmark test. Following the settings of (Shi et al. , 2023), we divide the original WikiMIA by length. Simultaneously, we construct StackMIAsub4 dataset (8267 samples in total, Appendix A) to conduct experiments in this paper. Specifically, we set May 1, 2023, as the cutoff date for non-member data. Following (Shi et al. , 2023), we sequentially select approximately balanced sets of member and non-member data by length. Additionally, we use GPT-3. 5-turbo5 to construct (Appendix A) synonymous rewritten data to test the stability of the methods under the condition of approximate memory (Ishihara, 2023). Referring to (Ippolito et al. , 2022), we set BLEU (Papineni et al. , 2002)> 0. 75 as a condition to ensure semantic consistency in the rewritten data