The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
Authors: Matthieu Meeus
Lukas Wutschitz
Santiago Zanella-Beguelin
Shruti Tople
Reza Shokri
Venue: ICML 2025 poster (2025)
Keywords: privacy
language models
synthetic data
Primary area: deep_learning->large_language_models
TL;DR: We design a MIA and specialized canaries to audit the privacy of synthetic text generated by LLMs.

Abstract: How much information about training samples can be leaked through synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to a false sense of privacy. In this paper, we assume an adversary has access to some synthetic data generated by a LLM. We design membership inference attacks (MIAs) that target the training data used to fine-tune the LLM that is then used to synthesize data. The significant performance of our MIA shows that synthetic data leak information about the training data. Further, we find that canaries crafted for model-based MIAs are sub-optimal for privacy auditing when only synthetic data is released. Such out-of-distribution canaries have limited influence on the model’s output when prompted to generate useful, in-distribution synthetic data, which drastically reduces their effectiveness. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries with an in-distribution prefix and a high-perplexity suffix that leave detectable traces in synthetic data. This enhances the power of data-based MIAs and provides a better assessment of the privacy risks of releasing synthetic data generated by LLMs.

The Canary's Echo:
Auditing Privacy Risks of LLM-Generated Synthetic Text
Matthieu Meeus 12
Lukas Wutschitz? Santiago Zanella-Béguelin Shruti Tople? Reza Shokri 2.3
Abstract
raises serious privacy concerns about unintended informa- tion leakage through synthetically generated text. In this paper, we address the critical question: to what extent does synthetic text generated by LLMs leak information about the real data it is derived from?
How much information about training samples can be leaked through synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to false sense of privacy. In this paper, we assume an adversary has access to some synthetic data generated by LLM. We design membership inference attacks (MIAs) that target the training data used to fine-tune the LLM that is then used to synthesize data. The significant performance of our MIA shows that synthetic data leak information about the train- ing data. Further, we find that canaries crafted for model-based MIAs sub-optimal for pri- vacy auditing when only synthetic data is released. Such out-of- fluence on the model's 's output when prompted to generate useful, in-distribution synthetic data, which drastically reduces their effectiveness. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries with an in-distribution prefix and high-perplexity suffix that leave detectable traces in synthetic data. This enhances the power of data- based MIAs and pro- vides better assessment of the privacy risks of releasing synthetic data generated by LLMs.
Prior methods to audit privacy risks insert highly vulnerable, disstri-iitiittionn examples, canaries (Carlini et al., 2019), into the training data and test whether they can be identified using membership inference attacks (MIAs) (Shokri et al., 2017). Various MIAs have been proposed, typically assum- ing an attacker with access to the trained model or its output logits (Carlini et al., 2019; Shi et al., 2024). In the context of LLMs, MIAs often rely on analyzing the model's behavior when prompted with inputs related to the canaries (Carlini et al., 2021; Chang al., 2024; Shi al., 2024). However, similar investigations are lacking in scenarios where LLMs are used to generate synthetic data and only this synthetic data is available to an attacker.
canaries have limited in-
Contributions. In this work, we study-for the first time the factors that influence information leakage from a synthetic data-corpus generated using LLMs. First, we introduce data- based attacks that only have access synthetic data, and not to the model used to generate it, and therefore cannot probe it with adversarial prompts nor compute losses or other statistics used in model-based attacks (Ye et al., 2022). We propose approximating membership likelihood using either a model trained on the synthetic data or the target example similarity to its closest synthetic data examples. We design our attacks adapting the state-of-the-art pairwise likelihood ratio tests as in RMIA (Zarifzadeh et al., 2024) and evaluate them on labeled datasets: SST-2 (Socher et al., 2013), AG News (Zhang et al., 2015) and SNLI (Bowman et al., 2015). Our results show that MIAS leveraging only synthetic data achieve AUC scores of 0. for SST-2, 0.68 for AG News and 0.77 for SNLI, largely outperforming a random guess baseline. This suggests that synthetic text can leak significant amount of information about the real data used to generate it.
1. Introduction
Large Language Models (LLMs)) can generate synthetic data that mimics human-written content through domain-specific prompts. Besides their impressive fluency, LLMs are known to memorize parts of their training data (Carlini et al., 2023) and can regurgitate exact phrases, sentences, or even longer passages when prompted adversarially (Zanella-Béguel Béguelin et al., 2020; Carlini et al., 2021; Nasr et al., 2023). This
'Imeriall College London Microsoft $ National Univer- sity of Singapore Correspondence to: Matthieu Meeus 4mm422@ic.ac.um
Second, we use the attacks we introduce to quantify the gap in performance between data- and model-based attacks. We do so an auditing scenario, designing adversarial canaries and controlling leakage by varying the number of times a canary occurs in the training dataset. Experimentally. we
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Canary's Echo: Auditing Privacy Risks of LLM--Genrratd Synthetic Text
find a sizable gap when comparing attacks adapted to the idiosynerasies of each setting: a canary would need to occur 8x more often to be as vulnerable against a data-based attack as it is against a model- attack (see Figs. la and ld).
2. Background and problem statement
Synthetic text generation. We consider a private dataset D == (z. (s,, 1 of labelled text records where s; represents a sequence of tokens (e.g. a product review) and 4, is a class label (e.g. the review sentiment). A synthetic data generation mechanism is probabilistic procedure map- ping D to a synthetic dataset D = (5; = (5., 4)) with desired label set {fa) Unless stated otherwise, we consider N = N. The synthetic dataset D should preserve the uttliity of the private dataset D, i..., it should preserve as many statistics of D that are useful for downstream anal- yses as possible. In addition, a synthetic data generation mechanism should preserve the privacy of records in D, i.c. it should not leak sensitive information from the private records into the synthetic records. The utility of synthetic dataset can be measured by the gap between the utility achieved by D and D in downstream applications. The fact that synthetic data is not directly traceable to original data records does not mean that it is free from privacy risks, On the contrary, the design of a synthetic data generation mechanism determines how much information from D leaks into D and should be carefully considered. Indeed, several approaches have been proposed to generate synthetic data with formal privacy guarantees (Kim et al., 2021; Tang et al., 2024; Wu et al., 2024; Xie et al., 2024). We focus on privacy risks of text generated by a pre-trained LLM fine-tuned on a private dataset D ( (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Specifically, we fine-tune an LLM Au on records (s:, (.) E D to minimize the loss in completing s; conditioned on a prompt template p(()). obtaining 0. We then query 0 using the same prompt template to build synthetic dataset D matching a given label distribution.
Third, we discover that canaries designed for model-based attacks fall short when auditing privacy risks of synthetic text. Indeed, privacy auditing of LLMs through model- bosel- MIAs relies rare, -i--tii-iiiiittoonnn sequences of high perplexity (Carlini et al., 2019; Stock et al., 2022; Wei et al., 2024; Meeus et al., 2024c). We confirm that model-based MIAS improve as canary perplexity increases. In sharp contrast, we find that high perplexity sequences, although distinctly memorized by the target model, are less likely to be echoed through synthetic data generated by the target model. Therefore, canary perplexity increases, the ca- nary influence on synthetic data decreases, making its mem- bership less detectable from synthetic data (see Figure 2). We show that low-perplexity, and even in-distribution ca- naries, while suboptimal for model-based attacks, are more adequate canaries in data-based attacks.
Next, we propose an alternative canary design tailored for data-based attacks based on the following intuition: (i) in-distribution canaries aligned with the domain-specific prompt can influence the generated output; and (ii) memo- rization is more likely when canaries contain sub-sequences with high perplexity. We construct canaries starting with an in-distribution prefix of length F, transitioning into an diftiiiirbtttoon suffix, increasing the likelihood that the model memorizes them and that they influence synthetic data. We show that, for fixed overall canary perplexity, the performance of attacks for canaries with in-distribution pre- fix and ouito--distribution suffix (0 < F < max) improves upon both entirely in-distribution canaries (F = max) and out-of-distribution canaries (F- = 0), across datasets (see Fig. 1 and Table 2).
Membership inference attacks. MIAs (Shokri et al., 2017) provide a meaningful measure to quantify privacy risks of machine learning models, due to its simplicity but also due to the fact that protection against MIAs implies protection against more devastating attacks such as attribute inference and data reconstruction (Salem et al., 2023). In MIA target model 0, an adversary aims to infer whether target record is present in the training dataset of A. Different variants constrain the adversary's 's access to the model. In our setting, we consider model- based adversaries that observe the output logits on inputs of their choosing of a model 0 fine-tuned on a private dataset D. We naturally extend the concept of MIAs to synthetic data generation mechanisms by considering data-based adversaries that only observe a synthetic dataset D generated from D.
Lastly, we evaluate our attacks synthetic data generated with formal privacy guarantees. We adopt the training-tinn method proposed by work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023) and finetune the target model on the private dataset using DP-SGD (Abadi et al., 2016) with e 8. We find the performance of the strongest data- based MIA to drop to random guess performance (AUC of 0.5), confirming that differential privacy constitutes strong defense.
Taken together, the proposed MIAs and canary design can be used to audit privacy risks of synthetic text. Auditing es- tablishes a lower bound on the risk, useful to take informed decisions about releasing synthetic data in sensitive applica- tions and also complements upper bounds on privacy risks from methods that synthesize text with provable guarantees.
Privacy auditing using canaries. A common method used to audit the privacy risks of ML models is to eval- uate the MIA vulnerability of canaries, i.e., artificial worst- case records inserted in otherwise natural datasets (Carlini et al., 2019). This method can also be employed to de-
2

---

The Canary's Echo: Auditing Privacy Risks of LLM--Gneeatted Synthetic Text
rive statistical lower bounds on the differential privacy (DP) guarantees of the training pipeline (Jagielski et al., 2020; Zanella-Béguelin et al., 2023). Records crafted to be out-of- distribution w.r.t the underlying data distribution of D give a good approximation to the worst-case (Carlini et al., 2019; Meeus et al., 2024c). Canaries can take a range of forms, such as text containing sensitive information (Carlini et al., 2019) and random (Wei et al.. 2024) or synthetically gener- ated sequences (Meeus et al., 2024c). Prior work identified that longer sequences, repeated more often (Carlini et al., 2023), and with higher perplexity (Meeus et al., 2024c) are better memorized during training and hence are more vul- nerable to model- -baed MIAs. We study multiple types of canaries and compare their vulnerability against model- and synthetic data-based MIAs. We consider a set of canaries {&r= = (Bs, / each crafted adversarially and inserted with probability 1/2 into the private dataset D. The resulting dataset is then fed to a synthetic data generation mechanism. We finally consider each canary अे as the target record of MIA to estimate the privacy risk of the generation mecha- nism (or the underlying fine-tuned model).
adapt state-of-the-ard methods used to compute membership
scores in model-based attacks to the data-based scenario.
3. Methodology
3.1. Computing the membership score
In Algorithm 1, the adversary computes a membership score 3 indicating their confidence that 0 was trained on & (i.e. that b = 1). We specify first how to compute a membership signal a for model- and data-based adversaries, and then how we compute 3 from adapting the RMIA methodology of Zarifzadeh et al. (2024).
3. 3.1.1. I. MODEL-BASER ATTACKS
The larger the target model f's probability for canary ā = (5, E). Po p())), as compared to its probability on reference models, the more likely that the model has seen this record during training. We compute the proba- bility for canary & the product of token-level probabil- ities for å conditioned on the prompt p(.). Given target
canary text 11,..., .n. we compute (5 p())) as Po( = IT= Po(t, p()), .t....,tty tz-1). We consider this probability the membership inference signal against model, i.c.
Threat model. We consider an adversary A who aims to infer whether a canary å was included in the private dataset D used to synthesize a dataset D. We distinguish between two threat models:
= (a p()).
(i) an adversary A with query-accest to output logits of target model 0 fine-tuned on D; and (ii) adversary A 5 with only access to the synthetic dataset D. To the best of our knowledge, for text data this latter threat model has not been studied extensively in the literature. In contrast, the privacy risks of releasing synthetic tabular data are much better understood (Stadler et al., 2022; Yale et al., 2019; Hyeong et al., 2022; Zhang et al., 2022). Algorithm shows the generic membership inference experiment en- compassing both model- and data-based attacks, selected by the synthetic flag. The adversary is represented by a stateful procedure A, used to craft a canary and compute its membership score. Compared
3.1.2. DATA-BASED ATTACKS
When the attacker only has access to the synthetic data D. we need to extract a signal purely from D that correlates with membership. We next describe two methods to com- pute a membership signal a based on D. For more details, refer to their pseudo-code in Appendix A.
n-gram model. The attacker first fits an n--rram model using D as training corpus. An n-gram model computes the probability of the next token wy in a sequence based solely on the previous n - 1 tokens (Jurafsky & Martin, 2024). The conditional probability of a token Wy given the previous 1 tokens is estimated from the counts of n--rams in the training corpus. Formally, P Pr-gom((y))
standard membership experiment, we consider a fixed private dataset D rather than sampling it, and let the adversary choose the target át. This is close to the threat model of unbounded DP, where the implicit adversary selects two datasets, one obtained from the other by adding one more record, except that in our case the adversary observes but cannot choose the records in D. The membership score 33 returned by the adversary can be tumndd into binary membership label by choosing appropriate threshold. We further clarify assumptions made for the adversary in both threat models in Appendix E.
C(ws-(n- (),,,,))))...
C(w (oe -1) where C(s) is the number of times the sequence s appears in the training corpus and V is the vocabulary size. We use Laplace smoothing to deal with n-grams that do not ap- pear in the training corpus, incrementing the count of every n- g-aam by 1. The probability that the model assigns to sequence of tokens
(Wi,..., wg) can be computed as P P (s) πī Pr--rram( (w, Wj-(n-1),- W)-1). With the n-gram model fitted on the synthetic dataset, the attacker computes the n-gram model probability of the target canary &t = ($, as its membership signal, i.e. a = Pr.gnmm (8). Intuitively, if the canary 2 was present in the training data, the generated synthetic data D will bet-
Problem statement. We study methods to audit privacy risks associated with releasing synthetic text. Our main goal is to develop an effective data-based adversary A in the threat model of Algorithm 1. For this, we explore the design space of canaries to approximate the worst-case, and
3

---

The Canary's Echo: Auditing Privacy Risks of LLne-Gneeattdd Synthetic Text
Algorithm I Membership inference against an LLM-based synthetic text generator
1: Input: Fine-tuning algorithm T, pre-trained model Og. private dataset D = {x; = (si, (.) labels ( prompt
template p(-), canary repetitions Prep- sampling method sample, adversary A
2: Output: Membership score 3
3: & + A(T. (UT D, (2) P(-))
[Adversarially craft canary (see Sec. 3.2))
4: b {0. 13
(Flip fair coin)
5: bf = then
6: 0 < T(oo, DU (m)mm)
(Fine-tune On with canary repeated Krep times)
7: else
8: 0 - 1-0 D)
(Fine-tune 0g without canary)
9: end if
10: ior 1... Ñ do
3, sample(8(p(E.))
(Sample synthetic records using prompt template)
12: end for
Ñ 13: D - (5. { i-1 14: if synthetic then
8+ 2()
(Compute membership score 33 of â, see Sec. 3.1.2 and algorithms in Appendix A}
16: else
17: B+ A(),2) i)
(Compute membership score 3 of ft, see Sec. 3.1.1)
18: end if
19: return 3
ter reflect the patterns of 5, resulting in the n-gram model assigning a higher probability to á than if was not present.
quickly becomes computationally prohibitive. We use the oftte-of-theaatt RMIA method (Zarifzadeh et al., 2024) to maximize attack performance with a limited number of ref- erence models M. Specifically, for the target model 6, we calculate the membership score of canary & using refer- ence models { as follows (details on applying RMIA to our setup are in Appendix B): (0
Similarity metric. The attacker computes the similarity be- tween the target canary text á and all synthetic sequences 5; in D using similarity metric SIM, i.e.
SIM($, 3,) for i 1...., N. Next, the attacker identifies the synthetic sequences with the largest similarity to S. With oz() the j-th largest similarity, the membership inference signal is computed as the mean of the l: most similar examples, i.e.
(2)
3.2. Canary generation
= Ein (ig). Intuitively, if & was part of the training data, the synthetic data D will likely contain sequences 5; more similar to å than if & was not part of the training data, resulting in a larger mean similarity. Various similarity met- rics can be used. We consider Jaccard similarity (SIM)ae), often used measure string similarity, and cosine similarity between the embeddings of the two sequences, computed using a pre-trained embedding model (SIMemb).
Prior work has shown that canaries with high perplexity are more likely to be memorized by language models (Meeus et al., 2024c). High perplexity sequences are less pre- dictable and require the model to encode more specific, non non-generalizable details about them. However, high per- plexity canaries are not necessarily more susceptible leak- age via synthetic data generation, as they are outliers in the text distribution when conditioned on given in-distribution prompt. This misalignment with the model's natural gen- erative behavior means that even when memorized, these canaries are unlikely to be reproduced during regular model inference, making them ineffective for detecting memoriza- tion of training examples in generated synthetic data.
3.1.3. COMPUTING RMIA SCORES
Reference models, also called shadow models, are surrogate models designed to approximate the behavior of a target model. MIAs based on reference models perform better but are more costly run than MIAS that do not use them, with the additional practical challenge that they require ac- cess to data distributed similarly to the training data of the target model (Shokri et al., 2017; Ye et al., 2022). Ob- taining multiple reference models in our scenario requires tne-ttunig a large number of parameters in an LLM and
To address this issue, take advantage of the greedy nature of popular autoregressive decoding strategies (e.g. beam search, top-k and top-p sampling). We can encourage such decoding strategies to generate text closer to canaries by crafting canaries with a low perplexity prefix. To ensure memorization, we follow established practices and choose
4

---

The Canary's Echo: Auditing Privacy Risks of LLne-Gneeratdd Synthetic Text
high perplexity suffix. Specifically, we design canaries a (), where & has an in-distribution prefix and out-of-distribution suffix. In practice, we split the orig- inal dataset D into a training dataset and a canary source dataset. For each record x = (s, () in the canary source dataset, we design a new canary _ = ($, E). We truncate to get an in-distribution prefix of length F' and generate a suffix using the pre-trained language model 0b. adjusting the sampling temperature to achieve a desired target per- plexity P Purget.. We use rejection sampling to ensure that the perplexity of the generated canaries falls within the range [0.9 Purge: 1.1 Purgell. We ensure the length is consistent across canaries, as this impacts memorization (Carlini et al., 2023; Kandpal et al., 2022). By adjusting the length of the in-distribution prefix, we can guide the generation of either entirely in mn-distribution or -fittiibbitttni canaries.
sampling with p 0.95, i.e. we sample from a vocabu- lary restricted to the smallest possible set of tokens whose total probability exceeds 0.95. We further ensure that the synthetic data bears high utility, and is thus realistic. For this, we consider the downstream classification tasks for which the original datasets have been designed. We fine- tune RoBERTa-baal ba-baa (Liu et al., 2019) on D and D and compare the performance of the resulting classifiers on held- out evaluation datasets. Details are provided in Appendix F, for synthetic data generated with and without canaries.
Canary injection. We generate canaries f = (s. E) as described in Sec. 3.2. Unless stated otherwise, we consider 50-word canaries. Synthetic canaries are generated using Mistral-7B (Jiang et al., 2023) as Pg- We consider two ways of constructing a canary label: (i) randomly sampling a label i from the distribution of labels in D, ensuring that the class distribution among canaries matches that of D (Natural); and (ii) extending the set of labels with new artificial label ( "canary""" only used for canaries (Artificial).
We insert each canary Trep times in the training dataset of target and reference models. When a canary is selected as member, the canary is repeated Reep times in the training dataset, while canaries selected as non-members are ex- cluded from the training dataset. As in prior work (Carlini et al., 2023; Kandpal et al., 2022; Meeus et al., 2024c), we opt for nrpp > 1 to increase memorization, thus facilitating privacy auditing and the observation of the effect of different factors on the performance of MIAs during ablation studies.
Membership inference. We compute the membership scores (ă) as described in Sec. 3.1. For one target model 0, we consider 1000 canaries â. of which on average half are included in the training dataset Reep times (members), while the remaining half are excluded (non-members). We then use the computed RMIA scores and the ground truth for membership to construct ROC curves, from which we compute AUC and true positive rate (TPR) at low false pos- itive rate (FPR) as measures of MIA performance. Across experiments, we use M = 4 reference models H', each trained on dataset Dar consisting of the dataset D used to train the target model 0 with canaries inserted. Note that although practical attacks rarely have this amount of infor- mation, this is allowed by the threat model of Algorithm I and valid as a worst-case auditing methodology. We ensure that each canary is a member in half (i.e. 2) of the reference models and a non- non-member in the other half. For the attacks based synthetic data, we using n-gram model and k = 25 for computing scores based on similarity. We use Sentence-BFi BERT (Reimers & Gurevych, 2019) (paraphrase Miniiee-MMMMMMMMM--- MiLLiMM------- L6-v2 from former as the embedding model.
4. Experimental setup
Datasets. We consider three datasets that have been widely used to study text classification: (i) the Stanford Sentiment Treebank (SST-2) (Socher et al., 2013), which consists of excerpts from written movie reviews with a binary sentiment label; (ii) the AG News dataset (Zhang al., 2015), which consists of news articles labelled by category (World, Sport, Business, Sci/Tech).; and (iii) the SNLI dataset (Bowman et al., 2015), which consists of premises and hypotheses labeled as entailment, contradiction or neutral. In all ex- periments, we remove examples with less than 5 words, bringing the total number of examples to 43 296 for SST-2 and 120 000 for AG News. For SNLI, we selected the first 100 records.
2 for computing scores
Synthetic data generation. We fine-tune the pre-trained Mistral-7B model (Jiang et al., 2023) using low-rank adap- tation (LoRA) (Hu et al., 2022). We use a custom prompt template p(-) for each dataset (see Appendix C). More de- tails on the implementation and parameters are provided in Appendix D. We sample synthetic data from the fine-tuned model 0 conditioned on prompts p().), following the same distribution of labels in the synthetic dataset D as in the original dataset D, i.e. L,- = E, for i = 1,.... N. To generate synthetic sequences, we sequentially sample completions using a softmax temperature of 1.0 and top-p (aka nucleus)
5. Results
5.1. Baseline evaluation with standard canaries
We begin by assessing the vulnerability of synthetic text using standard canaries. Specifically, we utilize both in- distribution canaries and synthetically generated canaries with target perplexity Purget 250, no in-distribution prefix (F = 0), neep = 12 and natural or artificial labels, as described in Section 4. Table I summarizes the ROC AUC for model- and data-based attacks.
5

---

The Canary's Echo: Auditing Privacy Risks of L.Me-Geerrated Synthetic Text
Canary injection
ROC AUC
Model .A
Synthetic A (2-gram)
Synthetic A (SIMaa)
Synthetic A² (SIMemb)
Dataset
Source
Label
In-distribution'
0.911
0.741
0.602
0.586
SST-2
Natural Artificial
0.999
0.620
0.547
0.530
Synthetic
0.999
0.682
0.552
0.539
In-distribution
0.993
0,676
0.590
0.565
AG News
Natural
0.996
0.654
0.552
0.506
Synthetic
Artificial
0.999
0.672
0.560
0.525
In-distribution'
0.892
0.718
0.644
0.630
SNLI
Natural Artificial
0.998 0.997
0.534
0.486
0.488
Synthetic
0.770
0.602
0.571
Constrained by in-distribatioo data, canaries consist of exactly 30 words (50 elsewhere).
Table 1. ROC AUC across datasets, threat models (model-based A and data-based A and MIA methodologies for standard. high perplexity canaries (target perplexity Parger = 250, no in-distribution prefix (P = 0) and frrep = 12). We give the ROC curves and TPR at low FPR scores in Appendix G, further ablations in Appendix H, and elaborate the disparate vulnerability of high perplexity canaries in model- and data-based attacks in Appendix I.
First, we find that MIAs relying solely on the generated synthetic data achieve AUC score significantly higher than a random guess (i.e. AUC = 0.5), reaching up to 0.74 7 for SST-2, 0.68 for AG News and 0.77 for SNLI. This shows that synthetic text can leak information about the real data used to generate it.
matches that of a model-based attack. Fig. la illustrates these results as set of ROC curves. We quantify this discrepancy by noting that the MIA performance for A² at reep 16 is comparable to A at Nlop 2 and for low FPR at neep = 1. We find similar results in Fig. 1d for AG
News. The MIA performance for A at Reep = 16 falls between the performance of A at neep Under these experimental conditions, canaries would need to be repeated 8 to 16x to reach the same vulnerability in data-based attacks compared to model-based attacks.
Next, we observe that the data-based attack using n-gram model trained on synthetic data to compute membership scores outperforms the two attacks leveraging similarity metrics: Jaccard distance between a canary and synthetic strings (SIM)ac) or cosine distance between their embed- dings (SIMemh). This suggests that information critical to infer membership lies in subtle changes in the co-occurrence of nn-grams in synthetic data rather than in the generation of many sequences with lexical or semantic similarity.
and nrep 2.
We provide additional results for the standard canaries as ap- pendices: TPR at low FPR scores in Appendix G, ablations for data-based MIA hyperparameters in Appendix H, and a discussion on the disparate vulnerability of high perplexity canaries in model- and data-based attacks in Appendix I.
We also compare MIA performance across different canary types under dasa- attacks. The AUC remains consis- tently higher than a random guess across all canaries. For SST-2 and AG News, the highest AUC score of 0.74 and 0.68 is achieved when using in-distribution canaries, while for SNLI the AUC of 0.77 is reached for synthetic canaries.
5.2. Specialized canaries for enhanced privacy auditing
To effectively audit privacy risks in a worst-case scenario, explore designing specialized canaries that are both mem- orized by the model and influential in the synthetic data.
First, we generate specialized canaries by controlling their target perplexity Piarges We evaluate MIAs for both threat models across a range of perplexities for canaries with natu- ral labels, using Thep = 4 for the model-based MIA A and
As another baseline, we test RMIA on the target model trained on D. assuming the attacker has access to the model logits (A4). This attack achieves near-perfect performance across all setups, highlighting an inherent gap between the performance of model- and data-hased MIAs. This suggests that, while fine-tuned model memorizes standard canaries well, the information necessary to infer their membership is only partially transmitted to the synthetic text.
neep = 16 for the data-based MIA A We explore wide range of perplexities, finding x 10a to align with random token sequences. Figure 2 shows the ROC AUC score versus canary perplexity. For the model-based attack .A", the AUC monotonically increases with canary perplexity, reaffirming that outlier records with higher perplexity are more vulnera- ble MIAS (Feldman & Zhang, 2020; Carlini al., 2022a;
To investigate the gap between the two attacks in more detail, we vary the number canary repetitions Teep to amplify the power of the Bata-based attack until its performance
6

---

The Canary's Echo: Auditing Privacy Risks of LeM--enerated Synthetic Text
0.8
0.8
0.8
1 Per 10
TPR
0.6
A
0.6
A Per 102t
TPR
0.6
F
10, Pl 10a
19, Par 10
0.4
A F= 10
A F
1 Mapp
16
A 10
0.2
AP,
0.2
A Puu 10"
30
A Map
A Pa. 109
A4
A" Per
10
0
0.2
0.4
0.8
0
0.2
0.8
0
0.2
FPR
HPR
IPR
(a) Canary repetitions Teepp
(b) Canary perplexity P Parge-
(c) Canary in-distribution prefix F.
Puger 31. F O.
Rep
0
4. Reep
16, F = 0.
Pungg
31. Plp
4. Tirep 16.
0.8
A PL
0.6
0.6
TPR
18
Pagy
TPR
1 PP 10
19, Pur 10a
A
0.1
1 Srep
0.4
A9, Per 10
0.4
A F
Pa 10
A 1P,F=20
Paay
Dap
0.2
A Pur 1
P 10a
10a
0.2
A
A F=
Dup
Kap
10
max
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
HPR
IPR
(d) Canary repetitions nhep-
(e) Canary perplexity Purget- mapp 4, mrpp 16, F = O.
Canary in-distribution prefix
Purget 31, F 0.
P Parget 31, nep 4, mep 16.
Figure ROC curves MIAs on synthetic data A compared to model- MIAS A on SST-2 (la-lc) and AG News (Id-If). We ablate over the number of canary insertions Thop in 1a, 1d, the target perplexity Purger of the inserted canaries in 1b, le and the length F of the in-distribution prefix the canary in Ic, 1f. Log-log plots in Appendix
the entire FPR range, while the data-based attack weakens, approaching AUC of 0.5 at high perplexities. This suggests that identifying susceptible canaries is straightforward for model-based privacy audits, but assessing the privacy risk of synthetic data requires careful balance between canary memorization and its influence on synthetic data.
Clearyy perfecty
(a) SST-2
(b) AG News
We now examine whether canaries can be crafted to enhance both memorization and influence on the synthetic data, mak- ing them suitable to audit the privacy risks of releasing synthetic data. In Sec. 3.2, we introduced a method that exploits the greedy nature of LLM decoding to design more vulnerable canaries. We craft a canary with low-perplexity, in-distribution prefix to optimize its impact on the synthetic dataset, followed by a high-perplexity suffix to enhance memorization. We generate this suffix sampling from the pre-trained LLM 8g with high temperature. Figures Ic and If illustrate the ROC curves for SST-2 and AG News, respec- tively, and Table summarizes the corresponding ROC AUC and TPR low FPR. We set the overall canary perplexity Purget 31 and vary the prefix length F' from F = 0 (fully
Figure 2. ROC AUC for synthetic canaries with varying perplexity (natural label). The model-based MIA A improves as canary perplexity increases, while the data-based MIA performance A f0 (2-gram) decreases. nepp Thop 16.
Meeus et al., 2024c). Conversely, for the data-based attack AD, the AUC initially increases with perplexity but starts to decline beyond certain threshold, eventually approaching random guess (AUC of 0.5). To further illustrate this, we present the complete ROC curve in Figures 1b and le for SST-2 and AG News, respectively. We vary the canary perplexity Pearges while keeping other parameters constant. As P uurget increases, the model-based attack improves across

---

The Canary's Echo: Auditing Privacy Risks of LLM--Geeraate Synthetic Text
@
5.4. Synthetic data with formal privacy guarantees
Dataset
F
ROC AUC
FPR=0.01
FPR=0.1
To mitigate any privacy leakage associated with the release of synthetic data, prior work has proposed to generate syn- thetic data with formal privacy guarantees, in particular differential privacy (DP). Methods used to generate syn- thetic text with DP guarantees mitigate MIAs by ensur- ing that any single training record exerts limited influence on synthesized data. These methods are broadly split into training-timg time (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023) and inference-time (Xie et al., 2024; Wu et al., 2024; Tang et al., 2024; Amin et al., 2024). Training-time time methods fine-tune pre-trained LLM with DP-SGD and then prompt this model to generate synthetic data. These methods leverage the post-processing property of DP to transfer the guarantees from the fine-tuned model to syn- thetic data. Because generating synthetic data from DP model does not consume additional privacy budget, they can generate unlimited amount of data with a fixed privacy budget. In contrast, inference-time methods use unmodified pre-trained models prompted on private data and inject cal- ibrated noise during decoding (Xie et al., 2024; Wu et al., 2024; Tang et al., 2024) or employ DP evolutionary algo- rithms to steer generation towards a distribution similar to the private data (Amin et al., 2024).
0.673
0.081
0.304
10
0.715
0.057
0.312
SST-2
20
0.725
0.069
0.318
30
0.760
0.069
0.410
max
0.741
0.101
0.408
0
0.692
0.089
0.309
10
0.646
0.053
0.276
AG News
20
0.716
0.069
0.321
30
0.710
0.055
0.333
0.676
0.039
0.314
Table 2. MIA performance (ROC AUC and TPR at low FPR) for data-based MIA A (2-gram) for canaries with varying length of in-distribution prefix (results from Figs. 1c,
synthetic canaries) to F max (in-distribution canaries). We observe that combining an in-distribution prefix (F 0) with a high-perplexity suffix (F < max) enhances attack effectiveness. For both datasets, the optimal AUC, and often also the optimal TPR at low FPR, for the MIA is reached for a prefix length 0 < F < max (see Table 2). This suggests that although the model's memorization of the canary stays consistent (as the overall perplexity remains unchanged), the canary's impact on the synthetic data becomes more promi- nent with longer in-distribution prefixes. We hypothesize that familiar low- oowperrpexxtyy prefixes serve as starting points for text generation, enhancing the likelihood that traces of the canary appear in the synthetic data.
We instantiate the training- toiniing-timeeeeee method, i.c. finetuning the target model with DP-SGD (Abadi et al., 2016) using the Opacus library (Yousefpour et al., 2021) and = 8. We follow the same setup from Section 5.1 and report the per- formance of the data-based MIA in Table 4. As expected, we find the AUC for the strongest data-based MIA (2-gram) to approach random guess performance (AUC of 0.5) when DP guarantees incorporated. This confirms that DP constitutes strong defense. We further find that the corre- sponding generated synthetic data maintains a high utility in downstream tasks. For instance, for synthetic data generated with e = 8, accuracy on SST-2 reaches 91.6%, compared to 91.5% for non-DP synthetic data and 92.3% for real data (see Appendix F).
5.3. Identifying the memorized sub-sequences
We analyze what information from canary leaks into the synthetic data that enables a data-based attack to infer its membership. For each canary = (&, E), we examine the synthetic data generated by a model trained on a dataset including (member) and excluding f (non-member). We leverage the M 4 reference models 6' used to develop the attack for 1000 specialized canaries from Fig. 1c. For each model 0', we count the number of n-grams in 3 that occur at least once in D' (Cunique). We also compute the me- dian Cmmdd and average Cavg counts of n-grams from á in D'. Table 3 summarizes how these measures vary with n. As n increases, the number of n-grams from the canary appearing in the synthetic data drops sharply, reaching Cmed = 0 for n = 4 for models including and excluding a canary. This suggests that any verbatim reproduction of canary text in the generated synthetic data is of limited length. Further, we observe only slight differences in counts between members and non-members, indicating that the signal for inferring membership is likely in subtle shifts in the probability distri- bution of token co- oo-occurrences within the synthetic data, as captured by the 2-gram model. We further analyze canaries with the highest and lowest RMIA scores in Appendix K.
Our results suggest that DP-generated synthetic data can achieve high utility, while strongly mitigating the success of data-based MIAs. Yet, achieving the right balance be- tween privacy and utility in DP synthetic text generation is likely context-dependent We hope that the privacy auditing framework adapted to actual threat models we here propose enables future work rigorously explore this trade-off.
6. Related work
MIAS against ML models. Since the seminal work of Shokri et al. (2017), MIAs have been used to study mem- orization and privacy risks. Model-based MIAs have been studied under varying threat models, including adversaries
8

---

The Canary's Echo: Auditing Privacy Risks of LLMe-Geerrated Synthetic Text
Cuniqus
Caee
Cwe
71
Member
Member
Non-member
Member
Non-member
1
45.97 2.8
45.1 + 3.0
819.6 + 702.6
821.5 + 727.9
7389.2 + 1648.4
7381.2 + 1650.0
2
29.6 5.6
28.0 5.5
4.5 = 6.5
3.5 : 5.5
198.0 109.3
195.9 107.5
4
4.9
3.7
4.1 3.2
0.0 = 0.0
0.0 + 0.0
1.3 2.6
1.2 2.5
8
0.1 = 1.0
0.1 1 0.6
0.0 0.0
0.0 0.0
0.0 = 0.1
0.0 = 0.0
Table Count statistics of n-grrams in canary that also appear in the synthetic data D' generated using reference models including and excluding & Number of r--grams in & that also appear in D' (Cunique), ). median (Cmes) and average (Cavg) counts of n-grams from & in D'. We report mean and std. deviation of these measures over all canaries (F' = 30, Punge = 31, Phpp = 16) for SST-2, Each canary & contains exactly 50 words and D' contains 685. = 45.4k words.
ROC AUC
GANs for images and location data. Privacy risks of syn- thetic tabular data have been widely studied, using MIAs based on similarity metrics and shadow models (Yale et al., 2019; Hyeong et al., 2022; Zhang et al., 2022). Stadler et al. (2022) compute high-level statistics, Houssiau et al. (2022) compute similarities between the target record and synthetic data, and Meeus et al. (2024a) propose trainable feature extractor. Unlike these, we evaluate MIAs on text generated using fine-tuned LLMs. This introduces unique challenges and opportunities, both in computing membership scores and identifying worst-case canaries, making our approach distinct from prior work.
Dataset
00
8
SST-2
0.620
0.48
AG News
0.654
0.52
SNLJ
0.534
0.49
Table ROC AUC across datasets for the strongest basad-aaed_aa A MIA (2-gram). for synthetic data without (e oo) and with DP guarantees (e = 8). We use the setup from Table I, i.e. synthetic canaries with natural labels, target perplexity Purga 250, no in-distribution prefix (F 0) and Plep = 12.
Vulnerable records in MIAs. Prior work found that some records (outliers) have a disparate effect on a trained model (Feldman & Zhang, 2020), making them more vulner- able to MIAs (Carlini et al., 2022a;b). Hence, specifically crafted canaries have been proposed to study memorization and for privacy auditing of language models, ranging from a sequence of random digits (Carlini et al., 2019; Stock et al., 2022) tokens (Wei et al., 2024) to synthetically generated sequences (Meeus et al., 2024c). Also for synthetic tabular data, outliers have been found to have increased privacy leakage (Stadler et al., 2022; Meeus et al., 2024a).
with access to model weights (Sablayrolles et al., 2019; Nasr et al., 2019; Leino & Fredrikson, 2020; Cretu et al., 2024), output probabilities (Shokri et al., 2017; Carlini et al., 2022a) or just labels (Choquette-Choo et al.. 2021). Most powerful MIAs leverage a large number of reference mod- els (Ye et al., 2022; Carlini et al., 2022a; Sablayrolles et al., 2019; Watson et al., 2021), while RMIA (Zarifzadeh al., 2024) achieves high performance using only few.
MIAs against language models. Song & Shmatikov (2019) study MIAs to audit the use of an individual's 's data during training. Carlini et al. (2021) investigate training data re- construction attacks against LLMs, sampling synthetic text and running model-based attacks to identify likely members. Kandpal et al. (2022) and Carlini et al. (2023) both find that repetitions in the training data make records more vulnera- ble. Shi et al. (2024) and Meeus et al. (2024b) use attacks to identify pre-training data. Various membership scores have been proposed, e.g. model loss (Yeom et al., 2018), lowest predicted token probabilities (Shi et al., 2024), changes in the model's 'ss probability for neighboring samples (Mattern et al., 2023), or perturbations to weights (Li et al., 2023).
Decoding method. Prior works study how decoding meth- ods like beam search (Zanella-Béguchl et al., 2020; Carlini et al., 2023), top-k sampling (Kandpal et al., 2022), de- caying temperature (Carlini et al., 2021) impact how often LLMs replicate information from their training data. We use fixed prompt templates and top-p sampling with 0.95 and temperature 1.0 to assess the privacy of synthetic text in a realistic regime rather than allowing the attacker to pick decoding method adversarially.
Reproducibility
base- MIAs in other scenarios. Hayes et al. (2019) train a Generative Adversarial Network (GAN) on synthetic images generated by a target GAN and use the resulting dis- criminator to infer membership. Hilprecht et al. (2019) ex- plore MIAs using synthetic images closest to target record. Chen et al. (2020) study attack calibration techniques against
We provide experimental details in Section 4 and Ap- pendix D. The datasets are publicly available, and we re- lease the code necessary to reproduce our results on Github: https: / / /aka ms mat.aaaaacaaaccc.....ooo
9

---

The Canary's Echo: Auditing Privacy Risks of LLM--Gneratedd Synthetic Text
Impact statement
Song, D. The secret sharer: Evaluating and test- ing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Secu- rity 19), pp. 267-284. USENIX Association, 2019. Noi:105555/3361138888 333.3333333...33335...3833333331
In this work, we propose a methodology to audit the privacy risks in LLM-generated synthetic data. Through a novel MIA, we quantify the potential for sensitive information leakage even in scenarios where the underlying model is inaccessible. We also identify that canary generation mecha- nisms found useful to study risks in model-basee attacks fall short in data-based attacks, and propose an improved canary generation mechanism optimal for data-based attacks.
Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, USENIX As- sociation, 2021. URL https:/ usenix. org/ http://feerroeeesssseeeeeeeeette
Taken together, the methods proposed in this work enable an auditor empirically estimate the privacy risks associated with synthetic text. Practitioners leveraging synthetic data as privacy-enhancing technology can use our tools to evaluate these risks before deploying synthetic text in downstream applications. In particular, our privacy auditing pipeline would be valuable when synthetic text data is proposed to extract utility from sensitive data (e.g. medical records, financial statements) to verify synthetic data generation implementations with formal privacy guarantees.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramèr, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Secu- rity and Privacy (S&P), pp. 1897-1914. IEEE, 2022a. doi: 10.:10.1110 100.1100//944211..00000 010.S4446221222229833336666
Carlini, N., Jagielski, M.. Zhang, C., Papernot, N., Terzis, A., and Tramèr, F. The privacy onion effect: Memoriza- tion is relative. Advances in Neural Information Process- ing Systems (NeurIPS 2022), 35: 355:13263-133223 13276, 2022b.
We hope this work advances the understanding of privacy risks in LLM-ggnnrated synthetic data and helps organi- zations and policymakers navigate the associated privacy- utility trade-offs effectively.
Carlini, N., Ippolito, D.., Jagielski. M., Lee, K., Tramèr, F,, and Zhang, C. Quantifying memorization across neural language models. In 11th International Conference on Learning Representations (ICLR 2023). OpenReview.net, 2023. URL https: /ttp::////////ee/e/eeeeee net/ forum? i id=TatRHT_Le
Acknowledgements
L.W. would like to thank Robert Sim for encouraging to work on this topic and Huseyin Inan for fruitful discussions on private synthetic data generation.
Chang, H., Shamsabadi, A. S., Katevas, K., Haddadi, H., and Shokri, R. Context-aware membership infer- ence attacks against pre-trained large language mod- els, 2024. URL https: /awxiaxxvvvvo/o org/ atsp//bb//// 13745. arXiv preprint.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS 2016), pp. 308-008-38 318. ACM, 2016.
Chen, D., Yu, N., Zhang, Y., and Fritz, M. GAN-leaks: A taxonomy of membership inference attacks against generative models. In 2020 ACM SIGSAC conference on computer and communications security (CCS 2020), pp. 343-362. ACM, 2020. doi:10.. 1111114332299.3411111 27711222997.4..8....
Amin, K., Bie, A., Kong, W., Kurakin, A., Ponomareva, N., Syed, U., Terzis, A., and Vassilvitskii, S. Private prediction for large-scale synthetic text generation. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 7244-7262, 2024.
Choquette- Choo, C. A., Tramèr, F., Carlini, N., and Pa- pernot, N. Label-only membership inference attacks. In 38th International conference on machine learning (ICML 202/), volume 139, pp. 1964-1974. PMLR, 2021. wtt.tttt: httpss///cooccceeeeeeeeee mlr. ..reess v139/ choquette wtw.ttteeechoo2aa.a..oo
Bowman, S. R., Angeli, G., Potts, C., and Manning, D. A large annotated corpus for learning natural language inference. In Mårquez, L., Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. 1. 10.18663/1/DD15-10000 URL https: / /aaalaannnlloyyyooeaeao oro/ /D15--155-11 1075.
Cretu, A.-M., Jones, D., de Montjoye, Y-A.,, and Tople, S. Investigating the effect of misalignment on membership privacy in the white-box setting. Proc. Priv. Enhancing Technol. 2024(3):4774555 2024. do:100.6653/PPPETS-2200
Carlini, N., Liu, C., Erlingsson, Ú. Kos, J., and
10

---

The Canary's Echo: Auditing Privacy Risks of LLMe-Geeraatdd Synthetic Text
Feldman, V. and Zhang, C. What neural networks mem- orize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems (NeurIPS 2020), 33:2881-2899 2020.
In 39th International Conference on Machine Learning
(ICMI. 2022), volume 162, pp- 10697-10707. PMLR, 2022. WRRLhtts://ppoceeeeiie 1162kkandpaa22..naa......aaa
mlr ......p presss/
Hayes, J., Melis, L., Danezis, G., and De Cristofaro, E. LOGAN: Membership inference attacks against gener- ative models. Proc. Priv. Enhancing Technol., 2019(1): 133-152, 2019. doi: 10t.::/4/////p/ss20000
Kim, K., Gopi, S., Kulkarni, J., and Yekhanin, S. Dif- ferentially private n-gram extraction. Advances in Neu- ral Information Processing Systems (NeurIPS 2021), 34: 5102-51 2021.
Hilprecht, B.. Härterich, M., and Bernau, D. Monte Carlo and reconstruction membership inference attacks against generative models. Proc. Priv. Enhuncing Technol., 2019 (4):232-249, 2019. 1oi:1002478/pppee 201-ss-2010-000000 9-00677
Kurakin, A., Ponomareva, N., Syed, U., MacDermed, L., and Terzis, A. Haresssing large-language models to generate private synthetic text, 2023. URL https:
/
01 184. arXiv preprint.
Houssiau, F., Jordon, J., Cohen, S. N., Daniel, O.,, Elliott, A., Geddes, J., Mole, C., Rangel-Smith, C., and Szpruch, L. TAPAS: a toolbox for adversarial privacy auditing of synthetic data. In NeurlPS 2022 Workshop Synthetic Data for Empowering ML Research, 2022. URL https: // //openreviee net. forum? d= 9hwskf1k722
Leino, K. and Fredrikson, M. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1605-1622. USENIX Association, 2020. URL
https: /top:://www.uue http://wwwwsseeix.xooo wwwp:eiiww...ggg/oeeeeeeeeeeeeee (ssnixsecurtt/////eeeetttttoool
Hu, J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang. L., Chen, W., et al. LoRA: Low-rank adaptation of large language models. In 10th International Conference on Learning Representations (ICLR 2022). OpenReview.net, 2022. URL ptttt::://ppnneviieee net/ forum? id=nZeVKeeFYE9.
Li, M., Wang. J., Wang. J. G., and Neel, S. MoPe: Model perturbation based privacy attacks on language models. 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pp. 13647-13660. ACI., 2023. doi: 1ec.10.1663/vv1 11883661/2023333 http///022...mmmmmmmm..//////
Hyeong, J., Kim, J., Park, N., and Jajodia, empirical study on the membership inference attack against tabular data synthesis models. In 31st ACM International Conference on Information & Knowledge Management (CIKM 22), pp. 4064-4068. ACM, 2022. 1eee1...11/351111 100.5111888.35545.
An
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M.. Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https: 1907. 11692. arXiv preprint.
Mattern, J., Jin, Z., Weggenmann, B., Schoelkopf, B., and Sachan, M. Differentially private language models for se- cure data sharing. In 2022 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP 2022). pp. 4860 4873. ACL, 2022. doi:10. htb.10.18653//122222.mmmm m23..
Jagielski, M., Ullman, J., and Oprea, A. Auditing differ- entially private machine learning: How private is private SGD? Advances in Neural Information Processing Sys- tems (NeurIPS 2020), 33:22205 2020.
Jang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-
Mattern, J., Mireshghallah, F., Jin, Z., Schölkopf, B., T. Membership infer-
Sachan, M., and ence attacks against language models via neighbourhood comparison. In Findings of the Association for Compu- lational Linguistics: ACL 2023, pp. 11330-11343, ACL, 2023. doi: 10011088653//1/2223 htt.:655///2222....nmnaaaaaaaa findings--cl..19.
A., Stock, P., Scao, T. I.., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. Mistral 7B, 2023. URL ht tps: //arxiv. otg/ abs www.aa/5533 2310 06825. arXiv preprint.
Jurafsky, D. and Martin, J. H. Speech and Language Pro- cessing: An Introduction Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. n.p., 3rd edition, 2024. URL https: //web. stondord. edu/ http://juraaaaauuuuu /sl33/. On- line manuscript released August 20, 2024.
Meeus, M., Guepin, F., Crequ, A.-M., and de Montjoye, Y.-A. Achilles' heels: vulnerable record identification in synthetic data publishing. In European Symposium on Research in Computer Security (ESORICS 2023), pp. 380- 399. Springer, 2024a. 100:100.10 0000///7778--31-117777 0 0_19.
Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models.

---

The Canary's Echo: Auditing Privacy Risks of LLM--Gnerattd Synthetic Text
Meeus, M., Jain, S., Rei, M., and de Montjoye, Y.- A. Did the neurons read your book? document- level membership inference for large language models. In 33rd USENIX Security Symposium (USENIX Secu- rity 24), pp. 2369-2385. USENIX Association, 2024b. URL hwwph:tpps.... www. www.ix. org.oooercc/ccceee/ecmoooooooo/e (been//sccrrrttyy//rreeeeetteeee
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 2017 IEEE symposium security and privacy (S&P), pp. 3-18. IEEE, 2017. doi: 109/P...//.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning. C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pp. 1631-1642. ACL, 2013, URL https: https////cccaatthhoog..o org/ 113--170.
Meeus, M. Shilov, I., Faysse, M., and de Montjoye, Y.-A. Copyright traps for large language models. In 41st International Conference on Machine Learn- ing (ICML 2024), volume 235, pp. 35296-35309. PMLR, 2024c. URL https: httpt::://pocceeaaiiiiee http://pcoceee....e//// mml.. press/ /1ees//v35/// m2555/eees222 html.
Song, C. and Shmatikov, V. Auditing data provenance in text-generation models. In 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2019), pp. 196-206. ACM, 2019. 16:110..145/222200.33333
Nasr, M., Shokri, R., and Houmansadr, A. Comprehen- sive privacy analysis deep learning: Passive and ac- tive white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium Se- curity and Privacy (S&P), pp. 739-753. IEEE, 2019. doi: 1ee.1..11///P.111...
Stadler, T. Oprisanu, B., and Troncoso, C. Syn- thetic data anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Secu- rity 22), pp. 1451-1468. USENIX Association, 2022. URL httph: /wwww uww.ix.eni.............. wtwpa//...ogcooofeenceeeeeeeeee neeulaaecurtt22///rreettttttaaan
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito. D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models, 2023. URL https: //tp:ii///ivvee//e/eaaeeeeeaeeee wwtp//gg//////oo/ee//.///.eeeeee preprint.
Stock, P., Shilov, I., Mironov, I., and Sablayrolles, A. De- fending against reconstruction attacks with Rényi dif- ferential privacy, 2022. URL https::/arxiv..t abs 2202. 07623. arXiv preprint.
abs//bb////// /tbs//311... 17035. arXiv
Reimers, N. and Gurevych, I. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019), pp. pp. 3982-3992. ACL, 2019. 1/111111865///1111914444444
Tang. X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F,, Lin, Z., Gopi, S., Kulkarni, J., and Sim, R. Privacy- preserving in-context learning with differentially private few-shot generation. In /2th International Conference on Learning Representations (ICLR 2024). OpenReview.net, 2024. URL htttps://opeereeee http:/iwwwweeeeeeeeeeeeeeeee forum? aalo2tt0pRRnC
Sablayrolles, A., Douze, M. Schmid, C., Ollivier, Y., and Jégou, H. White-box vs black-box: Bayes opti- mal strategies for membership inference. In 36th In- ternational Conference on Machine Learning (ICML 2019), volume 97, pp. 5558-5567. PMIR, 2019. URL URL sablayrolles19e
Watson, L., Guo, C., Cormode, G., and Sablayrolles, A. On the importance of difficulty calibration in membership inference attacks. In 10th International Conference on Learning Representations (ICLR 2022). OpenReview.net, 2021. URL https: /pw.reviewweeeeeeeeeeeeeee net forum? id=3eIrli0TwQ.
/ http://poocceeeeceeeeeee mlr wwl..prres/vv/
Salem, A., Cherubin, G., Evans, D., Köpf, B., Paverd, A., Suri, A., Tople, S., and Bégueli,,,,,,,in, S. SoK: Let the privacy games begin! A unified treatment of data inference privacy in machine learning. In 2023 IEEE Symposium Security and Privacy S&P), pp. 327-345. IEEE, 2023. doi: Noi::0.1100///P.. 109/10442255..../ 2023. 101 79281.
Wei, J. T.-., Wang, R. Y., and Jia, R. Proving membership in LLM pretraining data via data watermarks, 2024. URL https:/ http://awxvv..ggg..///// wtt.r//ga////2.....gg 10892. arXiv preprint.
Shi, W., Ajith, A., Xia. M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, Detecting pretraining data from large language models. In 12th International Conference on Learning Representations (ICLR 2024). OpenReview.net, 2024. AtLhttps: haattps://openreviee net/ forum? i=
Wu, T., Panda, A., Wang, J. T., and Mittal, P. Privacy- preserving in-context learning for large language models. In /2th International Conference on Learning Representa- tions (ICLR 2024). OpenReview.net, 2024. URL https: / /ppenreview net forum? d= 40P
12

---

The Canary's Echo: Auditing Privacy Risks of LLM--Genratted Synthetic Text
Xie, C., Lin, Z.,, Backurs, A., Gopi, S., Yu, D., Inan, H. A., Nori, H., Jiang, H., Zhang, H., Lee, Y. T., Li, B., and Yekhanin, S. Differentially private syn- thetic data via foundation model APIs 2: Text. In 41st International Conference on Machine Learning (ICML 2024), volume 235, pp. 54531-54560. PMLR, 2024. URL https: /ptpceeeicgcgssssee www. mll.press/ v235/xie240 html.
generation with differential privacy: A simple and prac- tical recipe. In 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1321-1342. ACL, 2023, doi:10.18800 101111666311222335 acl- long.74.
Zanella-Béuuelin, S., Wutschitz, I.., Tople, S., Rühle, V., Paverd, A., Ohrimenko, O., Köpf, B., and Brockschmidt, M. Analyzing information leakage of updates to natural language models. In 2020 ACM SIGSAC Conference on Computer and Communicaatinns Security (CCS 2020), pp. 363-375. ACM, 2020. doi: 10. 141113473/222.... 3.4478448
Yale, A., Dash, S., Dutta, R., Guyon, I., Pavao, A., and Bennett, K. P. Assessing privacy and quality of synthetic health data. In Conference on Artificial Intelligence for Data Discovery and Reuse (AIDR '19), pp. 1-4. ACM, 2019. doi: Nai:00.1145/3555 13359112..
Ganella-Bégueligg S., Wutschitz, L., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., and Jones, D. Bayesian estimation of differential privacy. 40th International Conference on Machine Learning (ICML 2023), volume 202, pp. 40624 40636. PMLR, ml r.rrsss wtt.eeeeeddiiis..m...e....eeee
Ye, J.. Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference at- tacks against machine learning models. In 2022 ACM SIGSAC Conference on Computer and Communica- tions Security (CCS 2022), pp. 3093-3106. ACM, 2022. 1oi:00.145//33446665566666
2023. URL
v202
Zarifzadeh, S., Liu, P., and Shokri, R. Low- cost high-power membership inference attacks. In 41st International Conference on Machine Learning (ICML 2024), volume 235, pp. 2024. URL. htt...ttttppp:///reeeeeee http:/poooeddnggee ngg. ml r.press// v235 htt//aaaiizza 2t5////arfizaaeh2255 htmml.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Pri- vacy risk in machine learning: Analyzing the connection to overfitting In s1 IEEE Computer Security Founda- tions Symposium (CSF 2018), pp. 268-282. IEEE, 2018. 10:990.1109/CSFF2000 8.00027.
PMLR,
Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., al. Opacus: User-friendly differential pri- vacy library in pytorch. arXiv preprint 11Xiv:2109.12298. 2021.
Zhang, X., Zhao, J. J., and LeCun, Y. Character-level con- volutional networks for text classification. In Advances in Neural Information Processing Systems (NIPS 2015), volume 28, 2015.
Zhang, Z., Yan, C., and Malin, B. A. Membership inference attacks against synthetic health data. J. Biomed. Inform., 125, 2022, 10i:10.1016/j.bbii ji..11111bi...221113777777
Yue, X., Inan, H., Li, X., Kumar, G., McAnallen, J., Sha- jari, H., Sun, H., Levitan, D., and Sim, R. Synthetic text
13

---

The Canary's Echo: Auditing Privacy Risks of LLM-GGneratedd Synthetic Text
A. Pseudo-code for MIAs based on synthetic data
here provide the pseudo-code for computing membership signals for both MIA methodologies based on synthetic data (Sec. 3.1.2), see Algorithm for the n-gram method and Algorithm 3 for the method using similarity metrics.
Algorithm 2 Compute membership signal using n-gram model
1: Parameter: 7-gram model order
2: Input: Synthetic dataset D [ - (8,, (.) Target canary = (8, E)
Oufput: Membership signal
4: C(w) - for all (n-1)- and n-grams
5: for = to N do
6: wi..... (k(s) 3;
7: for each n-gram (wy ( (1-1111-- :Is...; Wj) in 8; do
8: C(wj- ................................ wz) +=1
9: C(wy-(nn11 wij-1) +=1
10: end for
11:
end for
12:
V + Fi.w w E 3
The n-gram model is factored into conditional probabilities:
(Final n-gram model)
(a +
Pr Pn.gam (w; w Wg w--1) (1-)-(n-1)) wy-1) + V
14: W1..... Wk
(Compute probability of canary text 8)
15: a a I Pn.gram (105 W W))((--)) 11-11
16: return &
Algorithm 3 Compute membership signal using similarity metric
1: Parameter: Similarity metric SIM(s cutoff parameter k
2: Input: Synthetic dataset = { = (3), &)),, Target canary a = (3, 2)
3: Output: Membership signal a
4: for to N do
5: 0; < (1,($,5))
(Compute similarity of each synthetic example)
6: end for
7: Sort similarities o3 for i = 1....,N Ñ in descending order
8: Let (i(1). (((k) be the top-k similarities
9: a e & E ((())
(Compute mean similarity of the top-k examples)
return
B. Computation of RMIA scores
We here provide more details on how we adapt RMIA, as originally proposed by Zarifzadeh et al. (2024), to our setup (see
Sec. 3.1.3). In RMIA, the pairwise likelihood ratio defined as:
P(x 10)) P(x) )
( P(x (1=10)) P(z)
LRo 2) =
(1)
where represents the target model, x the target record, and z the reference population. this work, we only consider one target model 0 and many target records æ. As we are only interested in the relative value the likelihood ratio across target records, we can eliminate the dependency on the reference population 2,
14

---

The Canary's Echo: Auditing Privacy Risks of LLM--Genrated Synthetic Text
LRo( LRo(x)= P(x |0) P(x)
As suggested by (Zarifzadeh et al., 2024), we compute P(x) as the empirical mean of P(r (²)) across reference models {0,)/1
M P(a)
Σ P(æ 0,) M 1=1
(3)
To compute RMIA scores, we replace the probabilities in (2) by membership signals on target and reference models:
(1)
BO(x)
(4)
1 Ei=1 00 (r)
Note that when we compute (2(2)) as a product of conditional probabilities (e.g. when using the target model probability in the model- attack the n-gram probability in the data-based attack), we truly probability for (z)))) However, in the case of the data-based attack using similarity metrics, we use the mean similarity to the k closest synthetic sequences which does not correspond to true probability. In this case, we normalize similarities to fall in the range [0, 1] and use (2(r) as an empirical proxy for the probability P(x| ().
In practice, P(x 0) can an extremely small value, particularly when calculated as a product of token-level conditional probabilities, which can lead to underflow errors. To mitigate this, we perform arithmetic operations on prgpbobbasilities whenever possible. However, in the context of equation (4), where the denominator involves averaging probabilities, we employ quad precision floating- point arithmetic. This method is sufficiently precise to handle probabilities for sequences of up to 50 words, which is the maximum we consider in our experiments.
C. Prompts used to generate synthetic data
Table 5 summarizes the prompt templates p()) used to generate synthetic data for all datasets (see Sec. 4).
Dataset
Template p())
Labels f
SST-2
"This is sentence with l sentiment:
(positive, negative)
AG News
"This is news article about E:
World, Sport, Business, Sci/Tech) fentailing, neutral, contradicting)
SNLI
"A premise with hypothesis:"
Table 5. Prompt templates used to fine-tune models and generate synthetic data.
D. Implementation details
To generate synthetic data throughout the experiments in this paper, we fine-tune the pre-trained model Mistral-7B (Jiang et al., 2023) using LoRA with r 4, including all target modules (updating 10.7M parameters in total).
We optimized training hyperparameters for LoRA fine-tuning Mistral-7B on SST-2 by running a grid search over learning rate ([L x 10- 6 4 10 2 10 1,055,4 6 10-5,3 10-4, x 10 10-31) and batch size ((64, 128, 256]). We fine-tuned the models for 3 epochs and observed the validation loss plateaued after the first epoch. Based these results, we selected learning rate of 2 x 10-5, effective batch size of 128, sequence length 128, LoRA r = 4 and fine-tuned the models for I epoch. Figure 3 shows the validation cross-entropy loss for SST-2 over the grid we searched on and the train and validation loss curves for 3 epochs with the selected hyperparameters.
All our experiments have been conducted on a cluster of nodes with 8 V100 NVIDIA GPUs with a floating point precision of 16 (fpl 6). We built our experiments on two open- packages: (i) pr i vacy eecoyy--stimttttte imates which provides a
15

---

The Canary's Echo: Auditing Privacy Risks of L.M--Genrated Synthetic Text
Train Validation
Validation Heatmap
3.8
1es8
2.8
5.5
4e-00
2:7
2.7
3.4
5.0
Validation Loss
Loss
2e-25
2.8
2.7
2.7
Learning re-d5
4.5
2.9
2.5
3e-04
3.2
1e63
2.0
128 Batch Size
256
2
Epoch
(a) Grid search
Loss curve
Figure (a) Validation cross-entropy loss LoRA fine-tuning Mistral-7B on SST-2 varying the leamingg rate and effective batch size. (b)
Training and validation loss for best hyperparameters over epochs.
distributed implementation of the RMIA attack and (ii) dp-transformers which provides the implementation of the synthetic data generator.
E. Detailed assumptions made for the adversary
We clarify the capabilities of adversaries model- and data-based attacks according to the threat model specified in Section 2. note:
1. A model-based attack is strictly more powerful than data-based attack. This is because with access to the fine-tuned model 0 and the prompt template P(-), model-based attack can synthesize D for any set of synthetic labels and perfectly simulate the membership inference experiment for a data-based attack.
In both threat models, the adversary can train reference models M This assumes access to the private dataset D, and the training procedure of target model 0, including hyperparameters. This made clear in line 3 in Algorithm
3. In our experiments, we consider model-based attacks that use the prompt template p(.) to compute the model loss for target records, as specified in Sec. 3.1.1. Our data-based attacks use the prompt template p(-) to generate synthetic data D from reference models.
4. Only the model-based attack has query-access to the target model 0. The attacks used in our experiments use 0 to compute token-level predicted logits for input sequences and do not use white-box features, although this is not excluded by the threat model.
5. Only the data-based attack generates synthetic data from reference models, only this threat model leverages the sampling procedure s-mple(-).
Table 6 summarizes the adversary capabilities used in the attacks in our experiments.
F. Synthetic data utility
To ensure we audit the privacy of synthetic text data in realistic setup, the synthetic data needs to bear high utility. We measure the synthetic data utility by comparing the downstream classification performance of RoBERTa-base (Liu et al., 2019) when fine-tuned exclusively on real or synthetic data. We fine-tune models for binary (SST-2) and multi-class classification (AG News) for epoch on the same number real or synthetic data records using batch size 16 and learning rate 7) = 10-5,
We report the macro-averaged AUC score and accuracy on held-out test dataset real records.
16

---

The Canary's Echo: Auditing Privacy Risks of L.M--Genrrated Synthetic Text
Assumptions
Model-based MIA
Data-based MIA
Knowledge of the private dataset D used to fine-tune the target model (apart from knowledge of canaries).
Knowledge of the training procedure of target model 6.
Knowledge of the prompt template p((;) used to generate the synthetic data.
Query-access to target model 0, returning predicted logits.
Access to synthetic data D generated by target model 6.
Knowledge of the decoding strategy employed to sample synthetic data D (e.g., temperature, top-k).
Table 6, Adversary capabilities effectively used by attacks in our experiments.
Table 7 summarizes the results for synthetic data generated based on original data which does not contain any canaries. While we do see a slight drop in downstream performance when considering synthetic data instead of the original data, AUC and accuracy remain high for both tasks.
Classification
Fine-tuning data
Dataset
AUC
Accuracy
Real Synthetic
0.984
92.3 %
SST-2
0.968
91.5 %
Real Synthetic
0.992
94.4 %
AG News
0.978
90.0 %
Table Utility of synthetic data generated from real data without canaries. We compare the performance of text classifiers trained on real or synthetic data-hoth evaluated on real, held-out test data.
We further measure the synthetic data utility when the original data contains standard canaries (see Sec. 5.1). Specifically, we consider synthetic data generated from a target model trained on data containing 500 canaries repeated nepp = 12 times, so 6000 data records. When inserting canaries with an artificial label, we remove all synthetic data associated with labels not present originally when fine-tuning the RoBERTa-base model.
Canary injection
Classification
Dataset
Source
Label
AUC
Accuracy
In-distribution
0.972
91.6 %
SST-2
Natural Artificial
0.959
89.3 %
Synthetic
0.962
89.9 %
In-distribution
0.978
89.8 %
AG News
Natural
0.977
88.6 %
Synthetic
Artificial
0.980
90.1 %
Table Utility of synthetic data generated from real data with canaries (Neep 12). compare the performance text classifiers trained on real or synthetic data Bata-both evaluated on real, held-out test data.
Table 8 summarizes the results. Across all canary injection methods, find limited impact of canaries the downstream utility of synthetic data. While the difference is minor, the natural canary labels lead to the largest utility degradation. This makes sense, as the high perplexity synthetic sequences likely distort the distribution of synthetic text associated with a certain real label. In contrast, in-distribution canaries can seen as up-sampling certain real data points during fineiuunin, while canaries with artificial labels merely reduce the capacity of the model to learn from real data and do not interfere with this process as much as canaries with natural labels do.
17

---

The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
G. Additional results for MIAs using standard canaries
In line with the literature on MIAs against machine learning models (Carlini et al., 2022a), we also evaluate MIAs by their true positive rate (FPR) low false positive rates (FPR). Tables 9 and 10 summarize the MIA TPR at FPR=0.01 and FPR=0.1, respectively. We also provide the ROC curves for the data-based MIAs for both datasets, considering canaries with natural labels in Figure 4.
Canary injection
TPR @ FPR=0.01
Model _A'
Synthetic AD (2-gram)
Synthetic AD (SIM
Synthetic AD cmb)
Dataset
Source
Label
In-distribution
0.148
0.104
0.029
0.020
SST-2
Natural Artificial
0.972
0.042
0.018
0.024
Synthetic
0.968
0.057
0.000
0.030
In In-distribution
0.941
0.050
0.032
0.016
AG News
Natural 0.955
0.049
0.006
0.016
Synthetic
Artificial
0.990
0.053
0.041
0.022
Table True positive rate (TPR) false positive rate (FPR) of 0.01 for experiments using standard canaries (Sec. 5.1) across training datasets, threat models (model-based adversary A and data-based adversary AP) and MIA methodologies. Canaries synthetically generated with target perplexity Parget 250, with no in-distribution prefix (F = 0) and inserted Thep 12 times.
Canary injection
TPR FPR=01
Model A
Synthetic A (2-gram)
Synthetic A² (SIM)ac)
Synthetic A² (SIMemb)
Dataset
Source
Label
In-distribution
0.795
0.406
0.207
0.203
SST-2
Natural Artificial
0.996
0.191
0.114
0.128
Synthetic
1.000
0.277
0.142
0.142
In-distribution
0.982
0.314
0.158
0.168
AG News
Natural
0.990 0.996
0.271
0.114
0.114
Synthetic
Artificial
0.323
0.152
0.164
Table 10. True positive rate (TPR) false positive rate (FPR) of 0.1 for experiments using standard canaries (Sec. 5.1) across training datasets, threat models (model-based adversary AA and data-based adversary AP) and MIA methodologies. Canaries are synthetically generated with target perplexity Punge* 250, with no in in-distribution prefix (F" = 0) and inserted nepp 12 times.
H. Ablations for MIAS on synthetic data
Synthetic multiple Thus far, we have exclusively considered that the number of generated synthetic records equals the number of records in the real data, i.e., N N. We now consider the case when more synthetic data is made available to data-based adversary (A). Specifically, we denote the synthetic multiple perform for varying values of m. Figure 5 shows how the ROC AUC score varies as increases. As expected, the ROC AUC score for the attack that uses membership signals computed using a 2-gram model trained on synthetic data increases when more synthetic data available. In contrast, attacks based on similarity metrics do not seem to benefit significantly from this additional synthetic data.
R/N and evaluate how different MIAs
Hyperparameters in data-based attacks The data-based attacks that we presented in Sec. 3.1 rely certain hyperpa- rameters. The attack that uses n-gram models to compute membership signals is parameterized by the order n. Using a too small value for n might not suffice to capture the information leaked from canaries into the synthetic data used to train the n-gram model. When using a too large order n, on the other hand, we would expect less overlap between n- n-grams present in the synthetic data and the canaries, lowering the membership signal.
18

---

The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
1
0.8
0.8
0.6
0.6
TPR
TPR
0.4
0.4
0.2
Z-gram
0.2
2- gram
SIM cmm
k
25
SIM
k = 25
SIM jac
25
SIM A 25
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
FPR
FPR
(a) SST-2
(b) AG News
Figure 4. MIA ROC curves across data-based MIA methodologies for the SST-2 (left) and AG News (right) datasets. Canaries are synthetically generated with target perplexity of Purga 250 with natural label, with no in-distribation prefix (F 0) and inserted Nep 12 times.
Further, the similarity-based methods rely on the computation of the mean similarity closest synthetic records to the canary. When & is very small, c.g. 1, the method takes into account single synthetic record, potentially missing on leakage of membership information from other close synthetic data records. When A becomes too large, larger regions of the synthetic data are taken into account, which might dilute the membership signal among the noise.
Table 1l reports the ROC AUC scores of data- based attacks for different values of the hyperparameters n and k when using standard canaries (Sec. 5.1). We find that for both datasets, training a 2-gram model on the synthetic data to compute the membership signal yields the best performance. For the data-based MIAs relying on the similarity between the canary and the synthetic records, both when considering Jaceard distance and cosine distance the embedding space, we find that considering the k 25 closest synthetic records yields the best performance.
7-gram
SIMIac
SIMemb
Dataset
72
AUC
AUC
AUC
1
0.415
0.520
0.516
2
0.616
0.535
5
0.516
SST-2
3
0.581
10
0.538
10
0.519
0.530 4
25
0.547
25
0.530
1
0.603
0.522
0.503
2
0.644
5
0.525
5
0.498
AG News
3
0.567
10
0.537
10
0.503
4
0.527
25
0.552
25
0.506
Table I.. Ablation over hyperparameters of data-based MIAs. We report ROC AUC n and (see Scc. 3) Canaries are synthetically generated with target perplexity Purget 250, with natural label, with no in-distribution prefix (F 0), and inserted neep 12 times.
different values of the hyperparameters
Disparate vulnerability of standard canaries
We analyze the disparate vulnerability of standard canaries between the model-based attack and the data-base based attack that uses 2-gram model (as discussed Sec 5.1). Figure 6 plots the RMIA scores for both attacks on the same set canaries, which have either been included in the training dataset of the target model (member) or not member)... Note that the RMIA scores are used to distinguish members from oo-members, and that larger value corresponds to the adversary
19

---

The Canary's Echo: Auditing Privacy Risks of LLM-GGnerated Synthetic Text
1,0
Model
1,0
Model
Synthetic (2- (2-gram)
Synthetic (2-gram)
Synthetic (SIM)e k 25)
Synthetic (SIM,c k- 251
0.9
Synthetic (SIM k 25)
0.9
Synthetic (So k= 25)
Random guess baseline
Random guess baseline
MIA AUC
AUC
0.7
0.6
0.6
0.5
0.5
20
26
Synthetic multiple m
Synthetic multiple
Figure ROC AUC score for inereasing value the synthetic multiple m across data-based attack methods for SST-2 (left) and News (right). Canaries synthetically generated with target perplexity of Purget 250, with natural label, with in-distribution prefix (F 0), and inserted nepp 12 times.
being more confident in identifying record member, i.e., the record being more vulnerable.
First, note that the scores across both threat models exhibit statistically significant, positive correlation. We find Pearson correlation coefficient between the RMIA scores (log) for both methods of 0.20 (p-value of 2.4 10-19) and 0.23 (p-value of 1.9 x 10 10-13) for SST-2 and AG News, respectively. This means that a record vulnerable to the moodel-based attack tends to be also vulnerable to the data-based attack, even though the attacks differ substantially.
Second, and more interestingly, some canaries have disparate vulnerability across MIA methods. Indeed, Figure shows how certain data records which are not particularly vulnerable the model-based attack are significantly more vulnerable the data-based attack, and vice versa.
50
20
RMIA scores (log)- Synthetic (2-gram) AUC-0.624 .2620
0
0
scores (log) Synthetic (2- (2-gram) AUC=0.680
-2.5
5.0
-5.0
-7.5
0.0
-2.
-12.5
-15.0
Members
-7.
Members
Non-members
Non-members
Correlation=0.20s
-10.0
Correlation=0.244
-50
-60
-0
-20
20
25
-125
-100
-75
-50
-25
25
(log)- Model AUC=0.999
(log) Model AUC=0.999
(a) SST-2
(b) AG News
Figure 6. RMIA scores (log) for model- and data-based MIAS the same set of canaries. Results for both datasets SST-2 and AG News. Canaries are synthetically generated with target perplexity of Punger 250 with a natural label, inserted Neep 12 times.
20

---

The Canary's Echo: Auditing Privacy Risks of LLM--Genrated Synthetic Text
J. Low FPR ROC results
Figure shows log-log plots the ROC curves in Figure to better examine behavior of attacks at low FPR.
10P
A Per 10
T1R
10-
A Mepp
A
TPR
10
1 Per 10P
PLr
TPR
A
1 Pkyy
A Pa 10"
A
4 Plep
A. Thop
16
A
A Par
10a
AP F0 30
AP,F
A Zlep
10-2
A
10
10-2
1 =0=0=
A
Por 10
AP, F
10%
10
10
10
10
100
10
10
10"
TPR
FPR
FPR
(a) Number of canary repetitions Teep.
(b) Canary perplexity Parget.
(c) Canary in-distribution prefix F.
P urgeet
31, P
O.
Tep
4, Ti
16,
Panes
31, Pepp
10p
10
10
AD, Par
TPR
10-1
May
TIR
AD Per 10a
10
A A
1 Mlpp
AP, 10"
AD F
4 Map
4 PIN
4 Plep
1 Plop
16
1 Por
1 Par
A 20
A
1 Thep
A,, Par
10-2
A",
A Proz
A,, Par
max
10
10P
10-2
10
10"
FPR
FPR
FPR
(d) Number of canary repetitions nhep
(e) Canary perplexity Parget
(f) Canary in in-distribution prefix F. 4, ntup
Punger 31, F = 0.
Pep
4, Trep
1)
16, == O.
Panget 31, Tepp
16.
Figure 7. Log-log ROC curves of MIAs synthetic data A compared model-based MIAs A SST-2 (7a-7c) and AG News (7d-7f). We ablate over the number of canary insertions nhep in 7a, 7d, the target perplexity Purger of the inserted canaries in 7b, 7c and the length F of the in-distribution prefix in the canary in 7c, 7f.
K. Interpretability
To further understand the membership signal for data-based attacks, examine some examples in-depth.
Specifically. we consider the MIA for specialized canaries with F = 30, Pirgett = 31 and Thep = 16 for SST-2 from Figure 1c. Recall that for this attack, we consider 1000 canaries, 500 which are injected into the training dataset of one target model 0. We also train 4 references models {0]] where each the 1000 canaries has been included in exactly half. We focus on the best performing MIA based on synthetic data, i.c. the attack leveraging the probability of the target sequence computed using a 2-gram model trained on the synthetic data.
To understand what signal the MIA picks up to infer membership. we focus the canary most confidently, and correctly, identified member and the one most confidently, and correctly, identified as non- mn-mee.er. For this, we take the canaries for which the RMIA score computed using the target model and the reference models is the highest and the lowest, respectively.
Next, for each model (4 reference models, and target model), we report for this canary d;;
1. Whether the canary has been included in, à; E D (IN), excluded from, d; D (OUT), the training dataset of the model question, and thus to generate the synthetic data D (7. (Fi, 2
21

---

The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
2. The canary with the words that appear as 2-gram in the synthetic data D emphasized in bold face. Note that if, for instance, this is sequence of 3 words, e.g.. "the woodman seems", this means that all 3 words appear in 2-grams in the synthetic data, e.g., "the woodman" and "woodman seems".
The maximum overlapping sub-string between the canary and any synthetically generated record 8,. We define a sub-string sequence of characters, including white space, and also report its length as number of characters Lovertap
4. The mean, negative cross-entropy loss of the canary computed using the 2-gram model trained on the synthetic data. Formally, for canary 8; (W1, 02...... ..........:: Σド= log (P2 gram grammm() (Wj, w)-1))-
Tables 12 and 13 report this for the canary with the largest and lowest RMIA score, respectively.
First, we analyze the membership prediction made for the canary with the largest RMIA score (Table 12). Examining the reference models (f), we find little variation in the metrics we consider, regardless of whether the canary was included in the training dataset (IN) not (OUT). Specificaly. the number of overlapping 2-grams, the length of the longest overlapping sub-string, and the 2-gram loss remain largely unchanged across IN and OUT reference models.
In contrast, the target model f exhibits a strong signal, especially when compared to the reference models. Notably, the uncommon sequence "Embed from Getty Images Embed from Getty Images" appears in the synthetic data generated by the trained target model 0 but absent from the synthetic data of all 0% The signal is further reflected by a significantly lower 2-gram loss compared to the reference models, explaining the high RMIA score for this canary.
Overall, even for the most vulnerable canary, not all of its 2-grams appear in the synthetic data, and the longest overlapping sub-string accounts for only 52 out of 296 characters. This suggests that membership inference does not rely verbatim regurgitation of long sub-sequences. Instead, it detects subtler patterns, such as the presence of specific 2-grams shorter sub-strings. Such signal is effectively captured by the 2-gram loss and becomes especially meaningful when contrasted against values reached for the reference models using RMIA.
Second, we analyze the membership prediction for the canary with the lowest RMIA score (Table 13). In this case, the canary was not included in the target model's training dataset (OUT) and was correctly classified as non-member.
We observe minimal differences in the number of overlapping 2-grams and the length of the longest overlapping sub-string across IN and OUT reference models, as well as the target model. Instead, the most informative signal emerges from the 2-gram loss: it is lower for IN models than for OUT models, with the target model exhibiting the highest loss, resulting in the low RMIA score. These results again suggest that the information useful to infer membership based on synthetic data does not rely on the regurgitation of long sub-sequences, and instead arises from slight shifts in the probability distribution of co-occurrences of words the synthetic data, as captured by the 2 2-gram loss.
22

---

The Canary's Echo: Auditing Privacy Risks of LeM-Grnerated Synthetic Text
IN or
Canary
Max overlapping sub-string
2 2-gram loss
Model
OUT
(words present part of 2-grams in D' in bold)
81 (ref)
IN
"the woodman seems have directly influenced this girl- meets-girl love story, but even more reassuring is how its makers actually seem to understand what made allen 's romantic comedies work in the first place. Embed from Getty Images Embed from Getty Images Earlier this week, the case against the"
understand what made >>
8.21
Loverlup 25
82 (ref)
IN
"the woodman seems to have directly influenced this girl- meets-girl love story, but even more reassuring how its makers actually seem understand what made allen 's romantic comedies work in the first place. Embed from Getty Images Embed from Getty Images Earlier this week, the case against the"
<ally seem to understand Lovertap 24
8.19
8' (ref)
OUT
"the woodman seems to have directly influenced this girl- meets-girl love story, but even more reassuring is how its makers actually seem to understand what made allen romantic comedies work the first place. Embed from Getty Images Embed from Getty Images Earlier this week, the case against the"
seem understand what mas Loverlap 27
8.18
032 (ref)
OUT
"the woodman seems to have directly influenced this girl- meets-girl love story, but even more reassuring how its makers actually seem understand what made allen 's romantic comedies work in the first place. Embed from Getty Images Embed from Getty Images Earlier this week, the case against the"
<s work the first place> Lowerlap
8.18
0 (target)
IN
"the woodman seems to have directly influenced this girl- meets-gin love story, but even more reassuring is how its makers actually seem to understand what made allen romantic comedies work in the first place. Embed from Getty Images Embed from Getty Images Earlier this week, the case against the"
<e. Embed Getty Images Embed from Getly Images Es Loveclop 52
7.59
Table 12. Interpretahility of the best MIA (2-gram) based on synthetic data for specialized canaries with = 30, Purger 31 and Neep 16 for SST-2 from Figure Results across 4 reference models and the target model for the canary with the largest RMIA (most confidently and correctly identified member by the MIA). Words in bold appear in 2-grams in D'. The largest generated sub-sequence of the canary in D' corresponds to the maximum overlapping sub-string, not the longest sequence of words in bold.
23

---

The Canary's Echo: Auditing Privacy Risks of LeM-Greerated Synthetic Text
IN or OUT
Canary
Max overlapping sub-string
2-gram
Model
(words present as part of 2-grams in D' in bold)
loss
8 (ref)
IN
"give spark to chasing amy and changing lanes falls flat thinking man cia agent jack ryan in this summer 's new action film, the sum of all fears, " in theaters friday. if director philip noyce and writer aaron singer"
the of all fears, * Loverlap 30
7.80
0/ (ref)
IN
"give a spark to " chasing amy " and changing lanes " falls flat as thinking man cia agent jack ryan in this summer new action film, " the sum of all fears, " in theaters friday director philip noyce and writer aaron singer"
<, the sum of all fears w Lovertap 26
7.73
8/ (ref)
OUT
"give spark chasing amy and * changing lanes falls flat thinking man cia agent jack ryan in this summer 's new action film the sum of all fears, in theaters friday if director philip noyce and writer aaron singer"
the sum of all fears Lovertap 27
8.27
04 (ref)
OUT
"give spark to chasing amy and channing lanes falls flat as thinking man cia agent jack ryan in this summer new action film, the sum of all fears, " in theaters friday. if director philip noyce and writer aaron singer"
" "hhasing amy and changing lanes Loverlap 41
7.99
(target)
OUT
"give a spark to " "chasing amy and changing lanes falls flat as thinking man eia agent jack ryan this summer new action film, the sum of all fears, in theaters friday if director philip noyce and writer aaron singer"
" the sum of all fears, > Loverlap 30
8.30
best MIA (2-gram) based on synthetic for specialized canaries with 30, Purget and Nep 16 for SST-2 from Figure le. Results across reference models and the target model for the canary with the smallest RMIA score (most confidently and correctly identified as non-member by the MIA). Words in bold appear in 2-grams in D'. The largest generated sub-sequence of the canary 7'' corresponds the maximum overlapping sub-string, not the longest sequence of words bold.
Table 13. Interpretability
24


Reviews:
[{"review_id": "3FfzRW6XOo", "reviewer": "Reviewer_L9fq", "summary": "This paper aims at investigating the privacy risks of synthetic text generated by LLMs by developing a new membership inference attack (MIAs). The main novelty of the proposed approach is that the adversary model considered only has access to the synthetic text generated by the model and not the model itself. Two MIAs are proposed for this setting.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback; we provide detailed responses below. \n\n> Difference between model-based and data-based MIAs supported only through experiments on two datasets. A large set of variations of these experiments have been conducted, which demonstrate the robustness of the proposed approach.\n\nWe are glad the reviewer thinks our experiments demonstrate the robustness of our approach. We chose to investigate the gap between model- and data-based attacks in depth through detailed ablation studies rather than shallowly on a broader range of datasets. \n\nWe provide results for the *n*-gram based MIA on SNLI (same setup as Table 1), which we will include in a revision. Results suggest the same trends we report carry over to other datasets.\n\n|Canary injection|||||\n|-|-|-|-|-|\n|||AUC|TPR\\@0.01|TPR\\@0.1|\n|In-distribution||0.718|0.122|0.443|\n|Synthetic|Natural|0.534|0.016|0.111|\n||Artificial|0.718|0.061|0.412|\n\n> Difference between model-based and data-based attacks not well explained\n\nWe thoroughly describe the difference between model- and data-based MIAs in Sec. 2, including pseudocode in Alg. 1. Appendix E contains additional discussion on the difference between the threat models. We present concrete model- and data-based attacks in Sec. 3.1, including the calculation of membership inference signal and our adaptation of RMIA. We detail the choice of hyperparameters and how we evaluate attacks in practice in Sec. 4. We include additional details about model- and data-based attacks in Appendices A, B.\n\nWe welcome suggestions to make this clearer if what we already provide is not enough.\n\n> The process for generating a synthetic text is unconventional\n\nWe generate synthetic data as done conventionally (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023), by prompting the model on label-dependent templates (cf. Appendix C), so that text is generated based on the indication of the prompt.\n\n> The fact that the synthetic dataset should be the same size as the original is not realistic\n\nWe report on experiments on synthetic datasets 2-8x larger than the original in Appendix I.\n\nWe focus on generating synthetic data derived from a private dataset so that e.g. downstream tasks on the synthetic dataset have similar utility. Hence, in our main experiments we generate datasets matching the size and label histogram of the private data. This is the same setting studied by Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023.\n\n> Possibility that the training set and the canaries have been used during pre-training\n\nMistral-7B's training data is not public, so we cannot rule out SST-2/AG News being included in the training data. However, we aim to measure the difference in performance between model- and data-based MIAs on the fine-tuning dataset of a model used to synthesize data, which training and fine-tuning data overlap would affect similarly.\n\nWe can, however, rule out the presence of canaries in the training data when they are constructed artificially (in-distribution prefix F=0; Table 1, Fig. 1(a,b,d,e), Fig. 2). Especially at high perplexities, these canaries are likely absent from the training data. For canaries with F>0 and if parts of the datasets were included in pretraining, it would, if anything, make MIAs more challenging. We will include this discussion in an eventual revision.\n\n> No justification of the choice of parameters used for the two variants of the MIA proposed.\n\nWe discuss hyperparameter selection in Appendix H (paragraph “Hyperparameters in data-based attacks”). We consistently find the best performance for *n*=2 (*n*-gram MIA) and *k*=25 (number of closest synthetic records for similarly based MIAs), which we used in the main experiments. \n\n> differentially-private variants of the model training.\n\nWe will add a section on mitigations, focusing on fine-tuning the target model with DP-SGD before generating synthetic data as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on SST-2 [3]), we expect performance of model-based attacks to decrease to a random guess baseline under DP. Since data-based attacks underperform compared to model-based attacks and guarantees transfer to synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper. \n\n[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text \n\n[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models. \n\n[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).\n\n> Fig.5 legibility.\n\nThanks for pointing this out, we will address this.", "claims_and_evidence": "Overall, the claims with respect to the difference between model-based and data-based MIAs are supported only through experiments on two datasets, which raises some doubts on whether the observed results will carry to other datasets. However, a large set of variations of these experiments have been conducted, which demonstrate the robustness of the proposed approach.", "methods_and_evaluation_criteria": "The difference between model-based attacks and data-based attacks are not well explained. The process for generating a synthetic text is also unconventional as it seems to indicate that the objective of the model is to create texts that match desired labels while in general the text will be generated based on the indication of the prompt. In addition, the fact that the synthetic dataset should be the same size as the original dataset is not very realistic are most LLMs are usually trained on a very large corpus (e.g., possibly the whole Internet). This major issue should at least be acknowledged and discussed in the paper.", "theoretical_claims": "Currently, the authors do not discuss the possibility that the training set and the canaries have been possibly used during the training of the LLM rather than simply during the fine-tuning. In particular as the Stanford Sentiment Treebank and the AG news datasets have respectively been published in 2013 and 2015, there is a high chance that they have been seen in the training set of the LLM models considered.", "experimental_designs_or_analyses": "Overall, the experimental evaluation is well-explained and seems sound. There is however no justification of the choice of parameters that have been used for the two variants of the MIA proposed. The paper also lacks experiments for evaluating how the success of the proposed attack will fare against differentially-private variant of the model training.", "relation_to_broader_scientific_literature": "The authors have done a good job at reviewing previous works on membership inference attacks against LLMs and synthetic tabular data. The proposed approach is also well-situated compared to existing works although the adversary model considered is non-standard.", "essential_references_not_discussed": "Essential references, including recent ones, have been cited in the paper.", "other_strengths_and_weaknesses": "The main novelty of the proposed approach is the adversary model considered, which only leverages the synthetic produced, and proposes two MIAs for this setting.", "other_comments_or_suggestions": "Figure 5 in the supplementary material has some issues with the corresponding legends.", "questions_for_authors": "Do you have any way to verify if the two datasets considered for the experiments were not part already of the training set of the LLM considered?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbhQe4uw8w", "reviewer": "Reviewer_qpCC", "summary": "This paper proposes to audit the privacy risks from synthetic data generated by LLMs as synthetic data is becoming increasingly prevailing in different applications. The authors found that the typical canaries designed for model-based auditing were not effective for auditing the synthetic data. The paper proposed a new design for canaries that is better suited for auditing the data-based scenario. The method is analyzed empirically on benchmark datasets with various evaluation metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We provide responses for the concerns raised below. \n\n> (1) analyzing how different domains might impact the auditing efficiency\n\nIn Table 1, we also study the effect of which labels (or domains) the canaries belong to. In particular, we consider both ‘natural’ and ‘artificial’ labels associated with canary samples, where ‘natural’ corresponds to labels from the same distribution as the labels from the original dataset and ‘artificial’ corresponds to a new, canary-specific label (see Sec. 4). We observe a slight increase in MIA performance across all data-based MIAs, suggesting that more rare, potentially artificially crafted labels make canaries more vulnerable. We leave a more thorough study on this effect to future work and will add this to the discussion section. \n\n> (2) measuring the leakage against privacy-preserving methods\n\nWe will add a section on mitigations strategies for our novel MIAs, focusing on fine-tuning the target model with DP-SGD before generating synthetic data, as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on STT-2 [3]), we expect the performance of model-based attacks to quickly decrease to a random guess baseline under DP guarantees. Since data-based attacks underperform compared to model-based attacks, and guarantees transfer to the synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper. \n\n[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text \n\n[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models. \n\n[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).\n\n> (3) motivation for using n-gram for data-based attacks is not clearly described\n\nMany thanks for pointing this out. We have opted to train an n-gram model rather than a small, transformer-based model due to its simplicity and computational cost. Indeed, training the n-gram model on the synthetic data, both for SST-2 and AgNews takes less than 1 CPU minute. We further provide ablations of the value of n in Appendix H and Table 10, where we consistently find n=2 to be the optimal value. We will add the suggestion of training a small neural network instead to the discussion section. \n\n> (4) How does the size of synthetic data impact the auditing?\n\nWe provide ablations for different sizes of the synthetic data in Appendix H and Figure 5. While we observe an improved performance for the n-gram based MIA as more synthetic data is generated, we maintain our main analysis considering a synthetic dataset of equal size to the private dataset as this is more realistic and used in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023).  \n\n>  (5) How do different embedding models impact the auditing?\n\nFor similarly-based methods, we opted for paraphrase-MiniLM-L6-v2 from sentence-transformers as the embedding model, as it offers great performance in semantic search benchmarks [link](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html). As the performance of the MIA based on semantic similarity using this embedding model is outperformed by all other data-based MIAs (Table 1), we did not further ablate the choice of the embedding model. We leave this to future work and will add this to the discussion section.", "claims_and_evidence": "The claims are supported by the evidence under the assumptions that the author made.", "methods_and_evaluation_criteria": "The evaluation criteria makes sense to demonstrate the improvement in the auditing performance.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment designs are thorough. Though could be improved by analyzing how different domains might impact the auditing efficiency, e.g. would synthetic data generated for different domains in AG news make any difference in the auditing performance.", "relation_to_broader_scientific_literature": "The paper is related to better understanding of privacy leakage through synthetic data which is a novel and important topic in the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is considered a novel auditing scenario, as private synthetic data generation is more and more popular, understanding the privacy leakage from synthetic data is critical. \n2. The paper is well-written with clear methodology, and thorough analysis on why the existing canaries failed and how to craft canaries for synthetic data auditing.\n\nWeaknesses:\n1. The evaluation can be strengthened by measuring the leakage against privacy-preserving methods such as private evolution, and DP fine-tuning for synthetic data generation.\n2. The motivation for using n-gram for data-based attacks is not clearly described. Why n-gram is preferred, why not consider training a small neural network based model on synthetic data? How is n chosen?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does the size of synthetic data impact the auditing?\n2. For similarity scores based on embeddings, how do different embedding models impact the auditing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrN1oUamLW", "reviewer": "Reviewer_MC8u", "summary": "The paper investigates the privacy risks associated with releasing synthetic data generated by Large Language Models (LLMs). It explores how much information about the original training data can be extracted from such synthetic data, even when adversaries do not have direct access to the fine-tuned model.\n\n\n- **Synthetic Data Leakage:** MIAs using only synthetic data can detect membership with AUC scores significantly above random, showing that synthetic text leaks training information.\n- **Attack Comparison:** There's a gap between model-based and data-based attacks; canaries effective in one setting require much higher occurrence to be vulnerable in the synthetic data scenario.\n- **Improved Canary Design:** The paper proposes canaries with an in-distribution prefix and high-perplexity suffix, enhancing their detectability in synthetic outputs for more reliable privacy auditing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We provide responses for the concerns raised below. \n\n> (1) A minor limitation is that the experiments are conducted on only two datasets\n\nWe provide results for the *n*-gram based MIA for the SNLI dataset (for the setup from Table 1) below and will include this in the paper. These results suggest the same trends we report carry over to other datasets.\n\n|Canary injection|||||\n|-|-|-|-|-|\n|||AUC|TPR\\@0.01|TPR\\@0.1|\n|In-distribution||0.718|0.122|0.443|\n|Synthetic|Natural|0.534|0.016|0.111|\n||Artificial|0.718|0.061|0.412|\n\n> (2) How do attacks work on differentially private text generation?\n\nWe will add a section on mitigations strategies for our novel MIAs, focusing on fine-tuning the target model with DP-SGD before generating synthetic data, as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on STT-2 [3]), we expect the performance of model-based attacks to quickly decrease to a random guess baseline under DP guarantees. Since data-based attacks underperform compared to model-based attacks, and guarantees transfer to the synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper. \n\nWe leave other defense strategies (e.g. using paraphrasing techniques) as proposed by the reviewer for future work, and will elaborate on this in the discussion section. \n\n[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text \n\n[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models. \n\n[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The authors examine data-driven MIA, proposing a fresh framework that offers a more realistic assessment of threats compared to model-based MIA.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n\n- The work shifts the focus from traditional model-based MIAs to attacks based solely on synthetic data, addressing a realistic threat model where adversaries do not have direct access to the fine-tuned model.\n- By proposing specialized canaries that blend an in-distribution prefix with a high-perplexity suffix, the authors enhance the detection capability of data-based MIAs, making privacy auditing more effective.\n\n\nWeaknesses: \n\n- A minor limitation is that the experiments are conducted on only two datasets, which may not capture the full diversity of real-world scenarios. The effectiveness of the proposed techniques across different domains or larger-scale datasets remains to be validated.", "other_comments_or_suggestions": "How do attacks work on differentially private text generation (text-to-text privatization) [1, 2, 3, 4]? Recent studies [2, 4] have demonstrated that paraphrasing techniques can achieve a highly favorable privacy-utility trade-off. I encourage the authors, if they have time, to explore simple paraphrasing-based DP methods, as they are relatively easy to implement and serve as strong defenses. A brief discussion on defenses, supported by some results, would greatly benefit readers seeking defense strategies, and if the authors provide such insights, I would be happy to change my rating to strong accept.\n\nReferences: \n\n[1] Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations .WSDM 2020\n\n[2] The Limits of Word Level Differential Privacy, EMNLP 2022.\n\n[3] TEM: High Utility Metric Differential Privacy on Text, SIAM, 2023\n\n[4] Locally differentially private document generation using zero shot prompting, EMNLP 2023", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BB9u862rBa", "reviewer": "Reviewer_Ni1p", "summary": "This paper argues that the synthetic data generated by LLMs, which are finetuned on private data, poses privacy risks. They identify a new class of canaries suited for these data-based privacy risks, and show that by choosing an in-distribution prefix and out-of-distribution suffix they can greatly increase the vulnerability of the synthetic data.\n\n## update after rebuttal\nAfter the rebuttal, in which I mainly asked the authors to add some additional baselines/ablations, I keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We provide responses for the concerns raised below. \n\n> (1) one could imagine finetuning on D_tilde and then applying standard MIAs to the finetuned model\n\nMany thanks for pointing this out. We have opted to train an n-gram model on the synthetic data rather than a larger, transformer-based model due to its simplicity and computational cost. Indeed, training the n-gram model on the synthetic data, both for SST-2 and AgNews takes less than 1 CPU minute. We will add the suggestion of training a full LLM on the synthetic data instead to the discussion section. \n\n> (2) include a discussion on the privacy risks of D \n\nWe provide results on fully in-distribution canaries (randomly sampled from D, no out-of-distribution suffix or F=max) throughout our work (in-distribution results in Table 1, and results for F=max in Figure 2 (c,f) and Table 2). Due to the lower perplexity of in-distribution sequences, we recover that data-based MIAs work quite well for these samples, especially compared to the high-perplexity canaries commonly used for model-based attacks. However, in these experiments, we consider the member canaries repeated n_rep times in the training data with n_rep up to 12 and 16. From Figure 2(a,d), we learn that when n_rep decreases, the MIA performance drops to no better than a random guess baseline. We hence anticipate that, at least in our experimental setup, the privacy risks associated with sequences appearing only once in D remains low. We will elaborate on this in the discussion section. We believe this also answers the reviewer’s last question (i.e. reporting the performance of the data-based MIAs on non-canaries belonging to D as a baseline). \n\n> (3) Could you report CIs for some of the major results\n\nFor our main results (e.g. Table 1), we report ROC AUC as the performance of the MIA, representing an average performance over all (1000) canaries. \n\nGetting meaningful confidence intervals for these results requires training multiple (10+) target models, which is computationally quite expensive and was not feasible within the rebuttal period. We will run this for the SST-2 results in Table 1 to be included in a final version of the paper.\n\n> (4) The text in Figure 3a is quite small; could you make the plot more legible?\n\nThanks for pointing this out, we will increase the corresponding fontsize. \n\n> (5) Could you explore the effect of training data size on the success of the MIAs?\n\nWe share the reviewer’s intuition that as the size of the training dataset increases, the MIA performance likely decreases. As part of the rebuttal process, however, we have prioritized running other experiments and would leave this analysis to future work.", "claims_and_evidence": "I think all the claims in this submission are well-substantiated.  There are extensive ablations throughout the entire manuscript.", "methods_and_evaluation_criteria": "Yes, the metrics reported are standard in the MIA literature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. I checked the main MIA experimental design.", "relation_to_broader_scientific_literature": "Existing literature has found that machine learning models trained on private data pose substantial privacy risks as quantified by MIAs (Shokri et al. 2017, Carlini et al. 2022a, Shi et al. 2023). However, the privacy risks of synthetic data generated by LLMs has not been explored by these works. This submission initiates a thorough investigation into these new risks and show that one can construct canaries which can be identified in the synthetic data by simple MIAs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is very well written.\n2. The framing of the data-based attacks for the privacy risks of synthetic data is important for practice.\n3. The new construction of canaries with in-distribution prefixes and out of distribution suffixes is very original and interesting, and the authors do a good of ablating and thoroughly exploring this design space. \n\nWeaknesses:\n1. The authors do not fully explore the design space of MIAs for synthetic dataset. For example, one could imagine finetuning on $\\tilde{\\mathcal{D}}$ and then applying standard MIAs to the finetuned model to try to extract canaries from $\\mathcal{D}$. This is not a major concern because the MIAs solely based on synthetic data are already performant.", "other_comments_or_suggestions": "It might be worthwhile to include a discussion on the privacy risks of $\\mathcal{D}$ itself when it does not have any canaries and just given access to the synthetic data itself. I imagine that the performance of the MIAs would be much worse, since the canaries are specially designed to be memorized.", "questions_for_authors": "1. Could you report CIs for some of the major results, i.e. Table 1 and Figure 1?  \n2. The text in Figure 3a is quite small; could you the plot more legible?\n3. Could you explore the effect of training data size on the success of the MIAs? My intuition is that if the training data size goes to infinity, then one needs more repetitions of the canary. \n4. Could you report the performance of the data-based MIAs on non-canaries belonging to $\\mathcal{D}$ as a baseline? Are there privacy risks inherent in any finetuning data given synthetic generations from the finetuned model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the privacy risk of synthetic data generation. To this end, this paper proposes novel methods for generating canaries and testing for their leakage. 

Overall, the reviewers agreed that this was an interesting setting with good empirical evaluation. The results show promise and the paper was well-written.

However, there were also a few concerns that reviewers noted:
* Lacking justification for and explanation of the n-gram attack
* Some issue with the experimental design and # of datasets considered
* Possibly other attacks
* Considering models trained with DP.

Some of these were addressed within the rebuttal. Overall, the paper appears to be have improved during the rebuttal and provides a good contribution.
Decision: Accept (poster)