The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text

Authors: Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Beguelin, Shruti Tople, Reza Shokri
Primary area: foundation or frontier models, including LLMs
Keywords: Privacy, language models, synthetic data

Abstract:
How much information about training examples can be gleaned from synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to a false sense of privacy. In this paper, we investigate the design of membership inference attacks (MIAs) that target data used to fine-tune pre-trained LLMs that are then used to synthesize data, particularly when the adversary does not have access to the fine-tuned LLM but only to a synthetic data corpus. We demonstrate that using canaries crafted to maximize their vulnerability to attacks that have access to the model are sub-optimal for auditing privacy risks when only synthetic data is released. This is because such out-of-distribution canaries have limited influence on the model’s output when prompted to generate useful, in-distribution synthetic data, thus significantly limiting their vulnerability to MIAs. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries that leave detectable traces in synthetic data. Our approach significantly enhances the power of MIAs, providing a better assessment of the privacy risks of releasing synthetic data generated by LLMs.

Decision: reject
Reviewer scores: 5, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1729815576796

thread_type:
official_review

<thread_object>
creator: reviewer_ozg5

<content>
summary:
This paper studies membership inference attacks (MIAs) that target data used to fine-tune pre-trained LLMs, which are used to generate synthetic data. The authors focus on the case where the adversary does not have access to the fine-tuned model, but only to the generated synthetic data. In this case, prior MIAs are not applicable because even though canaries may be memorized by the target model, they are less likely to be present in the synthetic data generated by in-distribution prompts. The authors address this by constructing canaries starting with an in-distribution prefix and transitioning into an out-of-distribution suffix, where the former increases the likelihood of influencing the synthetic data, and the latter increases the likelihood that the model memorizes them.

strengths:
- The authors propose a new method for generating canaries optimized for this application.
- They run a thorough experimental evaluation to understand how the performance varies with the different parameters.

weaknesses:
- It would be great if the authors could discuss concrete applications of this work or scenarios where it is useful.
- This work only gives a lower bound on the privacy risks, and it is not clear how the results generalize.

questions:
- How would the results change if the adversary is able to choose the prompts for the synthetic data generation?
- How much hyperparameter tuning was done when training the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492194542

thread_type:
official_review

<thread_object>
creator: reviewer_d3yn

<content>
summary:
The paper investigates privacy auditing of synthetically generated text using MIAs. The authors demonstrate that standard canaries for model-based MIAs are ineffective for data-based MIAs. Hence, they design a new canary generation technique that improves data-based MIAs. The main challenge of generating effective canaries is ensuring that they align with the prompt, so they can at least be generated. But at the same time need to be unique enough so that they are memorized by the model. So they create a canary where the prefix contains a truncated original record, and the suffix is generated by a pre-trained LLM.

strengths:
1. The paper tackles an important problem of privacy auditing synthetic text generation from LLMs
2. The ideas in the paper are easy to follow
3. The result showing that increased canary perplexity leads to lower MIA AUC for synthetic data-based attacks is interesting

weaknesses:
1. I find the main results from 5.1 that current canary designs aren’t effective for MIAs on synthetic data a bit underwhelming. I feel like this is already well-known and intuitive that MIAs based on just the generated labels/tokens is much harder than having logits because you loose the uncertainty quantities that are usually exploited in standard MIAs. Unless I am missing something that is the novelty of the results in 5.1.
2. It’s unclear to me how the author’s proposed canary generation method improves over the standard canary generation. For example, how much does the canaries with in-distribution suffix improve over the baseline canaries for MIAs on synthetic data in terms of AUC? This would help with connecting the main results from 5.1 as I only see improvements stated in terms of TPR which feels disconnected from 5.1. 
3. There seems to be a TPR improvement for increasing prefix length on SST-2 in Figure 1, but marginal improvement for AG News. This ambiguity in results makes it inconclusive about the exact effect the prefix length has improving MIAs (large vs small). Unfortunately, we’d need to see more results on at least one more dataset.  
4. I guess generalizng points 1. and 2., I’m uncertain about the exact contribution of the paper. It is hard to find in the introduction and there is no conclusion section to make the contribution more concrete.

questions:
1. Are the membership scores using n-gram probability and similarity metric novel techniques that you are proposing?
2. The authors mention that memorization is more likely when there is an abrupt transition in entropy between sub-sequences. Could you discuss the intuition behind this?
3. Were experiments on low-perplexity prefix lengths of 40 and 50 performed? It would be great to provide those results, too.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707117659

thread_type:
official_review

<thread_object>
creator: reviewer_uo6v

<content>
summary:
This paper investigates membership inference attacks (MIAs) on synthetic text generated by fine-tuned Large Language Models (LLMs). It highlights that traditional MIAs, which rely on access to the model or its logits, are not directly applicable to scenarios where only the synthetic data is released.  The authors find that standard "canary" examples, designed to be highly vulnerable to model-based MIAs, are less effective when auditing the privacy risks of synthetic data because their influence on the generated text diminishes.  To address this, they propose crafting specialized canaries with an in-distribution prefix and an out-of-distribution suffix to improve their impact on synthetic data generation while maintaining memorizability. Their experiments demonstrate that this approach significantly enhances the power of MIAs on synthetic text, enabling a more accurate assessment of the privacy risks associated with releasing LLM-generated data. They also analyze the membership inference signal and find that it relies on subtle shifts in the word probability distribution within the synthetic data, rather than explicit regurgitation of canary subsequences.

strengths:
* Makes progress on an important problem by proposing new methods to perform MIA against synthetic data instead of directly with model access, and developing a sound methodology to design canaries for this task

* Thorough empirical evaluation providing quanlitative and quantitative analysis of the different aspects that affect MIA success

weaknesses:
* Lacks a centralized presentation of the overall auditing strategy. In particular, Algorithm 1 is presented as adding a single canary one time to the dataset, while in practice most of the evaluations add several canaries with multiple repetitions. The authors should considering spelling out the full auditing method with all hyper-parameters and design choices in a single algorithm block.

* The description of the threat model (L128-138) only focuses on the access mode available to the adversary. This needs to be expanded to capture the knowledge of the training pipeline available to the adversary (base dataset, training method, sampling method, etc). This is important because in some cases (e.g. when using shadow models and when designing canaries for synthetic data auditing) the adversary leverages some of this information to perform the attack, while in other methods the attacker uses less information. A table summarizing these differences would make it easier to compare the different approaches evaluated in the paper.

questions:
See weaknesses for points that I would like to see addressed in the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730919472511

thread_type:
official_review

<thread_object>
creator: reviewer_ylle

<content>
summary:
The paper investigates the use of MIAs to audit privacy in synthetically generated text. The authors have designed new canaries that begin with an in-distribution prefix (to maximize the chance of regeneration by the fine-tuned model) and an out-of-distribution suffix to maximize memorization.

strengths:
- The problem is intriguing, particularly as designing MIAs becomes challenging without access to the target model.
- The paper is well-written.

weaknesses:
- In Table 1, the authors use only 30 words for in-distribution canaries, compared to 50 words for synthetic canaries (as mentioned in a footnote). A larger canary size generally results in higher AUC-ROC numbers. Even for SST-2, the in-distribution canaries perform better than synthetic canaries, which diminishes the contribution of the authors regarding the efficiency of their canaries. This pattern is also visible in Tables 6 and 7.
- The AUC-ROC scores for model attacks in Table 1 suggest that the fine-tuned model is overfitted (likely due to a high n_rep=12). Knowing this, results of MIA scores for synthetic data are little bit low. The scores using SIM_jac and SIM_emb are nearly random, with only the 2-gram model scores slightly better. It also raises the question of the practicality of having high n_rep values like 12 or 16 in real-world synthetic data privacy auditing. These results do not strongly support the new MIA contribution. Additionally, Figure 1 (a and b) show near-random curves for n_rep=2 or 4.

questions:
- It would be helpful to see the MIA scores when transitioning from the last words of the prefix to the first words of the suffix in generated canaries to demonstrate their effectiveness.
- It would benefit the paper if the authors could highlight their contributions more effectively based on the results.
- How costly is it to compute the n-gram model when dealing with a large volume of synthetic data generated by the target model? A cost analysis for different n-grams and data sizes would be informative.
- Given a large corpus of synthetic data, would it be feasible to train a transformer-based attack model to predict the next word probabilities as the MIA scores?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732794838959

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised manuscript as part of the rebuttal

comment:
We have revised the manuscript to address comments from the reviewers: 
- We added results for in-distribution canaries to Figure 1(c,f) and Figure 6 (c,f). We find that canaries with an in-distribution prefix and an out-of-distribution suffix consistently lead to better MIA performance (in terms of higher AUC ROC score, but especially in terms of higher TPR at low FPR) than in-distribution or high perplexity OOD canaries normally used in model-based attacks (addressing comments by Reviewers yLLE and d3yN). 
- We have made explicit and clearer our contributions in the introduction (Reviewers yLLE and d3yN).
- In the introduction, we have elaborated on real-world applications and on how our work complements upper bounds on privacy risks when synthetic data is generated using models fine-tuned with differential privacy guarantees (Reviewer oZg5).
- We have modified Algorithm 1 to account for the number of canary repetitions and make it closer to the auditing method we use in experiments. We have also added an explicit parameter standing for the decoding strategy used to generate synthetic data (Reviewer uo6v).
- We have summarized the capabilities of adversaries in the data- and model-based threat models and what capabilities the attacks in our experiments effectively use in Appendix D, complementing the description in lines 128-138 and the formalization in Algorithm 1 (Reviewer uo6v).
- We added in Appendix J a description of hyperparameter tuning for fine-tuning LLMs (Reviewer oZg5).

We have also made numerous minor changes throughout the manuscript, improving the presentation, notational consistency and clarity. We moved Section 5.3 to Appendix K.1 to accommodate for the new content without compromising legibility.

Due to contingency on compute resources, we are still working on getting additional experimental results on an additional dataset (Reviewer d3yN) and leveraging a Transformer model instead of an $n$-gram model in attacks (Reviewer yLLE). We will elaborate on our progress in the following days as part of the discussion and will work diligently to include the results in an eventual camera-ready version of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission studies membership inference attacks (MIAs) that target data that is used to fine-tune pre-trained LLMs which are then used to synthesize data. The submission focuses on the setting where the adversary does not have access to the fine-tuned LLM, but only has access to the synthetic data corpus.The submission shows that using canaries designed to maximize the vulnerability to attacks that have access to the model are sub-optimal when auditing privacy risks in the setting where only the synthetic data is released.
While the submission studies an important problem, and presents an interesting approach, a concern was raised in the reviews about the empirical evaluation being limited. The authors attempted to include the evaluation on an additional dataset during the discussion period, but didn’t manage to do so. We think that adding such an additional evaluation would be needed before the paper is ready for publication.