Language Models are Advanced Anonymizers

Authors: Robin Staab, Mark Vero, Mislav Balunovic, Martin Vechev
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: privacy, anonymization, large language models

Abstract:
Recent privacy research on large language models (LLMs) has shown that they achieve near-human-level performance at inferring personal data from online texts. With ever-increasing model capabilities, existing text anonymization methods are currently lacking behind regulatory requirements and adversarial threats. In this work, we take two steps to bridge this gap: First, we present a new setting for evaluating anonymization in the face of adversarial LLM inferences, allowing for a natural measurement of anonymization performance while remedying some of the shortcomings of previous metrics. Then, within this setting, we develop a novel LLM-based adversarial anonymization framework leveraging the strong inferential capabilities of LLMs to inform our anonymization procedure. We conduct a comprehensive experimental evaluation of adversarial anonymization across 13 LLMs on real-world and synthetic online texts, comparing it against multiple baselines and industry-grade anonymizers. Our evaluation shows that adversarial anonymization outperforms current commercial anonymizers both in terms of the resulting utility and privacy. We support our findings with a human study (n=50) highlighting a strong and consistent human preference for LLM-anonymized texts.

Decision: accept (poster)
Reviewer scores: 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730070546912

thread_type:
official_review

<thread_object>
creator: reviewer_18jd

<content>
summary:
This paper proposes a LLM-based adversarial anonymization framework to address privacy risks. The authors use a feedback-guided approach where an LLM adversary attempts to infer personal attributes from a given text, and an anonymizer LLM iteratively modifies the text to reduce inference risks. The paper evaluates this method against traditional anonymization techniques and demonstrates superior performance in both preserving utility and privacy across several datasets.

strengths:
1. The proposed adversarial anonymization framework leverages the strengths of LLMs both as adversaries and anonymizers, showcasing a new application of LLMs in a privacy-preserving context.
2. The inclusion of a human study adds value to the evaluation by confirming the practical applicability of the framework and showing a preference for the LLM-anonymized text.

weaknesses:
While this might be an interesting application of LLMs to the field of anonymization, the core methodology introduces neither fundamentally new anonymization techniques nor a different way to use LLMs. It merely adapts existing concepts by leveraging the powers of LLMs. Thus, the contribution of novelty is limited for either the LLM or the privacy community.

One major limitation is that, in real life, texts to anonymize are normally very long, and due to the current method, which would be prohibitively expensive, it is practically not feasible in applications that demand real-time processing. The scalability and applicability of this framework are rather limited due to the enormous amount of documents it may need to work with iteratively. Another limitation is that privacy performance remains unpredictable due to heavy dependence on the capabilities of the LLM. This creates a dependency where consistency of anonymization outcome cannot be guaranteed and may even differ from model to model or update to update.

questions:
1. Could you provide more examples where the framework fails and explain why the LLM is unable to recognize these instances?
2. Do you think it’s feasible to distill this capability into a smaller model? in this way, we can reduce the computational cost.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730501182828

thread_type:
official_review

<thread_object>
creator: reviewer_epkh

<content>
summary:
In this work, the authors focus on the privacy scenario where online texts can be exploited to infer personal data. The authors utilize adversarial LLM inferences, which are highly performant in extracting personal attributes from unprotected texts, for evaluating anonymization and also use this adversarial model as "feedback provider" to another LLM whose goal is to anonymize texts. An iterative framework between these two LLMs lead to strong anonymization performance as shown by the authors in wide range of experiments, outperforming existing anonymizers and also aligning well with human preference.

strengths:
The presentation is very clear and the paper flows very well. Text anonymization is an important problem in the realm of privacy and the approach the authors introduce do improve the existing anonymization tools significantly. The evaluation section has extensive analysis, which is great. The reviewer really enjoyed reading this paper overall.

weaknesses:
In my opinion, the paper is very well written and the authors conducted extensive empirical studies to demonstrate the significant improvement of their approach compared to the existing text anonymization tools. I think my main concern is the scope and complexity of the approach appear quite limited, especially for a conference of this caliber. The approach is based on iterating two LLMs, one is anonymizing text and the other is trying to infer personal attributes. To me, this is like a cute application of LLMs but perhaps rather better suited for a workshop instead of this conference. In this sense, I am unsure about the fit.

questions:
1. Have you considered measuring utility by some downstream applications? E.g. if the texts are used for some analysis or for some task, how the performance changes from the original unprotected texts to the anonymized texts. Would you think this could also serve for useful utility metrics?

2. How can one turn this approach into a more comprehensive privacy-protecting tool? To my understanding, it currently builds on pre-defined set of attributes and the adversary LLM is trying to infer these attributes while the anonymizer LLM is trying to anonymize as oppose.  But it'd be hard to list all possible attributes that could lead to deducing personal information so any comments on scaling this approach would be appreciated.

3. Also related to my question above, formal privacy guaranteeing mechanisms like differential privacy ensures that even the existence of the data cannot be inferred from the analysis by any adversary. Although in this work the authors focus on anonymizing individual text snippets so that DP may not be applicable, however, it'd be interesting to find a common scenario where two approaches can be compared I think.
 
Minor: AzureLanguageService -> Azure Language Service
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723979166

thread_type:
official_review

<thread_object>
creator: reviewer_cnvx

<content>
summary:
This paper introduces a novel approach to text anonymization in the era of large language models. The authors present two main contributions: (1) a new evaluation framework that leverages LLMs for adversarial inference to measure anonymization effectiveness, and (2) an iterative anonymization pipeline that uses adversarial feedback to guide the text anonymization process. This framework offers improvement over the traditional span based formulation as contextual information leaks information as well. The authors conduct extensive experiments with various models and demonstrate that their approach achieves better privacy-utility tradeoffs compared to traditional span-based anonymization techniques such Azure Language Services. In their results, performing the procedure reduces the adversarial inference chance from 87% to 66%, and iterating the procedure with GPT-4 for three rounds further reduces adversarial inference success to ~45% while maintaining higher text utility than baseline methods. They validate their results with human annotation.

strengths:
- Novel approach that leverages LLMs' inference capabilities to measure privacy leakage in a more realistic way than traditional span-based methods.
- Comprehensive experimental evaluation across multiple models, attributes, and metrics, with clear ablation studies showing the benefit of the feedback-guided approach.
- Strong empirical results showing significant improvements over industry-standard tools like Azure Language Service, with detailed analysis of both privacy protection and utility preservation.
- Thoughtful consideration of practical concerns including computational costs, local deployment options, and regulatory compliance.
- Clear demonstration of how multiple rounds of anonymization can progressively improve privacy while maintaining readable text.

weaknesses:
- The ~41% remaining adversarial inference success rate after anonymization remains concerning for privacy-critical applications. The paper would benefit from deeper analysis of these failure cases.
- Limited domain evaluation, focusing primarily on data directly or indirectly from Reddit. Testing on other domains (medical, legal, etc.) would strengthen generalizability claims. It could also be reddit comments that mentions information from these domains. 
- The method requires pre-defining attributes to protect, which may miss unexpected privacy leaks. An automated approach to identifying sensitive attributes could be valuable.
- Cost analysis could be more comprehensive - while per-comment costs are reasonable (
(~$0.035), real-world applications with high volume could face significant expenses. In addition, the estimate is only for one turn, so to achieve the same level of privacy protection, this number might be more expensive.

questions:
- Have you explored methods to automatically determine the optimal number of iterations, perhaps based on inference confidence?
- How does the system perform when encountering privacy-sensitive attributes not explicitly listed in the input? Could it be extended to automatically identify such attributes?
- Can you provide specific examples of common failure cases where privacy leaks persist even after multiple rounds of anonymization? Understanding these patterns could help improve the approach.
- Have you considered how this approach might need to be adapted for different domains with varying privacy requirements and linguistic patterns?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732277729197

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for their feedback and their thorough evaluation of our work. 
We are pleased that reviewers find that our work introduces a more realistic and practical setting for privacy measurement (CnVX), provides a method that shows noticeably empirical improvements over existing solutions (CnVX, EPkh, 18jd) and has an extensive evaluation (CnVX, EPkh) including a human study (18jd).
We are also glad to hear that reviewers found the paper clear and thoughtful with respect to practical considerations (CnVX) as well as enjoyable to read (EPkh).

Alongside this rebuttal we have uploaded an updated version of our manuscript (with new content in purple), that contains various additional results and explanations to support our rebuttal. We responded to each reviewer separately below and are happy to further engage in the discussion in case of follow-up questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Building on prior work from Staab et al. (2023) that showed that LLMs can infer many personal attributes from online posts without needing explicit identifiers, the authors show that LLMs can still infer these attributes even after the posts have been "anonymized" by SOTA, industry-grade anonymizers. They then propose an iterative adversarial scheme to better anonymize these posts (essentially, prompting an LLM to see if they can still infer any personal attribute; and then editing the post accordingly).

Reviewers thought the paper was well-written, well-executed, and tackling an important problem. The main concern from some reviewers was about fit, given that the paper is largely prompting-based and does not introduce any other methodological or conceptual developments. I am less concerned about this issue because the paper provides an important contribution by showing that SOTA anonymizers still allow for significant inference of personal attributes and that simple iterative methods with an LLM in the loop can improve anonymization. While there are no "novel technical contributions", I think this paper carries a useful message for the community and can serve as a foundation for future efforts to build better privacy safeguards. Thus, I recommend acceptance. I encourage the authors to take the skeptical reviewer feedback into account in the framing of their paper.