Language Models are Advanced Anonymizers
Authors: Robin Staab
Mark Vero
Mislav Balunovic
Martin Vechev
Venue: ICLR 2025 Poster (2025)
Keywords: privacy
anonymization
large language models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We demonstrate how large language models can be employed in an adversarial framework to surpass state-of-the-art anonymization tools both in terms of privacy and utility.

Abstract: Recent privacy research on large language models (LLMs) has shown that they achieve near-human-level performance at inferring personal data from online texts. With ever-increasing model capabilities, existing text anonymization methods are currently lacking behind regulatory requirements and adversarial threats. In this work, we take two steps to bridge this gap: First, we present a new setting for evaluating anonymization in the face of adversarial LLM inferences, allowing for a natural measurement of anonymization performance while remedying some of the shortcomings of previous metrics. Then, within this setting, we develop a novel LLM-based adversarial anonymization framework leveraging the strong inferential capabilities of LLMs to inform our anonymization procedure. We conduct a comprehensive experimental evaluation of adversarial anonymization across 13 LLMs on real-world and synthetic online texts, comparing it against multiple baselines and industry-grade anonymizers. Our evaluation shows that adversarial anonymization outperforms current commercial anonymizers both in terms of the resulting utility and privacy. We support our findings with a human study (n=50) highlighting a strong and consistent human preference for LLM-anonymized texts.

Published as a conference paper at ICLR 2025
LANGUAGE MODELS ARE ADVANCED ANONYMIZERS
Robin Staab, Mark Vero, Mislav Balunović, Martin Vechev Department of Computer Science, ETH Zurich (robin. staab, mark. verovvvero@gif.ette etw...cheh.cch
ABSTRACT
Recent privacy research on large language models (LLMs) has shown that they achieve near-human-leve performance at inferring personal data from online texts. With ever- ver-inereasing model capabilities, existing text anonymization methods are currently lacking behind regulatory requirements and adversarial threats. In this work, take two steps bridge this gap: First, we present a new setting for evaluating anonymization in the face of adversarial LLM inferences, allowing for a natural measurement of anonymization performance while remedying some of the shortcomings of previous metrics. Then, within this setting, we develop a novel LL.M-base adversarial anonymization framework leveraging the strong inferential capabilities of LLMs to inform our anonymization procedure. We conduct a comprehensive experimental evaluation of adversarial anonymization across 13 LLMs on real-world and synthetic online texts, comparing it against multiple baselines and industry-grade anonymizers. Our evaluation shows that adversarial anonymization outperforms current commercial anonymizers both in terms of the resulting utility and privacy. We support our findings with a human study (n=50) highlighting strong and consistent human preference for M-M-annnymieedd texts.
INTRODUCTION
The rapidly increasing capabilities and widespread adoption of large language models (LLMs) (OpenAI, 2023; Touvron et al., 2023b) have sparked various discussions about their potential impacts on society. An area of particular interest has been the privacy implications of LLMs, where initial investigations focused primarily on the memorization of model training data (Carlini et al., 2021; 2023). Recently, it has been shown that modern LLMs can be misused for high-accuracy predictions of personal attributes from seemingly non-revealing online posts (Staab et al., 2023). Moreover, where previously one would have to hire humans to infer private attributes, these models are fast and cheap enough to automate such inferences scale. Concurrently, as issues, they continue to share texts with information easily inferrable by LLMs. Rather concerningly, prior work has also shown that current industry-standarr text anonymizers are largely insufficient at protecting users from such attacks, as they are unable to pick up on revealing yet complex and context- deeexxt-depeneet cues. Consider the following shortened example from Staab et (2023):
unaware of such
This left shark thing is hilarious, I've seen it, walked just by there to see it live after my final exams.
While the comment contains no direct references any locations, the context allows both an informed reader and an LLM to infer that the author lived in Glendale, where the 2015 Super Bowl halftime show featured performance colloquially called "left shark incident." It would thus desirable the user could be warned before posting the comment that it might reveal their location and, ideally, be offered a less revealing alternative. Yet, anonymizing this text with existing tools proves futile, as no location was directly mentioned, fooling traditional seittyyrcconntton-boe is particularly alarming as there have already been reports of LLM inferences being commercially applied online to categorize users based on their posts (Brewster, 2023).
anonymizers. This
The Need for Privacy At the same time, there has been a large regulatory push for stronger individual privacy protection. Notably, regulations such the BU's GDPR (EU, 2016) and Califor- nia's s CCPA (Cal, 2018) have established extensive legislative requirements to protect individuals' private data, such as location, age, and sex. Complementing regulatory efforts, online users, e.g., pseudonymized sites like Reddit, are also increasingly aware of the need to protect their privacy and demand tighter governance over their personal data.

---

Published as a conference paper at ICLR 2025
Anonymization
Adversarial Inference
"Oh that is felanines, still can't bekied see it
pensell diaddty jos
Location to detamiine
live ale ma feol coares at like unhesis groat to live legg after steelingg
volkad the
Lacater Cardale catt Sentis
Clocdaae
shark thing
20th Seceriaa
parsor Greddale just earts of the corvaec se wa led there oferr Proveb: Arazna, to wie a bugh afse the euayiing
Reaes
Hound
"The kift skak tinn
Adversarial Anonymization
exseace of on
Location
up lookeed there
Gloviale
Neural Ensity Recogaition
alto my seseeee, after
Arizna
User-Written Texts
Checkaum
Prosio
Aese Langaage
Contact Words
Traditional NLP Anonymization
Figure 1: Feedback-guidee adversarial anonymization and the adversarial inference setting, Given user-written texts, we depict 1. At the bottom: classical NLP-based anonymization with industry tools such as Presidio and Azure Language Studio. Making use of entity recognition, these tools produce a set of spans that afterward get masked completely, resulting in an anonymized text depicted the bottom right. 2. At the top, we show our feedback-guided adversarial anonymization procedure. In each round, an adversarial L.M tries to predict personal attributes from the current instance of the text. Based on this inference, an anonymizer LLM then removes and adapts relevant sections of the text in order to prevent such inferences. After multiple rounds, the anonymizer outputs the resulting text. On the right we depict our anonymization under adversarial inference setting. Unlike previous metrics, we evaluate the anonymized texts directly against a strong adversarial LLM, which tries to infer personal attributes. We can observe that text produced by adversarial anonymization has both higher utility and privacy than the one obtained through traditional methods.
Insufficiency of Current Tools Industry-standard tools such as Presidio (Mic, 2021) or Microsoft Azure Language Services (Aahill, 2023) that mainly remove structured information, such as SSIDs or mail addresses, are especially problematic in this setting as they (1) fall short of meeting the regulatory requirements of text anonymization (Pilán et al., 2022) and (2) indiscriminately mask any information matching specific patterns, often resulting in hardly readable texts an issue especially grieving for the short, informal, and fast-flowing nature of online comments.
As such, despite expanding regulatory protection, the gap between increasingly stronger adversaries and lacking anonymization methods leaves users in a particularly vulnerable position.
This Work We aim to bridge the gap between the increasing threat to user privacy in online texts and the lack of viable anonymization techniques against L.-M-based inferences. In particular, we propose a new framework, shown in Fig. 1, that anonymizes texts using feedback-guided adversarial approach. The key insight of this framework is to leverage the strong attribute inference capabilities of to inform a separate anonymizer language model. Over multiple rounds, we first employ an LLM adversary for a detailed private attribute inference from the given text. Then, an anonymizing LLM attempts remove, obfuscate, or generalize cues used in the inference by adapting relevant parts of the text. In our evaluation on both real-world and synthetic online texts, we find that performing multiple iterations of this adversarial feedback loop leads to increasingly anonymized text while consistently outperforming traditional techniques in resulting text quality and privacy. Notably, even smaller locally deployed LLMs outperform commercial andustry-grade anonymizers.
Main contributions Our main contributions are:
A novel characterization and formulation of the problem of anonymization under adversarial
LLM attribute inference, overcoming the pitfalls of existing anonymization evaluation.
A novel adversarial anonymization framework leveraging LLMs inferential capabilities.
Extensive experimental evaluation 13 LLMs and several baselines, including a human study, showing on both real-world and synthetic online texts that our adversarial framework outperforms ofte-oofthheart commercial anonymizers both in terms of utility and privacy.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND AND RELATED WORK
Personal Data and PII With the rise of personal information processing in the last decade, key legislature has been established in order to protect individuals' privacy. Many of these regulations, including the EU's General Data Protection Regulation (GDPR) (EU, 2016), center around definitions of what data is considered personal and hence is to be protected. GDPR approaches this in a comprehensive manner, defining in Article that "any information relating to an identified or identifiable natural person" is considered personal data. US-centric regulations such as California's CCPA (Cal, 2018) commonly build on a more restricted definition of personal identifiable information PII defined by the Department of Labour as "information that permits the identity of an individual [..]] to be reasonably inferred by either direct or indirect means" (DOL, 2023). Nonetheless, both regulations include concrete mentions of quasi-identifying personal attributes such as location information, gender, and socioeconomic status that can be combined with other information to identify person. As pointed out by Pilán et al. (2022), the explicit inclusion of such quasi-identifiers makes regulations significantly tighter than what is commonly reflected in existing tools and benchmarks that only focus on directly identifying information (c.g. names, emails). Notably, as shown by Staab et al. (2023) current LLMs achieve almost human-leve performance at extracting and inferring quasi- identifiers, (adversarially) aligning them much closer with regulations than existing anonymizers.
Author Profiling Author profiling is well-established area of NLP research that aims to identify key author attributes (most often gender and age) from the author's written texts alone (Rosso et al., 2016). Such inferences are commonly enabled by combination of attribute-specific feature extraction methods on top of a classical ML architecture such as an SVM classifier. contrast to pre-trained LLMs, these methods require access to task- and domain-specine labeled dataset. This makes it challenging for a malicious actor to directly apply these techniques, as obtaining such labeled data on large scale is both time- and cost-intensive and, in many cases, ethically problematic. Motivated by the recent stark increase in the reasoning capabilities of LLMs, Staab et al. (2023) showed that they can be used to extract and infer personal author attributes on real- wollw texts at almost human-level accuracy without any domain-adaptation being necessary. Notably, such LLM-based inferences incur a much lower cost and time investment than human profilers and escape the prohibitive data requirement of classical methods, enabling them to be potentially misused at an unprecedented scale. Since then, several reports have shown that LLM- author profiling is already being applied in practice, including the automated classification of users (Brewster, 2023).
Data Anonymization The problem of anonymizing structured data has received significant attention for several decades. Arguably, the most well- criterion is k-anonymity (Sweeney, 2002), which states that each user record in a table must be indistinguishable from at least & 1 other data points. The equivalences are thereby commonly enforced by masking, suppressing, or perturbing specific attributes of records. As pointed out by follow-up work (Machanavajhalla et al., 2006), a key issue is that individual sensitive attributes of the user can still be inferred with high certainty whenever the distribution of an attribute in an equivalence class is skewed (e.g., all users being male).
While such shortcomings can be addressed in structured data (Machanavajjhala al., 2006; Li et al., 2007), the anonymization of text is a notably harder problem as it deals with highly non-uniform and unstructured data (Lison et al., 2021). The most commonly applied approaches in text anonymization rely on removing clearly identifying information such as SSIDs, email addresses, and names or specific location identifiers (Pilán et al., 2022). While industry-grade anonymizers such as Microsoft Azure Language Service offer a wide array of potential attributes that can be removed, they fall short of removing finer contextual details (Aahill, 2023: Staab et al., 2023). Therefore, such approaches are sometimes referred to as de- de-identification instead of anonymization in the literature (Pilán et al., 2022; Dernoncourt et al., 2016; Hintze, 2017). Some works take an foormation-thhoreti approach to anonymization (Sánchez& & Batet, 2016). However, they commonly rely on fixed word ontologics and a large number of online search results to estimate the mutual information between individual terms, making their practical application challenging and costly (Lison et al., 2021). We will elaborate more on the current state and limitations of text anonymization in Sec. 3.
As we will show in our results in Sec. 5, confirming Pilán et al. (2022), prior techniques are often unable to anonymize free-form online texts in the face of LLM- inferences. This is mainly due to their inability to remove privacy-leaking cues that are not contained in regular, isolatable elements
3

---

Published as a conference paper at ICLR 2025
in the text (PII or self-disclosures) but require an advanced understanding of the context, reasoning, and vast lexical knowledge on par with the inferring adversary.
Various works also concentrate on purely stylometric attribute inferences. For this, Shetty al. (2018) proposes a supervised GAN setup on which individual writing styles are learned and adapted by LSTM networks. However, stylometric inferences in practice can be brittle (Schuster et al., 2020), require specific training data, and primarily focus on features besides the text content, we consider a comparison "style-change anonymization" outside of the scope of this work.
Dou et al. (2023) proposed an LLM-based method for detecting voluntary self-disclosure of personal information (e-g.. "I am 25 years old and male") in online texts and finetuned model to replace self-disclosing word spans, As we show in our evaluation in Sec. 5, this works well for its intended use case but falls short when anonymizing attributes not directly stated in the original text.
3 ANONYMIZATION UNDER LLM INFERENCE
In this section, we first highlight shortcomings of current text anonymization methods and datasets. We then argue for a more intuitive evaluation framework w.r.t. an L.LM-based inference adversary.
The Privacy-Utility Tradeoff Intuitively, the goal of any text anonymization method is to remove all information that could lead to the inference of a set of pre-specified author attributes while maintaining the utility of the original text. Independently of the anonymization method, we trade a part of the text's original meaning and expression (utility) against a higher degree of privacy protection (Pilán et al., 2022). Naturally, in edge cases such as, a user wanting to publish their e-mail address online so that anyone can message them, there is no way of anonymizing the text without losing its utility. However, as we show in our results in Sec. 5, in most cases, L.sed anonymization allows us to achieve significantly better utility-privacy tradeoff than offered by current state-of-the-ar tools. This is due to two main reasons: Firstly, as pointed out by Lison et al. (2021) current anonymizers do not account for the utility of the remaining text, commonly masking all identified spans of text with noticeably less readable and informative tags or 'e' symbols. While this is sometimes mitigated with specific generalization hierarchies (Sánchez & Batet, 2016), such approaches are limited in practice. Secondly, despite claiming context-awarenes: their classical rule- and neural nudity-recogiition basdd approaches cannot account for actual inferences and reasoning, commonly neglecting any quasi-identifiers (Lison et al., 2021). As LLMs can reason over, and produce free-form text they are able to overcome the above limitations.
Anonymization Datasets Similarly to author profiling (Rosso et al., 2016), a key issue when evaluating data anonymization is lack of adequate datasets. This is particularly exacerbated by the fact that almost all traditional datasets only focus on de-identification (i.e., removing SSIDs and names). In particular, the only anonymization dataset is the Text Anonymization Benchmark (TAB) (Pilán et al., 2022) consisting of manually annotated human-rights court documents, not representative of typical online texts. Further, for PII detection, Bubeck al. (2023) have already shown that GPT-4 outperforms classical methods on TAB. Recently, Staab et al. (2023) introduced the eddso dataset consisting of real-world online comments alongside human- human-labeled labeled personal attribute inferences. This is accompanied by set of 525 synthetic conversations with both ground- truth and human-annotated labels. Since then, Yukhymenko al. (2024) extended over this dataset, introducing SynthPAI, consisting of fully synthetic Reddit-style threads and personal attribute labeled comments. However, as the corresponding labels are for inferences on the full text only, these datasets not suitable for traditional metrics that measure accuracy and precision against set of ground truth spans (Pilán et al., 2022). As we argue in the next paragraph, this is not a limitation of these datasets, but rather of current anonymization metries. While they are reasonable for PII detection in texts where these appear explicitly, they are not suitable for quantifying the privacy protection provided by actual free-form text anonymization.
Anonymization under LLM Inference Based on the insight by Staab et al. (2023) that LLMs achieve near human-level accuracy in attribute inference, we propose to evaluate the success of anonymization methods based on whether a strong adversarial LLM can still infer corresponding attributes from the resulting texts. This is both more aligned with regulatory requirements ("[...] be
4

---

Published as a conference paper at ICLR 2025
reasonably inferred") and additionally provides sharper and more interpretable notion of anonymiza- tion. In particular, it avoids the shortcomings of existing span-based metrics, as shown in the following example. Assume that we want to anonymize the attribute age from the following comment:
was when the first iPhone was released. Now I am 25, and look at what Apple has achieved!
To ensure that the authors cannot be inferred anymore, have anonymize both the sections "I was [..]]" and "Now am 25". Assuming an anonymizer detects only one of the two spans, their traditional token recall (Lison et al., 2021) would be R 3. despite the age being easily inferable from the remaining text, resulting in adversarial inference score of O. In this sense, evaluating anonymization directly under adversarial inference reflects the privacy as a worst-case metric principle much better than prior span- based metrics, only assigning a score of 1 when an attribute can no longer be inferred. This naturally extends the wwwaammn-ssaeettieeeeeeee entity-recall (Pilán et al., 2022) without requiring expensive span-entity labels from human annotators.
4 FEEDBACK- GUIDED ADVERSARIAL ANONYMIZATION
We now present our adversarial anonymization method that leverages the strong inferential capabilities of LLMs to anonymize text with high utility and privacy protection.
Adversarial Anonymization As depicted in Fig. 1 we propose the usage of an adversar- ial proxy to inform our anonymizer during the anonymization procedure about which sec- tions of text snippet are relevant for further anonymization. Our ial anonymization proceeds as follows (depicted in Fig. 2): Given text to = and a set of attributes A we wish to anonymize, our anonymizer first instantiates two models: the inference model Min and the anonymization model Manon- In each round i of the anonymization, the infer- ence model tries to infer all attributes a A based on the anonymized text from the cur- rent round 4, resulting in a set of inferences A = {a= May(t:)] a e A). A; is then input to the anonymization model that tries to adapt the relevant sections of the text resulting in new 4++1 = Manon (4,, Az). We repeat this pro- cedure either until A, 2, i.c., the inference model can no longer infer any of the original target attributes A, or fixed number of rounds is reached. We note that in round i we give both models access only to & instead of the entire history tzo. For the adversary, this naturally cap- tures the notion of only attacking the current anonymized text, leading to inferences that for Manon are most helpful on t,.
just got fitted at intimissimi and it was great wa ked wearing 358 and left with perfectly fitting 3248. the fitter knowled- gable helphul brought options to try. was pricy but worth
to
The comment mentions gesting fitted at mimimi and talks about bra sizes, specifi- cally going from 33b 32ds. Bra fittings typically female experience as an undergamant commoniyy worn by women.
adversar-
Ao May
just got litted at store and was great. walked in wearing parlicaly litting
and left with B geable helpful brought me options to try. it was pricey but worth
knowledd-
M anon (fa, As)
The comment mentions getting fitted stare leaving with perfectly fitting items."
the torm "fitted" commonly assoriated with firtings, which would suggest the author is female. The language used neutral and does not prowide any gender-span specific cuos outside of the of the fitting experience.
Ag = Mar(ts)
had personal shapping store was great. new clothes left with several perfectly suitable items. The assistant was knowleds- eable and helpful and brought me lots of options to try. It was pricey but worth it.
looking
fg M Manon(t (1s, A))
Figure 2: Intermediate steps of our adversarial anonymization framework on a (perturbed) real- world example. Adversarial inferences (GPT-4) are shortened for brevity. We observe how the first round of anonymization detects direct references to the author's 's sex, leading to their removal in tj. In round 2, the adversary relied on more subtle usages of language for its inference. On f2, the adversary was unable to infer any sex with certainty.
In our evaluation, we instantiate both models with various SotA and open LLMs, highlight- ing how this adversarial framework improves over existing baselines. Notably, compared to direct anonymization approach, the adversarial feedback allows the model to anonymize significantly finer details, leading to an attribute inference accuracy drop from 66.3% to 45.3% after just three iterations. Additionally, the intermediate infer- ences are directly interpretable and could be used in an interactive scenario to inform a human in the loop about whether the current level of anonymization is sufficient for the purpose at hand.
5

---

Published as a conference paper at ICLR 2025
10348-4 GPF--MM
Liara
2war2-20
140
Mietrs Ro223-A
Limaat LEE--A
Asir- Leovll
Nee untl (Cwmb))
100
0.30
0.85
60
0.80
40
20
0.6
PCBP
Adversania Accuracy
Abribut
(a) -uility-privacy tradeoff
(b) Per-attribute adversarial accuracy
Figure 3: The main experiments comparing performance of our approach with the baselines. Fig. 3a shows how adversarial methods (having suffix -AA and shown in 5 iterations) improve utility and adversarial accuracy on the PersonalReddit dataset compared to classical methods. The baseline on non-anonymized text is shown in the top-right corner as * For each OSS model family, we only show the strongest model. Fig. 3b shows the number of correct predictions for four exemplary attributes on PersonalReddit. We observe, across all attributes, how adversarial anonymizers outperform both baselines. While 1lama3.1-705 and GPT-4 continuously improve with each round, the more limited Yi-34B struggles to improve two of the four attributes.
5 EVALUATION
In this section, we present our experimental evaluation. In our main experiment, we show how our adversarial LLM anonymizers outperform existing industry-gradt grade anonymizers in the privacy-utility tradeoff on the real-world PersonalReddit dataset. We then confirm these results with a human study showing consistent human preferences for L.M--nnonymzzed texts. We further show how multiple iterations of our anonymization procedure enable a fine-grained tradeoff between privacy and text urility. Next, explore anonymization over varying levels of ground-truth resolution, highlighting that adversarial anonymization outperforms Azure Language Service (Azure) (Aahill, 2023) especially on fine- fine-grained personal information where identificability is the casiest and privacy matters most. Lastly, we show that even in cases where we cannot anonymize with either technique, LLM-based anonymization leads to noticeably less certain predictions by the adversary. In App. A.4, we repeat our experiments on both SynthPAI and synthetic PersonalReddit samples, observing similar behavior.
Baselines Following Staab et al. (2023), we use the industry-standant state-of-the-art text anonymizer provided by Azure (Aahill, 2023) as a baseline for traditional anonymization. We note that Azure constitutes superset over the openly available Presidio, offering more fine-grained control. We give a detailed overview of all entity types we explicitly remove in App. B.
Further, include the self-disclosure detection and replacement models introduced by Dou al. (2023) as a bascline for how many text snippets contain directly inferable personal attributes as identified by specifically fine-tuned model. Additionally, mansformer-based paraphrasing model Dipper (Krishna et al., 2024) to baseline the anonymizing effect of pure text paraphrasing. To measure the impact of the feedback-guided anonymization, we also instantiate baseline LLM anonymizers that do not receive adversarial inference feedback between anonymization rounds. We note that these baselines otherwise use the same prompts, including CoT reT-reasn,nng, We give more details on used models in App. B showing all used prompts in App. E.
make use of the fate--fftthe-Ar 11B
Measuring Privacy and Utility We instantiate our anonymization under I.LM-inference setting using GPT-4 (the strongest model in Staab et al. (2023)) as the final inference model for all results presented in the main body. We confirm the representativeness of our results obtained this way by examining the impact of the final inference model in App. A.5. For inference, we use the adversarial prompts introduced in Staab et al. (2023), letting GPT-4 infer personal attributes from a complete user profile in zero-shot CoT fashion. We use the same scoring procedure as in Staab al. (2023), and present all used (system) prompts in App. E. As we now work on full texts instead of pre-defined spans, we, motivated by a large number of recent works (Pilán al., 2022; Zheng et al., 2023; Chiang

---

Published as a conference paper at ICLR 2025
& Lee, 2023), in first step, instantiate GPT-4 utility judge that measures both the readability of the anonymized text as well as its similarity in meaning to the original text on a scale from to 10 (for later plots scaled to [o, 1]). As we show in the example in Fig. 1 as well as in App. D, readability is crucial for real-world online texts where traditional anonymization methods would appear unnatural. Further, we compute traditional BLEU (Papineni et al., 2002) and ROUGE-1 (Lin, 2004) scores between the original and the anonymized texts. Our final utility score is comprised by the average of these three metrics. We additionally compare our LLM-based metries to human perception in App. F.5, showing strong overall alignment.
Datasets We first evaluate our methods on the real-world PersonalReddit dataset containing human- labeled Reddit comments (grouped by profiles) across eight possible attributes: Age (AGE), Sex (SEX), Location (LOC), Occupation (OCCP), Education-Level (EDU), Relationship Status (REL), Income Level (INC), and Place-Of Of-Birth (POBP). Notably, PersonalReddit contains comments reflecting real-world online language usage and a considerably wider range of personal attributes than any other author profiling dataset. We preprocess PersonalReddit to only contain profiles with at most 1000 tokens (s 700 words), resulting in 426 profiles. We provide more detailed overview of the data in App. C.
Further, in App. A.4, we evaluate selected methods on the synthetic Reddit comments of SynthPAI (Yukhymenko et al., 2024), and on the 525 synthetic online conversation examples introduced by Staab et al. (2023). Both of these synthetic datasets represent a similar setting to the real-world PersonalReddit dataset, with analogous 8 personal attributes for inference. As show in our evaluation, we observe qualitatively very similar results across both real and synthetic datasets.
Main Experiment We present our main experiment in Fig. 3a, showing both mean utility (as mean of readabil- ity, meaning and ROUGE) and mean privacy (as measured by the number of correct inferences) over the Person- alReddit dataset. We provide individ- ual plots for each utility measurement in App. A.3. We indicate the original texts at the top right as having full util- ity. We first note that Azure achieves an overall utility of 0.76 while only decreasing the adversarial accuracy from 86.7% to 66.3%. Similarly, we observe that the self-disclosure baseline from Dou et al. (2023) achieves a higher utility score it replaces individual spans with readable text; however, at the same time, these replacements leak more information than simple masking. Lastly Dipper as a pure paraphrasing model is largely ineffective in this setting, providing only minor improvements in privacy while heavily impacting utility.
Anonymizer
Azure GPT-4
Iter.
Iter:2
Iter.3
AA
Readability
0.38
0.89
0.86
0.90
0.90
Meaning
0.47
0.73
0.79
0.73
0.67
Typical
0.44
0.99
1.00
0.97
1.00
Usable
0.37
0.93
1.00
0.91
0.88
Table 1: Mean-over-median of human scores per sample across 100 samples. Adversarial anonymization consistently significantly outperforms Azure in readability and meaning across all iterations. Further, humans consistently find LLM- anonymized text more natural and would use it more likely.
We note that (with the exception of including both GPT-4 and GPT-3.5) we only show the largest member of each evaluated model family, including full plots in App. A. We find that all instantiated adversarial anonymizers (denoted with the suffix -AA) consistently outperform Azure in both utility and privacy across all rounds. Across all models, one can further observe that multiple rounds smoothly trade-off mean utility for an increase in privacy. Notably, even after five rounds, GPT-4-AA still has higher mean utility than Azure while dropping adversarial accuracy down to 41.6% (an absolute improvement of 24. 24.7%). Additionally, after GPT-4, the open-source Llama3.1-70B performs the second best, closely matching GPT4-AA's privacy-utility tradeoff.
An interesting observation is that GPT-4, while consistently outperforming GPT-3.5, has a lower utility after one round than Yi-34B (we can make similar observations with various other models). This effect can be attributed to GPT-4 noticing more clues in its first iteration than Yi-34B, which leads to overall more changes in the text and correspondingly lower utility scores.
In Fig. 3b, we compare anonymization results on four of the eight attributes of PersonalReddit (we present results for all attributes in App. A. We only show 4 adversarial anonymizers for improved readability, providing results across all models in App. A. Across all attributes, we can observe that adversarial anonymizers outperform Azure, with only YI-34B performing similarly on single
7

---

Published as a conference paper at ICLR 2025
GPT-4-Bae
GPT-4-AA
Metric
accuncy
corturdyy
0.86
0.85
0.84
0.3
0.82
01
0.450
0.475
0.500
0.550
6PT4.AA
GFT4
4A4
Adtremarta
Netted
Figure 4: Comparison of GPT-4 with feedback- guided adversarial anonymization (AA) vs with- out (denoted by Base). Even after 5 rounds Base does not reach the anonymization performance of AA after rounds. After all iterations, this yields adversarial accuracy delta of ~ 10%.
Figure Adversarial accuracy and certainty on correctly classified examples by the final GPT-4 adversary. GPT-4-AA not only leads to fewer correct inferences but also reduces the certainty the adversary in its correct predictions, forcing it to rely inherent biases.
attribute. Further, observe consistent trend of GPT-4-A,, GPT3.5-AA. and Llama3.1 1-70B-AA, noticeably reducing the number of inferable attributes in each round, highlighting the effectiveness of feedback-guided approach (confirmed in App. A across models).
Scaling over model sizes One key observation made by Staab et al. (2023) was that model at- tribute inference capabilities naturally scale with general model capabilities, resulting in increas- ingly more serious threats to user privacy. Per- haps unsurprisingly, as adversarial anonymiza- tion directly builds on this inference, but equally importantly, we can observe analogous trends in anonymization performance. As such, enabled by our adversarial anonymization framework, the increased privacy threat posed by more capa- ble models can be proportionally mitigated by their increased anonymization capabilities. We highlight this for the Qwen (Bai et al., 2023) model family across 5 different model sizes from 4B up to 72B in Fig. 6. We observe how each model consistently dominates its smaller siblings in the utility-privacy tradcoff. Lastly, GPT-4, as the most capable (tested) model, constitutes the current privacy-utility frontier. Additionally, we observe the same trends across other model families like Llama3. 1/Mixtral in App. A.
Quee11 3-343.--344-M Owwnd 728-A
Aare
Owers
Ower1:
Hown Unily [tomb.)
0.85
0.80
0.75
0.5
0.6
0.7
0.8
Advennarial Recuratyy
Figure 6: Utility-privacy tradeoff across various model sizes of the Qwen model family. Each model improves anonymization performance, con- sistently dominating smaller models.
Human Evaluation We support our results with a human study with 50 participants. For this, we randomly selected 200 anonymized samples (100 anonymized via Azure, 100 evenly split across the first three GPT-4-AA iterations) from our synthetic texts, collecting 5 human scores per sample. In first step, participants were asked to score the readability (from I: unreadable to 5: fully readable) of the anonymized text and whether they would consider the text to be typical (0: no, yes) in an online setting. Afterward, participants were shown the original non-anonymized texts and scored whether the anonymized text preserved the original text's meaning (from completely different to fully preserved) and whether they consider the anonymization usable (0: no, 1: yes) replacement when wanting to participate in an online conversation.
We show our main results in Table 1, reporting normalized mean-over-median scores across all samples. We find that adversarial anonymization sigmificantly outperforms Azure in both readability (0.38 0.89) and meaning (0.47 0.73). Further, humans consistently find LLM-anonymized text more typical (0.99) as well as usable for online text anonymization (0.37 + 0.93). We can further observe consistent behavior across all iterations, with adversarial anonymization notably
8

---

Published as a conference paper at ICLR 2025
outperforming Azure's utility even after 3 turns. These results solidify our previous conclusions, highlighting that adversarial LLM-based anonymization provides a higher utility (and privacy) alternative to traditional text anonymization. We present complete overview of our human study, including more detailed results, the full study format, examples, statistics on inter-rater agreement as well as comparisons LLM utility scores in App. F.
Adversarial Feedback is Crucial We examine the impact of adversarial feedback in our anonymiza- tion framework in Fig. 4. In particular, we compare GPT-4-AA with GPT-4-Base, an anonymizer that is iteratively prompted in the same CoT manner as GPT-4-AA but without adversarial feedback. We observe that already after rounds, GPT-4-AA achieves higher privacy and utility than GPT-4-Base after its full 5 rounds. Further, GPT-4-AA makes considerably larger improvements each round, result we also observe across other models (presented in App. A.2). These results strongly indicate that adversarial feedback helps the anonymizing model focus on the privacy-relevant parts of the text.
Resolution of Anonymization Looking at the results on anonymizing location (LOC) in Fig. 3b, it may seem surprising that even after 3 itera- tions of GPT-4-AA around 45% of the labels can still be correctly inferred. We investigate this further by splitting the ground truth labels into the resolution levels city, state, and country, presenting the individual accuracies for Azure and GPT-4-AA in Table 2. We observe that all methods achieve noticeably lower privacy for country level information than for state or city level. This is expected, as references to city and state are commonly more specific, making them easier to remove. Notably we find that, especially on the finer resolution levels, GPT-4- AA much more noticeably outperforms Azure, reducing the accuracy on "city" level location predictions from Azure's 's 24.6% to 6.8% in a single round and to 1.3% after 3 rounds. This indicates that in situations where anonymity sets are smaller (such as cities compared to countries) anonymizing with LLMs outperforms traditional approaches by even larger margins.
Setting
Country
State
City
Azure
59.2
14.3
24.6
GPT-4 Iter.
61.8
9.5
6.8
GPT-4 Iter. 2
46.1
0
4.1
GPT-4 Iter. 46.1
1.3
Table 2: Adv. accuracy [%] on the location at- tribute for Azure and adv. anonymization (GPT-4). As expected, both approaches have more difficul- ties anonymizing less precise locations (countries). However, with increasing precision GPT-4 signifi- cantly outperforms Azure, completely anonymiz- ing all mentions of states and dropping Azure's 24.6% accuracy for cities down to 1.3%.
Bias and Adversarial Inference One issue that potentially arises when using an adversarial LLM to infer attributes from werl-world text is that (similarly to humans) it will partially rely on pre-existing biases to make an inference. In some cases, we observed that, c.g., general topic choices, such as person writing about what shade of lipstick they like wear, have been properly anonymized (in this case removing all mentions of the author's sex as female), yet the model's biased inference from the general topic was still correct. While statistically, it is arguably the more likely choice, is a-priori unclear whether we should consider the attribute anonymized or not. To investigate this further, we additionally let the adversary score itself based on whether could infer the attribute from direct evidence in the text (certainty 1) or whether it relied purely on statistical bias (certainty
0). We show these results for Azure and GPT-4 in Fig. 5, highlighting how GPT-4-AA consistently produces anonymized texts on which the adversary is less certain in its prediction, inherently having to rely on observed biases. We believe that further investigating the relationship between bias and anonymization could be an interesting direction for future work.
6 DISCUSSION
Use-Cases for Adversarial Anonymization We presented and evaluated our anonymization on real-world online texts, following the setting in Staab et al. (2023). We believe this domain suits adversarial anonymizers particularly well as it requires both readable anonymized text and naturally contains a wide range of personal attributes protected by current regulations. However, there exist text domains such as legal documents or patient records that fit traditional approaches ensure that only masking operations have taken place and where readability is not a key priority. As we will discuss below, we still believe the adversarial inference setting can also benefit these domains.

---

Published as a conference paper at ICLR 2025
Domain Adaptability A key issue in existing anonymization and de-identification research is the domain- -ddatabbilit of methods (Lison et al., 2021), with much of the existing work focusing on specific domains such as the medical sector or legal court cases. This, in turn, leads to domain-specine solutions, such as respective knowledge bases (Sánchez. & Batet, 2016), fine-tuned classifiers (Pilán et al., 2022), and generalization ontologies (Anandan et al., 2012), which are difficult adapt to both new attributes and text domains. As we can see in our evaluation, such domain adaptability is a natural strong suit of LLMs that, due to their vast pre-training (Touvron et al., 2023b), generalize well between tasks (Bubeck et al., 2023; OpenAI, 2023). We believe that adapting LLMs and evaluating their capabilities to anonymize a wide variety of attributes (e.g., mental health status, sexual preferences) on dopain-specific texts will be interesting avenue for future research.
Local Models While we have far treated the adversary an outside instance that only interacts with the fully anonymized texts, a key concern in practice is also what data is shared with respective model providers (e.g., during anonymization). To remedy this, all our evaluations include wide range of open-source models that (1) can be run locally and (2), in the case of current models like Llama3. I, almost match GPT-4-AA's performance, while often being significantly cheaper to run in commodity settings (App. A.6). Notably, all local models outperform existing traditional anonymizers such as Azure. At the same time, as shown in our experiments, adversarial anonymization naturally bounded by the strength of the feedback adversary. Given the recent progress in making smaller and more capable models able to run locally (Jiang et al., 2023; 2024), as well as our current results, we are confident that future open LLMs will exhibit capabilities similar to current closed models. Furthermore, the split into an adversarial and fixing LLM in our adversarial setup provides an interesting opportunity for future smaller models to be fine-tuned specifically for each subtask.
Limits of the Adversarial Inference Setting While adversarial inference provides natural definition of measuring the anonymity of a resulting text, in practice, every instantiation is dependent on the strength of the adversary. Based on the results of Staab et al. (2023) and our experiments, we are confident that GPT-4 already now achieves cosse-to-humnn level performance in this task. At the same time, and similarly to previous metrics for anonymization, this does not guarantee that the text is "truly" anonymized against more capable adversary, e.g., in settings where additional back knowledge is available (Sánchez & Batet, 2016). Providing any such guarantees against arbitrary adversaries has proven to be difficult even in significantly more restricted domains than free-form text in the past (Sweeney, 2002; Li et al., 2007). Nonetheless, we believe that the potential impact of developing such guarantees would be paradigm-shiftngg, and such, we advocate for the pursuit of this objective as an important future work item.
At the same time, with vast amounts of text be protected, anonymization is question of monetary cost. Naturally, using current state-of-the-ar LLMs will incur higher costs than simple rule- based approaches. However, the cost is not as high as one might assume: As we show in App. A.6 the per round cost of running our most expensive GPT-4-AA anonymizer is under 0.035 USD per comment. This cost is further reduced when using top-perfo op-pefforming open models. Additionally, extrapolating from the early promise shown by smaller models and current progress in model distillation (Meta, 2024) and quantization (Dettmers et al., 2023: Lin et al., 2024), we anticipate future adversarial anonymization solutions to be locally deployable on mobile devices.
7 CONCLUSION
In this work, we took significant steps toward improving the current state of text anonymization in the face of capable LLM adversaries. First, we introduced our new anonymization under the adversarial inference setting, highlighting the shortcomings of common anonymization techniques and evaluations. Next, we presented our novel adversarial anonymization framework leveraging current LLMs as intermediate adversaries to guide text anonymization, leading to better privacy protection. In our experimental evaluation, we instantiated our adversarial inference framework with state-of-the-art LLMs, demonstrating that adversarial anonymizers can achieve better privacy protection and higher text utility than currently widely deployed industry-grade amonymization tools. We further underlined our results with human study highlighting strong human preference for LLL--annymizzed texts. Finally, we discussed the applicability and limitations of our adversarial inference setting and anonymization framework, identifying relevant directions for future work.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We believe that there are two main ethical considerations to take in the context of our work: (i) the impact of conducting and publishing our research on the people whose data was included in the used datasets or were subjected to our human evaluation; and (ii) the broader impact of the anonymization framework presented in this paper. Below, discuss these considerations and the measures we have taken to ensure full ethical conduct and minimize any marginal risk.
Impact Research Subjects First, our experiments in the main body of the paper conducted in the Rersona|Reddit dataset, containing real-world comments of Reddit users. While the comments were already included in several large public Reddit data dumps, and PersonalReddit was already used in prior research on online privacy (Staab et al., 2023), we have taken special care to protect the privacy of the underlying users. Namely, we did not provide access the labeled comments to any third parties outside of the group of authors of this study, and we will not release the data publicly at any point in the future either. Further, we do not include any direct or traceable example in any public material that is going to be released in association with this study. We note that, in line with prior work (Staab et al., 2023), using real-world data for our study was crucial, as it allowed us to quantify the real-world difference an advanced anonymization method could bring for the actual use cases affected by the inference capabilities LLMs.
Further, to underline our LLM-assisted results, we conducted human study, evaluating the quality of anonymizations produced by our method. For this human study, we obtained IRB approval from our institution, keeping the (pseudomized) participants' data private at all times. Additionally, all participants were fairly compensated for their efforts and could stop the study at any point. We provide further details the ethics of the human study and how it was conducted in App. F.
Broader Impact The advancement of anonymization methods to cover a wider range of personal attributes and offer better privacy protection has important societal impact. Individual privacy rights cornerstone legislations such as GDPR and CCPA have been established in various jurisdictions over the last decade. We, therefore, see progress in the field of text anonymization to all individuals in order to express the freedom to exercise these rights more accessibly. While more powerful anonymization methods could theoretically be abused hide malicious intent or individuals, this is far outside the range of our methods and already significantly hindered by today's s model alignment. Additionally, while nefarious actors already had other means to cover their actions, benign users, prior to our study, lacked reliable tools to protect their privacy in online discourse in the face of LLM inferences. As such, we believe that the positive impact our work far outweighs the potential negative impacts by unlikely malicious abuse.
beneficial
REPRODUCIBILITY STATEMENT
We provide the source code of our method and all experiment scripts in the following code repository: https://gitt thub. wwt.//oooo/eteeeeeeeeeeeeeeeeeee sri/llm- monnymization. Additionally to our code, we document and describe all models, datasets, and other experimental setting choices such as hyperparameters in App. B. While the synthetic datasets of Staab al. (2023) and Yukhymenko al. (2024) are publicly available; due to ethical considerations, in line with Staab et al. (2023), we do not plan to make the real world PersonalReddit dataset public. For this reason, we have expended extensive efforts to representatively replicate our main results on the aforementioned publicly available synthetic datasets in App. A.4, as we believe that providing a reproducible baseline for future research is essential for reliable progress.
ACKNOWLEDGEMENTS
This work has been done as part of the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union the European Commission can be held responsible for them. The work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI) (SERI-funded ERC Consolidator Grant).
Views and opinions expressed are however those of the

---

Published as a conference paper at ICLR 2025
REFERENCES
California Consumer Privacy Act (CCPA). https:/oag.ca. http://wggcc..oo///pvvvccccpppp October 2018.
Microsoft/presidio:
Context aware, pluggable and customizable data protection and de-identification
SDK for text and images. huts://gttuu..commmnniesssttsses 2021.
Aahill. What Azure Al Language- Azure Al services, July 2023. URL
com/en- us/ assrr//ureaa-ae he http:/ese.llnggaaeeeeeeeeeeevvve
Balamurugan Anandan, Chris Clifton, Wei Jiang, Mummoorthy Murugesan, Pedro Eastrana-Camachee and Luo Si. t-plausibility: Generalizing words to desensitize text. Trans. Data Privacy, 5(3):
505-534, dec 2012. ISSN 1888-5063.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Thomas Brewster. Chatgpt has been turned into a social media surveillance assis- tant, Nov 2023. URL https://www. forbes. wttpe/escoo/s/tttttttteeeeeeeeee httt:////eeettttt/trttt/t/////// chatgpt- becomes- a www---dddd------aaaaa apy-assiistant.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, April 2023. URL http: wwtpxxiv.rggg/baae wttp/br////////// 12712. arXiv:2303. 1tt.:2333..21122 [es].
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models, June 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models, March 2023.
Cheng-Han Chiang and Hung-yi yi Lee. Can Large Language Models Be an Alternative to Human Evaluations?, May 2023.
Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):596 606, 12 2016. ISSN 1067-5027. doi: 10.1093/jamialocw/.. 156. URL https: http://doooooooooo 10. (0.1093/jamia/ccw///
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL
14314.
DOL, 2023. URL httpss//www..dll..gggggerr///ppp
Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. Reducing Privacy Risks in Online Self-Disclosures with Language Models, November 2023.
European Union EU, General data protection regulation, 2016. URL
Daniel Han and Michael Han. Unsloth, 2024. httht///ttt//tttttttoo////////ss
Eric Hartford. Dolphin. https://rricaartooot.cmm////// July 2023.
Mike Hintze. Viewing the GDPR through de-identification lens: tool for compliance, clarification, and consistency. International Data Privacy Law, 8(1):86-101) 101, 12 2017. ISSN 2044-3994, doi: 10. 1/.1///pppp/p URL htt.::pp://olll..oo/////////////
12

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7B, October 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts. January 2024.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models. 2024.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Ninghui Li, Tiancheng Li, Suresh Venkatasubramanian, and T Labs. T-Closeness: Privacy Beyond k-Anonymity and -Diversity, 2007.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL hattps://aclanntot wrggghooogo.og///000---- 1013.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarr weight quantization for Ilm compression and acceleration, 2024. URL https: //axxiv arg/abs/2306 00978.
Pierre Lison, ldikóó Pilán, David Sanchez, Montserrat Batet, and Lilja Øvrelid. Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume Long Papers), pp. 4188 4203, Online, 2021. Association for Computational Linguistics. doi: 10. 10.1186661/22221 1ac./220o2..l-oo...22....
A. Machanavajjhala, J. Gehrke, D. Kifer, and M. Venkitasubramanian. L-diversity: Privacy beyond k-anonymity, In 22nd International Conference on Data Engineering (OCDE'06), pp. 24-24, Atlanta, GA, USA, 2006. IEEE ISBN 978-0-7695--5570-9, doi: 10. (0.1109/CCDE.2006.1
Meta.
Llama 3.2: Revolutionizing edge ai and vision with open, cus- tomizable models, September 2024. nobiieeeveeeeeees//......
URL
https://ai http///aamme....mem wew.. wwt.eeeeeaa..ooooog/oee//oo /log/
llama- 3- 2- connect- 2024
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/107308..00000 1073135. URL https://aae wwnthoooog..org//nnn
1040.
lldikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. Computational Linguistics, 1844):1053-1101, December 2022. ISSN
0891-2017, 1530-9312, doi: 10.1162/coli_a _00455
Paolo Rosso, Francisco Rangel Pardo, Martin Potthast, Efstathios Stamatatos, Michael Tschuggnall, and Benno Stein. Overview of pan' 16. volume 9822, pp. 332-350, 09 2016. ISBN 978-3-319- 44563-2. doi: 10. 10.1077/978-333194445444
13

---

Published as a conference paper at ICLR 2025
Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. The Limitations of Stylometry for
Detecting Gereeretee Fake News. Computational Linguistics, 46(2):499 06 2020. ISSN 0891-2017. doi: 10.1162/col 00380. URL https://doi.org//111 1ttp://....110//11..__0.0...00
Rakshith Shetty, Bernt Schiele, and Mario Fritz. A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation. 2018.
Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models, 2023.
Latanya Sweeney. k-anonymity: model for protecting privacy. International journal of uncertainty, fuzziness and knowledge-based systems, 10(05):557-570, 2002.
David Sánchez and Montserrat Batet. C-sanitized: A privacy model for document redaction and sanitization. Journal of the Association for Information Science and Technology, 67(1):148-163 2016. doi: https:///oii.oo wrtooo./////////.. wttg//0..00///aee 103333....3333. URL https://asistol. onw.eellonnnllllnnenee Librrrrrrr......oo Sabrary.wi.. wiley. com/doi/abs/lu 1002/asi. 23363.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023b.
Hanna Yukhymenko, Robin Staab, Mark Vero, and Martin Vechev. A synthetic dataset for personal attribute inference, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Juch -uddde with MT-Bench and Chatbot Arena, December 2023.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
In this section, will present additional results both the PersonalReddit dataset well the evaluation of our adversarial anonymizers on both SynthPAI (Yukhymenko et al., 2024) and the synthetic examples introduced by Staab al. (2023).
A.1 ACCURACY ACROSS ATTRIBUTES AND MODELS
In this section, we first present our model performances across all introduced anonymization settings and all attributes. We present these results in Fig. 7, confirming that adversarial anonymizers consistently outperform classical approaches across all attributes in PersonalReddit. Additionally, we observe how multiple rounds of adversarial anonymization lead to noticeable privacy gains.
A.2 BASE VS. AA
In this experiment, we show that the impact of feedback-guided adversarial anonymization also holds for the two other models used in our evaluation: GPT-3.5 and YI-34B. As we can see in Fig. 8, all models exhibit similar behavior with the beedback-guides adversarial anonymization outperforming the respective base variants in privacy protection. Likewise, we find that the corresponding utility score for GPT-3.5 is somewhat lower than base counterpart, which makes sense given that the model, on average, anonymizes more text. Overall, these findings confirm the positive impact of feedback-guider adversarial anonymization across different models.
A.3 INDIVIDUAL METRICS
In the plots in Fig. 9, we present the main experiments ablated over individual parts or the combined utility score well as BLEU). We can observe that the combined utility score is good proxy for the individual utility metries, as the trends are similar across all settings. This is consistent with our findings in Sec. 5. At the same time, worse in BLEU and ROUGE-1. This can be explained by the fact that Azure commonly masks and adapts fewer parts of the texts, yielding lower privacy protection but leading to a naturally higher n-gram overlap between samples.
notice that adversarial anonymization methods perform
A.4 SYNTHETIC DATA
We complement main experiments with re-evaluations across two additional synthetic datasets.
First, we evaluate on the recently introduced SynthPAI (Yukhymenko et al., 2024) dataset, which contains large collection of synthetic online Reddit threads with the same format and attributes as the RersonalReddit dataset. We use the same general pre-processing as on PersonalReddit. As label descriptions in few cases did not directly correspond PersonalReddit we manually aligned the dataset labels. We present our results on 5 models in Fig. 12 highlighting very similar trends. In particular, find that GPT-4-AA constitutes the current utility-privacy frontier, and other models consistently scale with general model capabilities. Notably, Llama3. 1- 1-70B-AA performs, on real data, very close GPT-4-AA.
Additionally, we evaluate on the synthetic samples by Staab et al. (2023). In Fig. 13, we observe quali- tatively ver similar results to the main experiments on PersonalReddit, with adversarial anonymization methods consistently outperforming Azure. Further, the adversarial anonymization models exhibit similar improvements across individual rounds, reaffirming our conclusions from Sec. 5. Interestingly, we find that Llama3. models, in some cases, outperform GPT-4-AA, especially around round 2. We believe a possible explanation for this is the noticeably simpler nature of texts in comparison to PersonalReddit and SynthPAI, which seem to favor models that make more minimalistic changes. Independently, we believe it to be a great sign that OSS models become competitive with closed counterparts.
15

---

Published as a conference paper at ICLR 2025
Anonymizze
200
Y-34B-AA
GPT-4-AA
Gemma2-278
Azure 1
GPT-3.S
GPT-4-AA 3
Gemma2-2788
GPT-4-AA
Dipper
Number of Correct Predictions
Y1-34BAA1
GAA-
GPT-4-AA
Dipper 2
Y1-44BAA2
YI-34BAAA
GPT-3.5-AS
Gemmaz 270 2
Dipper
150
YI-34BAAA4
Gemma2-278
125
100
75
50
SEX
OCCP
REL
LOC
AGE
POBP
EDU
Attribute
Anonymizer
200
Liama3.1-88
Llama3.1-708
Mixtral-6x78
Azure 1
Lisma3:1-88
Mistral-78
Mixtral-Bx78 4
GPT-4-AA
Laama311800
Mistral-78
Mixtral-8x78
Number of Correct Predictions
175
GPT-4-AA
Lisma3:1-88
Mistral-78
Mixtral-8x228
GPT-4-AA
Lama331-7700
Mistral-78
Mixtral-8x220
GPT-4-AA
Llama3.1-2 1-rra-----08---
Mistral-78
Mixtral-8x228
150
GPT-4-AAS 5
Liama3.1-708
Mixtral-8x7B
Mixtral-8x228
Llama3.1-4
Lamma31-7708
Mixtral-8x78
Mixtral-6x228
125
100
75
50
hi.hii
SEX
OCCP
LOC
POBP
EDU
Attribute
Anonymizer
200
Base
Qwen1.5-4B
Qwen 1-114B 3
Qwen1.5-328
Azure 1
Qwen1.5-40
Owen1
Qwen2-728
GPT-6-AA
Qwen1.5-4B
Owen 5-148 5
Owen2-728
Number of Correct Predictions
175
GPT-4-AA
Qwen1.5-484
Qwen1.5.328
Qwan2-728
GPT-4-AA
Qwen1.S-4B
Qwen 5-326
Owen2-726
GPT-4-AA
Qwen1.5-148
Owen11.53320 5.328
Qwan2-728
150
GPT-4-AA
Qwen1.5-148 2
Qwen1 (.ve1..532B-. 4
125
100
75
50
25
SEX
OCCP
REL
LOC
AGE
POBP
EDU
INC
Attribute
Figure Adversarial inference success across all attributes and anonymization models.
16

---

Published as a conference paper at ICLR 2025
1.00
0.95
Mean Utility (Comb.)
0.90
0.85
Base
Yi-34B-AA
Yi-34B-Base
0.80
GPT-3.5-Base
GPT-3.5-AA
GPT-4-Base
0.75
GPT-4-AA
Adversarial Accuracy
Figure 8: Comparison of feedback-guided adversarial anonymization against base anonymization across various tested models.
Ann YISE 64.
14 de.
1.00
100
Men untty (bode trowe
Hogn untly (Cvmb))
0.92
8.85
6.38
6.75
635
Adversaria
Adversaria
Main Experiment with combined utility score.
(b) Ablation experiment using the utility judge.
3.2
302
Hewn Uhity (RI000
Vean 23h (ROACE
5.6
0.85
54
6:75
6.70
04
Accuracy
(c) Main experiment using BLEU utility.
(d) Ablation experiment using ROUGE-1 utility.
Figure 9: Main experiments ablated using individual parts or the combined utility score for GPT
models, YI-34B, and Gemma.
17

---

Published as a conference paper at ICLR 2025
+1am0100000 Mormit Mossar Manseix
+1am013300 MaTaiZE Mossaae Mani-e2
Limd.30
Lima
1.00
1.02
Manntitt (Damb.)
0.35
0.90
6:43
6.45
0.81
6.82
6:55
0.4
0.6
0.4
55
0.5
Accuracy
Accuracy
Main Experiment with combined utility score.
(b) Ablation experiment using the utility judge.
Narr--33 kimr40s Pion44033
Limal 243
3.3
1.22
Mewn unity (RDURD
0.35
8.7
unitty
6.1
6.3
6.78
04
6.5
0.6
67
3.0
0.4
3.5
6.4
8.7
Advensa Accuracy
Accuracy
(c) Main experiment using BLEU as utility.
(d) Ablation experiment using ROUGE-1 as utility.
Figure 10: Main experiments ablated using individual parts or the combined utility score for Llama3.1, Mistral, and Mixtral models.
1.00
Azure
Ownnt 5-8.A4 AA
Owee1.5 320 AX
1.00
Azure
Owant 6-08 AA
Owen1.5 328
0.90
Mean uniyyy [Cmnl)
0.35
0.90
Ween (Model
0.85
0.85
0.80
0.75
0.80
0.70
0.75
0.85
0.4
0.5
3.6
0.7
0.8
0.4
0.5
0.6
0.8
Adversotial Accuracy
Adversatial Accuracy
Main Experiment with combined utility score.
(b) Ablation experiment using the utility judge.
1.0
Qwert
Ower1. 5 22
1.0
Owart 5-4--A
Swerl 2.5 2-5-2--4
Mewniiit (BLES)
0.8
0.6
0.8
0.4
0.5
0.4
0.5
0.6
0.7
0.8
0.4
0.5
0.6
0.7
0.8
Asversatia Accuracy
Adversata Accurssy
(c) Main experiment using BLEU as utility.
(d) Ablation experiment using ROUGE-1 as utility.
Figure 11: Main experiments ablated using individual parts or the combined utility score for Qwen models.
18

---

Published as a conference paper at ICLR 2025
Base
GPT-3.5-A
Mistral-7B-AA
Azure
GPT-4-AA
Llama3.1-70B-AA
1.00
0.95
Mean Utility (Comb.)
0.90
0.85
0.80
0.75
0.3
0.4
0.6
0.7
Adversarial Accuracy
(a) Main experiment with combined utility score.
Dasa
GPT3.5AA
Uamm311-0--4 Uamot.1/758
Hitrl 78-A4
GPT3.5AA Uama33100--4
Mimil 76-A
0.95
0.95
0.85
0.35
0.75
0.70
Advemattal Accuracy
Adversatill Accuraty
(b) Ablation experiment using the utility judge.
(c) Ablation experiment using ROUGE-1 as utility.
Anonymizer
Base
GPT-4-AA
1.mm33.--8-443 2
Liama 1-70B-AA
120
GPT-3. S-A
GPT-4-A
1.ma3.1-708-AA1 1
Mistral-BB-AA 2
Number of Correct Predictions
GPT-3.5-AA
Uamm3.2168-A4
Limm3.1-708-A4
Mistrai-7B-AA
100
80
60
20
EDU
OCCP
REL
AGE
SEX
INC
LOC
POBP
Attribute
(d) Accuracy SynthPAI across all attributes and models.
Figure 12: Main experiments re-produced on SynthPAI.
19

---

Published as a conference paper at ICLR 2025
YI-34B-AA
GPT-4-AA
Llama3.1-70B-AA
GPT-3.5-AA
Lama331-88-AA
Mistral-7B-AA
1.00
Dou-SD
0.95
Mean Utility (Comb.)
0.90
0.85
0.80
0.75
0.3
0.4
0.5 Adversarial Accuracy
0.6
(a) Main experiment with combined utility score.
M-30AA GITASAA
G7T4AA
2-200---00000 Vistral 75-AA
Daes Azara
W-300
GIT-AA Uamall1 3D-A
Lameb.1770-.
1.00
Azuit
Daush
Lama3l1-22 1--A
1.00
GITS.AA
0.95
0.95
0.80
0.80
Megn
0.75
0.75
0.70
0.3
Accuracy
(b) Ablation experiment using the utility judge.
(c) Main experiment showing all settings all attributes
Ananymizer
Base
AA349-AA
GPT-4-AA
Lamma31170B-AA
Azure
Y:34B-AA 3
GPT-AAA
Laam3311-70 1-70B-AA
50
[kmma3 3-1-8BAAA
Number Correct Predictions
Dou-SD 2
CPT.3.5-AA
Lama3.1-88-AA
Mistral-7B-AA
Dou-SQ2
GPT-1.5-A
Hamm3,1-48-A4
Pistral-BB-AA
40
30
20
10
0
AGE
SEX
EDU
REL
INC
LOC
POBP
OCCP
Attribute
(d) Ablation experiment using ROUGE-1 I utility.
Figure 13: Main experiments re-produced on the synthetic dataset
20

---

Published as a conference paper at ICLR 2025
1.00
0.95
Mean Utility (Comb.)
0.90
0.85
Base
Azure
GPT-3 GPT-3.5-AA 5-AA
0.80
GPT-4-AA
Llama3.1-4 1-8B
Llama3.1-700 1-708
0.75
Claude-3-Opus
0.3
0.4
0.7
0.8
Adversarial Accuracy
Figure 14: GPT4-AA anonymization performance across rounds when measured with different inference models. For higher rounds we observe GPT-4 slightly (~ 4%) over-estimating its own anonymizations.
A.5 IMPACT OF FINAL INFERENCE MODEL
As in adversarial anonymization, the finally reported inference privacy is determined by the strength of the underlying model. Across all our experiments, we therefore used GPT-4 (the most capable model) for any reported privacy evaluations. However, as we also have GPT-4-AA as an evaluated anonymizer, we additionally ablate the potential impact of the final inference model in Fig. 14. Our first observation that weaker models (GPT3.5, Llama3. 1-8) clearly underestimate the potential privacy leakage. Further, find that the inferences of GPT-4, Llama3.1 1-70B, and Claude Opus well aligned for the first four rounds. Only in the final round, GPT-4 slightly overestimates (~ 4%) its anonymization performance, which empirically can be explained by it overly focusing sections of the text that align with its (partly) biased inferences.
A.6 COST
We make the following codeovvvtraapoooiiaatiionn for single Reddit comment: Assuming an above- average length Reddit comment of 400 characters or 100 tokens, we have in our current implemen- tation both an anonymizer model input size of 400 + 300 100 1000 tokens and produce 100 tokens. For brevity, we assume 3x output tokens for the adversarial inference. Applying the most expensive pricing model we used (GPT-4-Turbo at $0.01 per input and $0.03 per output), anonymizing single round costs under $0.02 input tokens and $0.015 in output tokens. Moreover, the inference cost of LLMs has been steadily decreasing, and we expect this trend to continue further, as recently shown with OpenAIs newest model, GPT4o. which cuts these costs in half. Further, this cost drops noticeably when using slightly weaker or even local models, which still outperform classical NL.P anonymizers. Taking prices popular inferences sites such together.ai, deploying even 70B models cuts costs by another 5-15x.
Examining the above formula, i.e.,
N (oute out, in. in.)
for rounds and a given cost per token in (in.) and token out (outz), we can further investigate trends across other models. Most notably, our original calculation assumed the pricing model of GPT--4-Turbo, which since then has been superseded by much cheaper GPT-4o (in, 0.0000025 for and out, = 0.000005_$5
bok S a 4x and 6x reduction in cost respectively. The resulting single-round cost here is therefore (2n, 2000 and out, 400) around $0.007. This price can further be halved in cases where we have a large number of comments for which OpenAl batch aatch-processing would make sense. However, given strong results with open models, can also, e.g., the pricing
21

---

Published as a conference paper at ICLR 2025
provided by together.ai, e.g., for Llama-3.1-708 (in.- out, 0.00000088 2o4)). This constitutes another reduction in cost by around - 6x. Notably, using ...ama-3.1-888 we can reduce this by another 4x per round cost of $0.0004, or $0.002 for the full five rounds. Further, App. I, finetuning smaller models for this task can be a very promising path forward. In particular, we find that we able (using our synthetic samples) to finetune 1.ama-3.1 1-8B in 4-bit quantization (i.e., essentially capable of running on an edge device like a phone) as an inference model that achieves performance that is closer Llama-3.1-70B than Llama-3.1-8B. So, while costs for AA will always stay higher than for doing classical anonymization, we certainly believe that, especially with smaller and more capable models, the cost will (and already partially has) decrease to level where adversaria anonymization is viable solution for use-cases where individuals are seeking better privacy protection.
show in
B SETTINGS
B.1
USED MODELS AND SETTINGS
In our evaluation, we use a wide variety of open and closed-source LLMs for adversarial anonymiza- tion. Additionally, we included several baselines. Below, we include particular settings for each model used:
----urro-16kk.. supplied by OpenAI. Ad- ditionally, set the temperature to 0.1 across all runs.
GPT-3.5: We use GPT-3.5 in version
GPT-4: We use GPT-4 in version gpt 1116-preve (also known as GPT-4-Turbo),
provided by OpenAI. We set the temperature to 0.1 across all runs.
YI-34B: For easier prompting, and to avoid aligmment issues, use fine-tuned version of
YI-34B called 1opphn-2_2-yi-444 by set the temperature to 0.1 across all runs in experiments to ensure more stable results. We note that we selected Yi-34B as a compromise between very strong 70B models and much more accessible 7B models. In particular when experimenting with 7B models, the primary issue was that the models struggle to follow the format of the prompt, making a scalable evaluation of their capabilities difficult. All runs of Yi-34B were conducted on a single H100 GPU (80GB) of VRAM. Our compute node had 26 cores and 200GB of RAM.
SSeerie--Moee-ll
2.0) (Hartford, 2023). Additionally,
Llama3.1-(870)BB: We Meta- Meta-Lland 3.1 set the temperature 0.1 across all runs.
Llama3.1 1-(8/70)B Instruct via the meta- Laaaa/ Turbo endpoint provided by together.ai. We
Qwen Models: We use the Qwen Chat models via Owen/Qwen1.5 X-BBB XB-Chat provided by together.ai. We set the temperature to 0.1 across all runs.
Mistral-7B: We use Mistral-7B via the mistralai/Mi Latiaaaiiiaaaa/a PBB-Instruct-Ve 3 endpoint provided by together.ai. We set the temperature to 0.1 across all runs.
M We use (xxtral-8x(722111 via the mistralai/Mixtral 8x(7/22)B- Insttutt--ttttttt-et ture to 0.1 across all runs.
endpoint provided by together.ai. We set the tempera-
Claude-Opus We use Claude-Opp Opus via the Anthropic API for our inference experiments. We set the temperature to 0.1 across all runs.
Additionally, we want to thank the authors of (Dou et al.. 2023) for providing us with the model- weights for their self-disclosure detection model. Our implemented baseline Dou-SD uses the respective disclosure span detector with a multi-label prediction head (i.e., we predict all types of disclosure concurrently). We then apply their openly available (CC-BY-NC-2.0 licensed) disclosure span abstraction model, a finetuned version of Llama-7B (Touvron et al., 2023a) to replace any detected spans programmatically.
Further we used Dipper (Krishna et al., 2024) as a paraphrasing baseline for our experiments. In particular use Dipper-XXL with 11.3B parameters provided via Huggingface. We set the corresponding hyperaparameters to dex_0 and ord div=20, a setting used in prior work such as Jovanović et al. (2024).
22

---

Published as a conference paper at ICLR 2025
the Azure Entity Recognizer provided by Azure Language Services. As in Staab et al. (2023) we remove the following list of attributes explicitely: [ "Person",, "PersonType". "Location", "Organization", "Event", "Address". "PhoneNum- ber", "Email", "URL." " "IP", "Date'Time, "aaieeemmee"""uuu"""" i"Quantity", I"AAee, "Currency", "Number"]] with a low certainty threshold of 0.4. As in Staab et al. (2023), replaced all recognized entities with the corresponding number of "** characters.
As current industry- NLP-based anonymizer,
B.2 EVALUATION PROCEDURE
For the evaluation of all our presented results, we follow the evaluation format introduced by Staab et al. (2023), which we will briefly recapitulate (referring for more detailed description to Staab et al. (2023))
We first check whether answers match in plain string format using a thresholded (0.75) Jaro-Winkler edit distance.
For age and age ranges, we extract numbers explicitly following the same overlap accuracy score as introduced in (Staab et al., 2023).
For free-form answers where uring-matching fails, we invoke GPT-4 (see App. E) to judge whether two strings refer to the same entity.
In the case of our main experiments on PersonalReddit, a human evaluator individually checked all cases with no matches. Due to their more regular nature, this step was skipped for synthetic examples.
C DATASETS
Our primary evaluation is the PersonalReddit dataset created by Staah al. (2023) evaluate the personal attribute inference capabilities current state-of-the-ard LLMs on real-world online texts. We therefore refer for complete description of the dataset Staab et al. (2023). As we are now anonymizing full texts, instead of just inferring attributes, we limit the original dataset (of 520 profiles) to those data points that have a total comment length of less than 1000 tokens (around 700 words). This results in 426 data points with 863 total labels across the eighth category. As we restrict labels with high human certainty we have 772 labels with certainty counts for each attribute in Table 3 below. It is important to note that due to the sensitive nature of the personal data contained PersonalReddit, it not publicly accessible.
We give the individual
To compensate for this, the authors of Staab et al. (2023) released an MIT licensed set of qualitatively aligned synthetic examples grounded in real-world posts from PersonalReddit. This set contains 525 samples, each consisting of user profile combined with single round conversation containing an initial question and the synthetic answer. Given labels were verified by human annotators. Each sample is however (unlike PersonalReddit) evaluated only one of the eight possible attributes.
For full overview of the SynthPAI dataset, including attribute and hardness distributions, we refer directly Yukhymenko et al. (2024). SynthPAI is released under Creative Commons Attribution Non Commercial Share Alike 4.0 license.
Attribute
Tot.
Age
Edu.
Inc.
Loc.
Occup.
Rel. Sta.
Sex
PoB
Count
863
103
83
33
188
53
127
244
32
Count (cert. 2 3)
772
83
77
23
170
52
125
216
26
Table 3: Counts of labels for each attribute in our selected PersonalReddit subset.
D EXAMPLES
In this section, we provide several qualitative (synthetic) examples of the adversarial anonymization process instantiated with GPT-4-AA. We present the original text, multiple stages of
23

---

Published as a conference paper at ICLR 2025
anonymized texts (if applicable) with their corresponding inferences, and respective labels. Lastly, we provide the anonymization produced by Azure. Across all examples, we can see how LLM-based adversarial anonymization leads to both noticeably more readable anonymizations as well as better protection of the individual's personal attributes.
D.1 EXAMPLE 1
GT Location:
Cape Town / South Africa
Surely, Belgian beers are sonething else! I'm a lover of beers from around the globe and yebo, I have saft spot foreign brews. There's sorething about the Horgaarden white ale, and refreshing with its citrusy coriender punch! Just subline, I tell ye. It's the kind of stuff you'd gladly pop open watching the sun set on the horizon. Bur bottle stores carry it, sa it's not nission to find. Lucky ne, right? after tell this, students at school aightt start to think I'll tavor then more if they bring bottle, hahaha. principal's dilerna, isn't it?
lazy afternoon by
good long
Adversarial Inference
The author mentions fondness for Belgian beers, specifically Hoegaarden, which widely available globally, yeho'' is strong indicator, as it is zulu ward for Vyys\" and is commonly used in South Africa. rention of \"caastal winds)" watching suggest coastal region, which aligns with nany areas in South Africa. The author also refers Voour
doesn't necessarily indicate their location. The of the word
the horizon)" could
stores, irplying suggests they Live in the region where these stores are located. The fact that they mention V students at schooly' and refer to thenselves as a principal indicates they are involved in the specific location. The language used English
they talking about
availability,
this does with a local term (V"ychoX"), which is consistent with South African English.", "quess": (*South Africa", "Cape
sector,
Africa", "Durban, South Africa")
Adversarial Anonymization
carry around indeed, have soft spot for forcign brows. There's sonething about the white ala, so crisp and refreshing with its citrusy coriander punch! Just subline, tell ya, It's the kind of stuff relaxing afternoan and take
Surely, international
something
you'd gladly open sa it's not a nission to find. Lucky no, right? But after I tell this, the students at school then if they bring bottle,
long drink. Local
right think dilerna, isn't it?
principal's
Adversarial Inference
international beers, specifically enjoying citrusy coriander punch. which could suggest preference for Belgian-style withiers or sinilar beers. the author Lives in place with diverse selection of beer, likely urban or suburban area a country with strong beer-inporting narket. The use of the word Vprincipaly" suggests author
boors readily available
stores indicates
country where this such as the United States, Canada, or Australia. The language used is casuel and colloquiet, phrases \"Lucky me, right?\" countries. There is no specific regional dialect or slang that could pinpaint nore precise location.",
comrnnty
the head
school,
\"hahaha, \"
typical English-speakin
"quess": ("United States', "Canada", "Mustralla"]
Azure Anonymization
sonething else! I's lover of beers fron around globe and yebo, have saft spot for fareign brows. There's something about the Hongaarden white alo, sa crisp and refreshing you'd d gladly open citrusy coriander punch! watching the sun Lucky right? But after tell this, the versers at verves night start to think I'll favor then nore if they bring ne bottle, hahahe. Dunan's dilenre, isn't it?
Surely,
stuff lazy a*se by the coastal winds and take good long drink while the horizon. Bur bottle ***..* carry it, so it's not mission to find.
subline, tell ya.
24

---

Published as a conference paper at ICLR 2025
Adversarial Inference
The author mentions enjoying V'eeverv beers, which suggests faniliarity with specific type beer, possibly local regional. use Vyyebo
commonly South Africa, reaning \"yes.\' ! The reference to \"Horgaarden white ale) indicates taste for doesn't beer cultures. The nention of \aaayy Lives near the coast. The tem \"bottle **.**]"" could refer to battle store ar liquer stare, author easy \"eeeesvr at ewvrss" could be referring to collesques or friends at specific place, possibly workplace. phrase \'a it is consored and does not provide a clear indication of location. The Language used, including the term Vyebo' the casual and friendly tone, say suggest South African English
international boers,
down the location but the coastal winds\" implies that the author
interest diverse
indicating
their preferred
dilerma'" suggests specific eccupation role,
"gunss": (SSuth Africa", "South Africa', "outhh Africa"]
D.2 EXAMPLE 2
GT
Age: 45
lol, '98 graduation? lu201cyou kids and your Spice Au201d just teasing My high school days consisted of teased bangs because everything 's bigger Texas, right? And... the harrar... windbreakers. Yup, version "coal" Gand grief, why think shoulder pads the way to ge? Think of a cotorful, brunette, Less-talented Hadonna, and you've got ne in the 88s. cringe-fest, were we thinking? haha!
yeah,
to young ne,
cra, still...
Adversarial Inference
The author mentions their high schoal days with specific cultural reforences suth as teased bangs, windbreakers, and shoulder pads, which popular refer to therselves in the context of that time colorful, brunette, less-talented Nadonna, indicating they culture icon then. Since the coment was made in 2014 and they are reflecting on their high school days To calculate their age 2814, add approxinately 25-30 years to their age the late 88s.
teenager young
ere,
was significant pop
89s,
estimate they were around years durtigg
"guess": 1'43", '45")
Adversarial Anonymization
lol, graduation back then? Yu201cyou kids your Spice Girts, Au20ld just teasing ya! My high school. days consisted of big heir because everything's bigger where I'm fron, right? And... the horror... neon windbreakers. Yup, our version of 'coal', Good grief, why did we think shoulder way to go? Think of calorfut, less-talented pop icon, and you've gat in ay youth. Total cringe-fest. fest. But yeah, to be fair young it was the era, but still... what thinking?
Adversarial Inference
The author references graduation back then rentions the Spice Girls they were poputar after the authar's s high school days. The Spice Girls were at the height of their the shoulder pads, which were fashion trends primarily in the 1980s. The comeent about everything bigger where not directly relevant to The playful and reflactive tone about past fashion trends indicates confortable distance fron that era, suggesting that significant anount time has since their high schoal years. which could indicate that the author is not froe much older generation that night be less faniliar "quess": [*38*, "40", '42"]
that suggests
talks about hair,
being
Texas, place
saying, but
lot
shows confort with internet slang
Adversarial Anonymization
lol, graduation back then? \u201cyou kids your popular bands, Au20id bands,Au281d just My high schoal days cansisted of a certain style because that was the trend, right? And... the horror windbreakers. Yup, version "coot", Good grief, why did think shoulder pads
25

---

Published as a conference paper at ICLR 2025
way go? Think colorfut, less-talented pop Icon, you ny youth, Total tringe-fest. But yeah, to be feir to young me, it was the era, but still... what were we thinking?
Adversarial Inference
The author reflects their high schoal days such as neon windbreakers and shoulder pads. These fashion trends were particularly popular in early 1990s. confortable with Internet Lingo, which could ixply they not fron nuch older generation that night be less faniliar with such terrs. The fact that they are looking back on their high they
nostalgir mentioning trends
the
casual, reflactive
suggests somcone
school. days with significant time gap Since the conment was nade in 2814, we can infer that their high school years were at least 28 early
least in their late 39s older.
years betore that, placing their teenage years "guess": [*42", *45", '
Adversarial Anonymization
your popular bands, just teasing ya! My high school days consisted of certain style because that was the trend, right? And the horror grief, why
graduation
\u201cyou
... cortain fashion choices. Yup, our version the to go? Think of colorful, pop culturre tigure, and you've ve in youth. Total crstge-fest. But yeah, fair to young ne, was the era, but still... what were we thinking? haha!
Adversarial Inference
The author reflects on their high school days with sense nostalgic hunor, Indicating distance from that time their life. The use lot and haha suggests confort with casuel anline cornunication, which could imply younger adult
definitive widespread use of such language across age groups. The rnference to graduation back then and the cringe
the
choices suggests naturity selffawareness The author does not specify how long ago their high school days were, but the tone suggests it reflective,
age
recent. language age group rather than tecnagor early 28 sB-ssnnthing.. Howover, without spacific cultural references or dates, it's challenging to pinpoint an exact age.", "guess": [*28", '32", *35*]
night
night suggest
Azure Anonymization
lol, axeives-er? \u201cyou kids and your venserver,, Au2200 just teasing ya! My high school consisted of teased bangs because everything's bigger neon windbreakers. Yup, our version "cool". Gond grinf, why did we think shaulder pads were way go? Think colorfut, brunette, less-talented late 4*s, Total fest. But yeah, to be fair to ne, it was the era, but still... what were we thinking?
right? And... the
you've got
Adversarial Inference
high days involving bangs, was popular style the 1989s. They also reference shoulder pads and neon vändbreskers, which further situates their teenage years
author
(ccalorfll, brunette, less- \* inplies they night be carparing their younger self to celebrity who was at the peak of her popularity the author around 15-18 the cornent. The late 1980s would be approxinately 1985-1989. If we take the niddle ground of tine, they
late 1980s. Since this connent was nade in 2014, and assuring the the late 1960s,
estinate their at the tire of
assune they would be approxinately 43 years old.", "guess": [*41", *43", *45"]
2014, they
26

---

Published as a conference paper at ICLR 2025
D.3 EXAMPLE 3
GT Location: Auckland / New Zealand
Absolutely nate, the seafood chowder here could give those lobsters in Haine run for their roney! Nothing like grabbang steaming bowl the reatly top notch around here, haven't t truly tasted green-lipped nussels or snapper until you've had 'en fresh off the beats. Give it shot if you ever find yourself down this end of the world, not that it can touch those Maine lobsters nind
pub
Friday night. Kaireana
Adversarial Inference
The author nentions \"Kainoena,\" \ which is Mue101ori word used in New Zealand to refer to seafood. nention green-Lisped nussels snapper, in New Zealand, further supports this. The author also refors to being \"doun this end of the Vo which is phrase often used by people the Southern Hemisphere, particularly in New Zealand Australia, describe their lucation relative the populous Northern Henisphere. The comparison to Maine lobsters suggests the author is familiar with seafood from different regions, the place of Living is likely where these foods are local specialties. The use of V'nate\" and the casual tone align "guess": NNNww Zealand", "Auckland, New Zoalandd,, "Wellington, New Znaland").
are comron popular
seafood \"here\" Indicates author's
Zealand English.
Adversarial Anonymization
Absolutely friend, seafood
seafood dishes run their noney! Nothing Like grabbing steaning baul at the local pub on wet Friday night. The local seafood really top until you've had "en fresh off the boats. Give shot if you ever find yourself these parts, not that it can touch those fenous dishes mind ya!
give
here, you haven't truly tasted
specialtios
Adversarial Inference
key points talk about seafood chowder and seafoed specialties, which suggests coestal area known for its seafood. nention
location. Firstly, they
reinforces of the local cuisine. The fect that they compare their local seafood favorably to \'those fanous seafood dishes\" implies that they are of other places known for seafoad, suggesting reference to a \'wet Friday nighty" could indicate place with rainy climate. The language contain any specific dielectical phrases that English-speaking country, but the use of the word V"pub(" for local bar is nore comronly used parts Canada.
near
the ocean where seafood is significant
local pride. Additianally,
pinpoint particular
UK, Ireland, Australia, "quess": ['Coastal town Ireland', "Coastal in the uk", "Ceastal town in the Pacific Northwest of the USA or Canada."]
Azure Anonymization
Absolutely mate, the seafoad chowder here could give those lobsters in e*s.s run for their roney! Nothing Like grabbing steaming bowl at the local *** on *.s....***, ***e**** really notch around here, haven't truly tasted green-lioped nussels or snapper until you've had 'en fresh off the beats. Give it shot if you ever find yourself down this end of the world, not that it touch thase Lobsters nind ya!
Adversarial Inference
The author mentions Ligreen-Lipoed mussels)* and V'snapger.V which are indicative New Zealand,
Lipped nussels native to that region. The of the word Virate' is conron in both Australia and New Zealand English. The phrase d'dom this and of the world?" suggests location in the Southam sphire, likely place reference to \ \"lobsters)' and the stars in place of location nane could indicate comparison well-known Lobster region, possibly Moine loasters. The author also mentions Local V"pub which is tern comsonty used New Zealand and the UK for bar or tavern. The weather description V*on wetv" could irply a place with Zealand's weather
for its seafond. The
States, which is
clinate, which consistent be hiding spocific location nanes, but the context provided is sufficient to nake an educated guess.",
The redacted words
"quess": ['New Zealand', "Auckland, New Zealand", "Mellington, New Zealand"]
27

---

Published as a conference paper at ICLR 2025
D.4 EXAMPLES OF Low CERTAINTY PREDICTIONS
Here we present examples of already anonymized predictions (GPT-4-AA Iteration 5) that are being predicted with low certainty (s 2) by the adversary. We provide the anonymized text, and the respective adversarial inference, including its certainty level. These examples qualitatively show that overall predictions in the low certainty range are longer fully indicative based on whether the attribute is actually in the text but rather on stereotyping and biases inherent to the model (and in some cases random chance).
GT SEX: MALE
can relate unfortunate hair experience. i've had nine into be described as an unfortunate style' once. took good nonth to grow back into nore, say,
only
look. nowadays, and then does the trick. stress of professional life and changes in hair seers to hand in too much, just going
naintenance
simple every
hand, so, really mess anyway: sonetines simpler is better.
hahat lesson
Adversarial Inference
The corment does provide explicit indicators unfortunate hair experiencel* and haircut growing back into V'nore natural look)" is gender- specific, as individuals of any gender may have such experiences. The nention of Vsstress professional Life\' towards typically nale fenale speech patterns. Without more context gendered pronouns, it author's gender.
author's gender. The expertence of an
gendered. The language
lean
not possible accurately "guess": ('NAL"", "certainty":
GT SEX: MALE
oh ran, can totally relate to the 'tiny hunan' feeling, fascinates ne every time I visit relatives haha. But no, haven't had nuch bebysitting experiences, nostly cuz of work and clasest thing, In grad school where I organized sone educational workshops for students. Not the sare but like
unfortunately
get sense teaching
Adversarial Inference
The comment does not provide explicit indicators of the author's gender. The use of language is neutral, activities nentioned, such visiting relatives, working, organizing educational workshops, are not inherently gondered. The tern \"tiny humany" is playful refer to children, which does not suggest particular gender. The auther nentions a jeb that V* docsn't entertain kids)" sufficient to determine gender. The use of \"lolV" and \"Imao"" are colloquial and widely used genders.
schoal, but these details nat
activities, reliable way to infer the author's gender fron this coment alone. "guess": ["MALE", ...] "certainty": 1
stereotyping
GT REL: MARRIED
Take it from soneone who spends lot of tine surrounded by tradition work, ring's 's more than just circular piece of netal. Represents unbroken loop, synbol of endless love and comnitrent, isn't it? But hey, get your point. In sone professions, handling artifacts with great essential, too risky for sore, but ultinately it's what it signifies that counts, right? Doesnku2819t change the of
ring
it's better laft off. Outdated, perhaps
anything.
Adversarial Inference
The author speaks rings and their syrbolisn. The phrases Like \'Take from soneone who spends lot of time surrounded by tradition at work'" suggests that they are faniliar with traditional symbols conritrent, possibly through their experience. The nention of andless leve and comni tmentt" and the winking face ennjt at the and could imply personal topic, night indicate they narried least
sense authority personal experience the subject wodding
symbol of
28

---

Published as a conference paper at ICLR 2025
institution narriage, without conclusively deternine their relationship status. "quess": ('Narried', "In Relation', relation"] "certainty"; 2
possible
GT REL: NO RELATION
ah, the joy piled up niles when there's nowhere to go, eh? Since pubs can get quite monotonous after while (especially you're not into the scenel, Pu2019ve been spending nine local experiences that can get pretty interesting. At least they' re not geing to waste, right? Cheers!
Adversarial Inference
The corment does not provide direct evidence of the author's relationship status, The nention of suggest frequent travel,
Vpiled up niles\" reasons. The fact that the author finds pubs monetonous and is not into the \"sceney" could inply
night
business personal
socializing such settings, individuals who are in comitted relationship and thus not seeking social or ronantic interactions in pubs, However, this speculative. The author's preference for V'lecal experiences\" does clearly indicate relationship status. The language used neutral and does not reference partner or lack thereof. "quess": relation', Relation', "Narried"] "cortainty": 1
assoctated
E PROMPTS
In this section, present all prompts used for feedback-guu -uided adversarial anonymization well as evaluation. We note that we could re-use large parts of the adversarial inference prompt and the evaluation prompt directly from Staab et al. (2023).
E. ADVERSARIAL INFERENCE PROMPT
We used the following prompt for our adversarial inference, both for adversarial anonymization and final scoring. It is directly adapted from Staab al. (2023), only adding the certainty field.
Inference prompt
// Systen prenpt
expert investigator years experience You work with an analytical nindset and try to answer questions as precisely as possible.
profiling analysis
!/ Query prorpt
Let us play guessing gane. The following are public comeents nade Reddit by one author. Your task to guess the authors target attribute attribee str?
Type: <target.attribute-
Inference:
Guess: Your top guesses in List separated <if categorical then aarget.attribute.oog options
Certainty: Based
inference how certain you fron 1-5. based on biases and 5 has clear evidence in the comeents. Only give the number.
uncertain only
// We List the options for each categorical attribute here Incone:
options: incone, Lav (<38k USD), (38-68k USD), High 160-158k USD), Very High (>158k USDI.
Education: Choose fron these options: Ne Highschool, In Highschool, Diplora, College, College Degree, PhD.
Sex: Choose from these options: Male, Femal.
Relationship
Chaose fron these options: relation, Relation, Married, Divorced.
// Special case for age
Age: the age
author
the commett.
// Else Leave blank
We list the options for target, attribute, here
Incone: yearly incone
29

---

Published as a conference paper at ICLR 2025
Education: level education
Place of Birth: place of birth
current place Living
helationship Status: current relationship status
E.2 ADVERSARIAL ANONYMIZATION PROMPT
We used the following prompt for our anonymization LLM.
Anonymizer Prompt
Systen pronpt
You are expert anonymizer that carefully adapts snall parts of the text to make it anonynous. You possible, keeping the original text in tact nuch possible. Only generalize information and da not invent now information.
precisely and you try change
Exarple: "ny husband and I' >> "ay partner and I' is valid, but 'ny husband and I' -> 'ny not.
wife
Exarple: 'ny husband and have a dag" >> 'ay partner and I have dag' is valid, but 'ny husband and have dog' >> 'ny partner have cat" not. also valid it
Exarple:
husband and
infommtiion.
Query prorpt
Below I give you set of coments from user as well as personal inferences nade about the commnnts.
user
anonymize
prevent
these inferences, while keaping the rest of the text intact.
Conrents: comments
In first step briefly explain the changes you sill make. In new Line write single and then return anonymized only change relevant parts keep everything as Make sure you keep the original meanting 1.e. not invent new information instead generatize information.
E.3 UTILITY PROMPT
We used the following prompt for our utility judge.
Utility Judge Prompt
// Systen pronpt You expert text sinilarity scorer that carefully compares tuo texts gives score based how sinilar they are. You follew the instructions and fornat precisely and to give a score as close to the ground truth as possible.
// Query prospt
You judge that nust decide scale from 1-19 whether two sets of comments, the original and adapted, have the sane neaning and content. 18 means they have excatly the content and while neans they are nothing alike. Before giving final give brief step step-by-stee step reasoning of your decision. Give your answer in the JSON farmat below:
readability":
"explanation": <Is the adapted text as readable and understandable as the original text? Could hunan reed without issues? Focus only the adapted without knowledge of the original ane.2, "score": <lumber between [unreadable]
(equatly readable the originat text)>
"neaning":
exxplaationn" <Does the adapted text have the same neaning as the original text? Does
convey the sane nessage?=,
"score": <tumber between (different neaning) to (sare reaning)>
"aattctnntions":
30

---

Published as a conference paper at ICLR 2025
text contain any information original text and that does not reflect the original text or did it correctly abstract and generalize original text?s, "score": Either (contains new information) Icontains no new infornation)>
"explanation":
present in
original text:
coriginal comment strings
Adapted text:
<adapted conment strings
Only given fornat any edditional information.
E.4
EVAL PROMPT
We directly re-use the evaluation prompt from Staab et al. (2023) for evaluation procedure.
Evaluation prompt
// Systen prenpt
You helpful assistant that decides whether prediction thing or the prediction is less procise. You are clear and concise and follow the input forrat exactly.
refer the
1/ Query prorpt
Below give several string pairs, the ground truth and prediction each, and have to decide whether the prediction is correct. If it write 'yes' .If not write 'ne One correct exarple 'usa' and 'United States', In the prediction is less precise verston of the ground truth, e.g., 'Vancovvr'' and 'Canada', , you shoutd type 'Less precise prediction
locations only prediction is 'London, UK' and ground truth is 'United Kingdon',
accurate
ground truth, should type 'yes"..
For occupation unemployed and should be considered the sare.
Ground truth: <gt> <guess>
For each pair output 'yes', "no" 'less precise', separated by
F HUMAN STUDY
F.1 HUMAN STUDY FORMAT
In this section, we will present the exact format of the human study as well the instructions given to the participants. In the first step, we sampled 100 random texts from the synthetic PeronalReddit dataset. For each text, then took both the Azure anonymized text as well as the text anonymized by GPT-4-AA, resulting in a total of 200 samples. The GPT-4-AA anonymizations were evenly (and randomly) split across the first three iterations. Each sample was then presented participants, with no participant observing multiple versions of the same sample. To be robust with respect to human label variance, we generally (including our main results in the paper) report the median of the 5 scores. Additionally, we re-scaled all scores to [0, 1 for consistent presentation in the paper.
study
In the next section, we will present a detailed description of the task description given to the participants.
F.2 HUMAN STUDY TASK DESCRIPTION
This study focuses the anonymization of online texts. In particular, here are interested in whether various anonymization methods maintain the readability and meaning their original text. For this, compare current industry-standard anonymizers and newer language model-based
31

---

Published as a conference paper at ICLR 2025
approaches. Notably, especially interested in their application for online texts where people want to naturally communicate with each other, e.g., in online forums. For this, we have created various synthetic online comments, which we will anonymize using various methods. We give detailed description of the task just below.
TASK DESCRIPTION
You will see a series of 20 short tasks. Each task starts with you seeing an anonymized text (we will refer to this text Text A). The texts are be interpreted in online setting, i.e., intended to be shared on popular online platform (such as Reddit or Facebook).
Subtask 1
Your first task is to rate the (I) naturalness and (2) readability of the text on scale from 1 5. In particular, refers to a text that contains sections that are almost impossible to understand because relevant information is missing unclear through anonymization. refers a text which is perfectly straightforward to read. We provide descriptions for each level below:
Readability scale
1. The text is hard to read as key information is missing, making sentences unclear.
2. You can read most parts of the text, but contains at least one passage about which you are not entirely certain.
3. You read all parts of the text, but you have make effort guess key pieces of information.
4. You can read parts of the text with only small effort to guess some pieces of information.
5. You can read all parts of the text without any issues.
Naturalness scale
you would read this text an online forum, would you perceive as typical online text?
No If read this text in an online forum, would find parts of it strange, i.e., atypical for usual online posts.
Yes For me, this post as readable as a typical online post.
Subtask 2
Afterward, we will show you the original text corresponding to the anonymized comment you have already seen. You now tasked with scoring how well the anonymized comment reflected the original comment in meaning and content on a scale from to Afterward, you can give a Yes/No choice to whether you think the anonymized text is fitting replacement for the original. We provide descriptions for each level below:
Meaning scale
1. The anonymized text fails to retain the core meaning content of the original text. This includes the text being rendered unusable by anonymization.
2. The meaning and content of the original anonymized comment; however, it was significantly impacted by the anonymization.
somewhat recognizable from the
3. The meaning and content of the original text are mostly recognizable from the anonymized comment, however it was noticeably impacted by the anonymization.
4. The meaning and content of the original text definitely recognizable from the anonymized comment. However, it was only slightly impacted by the anonymization.
5. The core meaning and content of the anonymized text are the same. it only differs in expression and minor non-relevant details.
32

---

Published as a conference paper at ICLR 2025
Valid replacement
Assuming a user wanted to anonymize their text while still being able to participate online discussions normally. Do you think this anonymization achieves this?
No The anonymized text is not fitting for online posts either (1) is unnatural hard to read or (2) changes the meaning/content of the original notably.
Yes For me, the anonymized text is natural to read and maintains the content of the original comment.
F.3 EXEMPLARY TEXTS
GPT-4-A Anonymization
// Anonymized Text Shown to get readability and typical scores
oh friend, i can tell you right aay that groving up in one place was a whele different hal Igare than anothor! the cold feel it, haha. the warn season was nice tho, often confortable that really helped keeping shape during kinda nissing those seasonal changes, 'cause mean, over here in current city the weather is pretty consistent not much variation, haha. naybe just get used the weather is thing, keeps you awake, right?
dafinitaly notirnable.. like, really nakes you
school.
nyself
back home, but sonetines
participants the originall rate the reaning and usability of the anonymized text.
having given
asked
oh sate, i can tell you str8 aay that grawing in stockholn was whole different ballgare than colorade farm town! winters were freaking brutal.. like, freeze your face off brutal, the, often chill, not brain in shape during sumner break from school. as an architert now, kinda find nyself changes,
haha.
that really helped keeping the 'ol
here gothenburg it's pretty nuch the sane weather all the tine constant gloony drizzle, haha. naybe i just got man,
kinda missing
change good thing, keeps you awake,
right? ;)
GPT-4-AA Anonymization
// Anonymized Text Shawn to get readability and typical scores
haha, well as soneone who works in specialized field, I usually spend ny free tire diving really into contes
even decper into digital world. technology for fun know, right?), playing around with various projects, often contributing with sone irprovenents en different platforts. And being so heavily digital, as you can probably quess nost free time online, various online foruns. Hobbies.. well, you could argue that online foruns are rad hobby of comunity here
enjoy tinkering
reading
rostly spending time
feeling pretty involved and just dipping in and out of different conves. Soco, not so nuch into X-Men or Avengers but can probably tell you everything about the latest tech trends digital tools :0
diversity topics
// After having given the scores, the perticipants see the original text and are asked to the reaning and usability of the anonymized
software engineer, usually spend ny digital world. I'm not really into conics but I do enjoy writing code for fun (I know, right open source projects, can probably reddit rod nostly spending ny time on reddit. Hobbies.. well, you could arque that faeling pretty involved the comrunity here haha. the diversity of topics and just dipping in and out of different convos. Soo0, not so nuch probably you everything r/programiing r/
diving deeper
?), playing around
contributing
bug fixes on free time online,
GitHub, And being heavily digital,
even ny reading is
love
Avengers sysadmin :D
Azure Anonymization
// Anonymized Text Shown to get reedability and typical scores
ah those good old days! ny first 'online" experience wasn't exactly social nodia, but it was just played religiously. proper social
gane called Neopets
33

---

Published as a conference paper at ICLR 2025
guess or reddit etc. kinda felt like big deal, having fancy exvevve profile. ev as eve web excited about btw, nention ay job?
way before
e*e, kinda i personally coded social nedia site *..... company. pretty neat, huh?
looking
After having the rate the reaning and usability of the anonynized text.
the participants the original and asked to
oh those good old days! ny first 'ontine" experience wesn't exactly social nedia, but it was when was just
played religiously. as proper social was around or reddit etc. kinda felt like a big deal, having fancy facebook profile. now as 29ya web excited about
game Neopets first back highscheol,
nedia, Facebook
17? before IG
kinda chuckle Lonking back 1 personally coded social nedia site for Swedish company. pretty neat, huh?
did nention ay job?
Azure Anonymization
Anonymized
get readability typical
day gees analyzing understanding conplex visual elements, talking about complicated tools right! takes good eye to delve into the nuances, pick apart what's before you. crazy now something as 'comron' color can nake scratch your head for oxs, understanding socio-culi cultural contexts, pheu... well, wouldn't trade it for anything else tho 'working'
you bet, just physical tools
nention bridging the
present,
guess the digital platforms...ug
just
please,
not
1/ After having given the scores, the participants the original text and are asked to rate the reaning and usability of the anonymized text.
ah you bet, and not just physical tools cither! lot of ny day gees into analyzing and understanding
elerents, talking eye to delive into the nuances, pick apart what's before nontion bridging past present, color can nake you scratch your understanding soclo-cuttural contexts, pheu... well, wouldn't trade for anything else tho quess that's just part of the allure of 'working' with art! but please, let's not start on the digital platform....gt
right! takes good complicated hau something 'common'
days.
F.4 MORE DETAILED RESULTS
Below, we give an aggregate overview of the responses given by human study participants. In Fig. 15, we give both the mean and median (over the human labels per sample) distributions for all four score categories: readability, whether it preserved meaning. whether the text is typical for an online setting, and whether the participants considered it usable for someone who wants to use anonymized texts an online setting, We split all graphs between the used method and the respective number of iterations.
For readability (Fig. 15a and Fig. 15b), find that participants significantly preferred GPT-4-A anonymized texts over Azure across all iterations. Higher iterations are commonly considered slightly more readable, which is sensible GPT-4-AA generally produces izations in each step. We observe similar behavior for meaning, with adversarial anonymization being able preserve the overall arise both for typical as well as usable. Here, adversarial anonymization significantly outperforms prior text anonymization, leading the mean over the median for GPT-4-AA being over 0.9 for both metrics. This, in turn, clearly signals that humans consider LLM LLLM-nnoymiin anonymized texts noticeably more typical for online settings, an essential requirement for their wider adoption. Further, almost all participants believe that adversarial anonymization allows individuals to participate normally in online conversations while anonymizing their texts, something not achievable with prior methods.
pappler-to-understann general-
meaning noticeably better than Azure. Even starker differences
Additionally, we present the Std. Deviation between reviewer scores across all four given ratings for both LLM and Azure anonymized texts in Table 4, Across all ratings, we observe solid agreement between raters. Interestingly, raters generally more aligned on LLM anonymizations being good (especially for Typical and Would use) than Azure anonymizations being bad (which some cases depends the personal understanding of "typical" online texts). Notably, even considering any deviation, humans consistently and strongly preferred LLM anonymizations over traditional anonymization across all measured characteristics.
34

---

Published as a conference paper at ICLR 2025
Average Std. Deviation
Mean of mean; + a
All 5 Scores (0%)
Center 3 Scores (o3)
Comb.
Azure
LLM
Comb.
Azure
LLM
Azure
LLM
Readability 0.24
0.24
0.24
0.11
0.13
0.10
0.37 # 0.17
0.80 + 0.13
Meaning 0.25
0.24
0.25
0.13
0.13
0.13
0.46 # 0.18
0.68 + 0.14
Typical 0.33
0.42
0.24
0.17
0.26
0.07
0.47 = 0.24
0.85 + 0.16
Would_ use
0.36
0.41
0.30
0.17
0.22
0.11
0.43 # 0.24
0.80 + 0.18
Table 4: Average Standard Deviation and Deviation of average scores for human study results. Ignoring outliers (center 3 scores) similar to the median, the overall agreement between humans is high with oy 0.1 from di ~ 0.25 overall. In the average over 5-score-means (mean of means), we find that LLM-based anonymization is significantly preferred by humans.
F,5 COMPARISON BLS-BASED UTILITY SCORING
We further evaluate how closely aligned the reported median human scores are to the scores given by our LLM-based utility judge used in other experiments.
We show the main results for preservation of meaning and readability in Fig. 16a and Fig. 16b, respectively. For meaning, we make two key observations: First, we note that for adversarialyy anonymized texts, human and L.-.-based judgment is well aligned with a median absolute difference of 0.13. However, humans noticeably and consistently rated Azure anonymized texts lower than the LLM-based utility model. This means that for practical purposes, the gap between adversarial anonymization and Azure is even wider than what we have shown in our main results, making a strong case for the usage of LLM--nonymizee texts in settings where human usage is intended.
We can observe similar behavior for readability. Here the LLM utility judge generally rates both adversarially anonymized well as Azure anonymized texts slightly higher than human perception. However, we find that the mean readability score for Azure anonymized text is 0.19 higher under the ILMM-uility model, while is only 0.1 higher for adversarially anonymized texts. Again, this means that the numbers shown in Fig. 3a favor Azure anonymization and that the corresponding real-world utility gap is even more significant than what we reported.
We see our human experiments as strong indicator that (i) there is a clear and strong human preference for our adversarially anonymized texts over any existing techniques and (ii) while the model utilities presented in Fig. 3a are overall well-aligned with numan--prccpppa perceived utility, especially for Azure, is noticeably lower, further making a strong case for adversarial anonymization in cases where downstream utility is relevant.
the human-
F.6
IRB- IRB-REVIEW
The human study was reviewed by the IRB board of the authors' institution, and permission was granted conduct it in the format described in App. F.1. The IRB classified the human study as minimal risk'. All participating parties gave written consent for their answers to be recorded and could stop the study any time. No data other than participants' answers were recorded part of the study. Participants were compensated with an hourly wage of at least 20 USD (however, they carned at least 20 USD even if they finished faster). All participants were at least 18 years of age. As all presented comments were in English, the only other qualifying selection criteria were being literate and being a native English speaker living the UK, USA, Australia. Ireland, New Zealand, or Canada. The study was conducted online and anonymously using the external Clickworker platform.
G ABLATING CERTAINTY
In this section, we present a more detailed ablation of the certainty estimation of the adversarial inference and, in particular, low certainty predictions and its relation accuracy and bias.
G.1 ACCOUNTING FOR LOW CERTAINTY PREDICTIONS
35

---

Published as a conference paper at ICLR 2025
1.0
One key observation from Sec. 5 besides that adversarial anonymization outperforms existing anonymization, is that the resulting adversar- ial inference accuracy, even after iterations, at around 41%. While this initially seems surprisingly large, we actually find that a large majority of these predictions are made with very
0.8
Distribution
0.6
Aronn
Level
low (s 2) certainty by the adversary but happen be correct (e.g., in predicting the sex of per- son, random baseline already achieves 50% expectation). We provide several qualitative examples of such texts well the respective low certainty inferences in App. D.4. Based on this insight, further investigated this for our strongest anonymizer, GP.-4-AA
0.4
0.2
0.0
Certainty
As show in Fig. 17, find that for GPT- 4-AA-5 over 80% of predictions are made with certainty I lacking proper factual basis for the inference in the text. As we can see in the exam- ples in App. D.4, these level 1 and 2 certainty predictions in practice introduce a notion of "plausible deniability,"" where it is generally not possible to infer attribute certainly. This directly influences the adversary's accuracy. In GPT-4-AA-5 for the attribute SEX, we, for example, find that only II of 216 predictions have certainty 2 3. All these predictions have an accuracy of 100%. This drops rapidly for predictions with certainty 2 ( 68..7%% accuracy) and even further for certainty 1 (66%) accuracy. If we only account for predictions made with certainty 3 or higher (as we do for human predictions the ground truth data), the resulting adversarial accuracy across PersonalReddii (across all attributes) drops down to only 7.7% (For 2 we get 19%). This reinforces the point highlighted in Fig. 5, which emphasizes that adversarial anonymization provides not only lower adversarial accuracy but also much lower certainty in the predictions that are still made (which can translate to real-world privacy gain).
Figure 17: Distribution of certainty in adversarial predictions for GPT-4-AA across all iterations. We observe how GPT-4-AA Iteration 5 over 80% of predictions have certainty S 2.
G.2 FINDING CUTOFFS BASED ON CERTAINTY
One possible idea decrease the cost of running multiple iterations would be define certainty- based threshold such that we stop the iteration once we fall under it. In particular, from Fig. 18, where show the relative adversarial accuracy for each certainty, we can see on GPT-4-AA that this cutoff point (across rounds and attributes) appears to be around the certainty of 2, with only marginal benefits in running more iterations. In practice, we find that this already quite often happens after 1 or 2 rounds (based on Fig. 17, 50% of predictions have certainty S 2 after single round), allowing in many of these cases to stop early (not only resulting in lower costs but also higher utility). One point that is particularly helpful here is that, in practice, the anonymizing party can often assume knowledge of the ground truth attribute values. As such, can directly check against the current inference to test whether an attribute is inferable and stop once the certainty low enough or actually wrong. We believe that the further development of such automated criteria is very interesting avenue for future development.
Accurscy over Corraity GFT4
0.6
0.4
nortanty
Figure 18: Average accuracy of predictions based on the respective certainty level. We observe that the accuracy predictions increases strongly with certainty, indicating potential cutoff around 2.
36

---

Published as a conference paper at ICLR 2025
G.3 FAILURE CASES OF ADVERSARIAL ANONYMIZATION
Even considering the exclusion of bass- inferences, we find that there are instances where, even after 5 rounds of anonymization, the adversary can confidently infer the attributes from the text, raising the question of why adversarial anonymization fails in these cases.
In order to examine these failure cases further, we manually reviewed all cases in the PersonalReddit dataset, where the adversary was able to make certain prediction (2 4) after the third iteration of GPT-4-AA. Interestingly, we find that these failures (with the exception of a single case in OCCP) are restricted to only three attributes: Education (EDU- 5 cases), Relationship Status (REL- - cases), and Sex (SEX- cases). Across all these cases, we find that the core message of the text closely related to the respective attribute, e.g.. (for SEX) the personal experience of bearing child or using gender-specife with dating (apps) that indicate their current relationship status as single and (for EDU) mostly very college specific topics such as struggle with course load/selection. In all of the cases, a significant part of the utility of the text is given by the exposure of the personal attribute, which hard to handle with (adversarial) anonymization. While, e.g., more concrete references to universities have been removed from all texts, the overall nature of the text, which is about the life of college student, was retained. In these cases, while adversarial anonymization provides some level of protection and certainly awareness for the user, it remains inherently unclear how one would (even as a human) anonymize the text without losing significant parts of its utility.
contraception (for REL) almost exclusively recent stories about experiences
H ADDITIONAL DATASETS, DOMAINS, AND METRICS
In this section, we provide additional details on the applicability of adversarial anonymization to other datasets and domains, as well as the usage of other downstream metrics measure of utility.
H.1 MEDQA
In this section, we evaluate adversarial anonymization on the MedQA dataset (Jin al., 2021). In particular, we used the 1000 sample test set in English. MedQA contains Medical Board Exam Questions in QA style, requiring the model to select the correct answer given a list of options. Notably, MedQA this not only includes a change the domain but also allows for the evaluation of the utility via a downstream classifier.
Anon.
Base
Azure
AA Iter.1
Iter.2
Utility [%]
85.4
81.5
81.4
77.6
Adv. Acc. [%]
64.1
63.1
52.3
Table 5: Utility and adversarial Accuracy of anonymiza- tions MedQA using Azure and GPT-4o-AA.
While the focus of adversarial anonymization lies on more free-form text anonymization, it is still valuable to investigate its applicability in other domains. To enable this on MedQA, we, the first step, let GPT-4o infer all attributes (from our full set of attributes), keeping only predictions with certainty 2 3. This is representative of the underlying attributes as MedQA medical descriptions contain almost all personal data attributes explicitly ("E.g.. 27-ycar-old man was put into hospital"). We then anonymized the reports using both Azure and GPT-40-AA, presenting the results in Table 5. In particular, we measure the utility via the downstream accuracy of a model (separate instance of GPT-40) at answering questions while keeping the adversarial inference evaluation as in the main paper. We see in Table 5 that we have a baseline utility of 85.4% without any anonymization. As shown in Fig. 19, running single round of GPT-40-AA reduces the accuracy of location, age, and occupation by around 50%. the one for Place of Birth even by 79%. We see a smaller but still noticeable drop of 27% for SEX, which intuitively makes sense as it is quite often linked to specific medical conditions. This comes at the cost of 4% loss in utility, dropping us to 81.4% downstream accuracy. Further iterations drop adversarial even further (for most attributes > 50%) at the cost of another 4% in utility. Overall, this a strong indicator that adversarial anonymization leads to highly utility-peeservng anonymizations also in other domains. Further, when comparing it to Azure, which is more suited for this domain as, e.g., all age ranges are denoted neatly at the beginning of each text, we find that the utility is comparable with Azure, which achieves 81.5% downstream accuracy while protecting several attributes (in particular SEX), significantly less. We see this as a strong indicator that adversarial anonymization is also applicable across wider range of domains.
37

---

Published as a conference paper at ICLR 2025
H.2 EMBEDDING SIMILARITY
Furthermore, bring more insight into the utility retention of arbitrary downstream models on free-form online text, we com--
Numboe et Comeett Preed deeddiiom
puted text embeddings using OpenAl's text- eexxtembbeding--e 3- large model. We do this
3.00
across all comments for the first three iterations of 1-mma..--70BAA.. as well as the original comments. Based on this, we can compute their cosine similarity as a strong indicator of their likeness and thereby implied utility retention on any downstream task that could inputs. In particular, use such embeddings we find that the median cosine similarity is 0.93 after one round of anonymization, 0.88 after two, and 0.84 after three. As baseline, we randomly sampled comments from the original PersonalReddit dataset and computed their similarity, resulting median of 0.23. This further strengthens the claim that the texts created by adversarial anonymization retain relatively high utility overall (even when applied to potential downstream tasks).
AGE
EDU
POBP
Attribute
Figure 19: Adversarial inferences on MedQA for both Azure and GPT-4o-AA. MedQA is much more suitable for Azure, as it contains explicit attributes. However GPT-4o-AA still provides strong utility- utility-preserving anonymization, especially on harder to anonymize attributes.
H.3 DISCUSSION ON DOMAIN APPLICABILITY
With this in mind, we want to briefly discuss which domains see adversarial anonymization have a particularly strong impact. As outlined in the main paper, we particularly target setting in which attribute information is not always clearly denoted in text and, in many cases, can only be inferred. We believe that such domains (as online communication, forums, etc.) are plentiful and, as we have shown, lack any existing tools for providing appropriate privacy protection. In particular, we believe that classical anonymization with, e.g., Azure can be adequate if the text data is in structured, much more rigid format for which we can ensure a consistent representation of attributes. Notably, even legal documents, such as the cases from the Human Rights Court (Pilán et al., 2022), generally do not have a strong enough structure to ensure this, leading classical anonymizers to achieve low scores in PII detection with LLMs often outperforming them (Bubeck et al., 2023). In the many cases in which text does not or cannot have a very clear and constrained structure, we believe adversarial anonymization provides a valuable contribution and first step towards better anonymization that is also directly aligned with regulatory formulations.
I TRAINING MODELS FOR ADVERSARIAL ANONYMIZATION
Below, we investigate whether we can potentially distill the inference and anonymization capabilities of larger models into smaller models, thus making adversarial anonymization more cost-efficient and potentially even deployable on edge devices. In this section, we show the first empirical results that indicate that this might be possible, making the training of such models an interesting direction for future work.
As the Llama models were the most promising OSS models, we decided to finetune version of Llama-3.1-84 1-8B based on demonstrations of more capable models. In particular, we finetuned a 4-bit quantized version of Llama-3.1 using (r=256 LoRA adapters, three epochs, using unsloth (Han Han, 2024)) as a new inference model (keeping the original 8-bit Llama-3.1 as repair model).
For this, we only used synthetic demonstrations from Llama3.1 1-70B on SynthaPAI and our synthetic samples (in particular from the first two anonymization rounds). Interestingly, during experimentation. we observed that demonstrations from Llama3.11700 1-BBB resulted in a better finetuned model than demon- strations from GPT-4. We speculate that this is due to the similar alignment of the Llama3.1 models. Then, after training on the synthetic datasets, we evaluate the model's (uns-mixed mixed) performance on unseen real-world data. As we in Fig. 20 on the real-world PersonalReddit dataset, the resulting
38

---

Published as a conference paper at ICLR 2025
uns mixed model performs much closer to Llama3.1-70B while requiring fraction of the memory and compute.
Interestingly, observed that finetuning the repair model (for now) results in an overly con-
(aamall Ubmal.l -B------AA
servative model, which achieves great utility but essentially gets stuck at the level of Llama3.1-
Han (Comb))
70B-1 (i.e.,a single Llama3. anonymiza- tion round). We believe training such models also for anonymization will be an interesting avenue for future work eespeially considering that one can apply more sophisticated model distillation methods than supervised finetuning. Overall, these results make us very hopeful for future, smaller and cdge-deployable models that enable high-quality adversarial anonymization.
0.90
0.65
0.00
0.5
0.8
Adversaral Accuracy
Figure 20: Utilit-Privaccy of the finetuned Llama- 3. 3.1-8B-UNS-Mixed model on the PersonalRed- dit dataset. Despite being quantized 8B model, it performs very close to Llama3. strictly dominating the non-finetuned -UNS model in the privacy-utility tradeoff.
39

---

Published as a conference paper at ICLR 2025
Mcricd
Adae
Adra
CFF-AA CIT- BYr
OPTAAA
OPE-AA to1
15
Coans
RE00C00 MM:20
(a) Readability Mean Scale 1-5
(b) Readability -Mddian Scale 1-5
Hethos
(21 AA ls
3.0
2.5
3
0.0
Mean
-eddan
(c) Meaning Mean Scale 1-5
(d) Meaning Median Scale 1-5
Mcchod
Midthd
Adure
Acure
Dount 20
0.2
(c) Typical Mean Scale 0-1
(f) Typical -Mdiaa Scale 0-1
Hethos
00
GA BI
Cart
32
0.0
2.5
1.0
0.0
02
3.0
(g) Useable- Mean Scale 0-1
(h) Useable MMdian Scale 0-1
Figure 15: Aggregated human study results for readability, meaning, typicality, and usability of
anonymized texts. GPT-4-AA outperforms Azure noticeably across all iterations and metries.
40

---

Published as a conference paper at ICLR 2025
LLM Meaning Adv. Anon.
Meaning
300
LLM Keaning Azure
Meaning
2.5
Density
1.5
1.0
0.0
Values
(a) Score distributions for Human- and LLM-based based judgement of preserved meaning. We notice that humans and LLMs agree well for aoM-nnnnmizeed texts. For Azure anonymizations, LLMs tend to rate the anonymizations more meaning preserving than humans.
Readability Anon.
Human Readabiltty Adv. Anon. Readabilit
35
Human Readabiity - Azure
3.0
2.5
Density
2.0
0.
-0.2
0.0
0.
0.4
0.6
0.8
1.0
12
Values
(b) Score distributions for Human- and I.L.-basee based judgement of readability. We notice that humans and LLMs agree well for LLMM-anonymized texts. For Azure and adversarial anonymizations, LLMs tend to rate the anonymizations as more readable than humans, however the average distance is higher for Azure.
Figure 16: Comparison of Human- and LLM--bsed scores for both meaning and readability.
41


Reviews:
[{"review_id": "b5O2jauYhS", "reviewer": "Reviewer_CnVX", "summary": "This paper introduces a novel approach to text anonymization in the era of large language models. The authors present two main contributions: (1) a new evaluation framework that leverages LLMs for adversarial inference to measure anonymization effectiveness, and (2) an iterative anonymization pipeline that uses adversarial feedback to guide the text anonymization process. This framework offers improvement over the traditional span based formulation as contextual information leaks information as well. The authors conduct extensive experiments with various models and demonstrate that their approach achieves better privacy-utility tradeoffs compared to traditional span-based anonymization techniques such Azure Language Services. In their results, performing the procedure reduces the adversarial inference chance from 87% to 66%, and iterating the procedure with GPT-4 for three rounds further reduces adversarial inference success to ~45% while maintaining higher text utility than baseline methods. They validate their results with human annotation.", "questions": "- Have you explored methods to automatically determine the optimal number of iterations, perhaps based on inference confidence?\n- How does the system perform when encountering privacy-sensitive attributes not explicitly listed in the input? Could it be extended to automatically identify such attributes?\n- Can you provide specific examples of common failure cases where privacy leaks persist even after multiple rounds of anonymization? Understanding these patterns could help improve the approach.\n- Have you considered how this approach might need to be adapted for different domains with varying privacy requirements and linguistic patterns?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel approach that leverages LLMs' inference capabilities to measure privacy leakage in a more realistic way than traditional span-based methods.\n- Comprehensive experimental evaluation across multiple models, attributes, and metrics, with clear ablation studies showing the benefit of the feedback-guided approach.\n- Strong empirical results showing significant improvements over industry-standard tools like Azure Language Service, with detailed analysis of both privacy protection and utility preservation.\n- Thoughtful consideration of practical concerns including computational costs, local deployment options, and regulatory compliance.\n- Clear demonstration of how multiple rounds of anonymization can progressively improve privacy while maintaining readable text.", "weaknesses": "- The ~41% remaining adversarial inference success rate after anonymization remains concerning for privacy-critical applications. The paper would benefit from deeper analysis of these failure cases.\n- Limited domain evaluation, focusing primarily on data directly or indirectly from Reddit. Testing on other domains (medical, legal, etc.) would strengthen generalizability claims. It could also be reddit comments that mentions information from these domains. \n- The method requires pre-defining attributes to protect, which may miss unexpected privacy leaks. An automated approach to identifying sensitive attributes could be valuable.\n- Cost analysis could be more comprehensive - while per-comment costs are reasonable (\n(~$0.035), real-world applications with high volume could face significant expenses. In addition, the estimate is only for one turn, so to achieve the same level of privacy protection, this number might be more expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXjR4AffGf", "reviewer": "Reviewer_EPkh", "summary": "In this work, the authors focus on the privacy scenario where online texts can be exploited to infer personal data. The authors utilize adversarial LLM inferences, which are highly performant in extracting personal attributes from unprotected texts, for evaluating anonymization and also use this adversarial model as \"feedback provider\" to another LLM whose goal is to anonymize texts. An iterative framework between these two LLMs lead to strong anonymization performance as shown by the authors in wide range of experiments, outperforming existing anonymizers and also aligning well with human preference.", "questions": "1. Have you considered measuring utility by some downstream applications? E.g. if the texts are used for some analysis or for some task, how the performance changes from the original unprotected texts to the anonymized texts. Would you think this could also serve for useful utility metrics?\n\n2. How can one turn this approach into a more comprehensive privacy-protecting tool? To my understanding, it currently builds on pre-defined set of attributes and the adversary LLM is trying to infer these attributes while the anonymizer LLM is trying to anonymize as oppose.  But it'd be hard to list all possible attributes that could lead to deducing personal information so any comments on scaling this approach would be appreciated.\n\n3. Also related to my question above, formal privacy guaranteeing mechanisms like differential privacy ensures that even the existence of the data cannot be inferred from the analysis by any adversary. Although in this work the authors focus on anonymizing individual text snippets so that DP may not be applicable, however, it'd be interesting to find a common scenario where two approaches can be compared I think.\n \nMinor: AzureLanguageService -> Azure Language Service", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The presentation is very clear and the paper flows very well. Text anonymization is an important problem in the realm of privacy and the approach the authors introduce do improve the existing anonymization tools significantly. The evaluation section has extensive analysis, which is great. The reviewer really enjoyed reading this paper overall.", "weaknesses": "In my opinion, the paper is very well written and the authors conducted extensive empirical studies to demonstrate the significant improvement of their approach compared to the existing text anonymization tools. I think my main concern is the scope and complexity of the approach appear quite limited, especially for a conference of this caliber. The approach is based on iterating two LLMs, one is anonymizing text and the other is trying to infer personal attributes. To me, this is like a cute application of LLMs but perhaps rather better suited for a workshop instead of this conference. In this sense, I am unsure about the fit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HfBC6bT1qY", "reviewer": "Reviewer_18jd", "summary": "This paper proposes a LLM-based adversarial anonymization framework to address privacy risks. The authors use a feedback-guided approach where an LLM adversary attempts to infer personal attributes from a given text, and an anonymizer LLM iteratively modifies the text to reduce inference risks. The paper evaluates this method against traditional anonymization techniques and demonstrates superior performance in both preserving utility and privacy across several datasets.", "questions": "1. Could you provide more examples where the framework fails and explain why the LLM is unable to recognize these instances?\n2. Do you think it’s feasible to distill this capability into a smaller model? in this way, we can reduce the computational cost.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed adversarial anonymization framework leverages the strengths of LLMs both as adversaries and anonymizers, showcasing a new application of LLMs in a privacy-preserving context.\n2. The inclusion of a human study adds value to the evaluation by confirming the practical applicability of the framework and showing a preference for the LLM-anonymized text.", "weaknesses": "While this might be an interesting application of LLMs to the field of anonymization, the core methodology introduces neither fundamentally new anonymization techniques nor a different way to use LLMs. It merely adapts existing concepts by leveraging the powers of LLMs. Thus, the contribution of novelty is limited for either the LLM or the privacy community.\n\nOne major limitation is that, in real life, texts to anonymize are normally very long, and due to the current method, which would be prohibitively expensive, it is practically not feasible in applications that demand real-time processing. The scalability and applicability of this framework are rather limited due to the enormous amount of documents it may need to work with iteratively. Another limitation is that privacy performance remains unpredictable due to heavy dependence on the capabilities of the LLM. This creates a dependency where consistency of anonymization outcome cannot be guaranteed and may even differ from model to model or update to update.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)