<figure><img src="image_1.png" /></figure>

HAL
open science

# Towards the Anonymization of the Language Modeling

Juliette Sénéchal, Antoine Boutet, Lucas Magnana, Hélain Zimmermann

► To cite this version:

Juliette Sénéchal, Antoine Boutet, Lucas Magnana, Hélain Zimmermann. Towards the Anonymization of the Language Modeling. 2025. hal-05223726

HAL Id: hal-05223726

https://lilloa.hal.science/hal-05223726v1

Preprint submitted on 26 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Towards the Anonymization of the Language Modeling

Antoine Boutet

INSA Lyon, Inria, CITI, UR3720
69621 Villeurbanne, France
antoine.boutet@insa-lyon.fr

Juliette Sénéchal

Université de Lille
Lille, France
juliette.senechal@univ-lille.fr

Lucas Magnana

Inria, INSA Lyon, CITI, UR3720
69621 Villeurbanne, France
lucas.magnana@inria.fr

Hélain Zimmermann

Ensimag
38400 Saint-Martin-d'Hères, France
helain.zimmermann@grenoble-inp.org

## ABSTRACT

Rapid advances in Natural Language Processing (NLP) have revolu-
tionized many fields, including healthcare. However, these advances
raise significant privacy concerns, especially when pre-trained
models fine-tuned and specialized on sensitive data can memo-
rize and then expose and regurgitate personal information. This
paper presents a privacy-preserving language modeling approach
to address the problem of language models anonymization, and
thus promote their sharing. Specifically, we propose both a Masking
Language Modeling (MLM) methodology to specialize a BERT-like
language model, and a Causal Language Modeling (CLM) methodol-
ogy to specialize a GPT-like model that avoids the model from mem-
orizing direct and indirect identifying information present in the
training data. We have comprehensively evaluated our approaches
using a medical dataset and compared them against different base-
lines. Our results indicate that by avoiding memorizing both direct
and indirect identifiers during model specialization, our masking
and causal language modeling schemes offer a good tradeoff for
maintaining high privacy while retaining high utility.

## KEYWORDS

Language Models, Privacy, NLP, LLM, Anonymization

# 1 INTRODUCTION

The healthcare sector is a major generator of sensitive data col-
lected from different sources. The exploitation of this valuable data
presents many promises such as improving the quality of care, ac-
quiring a better knowledge of the healthcare system, identifying
disease risk factors, aiding in diagnosis, personalized care to name a
few. A large amount of this data is unstructured text documents in
the form of medical reports. With the rise of Machine Learning (ML)
and the advent of Natural Language Processing (NLP), language
models are increasingly used to automate the processing of these
medical reports [12, 22, 56, 61].

Patient medical records are extremely sensitive and private data. Their use and dissemination are therefore subject to numerous regulations such as HIPAA (Health Insurance Portability and Accountability Act), for the USA, or GDPR (General Data Protection Regulation) for Europe. In these regulations, one of the main prerequisites for the dissemination of medical data is to remove any element that allows a patient to be directly identified (i.e., de-identification or pseudonymization) or indirectly (i.e., anonymization). Recent advances in NLP [45] based on neural networks have democratized

their use. Since the advent of ChatGPT, NLP models are not limited
to text generation and can include several tasks, such as classifi-
cation tasks or Named Entity Recognition (NER) which therefore
allows for de-identification of free texts if it is trained to recognize
identifiers. For example, Johnson et al. proposed to use a neural
network based on a BERT architecture [20] to detect a number of
identifying elements in medical documents. More recently, different
hospitals have also explored the feasibility of using NLP models to
automatically pseudonymize text documents, means hiding specific
direct identifiers named Personally Identifiable Information (PII)
from their clinical data warehouse [40, 50]. The set of PII to con-
sider is recommended by Data Protection Authorities (DPA) <fnref n="1" /> and
includes names, hospitals, addresses, birth dates, patient numbers
for instance.

Although a large number of works only exploit NER to pseudonymize
documents [47], pseudonymization is not sufficient for text anonymiza-
tion [38]. Indeed, a link between an individual and their data can
be found from indirect identifying information. Therefore, these
indirect identifiers must also be protected (e.g., removed) in order
to better anonymize texts. This operation cannot rely on a classifi-
cation or NER task as there is no a priori information to identify
these indirect identifiers, it depends on the context and the dataset
considered. To our knowledge, no NLP model training methodology
has been proposed for taking into account indirectly re-identifying
information in the protection of medical reports or documents from
other domains.

The need for information sharing in the medical field goes be-
yond data sharing and now concerns learning models. As language
models specialized on clinical reports improves their performance
for downstream clinical NLP tasks [17, 26, 28], some hospitals or
healthcare centers also want to be able to share pre-trained models
specialized and fine-tuned on their own data with other medical
centers. For example, a hospital specialized in oncology may train a
classification model with its own data and would like to share this
model with other hospitals. In this context, the model specialized
with the medical reports of the first hospital must not be able to re-
gurgitate sensitive data used during its specialization (or return any
other information allowing to deduce them) when it will be used by
the second hospital. The attack surface on models trained on per-
sonal and highly sensitive data is still poorly understood [16, 29, 57].

<footnote n="1">¹For instance, in Section 5.2.1.1 of the reference document relating to the processing of personal data implemented for the purpose of creating data warehouses in the health sector from the French DPA (i.e., CNIL): https://www.cnil.fr/sites/cnil/files/atoms/files/referentiel_entrepot.pdf</footnote>
A number of threats are related to the memorization and possible
leakage of sensitive information used during model training. There
are some known privacy vulnerabilities involving training data
associated with learning models, such as memorization, data re-
construction, and membership inference (i.e., identifying elements
used during the training or the fine-tuning).

Memorization of information by a model is not a problem in
itself, it is even necessary for learning. However, this memorization
becomes a problem when the training information is not gener-
alized enough by the model which reproduces large portions of
training data verbatim or discloses some sensitive information. And
more specifically, the risk to privacy is increased when this prob-
lematic memorization concerns identifying information. Indeed,
memorizing identifiers that leads to the regurgitation of these iden-
tifiers can clearly be used to infer that the information of a specific
individual was used during the training of the model (i.e., mem-
bership inference). Thus, in order to share a model without risk
of regurgitation of personal information, it is necessary to ensure
that a model has not memorized both direct and indirect identifiers.
However, as far as we know, no methodology has been proposed
to help model designers, especially hospitals, to "anonymize" their
learning models before sharing them.

As part of investigations into the training of AI models, the Irish
DPA has asked the European Data Protection Board (EDPB) to issue
an opinion on the development and deployment of AI models <fnref n="2" />
in light of the GDPR. More precisely, the opinion addresses the
issues of AI models trained with personal data, regardless if the
model is designed to provide information about individuals whose
data is included in their training sets or not. The opinion therefore
considers that an AI model trained on personal data cannot be con-
sidered anonymous in all cases, and that in most cases, an analysis
will have to be conducted by the data controller to demonstrate
that such a model is anonymous. To be considered anonymous,
the likelihood of regurgitation of personal information must be
insignificant taking into account all the means reasonably likely to
be used.

To reduce privacy risks, mitigation techniques have been pro-
posed such as learning with Differential Privacy (DP) guarantees [2]
or pruning strategy [55]. While these mitigation techniques in-
crease the probability of the adversary to make a wrong decision
or inference, these countermeasures also significantly degrade the
accuracy of the model, making them difficult to use in practice.
Finally, these techniques are data agnostic and do not treat identify-
ing data differently from other data. DP was originally designed as a
countermeasure to membership inference attacks (MIAs). MIAs aim
to predict whether a particular record belongs to a given model's
training dataset. This analysis addresses not only model privacy
auditing but also copyright infringement detection. The underly-
ing concept of this attack relies on the fact that a model behaves
differently depending on whether it is processing data that it has
already seen during training or not. Most of the state-of-the-art
attacks attempt to capture this change in behavior and require

training multiple models, making the attack very expensive. Fur-
thermore, the performance of existing MIAs on LLMs is more and
less conclusive [15, 19, 33].

In this paper, we propose privacy-by-design language model-
ing approaches to address the problem of anonymizing language
models. Specifically, we propose both a masking language mod-
eling (MLM) methodology to specialize a pre-trained BERT-like
language model, and a causal language modeling (CLM) method-
ology to specialize a pre-trained GPT-like model that avoids the
model from memorizing direct and indirect identifying information.
To achieve this goal, our fine-tuning approaches named PPMLM-
BERT and PPCLM-GPT (for a BERT and GPT model, respectively),
first identify directly and indirectly identifying information in the
data corpus used for the specialization. While direct identifiers are
pinpointed by exploiting a off-the-shelf NER model, indirect identi-
fication words are identified as words used only by a single patient
(which goes beyond the directly identifying words). This ensures
that the memorized words were used in the data of at least two
individuals (i.e., thus making the k=2 individuals indistinguishable,
note that the value of k can be parameterized in our solution). Then,
the language modeling to specialize and fine-tune the model on the
training data (i.e., the random masking or the next-word prediction
for BERT and GPT models, respectively) avoids using directly and
indirectly identifying words. Therefore, the risk of regurgitation or
inference of personal information during the use of the model is
drastically reduced.

We comprehensively evaluated our approaches using medical
reports and considering a wide range of comparison baselines (e.g.,
the use of pseudonymized data for training, DP). We quantified
the impact of our solution on the resulting model's performance
using multiple utility metrics (e.g., the ability to predict words and
tokens, training a downstream classification task), as well as on
privacy metrics based on the model's ability to predict identifiers or
through MIAs. We show that ignoring direct and indirect identifiers
in the language modeling improves privacy while maintaining good
utility. To enable reproducibility of results and appropriation of the
solution, code and public data as well as other associated research
artifacts are available<fnref n="3" />.

To summarize our contribution, this work addresses the too-
often-seen limitation that relying on NER models to detect identi-
fying information and simply removing it from the training data is
sufficient to adequately protect privacy. We believe that recognizing
this limitation and proposing a solution could be useful in practice
to prevent models supposedly without personal information (i.e.,
who have only exploited a NER) from being freely exchanged on
Hugging Face.

The paper is organized as follows. Section 2 reviews the back-
ground and related work while Section 3 defines the threat model.
Section 4 describes our privacy-preserving language modeling for
both descriptive and generative models. Section 5 then presents
the experimental setup while the evaluation results are reported
in Section 6. Finally, we discuss the limitations and future work
Section 7 before concluding Section 8.

<footnote n="2">²https://www.edpb.europa.eu/news/news/2024/edpb-opinion-ai-models-gdpr-principles-support-responsible-ai_en</footnote>

<footnote n="3">³Code for reproducibility: anonymized for submission</footnote>
## 2 BACKGROUND AND RELATED WORK

This section presents a comprehensive background and related work on NLP (Section 2.1), how NLP models can be used to improve privacy (Section 2.2), the privacy leakages associated to the memorization of the model (Section 2.3), and mitigation strategies (Section 2.4).

### 2.1 Natural Language Processing

Natural language processing (NLP) is the process of understanding and processing textual data using machine learning (ML) models. The field experienced a breakthrough in 2017 with the advent of the Transformer [54]. This new architecture revolutionized translation at the time. It consists of an encoder-decoder neural network with a parallel computational scheme that uses positional encoding and various attention mechanisms [5]. The goal of the encoder is to embed (i.e., transform into vectors) the input sentences. Each word is embedded in a latent space, taking the entire sentence as context. The decoder, in turn, learns to translate these latent vectors into new sentences. Each token (words or subwords) is predicted sequentially, paying attention to the input and the previously predicted words.

Two trends followed in 2018 with BERT models [14] that focus on the encoder part of the Transformer and GPT models that use the decoder. The former are very effective for classification tasks. They are pre-trained on huge unlabeled datasets to learn very complex word embeddings. It is then possible to fine-tune such models on specific data and even learn new tasks by adding just a few layers to the model. For example, a hospital could train a BERT model to classify medical records according to different pathologies. GPT models on the other hand are generative models. They consist of a very large language model (LLM) that has learned to imitate human expression. The output sentences are the most likely answers according to the model, based on probabilities that it has learned by observing sentences in its huge training dataset. They can be used to create chatbots such as chatGPT, which is derived from a GPT-4 base. OpenAI then extended its training with supervised and reinforcement phases to ensure that responses are non-toxic and do not include fake news [1, 21].

Since then, many new models inspired by Transformer have emerged. For example, RoBERTa [31] which is a much larger version of BERT, or distilBERT [41] which uses knowledge distillation to produce a smaller model. Models in other languages have also emerged, such as CamemBERT [32] for French. This model specialization on a language or on a specific data corpus such as ClinicalBERT [37] or DrBERT [28] with clinical reports is based on language modeling. Language modeling consists of predicting the words in a document (Figure 1). On the one hand, causal language modeling (CLM) consists of predicting words sequentially, from left to right (Figure 1(b)). This is the most common approach used in text generation with a GPT-type model. On the other hand, Masked Language Modeling (MLM) is when the prediction task is done anywhere in the text and in any order, in other words chosen randomly (Figure 1(a)). This is the approach mainly used for language understanding tasks with a BERT-type model. In both cases, the prediction is based on the context. This means that the words before the mask (in the CLM case) or the words around the mask (in the MLM case) allow the model to identify what the word

<figure><img src="image_2.png" /><figcaption>**Figure 1:** Masked language modeling (MLM) involves predicting words anywhere in the text and in any order, while causal language modeling (CLM) involves predicting words sequentially, from left to right.</figcaption></figure>

to fill could be. This language modeling step allows a model to be specialized on a specific corpus. For example, it has been shown that specializing on clinical reports improves the model's performance for downstream clinical NLP tasks [17, 26, 28].

### 2.2 Leveraging NLP for Privacy

The prospects for improving privacy offered by new NLP models are numerous, both on human-generated textual content (e.g., free text, source code) and on speech after transcription from voice to text. For example, [23] uses NLP models to analyze privacy-related feedback or [3] analyzes emergency calls to correctly assess and predict the severity of the situation. These models could also be used to analyze source code and detect security or personal data leaks [35, 46], or to analyze speech and detect sensitive elements in order to remove them from recordings [36].

In order to exploit or share their patients' information for research purposes while ensuring patient confidentiality, hospitals have begun to exploit language models to de-identify clinical reports [24, 40, 50, 51, 59]. De-identification of text-based clinical reports involves removing or replacing Personally Identifiable Information (PII) from electronic health records. Although strict anonymization is considered a very difficult task, this pseudonymization (e.g., identifying and replacing PII with a plausible substitute) makes it difficult to re-establish a link between the patient and their data and is considered sufficient protection for research purposes. Personally identifiable information is extracted from a list that may vary from country to country and includes, for example, address, date, date of birth, hospital, patient ID, email, visit ID, first name, last name, telephone, city, zip code and social security number.

The common pipeline of NLP models to perform de-identification of clinical reports is similar to the one reported in Figure 2 which describes the workflow adopted by Hospices Civils de Lyon (HCL) [40]. The base model (A) is a CamemBERT [32] which is a BERT [14] model specialized on French texts. This specialization of the model towards French is done via a filling-masking task (i.e. random holes are added to the sentences in which the model learns to fill them). Following the same specialization process via a filling-masking task, this CamemBERT model is then refined on medical texts to have a better statistical understanding of the language used in medical
<figure><img src="image_3.png" /><figcaption>Figure 2: Workflow used by the Hospices Civils de Lyon (HCL) for de-identifying clinical reports.</figcaption></figure>

reports. Finally, model A' is refined to B' to detect personal iden-
tification tokens, through the training of a NER using a manually
labeled dataset.

The literature shows that the resulting models are able to out-
perform de-identification using only regular expressions and man-
ual rules to remove private tokens (i.e., up to 0.99 F1-Score with
NLP models versus 0.85 with manual rules only [50]). While these
models can represent a very effective clinical text de-identification
tool, only a few of these studies evaluate the utility loss due to
de-identification and none of them evaluate the privacy leakage of
NLP models due to memorization of training data [29]. This latter
attack surface is detailed in the following subsection.

## 2.3 Privacy Leaksages

Large language models (LLMs) are trained on very large datasets. For example, training chatGPT required years of crawling the Internet. Therefore, a lot of personal data such as people's addresses was used during training. BERT models, on the other hand, are typically fine-tuned for specific tasks with domain-oriented data. In the medical domain, datasets typically include sensitive patient records. In both cases, the problem is that the models can regurgitate and leak information from the training data after deployment [7], which is a significant privacy leak and has implications for the practical and legal aspects of such models [13].

A central question in this context concerns the extent to which
language models memorize their training data [7, 8, 34, 43, 65].
However, defining memorization for language models is challeng-
ing, and many existing definitions and notions have been proposed
depending on whether the memorization concerns copyrighted
content or personal and sensitive content. In relation to privacy,
we can notably cite extractable memorization (Section 2.3.1) and
membership inference (Section 2.3.2).

2.3.1 Extractable memorization. Extractable memorization is a type of attack that aims to use the model to infer information from the original data [30]. This attack mainly concerns text generation models, such as GPT. These models are trained to produce text based on what they have seen during training. However, the model is not expected to be a basic parrot and repeat exactly the sentences

it has seen. This is especially concerning if the data it is repeating
is sensitive. This has been shown to be the case with GPT-2 for
example, from which the names and addresses of individuals can
be extracted [9].

In [7], the term *k-extractability* is used to refer to the sequences
that can be extracted from the model when an input sequence of
length *k* is requested. The lower the *k*, the easier it is to extract the
sequence. We therefore expect a model to have the highest possible
*k* on private queries. This measure, however, does not capture re-
gurgitations that are not perfect, which can lead to an illusion of no
extractable memory. Compressible memorization [43] extends this
definition by evaluating how short the minimal requested sentence
(or prompt) that elicits the sequence.

2.3.2 Membership inference. Membership inference attack [6] (MIA) is a more common inference attack in machine learning, which aims to infer whether a specific data was used in the training data of a target model. This can be a problem, for example, if a hospital has trained a model to detect risk of cancer recurrence and you learn that your colleague's data was used in the training. You will have indirectly learned that she had cancer.

There are different techniques that can be used to perform a MIA
attack depending on if the adversary has an access to the model pa-
rameters (i.e., white-box access), or access to a ground-truth subset
of member and non-member samples. One technique consists to
analyze the loss of member and non member samples [62], another
one is to use multiple shadow models [44, 60] trained to mimic the
behavior of the target model on an auxiliary dataset. An adversarial
model is then trained to infer membership from the loss or from
shadow models. Another method [65] is based on comparing the
performance of the target model trained on a dataset with a specific
input, with a second model trained without it. As ML models are
supposed to learn general information, one piece of data (even rare,
outlier or mislabeled samples) is not supposed to be memorized
and significantly changed the model's performance. By repeating
this operation many times with different subset, it is possible to
identify counterfactually memorized data.

Although membership inference attacks have been used to quantify memorization risks [4, 15, 27, 33, 55], it may not be applicable in a practical scenario due to the high dimensional input space (i.e., too expensive in terms of calculation). By ignoring and not memorizing the direct and indirect identifiers of the training data during language modeling, our privacy-by-design scheme avoids membership inferences targeted on identifiers.

Being able to audit privacy at low-cost is important and chal-
lenging [64]. Indeed, using MIAs that require training multiple
models is intractable for LLMs. We measure the privacy level of our
solution through a low-cost MIA attack that relies on the analysis
of the model's predictions. More precisely, the prediction of a word
associated with an individual (i.e., a direct or indirect identifier) is
considered a privacy leakage, as it betrays the fact that the model
has seen this data during its training. This attack is consistent in
the medical use case with off-the-shelf foundation models that have
never seen the data used for language modeling.
## 2.4 Mitigation strategies

The most popular approach to mitigating privacy risks is Differen- tial Privacy [18] (DP). DP is a mathematical property that a model must satisfy in order to disclose as little information as possible. This property requires the model to learn a limited amount of infor- mation at each training step. More formally, the probability that a model guesses the correct output for a given input must not increase too much each time the model sees that data:

$$ \forall(x, y), \log P(M_D(x) = y) < \epsilon \log P(M_{D+x}(x) = y), $$

where x, y represent data and its label, M<sub>D</sub> a model trained on dataset D and ϵ the privacy budget. The lower ϵ is, the more private the model is. The most popular method to apply DP in machine learning is DP-SGD: Differentially-Private Stochastic Gradient Descent [2]. The idea is to apply DP during the training phase by clipping gradient updates and adding centered noise at each step. DP is known to significantly decrease the accuracy of the model [27] and privacy budget management is difficult.

To reduce privacy risks, Chen et al. [10] proposed to relax the loss in order to reduce the distinguishability between the training and testing loss distributions, and Ruizhe et al. [11] proposed a method to localize specific neurons responsible for memorizing PII in LLMs through adversarial training and then disable these neurons.

## 3 THREAT MODEL

Since the training data contained personal information, the adver- sary's goal is to identify the individuals whose information was used in training the model. To achieve this goal, the adversary takes advantage of direct and indirect identifying data from its auxiliary knowledge. Both direct and indirect identifiers are complementary. Direct identifiers include information associated with a natural person such as first and last name, a telephone or a social secu- rity number. In case of medical data, a list of direct identifiers to cosider has been recommended by the Data Protection Authority<fnref n="4" />. However, direct identifiers fail to detect all terms that can lead to distinguishing an individual. This distinguishing nature is cap- tured by indirect identifiers. Indirect identifiers is information that is uniquely used in an individual's data compared to the rest of the population. For example, if the term vertebra is used only in a patient's data, this term is indirectly identifying compared to the rest of the corpus. In other words, the individual is distinguishable from others with the use of this term. This distinguishable nature depends on the adversary's auxiliary knowledge. We consider an adversary which has collected information about individuals. This information can be user profiles from Web pages or mobile applica- tions, medical reports, or any other text-type information describing an individual. These texts can be plain or pseudonymized and come from web scraping or from a legally or illegally acquired dataset. In a pseudonymized dataset, directly identifying information will be removed, but indirect identifiers will still be present. The num- ber of direct and indirect identifiers will vary depending on the

<figure><img src="image_4.png" /><figcaption>Figure 3: Detection of the identifiers: 1) a off-the-shelf NER model can detect the direct identifiers while by constructing a bipartite graph between individuals and the words used in their documents, we can easily identify indirectly identifiers (i.e., words pointed to only by one individual).</figcaption></figure>

amount of information the attacker is able to retrieve on each in- dividual, as well as the similarity of the retrieved information and the information used during training of the target model.

While the direct identifiers can be detected by a off-the-shelf NER model <fnref n="5" />, indirect identifiers have to be detected. To achieve this goal, an analysis of the data corpus can be carried out. Figure 3 depicts a bipartite graph which relates the individuals represented in the data corpus to the words used across the entire dataset. More formally, it is a Bipartite Graph G = (V, E) where the vertices of this graph are the users and words of the dataset V = (U ∪ I). The use of a word in a user's data is represented in G by an edge (u, i) ∈ UI. In this example, we can see that the word w1 and w4 are only used in the data of individual u1. Note that a direct identifier can also be a word used only in an individual's data (i.e., an indirect identifier), this is the typical case of a last name.

These identifying data (direct and indirect) can pose a confi- dentiality problem if they are memorized by the model after the training phase. Indeed, an identifier memorized by the model and proposed (i.e., predicted in a masking phase) or regurgitated (i.e., proposed to complete a sequence) during the model's use would betray its use during training (i.e., it represents a membership in- ference). As illustrated in Figure 4, the model may be required to memorize contextual information and that used as a mask during the supervised training phase<fnref n="6" />. In this example, the fact that the model proposes the word w4 in the testing phase means that an adversary could infer that user u1's data was used during train- ing, because this word only appears in this individual's data in the attacker's auxiliary knowledge. Note that the indirect identifiers detected from the attacker's auxiliary information are most likely

<footnote n="4">⁴The list of direct identifiers recommended by the french DPA: https://www.cnil.fr/sites/cnil/files/atoms/files/referentiel_entrepot.pdf.</footnote>

<footnote n="5">⁵EDS-Pseudo trained from Clinical Data Warehouse for instance - https://github.com/aphp/eds-pseudo</footnote>

<footnote n="6">⁶We discuss context-related memorization in Section 7 and show that its impact on confidentiality is very limited compared to memorizing the words used in masks.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4: During its specialization, the model will mainly memorize the words that were used for language modeling masking, this memorization can be used by an adversary to infer membership.</figcaption></figure>

different from the indirect identifiers from the dataset used to fine-
tune the model. This difference can lead to incorrect predictions
for the adversary.

4 PRIVACY-PRESERVING LANGUAGE
MODELING

In order to prevent the model from memorizing personal informa-
tion, the privacy-preserving language modeling that we propose
ensures that the choice of mask used during the specialization of a
pre-trained model avoids directly and indirectly identifying words.
To achieve that, this solution is based on a preprocessing that is
responsible for identifying both the direct and indirect identifiers.
We first present this preprocessing step (Section 4.1) before explain-
ing how the proposed masked and the causal language modeling
work (Section 4.2 and Section 4.3, respectively).

4.1 Preprocessing: Building a Blacklist

The preprocessing step focuses on identifying identifiers that the
model must not to memorize. First, we use an off-the-shelf NER
model to directly identify identifying words associated with certain
categories. This NER model can be adapted to include more or less
categories depending on the field of specialization. Then, we build
the bipartite graph G between the individuals of the corpus and the
words used in their documents in order to identify indirect iden-
tifiers (similar to Figure 3). To do this, once the graph is built, we
identify the words that are only linked to one individual. Finally, all
these (directly or indirectly) identifiers are kept in a kind of black-
list that is used during the following masked or causal language
modeling.

If we make the link with the notion of k-anonymity[49], which
defines that a data is anonymous if it is used by at least k different
users, all the words used as masks to specialize the model in the
language modeling step are at least used by two people (i.e., k=2).

This makes the task of an adversary wishing to infer the mem-
bership of an individual more difficult. We can easily change the
exploitation of the bipartite graph to consider a larger value of k for
the identification of indirectly identifying information (e.g., words
that are linked to 1 or 2 users to have a k=3).

In addition to being able to vary the value of k to protect indirect
identifiers, and to use a NER adapted to the domain of specialization
to identify direct identifiers, it should be noted that the cost of this
preprocessing is reduced.

4.2 Privacy-Preserving Masked Language Modeling

Compared to the standard masked language modeling scheme that randomly chooses terms to mask during supervised learning, our privacy-preserving masked language modeling ensures that directly or indirectly identifying terms are never masked by leveraging the blacklist constructed during the preprocessing step. More precisely, the choice of the mask is made randomly among the words that are not in the blacklist.

However, the identifying words (i.e., the words kept in the black-
list) can be used in the context of a mask to be predicted. Their
use improves the prediction of masked terms without introduc-
ing a significant risk of memorization leading to regurgitation. We
evaluate and compare the impact on the utility and privacy trade-
off of memorizing identifying words as a mask or as a context in
Section 6.2.

4.3 Privacy-Preserving Causal Language Modeling

Similar to privacy-preserving masked language modeling, our pri-
vacy preserving causal language modeling ensures that masked
words are not directly or indirectly identifying words. However,
since masked words are always at the end of the sequence to be
predicted, we exploit the blacklist created during the preprocessing
step to check that this last term is not an identifier. If it is, we modify
it and replace it with a padding token and add the next word for
prediction. The advantage of using a padding token is that it is not
used in the loss calculation during the fine-tuning, and therefore
does not degrade the model’s predictions.

Once the last masked word is predicted, we exploit the real
sequence to mask the next word. This means that an identifier
word can be part of the context of a masked word to be predicted.
As in the case of masked language modeling, the exploitation of an
identifying word in the context does not introduce a notable risk
of regurgitation but improves the prediction (Section 6.3).

5 EXPERIMENTAL SETUP

We performed an extensive evaluation of our privacy-preserving
language modeling scheme using medical datasets (Section 5.1) and
considering both off-the-shelf BERT and GPT pre-trained models
(Section 5.2). To capture the full impact of our proposed solutions,
we considered a set of both utility and privacy metrics (Section 5.3)
and considered different comparative baseline approaches (Sec-
tion 5.4). Finally, we describe our methodology (Section 5.5).
## 5.1 Datasets

To conduct our experiments on health data, we considered the N2c2 (National NLP Clinical Challenges) datasets [25]. More precisely, we leverage two datasets gathering medical discharge summaries (i.e., english free text). In the first one [53], 928 records have been annotated for replacing all authentic Personally Identifiable Information (PII) with realistic surrogates. These PII s fall into different categories: six of the seventeen textual PII categories listed by HIPAA (only these six categories appear in the data: patients, locations, dates, IDs, phone numbers, and ages), and two additional categories, doctors and hospitals, resulting in eight PII categories in the dataset. In the second dataset [52] almost 500 records have been annotated by pulmonologists. The pulmonologists were asked to classify patient records into five possible smoking status categories as past smoker, current smoker, smoker, non-smoker, and unknown. We leverage this dataset to assess the utility degradation of our privacy-preserving language modeling through a downstream classification task.

## 5.2 Model Architecture

In order to cover the different use cases of language models (i.e., the generative models and the descriptive text models), we consider both the BERT architecture (i.e., the base model case composed of 12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters) and the GPT-2 architecture [39] (the smallest version of GPT-2, with 124M parameters) from Hugging Face<fnref n="7" />. We did not consider larger model versions due to resource limitations, and we also discarded distilled version of these models [42].

## 5.3 Metrics

We consider both utility and privacy measures in our evaluation to better assess the trade-off between protecting training data and the performance of the model in language modeling.

5.3.1 **Utility.** For the utility evaluation of BERT-like models, we quantify the ability of the models to predict masked terms well (measured by the prediction accuracy), and the model’s performance in classification by creating a downstream task (measured by the F1-Score metric on the prediction of the smoking status). For GPT-like models, the utility is evaluated by the model’s ability to generate the right tokens in the right places (measured by the tokens accuracy). To evaluate the quality of the generated text, we also consider the BLEU and ROUGE scores. While the BLEU (Bilingual Evaluation Understudy) score measures the similarity between the generated text and the original text using n-grams (i.e., contiguous sequences of n words), the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score calculates the recall of common n-grams.

5.3.2 **Privacy.** To assess the model' privacy, we study its ability to predict identifiers (direct and indirect) contained in the dataset used to specialize the model. An individual's presence is revealed as soon as at least one of their identifiers is predicted or regurgitated by the model, whether this prediction is in the correct place in the original text or elsewhere. We use the metric *Privacy* to measure the ability of a model to predict any identifying term (i.e., direct

or indirect identifier). A Privacy=1 means that no identifiers are predicted or regurgitated by the model, and a Privacy=0 means that all identifiers are leaked. The metrics named *Direct Privacy* and *Indirect Privacy* measure, respectively, the ability of the model to predict only directly or indirectly identifying terms. We also considered MIAs to measure an adversary's ability to correctly predict whether a patient's reports were used in model fine-tuning. Specifically, we considered a state-of-the-art MIA attack based on the loss [15] through the MIMIR package<fnref n="8" />, and an attack that infers membership if the model predicts any identifiers present in the adversary's auxiliary knowledge.

## 5.4 Comparative Baselines

We compare our privacy-preserving language modeling scheme to different comparative baselines for both Masked Language Modeling (e.g., BERT-like models) and Causal Language Modeling (e.g., GPT-like models). For the former language modeling, our solution is named PPMLM-BERT, and for the latter our solution is named PPCLM-GPT.

* **Masked Language Modeling (MLM):** First, we consider a classical specialization of a BERT model (named MLM-BERT). Next, we consider a baseline that performs the same specialization but using a pseudonymized dataset for the fine-tuning (baseline named MLMA-BERT). Finally, we consider two variants of our solution where only direct or indirect identifiers are protected from memorization (baselines named DPPMLM-BERT and IPPMLM-BERT, respectively).
* **Causal Language Modeling (CLM):** We consider a classical specialization of a GPT model (named CLM-GPT) that consists in iteratively predicting each term of a sequence of the dataset used for the specialization. We also consider a specialization using a pseudonymized dataset (baseline named CLMA-GPT), and two variants of our solution that protect only directly or indirectly identifying terms (baselines named DPPCLM-GPT and IPPCLM-GPT, respectively). Lastly, we consider a state-of-the-art baseline using a differentially private fine-tuning of pre-trained language models [58, 63] integrating the Opacus library<fnref n="9" />.

## 5.5 Methodology

This section provides the methodological details we followed to perform our evaluations. First, when datasets are pseudonymized before being used for model fine-tuning (i.e., in the case of MLMA-BERT and CLMA-GPT), direct identifiers are replaced by the term "X" in the text. Then, BERT-like models were trained by randomly replacing a subset of tokens from the sequence by the `<mask>` token, and asking the model to predict them using cross-entropy loss. In our setting, 15% of the words are randomly selected. In the case of our solution, identifying words are excluded from this random selection. If a word chosen to be masked consists of several tokens, all of them are masked. At each epoch, 15% of non-identifying words are masked.

Finally, GPT-like models were trained by sequentially masking the last word in a sequence. In our solution, if this word is a direct

<footnote n="7">⁷Hugging Face - https://huggingface.co/</footnote>

<footnote n="8">⁸MIMIR - https://github.com/iamgroot42/mimir</footnote>

<footnote n="9">⁹Opacus - https://opacus.ai/</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 5: Cumulative distribution of the number of identifiers (both direct and indirect ones), and words per patient: half of the patients have more than 20 indirect identifiers, and almost all patient have at least 3 indirect identifiers.</figcaption></figure>

or indirect identifier, it is replaced by a specific padding token that is not taken into account during fine-tuning (i.e, for the computation of the loss). Unlike CLMA-GPT (the baseline that finetunes a GPT model with a pseudonymized dataset), padding tokens are not used in the computation of the loss compared to the term "X" (i.e., the anonymized equivalent of the identifiers) which is taken into account in the loss.

All measurements were performed after 4, 8, 16, 32 and 64 epochs to observe their evolution. We used 16 batches of 512 tokens for training. For language modeling, the learning rate starts at 1e-4 and decreases linearly until the end of finetuning. For classification, we took the finetuned model after 64 epochs and finetuned it again by freezing all the BERT layers (to not bias the model and introduce other leakage) except the last one and the classification layer following the methodology described in this study [48]. The learning rate this time starts at 0, 10% of the total number of warmup steps with the learning rate increasing until reaching 2e-5 then decreasing linearly on the remaining 90% steps. All the computation has been parallelized on a hybrid GPU/CPU computing farm.

# 6 EVALUATION

This section presents a comprehensive evaluation of our privacy-preserving language modeling scheme. We first analyze the distribution of identifiers in the dataset (Section 6.1). We then evaluate the utility and privacy trade-off of both masked and causal language modeling (Section 6.2 and Section 6.3, respectively). Finally, we evaluate the impact of MIAs (Section 6.4) before evaluating the impact of considering n-grams as potential indirect identifiers (Section 6.5). We show that memorization of identifiers can lead to the membership inference of individuals whose data were used in training. Furthermore, we show that protecting the direct identifier only through pseudonymization is not enough to protect the model against a risk of memorizing indirect identifiers.

<figure><img src="image_7.png" /><figcaption>Figure 6: Cumulative distribution of the number of patients who share words in their reports: 60% of distinct words in the corpus are used by only one patient and therefore represent indirect identifiers.</figcaption></figure>

## 6.1 Identifiers Analysis

In this section, we first analyze the distribution of direct and indirect identifiers in the N2C2 dataset, as well as the number of words in each patient's reports (Figure 5). We can see that almost all patients have indirect identifiers and half of them have more than twenty indirect identifiers. We can also see that direct identifiers are about twice as numerous as indirect identifiers. Finally, a fraction of indirect identifiers (i.e., words exploited in the documents of a single patient) are also direct identifiers.

Figure 6, in turn, shows the cumulative distribution of the number of patients who share words in their reports. The plot distinguishes distinct words from the total number of words by including repetitions. The distribution shows that 60% of distinct words in the corpus are used by only one patient and therefore represent indirect identifiers. However, the use of these words represents less than 3% of the total occurrences of words present in all reports. In other words, the 40% of words used in reports of multiple patients represents more than 97% of the total volume of words used.

## 6.2 Utility and Privacy Tradeoff for Masked Language Modeling

We now evaluate the utility and privacy tradeoff for our masked language modeling scheme. Figure 7 illustrates this tradeoff for PPMLM-BERT and other comparative baselines. First, the results show that without a specific mechanism to prevent the model from memorizing identifiers (i.e., classical MLM-BERT), privacy is significantly degraded, from 0.87 after 4 epochs to 0.66 after 64 epochs. This decrease in privacy means that the model learns more and more identifiers over the epochs and is led to predict them more along the training. Second, training without memorizing indirect identifiers (i.e., IPPMLM-BERT) provides about the same level of utility while increasing privacy. This small increase in privacy is due to the reduced number of indirect identifiers. Training without memorizing the direct identifiers, in turn, significantly improves
<figure><img src="image_8.png" /><figcaption>Figure 7: Utility and privacy evaluation for MLM: by avoiding memorizing both direct and indirect identifiers during model specialization, our PPMLM-BERT solution offers the best tradeoff in maintaining high privacy while retaining high utility.</figcaption></figure>

privacy. Indeed, since the number of direct identifiers is signifi-
cantly larger than the number of indirect identifiers, the increase in
privacy is significantly more pronounced. The utility, on the other
hand, is similar. Then, training using a pseudonymized dataset (i.e.,
leveraging a training dataset without direct identifiers, MLMA-BERT)
provides a similar level of privacy but reduces the utility. This re-
duction in utility comes from a context used for prediction that does
not benefit from direct identifying terms. Finally, our PPMLM-BERT
solution offers the best utility and privacy tradeoff where the pri-
vacy remains stable around 0.98 regardless of the number of epochs,
meaning that the model does not learn any identifier directly or
indirectly during the fine-tuning. Regarding utility, PPMLM-BERT
also offers the best utility. This high utility is due to the wealth of
information (i.e., using all terms including identifiers) in the context
used to predict a term. In Appendix A.1, we examine the behavior of

<figure><img src="image_9.png" /><figcaption>Figure 9: Utility and privacy evaluation for CLM: by avoiding memorizing both direct and indirect identifiers during model specialization, our PPCLM-GPT solution offers the best tradeoff in maintaining high privacy while retaining high utility in the generation.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Utility assessment for MLM through a down-stream classification task: all baselines provide roughly the same level of F1-Score.</figcaption></figure>

PPMLM-BERT and other baselines if privacy leakages are measured
only from direct or indirect identifiers.

We now evaluate the utility of the models through a classifica-
tion task. As described in Section 5.3, this classification task aims
to predict the patient's smoking, and the classification models are
trained as a downstream task from the specialized model on the
dataset. Figure 8 illustrates the F1-Score of the classification model
after 8 epochs. Counterintuitively, the results show that models that
protect identifiers display slightly better accuracy. These results
tend to show that identifiers (i.e., information that only affects a sin-
gle individual) reduce the model's ability to generalize knowledge
and classify correctly.

## 6.3 Utility and Privacy Tradeoff for Causal Language Modeling

This section evaluates the utility and privacy trade-off of our PPCLM-GPT compared to other comparative baselines. Figure 9 presents the utility through the prism of token accuracy as a function of privacy assessed by the model's ability to predict both direct and indirect identifiers during generation. The results show similar observations to those of PPMLM-BERT but with much better privacy for PPCLM-GPT compared to other baselines (up to 20% more privacy). Without any protection (i.e., CLM-GPT which corresponds to a classic fine-tuning), the model learns the identifiers during training. After 64 epochs, this model is even able to regurgitate the direct and indirect identifiers it saw during training (revealed by a token accuracy close to 1). IPPCLM-GPT (i.e., protecting only indirect identifiers) provides a similar level of privacy to classic fine-tuning because indirect identifiers are small in number. However, the utility is limited to 80%, which means that the model does not regurgitate indirect identifiers because they were not learned during training. DPPCLM-GPT and CLMA-GPT, in turn, provide similar results with privacy decreasing to 80% privacy after 64 epochs. This decrease means that the model learns indirect identifiers over the epoch and tends to regurgitate
<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /><figcaption>**Figure 10:** A GPT model fine-tuned through our privacy-preserving language modeling provides roughly the same BLUE and ROUGE score than a model fine-tuned with pseudonymized data, while a model without any protection provides better utility (15% after 32 epochs).</figcaption></figure>

them more frequently. Finally, DP drastically impacts the model's utility. Results show that token accuracy is lower than 0.3. Even worse, since DP is content-independent, privacy leakage increases slightly with the number of epochs, indicating that the model tends to increase memorization and regurgitation of identifiers as fine-tuning progresses. In Appendix A.2, we examine the behavior of PPCLM-GPT and other baselines if privacy leakages are measured only from direct or indirect identifiers.

We now evaluate the usefulness of the fine-tuned GPT models through text generation quality assessment metrics (Figure 10). More precisely, we leverage the BLUE and the ROUGE scores which rely on n-grams overlap to measure similarity and recall between the text generated by the model and the reference text (i.e., the original one in our case). Figure 10(a) and Figure 10(b) depict these scores over the number of epochs for a fine-tuned GPT model without any protection (i.e., CLM-GPT), for a GPT model fine-tuned

<figure><img src="image_13.png" /><figcaption>**Figure 11:** ROC curve for the Membership Inference Attack with log-log scaling to emphasize the low-FPR regime.</figcaption></figure>

with pseudonymized data (i.e., CLMA-GPT), and for our privacy-preserving modeling scheme PPCLM-GPT. Results show that for both utility metrics, CLMA-GPT and PPCLM-GPT provide roughly the same level of utility, while the solution without protection improves by 15% the utility after 32 epochs.

## 6.4 Impact of Membership Inference Attacks

To evaluate the accuracy of the membership inference attacks we conducted a ROC (Receiver Operating Characteristic) analysis which consists of comparing the positive predictions (i.e., membership guesses) over negative (non-membership) predictions. Specifically, we consider the true-positive rate (TPR) at low false-positive rates (FPR) for an attack conducted on a fine-tuned BERT model. The ROC curve (Figure 11) depicts the AUC (Area Under the Curve) for a model specialized without protection (i.e., MLM-BERT), a model specialized from pseudonymized data (i.e., MLM-BERT), and a model specialized following our scheme (i.e., PPMLM-BERT). The dashed line shows AUC=0.5 which corresponds to random guess. Results show that for low false-positive rates (i.e., FPR at 1%), the true-positive rate is around 0.9% for both MLM-BERT and PPMLM-BERT, and around 2.5% for MLM-BERT. The AUC of PPMLM-BERT is slightly lower than the AUC of both MLM-BERT and MLM-BERT (i.e., 0.59 versus 0.61) that tends to say that the MIA applied on the model fine-tuned following our privacy-preserving language modeling scheme is slightly less efficient than on fine-tuned models without protection or using pseudonymized data.

A model can predict or regurgitate identifying terms. Let us now see if this prediction of identifiers can be used to infer membership at low cost, i.e., an individual will be considered as a member if at least one of their indirect identifiers is predicted by the model. Figure 12 depicts the precision, recall, and F1-Score of this low-cost membership inference when the adversary is able to construct the same bipartite graph as the one used in the fine-tuning (i.e., the adversary has the data used to fine-tune the model, which corresponds to a privacy audit carried out by the designer on his own model). The results for a fine-tuned model without protection
show that it is possible to infer membership with high precision and a recall close to 0.7. Conversely, when the model is fine-tuned with our privacy-preserving solution, the recall drops to 0.1, drastically reducing the attacker's ability to infer membership.

<figure><img src="image_14.png" /><figcaption>**Figure 12:** Inferring membership at low cost: by inspecting the identifiers predicted or regurgitated by the model, an adversary is able to predict membership with a good accuracy.</figcaption></figure>

Figure 13, in turn, illustrates the F1-Score of the membership inference attack as a function of the number of epochs (i.e., overfitting) for unprotected language modeling (i.e., MLM-BERT) and our privacy-preserving scheme (i.e., PPMLM-BERT). The results show that, without protection, membership inference increases with the number of epochs, indicating a strong correlation between overfitting and the ability to predict membership. Conversely, membership inference for PPMLM-BERT remains stable as training progresses.

<figure><img src="image_15.png" /><figcaption>**Figure 13:** While membership inference increases with the number of epochs (i.e., overfitting) in the case of unprotected language modeling, our privacy-preserving language modeling scheme is not affected by overfitting.</figcaption></figure>

## 6.5 Considering n-grams

The proposed solution only considers isolated words as indirect identifiers. However, these indirect identifiers can be composed of multiple terms (i.e., n-grams of variable size n). Extending our

<figure><img src="image_16.png" /><figcaption>**Figure 14:** Distribution of the number of indirect identifiers by considering n-grams as potential identifiers: the number of indirect identifiers does not increase after n=2.</figcaption></figure>

solution to consider these n-grams as identifiers involves generating a bipartite graph accounting them. We first analyze the dataset to quantify the number of n-grams detected as indirect identifiers (i.e., unique to a patient). Figure 14 illustrates the distribution of the number of n-grams as indirect identifiers per patient for different values of n. Results show that, on average, the number of indirect identifiers doubles if we consider n from 1 (i.e., a word unique to a patient) to 2 (i.e., a sequence of two words unique to a patient). Considering n greater than 2 does not significantly increase the number of indirect identifiers.

We then extend our solution to consider n-grams as potential indirect identifiers. Figure 15 quantifies the number of identifiers predicted or generated by the fine-tuned BERT or GPT models for n from 1 (i.e., a single word) to 3 (i.e., a unique sequence of 3 words). We plot the results for models accounting both n=1 (dash lines) and n=3 (continuous lines) during our language modeling, and models using comparative language modeling baselines. First results show that for BERT and GPT models fine-tuned without protection (i.e., MLM-BERT or CLM-GPT) or fine-tuned with pseudonimized data (i.e., MLMA-BERT and CLMA-GPT), the number of identifiers increases according to the value of n. For instance, for CLM-GPT, the number of identifiers composed of a single word is around 30,000, while this number increases to more than 80,000 when the identifiers can also be composed of a sequence of two words. Similarly, when our language modeling scheme considered only single words as indirect identifiers during the fine-tuning, the model can still regurgitate sequences of two-word identifiers. However, when our language modeling scheme considered identifiers up to n=3 during the fine-tuning, the number of regurgitated identifiers remains stable, which is consistent because these 3-grams have been included in the blacklist of terms not to be memorized by the model.

## 7 LIMITATIONS AND DISCUSSIONS

The work presented in this paper represents a practical step towards protecting the training data against privacy leakage through
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /><figcaption>**Figure 15:** By leveraging the bipartite graph to account the n-grams as potential indirect identifiers, our privacy-preserving language modeling can avoid the prediction or the regurgitation of these identifiers.</figcaption></figure>

the fine-tuning of pre-trained LLMs. There are, however, some limitations in several areas. Although the study and results have been validated on one dataset, this dataset is small in size. A validation on a larger dataset requires more computational resources and is left as future work.

Another limitation is that the identification of direct identifiers of our privacy-preserving scheme relies on a NER (Named Entity Recognition), and the quality of this NER can impact the performance of the protection of our solution. This limitation is common to all pseudonimization schemes relying on a NER and affects both the protection itself by missing identifying words, and the privacy metrics by not counting certain leaks of personal information. However, our solution is not impacted by misspelled and/or foreign words. Indeed, a common problem of NER in free text is the management of misspelled and/or foreign words that cannot be found in dictionaries. These words can be identifiers and not be protected. It is worth noting that the advantage of our solution is that these

words will be identified as indirectly identifiers in our solution and will not be memorized.

It is also worth noting that the cost of our preprocessing step (i.e., building the bipartite graph) is significantly lower than using NER and fine-tuning the model. For example, when fine-tuning a BERT model (with 64 epochs) using a large dataset (50,000 documents), building the bipartite graph takes 0.3% of the overall fine-tuning time, while using NER takes 7%.

The language modeling scheme presented here avoids the model from memorizing direct and indirect identifying information present in the training data (including medical datasets and sensitive data) used for the specialization of off-the-shelf foundation LLMs. On the basis of the article 9 of the GDPR and on the Opinion 28/2024 of the EDPB on certain data protection aspects related to the processing of personal data in the context of AI models<fnref n="10" /> as well as the AI Privacy Risks & Mitigations Large Language Models (LLMs) report<fnref n="11" />, it is possible to say the fine-tuning of the LLM does not give birth to an fully anonymised model, due to the fact that the full anonymisation of the model can only be reached by an anonymisation process during the data preparation and preprocessing and the training phases of the foundation model<fnref n="12" />. Due to the fact that the model, before fine-tuning, was not, from the outset, anonymised, what is obtained is anonymization of the secondary learning data set used during fine-tuning, but not anonymization of the set of personal data on the basis of which the foundation model learned beforehand. In other words, a lot of personal data can be regurgitated by the fine-tuned model, even if it is not the health data on the basis of which the fine-tuning took place. Furthermore, interactions between the training data of the foundation model and the data used for fine-tuning are not controlled and may give rise to hallucinations. Consequently, the requirement set out in the EDPB opinion of an "insignificant probability of personal data regurgitation" to qualify the fine-tuned model as anonymous is not met (Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models). Consequently, this means that the fine-tuned model will have to respect the conditions set out by the EDPB, in the three scenarios in the presence of a re-use described in their opinion <fnref n="13" />.

## 8 CONCLUSION

This article presents a methodology to prevent a pre-trained LLM from memorizing direct or indirect identifiers linked to individuals present in a dataset used to specialize it. This absence of identifier memorization facilitates model sharing by limiting the risk of it regurgitating identifying information during further exploitation. Our privacy-preserving language modeling scheme takes into account masked language modeling (i.e., for BERT-like models) and causal language modeling (i.e., for GPT-like models). We exhaustively evaluated our approach and demonstrated that it offers

<footnote n="10">¹⁰https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_ai-models_en.pdf</footnote>

<footnote n="11">¹¹https://www.edpb.europa.eu/our-work-tools/our-documents/support-pool-experts-projects/ai-privacy-risks-mitigations-large_en</footnote>

<footnote n="12">¹²AI Privacy Risks & Mitigations Large Language Models (LLMs) report, p. 25 et p. 33</footnote>

<footnote n="13">¹³Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, p. 4.</footnote>
the best tradeoff between utility and privacy compared to other baselines.

## REFERENCES

[1] 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

[2] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. (oct 2016). https://doi.org/10.1145/2976749.2978318

[3] Marianne Abi Kanaan, Jean-François Couchot, Christophe Guyeux, David Laiymani, Talar Atechian, and Rony Darazi. 2023. A methodology for emergency calls severity prediction: from pre-processing to BERT-based classifiers. In *IFIP Advances in Information and Communication Technology*, Vol. 675. Leon, Spain. https://doi.org/10.1007/978-3-031-34111-3_28

[4] Michael Aerni, Jie Zhang, and Florian Tramèr. 2024. Evaluations of Machine Learning Privacy Defenses are Misleading. In *Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security* (Salt Lake City, UT, USA) (CCS '24). Association for Computing Machinery, New York, NY, USA, 1271–1284. https://doi.org/10.1145/3658644.3690194

[5] J Alammar. (2018). *The Illustrated Transformer*. Retrieved from https://jalammar.github.io/illustrated-transformer/

[6] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In *2022 IEEE Symposium on Security and Privacy (SP)*. IEEE, 1897–1914. https://doi.org/10.1109/SP46214.2022.9833649

[7] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. arXiv:2202.07646 [cs.LG] https://arxiv.org/abs/2202.07646

[8] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jenej Kos, and Dawn Song. 2019. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. arXiv:1802.08232 [cs.LG] https://arxiv.org/abs/1802.08232

[9] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. Extracting Training Data from Large Language Models. CoRR abs/2012.07805 (2020). arXiv:2012.07805 https://arxiv.org/abs/2012.07805

[10] Dingfan Chen, Ning Yu, and Mario Fritz. 2022. RelaxLoss: Defending Membership Inference Attacks without Losing Utility. arXiv:2207.05801 [cs.LG] https://arxiv.org/abs/2207.05801

[11] Ruizhe Chen, Tianxiang Hu, Yang Feng, and Zuozhu Liu. 2024. Learnable Privacy Neurons Localization in Language Models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 256–264. https://doi.org/10.18653/v1/2024.acl-short.25

[12] Xianlai Chen, Jiamiao Lin, and Ying An. 2022. DL-BERT: a time-aware double-level BERT-style model with pre-training for disease prediction. In *2022 IEEE International Conference on Big Data (Big Data)*. 1801–1808. https://doi.org/10.1109/BigData55660.2022.10020513

[13] A. Feder Cooper, Katherine Lee, James Grimmelmann, Daphne Ippolito, Christopher Callison-Burch, Christopher A. Choquette-Choo, Niloofar Miresghallah, Miles Brundage, David Mimno, Madiha Zahrah Choksi, Jack M. Balkin, Nicholas Carlini, Christopher De Sa, Jonathan Frankle, Deep Ganguli, Bryant Gipson, Andres Guadamuz, Swee Leng Harris, Abigail Z. Jacobs, Elizabeth Joh, Gautam Kamath, Mark Lemley, Cass Matthews, Christine McLeavey, Corynne McSherry, Milad Nasr, Paul Ohm, Adam Roberts, Tom Rubin, Pamela Samuelson, Ludwig Schubert, Kristen Vaccaro, Luis Villa, Felix Wu, and Elana Zeide. 2023. Report of the 1st Workshop on Generative AI and Law. arXiv:2311.06477 [cs.CY] https://arxiv.org/abs/2311.06477

[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

[15] Michael Duan, Anshuman Suri, Niloofar Miresghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do Membership Inference Attacks Work on Large Language Models? In *Conference on Language Modeling (COLM)*.

[16] Henri Duprieu and Nicolas Berkouk. 2024. *Techniques d'audit des grands modèles de langage*. Technical Report. Commission Nationale Informatique et Libertés (CNIL). https://hal.science/hal-04782667

[17] Basile Dura, Charline Jean, Xavier Tannier, Alice Calliger, Romain Bey, Antoine Neuraz, and Rémi Flicoteaux. 2022. Learning structures of the French clinical language: development and validation of word embedding models using 21 million clinical reports from electronic health records. arXiv:2207.12940 [cs.CL] https://arxiv.org/abs/2207.12940

[18] Cynthia Dwork. 2006. Differential privacy. In *International colloquium on automata, languages, and programming*. Springer, 1–12.

[19] Cédric Eichler, Nathan Champeil, Nicolas Anciaux, Alexandra Bensamoun, Héber H. Arcolezi, and José Maria De Fuentes. 2025. Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction. In *Web Information Systems Engineering – WISE 2024, Mahmoud Barhamgi, Hua Wang, and Xin Wang (Eds.). Springer Nature Singapore, Singapore*, 441–456.

[20] Johnson et al. 2020. Deidentification of free-text medical records using pre-trained bidirectional transformers. (2020). https://pubmed.ncbi.nlm.nih.gov/34350426/

[21] Farshid Faal. 2022. *Reinforcement Learning for Mitigating Toxicity in Neural Dialogue Systems*. Ph. D. Dissertation. Concordia University.

[22] Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. 2024. Healai: A healthcare Ilm for effective medical documentation. In *Proceedings of the 17th ACM International Conference on Web Search and Data Mining*. 1167–1168. https://doi.org/10.1145/3616855.3635739

[23] Hamza Harkous, Sai Teja Peddinti, Rishabh Khandelwal, Animesh Srivastava, and Nina Taft. 2022. Hark: A Deep Learning System for Navigating Privacy Feedback at Scale. In *2022 IEEE Symposium on Security and Privacy (SP)*. 2469–2486. https://doi.org/10.1109/SP46214.2022.9833729

[24] Tzvika Hartman, Michael D Howell, Jeff Dean, Shlomo Hoory, Ronit Slyper, Itay Laish, Oren Gilon, Danny Vainstein, Greg Corrado, Katherine Chou, et al. 2020. Customization scenarios for de-identification of clinical notes. BMC medical informatics and decision making 20, 1 (2020), 1–9. https://doi.org/10.1186/s12911-020-1026-2

[25] Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2020. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. *Journal of the American Medical Informatics Association* 27, 1 (2020), 3–12. https://doi.org/10.1093/jamia/ocz166

[26] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2020. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. arXiv:1904.05342 [cs.CL] https://arxiv.org/abs/1904.05342

[27] Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership Inference Attack Susceptibility of Clinical Language Models. arXiv:2104.08305 [cs.CL] https://arxiv.org/abs/2104.08305

[28] Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023. DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains. arXiv:2304.00958 [cs.CL] https://arxiv.org/abs/2304.00958

[29] Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Online, 946–959. https://doi.org/10.18653/v1/2021.naacl-main.73

[30] Ximeng Liu, Lehui Xie, Yaopeng Wang, Jian Zou, Jinbo Xiong, Zuobin Ying, and Athanasios V. Vasilakos. 2021. Privacy and Security Issues in Deep Learning: A Survey. IEEE Access 9 (2021), 4566–4593. https://doi.org/10.1109/ACCESS.2020.3045078

[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs.CL] https://arxiv.org/abs/1907.11692

[32] Louis Martin, Benjamin Müller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. CamemBERT: a Tasty French Language Model. CoRR abs/1911.03894 (2019). arXiv:1911.03894 http://arxiv.org/abs/1911.03894

[33] Fatemehsadat Miresghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. arXiv:2203.03929 [cs.LG] https://arxiv.org/abs/2203.03929

[34] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable Extraction of Training Data from (Production) Language Models. arXiv:2311.17035 [cs.LG] https://arxiv.org/abs/2311.17035

[35] Ahmet Okutan, Ali Shokri, Viktoria Koscinski, Mohamad Fazelinia, and Mehdi Mirakhorli. 2023. A Novel Approach to Identify Security Controls in Source Code. arXiv:2307.05605 [cs.SE] https://arxiv.org/abs/2307.05605

[36] Michele Panariello, Natalia Tomashenko, Xin Wang, Xiaoxiao Miao, Pierre Champion, Hubert Nourtel, Massimiliano Todisco, Nicholas Evans, Emmanuel Vincent, and Junichi Yamagishi. 2024. The VoicePrivacy 2022 Challenge: Progress and Perspectives in Voice Anonymisation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024), 3477–3491. https://doi.org/10.1109/TASLP.2024.3430530

[37] Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. arXiv:1906.05474 [cs.CL] https://arxiv.org/abs/1906.05474
[38] Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. arXiv:2202.00443 [cs.CL] https://arxiv.org/abs/2202.00443
[39] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. https://api.semanticscholar.org/CorpusID:160025533
[40] Antoine Richard, François Talbot, and David Gimbert. 2023. Anonymisation de documents médicaux en texte libre et en français via réseaux de neurones. In *Plate-forme Intelligence Artificielle 2023 (PFlA2023) - Journée Santé & IA*. Association française pour l'Intelligence Artificielle (AflA) and Université de Strasbourg and Association française d'Informatique Médicale (AIM), Starsbourg, France. https://hal.science/hal-04139391
[41] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistILBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs.CL] https://arxiv.org/abs/1910.01108
[42] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistILBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs.CL] https://arxiv.org/abs/1910.01108
[43] Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. 2024. Rethinking LLM Memorization through the Lens of Adversarial Compression. arXiv:2404.15146 [cs.LG] https://arxiv.org/abs/2404.15146
[44] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In *2017 IEEE symposium on security and privacy (SP)*. IEEE, 3–18. https://doi.org/10.1109/SP.2017.41
[45] Sushant Singh and Ausif Mahmood. 2021. The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures. CoRR abs/2104.10640 (2021). arXiv:2104.10640 https://arxiv.org/abs/2104.10640
[46] Tanmay Singla, Dharun Anandanyuvaraj, Kelechi G. Kalu, Taylor R. Schorlemmer, and James C. Davis. 2023. An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures. In *Proceedings of the 2023 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (Copenhagen, Denmark) (SCORED '23)*. Association for Computing Machinery, New York, NY, USA, 5–15. https://doi.org/10.1145/3605770.3625214
[47] Till Speicher, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, and Evimaria Terzi. 2024. Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications. arXiv:2407.19262 [cs.CL] https://arxiv.org/abs/2407.19262
[48] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2020. How to Fine-Tune BERT for Text Classification? arXiv:1905.05583 [cs.CL] https://arxiv.org/abs/1905.05583
[49] Latanya Sweeney. 2002. k-anonymity: A model for protecting privacy. *International journal of uncertainty, fuzziness and knowledge-based systems* 10, 05 (2002), 557–570. https://doi.org/10.1142/S0218488502001648
[50] Xavier Tannier, Perceval Wajsbürt, Alice Calliger, Basile Dura, Alexandre Mouchet, Martin Hilka, and Romain Bey. 2023. Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehouse. arXiv:2303.13451 [cs.CL] https://arxiv.org/abs/2303.13451
[51] Yakini Tchouka, Jean-François Couchot, Maxime Coulmeau, David Laiymani, Philippe Selles, and Azzedine Rahmani. 2023. De-Identification of French Unstructured Clinical Notes for Machine Learning Tasks. arXiv:2209.09631 [cs.CR] https://arxiv.org/abs/2209.09631
[52] Özlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Kohane. 2008. Identifying Patient Smoking Status from Medical Discharge Records. *Journal of the American Medical Informatics Association* 15, 1 (01 2008), 14–24. https://doi.org/10.1197/jamia.M2408 arXiv:https://academic.oup.com/jamia/article-pdf/15/1/14/2339646/15-1-14.pdf
[53] Özlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the State-of-the-Art in Automatic De-identification. *Journal of the American Medical Informatics Association* 14, 5 (09 2007), 550–563. https://doi.org/10.1197/jamia.M2444
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. CoRR abs/1706.03762 (2017). arXiv:1706.03762 http://arxiv.org/abs/1706.03762
[55] Yijue Wang, Nuo Xu, Shaoyi Huang, Kaleel Mahmood, Dan Guo, Caiwen Ding, Wujie Wen, and Sanguthevar Rajasekaran. 2022. Analyzing and Defending against Membership Inference Attacks in Natural Language Processing Classification. In *2022 IEEE International Conference on Big Data (Big Data)*. 5823–5832. https://doi.org/10.1109/BigData55660.2022.10020711
[56] Qiang Wei, Xu Zuo, Omer Anjum, Yan Hu, Ryan Denlinger, Elmer V. Bernstam, Martin J Citardi, and Hua Xu. 2022. ClinicalLayoutLM: A Pre-trained Multi-modal Model for Understanding Scanned Document in Electronic Health Records. In *2022 IEEE International Conference on Big Data (Big Data)*. 2821–2827. https://doi.org/10.1109/BigData55660.2022.10020569
[57] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba

Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks Posed by Language Models. In *Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency* (Seoul, Republic of Korea) (FACCT '22). Association for Computing Machinery, New York, NY, USA, 214–229. https://doi.org/10.1145/3531146.3533088

[58] Lukas Wutschitz, Huseyin A. Inan, and Andre Manoel. 2022. dp-transformers: Training transformer models with differential privacy. https://www.microsoft.com/en-us/research/project/dp-transformers.
[59] Xi Yang, Tianchen Lyu, Chih-Yin Lee, Jiang Bian, William R. Hogan, and Yonghui Wu. 2019. A Study of Deep Learning Methods for De-identification of Clinical Notes at Cross Institute Settings. In *2019 IEEE International Conference on Healthcare Informatics (ICHI)*. 1–3. https://doi.org/10.1109/ICHI.2019.8904544
[60] Jiayuan Ye, Aadya Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced Membership Inference Attacks against Machine Learning Models. In *Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security* (Los Angeles, CA, USA) (CCS '22). Association for Computing Machinery, New York, NY, USA, 3093–3106. https://doi.org/10.1145/3548606.3560675
[61] Hangu Yeo, Elahe Khorasani, Vadim Sheinin, Irene Manotas, Ngoc Phuoc An Vo, Octavian Popescu, and Petros Zerfos. 2022. Natural Language Interface for Process Mining Queries in Healthcare. In *2022 IEEE International Conference on Big Data (Big Data)*. 4443–4452. https://doi.org/10.1109/BigData55660.2022.10020685
[62] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In *2018 IEEE 31st Computer Security Foundations Symposium (CSF)*. IEEE Computer Society, Los Alamitos, CA, USA, 268–282. https://doi.org/10.1109/CSF.2018.00027
[63] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautham Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022. Differentially Private Fine-tuning of Language Models. arXiv:2110.06500 [cs.LG] https://arxiv.org/abs/2110.06500
[64] Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. 2024. Low-Cost High-Power Membership Inference Attacks. arXiv:2312.03262 [stat.ML] https://arxiv.org/abs/2312.03262
[65] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021. Counterfactual Memorization in Neural Language Models. CoRR abs/2112.12938 (2021). arXiv:2112.12938 https://arxiv.org/abs/2112.12938

# A PRIVACY LEAKAGE ON DIRECT OR INDIRECT IDENTIFIERS

## A.1 MLM

A model can leak information by predicting or regurgitating direct or indirect identifiers. Let us analyze the behavior of PPMLM-BERT and the other baselines if we consider only the prediction of direct identifiers or indirect identifiers as privacy leakage (Figure 16). In the former case (Figure 16(a)), results show that MLM-BERT and IPPMLM-BERT exhibit a close utility and privacy tradeoff. Indeed, since indirect identifiers are not taken into account by the privacy measure, our mechanism for not memorizing indirect identifiers has no observable effect. Consequently, only the solutions that protect direct identifiers exhibit high privacy (i.e., PPMLM-BERT, MLMA-BERT, and DPPMLM-BERT). However, we observe that MLMA-BERT exhibits the lowest utility. This lower utility comes from the fact that all direct identifiers are pseudonymized in the training data and are not available in the context to improve the model's prediction. Once again, PPMLM-BERT offers the best trade-off between utility and confidentiality because it allows to benefit from direct identifiers in the context to improve predictions. By comparing the trade-off between MLMA-BERT and DPPMLM-BERT, we can evaluate the impact of memorization related to the use of directly identifying words in the context for mask prediction. We can see that the privacy impact is very small, and that using a context that exploits direct identifiers slightly improves the utility.
<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /><figcaption>Figure 16: Evaluating the utility and privacy tradeoff for MLM by biasing the privacy metric toward direct identifiers or indirect identifiers.</figcaption></figure>

<caption>Figure 17: Evaluating the utility and privacy tradeoff for CLM by biasing the privacy metric toward direct identifiers or indirect identifiers.</caption>

Similarly, if the privacy measure only captures the prediction of indirect identifiers as a privacy leakage (Figure 16(b)), only solutions that carefully protect this category of identifiers exhibit high privacy, this is the case for PPMLM-BERT and IPPMLM-BERT. The privacy degradation as a function of the number of epochs for the other baselines highlights the fact that they progressively learn indirect identifiers and predict them more and more often during training.

## A.2 CLM

Figure 17 illustrates the trade-off between utility and privacy when privacy is measured solely through quantifying the leakage of direct (Figure 17(a)) or indirect (Figure 17(b)) identifiers. The results are similar to the observations made with masked language modeling. More precisely, when privacy only takes into account direct identifiers (i.e., this corresponds to pseudonymization), it is the baselines that protect direct identifiers that provide the best compromise. Conversely, if privacy only takes into account indirect identifiers, it is the baselines that protect indirect identifiers that provide the

best compromise. This clearly shows that pseudonymization is not enough to protect a model against identifiable memorization, and that protection of both direct and indirect identifiers like done by our solution is necessary. Finally, the use of identifying words in the context for prediction has almost no impact on the utility and privacy tradeoff (comparison between DPPCLM-GPT and CLMA-GPT).