# A quantitative approach to the GDPR's anonymisation and "appropriate technical and organisational measures" tests

Nils Holzenberger, Winston Maxwell

► To cite this version:

Nils Holzenberger, Winston Maxwell. A quantitative approach to the GDPR's anonymisation and "appropriate technical and organisational measures" tests. Computer Law & Security Review, 2025, 59, pp.106173. 10.1016/j.clsr.2025.106173. hal-05368761

HAL Id: hal-05368761

https://hal.science/hal-05368761v1

Submitted on 17 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# A quantitative approach to the GDPR’s anonymisation and “appropriate technical and organisational measures” tests

Nils Holzenberger ID*, Winston Maxwell

Télécom Paris, Institut Polytechnique de Paris, 19 place Marguerite Perey, 91120 Palaiseau, France

## ARTICLE INFO

**Keywords:**

Differential privacy

European General Data Protection Regulation

Privacy-enhancing technology

Anonymisation

Pseudonymisation

Personal data

Machine learning

## ABSTRACT

This article examines two tests from the European General Data Protection Regulation (GDPR): (1) the test for anonymisation (the “anonymisation test”), and (2) the test for applying “appropriate technical and organisational measures” to protect personal data (the “ATOM test”). Both tests depend on vague legal standards and have given rise to legal disputes and differing interpretations among data protection authorities and courts, including in the context of machine learning. Under the anonymisation test, data are sufficiently anonymised when the risk of identification is “insignificant” taking into account “all means reasonably likely to be used” by an attacker. Under the ATOM test, measures to protect personal data must be “appropriate” with regard to the risks of data loss. Here, we use methods from law and economics to transform these two qualitative tests into quantitative approaches that can be visualized on a graph. For the anonymisation test, we chart different attack efforts and identification probabilities, and propose this as a methodology to help stakeholders discuss what attack efforts are “reasonably likely” to be deployed and their likelihood of success. For the ATOM test, we use the Learned Hand formula from law and economics to chart the incremental costs and benefits of privacy protection measures to identify the point where those measures maximize social welfare. The Learned Hand formula permits the negative effects of privacy protection measures, such as the loss of data utility and negative impacts on model fairness, to be taken into account when defining what level of protection is “appropriate”. We apply our proposed framework to several scenarios, applying the anonymisation test to a Large Language Model, and the ATOM test to a database protected with differential privacy.

## 1. Introduction

Machine learning raises a host of privacy risks and controversies. Privacy Enhancing Technologies (PETs) such as differential privacy can mitigate those risks, first by rendering data non-personal, i.e. fully anonymised. PETs are also used to reduce the risk that personal data will be improperly disclosed thereby fulfilling the obligation to apply “appropriate technical and organisational measures” (ATOM) to protect personal data.

This article examines the test for full anonymisation under the European General Data Protection Regulation (GDPR)<fnref n="1" /> (the “anonymisation test”), and the test under the GDPR for applying “appropriate technical and organisational measures” to protect personal data (the “ATOM test”). Both tests have given rise to legal disputes and differing interpretations among data protection authorities (DPAs) and courts. Both tests depend on vague legal standards. Under the first test, data are

sufficiently anonymised when they are immune from re-identification using “all means reasonably likely to be used” by an attacker. Under the second test, measures to protect personal data must be “appropriate” with regard to the risks of data loss, the state of the art, the costs of implementation and the nature, scope, context and purposes of processing. In this article we use methods from law and economics to transform these qualitative tests into quantitative tests that can be plotted on graphs to compare different levels of identification risk based on different attack scenarios. The graphs will help determine whether the use of PETs is sufficient to be deemed “appropriate” under the ATOM test, and sufficient to resist “reasonably likely” means of attack under the anonymisation test. These graphs will help GDPR compliance teams and data scientists discuss subjects such as the appropriate privacy budget in differential privacy that achieves the right balance between privacy protection and data utility. The graphs

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: nils.holzenberger@telecom-paris.fr (N. Holzenberger).</footnote>

<footnote n="1">¹ Regulation Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC.</footnote>
will help discuss whether, for example, trained Large Language Models (LLMs) contain personal data used during training, and what level of identification attacks are “reasonably likely”.

First, a word about PETs and “pseudonymisation”: in part of the technical literature on PETs, pseudonymisation means the removal of direct identifiers or replacing them with a random pseudonym [1]. In the GDPR, pseudonymisation refers to “the processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately and is subject to technical and organisational measures to ensure that the personal data are not attributed to an identified or identifiable natural person”.<fnref n="2" /> In this article, we consider pseudonymisation as a form of PET and will generally use the broader term PET to avoid confusion. As we will show in the examples below, a PET such as differential privacy can achieve three possible GDPR outcomes based on the level of protection: weak protection that satisfies neither the ATOM test nor the anonymisation test; medium protection that satisfies the ATOM test but not the anonymisation test; strong protection that satisfies both the ATOM test and the anonymisation test. Many forms of PET in the first two categories would be considered pseudonymisation. However, in some circumstances pseudonymisation can fall into the third category, as recently confirmed by the Advocate General of the CJEU.<fnref n="3" /> Thus pseudonymisation and anonymisation are not necessarily exclusive.

The use of PETs to reduce the probability of identification is a fast-
evolving scientific field. The main categories of PETs include homomor-
phic encryption, synthetic data, differential privacy, zero-knowledge
proofs, synthetic data and federated learning [4-7].

Deep learning models such as LLMs are themselves a form of data
compression, containing blurry representations of the data seen during
training [8]. The model captures the essential trends in the data, while
the rest is thrown away as noise. Viewed from this angle, a trained
model, if it is published instead of the training data, could itself be
considered a form of PET with regard to the training data.

We approach the question of anonymisation and ATOM from a cross-disciplinary perspective combining law, economics and computer science. By trying to translate the GDPR anonymisation and ATOM tests into mathematical expressions and graphical representations, we hope to facilitate discussion between GDPR compliance teams and data science teams on the application of PET to training data and trained models, including the use of differential privacy. Statisticians have long studied anonymisation techniques that permit release of microdata in certain environments [9,10]. Scholars in the “statistical disclosure control” or “SDC” field have developed methods for evaluating risks of re-identification based on different anonymisation techniques [11], as well as methods for measuring trade-offs between protective measures and data utility [12]. In parallel, computer science scholars have explored PETs and anonymisation (see works cited in Gadotti et al. [1]). Attempts to connect SDC, computer science, legal and economic approaches are relatively rare, examples including Kesari [13], Elliot et al. [10], Ohm [14] and Selzer et al. [15]. We are not aware, however, of any article that attempts to apply a law and economics and graph approach to the GDPR’s anonymisation test and ATOM test. This, and the fact that we address recent legal and technological developments, are the original contributions of our article.

As we will see, translating intentionally vague legal concepts into
mathematical formulas and graphs requires simplifying assumptions
and certainly does not replace nuanced legal analysis. However, using
law and economics approaches and graphs to illustrate GDPR tests

should help both data scientists and GDPR specialists think about
the problem from the other's perspective, and identify areas where
the two approaches converge. The graphs will generally show three
zones: a zone where all can reasonably agree that the relevant GDPR
test, anonymisation or ATOM, is not met, a middle zone where the
outcome is uncertain, and a third zone where all can reasonably agree
that the relevant GDPR test is met. Because our approach relies on
simplifying assumptions, it will not help resolve individual cases falling
in the middle zone. However, using our graphs to identify the two
extreme sides of the graph will help stakeholders reach a consensus
on the right level of protection, whether to satisfy the ATOM test, or
the anonymisation test. The remainder of the article is structured as
follows.

Section 2 of the article will summarize case law of the CJEU, and the position of the European Data Protection Board (EDPB) on the GDPR's anonymisation and ATOM tests. Section 3 will present how the GDPR's anonymisation test and the ATOM test could be transformed into measurable quantities under a law and economics approach, while also pointing out the limitations of this exercise. Section 4 will then apply these quantitative approaches to four examples, one involving the well-known Netflix challenge, another involving a hypothetical LLM that might contain personal data, a third involving a trained facial recognition model, and a fourth involving transport data protected by differential privacy. Each of these examples will be presented using graphs to show different levels of attack (i.e. identification) efforts and/or different levels of protective measures, and how these variables affect the probability of identification. Except for the Netflix challenge, these examples are contrived, and serve only to illustrate how graphical representations might help in discussing different anonymisation and ATOM thresholds, and the assumptions that underlie them. Section 5 will summarize the main lessons from these examples.

## 2. Summary of case law on the GDPR's anonymisation and ATOM tests

### 2.1. *The anonymisation test*

The question of what constitutes personal data under the GDPR and its predecessor the 1995 Data Protection Directive<fnref n="4" /> has been debated for almost 20 years [16–22]. The relevant test is whether a person is identifiable, directly or indirectly, when taking account of “all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly”.<fnref n="5" /> The first area of debate focused on whether the reference to “the controller or… another person” means that the test should take into account any possible attacker in the world, including the data controller herself (the “objective” standard), or only a particular attacker, such as the person to whom the data have been made available (the “subjective” standard) [19,20].

The European Data Protection Board (formerly called the Article 29 Data Protection Working Party) said that “an effective anonymisation solution prevents *all parties* from singling out an individual in a dataset, from linking two records within a dataset (or between two separate datasets) and from inferring any information in such dataset”.<fnref n="6" /> Reference to “all parties” suggests an objective standard. The CJEU’s 2016 Breyer decision,<fnref n="7" /> which addressed the question of whether dynamic IP addresses are personal data, adopted the subjective approach, stating

<footnote n="2">² GDPR, article 4(5); for a more complete discussion of pseudonymisation,
see The European Data Protection Board [2], Fischer-Hübner et al. [3].</footnote>

<footnote n="3">³ CJEU, Opinion of Advocate General Spielmann, European Data Protection Supervisor v. Single Resolution Board, Case C-413/23 P, 6 February 2025, ECLI:EU:C:2025:59, point 51.</footnote>

<footnote n="4">⁴ Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data.</footnote>

<footnote n="5">⁵ GDPR recital 26.</footnote>

<footnote n="6">⁶ EDPB, Opinion 05/2014 on Anonymisation Techniques, WP216, 10 April 2014, p. 9, italics are ours.</footnote>

<footnote n="7">⁷ CJEU, Breyer v. Germany, Case C-582/14, 19 Oct. 2016, ECLI:EU:C:2016:779.</footnote>
that dynamic IP addresses, in the hands of a web publisher, constitute personal data because the web publisher has legal means to require the internet service provider to reveal the identity of the subscriber corresponding to a given IP address. Under the subjective approach, the focus is on one attacker with access to the data. The subjective approach was confirmed in two 2023 decisions, one by the CJEU's lower general court,<fnref n="8" /> and one by the CJEU's Third Chamber.<fnref n="9" /> The 2023 decision before the general court is currently being reviewed on appeal by the CJEU. In that appeal, the Advocate General confirmed that the anonymisation test should be applied taking into account the situation of a particular person holding the data, confirming the subjective approach.<fnref n="10" /> In this case, a European agency in charge of bank restructuring, the SRB, provided to the audit and consulting firm Deloitte comments from individual shareholders and creditors involved in a restructuring plan, replacing individual names with numerical codes. This is a classic form of pseudonymisation. The European Data Protection Supervisor found that the data in the hands of Deloitte were personal data but did not verify whether Deloitte had the legal or technical means to re-identify the authors of the comments. This approach is wrong, according to the Advocate General, because the question of anonymisation must take into account the reasonable means available to the person or persons with access to the data. In the hands of Deloitte, the data may well be considered anonymised if Deloitte has no means to identify the authors of the comments. In the hands of the SRB, the same data would not be considered anonymised because the SRB has the codes to match comments to particular individuals. Thus the anonymisation test must be applied taking into account the position of a particular person. The Advocate General's opinion is not binding on the court but is noteworthy because it considers that pseudonymised data can be considered personal in the hands of person A, but not in the hands of person B. In 2024, the CJEU clarified that under the subjective approach only persons unfamiliar with the data subject's personal background should be analyzed as potential attackers.<fnref n="11" /> The case involved a press release issued by the European Anti-Fraud Office (OLAF) which contained certain information about a researcher without citing her name. The CJEU said anonymisation should be considered from the point of view of journalists or other persons unfamiliar with the researcher's personal background. This suggests that the data subject should be excluded from the analysis, because he or she already has access to information that would facilitate re-identification. The subjective approach converges with the functional anonymisation approach proposed by Elliot et al. [10], who argue that the question of anonymisation should be asked not about the data itself, but about the data in a particular data environment.

The second area of debate focused on whether the anonymisation test required that identification be practically impossible (the “absolute” approach), or whether it was sufficient for re-identification to be unlikely (the risk-based approach) [20]. In the Breyer decision, the CJEU clarified that data are sufficiently anonymised if the risk of identification is insignificant when taking into account all means reasonably likely to be used. ‘According to the Court, ‘all means reasonably likely to be used” does not include attacks that are “prohibited by law or practically impossible on account of the fact that it requires a disproportionate effort in terms of time, cost and manpower, so that the risk of identification appears in reality to be insignificant”.<fnref n="12" />

When determining what means are reasonably likely to be used to identify persons, the CJEU indicated that we should take into account “all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments”.<fnref n="13" /> More recently, the CJEU confirmed that we should assess the risk of identification taking into account only means available to the persons with access to the data,<fnref n="14" /> again confirming the subjective approach. Finally, determining whether data are linked to a particular person requires taking into account the content, purpose and effect of the information in question.<fnref n="15" /> Thus the nature of the data and, by consequence, the motivation of the attacker, must be taken into account. For example, when data generate a high potential interest, such as information on corruption of a public figure, we can assume that persons seeking to find the identity of the person will be more motivated than they would be to re-identify more mundane data.<fnref n="16" /> This risk-based approach to anonymisation is consistent with Elliot et al.'s functional anonymisation, which takes account only of “empirically plausible [threats] rather than the logically possible ones” [10].

A last subject of debate is whether personal data that have been rendered anonymous become free from all legal constraints under a “release and forget” approach [10,14]. In the SRB case, the Advocate General emphasized that the SRB, as data controller, continued to owe duties to data subjects, including informing them of the transfer of data to Deloitte.<fnref n="17" /> This is consistent with the subjective approach focusing on a particular recipient of the data and a certain data environment. If the data are available for the whole world to see, then the anonymisation test would have to focus on all means reasonably likely to be used by any person in the world. As we will see below, this question becomes particularly important in the context of publicly available LLMs compared to LLMs limited to particular use environments. The debate between the objective and subjective approach, and between the absolute and risk-based approach has not been definitively settled, although the recent opinion of the EDPB on AI models seems to show consensus on the subjective and risk-based approaches [23]. In the remainder of this article, we will adopt the subjective and risk-based approaches, which we think better reflect the current case-law of the CJEU and the most recent EDPB opinion on AI models [23].<fnref n="18" />

The question of anonymisation is debated right now in the context of LLMs. In its 29 April 2024 complaint against OpenAI,<fnref n="19" /> NOYB asserts that ChatGPT is a filing system that processes personal data, and concludes that individuals have a right to access and rectify their data under the GDPR.<fnref n="20" /> It has been well documented that LLMs appear to memorize in their billions of parameters and weights much of the data seen during the training phase [24–32]. Certain kinds of prompts

<footnote n="13">¹³ GDPR, recital 26.</footnote>

<footnote n="14">¹⁴ CJEU, Gesamtverband Autoteile-Handel eV v. Scania, Case C-319/22 P 9 Nov. 2023, ; OC v. European Commission, Case C-479/22 P 7 March 2024.</footnote>

<footnote n="15">¹⁵ CJEU, Nowak v. Data Protection Commissioner, Case C-434/16 20 Dec. 2017, ECLI:EU:C:2017:994 point 35.</footnote>

<footnote n="16">¹⁶ OC v. European Commission, Case C-479/22 P 7 March 2024, ECLI:EU:C:2024:215 point 63.</footnote>

<footnote n="17">¹⁷ CJEU, Opinion of Advocate General Spielmann, European Data Protection Supervisor v. Single Resolution Board, Case C-413/23 P, 6 February 2025, ECLI:EU:C:2025:59, point 72.</footnote>

<footnote n="18">¹⁸ The United States's standard for anonymisation of health data is not that different from the GDPR's. The Code of Federal Rules states that data are anonymized if a person knowledgeable in the application of state of the art anonymization techniques “determines that the risk is very small that the information could be used, alone or in combination with other reasonably available information, by an anticipated recipient to identify an individual who is a subject of the information”. CFR, §164.514 (b)(1)(i).</footnote>

<footnote n="19">¹⁹ NOYB v. OpenAI, Complaint filed before the Austrian Data Protection Authority 29 April 2024 https://noyb.eu/en/chatgpt-provides-false-information-about-people-and-openai-cant-correct-it.</footnote>

<footnote n="20">²⁰ NOYB complaint, paragraph 6.</footnote>

<footnote n="8">⁸ Single Resolution Board v European Data Protection Supervisor Case T-557/20, 26 April 2023, ECLI:EU:T:2023:219.</footnote>

<footnote n="9">⁹ Gesamtverband Autoteile- Handel v Scania, Case C-319/22, 9 November 2023, ECLI:EU:C:2023:837.</footnote>

<footnote n="10">¹⁰ CJEU, Opinion of Advocate General Spielmann, European Data Protection Supervisor v. Single Resolution Board, Case C-413/23 P, 6 February 2025, ECLI:EU:C:2025:59, point 59.</footnote>

<footnote n="11">¹¹ OC v. European Commission, Case C-479/22, 7 March 2024, ECLI:EU:C:2024:215 point 63.</footnote>

<footnote n="12">¹² CJEU, Breyer v. Germany, Case C-582/14, 19 Oct. 2016, ECLI:EU:C:2016:779, point 46.</footnote>
can cause the model to reproduce personal information seen in the training data, except that the information may be false, resulting from hallucinations [27].

A discussion paper circulated by the Hamburg data protection commissioner considers that LLMs do not contain personal data under the GDPR's anonymisation test [33]. According to the paper, a successful privacy attack requires access to the original training data to validate the correlation revealed by the LLM after a prompt, such as “who lives at 121 Main Street?”. The output “John Doe lives at 121 Main Street” is only a probabilistic correlation. The Hamburg Commissioner says that the user has no way of knowing if the information is true or not. The research community may deploy more sophisticated means to validate that the information is indeed personal data contained in the training data and not a hallucination, but those means would be “disproportionate” for the average user according to the Commissioner, and might also be deemed unlawful.

The Hamburg Commissioner's analysis is noteworthy for two reasons. First, the Commissioner focuses on the means reasonably likely to be used by an average user, discounting the case of motivated research teams. This seems to contradict the CJEU's approach in the OLAF case which took into account not an average reader of the European Anti-Fraud Office's press release, but a motivated journalist with access to other information to fill in the gaps of a press release.<fnref n="21" /> Second, the Hamburg Commissioner suggests that a probabilistic correlation between the words “121 Main Street” and the words “John Doe” is insufficient linkage to constitute personal data, because the words John Doe could be totally fabricated. The Hamburg DPA contrasts this uncertain output with deterministic identifiers such as vehicle numbers or IP addresses, which the CJEU has confirmed are personal data in the hands of persons able to create a link with vehicle owners or internet subscribers. This raises an important question: if a model generates what appears to be personal data based on a probabilistic approach, what probability is sufficient for the output to constitute personal data? This question is connected with the debate on whether LLMs have a duty to provide “truthful” output [34]. Respect of privacy would push the LLM provider to encourage *untrue* output, whereas protecting end-users and third parties from false, misleading and defamatory content would push the LLM provider to encourage *truthful* output, thus highlighting another trade-off between privacy and other values [34]. The EDPB opinion on AI models [23] makes a distinction between LLMs designed to provide truthful personal data, and LLMs not intended to be used to provide personal data. The former naturally process personal data whereas for the latter, the question is more complex, requiring a case-by-case analysis.

The NOYB lawsuit, the literature on data leakage of LLMs, and the paper of the Hamburg Commissioner illustrate major differences in opinions. The EDPB opinion takes a middle ground, arguing that LLMs should be presumed in most cases to contain personal data, but that this presumption can be reversed if the data controller can show that given the particular context in which the LLM is trained and deployed, the likelihood of obtaining personal data from the model is insignificant. The EDPB mentions differential privacy, data filtering during the training stage, or at the output stage, as well as usage restrictions, as affecting the anonymisation test as applied to LLMs. The EDPB thus adopts a subjective approach, accepting that the use environment and population of potential users will affect the anonymisation question. The EDPB also adopts a risk-based approach, admitting that the identification risk need not be zero, but merely “insignificant” given means reasonably likely to be used by the relevant attacker [23].

The debate around LLMs highlights the importance of identifying different attack scenarios, evaluating the likelihood of their use, by whom, and then charting the probability of successful identification

as a function of each attack scenario [32]. As pointed out by the EDPB [23], an open-source LLM will naturally broaden the population of potential attackers, thereby making it more likely to fail the anonymisation test compared to a proprietary model whose population of users is restricted.

## 2.2. The appropriate technical and organisational measures test

As noted above, PETs including pseudonymisation, help satisfy the GDPR obligation to apply “appropriate technical and organisational measures” (ATOM) to protect personal data.<fnref n="22" />

The obligation to deploy ATOM flows from Articles 24, 25(1) and 32 of the GDPR, all three of which require measures adapted to risks. Article 24 imposes a general obligation to implement appropriate technical and organisational measures based on an analysis of risks for individual rights and freedoms. Article 25 states that appropriate measures such as pseudonymisation should be implemented at the time of the determination of the means for processing and at the time of the processing itself. Article 32 focuses on data security, requiring appropriate technical and organisational measures to ensure a level of security appropriate to the risk.

To design “appropriate” measures, the data controller must conduct a risk assessment, “taking into account the likelihood of the risks identified and their severity”.<fnref n="23" /> Based on these identified risks, the measures implemented by the data controller must take “into account the state of the art, the costs of implementation and the nature, scope, context and purposes of that processing”.²⁴

The question of what measures are “appropriate” has led to debate on whether data controllers have an obligation of ensuring successful protection (an obligation of result), or only an obligation of means [35–37]. In the context of Article 25, the EDPB has said that measures will be appropriate if they are effective in preventing the identified risks [38]. According to the EDPB, the cost of implementation may be taken into account, as long as the cost does not deteriorate effectiveness. This led in practice to a presumption that if a risk materializes, then protection measures were necessarily “inappropriate” [39].

The CJEU recently pushed back on this approach, pointing out, in the context of Article 32, that “appropriate” measures are not necessarily intended to eliminate all risks.<fnref n="25" />

Some residual risk is allowed, according to the Court. The appropriateness of measures should be judged based first on a concrete risk analysis, factoring in the severity and probability of identification. The appropriateness of mitigation measure should then be evaluated with regard to those identified risks, taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing.<fnref n="26" />

In Section 3.2, we propose using the Learned Hand formula, a method from law and economics to define negligence, to help answer the question what level of ATOM is “appropriate”, arguing that the appropriate level of care is the level that maximizes social welfare. In Section 4, we then apply the method to a concrete example involving differential privacy.

PETs such as differential privacy degrade the utility of the data, including for training machine learning models, resulting in a privacy-utility trade-off [1,10,12,40–42]. They can also reduce model fairness [13,43,44]. Current research on PETs, e.g. Singh et al. [45], attempts to reduce the extent of the trade-off, but in most situations, a trade-off remains.

A key question we will address below is whether adverse effects on model accuracy and fairness can be taken into account as a cost when

<footnote n="21">²¹ OC v. European Commission, Case C-479/22, 7 March 2024, ECLI:EU:C:2024:215 point 63.</footnote>

<footnote n="22">²² GDPR articles 24, 25 and 32.</footnote>

<footnote n="23">²³ CJEU, VB v Natsionalna agentsia za prihodite, Case C-340/21, ECLI:EU:C:2023:986 point 42.</footnote>

<footnote n="25">²⁵ Ibid., point 29.</footnote>

<footnote n="26">²⁶ Ibid., point 42.</footnote>
deciding on “appropriate” technical and organisational measures under our law and economics approach. We are not aware of any court or data protection authority in Europe taking loss of utility or fairness into account when assessing the appropriateness of ATOM, except in the context of processing personal data for scientific research under Article 89 of the GDPR. Article 89 GDPR requires “appropriate” safeguards. National laws and data protection authorities have taken account of the loss of utility when determining what protection measures are appropriate in the research context [46]. However, in other data processing contexts, including in machine learning, the loss of data utility and model fairness have not to our knowledge been explicitly considered when determining the “appropriate” level of ATOM. Because the AI Act requires accuracy and fairness for high-risk systems, the trade-off between privacy on the one hand, and accuracy and fairness on the other, will now have to be addressed more directly. Our law and economics approach provides one way for that to be done.

# 3. A law and economics approach to quantifying the anonymisation and ATOM tests

## 3.1. Quantifying the anonymisation test

Under classic economic analysis, we assume that economic agents, including an attacker seeking to identify individuals in data, are rational [13]. This means that an attacker will only deploy attack methods where the anticipated benefits of the attack exceed the costs [47,48]. The rational attacker will be selfish, i.e. she will not take into account societal harms or benefits, only benefits and harms for herself. Among the costs taken into account, the attacker will count her time investment, the equipment costs, the salaries of helpers, opportunity costs, and the risks of criminal or civil penalties [47,49]. Opportunity costs may include the ability to obtain the same information from a less costly source, for example if the information is available on the internet [10]. The CJEU Breyer case law says that we should take into account only “lawful” methods of attack.<fnref n="27" /> However, the economic analysis of cyberattacks generally includes unlawful methods in which the attacker factors in as a cost the likelihood and severity of punishment [47–49].

After calculating the costs, the rational attacker will calculate the anticipated benefits, which are equal to (i) the value to the attacker of obtaining the re-identified data, for example by selling them on the black market, or the reputation gain<fnref n="28" /> associated with the attacker successfully re-identifying data [47], multiplied by (ii) the probability of success given the protection technology used. The value of success multiplied by the probability of success is the expected pay-out. This pay-out will then be compared to the attacker's costs and will motivate the attacker to keep deploying efforts up to the point where her costs exceed the anticipated benefits. The efforts up to this threshold are “reasonably likely” to be used by a rational attacker.

The pay-out curve assumes a fixed level of protection measures. If we increase the level of protection measures, the probability of success goes down, and the level of justified effort by the attacker also goes down. Leeson and Coyne [49] illustrate this through a supply and demand curve. The upward slope of the supply curve in Fig. 1 shows that a higher expected payout on the y-axis justifies more attack efforts on the x-axis. The demand curve in this diagram (“DD”) shows the amount of hacking the defender is willing to tolerate, which can be translated into the price the defender would be willing to pay to prevent hacking. When the defender faces a high risk of loss, the defender will tolerate almost no hacking and will invest a high price in preventive

<figure><img src="image_4.png" /><figcaption>Fig. 1. The market model of hacking.</figcaption></figure>

Source: Leeson and Coyne [49]

measures. Conversely, when the loss risk is low, the defender will tolerate a higher quantity of hacking and pay less for protective measures. This is why the demand curve slopes down. The level of reasonably likely attack efforts will be situated at the intersection of the supply and demand curves.

Marsh et al. [11] present the risk of re-identification as the product of the probability that an attack will occur, and the probability of success should an attack occur. Our approach addresses the same factors, but from a different angle. We focus on the “means reasonably likely to be used” by an attacker based on the attacker’s perception of the likelihood of success given different levels of effort, and the amount of pay-out if the attack succeeds. This angle assumes that an attack can always be successful given enough effort, but that the maximum level of justified effort will be based on the attacker’s perception of his or her likely gain.

## 3.2. Quantifying the ATOM test

The ATOM test involves determining the level of “appropriate technical and organisational measures” under the GDPR. In the anonymisation test, we focused on the reasonably likely efforts of the attacker. In the ATOM test, we focus on the appropriate level of efforts of the defender. “Appropriate” efforts should be those we expect of a reasonable data holder, acting prudently with her interests in mind but also society’s. Contrary to the situation of the attacker, we expect the data holder to align her conduct with an ideal “reasonable person’s” conduct [10,50–52]. A reasonable person takes into account the harm her conduct might cause to others and adjusts her conduct accordingly. The incentive for the data holder to align her conduct with that of an ideal “reasonable person” is generally provided by liability mechanisms, administrative or criminal sanctions, and potential harm to reputation [48,51–53]. We do not expect the data holder to be altruistic. We expect her to factor in the cost, via risks of liability, other sanctions and reputational damage of not attaining the expected “reasonable person” level of care. In our case, sanctions for GDPR violation and bad publicity will motivate the data holder to bring her conduct to the “appropriate” or “reasonable” level [54]. (We are assuming here that the term “appropriate” is synonymous with “reasonable”.)

Under a law and economics approach, the appropriate level of protective measures is the level that maximizes social welfare, or put differently, minimizes social costs [52,53]. This level is attained at the point where two curves intersect. The first curve, usually referred to as “B” for burden, is the total costs resulting from each additional unit of protective measure, including not only the costs borne by the defender, but also costs to society of each additional unit of protection. This last point is particularly important, because protective measures can

<footnote n="27">²⁷ CJEU, Breyer v. Germany, Case C-582/14, 19 Oct. 2016, ECLI:EU:C:2016:779, point 73.</footnote>

<footnote n="28">²⁸ The HackerOne Leaderboard is an example of reputational gain sought by hackers.</footnote>
at some point reduce the utility of the defender's activity for society. For example, if a data holder is researching causes for cancer, adding additional noise into the dataset to protect privacy will reduce the utility of the data for cancer research. The societal costs resulting from this marginal decrease in utility should be included in the “B” curve as a cost of an additional unit of privacy protection. Intuitively, this is understandable because the most effective way to prevent a risk from occurring is to refrain from engaging in the activity in the first place. Yet to maximize social welfare, we often want useful but risky activities to continue, provided the risk is reduced to a reasonable level. As pointed out by Elliot et al. [10], the goal of protection measures “is not solely to produce safe data (and it cannot produce risk-free data); it needs also to produce useful data”.<fnref n="29" /> This goes to the heart of risk-based regulation.

The other line in the graph, called “PL”, shows the benefits derived from each additional unit of protective measure, which consists of the reduced risks of data disclosure. The benefit is typically calculated in terms of harm avoidance. L is the total harm resulting if the harm (e.g. a data re-identification) materializes, and P is the probability that the harm will materialize. What we generally see is that the first preventive measures have a large impact on reducing P, but that additional protection measures have a progressively weaker impact on P, such that at some point an additional unit of prevention will cost more than the expected additional benefit for society in terms of loss avoidance. For example, in the case of cancer research, replacing patient names and other directly identifying information with a code may reduce the probability of harm from 100% to 2% while retaining most of the utility of the data, but to reduce the harm from 2% to 0.1% may require much more costly privacy protection measures including those that reduce the utility of the data for researchers. The extra cost, for the data holder and for society, may not be worth the marginal decrease in risk. In its NAP decision the CJEU confirmed that the GDPR’s “risk-based” approach to protection does not attempt to provide 100% protection against risk.<fnref n="30" /> Some residual risk is expected and acceptable. Under a law and economics approach, the acceptable level of residual risk is situated where the two lines, B and PL, intersect. This method comes from a 1947 case in the United States decided by Judge Learned Hand,<fnref n="31" /> hence the name “Hand formula”. In his 1947 decision, Judge Learned Hand evaluated the level of care that should be exercised by a barge owner to prevent harm to other vessels:

> "Since there are occasions when every vessel will break from her moorings, and since, if she does, she becomes a menace to those about her; the owner's duty, as in other similar situations, to provide against resulting injuries is a function of three variables: (1) The probability that she will break away; (2) the gravity of the resulting injury, if she does; (3) the burden of adequate precautions. Possibly it serves to bring this notion into relief to state it in algebraic terms: if the probability be called P; the injury, L; and the burden, B; liability depends upon whether B is less than L multiplied by P: i.e., whether B > PL".<fnref n="32" />

The Hand formula is presented graphically in Fig. 2.

### 3.3. Limitations to the law and economics approach

The foregoing approaches to define what level of attack is “reasonably likely” and what protective measures are “appropriate” rely on simplifying assumptions. The most important and contestable assumption is that every harm or benefit, including harms to victims

<figure><img src="image_5.png" /><figcaption>Fig. 2. The Hand formula or PL-B curve.</figcaption></figure>

Source: Inspired by Posner [52].

and to society, can be quantified. Turning everything into monetary units is of course unrealistic and contestable. Critics of utilitarianism believe that individual human rights can never be traded off against collective interests in a cost-benefit analysis [55]. Economists have methods for converting almost anything, even individual rights, into measurable units, but every approach can be challenged as putting an arbitrary price tag on priceless values. Economists retort that their efforts are not designed to put a price tag on immeasurable rights, but to permit comparisons between different regulatory options to assess their relative impact and choose the one that is likely to yield the lowest net social costs.

The second major limitation of classic economic analysis is that it assumes that every actor is rational [50]. In reality, humans, including attackers, are often not rational. Consequently, our model of a “rational” attacker will not take into account the motivation of attackers who are not rational.

A third objection is that translating the conduct of a reasonable person into one that maximizes social welfare ignores many equitable concepts like fairness and distributive justice [50,56].

A fourth limitation relates to how we calculate the “B” curve in the Hand formula. Traditional law and economics approaches would take the drop in utility of data and fairness as a cost of additional protection measures. The privacy-utility tradeoff has been studied in [10,12,42], and is particularly relevant in activities that generate high social benefits such as scientific research [46]. But to our knowledge, except in the context of scientific research [46], decrease in utility has never been a factor taken into account by any data protection authority or court decision in interpreting ATOM under the GDPR.

### 4. Applying quantitative analysis to use cases

In this section we examine different scenarios for re-identification attacks and attempt to transform them into graphs to gain insight into the anonymisation and ATOM tests. An attack may involve randomness: either on the side of the attacker or defender, as in differential privacy. An attack can involve variable levels of effort: more or less queries, cross-referencing of more or fewer sources of information. In the following, we use a probabilistic framework. Let E be the variable capturing the level of effort.<fnref n="33" /> Let S be the random variable equal to 1 when the attack succeeds, 0 otherwise. We are

<footnote n="29">²⁹ In Elliot et al. [10], this quotation refers to anonymisation measures, not ATOM under the GDPR, but the underlying idea is that risks should be reduced to a reasonable level taking into account data utility.</footnote>

<footnote n="30">³⁰ CJEU, VB v Natsionalna agentsia za prihodite, Case C-340/21, ECLI:EU:C:2023:986 point 29.</footnote>

<footnote n="31">³¹ *United States v. Carroll Towing Co.*, 159 F.2d 169 (2d.Cir. 1947).</footnote>

<footnote n="32">³² *United States v. Carroll Towing Co.*, 159 F.2d 169, p. 173.</footnote>

<footnote n="33">³³ E can be expressed in terms of money, time spent, manpower...</footnote>
interested in the following statement: An attacker employing an effort of $\delta$ will be able to re-identify the data with probability at most $\epsilon$. This defines $(\epsilon, \delta)$-re-identification protection. In mathematical terms: $p(S = 1|E = \delta) \le \epsilon$. In our examples, we plot the function $f(x) = p(S = 1|E = x)$, to illustrate the relationship between attacker effort and probability of success. This expression of the function will help stakeholders discuss minimum and maximum values for $\epsilon$ and $\delta$, and narrow down the range of values that would be considered reasonable attack scenarios, i.e. “means reasonably likely to be used”.

Under the framework of $(\epsilon, \delta)$-re-identification protection, the data subjects can re-identify their own data with probability 1 and no effort. Including data subjects as potential attackers would thus break our framework. However, in practice, the data subjects gain nothing from re-identifying their own data, thus virtually not contributing to the risk. This is consistent with the CJEU's decision, in a case involving an allegedly anonymised press release, to exclude from its evaluation of reasonable attack scenarios a re-identification effort by someone who is already familiar with the person's professional background.³⁴

Some of the curves that we introduce in this section go slightly beyond trade-off curves like the R-U confidentiality map [9,12], in that we interpret R and U in terms of monetary amounts. Our definition of $(\epsilon, \delta)$-re-identification protection is related to the work of Marsh et al. [11], who examine re-identification of individuals in census data, in the context of SDC. Marsh et al. [11] examine the probability of re-identification as

$$p(\text{identification}) = p(\text{attempt})p(\text{identification}|\text{attempt})$$

The analysis of Marsh et al. [11] focuses on the presence of individuals across multiple census samples, and the probability that one individual may be linked from one sample to the other. This makes Marsh et al. [11] analogous to Narayanan and Shmatikov [57] (see below). The analysis of Marsh et al. [11] does not take effort into account, where we would examine the effort involved in obtaining additional identifying information. Instead, we examine the probability of re-identification as

$$p(\text{identification}) = \int_{x} p(E = x)p(\text{identification}|E = x)$$

where $x$ represents the amount of effort that the attacker invests. As will be further clarified below, we will assume that the attacker will aim towards a most profitable level of effort.

## 4.1. The Netflix challenge

The Netflix challenge is a well known test in which attackers attempted to re-identify an anonymous dataset provided by Netflix. The Netflix challenge is referred to in the EDPD's 2014 opinion on anonymisation techniques [18].

In the context of the Netflix challenge, Narayanan and Shmatikov [57] showed how it was possible to re-identify data that were seemingly anonymized. Their approach was probabilistic: under a property of the database called sparsity, it is possible to re-identify items from the database with high probability. The sparsity property itself is defined with probabilities. Under high sparsity, most items in the database have few neighbors, making it easier to identify an individual; under low sparsity, most items have many neighbors, making it more difficult to identify an individual. The approach of Narayanan and Shmatikov [57] directly translates to the framework of $(\epsilon, \delta)$-re-identification protection. Using Theorem 3 from Narayanan and Shmatikov [57], we can establish that, under $(1 - \epsilon - \delta, \epsilon)$-sparsity,<fnref n="35" /> $p(S = 1|E = m) = \frac{1}{1+\frac{N(1-\delta)^m}{m}}$. In that context, $N$ is the size of the database, $m$ is the number of features in the database that the attacker has access to, and $\delta$ is a measure of

<figure><img src="image_6.png" /><figcaption>Fig. 3. Expected reward of attack as a function of attacker's effort, for different settings of sparsity. The Netflix database+ is (.2,.1)-sparse.</figcaption></figure>

the data's sparsity — the larger $\delta$, the sparser the data and the more inherently easy it is to identify an individual. We interpret $m$ as a measure of effort: every additional attribute is costly to obtain, while the cost of running a re-identification attack is negligible.

In the Netflix challenge, $N = 480189$ and we estimate from Figure 1 in Narayanan and Shmatikov that $\delta = .7$ and $\epsilon = .1$. Further, since the reward for the Netflix prize was $1M, we assume that a successful re-identification attack would be worth $1M to the attacker. For the sake of the example, we further assume that the attacker would have to spend $65k to obtain an attribute. This assumption is artificial, but permits us to show how different levels of attack effort relate to probable pay-outs under the Netflix challenge. With these assumptions, Fig. 3 shows the attacker's expected reward as a function of the money (i.e. effort) spent on the attack. We additionally plot this same curve for different values of sparsity.

The characteristic S-shape of the curve remains unchanged, regardless of the data's sparsity. This shows how there is a critical threshold of effort, below which re-identification is hopeless, but above which it becomes very realistic. In Fig. 3, Attacker 1 is not deploying sufficient effort for any of the 3 sparsity settings, while Attacker 3 would be successful in any of the 3 settings. Attacker 2's effort is sufficient for (.1,.1)-sparsity, but not for the other 2 settings.

Further, it is reasonable to expect the attacker to only undertake attacks where the expected reward is greater than the cost involved. Those correspond to the region above the red dotted line. This means that the (.3,.1)-sparse database, being below the red dotted line, would never be attacked. The (.2,.1)-sparse database might be attacked by attackers having at least $780k available for an attack. For the (.1,.1)-database, that threshold would be at $585k.

By characterizing different attack profiles, and comparing how they rank in probability of success, it may be possible to narrow down the discussion on what attack profiles fall within the “means reasonably likely to be used” standard. Also, if we think of sparsity as a proxy for protective measures such as PETs, this graph shows how the level of protective measures affects the likelihood of attack. The blue line represents a level of protective measures that would make any level of attack non-cost-effective.

## 4.2. Personal data in a trained LLM

In our second example, we will examine whether an LLM contains personal data that it saw during training. To some extent, like most neural network models, LLMs can be said to memorize some of their

<footnote n="35">³⁵ Narayanan and Shmatikov [57] use the symbols $\epsilon$ and $\delta$. We keep with their notation, even if this creates an unfortunate confusion with the $\epsilon$ and $\delta$ of re-identification protection.</footnote>
training data. See for instance Huang et al. [58], who quantify the
ability of an LLM to reproduce email addresses seen in the training
data. Out of precaution, Samsung banned employee use of ChatGPT,
after Samsung source code was entered into the chatbot's interface.<fnref n="36" />
Exchanges with the chatbot are saved and potentially trained on, so
that there was a chance that subsequent users could extract the source
code by interacting with ChatGPT. This shows how confidential data
may be leaked by users of the chatbot, even if the underlying LLM was
trained on public data.

The question of whether a trained model such as an LLM contains
some of the personal data it saw during training has received consider-
able attention (see Section 2.1 and Carlini et al. [29]). The legal impacts
are significant because if a trained model retains personal data and
processes them to respond to queries, GDPR-related obligations will
apply, some of which will be difficult or impossible to comply with.<fnref n="37" />
By contrast, if the trained model does not contain personal data from
the training dataset, the GDPR will not apply, or will apply only insofar
as a prompt contains personal data. Following the example of The
European Data Protection Board [23], we exclude the case of an LLM
that is designed to reproduce truthful information of persons seen in the
training dataset. We also acknowledge that the simple test presented
below would have to be conducted on the LLM in its operational con-
figuration, taking into account technical and organisational limitations
imposed by the deployer [23].

The examples cited above amount to extracting an atomic piece of
information from an LLM. To simplify, we will consider the case where
an LLM was trained on non-public text containing the sentence "Jane
Doe (410 516 5201) is our office clerk at Acme, Inc". The goal of the
attacker is to extract the phone number from the LLM, having access
to a simple chatbot interface to the model. Instead of a phone number,
it could be other sensitive data such as physical address or financial
information, or even a snippet of code, as in the case of Samsung. The
attacker may ask the LLM to complete an input prompt. The LLM's
response is randomized, and the attacker can ask for as many samples
as desired. Here is the attack scheme:<fnref n="38" />

* The attacker asks the LLM to complete the prompt "The phone number of Jane Doe is" and runs *N* queries.

* The attacker calls up every number generated, verifying whether it is indeed that of Jane Doe.

Let $q$ be the probability of the LLM completing the prompt with the correct answer (“410 516 5201”). This quantity $q$ can be measured by the LLM provider. The probability of the correct number being in the sample is $1 - (1 - q)^N$. Assume that, roughly, the LLM produces $N$ distinct samples. The effort involved in this attack is to run $N$ queries, and to place $N$ phone calls. To simplify, the effort is proportional to $N$. Fig. 4 plots $f(x) = p(S = 1 | E = x)$.

This example shows why it is so difficult to determine whether
an LLM “remembers” personal data and can leak the data after an
attack. The main difficulty is that the LLM will fabricate (“hallucinate”)
outputs and may, or may not, leak the original personal data. Fig. 4
shows that in the Jane Doe example, it would take more than 800
phone calls to start to have a reasonable chance (> 50%) of obtaining
the right number. However, the right number might also emerge after
only one prompt and one phone call, although that is unlikely. This

<figure><img src="image_7.png" /><figcaption>Fig. 4. Probability of re-identification as a function of number of queries, with $q=0.1\%$.</figcaption></figure>

example raises the question of when a statistical probability is high
enough to be considered information about a person [22,42]. We do
not have an answer to this. Fig. 4 does not tell us the pay-out in case
an attack is successful. If the pay-out is high — for example the private
number of a politician or movie-star — then a high number of phone
calls can be expected.

The process of calling up many phone numbers is tedious. In this
contrived example, it is the only way of verifying information about
an ordinary person. For a public figure, there might be other ways of
verifying information produced by an LLM, for instance using publicly
available sources. A court or DPA would have to decide what level
of probability of re-identification is low enough such that the “risk of
identification appears in reality to be insignificant”.³⁹

4.3. Membership inference attack against facial recognition models

Our third example is a membership inference attack against a facial
recognition model: given an image, determine whether it represents
someone whose face was used to train the model. Facial recognition
models are very different from LLMs, both in terms of neural network
architecture, training data, and training objective. Attacks against each
of these models thus differ widely. The digital image given as input
to the attack was not used during model training, although the image
might represent the same person as one in the training dataset. Li et al.
[59] propose such a membership inference attack: assuming access to
some images with labels,<fnref n="40" /> infer membership, based on the embeddings
produced for these images by the model under attack. The membership
inference problem is a classification problem. For evaluation, Li et al.
[59] use two datasets (Market-1501 and PRID-2011), and measure
accuracy, precision and recall of the membership inference attack.
Further, the authors measure these metrics for different amounts of
training data available to the attacker, resulting in the paper’s Table
3.

This is an example of empirically measuring $P(S = 1|E \le \delta)$ — in
contrast to our previous 2 scenarios. Here, this probability corresponds
to precision: *given that the model has predicted that an image is from
someone whose face was used for training, how likely is it that this prediction*

<footnote n="36">³⁶ https://www.forbes.com/sites/siladityaray/2023/05/02/samsung-bans-chatgpt-and-other-chatbots-for-employees-after-sensitive-code-leak/.</footnote>

<footnote n="37">³⁷ NYOB v. OpenAI, Complaint filed before the Austrian Data Protection Authority 29 April 2024 https://noyb.eu/en/chatgpt-provides-false-information-about-people-and-openai-cant-correct-it.</footnote>

<footnote n="38">³⁸ We acknowledge that there are many different architectures and access modalities for LLMs (proprietary, open-source...). These and other factors impact how a given LLM can be attacked. Here, we make minimal assumptions, and thus lay out a simple attack.</footnote>

<footnote n="40">⁴⁰ There are 3 labels:
– member of the training dataset
– not member of the training dataset but image of a person member of the training dataset
– none of the above.</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 5. Precision of membership attack for varying amounts of samples from the training data available.</figcaption></figure>

Source: Data taken from Table 3 of Li et al. [59]

is correct? Table 3 in Li et al. [59] provides us with this precision measure, for varying amounts of data available to the attacker. We can assume that the amount of effort the attacker deploys is proportional to that amount of data — for instance, assuming a quantum of effort is involved in obtaining, legally or illegally, each data sample.

Fig. 5 shows the probability of success of the attack as a function of the number of examples available to the attacker. Li et al. [59] have attacked 2 models, each built from a distinct dataset. Neither of these models properly resists the inference attack — this is one of the paper’s points. The 2 curves nonetheless have distinctive shapes.<fnref n="41" /> For the training dataset called Market-1501, the probability of success of the attack starts out at 75% and steadily climbs up to 86% as more data becomes available to the attacker. In contrast, for models trained on PRID-1501, the probability of success of the attack stays between 69 and 71%. This suggests that the model trained on Market-1501 would likely never be considered anonymous under GDPR standards, whereas the model trained on PRID-1501 might qualify as anonymous, if a probability of re-identification of 70% is considered “insignificant”. A 70% probability is not as high as it sounds for a binary classifier, since the baseline probability for a binary choice is 50%, the same as flipping a coin.

### 4.4. Differential privacy to protect mobility data

Our last example involves differential privacy, where we examine both anonymisation, and the concept of “appropriate” technical measures based on different privacy budgets. Differential privacy mechanisms provide strong theoretical guarantees against privacy breaches, while enabling data science applications [60]. Examples of applications include protecting the personal data collected by the U.S. Census Bureau [61] and security metrics collected across personal computers [62]. Data poisoning would be another way of protecting against privacy breaches.

Here, we put forward a simple attack. Operators of public transportation collect data about users’ journeys as they tap their transit cards on the fare reader. This data can be used to plan train schedules and allocate resources to match users’ needs. As part of a push towards

open data for innovation, operators of public transportation may be required to publish transportation data. Without any anonymization, this data could be used to know a specific person’s past trips and infer future ones, such as those related to daily commutes. Some French cities have already put in place open transportation data initiatives.<fnref n="42" /> Here, we assume that the maintainers of the database have made it accessible to researchers to develop predictive algorithms, but have implemented a differential privacy mechanism to protect the data of individuals. Let us assume the attacker wants to infer whether John ends his daily commute at a specific train station, using a database of transportation records. The attacker thus runs a query targeted at John’s record in the database, and the query returns a binary answer.<fnref n="43" /> The differential privacy mechanism will add random noise to the answer before returning it to the attacker, so that there will always be doubt about the answer’s truthfulness.<fnref n="44" /> However, since the database is meant to be used to build machine learning algorithms, the parameters of the random mechanism are published openly. As a result, while an individual sample yields little information, an aggregate of samples might yield more. Ideally, a differential privacy mechanism should not allow an attacker to run the same query multiple times. But it is possible that the attacker has access to multiple user accounts, or that the same database is offered by multiple data holders, each running a different version of differential privacy. Here, we assume that the attacker finds a way to obtain multiple random samples, running the same query multiple times, but each additional query generates a cost. Thus the number of samples represents the cost of the attack.

Fig. 6 shows the probability of success of the attack as a function of the number of queries, for varying privacy budgets. The $\epsilon$ in $\epsilon$-differential privacy is the privacy budget. It measures how much privacy the data holder is willing to release about individuals in the database. The smaller the budget, the less information is available to potential attackers. In line with this intuition, Fig. 6 shows how, for a fixed effort, the probability of success decreases with decreases in the privacy budget. In addition, the blue curve is shallower than the orange curve, which is shallower than the green curve: when the privacy budget is smaller, the increase in success rate that an attacker gets out of each additional sample is smaller.

Let us extend this example to see whether we can determine what privacy budget corresponds to an “appropriate” level of technical measure under the ATOM test.<fnref n="45" /> To do this we need to use the Hand formula, with the two intersecting curves, B and PL. Let us consider a scenario where the attacker has a monetary incentive to find out information about individuals’ habitual locations. Typically, this information could be sold (illegally) to third parties. The attacker tries to infer individuals’ attributes by running queries against a differentially private database. As mentioned above, differentially private databases do not generally permit multiple queries, but in our example the attacker has found a way to get around this restriction at a fixed cost per query. A correct inference yields a net reward for the attacker, so the

<footnote n="41">⁴¹ This difference is due to the underlying dataset and the images used as input for the attack, since both models share the same training process and architecture. PRID-2011 is simply more vulnerable than Market-1501 to membership attacks.</footnote>

<footnote n="42">⁴² See https://www.data.gouv.fr/fr/organizations/regie-autonome-des-transports-parisiens-ratp/datasets/ and https://www.data.gouv.fr/fr/datasets/deplacement-mobilite-equipements-et-services/.</footnote>

<footnote n="43">⁴³ Even if all directly-identifying information has been removed, there might be a set of features that uniquely identifies John in the database, such as the combination of date of birth, sex and zip code. This uncommon combination can be queried, and is equivalent to querying directly for John’s record. For a discussion of this type of re-identification, see Marsh et al. [11] and Narayanan and Shmatikov [57].</footnote>

<footnote n="44">⁴⁴ Typically, the mechanism will flip two (biased) coins before answering: if the first coin lands on heads, it will answer truthfully. Otherwise, the answer is “yes” if the second coin landed on heads, else “no”. The higher the chance of the first coin landing on tails, the more private the database, as defined by differential privacy.</footnote>

<footnote n="45">⁴⁵ For the purposes of this example, we will assume that differential privacy is the only protective measure, and that there are no organisational measures.</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 6. Probability of success of a re-identification attack against differentially private databases as a function of the number of queries run by the attacker.</figcaption></figure>

higher the probability of a correct inference, the higher the expected reward. The attacker sets the level of effort so as to maximize expected pay-out, which is expected reward minus cost of effort.

The y-axis in Fig. 7 shows the attacker's net expected pay-out. Fig. 7 shows that, for a privacy budget of $\epsilon_C$, the pay-out is always negative, regardless of effort invested. This means that a reasonable attacker would not carry out such an attack. We interpret this as successful anonymisation under the GDPR.

With a privacy budget of $\epsilon'$, the attacker's pay-out stays negative but barely reaches the break-even point. We can say that the privacy budget of $\epsilon'$ is the borderline separating successful anonymisation from unsuccessful anonymisation. Anything above this budget is not successful anonymisation, anything below is successful anonymisation. We see that for higher values of the privacy budget (e.g. $\epsilon_B$ in Fig. 7), the attacker's pay-out is positive for a range of effort levels.

Fig. 7 helps us with the anonymisation test. We see that anonymisation is clearly achieved with a privacy budget of $\epsilon_C$. Now let us look at the ATOM test, taking into account the decrease in utility caused by differential privacy. Let us assume that this database could be used to design algorithms that help anticipate peaks in demand and needs for maintenance. The value of these predictive algorithms depends on their accuracy, which in turn is hampered by the privacy budget: the smaller the budget, the lower the algorithms' accuracy and thus utility [42]. For the sake of illustration, we assume a simple model of utility, with the burden of implementing differential privacy being inversely proportional to the privacy budget: for a large privacy budget, the burden goes to 0 because the dataset maintains its utility, while for a privacy budget of 0, the burden goes to a fixed number, representing the loss in utility of the data, for example the value that these predictive algorithms would bring to a single individual. Fig. 8 shows how the burden (the “B” curve in the Hand formula) changes with the privacy budget, and how the expected loss to the victim and data holder (the “PL” curve in the Hand formula) changes with the privacy budget. The latter is computed as the probability of a successful attack, multiplied by the damage to the victim and the data holder. The level of effort of a likely attack is that which yields an optimal pay-out for the attacker, and can vary as a function of the privacy budget. The optimal privacy budget is where both curves intersect, at $\epsilon''$. Fig. 8 shows the “B” and “PL” curves (see Section 3.2) for this attack on a differentially-private database. This gives us an idea of how to apply the Hand formula to differential privacy.

Fig. 8 has roughly 3 regions. As we saw from Fig. 7, for a privacy budget small enough, the database is successfully anonymised: for any privacy budget smaller than that, the attacker will not expect to profit

from the attack. This corresponds to the right side of the graph (region C), for $\epsilon < \epsilon'$. For a privacy budget between $\epsilon'$ and $\epsilon''$ (region B), the database is not successfully anonymised, because the rational attacker can reasonably expect to profit from the attack. But in this zone the burden of deploying differential privacy is higher than the expected loss, which means that the data holder, while not fully anonymizing the data, has deployed “appropriate” technical measures from a law and economics standpoint, thereby satisfying the ATOM test. For a privacy budget greater than $\epsilon''$ (the left side of the graph, region A), the database is not successfully anonymized, and the burden of deploying privacy enhancing technology being lower than the expected loss, the data holder is not compliant with the GDPR's requirement to deploy “appropriate” measures using the Hand formula.

To summarize, using the letters in Fig. 8:

(A) For $\epsilon > \epsilon''$, the data is neither anonymised nor sufficiently protected by the data holder, failing both the anonymisation test and the ATOM test.

(B) For $\epsilon \in [\epsilon', \epsilon'']$, the data is not anonymised, but sufficiently protected by the data holder, failing the anonymisation test but satisfying the ATOM test.

(C) For $\epsilon < \epsilon'$, the data is successfully anonymised.

In our example of transport data used for research in the public interest, the “B” curve slopes steeply up to the right, reflecting the large social cost associated with incremental losses in utility of the dataset for prediction. For another prediction problem, for example predicting what advertisement an internet user is likely to click on, it could be argued that the incremental loss of accuracy will correspond to lower social costs, resulting in a flatter B curve. The flatter B curve would move the intersection between B and PL to the right, meaning that more protective measures would be justified under the GDPR's ATOM test than would be the case of research with transportation data. This seems consistent with the GDPR's language that requires data controllers to consider the nature, scope, context and purposes of processing when evaluating the appropriateness of protective measures.<fnref n="46" />

## 5. Conclusions drawn from our examples

The question of whether a dataset is anonymised carries heavy consequences. An anonymised dataset can be shared with others, combined with other data, and used to train machine learning algorithms with few regulatory constraints on the recipient of the anonymised dataset. A non-anonymised dataset cannot be shared or otherwise processed without European General Data Protection Regulation (GDPR) compliance. Defining the frontier between anonymised data and non-anonymised data is difficult and different stakeholders will have different views on where the frontier should lie. The GDPR also imposes an obligation to apply ATOM to personal data, which depends on how one interprets the term “appropriate” in a given circumstance. Privacy Enhancing Technologies (PETs) play an important role, both in anonymisation and in ATOM.

We think that charting different attack efforts and probabilities on a graph can help stakeholders discuss whether data are sufficiently anonymised to satisfy the GDPR anonymisation test, and whether technical and organisational efforts are “appropriate” under the GDPR ATOM test. These graphs can help identify cases where all stakeholders agree that the dataset is anonymous, i.e. probability of re-identification near zero for almost any kind of attack, and cases where all stakeholders can agree that the dataset is not anonymous, i.e. a high probability of re-identification for reasonably foreseeable attack efforts. In between these two points, there will be a range of scenarios and probabilities that are non-consensual. In this gray zone, different stakeholders will

<footnote n="46">⁴⁶ CJEU, VB v Natsionalna agentsia za prihodite, Case C-340/21, ECLI:EU:C:2023:986 point 42.</footnote>
<figure><img src="image_10.png" /><figcaption>**Fig. 7.** Left: an attacker's bottom-line pay-out (expected reward minus cost of effort) as a function of effort invested, for different values of the privacy budget. $\epsilon_A$, $\epsilon_B$ and $\epsilon_C$ belong to the corresponding regions of Fig. 8. Right: an attacker's bottom-line payout, for different combinations of effort and privacy budget. Units are arbitrary.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Fig. 8.** Expected loss for victims and data holder; and burden of deploying differential privacy, as a function of privacy budget. The blue line is somewhat irregular because the attacker's optimal level of effort suffers from threshold effects. See Appendix A.3 for details.</figcaption></figure>

have different opinions on whether the probability of re-identification is sufficiently low for the data to be considered anonymous and whether ATOM are “appropriate”. The process of putting scenarios and probabilities into a graph will help people identify the parameters on which they disagree: for example whether they disagree on the “means reasonably likely” to be deployed in an attack, or on the probabilities that result.

To address the question of appropriate technical and organisational measures to protect personal data pursuant to the GDPR's ATOM test, we use an approach from law and economics to determine that a “reasonable” level of protective measures would be the level that maximizes social welfare, i.e. total social benefits minus total social costs. As pointed out by the CJEU,<fnref n="47" /> the GDPR's risk management approach does not aim to eliminate all risk, but to reduce risk to an acceptable level. Protective measures such as PETs can reduce data utility and impact model fairness, creating a privacy-utility tradeoff and privacy-fairness tradeoff. We argue that the loss of data utility should be taken into account as a social cost when determining the appropriate level of PETs, such as the appropriate level of the privacy budget in differential privacy. We recognize, however, that this approach has not been used to date by courts or data protection authorities (DPAs) in Europe outside the context of processing for scientific research under Article 89 of the GDPR.

The use cases and associated graphs in this article reveal several insights. First, in the Netflix challenge we saw that the probability of re-identification can resemble an “S” curve, with probability suddenly jumping from low to high once a certain attack effort is reached. This can simplify discussions on what an acceptable level of risk is, because the risk will go directly from very low (acceptable) to very high (unacceptable), avoiding the murky middle ground. We also see from the graphs that higher protective measures such as PETs, or lower sparsity, will move the probability curve to the right, requiring more attack efforts to reach the same level of probability of re-identification.

The “means reasonably likely to be used” by an attacker will depend on the anticipated pay-out for the attacker, which is linked to the value of re-identification for the attacker, the cost of attack efforts, and the probability of attack success given a fixed level of PETs. Where an attack is never cost-effective, data are anonymised. The Large Language

<footnote n="47">⁴⁷ CJEU Case C-C-340/21, point 29.</footnote>
Model (LLM) example illustrates the problem of not knowing whether the output is actual personal data. The main cost for re-identification for LLM output is verifying whether output, such as a phone number, is actual personal data or hallucinated data. This requires additional verification measures such as checking the original training dataset, or making phone calls, as in our example. The LLM example also shows that the likelihood of revealing personal data is always probabilistic. It is possible that the very first privacy attack will yield a successful hit, even though the chance of that happening is low.

The facial recognition example shows that the training dataset, and in particular its distribution, can significantly change the likely success of membership inference attacks. It also shows how, in the absence of a closed-form mathematical expression for $(\epsilon, \delta)$-re-identification protection, the probability of success of an attack can be measured empirically, pointing at the need for benchmarks to measure our $\epsilon$ at various levels of $\delta$, for existing models and datasets. Finally, the transportation data example shows how privacy budgets in differential privacy affect the likelihood of successful attacks. The graphs from this example permit us to identify three scenarios: one corresponding to successful anonymisation (privacy budget of $\epsilon'$ or less), one corresponding to unsuccessful anonymisation but technical measures that are “appropriate” (privacy budget between $\epsilon''$ and $\epsilon'$), and one corresponding to unsuccessful anonymisation and ATOM that are below “appropriate” (privacy budget above $\epsilon''$). We also show in this example how the loss of data utility can be transformed into monetary costs and used to help determine where a “reasonable” privacy budget would be situated.

Our approaches rely on simplifying assumptions, and are therefore far from the nuanced reasoning that would be applied by a DPA or court in a particular case. Our approach also takes into account fewer risk factors than those taken into account by scholars in the SDC field [9,10]. Our method presents nevertheless a straightforward way to estimate what attack efforts are “reasonably likely”, and what technical and organisational measures are “appropriate”, using classic law and economics tools. While the quantitative exercise we propose has many limitations, it can be an important tool for illustrating the effects of different levels of PETs, including impacts on data utility. It will also help the designers of PETs to better understand what it means to make personal data anonymous under the GDPR and how “appropriate” technical measures such as differential privacy might be gauged.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Appendix. Background to Section 4.4

This section describes how the figures in Section 4.4 were derived.

### A.1. Differential privacy mechanism

In our example, the attacker queries an $\epsilon$-differentially private database for a binary attribute. Before returning the answer, the differential privacy mechanism makes the following decision:

* With probability $\frac{e^c-1}{e^c+1}$, return the true answer.

* With probability $\frac{2}{e^c+1}$:

- With probability $\frac{1}{2}$, return 0.

- With probability $\frac{1}{2}$, return 1.

This results in an $\epsilon$-differentially private mechanism.

### A.2. Probability of re-identification

The attacker runs $n$ queries against the mechanism laid out above. In a properly implemented $\epsilon$-differentially private mechanism, query results would be computed once, then cached. As a result, the attacker's query $Q$ would always return the same result, and it would be impossible to extract the true answer.

Let us assume a more faulty environment, where it is possible to run $Q$ multiple times, with a different random draw every time. This could be achieved e.g. by running queries that are paraphrases of the original query; by querying different versions of the database $D$, each having their own implementation or cache of the differential privacy mechanism (with identical parameter $\epsilon$); or by creating multiple user accounts and running the same query against $D$. This could be achieved differently, depending on the implementation of the differential privacy mechanism. In any case, the effort or cost involved in the attack would be proportional to the number of queries run.

Running $n$ queries, $Q$ returns the value 0 for $N_0$ queries, and 1 for $N_1$ queries. $N_0$ and $N_1$ are random variables. Without loss of generality, let us assume that the correct value is 0. Let $\theta$ be the attacker's estimate of the attribute, based on $N_0$, $N_1$ and $\epsilon$. The maximum likelihood estimation of $\theta$ is $\theta^{MLE} = 0$ if $N_0 > N_1$ else 1, and we assume that this is what the attacker will do. The probability of success is then $p(S = 1 | E = n) = p(N_0 > N_1) = \sum_{k=0}^{\lfloor \frac{n}{2} \rfloor} p(N_1 = k, N_0 = n-k)$.

With independence assumption,

$$ 
\begin{aligned}
p(S = 1 | E = n) &= \sum_{k=0}^{\lfloor \frac{n}{2} \rfloor} \binom{n}{k} \left( \frac{2}{e^\epsilon + 1} \right)^k \left( \frac{e^\epsilon - 1}{e^\epsilon + 1} + \frac{2}{e^\epsilon + 1} \right)^{n-k} \\
&= \frac{1}{(1 + e^\epsilon)^n} \sum_{k=0}^{\lfloor \frac{n}{2} \rfloor} \binom{n}{k} e^{(n-k)\epsilon}
\end{aligned}
$$

For $\epsilon = 0$ (maximal privacy): $p(S = 1 | E = n) = \frac{1}{2^n} \sum_{k=0}^{\lfloor \frac{n}{2} \rfloor} \binom{n}{k} = \frac{1}{2}$ for odd $n$. This is the same as random guessing. Note that this setting is useless for data science, as it is impossible to learn anything about the true distribution.

For $\epsilon \to \infty$ (no privacy): the term $e^{n\epsilon}$ dominates, yielding $p(S = 1 | E = n) \to 1$, meaning that the attacker will almost surely get the true answer, regardless of effort.

### A.3. Attacker reward and effort

In Fig. 7, we show the attacker's reward as a function of effort. The effort is the number of queries $n$ times the cost of a single query, $C$. The reward is computed as $r_\epsilon(n) = R(p(S = 1 | E = n) - \frac{1}{2}) - Cn$ where $R$ is the reward of a successful attack. For the purpose of drawing the figure, we set $R = 300$ and $C = 1$. The offset of $\frac{1}{2}$ is meant to account for the base probability of the attribute: an attack against the database is only worth it if it raises the probability of a correct guess substantially above $\frac{1}{2}$. Note that $r_\epsilon(n)$ depends on $\epsilon$ because $p(S = 1 | E = n)$ does.

In Fig. 8, we compute the attacker's optimal level of effort. It is set to $n(\epsilon) = \arg\max_{n \in \mathbb{N}} r_\epsilon(n)$. Since we take the maximum over a discrete set, the function $f'(\epsilon) = p(S = 1 | E = n(\epsilon))$ can make sharp jumps.

### A.4. Hand formula

In Fig. 8, the expected loss to victim and data holder (blue curve) is given by $Lp(S = 1 | E = n(\epsilon))$, where $L$ is the loss incurred by the victim if an attack is successful. For simplicity, we set $L = R$.

The burden of deploying differential privacy (orange curve) is set to $B(\epsilon) = \frac{R_0}{1+\epsilon}$. To plot Fig. 8, we set $B_0 = 500$. In practice, the shape of $B(\epsilon)$ depends on the types of algorithms that can be developed with the data, and on how differential privacy impacts their utility.
References

[1] Gadotti A, Rocher L, Houssiau F, Creu AM, de Montjoye Y-A. Anonymization: The imperfect science of using data while preserving privacy. Sci Adv 2024;10.

[2] The European Data Protection Board. EDPB guidelines 01/2025 on pseudonymisation. 2025.

[3] Fischer-Hübner S, Hansen M, Hoepman JH, Jensen M. Privacy-enhancing technologies and anonymisation in light of GDPR and machine learning. In: Privacy and identity management. 2022.

[4] OECD. Emerging privacy-enhancing technologies. (351). 2023. URL https://www.oecd-ilibrary.org/content/paper/bf121be4-en.

[5] Information Commissioner's Office. Privacy enhancing technologies. 2024. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-sharing/privacy-enhancing-technologies/.

[6] Society R. From privacy to partnership - the role of privacy enhancing technologies in data governance and collaborative analysis. 2023. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-sharing/privacy-enhancing-technologies/.

[7] Biden JR. Executive order on the safe, secure, and trustworthy development and use of artificial intelligence. 2023.

[8] Chiang T. ChatGPT is a blurry JPEG of the web. New Yorker 2023.

[9] Duncan G, Elliot M, Salazar González JJ. Statistical confidentiality: Principles and practice. 2011. http://dx.doi.org/10.1007/978-1-4419-7802-8.

[10] Elliot M, O'Hara K, Raab C, O'Keefe CM, Mackey E, Dibben C, Gowans H, Purdam K, McCullagh K. Functional anonymisation of personal data and the data environment. Comput Law & Secur Rev 2018;34(2):204–21. http://dx.doi.org/10.1016/j.clsr.2018.02.001, URL https://www.sciencedirect.com/science/article/pii/S0267364918300116.

[11] Marsh C, Skinner C, Arber S, Penhale B, Openshaw S, Hobcraft J, Liesvesley D, Walford N. The case for samples of anonymised records from the 1991 census. J R Stat Soc: Ser A (Statistics in Society) 1991;154(2):305–40.

[12] Duncan G, Keller-McNulty S, Stokes L. Disclosure risk vs. data utility: the RU confidentiality map. Technical Report, (Technical Report 142). Duncan, George and Keller-McNulty, Sallie and Stokes, Lynne; 2004.

[13] Kesari A. The privacy-fairness-accuracy frontier: A computational law & economics toolkit for making algorithmic tradeoffs. Proc the 2022 Symp Comput Sci Law 2022.

[14] Ohm P. Broken promises of privacy: Responding to the surprising failure of anonymization. UCLA L. Rev 2009;57:1701.

[15] Selzer A, Woods DW, Böhme R. An economic analysis of appropriateness under article 32 GDPR. Eur Data Prot Law Rev 2021.

[16] Purtova N. The law of everything. Broad concept of personal data and future of eu data protection law. Law, Innov Technol 2018;10:40–81.

[17] WP136. EDPB opinion 4/2007 on the concept of personal data. 2007.

[18] WP216. EDPB opinion 5/2014 on anonymisation techniques. 2014.

[19] Leitner L, Maldoff G, Lee M. Anonymisation through separation: what recent cases teach us about the eu's anonymisation standard. Priv & Data Prot 2024;24:3.

[20] Finck M, Pallas F. They who must not be identified—distinguishing personal from non-personal data under the GDPR. Int Data Priv Law 2020;10(1):11–36, URL https://doi.org/10.1093/idpl/ipz026.

[21] Weitzenboeck EM, Lison P, Cyndecka M, Langford M. The GDPR and unstructured data: is anonymization possible? Int Data Priv Law 2022;12(3):184–206.

[22] Rubinstein IS, Hartzog W. Anonymization and risk. Wash L. Rev 2016;91:703.

[23] The European Data Protection Board. EDPB guidelines 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models. 2024.

[24] Jegorova M, Kaul C, Mayor C, O'Neil AQ, Weir A, Murray-Smith R, Tsaftaris A. Survey: Leakage and privacy at inference time. IEEE Trans Pattern Anal Mach Intell 2021;45:9090–108.

[25] Das BC, Amini MH, Wu Y. Security and privacy challenges of large language models: A survey. 2024. ArXiv, arXiv:2402.00888.

[26] El-Mhamdi E-M, Farhadkhani S, Guerraoui R, Gupta N, Hoang LN, Pinot R, Stephan J. On the impossible safety of large AI models. 2022. ArXiv, arXiv: 2209.15259.

[27] Nasr M, Carlini N, Hayase J, Jagielski M, Cooper AF, Ippolito D, Choquette-Choo CA, Wallace E, Tramèr F, Lee K. Scalable extraction of training data from (production) language models. 2023. ArXiv, arXiv:2311.17035.

[28] Carlini N, Tramèr F, Wallace E, Jagielski M, Herbert-Voss A, Lee K, Roberts A, Brown TB, Song DX, Erlingsson Ú, Oprea A, Raffel C. Extracting training data from large language models. In: USeNIX security symposium. 2020.

[29] Carlini N, Ippolito D, Jagielski M, Lee K, Tramèr F, Zhang C. Quantifying memorization across neural language models. In: The eleventh international conference on learning representations, ICLR 2023, kigali, rwanda, May 1-5, 2023. OpenReview.net; 2023, URL https://openreview.net/forum?id=TatRHT_1cK.

[30] Veale M, Binns R, Edwards L. Algorithms that remember: model inversion attacks and data protection law. Philos Trans Ser A, Math Phys Eng Sci 2018;376.

[31] Rosenthal D. Language models with and without personal data. 2024. https://www.vischer.com/en/knowledge/blog/part-19-language-models-with-and-without-personal-data/.

[32] Smith V, Shamsabadi AS, Ashurst C, Weller A. Identifying and mitigating privacy risks stemming from language models: A survey. 2023, ArXiv, arXiv:2310.01424.

[33] The Hamburg Commissioner for Data protection and freedom of information. Discussion paper: Large language models and personal data. 2024. https://datenschutz-hamburg.de/fileadmin/user_upload/HmbBfDI/Datenschutz/Informationen/240715_Discussion_Paper_Hamburg_DPA_KI_Models.pdf.

[34] Wachter S, Mittelstadt B, Russell C. Do large language models have a legal duty to tell the truth? R Soc Open Sci 2024;11. URL https://api semanticscholar org/ CorpusID:269166993.

[35] Rubinstein IS, Good N. The trouble with article 25 (and how to fix it): the future of data protection by design and default. Int Data Priv Law 2019;10(1):37–56, URL https://doi.org/10.1093/idpl/ipz019.

[36] Bygrave LA. Data protection by design and by default: Deciphering the eu's legislative requirements. Oslo Law Rev 2017;4:105–20.

[37] Selzer A. The appropriateness of technical and organisational measures under article 32 GDPR. Eur Data Prot Law Rev 2021;7:120–8.

[38] The European Data Protection Board. EDPB guidelines 4/2019 on article 25 data protection by design and by default, version 2.0. 2020.

[39] Christakis T. L'importance de la pseudonymisation à l'ère de l'intelligence artificielle: au-delà du binaire. 2024. https://cabinet-samman.com/2024/10/02/limportance-de-la-pseudonymisation-a-lere-de-lintelligence-artificielle-au-deladubinaire/.

[40] Pannekoek M, Spigler G. Investigating trade-offs in utility, fairness and differential privacy in neural networks. 2021. ArXiv, arXiv:2102.05975.

[41] Prokhorenkov D, Cao Y. Towards benchmarking privacy risk for differential privacy: A survey. Proc the 10th ACM Int Conf Syst for Energy-Efficient Build Cities, Transp 2023.

[42] Wu FT. Defining privacy and utility in data sets. Univ Color Law Rev 2013;84:1117.

[43] Agarwal S. Trade-offs between fairness and privacy in machine learning. In: IJCAI 2021 workshop on AI for social good. 2021.

[44] Calvi A, Malgieri G, Kotzinos D. The unfair side of privacy enhancing technologies: addressing the trade-offs between PETs and fairness. Proc the 2024 ACM Conf Fairness, Account, Transpar 2024.

[45] Singh T, Aditya H, Madisetti VK, Bahga A. Whispered tuning: Data privacy preservation in fine-tuning LLMs through differential privacy. J Softw Eng Appl 2024. URL https://apisemanticscholar.org/CorpusID:267149066.

[46] Kindt E, Fontanillo López CA, Bergholm J, Czarnocki J, Kanevskaia O, Herveg J, Benmayor A. Study on the appropriate safeguards under Article 89 (1) GDPR for the processing of personal data for scientific research. Technical report. 2021.

[47] Hui KL, Zhou J. The economics of hacking. In: Oxford research encyclopedia of business and management. 2020.

[48] Becker GS. Crime and punishment: An economic approach. J Political Econ 1968;76(2):169–217.

[49] Leeson PT, Coyne CJ. The economics of computer hacking. JL Econ & Pol'Y 2005;1:511.

[50] Landes WM, Posner RA. The Economic Structure of Intellectual Property Law. Harvard University Press; 1987.

[51] Grady MF. A new positive economic theory of negligence. Yale Law J 1983;92:2.

[52] Posner RA. Economic analysis of law, 9th. Aspen case book series, 2014.

[53] Shavell S. Foundations of economic analysis of law. The Belknap Press of Harvard University Press; 2004.

[54] Rubinstein I. Regulating privacy by design. Berkeley Technol Law J 2012;26:1409.

[55] Schroeder CH. Rights against risks. Columbia Law Rev 1986;86:495, URL https://api semanticscholar org/CorpusID:153269616.

[56] Wright RW. Hand, Posner, and the myth of the 'Hand Formula'. Theor Inq Law 2003;4.

[57] Narayanan A, Shmatikov V. Robust de-anonymization of large sparse datasets. In: 2008 IEEE symposium on security and privacy (SP 2008), 18-21 May 2008, oakland, california, USA. IEEE Computer Society; 2008, p. 111–25. http://dx.doi.org/10.1109/SP.2008.33.

[58] Huang J, Shao H, Chang KCC. Are large pre-trained language models leaking your personal information? In: Goldberg Y, Kozareva Z, Zhang Y, editors. Findings of the association for computational linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, p. 2038–47. http://dx.doi.org/10.18653/v1/2022.findings-emnlp.148, URL https://aclanthology.org/2022.findings-emnlp.148/.

[59] Li G, Rezaei S, Liu X. User-level membership inference attack against metric embedding learning. CoRR 2022. arXiv:2203.02077.

[60] Dwork C, McSherry F, Nissim K, Smith AD. Calibrating noise to sensitivity in private data analysis. J Priv Confidentiality 2016;7(3):17–51. http://dx.doi.org/10.29012/JPC.V7I3.405.

[61] Abowd JM. The U.S. census bureau adopts differential privacy. In: Guo Y, Farooq F, editors. Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2018, London, UK, August 19-23, 2018. ACM; 2018, p. 2867. http://dx.doi.org/10.1145/3219819.3226070.

[62] Erlingsson Ú, Pihur V, Korolova A. RAPPOR: randomized aggregatable privacy-preserving ordinal response. In: Ahn G, Yung M, Li N, editors. Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, scottsdale, AZ, USA, November 3-7, 2014. ACM; 2014, p. 1054–67. http://dx.doi.org/10.1145/2660267.2660348.