<figure><img src="image_1.png" /></figure>

HAL
open science

# On the Duality of Privacy and Fairness

Mário S. Alvim, Natasha Fernandes, Bruno D Nogueira, Catuscia Palamidessi, Thiago V A Silva

► To cite this version:

Mário S. Alvim, Natasha Fernandes, Bruno D Nogueira, Catuscia Palamidessi, Thiago V A Silva. On the Duality of Privacy and Fairness. CADE 2023 - International Conference on AI and the Digital Economy, Jun 2023, Venice, Italy. p. 46 - 48. [hal-04407491](https://hal.science/hal-04407491v1)

HAL Id: hal-04407491

https://hal.science/hal-04407491v1

Submitted on 20 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On the Duality of Privacy and Fairness
(Extended Abstract)

Mário S. Alvim¹, Natasha Fernandes², Bruno D. Nogueira¹, Catuscia Palamidessi³, and Thiago V.A. Silva¹

¹UFMG, Brazil

²Macquarie University, Australia

³Inria and LIX, École Polytechnique, France

## Abstract

When a machine learning model operates over data about individuals, there are two common concerns. On one hand, if the model’s output (i.e., its prediction) allows for information inferences about an individual’s sensitive attributes, we have a *privacy* issue. On the other hand, if the individual’s sensitive attributes can unduly influence the model’s output, we have a *fairness* issue. Recently, the interplay between these two concerns has gathered growing attention both in the scientific community and in society as a whole. In this work, we extend the framework of *quantitative information flow* to formally capture fairness and privacy as duals of each other, and give first steps toward a novel characterization of their relationship.

## 1 Introduction

The increasing ubiquity of machine learning (ML) has brought not only significant benefits to modern society, but also some pressing concerns. One such concern is that ML models can be *unfair*, in the sense that they may unduly use individuals' sensitive attributes (such as race or gender) to make decisions, thereby creating or reinforcing biases against certain groups. A second concern is that ML models may *not be private*, i.e., they may improperly leak individuals' sensitive attributes in the course of their execution. These issues, and their interrelationship, have gained increased attention [1, 2, 8, 9, 12, 19, 22].

In this work, we employ the framework of *quantitative information flow (QIF)* [4] to investigate the interplay between fairness and privacy in ML. QIF has been successfully applied to a wide range of privacy-related problems [5, 10, 11, 15–17], but, to the best of our knowledge, not yet to fairness. Here we report first steps in modeling fairness in QIF as a dual concept to privacy, and conjecture a formal trade-off between them.

## 2 Preliminaries

In this section, we briefly review some key concepts from QIF, ML, privacy, and fairness needed for this work.

**Quantitative information flow (QIF) and privacy.** QIF is concerned with measuring the amount of information that a system leaks about its (secret) inputs through its (observable) execution behaviour to an *adversary*, which is an entity trying to infer sensitive information.

Before the system is run, the adversary has some *a priori knowledge* about the secret values, represented by a *prior distribution* (or simply a *prior*) $\pi: D\mathcal{X}$ on the set $\mathcal{X}$ of possible values for the secret. (Given a set $S$, we denote by $D^S$ the set of all probability distributions over $S$.) We let $\pi_x$ denote the prior probability of secret value $x \in \mathcal{X}$. We can measure the *prior vulnerability* of prior $\pi$ as the adversary's probability

of guessing the secret value correctly in one try. This is formalised as *Bayes vulnerability*, defined as $V(\pi) = \max_{x \in \mathcal{X}} \pi_x$.

A system is modeled as an (*information-theoretic*) channel $C: \mathcal{X} \to D\mathcal{Y}$ mapping (in a possibly probabilistically way) secret inputs from $\mathcal{X}$ to observable behaviours from a set $\mathcal{Y}$. We let $C_{x,y}$ denote the probability of channel $C$ outputting $y \in \mathcal{Y}$ when its input is $x \in \mathcal{X}$. We assume that the adversary knows how the system works (i.e., the channel $C$) and, from that and the prior $\pi$, can derive a joint distribution $\pi \triangleright C: D(\mathcal{X} \times \mathcal{Y})$ on inputs and outputs of a system. More precisely, $(\pi \triangleright C)_{x,y} = \pi_x C_{x,y}$ for all $x \in \mathcal{X}$, $y \in \mathcal{Y}$. This joint represents the adversary's *a posteriori knowledge* about the secret (i.e., after the system is run), and from it we can compute the *posterior Bayes vulnerability* of the secret as $V[\pi \triangleright C] = \sum_{y \in \mathcal{Y}} \max_{x \in \mathcal{X}} (\pi \triangleright C)_{x,y}$. This value represents the adversary's expected probability of guessing the secret correctly in one try after the system is run.

The *Bayes (information) leakage* caused by system $C$ under prior $\pi$ can be quantified as the ratio by which the system's execution increases the secret's Bayes vulnerability, and it is given by $\mathcal{L}(\pi, C) = V[\pi \triangleright C]/V(\pi)$. This leakage from secret inputs to observable outputs is a measure of the system's *privacy*.

**Machine learning and fairness.** In a *classification problem*, a ML model receives as input a vector of *features* for an individual and outputs a prediction for another feature of this individual. As usual in the fairness literature, we consider *binary classification problems*, in which the model’s input is a single, binary sensitive attribute, and its output is a single, binary predicted class. More precisely, the input is a sensitive attribute $x$ taking values $s_0/s_1$ indicating, resp., whether or not the individual belongs to a protected group (e.g., an ethnic minority/majority), and the output $y$ is a class prediction taking values $+/-$ indicating, resp., a desirable or non-desirable outcome (e.g., acceptance/rejection to a university application). The unfairness of a model is often measured as *statistical disparity* [7, 14, 20, 21, 23], i.e., the dependence level of the classification outcome on the sensitive attribute.
**Acknowledgements.** Mário S. Alvim, Bruno D. Nogueira and Thiago V. A. Silva were partially funded by CNPq, CAPES, and FAPEMIG. Catuscia Palamidessi was funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme. Grant agreement № 835294.

## References

[1] Sushant Agarwal. Trade-offs between fairness, interpretability, and privacy in machine learning. Master's thesis, University of Waterloo, 2020.

[2] Mohammad Al-Rubaie and J Morris Chang. Privacy-preserving machine learning: Threats and solutions. *IEEE Security & Privacy*, 17(2):49–58, 2019.

[3] Mário S. Alvim, Miguel E. Andrés, Konstantinos Chatzikokolakis, Pierpaolo Degano, and Catuscia Palamidessi. On the information leakage of differentially-private mechanisms. *Journal of Computer Security*, 23(4):427–469, 2015.

[4] Mário S. Alvim, Konstantinos Chatzikokolakis, Annabelle McIver, Carroll Morgan, Catuscia Palamidessi, and Geoffrey Smith. *The Science of Quantitative Information Flow*. Springer, 2020.

[5] Mário S. Alvim, Natasha Fernandes, Annabelle McIver, Carroll Morgan, and Gabriel H. Nunes. Flexible and scalable privacy assessment for very large datasets, with an application to official governmental microdata. *Proc. of the 22nd Privacy Enhancing Technologies Symposium (PoPETS 2022)*, 2022(4):378–399, 2022.

[6] Mário S. Alvim, Natasha Fernandes, Annabelle McIver, and Gabriel H. Nunes. On privacy and accuracy in data releases (invited paper). In *Proc. of the 31st International Conference on Concurrency Theory (CONCUR 2020)*, volume 171 of *LIPIcs*, pages 1–1:1–1:18, 2020.

[7] Solon Barocas and Moritz Hardt. NIPS 2017 Tutorial on Fairness in Machine Learning, 2017.

[8] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In *Proc. of the 2nd Conference on Fairness, Accountability and Transparency (FAT 2019)*, pages 77–91. PMLR, 2018.

[9] Hongyan Chang and Reza Shokri. On the privacy risks of algorithmic fairness. In *Proc. of the 6th IEEE European Symposium on Security and Privacy (EuroS&P 2021)*, pages 292–303. IEEE, 2021.

[10] Konstantinos Chatzikokolakis, Natasha Fernandes, and Catuscia Palamidessi. Comparing systems: Max-case refinement orders and application to differential privacy. In *Proc. of the 32nd IEEE Computer Security Foundations Symposium (CSF 2019)*, pages 442–457, 2019.

[11] Giovanni Cherubin, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. F-BLEAU: fast black-box leakage estimation. In *Proc. of the 40th IEEE Symposium on Security and Privacy (S&P 2019)*, pages 835–852, 2019.

[12] Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning. *Communications of the ACM*, 63(5):82–89, 2020.

[13] Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic Decision Making and the Cost of Fairness. In *Proc. of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017)*, page 797–806, 2017.

[14] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In *Proc. of the 3rd Innovations in Theoretical Computer Science Conference (ITCS 2012)*, page 214–226, 2012.

[15] Danilo Fabrino Favato, Gabriel Coutinho, Mário S. Alvim, and Natasha Fernandes. A novel reconstruction attack on foreign-trade official statistics, with a brazilian case study. *Proc. of the 22nd Privacy Enhancing Technologies Symposium (PoPETS 2022)*, 2022(4):608–625, 2022.

[16] Natasha Fernandes, Mark Dras, and Annabelle McIver. Processing text for privacy: an information flow perspective. In *Proc. of the 22nd International Symposium on Formal Methods (FM 2018)*, pages 3–21. Springer, 2018.

[17] Mireya Jurado, Catuscia Palamidessi, and Geoffrey Smith. A formal information-theoretic leakage analysis of order-revealing encryption. In *Proc. of the 34th IEEE Computer Security Foundations Symposium (CSF 2021)*, pages 1–16. IEEE, 2021.

[18] Faisal Kamiran, Indrè Žliobaitė, and Toon Calders. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. *Knowledge and Information Systems*, 35:613–644, 2013.

[19] Mohammad Mahdi Khalili, Xueru Zhang, Mahed Abroshan, and Somayeh Sojoudi. Improving fairness and privacy in selection problems. *Proc. of the 35th AAAI Conference on Artificial Intelligence (AAAI 2021)*, 35:8092–8100, 05 2021.

[20] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In *Proc. of the 31st International Conference on Neural Information Processing Systems (NIPS 2017)*, page 4069–4079, 2017.

[21] Camelia Simoiu, Sam Corbett-Davies, and Sharad Goel. The problem of infra-marginality in outcome tests for discrimination. *The Annals of Applied Statistics*, 11(3):1193–1216, 2017.

[22] Tianqing Zhu, Dayong Ye, Wei Wang, Wanlei Zhou, and Philip S. Yu. More than privacy: Applying differential privacy in key areas of artificial intelligence. *IEEE Transactions on Knowledge and Data Engineering (TKDE)*, 34(6):2824–2843, 2020.

[23] Indre Žliobaite. On the relation between accuracy and fairness in binary classification. *CoRR*, abs/1505.05723, 2015.