Individual Fairness of Data Provider Regarding Privacy Risk and Gain
Authors: Toshiki Shibahara
Takayuki Miura
Masanobu Kii
Atsunori Ichikawa
Venue: Submitted to ICLR 2023 (2023)
Keywords: differential privacy
privacy-preserving machine learning
lower bound

Abstract: Fairness and privacy risks are important concerns of machine learning (ML) when deploying ML to the real world. Recent studies have focused on group fairness and privacy protection, but no study focuses on individual fairness (IF) and privacy protection. In this paper, we propose a new definition of IF from the perspective of privacy protection and experimentally evaluate privacy-preserving ML based on the proposed IF. For the proposed definition, we assume that users provide their data to an ML service and consider the principle that all users should obtain gains corresponding to their privacy risks. As a user's gain, we calculate the accuracy improvement on the user's data when providing the data to the ML service. We conducted experiments on the image and tabular datasets using three neural networks (NNs) and two tree-based algorithms with differential privacy guarantee. The experimental results of NNs show that we cannot stably improve the proposed IF by changing the strength of privacy protection and applying defenses against membership inference attacks. The results of tree-based algorithms show that privacy risks were extremely small without depending on the strength of privacy protection but raise a new question about the motivation of users for providing their data.

Under review as a conference paper at ICLR 2023
INDIVIDUAL FAIRNESS OF DATA PROVIDER REGARDING PRIVACY RISK AND GAIN
Anonymous authors Paper under double-blind review
ABSTRACT
Fairness and privacy risks are important concerns of machine learning (ML) when deploying ML to the real world. Recent studies have focused on group faimess and privacy protection, but no study focuses on individual fairness (IF) and pri- vacy protection. In this paper, we propose a new definition of IF from the per- spective of privacy protection and experimentally evaluate based on the proposed IF. For the proposed definition, we assume that users pro- vide their data to an ML service and consider the principle that all users should obtain gains corresponding to their privacy risks. As a user's gain, we calculate the accuracy improvement the user's data when providing the data to the ML service. We conducted experiments on the image and tabular datasets using three neural networks (NNs) and two tree-based algorithms with differential privacy guarantee. The experimental results of NNs show that we cannot stably improve the proposed IF by changing the strength of privacy protection and applying de- fenses against membership inference attacks. The results of tree- rre-based algorithms show that privacy risks were extremely small without depending on the strength of privacy protection but raise new question about the motivation of users for providing their data.
MI.
INTRODUCTION
As machine learning (ML) services trained with users' data become increasingly popular, privacy risks of memorizing training data have been gaining attention (Shokri et al. 2017, Jagielski et al.) 2020 Nasr et al., 2021; Malek Esmaeili et al., 2021). To prevent privacy leakage through trained models, pnrvacy-preserve standard. For example, DP-SGD (Song et al. 2013 Abadi et al. 2016) is used for training neural networks (NNs) based on stochastic gradient descend (SGD) with DP guarantee, and DPBoost (Li
ML based on differential privacy (DP) (Dwork et al., 2006) is de facto
ct al. 2020) and DPXGBoost (Grislain & Gonzalvez 2021 are used for training tree-based models with DP guarantee.
When applying ML to the real world, fairness is another important concern about ML. Recent studies have begun focus both privacy protection and fairness: the difference in the effect of DP on
majority and minority groups (Bagdasaryan et al.] 2019: [Pujol et al.) 2020; Farrand et al., 2020
Tran al. 202I), the difference in vulnerabilities against membership inference attacks (MIAs)
between majority and minority groups (Zhang et al. 2020 Zhong et al. 2022), and methods for guaranteeing both group fairness and DP (Xu et al.) 2019 (020). All these
These studies have focused on group fairness, i.e., fairness between majority and minority groups Assuming situations where users decide whether to provide their data to ML services, individual fairness (IF), i.e., fairness between individual users, is also important for the decision. However, no study has focused on IF and privacy protection.
In this paper, we investigate Siivacy-preseving ML from the perspective of both IF and privacy protection. To this end, we propose a new definition of IF from the perspective of privacy protection and experimentally evaluate privacy-preservee provide their data to an ML service, we define the proposed IF based on the principle that all users should obtain gains corresponding to their privacy risks. Furthermore, we discuss the relationship between the proposed IF and prior IF for classification and validate the proposed IF using synthetic data.
ML based on the proposed IF. Assuming that users

---

Under review as a conference paper at ICLR 2023
We extensively evaluate srivacy-rreeervngg ML in terms of the proposed IF. Using two image datasets, we evaluate a six-layer convolutional NN (CNN) and ResNet18 (He et al. 2016) trained with DP-SGD (Song et al. 2013 Abadi et al. 2016). Using two tabular datasets, evaluate a five- layer fully connected NN trained with DP-SGD, DPBoost (Li et al. 2020), and DPXGBoost Griss user's privacy risk, calculate lower bound of DP parameter e (Jagielski et al., 20200] Malek Esmacili et al. 2021). As a user's gain, we calculate the accuracy improvement on the user's s data when providing the data the ML service. Since the accuracy improvement means that the utility of the ML service increases for the user, we can regard the accuracy improvement, i.e., the utility iccrasse, as the user's gain.
lain & Gonzalvez 2021). In the evaluation,
The results were different for NNs and tree-based algorithms. The main findings are as follows.
The results of NNs show that unfairness in terms of the proposed IF was large without de- pending on the strength of privacy protection because some users' gains were small com- pared with their privacy risks. These results show that we cannot improve the proposed IF by adjusting the strength of privacy protection.
We further evaluated the proposed IF when applying defenses against MIA to NNs. No de- fense improved fairness without depending on the settings (i.c., datasets, NNs, and strength of privacy protection), and fairness was degraded by the defenses in some settings. These results show the need for a method that stably improves the proposed IF of NNs without depending on the settings.
The results of tree-based algorithms show that privacy risks and gains were extremely small without depending on the strength of privacy protection. For tree-based algorithms, the proposed IF does not seem to be important, but these results raise new question about the motivation of users for providing their data. For example, some users are unwilling to provide their data if their data do not improve the ML service.
PRELIMINARIES
Individual fairness. IF is main concept of algorithmic faimess along with group faimesss. IF is proposed for the classification task based on the principle "similar data should be classified similarly" (Dwork et al., 2012). Let input space be V, set of output classes be A, probabil- ity distributions over output classes be A(A), a mapping from an input to an output, i.e., an ML model, be M V - (AA), and distance in input and output space be d : V x V + R and D: ((A) x A(A) - IR. If an model is a Lipschitz mapping, the model satisfies the principle of IF. Definition (Lipschitz mapping). A mapping M : V + A(A) satisfies the (D, (D,d)-Line di-Lishchitz prop- erty if for any x, y E V. the following holds:
D(M(2), M()) S d(z,v/
d and D need to be designed for each task. An example of d a Mahalanobis distance without using features correlated with sensitive attributes such as races and genders.
Another definition based on the same principle is proposed by relaxing the Lipschitz property.
Definition 2 (e-ô-IF (John et al. 20200). A mapping M is e-d-individually fair if for all xr y such that d(x y) S 6, the following holds:
M(x) M()) 6.
Note that in practice, task-specific loss needs to be considered in addition to these definitions of IF for building a fair and accurate model.
Differential privacy. DP (Dwork et al., 2006) is standard definition of privacy protection for statistical data analysis. In DP, we consider neighboring datasets Do and D; differing by only one sample. An example is adding one sample (x', t/') to Do make D,, i..., D₁ Do U {(x',())
Definition 3 (Differential Privacy). A randomized mechanism M : D - IR is (e, -iifffrrrettiallyy private if for any neighboring datasets Do. D, and for any output range C IR, the following holds:
Pr(M(Da) E $] < e Pr(M(Di) E 8.
2

---

f M(D,) Under review as a conference paper at ICLR 2023
A small constant, e.g., 10-5, is typically used for 6, and represents the privacy risk of the mecha- nism. Privacy-preserving ML guarantees an upper bound of e with theoretical analysis.
Lower bound of DP parameter e. From the practical perspective, the lower bound of the DP parameter c is empirically studied by in-
Attacker
Trainer
stantiating attackers against privacy-preserving
D1 Do ((x',y)
Dg. D4
ML (Jagielski et al. 2020, Nasr et al. 202T]
bc $ {0.1)
Malek Esmaeili al. 2021). We explain the
basic idea of these methods following the prior
lt {((('))))
work (Jagillski et al., 2020). This method is based on a game of an attacker and trainer shown in Fig. First, the attacker prepares two neighboring datasets Do and Dj. Here we con- sider D2 is prepared by adding a target sample (x',y') ) to Do. Second, the trainer randomly se- lects one of the neighboring datasets Do, where b is the index of the selected dataset. The trainer builds the model f using D, with a privacy- preserving MI. algorithm M and returns the loss = {(((x') (xx)) ,/y) to the attacker. Third, the attacker predicts which dataset is used training with an algorithm A(Do. D1,() and sends the prediction to the trainer. Since a small loss indicates that the sample (æ', 3/) is used in training, a commonly used algorithm predicts that the dataset is D2 if the loss is less than the threshold. Finally, the trainer checks if the prediction is correct.
b' - A(D,.
b'
chock if b' b
Figure 1: Game for calculating lower bound of e
To calculate the lower bound, this game is repeated many times, e.g., 1,000 times. If M (c, differentially private. the false positive rate (FPR) and false negative rate (FNR) of the games are FNR S 1 and FNR CFPR SPRS1-1
bounded (Kairouz et al. 2013) by FPR
Given 6, the maximum satisfying the above inequalities is the empirical c. Its lower bound (LB is defined using the upper bounds of FPR and FNR calculated with the Clopper-Pearsee method (Clop- per & Pearson 1934):
1 - 5- FPR 1 1 8 1-8-PNRUE FNRUB
LLB max log
log
FNRUB
FPRUB
Assuming that the target sample (æ', 3²) is data provided by user, can estimate the user's privacy risk by ELB.
3 INDIVIDUAL FAIRNESS FOR PRIVACY PROTECTION
We propose a new definition of IF from the perspective of privacy protection assuming that users provide their data an ML service. Additionally, we discuss the relation between the proposed IF and the prior IF for the classification task and validate the proposed definition using synthetic data.
3.1 PROBLEM SETTING
Before describing the proposed IF, we explain the ML service and user that we assume in this paper.
Machine learning service. We assume service using an ML model trained with users' data. The service accepts inputs from users and returns the predictions made by the model. In parallel with running the service, the service provider continues to collect users' data and add them to the training dataset. different users have different traits of data, data needs to be collected for accurately predicting new users' data. Regarding data collection, provide their data to the service or not. For example, when they start to use the service, they are asked whether they consent to share their data. If they consent, their data will be shared with the service and added to the training dataset. Examples of the services are facial expression recognition, handwritten text recognition, product recommendation, and medical diagnosis.
assume that users can select whether to
User. We assume that users the above ML service expect that their data are accurately classified by the service. The users decide whether to provide their data depending on the expected privacy
3

---

Under review as a conference paper at ICLR 2023
risks. If the users consider that the risks are high, they decide not to provide their data. If the users consider that the risks are low, they decide to provide their data.
The typical privacy risk is caused by the memorization of the ML model. The memorization leaks information on the training dataset from the outputs of the model. Such privacy risk is empirically measurable by calculating the lower bound of DP parameter as described in Section 2 In this paper, we define IF regarding privacy protection when users provide their data.
3.2 PROPOSED DEFINITION OF INDIVIDUAL FAIRNESS
We define IF between users who provide their data to the ML service from the perspective of privacy protection. Since a privacy risk, i.e., the lower bound of c, can be calculated with the method described in Section 2 IF regarding the privacy risks could be defined by referring to IF for the classification task. Two naive definitions are as follows.
A training algorithm is individually fair if users having similar data face similar privacy risks. This definition is inadequate because a difference in privacy risks can be large if users' data are dissimilar. For example, users having outlier data may have to tolerate high risks despite the fact that users having ordinary data face low risks.
A training algorithm is individually fair if the difference in privacy risks between any pair of users is small. This definition can be satisfied by reducing privacy risks with privacy-preserving ML because the differences are small if all users' privacy risks are small. However, strong privacy protection with DP is known to degrade classification per- formance (Abadi et al. 2016). For this reason, this definition seems to be impractical.
As described above, IF for privacy protection cannot be adequately defined in naive ways. To solve this problem, we admit that there are differences in privacy risks between users and define IF by additionally considering gains that users obtain in response to providing their data. We consider that training algorithm is individually fair if all users obtain gains corresponding their privacy risks. In other words, an algorithm is fair if users facing high privacy risks obtain large gains and if users facing low privacy risks obtain small gains. In contrast, algorithm is unfair if users obtain much larger or smaller gains than expected from their privacy risks.
In this paper, we calculate user's gain by the accuracy improvement on the user's data. Since the user expects that their data is accurately classified by the ML service, the accuracy on the user's data corresponds to the utility of the service. If the accuracy is improved by providing the data, we can regard the accuracy improvement, i.c., the utility increase, as the user's gain. Note that we discuss other types of gains in Section 5 We estimate the accuracy improvement using shadow models. We use a dataset Do already collected by the service and a user's sample (I), We train ny shadow models {fout]] 1n. using Du, i.e., without the user's sample, and n shadow models {fin Hi using Do U {(I,, 3:) i.e., with the user's sample. Then we count Cout and can: the number of shadow models accurately predicting the user's sample (æ:, y:) among {Sort 1n, and 1., [[n)) respectively. The accuracies on the sample (æ;; Us) when models are trained without and
(Jin
with (a., y1) are estimated by Sem and fin, The accuracy improvement, i.c., the gain, is calculated by 9. = Sim Cams, Note that we use the accuracy improvement instead of the loss decrease because the utility of the ML service does not increase even if the loss decreases unless the falsely predicted user's sample becomes accurately predicted. As the user's privacy risk Tt, we calculate the lower bound of using (z), y/) as the target sample.
Using the above users' privacy risks and gains, we define IF for privacy protection. Since admit the difference in privacy risks between users, we focus on the difference in the tradeoff of privacy risks and gains. Specifically, we consider that training algorithm is individually fair if all users' tradeoffs are similar. We denote privacy risks and gains of users U = {u; Jm as R = {ri} and G = {" respectively. To evaluate the tradeoff, we normalize R and G that their means and variances are 0 and 1, respectively. The normalized risk and gain of the user are denoted as of
rf r:ード= and g: = 2:-1',, where jr and B are means of R and G, and o² and og are variances completely correlated with risks and all users'
G,
of R and G. In an ideally fair situation, gains tradeoffs are the same. In this case, we have gf = r for all users. Based on this insight, we define IF by focusing the difference between ideal gain (i.e., r() and actual gain (i.e., 9.).
4

---

in the Under review as a conference paper at ICLR 2023
5.0
4.0
Unfairness
Unfaireess
6.0.
2.0
10
15.0
0.0
-1.
10.5
0.0
0.5
1.0
00
10
0.0
0.1
0.3
0.5
0.7
0.9
0.3
0.3
0.5
16.7
(a)
(b)
(c)
(d)
Figure 2: Unfairness and distribution of dę. (a) and (c) show the mean and standard deviation of unfaimeess calculated 100 times. (b) and (d) show the distribution of d, calculated using 100
generated samples.
Definition 4 (6-IF regarding privacy risks and gains), Let = {r() and G' { be the normalized privacy risks and gains so that their means are 0 and variances are 1. A training algorithm is o-individually fair if for all users u, E U, the following holds:
1 r - 6.
When G' is completely correlated with Rt, we have 8 = 0. d expected increase as the correlation becomes weaker. When G" is inversely correlated with R', we have large ó. Note that even if most users have gains corresponding to their risks, if only one user has a small or large gain compared with the risk, we have large ó. This is because definition is based the worst-case same way as Definition and Definition of IF for the classification task.
Based on this definition, we can evaluate the unfairness å a training algorithm by using empirically calculated Rt and G²:
8 max 99 r/l. U
Additionally, we can investigate a training algorithm in detail in terms of the proposed IF using each user's difference between the gain and risk: d; g
3.3 RELATION TO 6-5-IF
Here, we discuss the relation between the proposed 5-IF and c-5-IF Definition c-5-IF guarantees that outputs of an ML model are similar if input data are similar. In the setting of this paper, we can consider e-8-IF guaranteeing that users' gains are similar their privacy risks are similar. Specifi-
cally, we modify 6-5-IF by replacing v, y with r..Jy r and (x). (y) with 9e, fj. For the distance of privacy risks, we simply calculate the absolute difference of privacy risks: d(r,,, ry) I ryl- If the modified e-6-IF is satisfied, for all users such that [r; ryl I e, we have 9 95 8. The proposed 5-IF has a relation to the modified c-5-IF.
Proposition 1. If a training algorithm satisfies 8-IF regarding privacy risks and gains, the algorithm also satisfies c- -IF in the setting where inputs privacy risks and outputs gains. Here, 8' "a 50
For the proof, please refer to Appendix A Note that the converse is not true because e-6-IF does not hypothesize the correlation between privacy risks and gains.
3.4 VALIDATION USING SYNTHETIC DATA
We validate the proposed using two synthetic data. described above, unfairness 8 is expected to be small if g' is correlated with r', and Ō is expected to be large if g' is inversely correlated with r'. We verify this expectation using the first synthetic data. Specifically, we generate [ using a two- dimensional Gaussian distribution with a mean = [ and covariate matrix = = []. changing from -1.0 to 1.0 by 0.2. For each we calculate 100 times using 100 generated samples. We show examples of synthetic data in Fig. 5 Appendix and unfairness 8 in Fig. a)). As expected, i was 0 when g' was completely correlated with r', and å was the largest when g' completely
5

---

Under review as a conference paper at ICLR 2023
inversely correlated with r'. When g' did not correlate with r', i..., 0.0, was close to 4.0. This result shows that training algorithm is unfair if å is close to 4.0 larger. Figure 22b) shows the distributions of d: = g -- r Since the Gaussian distribution generates many data close to the mean, the number of users having d, close to was large. The maximum and minimum values of d, becomes large and small as decreases. Note that based on the modified e-0-1F, unfairness is small when c 1.0 because users having similar r' obtain similar g'. This is because the correlation between r' and g' is not assumed in the modified e-5-IF.
We generate the second synthetic data to confirm that the distribution of d; is useful. The second synthetic data consists of three types of users. The first type of users obtain g' corresponding to r', and we generate| [ using a owo-dimensional Gaussian distribution with μ =[8] and Σ = 1 0.8]. The second type of users obtain small g' compared with r', and we generate gi using a distribution
with μ [-2] and Σ = 1 o. The third type of users obtain large g' compared with r', and generate [':] using distribution with μ [=]] and Σ [0] o. We generate 100 samples by changing the ratio of unfair users from 0.1 to 0.9 by 0.1. When the ratio is a, we generate (1 - a) x 100 samples using the first distribution, 50 samples using the second distribution, and a x 50 samples using the third distribution. Examples of the symthetic data are shown in Fig in in Appendix. For each ratio a, we calculate unfairness 100 times as shown in Fig. Cc.. Since the generated data always contained unfair users, unfairness was large without depending on a. Even though unfairness is similar, Fig. dd) shows the difference in the distributions of d, When was small, the number of unfair users was small, and the majority of users had d; close to O. When was close to 0.5, there were various types of users; d, was close to O, large, and small. When was large, the number of unfair users was large, and the majority of users had large or small d,.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
We describe important points of the setup. For more details, please refer to Appendix B
Dataset. We used two image datasets and two tabular datasets containing user information. The image datasets are FEMNIST and Celeba, both made by [Caldas et al. (2018). FEMNIST contains x8x28 gray scale handwritten digits for 10 class-classffication. Celeba contains 64 facial im- ages for classifying "smile" and "not smile". The tabular datasets are Adult (Kohavi & Becker 1996) and Texas Texas Department of State Health Services 2013). Adult contains users' attribute vectors of size 108 for classifying whether their income is larger than 50k. Texas contains patients' attribute vectors of size 72 for classifying whether their total charge is larger than 50k. For each dataset, we selected 100 users from the test datasets to evaluate faireess. We selected variety of users from ordinary to outlier users to estimate the overall trend of the proposed IF. For the detailed procedure, please refer to Appendix B
Machine learning algorithm and hyperparameters. We used two NNs for the image datasets (ConvNet and ResNet18) and one NN (FC) and two tree-based algorithms (DPBoost and DPXG- Boost) for the tabular datasets.
ConvNet: This is a six-layer CNN designed by referring to the prior work (Nasr et al. 2021). ConvNet consists of two convolutional layers, two max-pooling layers, and two fully connected layers. We trained ConvNet with DP-SGD and changed the strength of privacy protection using different variances of the noises: a = 0.1, 0.3, 0.5, and 0.7.
ResNer18: This an 18-layer CNN with the shortcut connections (He et al. 2016). We trained ResNer18 with DP-SGD using the same hyperparameters as ConvNet.
FC: This a fayer fully connected NN. The number of units in the intermediate layers is 500, and their activation functions are ReLU, We trained FC with SGD using the same hyperparameters as ConvNet.
DPBoost: This is differentially private Gradient Boosting Decision Trees (Li et al. 2020 We changed the strength of privacy protection using different values of total budget: 100, 50, 10, and 5.

---

Under review as a conference paper at ICLR 2023
0.
01905
FEMIETT Rechiats
Unfarraess
Privace stck
0.80
cubde Bubieel
Gain
0338
0.85
0.76
Protaction leval
Protection leval
Protection lovel
(a)
(b)
(c)
10
0.8-
0.06
Performenes
0.900
Unfaizness
Privacy rilk
0.6.
Gain
0.-
TT Ad4i
0.00
BiXtiikkoet 1hooe Those BiSSiileeet
0.350
-0.44
Protection level
Protectiio level
Protection Sevel
Protection
(c)
(f)
(h)
Figure 3: Main experimental results. (a) and (e) show unfairness when using the image and tabular datasets. (b)/()) and (c)(() show means and standard deviations of users' privacy risks and gains. (d) and (h) show classification performance on data provided by users, i.e., training datasets. The classification performance (accuracy for multi-class classification and area under the curve (AUC) for binary classification) is calculated using 10 shadow models, and the mean and standard devia- tions are shown. The protection level [1, 2, 3, 4] corresponds to [0.1, 0.3, 0.5, 0.7] for NNs, total budget = [100, 50, 10, 5] for DPBoost, and for DPXGBoost.
= [100, 10, 1, 0.1]
DPXGBoost: This is differentially private XGBoost proposed for improving scalabil- ity (Grislain & Gonzalvez 2021). We changed the strength of privacy protection using different values of dp.eps lon ceon.per.trree 100, 10, 1, and 0.1.
4.2 EXPERIMENTAL RESULTS
We evaluated the unfairness, privacy risks, gains, and classification performance changing the strength of privacy protection as shown in Fig. 3 When calculating the privacy risks using the NNs, we repeated the game shown in Fig. in Appendix(Bi 1,000 times to improve efficiency. When using the tree-based algorithms,, we repeated the game shown in Fig. 1,000 times for each user. As detailed results, we show the distributions of d; in Fig. Rin Appendix.
The results were different for NNs and tree-hased algorithms. The results of NNs show that privacy risks, gains, and classification performance decreased as privacy protection became stronger. The unfairness of NNs was large without depending on the strength of privacy protection. These results show that we cannot improve the proposed IF by changing the strength of privacy protection. To look deeper into the results, show users' privacy risks and gains in Fig. 4 when using ConvNet and a 0.1. The results show that privacy risks and gains of the majority of users had positive correlation, but some users' gains were small compared with their risks. Such users were the main cause of unfaimeess. Figure 8 in Appendix shows such users were also the cause of unfairness in other settings.
The results of tree-based algorithms were surprising and show that privacy risks and gains were extremely small without depending on the strength of privacy protection. Unfairness was large without depending on the strength of privacy protection. Even though the unfaireess of tree-based algorithms was large, users do not seem to consider that the unfairness magnitude of privacy risks and gains was similar and extremely small for all users. However, this result raises a new question about users' motivation for providing their data. For example, some users are unwilling to provide their data if their data do not improve the MI. service.
problem because the
7

---

Under review as a conference paper at ICLR 2023
Table 1: Changes in unfairness on FEMNIST
011
Geiin
FEMNISTICoWWNt
FEMNIST/ReaNLLE
0.3
0.5
0.7
0.1
0.5
0.7
1.02
-1.13
0.74
-0.14
01.02
-0.88
-1.17
2.68
Tup2
-0.27
-1.11
0.19
0.78
0.44
-0.85
-0.28
1.55
02
1.5
10
40
ud
Round1
0.01
0.89
-0.73
-0.36
1.11
-0.82
0.72
-0.15
Privacy disk
Privacy risk
Round2
0.04
0,52
-0.01
-0.85
0.77
-0.89
0.04
1.62
(a) FEMNIST
(b) Celeba
Round3
-0.00
0.65
0.47
0.13
0.43
-1.15
0.30
1.64
Temp5
0.09
-0.58
-1.09
-0.90
-0.08
1.17
2.24
-0.22
Temp15
0.30
-0.58
-1.58
1.75
0.19
0.35
1.97
2.62
Figure 4: Privacy risks and gains of users
when using ConvNet and = 0.1
Table 2: Changes in unfairness on Celeba, Adult, and Texas
ColebaComvNet
Celeha/ResNet188
Adalv/FC
Texas/FC
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
Roundl
0.06
-0.66
-0.22
0.61
1.26
-2.92
-0.88
-0.36
-0.41
-1.73
-0.55
0.31
0.20
-1.68
Round2
0.70
0.42
-0.74
-0.77
0.61
0.80
-3.92
-1.27
1.82
-0.17
-0.75
-0.04
-1.04
-0.72
-1.10
-1.65
Round3
0,97
0.43
-0.52
-0.87
0.44
-3.25
0.70
0.86
-0.06
-1.31
0.28
-0,34
-0.52
-0.54
-1.19
Temp5
0.00
-0.00
0.09
-0.01
0.02
-0.03
-0.04
0,01
-0.02
0.01
-0.01
-0.00
0.00
-0.01
-0.01
-0.08
Temp15
0.01
.0.01
4.3 EFFECTIVENESS OF DEFENSES AGAINST MEMBERSUIP INFERENCE ATTACK
The experimental results of NNs indicate that the proposed IF be improved by reducing the privacy risks of users whose gains are small. Hence, we further investigate the unfairness when applying defenses against membership inference attacks (MIAs). Since the proposed IF impor- tant only for NNs based on the experimental results, we investigate changes in unfairness when applying defenses NNs. In order not decrease gains, we used three defenses that do not affect classification performance but can reduce privacy risks.
Topk:
This defense outputs predictions on k classes with the largest confidence. This defense makes MIA difficult when the confidence of the correct class is small. We expect that this defense reduces the privacy risks of users whose accuracies are low. Note that this defense can be applied to only multi-class classification, i.e., FEMNIST. We used k 1,2.
Roundd: This defense rounds confidence to d decimal places. This defense can make MIA difficult because attackers cannot use small changes in confidence. We used d = 2,
Temp/: This defense replaces temperature parameter of the softmax function with I. This defense can make MIA difficult because attackers cannot use small changes confidence. We used 5, 15.
Tables [ changes in unfairness when applying the defenses. The negative and positive values mean that unfairness was decreased and increased by applying the defenses, respectively. We also show changes in privacy risks in Tables 6 in Appendix because the defenses affect privacy risks as well. The effects differ depending on defenses. Topk: improved the fairness when the privacy protection is relatively weak, i.e., = 0.3. Topl and Top2 improved the faimess to similar extent. Roundd improved the fairness when the privacy protection is strong, i.e., a = 0.5 or 0.7. Roundl- 3 improved the fairness to similar extent except for FEMNIST and ConvNet. Round3 was not effective for FEMNIST and ConvNet. Temp/ improved the faimess of FEMNIST and ConvNet, but did not improve the fairness of the other settings. All defenses effective in some settings, but defense improved faireess in all settings. In contrast, unfaimess was increased by the defenses in some settings. These results show that method is required for improving the proposed IF of NNs.
DISCUSSION
Necessity of the propused individual fairness. As aforementioned in Section 4.2 the proposed IF does not seem to be important for tree-base brsed algorithms because both privacy risks and gains were extremely small. Here, we discuss the necessity of the proposed IF for NNs. The experimental re- sults show that privacy risks decrease as the strength of privacy protection increases. With extremely
8

---

Under review as a conference paper at ICLR 2023
strong privacy protection, all users' privacy risks expected be negligibly small. In such case, the proposed IF does not seem to be important. However, strong privacy protection for NNs deteri- orates classification performance as shown in Fig. d.). For this reason, using an extremely large a is impractical. Using a moderate a that can limit privacy risks to an acceptable level and building an individually fair model is reasonable in practice.
Possibility of other gains. In this paper, we calculated gains by the accuracy improvement on users' data. However, the gains are not limited to the aceuracy improvement, and could assume other gains such as premium service and a monetary reward. Even when using such gains, we can evaluate IF based on our definition. Since all users need to obtain the gains corresponding their risks to achieve small ó in our definition, the gains are required to be adjustable depending on users' risks.
RELATED WORK
Fairness and privacy protection. Related work regarding fairesss and privacy protection is divided into three lines. The first and main line of work studying the relation between group fairness and privacy protection. Many studies have investigated the effect of privacy protection with DP on group fairness (Bagdasaryan al., 2019 Pujol et 2020; Farrand et al., 2020, Tran et al. 2021). All studies show that privacy protection deteriorates faireess. One study focused on the relationship between vulnerability against MIA and group fairness (Chang & Shokri 2021). This
study shows that there is tradeoff between vulnerability and fairness because training data needs to be memorized to make a model fair.
The second line of work is studying the difference in privacy risks depending groups (Zhagg cet al. 2020; Zhong et al. 2022). These studies employ MIA to evaluate privacy risks and show that minority groups face larger privacy risks than majority ones. The studies further show that privacy protection with DP decreases the difference in privacy risks between minority and majority groups
The third line of work is proposing methods satisfying both privacy protection and group faimesss Xu et al. (2019) proposed a logistic regression guarantccing both DP and group fairness, and Xu al. (2020) extended DP-SGD for reducing unfairness by adjusting the clipping of each class.
Lower bound of DP parameter c. In this paper, when calculating the lower bound, we assume a realistic attacker who can access outputs of the trained model via API. Not only such realistic attacker but also stronger attackers were proposed to calculate lower bounds (Nasr et al.) 2021). The stronger attackers can access intermediate models during training and manipulate the whole dataset. In this paper, we assume a realistic attacker because the trained models are assumed to be carefully protected by ML services.
7 CONCLUSION
In this paper, we propose a new definition of individual fairness (IF) from the perspective of privacy protection and experimentally evaluate -wwccy-preserving machine learning (ML) based on the pro- posed IF. For the proposed definition, assume that users provide their data an ML service and consider the principle that all users should obtain gains corresponding to their privacy risks. Fur- thermore, we discuss the relationship between the proposed IF and prior IF for classification and validate the proposed IF using synthetic data.
We conducted experiments on the image and tabular datasets using three neural networks (NNs) and two tree-based algorithms with differential privacy guarantee. The experimental results of NNs show that we cannot stably improve the proposed by changing the strength of privacy protection and applying defenses against membership inference attacks. These results show the need for method that stably improves the proposed IF of NNs. The results of tree-based algorithms show that privacy risks and gains protection. For tree-basee based algorithms, the proposed IF seems not to be important, but these results raise new question about the motivation of users for providing their data.
extremely small without depending
the strength of privacy

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 23rd ACM Conference on Computer and Communications Security, pp. 308-318, 2016.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate im- pact on model accuracy. In Proceedings of the 32nd Advances in Neural Information Processing Systems, 2019.
Sebastian Caldas, Sai Mcher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 2018.
Hongyan Chang and Reza Shokri. On the privacy risks of algorithmic fairness. In Proceedings of the 6th IEEE European Symposium Security and Privacy, pp. 292-303, 2021.
Charles J Clopper and Egon S Pearson. The of confidence fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404-413, 1934.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private sata analysis. In Proceedings of the 3rd Theory of Cryptography Conference, pp. 265-284, 2006.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Con- ference, pp. 214-226, 2012.
Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair: Impact of data imbalance utility and fairnsss in differential privacy. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, pp. 15-19, 2020.
Nicolas Grislain and Joan Gonzalvez. Dp-xgboost: Private machine learning at scale. arXiv preprint arXiv:2110.127700 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? arXiv preprint arXiv:2006.077000 2020.
Philips George John, Deepak Vijaykeerthy, and Diptikalyan Saha. Verifying individual fairness in machine learning models. In Proceedings of the 2020 Conference on Uncertainty in Artificial Intelligence, pp. 759-758. PMIR, 2020.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1376- 1385, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 69800 2014.
Ronny Kohavi and Barry Becker. Adult data set. https///acchiie..ss....oc////aat 1996.
Qinbin Li, Zhaomin Wu, Zeyi Wen, and Bingsheng He. Privacy-preserving gradient boosting deci- sion trees. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 784-791, 2020.
Mani Malek Esmaeili, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramer. Antipodes of label differential privacy: Pate and alibi. In Proceedings of the 35th Advances in Neural Information Processing Systems, pp. 0934-6945.0 2021.
Meta Platforms, Inc. Building an image classifier with differential privacy. https://opacus.a/uuooiaaaae image classifier, 2022.
10

---

Under review as a conference paper at ICLR 2023
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In Proceedings of the 42nd IEEE Symposium Security and Privacy, pp. 866-882, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. http:///githb..coo//yxxooo///pop 2017.
David Pujol, Ryan McKenna, Satya Kuppam, Michael Hay, Ashwin Machanavajihala, and Gerome Miklau. Fair decision making using privacy-proteeted data. In Proceedings of the 2020 Confer- ence on Fairness, Accountability, and Transparency, pp. 189-199, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In Proceedings of the 38th IEEE Symposium Security and Privacy, pp. 3-18, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with dif- ferentially private updates. In Proceedings of the 1st IEEE Global Conference on Signal and Information Processing, pp. 245-248, 248, 2013.
Texas Department of State Health Services. Hospital discharge data public use data file. http://ww......eeooooocccccoaaaa
2013.
Cuong Tran, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. In Proceedings of the 34th Advances in Neural Information Processing Systems, pp. 27555-27565, 2021.
Depeng Xu, Shuhan Yuan, and Xintao Wu. Achieving differential privacy and fairness logistic regression. In Proceedings of the 2019 World Wide Web Conference, pp. 594-5994999999 2019.
Depeng Xu, Wei Du, and Xintao Wu. Removing disparate impact of differentially private stochastic gradient descent on model accuracy. arXiv preprint arXiv:2003.0300 03699, 2020.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint 1rXiv:2109.12296 2021.
Bo Zhang, Ruotong Yu, Haipei Sun, Yanying Li, Jun Xu, and Hui Wang. Privacy for all: Demystify vulnerability disparity of differential privacy against membership inference attack. arXiv preprint arXiv:2001.08855, 2020.
Da Zhong, Haipei Sun, Jun Xu, Neil Gong, and Wendy Hui Wang. Understanding disparate effects of membership inference attacks and their countermeasures. In Proceedings of the 2022 ACM Asia Conference on Computer and Communications Security, pp. 959-974, 2022.
A PROOF OF PROPOSITION 1
Proof. From Definition if a training algorithm satisfies 8-1F, for a user u, having privacy risk r and gain g. the following holds:
r/ 8.855 g S <r/ &r/+6. +
Unnormalizing Ti and of with
r/ rI-Me and gf or
g. and gi satisfy the following:
0g
0g (r: (4)) 0g5 + My gi
0g (r; pr) + σg Fg.
o
or
Considering user uy having privacy risk ry = + the gain 9n satisfies the following:
Og (ri + (4) - σ + M1 ,و or
Tg (r: c -44) 0go + Mg-
or
11

---

Under review as a conference paper at ICLR 2023
Prinacy zack
Prinacy risk
Privacy risk
Pricacs riak
Privacy rak
(b) 0.5
c= 0.0
(d) c 0.5
c= 1.0
Figure 5: First type of synthetic data with different
Gom
1
Pltvary rik
Privacy rick
Privaay risk
Privary risk
Privacy Hok
(a) 0.1
(b) 0.3
(c) 0.5
(d) = 0.7
(e) a 0.9
Figure 6: Second type of synthetic data with different a
The difference between gains of users u, and My such that d(rs, ry) = -ra-- --ryl S c ttakes the
supremum when r- ri + c rj r;
- + 20, = 8'. max 19. 1yو d(r..ry)se
Or
This shows that a training algorithm satisfies 6-8'
B DETAILED EXPERIMENTAL SETUP
For the experiments of NNs, we used PyTorch v1.10.1 (Paszke et al. 2017) and Opacus v1.0.0 (Yousefpour et al. T02T for implementation and conducted experiments NVIDIA Tesla V100 16GB with CUDA 11.3.1.
Dataset. We used two image datasets and tabular datasets containing information.
This dataset consists of 28x28 gray scale handwritten images written by 3,500 users (Caldas al. 2018). We randomly selected 500 users from users having more than 100 images of digits 0-9. We used 59,556 images of the selected users as the training dataset Do and trained models for 10-class classification.
FEMNIST:
Celebu: This dataset consists of images of 9,343 users for facial expression recogni- tion (Caldas et al.] 2018). We resized images to 64 64 and randomly selected 2,000 users from users having more than 10 images. We used 46, 1 16 images of the selected users as Do and trained models classifying "smile" and "not smile".
Adult: This dataset consists of 14 types of users' attributes extracted from the 1994 US Census database (Kohavi & Becker 1996). We used one-hot encoding for categorical attributes, and the size of encoded feature vectors is 108. We randomly selected 39,073 users as Do and trained models classifying whether a user's income is larger than 50k.
Texas: This dataset consists of patients' attributes (Texas Department of State Health Ser- vices 2013). We used 12 types of attributes: DISCHARGE, TYPE.ON OF ADMISSION PAT_STR PATESTATE, PAT-STATUS, SEX_CCDE, RACE, ETHNICITY, ADMIT_WER TOMERDYEEKDAY PAT_AGE RISK MISK.MORTALITY
LENGTH _STAY. We used one-hot encoding for categorical attributes, and the size of encoded feature vectors is 72. We randomly selected 50,000 users Dg and trained models classifying whether pa- tient's TOTAL.CHARGES is larger than 50k.
We normalized each channel of FEMNIST and Celeba that its mean and variance are and 1, respectively. We normalized each element of Adult and Texas so that its minimum and maximum values 0 and 1, respectively.
12

---

Under review as a conference paper at ICLR 2023
Table 3: Hyperparameters
Algorithm
Hyperparameter
Candidates
Selected
Patch size
x 3, 5 5
5 x 5
Output channel size
32, 64, 128
64
ConvNet
Number of units in the FC layer
256, 512, 1024
512
Batchsize
64, 128, 512
512
Number of layers
3,5, 10 100, 500, 1,000
5
FC
Number of units
500
num num_leaves
20, 30, 50, 100
30
max_dep max_depth
3, 6, 10
10
DPBoost
my trees
20, 30,
50
inner_ ainerbbosttround
20, 30, 50
50
n trees subsample
10, 20, 30
20
DPXGBoost
0.1, 0.2, 0.3
0.2
max_depth
3, 6, 10
6
Table 4: Architecture of ConvNet
Layer
Type (Activation)
Patch
Output channels
Convolution (ReLU)
5 x
64
2
Max pooling
2
64
3
Convolution (ReLU)
5 X
64
Max pooling
2
64
5
Fully connected (ReLU)
512
Fully connected (Softmax)
10 or
For each dataset, we selected 100 from the test datasets to evaluate fairness. Even though 100 users are part of the users, we can estimate the overall trend of the proposed IF by selecting users considering their variety. To select variety of users from ordinary outlier users, we trained 10 shadow models using the training dataset and calculated the accuracy of each user's data. We sorted users in descending order, selected percentiles, and selected 20 users with the lowest accuracies. Since each user has multiple data in image datasets, we selected data with the lowest accuracy from each user's data for the evaluation. We calculated the accuracy using ConvNet (described below) and 0.1. Note that each user has one sample of data in tabular datasets.
users each from x 100, x 100, x 100, and 3 100
Machine learning algorithm and hyperparameters. We used three NNs and two tree-based al- gorithms. We selected the best hyperparameters in terms classification performance on the test dataset. For classification performance, calculate accuracy for multi-class classification, i.e., FEMNIST, and AUC for binary classification, i.e., Celeba, Adult, and Texas. The candidates of hyperparameters and selected experiment and set millisecond obtained with t time. time! () as random seed.
shown in Table We used different random seeds for each
ConvNer: This is six-layer CNN designed by referring to the prior work (Narr et al. 2021). ConvNet consists of two convolutional layers, two max max-pooling layers, and two fully connected layers. The details of the architecture
shown in Table 4 We optimize ConvNet with DP-SGD using commonly used hyperparameters: the number of epochs is 50, the batchsize is 512, the optimizer is Adam (Kingma & Ba 2014). the learning rate 7j is
0.001, the clipping threshold of the gradient C = 1, and the privacy parameter = 10-5 We changed the strength of privacy protection using different variances of the noises:
0.1, 0.3, 0.5, and 0.7. The corresponding upper bounds of by theoretical analysis are 42,967.4 217.0, 36.1, and 12.5.
ResNer18: This is a 18-layer CNN with the shortcut connections (He et al. 2016). To apply DP-SGD to ResNet18, replaced the BatchNorm layers with GroupNorm layers referring to the tutorial of Opacus (Meta Platforms. Inc. (2022). BatchNorm layers cause a privacy violation because they means and variances regarding samples in minibatch.
13

---

Under review as a conference paper at ICLR 2023
Attacker
Trainer
Da. [xx
b D bn + {0.1}a
D
for in l,... n do if 1 then (()
D. f MD;)) for in 1,.., n da 4 e(f()),))
4 .
for in 1.... do
by' + -ADo, 1.14)
bi.... b/2
for in 1,... do check if b/ bç
Figure 7: Multiple-sample game for efficient calculation of lower bound
The means and variances make dependencies between samples in minibatch and violate DP. For this reason, we fix ResNet18 with ModuleVall i ddttor. For optimization, we used the DP-SGD and the same hyperpatrammeters as ConvNet.
FC: This is five-layer fully connected neural network. The number of units in the inter- mediate layers is 500, and their activation is ReLU. For optimization, we used the DP-SGD and the same hyperparameters as ConvNet.
DPBoost: This is differentially private Gradient Boosting Decision Trees (Li al. 2020). DPBoost obtains a tighter sensitivity bound with Gradient-based Data Filtering and Geo- metric Leaf Clipping. We changed the strength of privacy protection by specifying the upper bounds of e using different values of total 100, 50, 10, and 5.
DPXGBoost: This is a differentially private XGBoost proposed for improving scalabil- ity (Grislain & Gonzalvez 2021). We changed the strength of privacy protection using different values of dp.eps appsili popsiion.per won.poo..trreeeeeeeeeeeeeeeeeeee 100, 10, 1, and 0.1. The corresponding upper bounds of by theoretical analysis are 1,967.8, 88, 167.8 5.9, and 0.4.
Efficient calculation of lower bound. One drawback of the lower bound calculation is computa- tional cost. Specifically, when the game is repeated 1,000 times, 1,000 models need to be trained with privacy-preserving ML in total. In our experiments, we calculate the lower bounds for 100 users and need to train 100 x 1, 000 models. When using NNs, the computational cost is extremely expensive. To tackle this problem, we use multiple-sample game shown in Fig. 7 designed by referring to efficient method for label DP (Malek Esmaeili et al. 2021). In the multiple-sampp game, the attacker sends dataset Do and n target samples {(x,, Vi randomly decides whether to use each target sample for training. In Fig. 7 b, represents whether the sample (æ,, y/s) is used for training. The trainer builds a model using the dataset Do and the selected target samples with privacy-preserving
7 ==1 to the trainer. The trainer
ML algorithm M and returns the losses the target samples to the attacker. The attacker predicts whether each sample is used for training with the dataset Do. the target samples {(>;, );)) ) and the loss I; on the sample (ær,, yz). Finally, the trainer checks if the predictions are correct. We repeat the multiple-sample game multiple times and calculate the lower bound E following Eq. [ for each target sample. When we use 100 samples in our game, we can reduce the number of training by 1/100.
We validated the reliability of the multiple-sample game using users' data with the lowest accuracies. We calculated the lower bounds with the game shown in Fig. andd the multiple-sample game shown in Fig. 7 and checked if the lower bounds were close to each other. We repeated both games 1,000 times each using DP-SGD parameter = 0.1 and used 100 target samples in the multiple-sampe game. Table (sshos that we obtained good approximations of the lower bounds with the multiple- sample game.
14

---

Under review as a conference paper at ICLR 2023
Table 5: Lower bounds of calculated with one-sample and multiple-ssmple games. The one-sample game a game shown Fig.
Dataset
Network
One-sample
Multpplesampple
FEMNIST
ConvNet
2.00
1.98
FEMNIST
ResNet18
2.99
2.58
Celaba
ConvNet
3.21
3.35
Celaba
ResNet18
2.53
2.35
Adult
FC
1.73
1.52
Texas
FC
1.00
1.05
2.0
-2.0
-4.0
0.1
0.7
0.5
0.7
(a) FEMNIST//onwNet
(b) FEMNIST/ResNc11
2.0
2.0
0.0
4..0
2.0
-6.0
8.00
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
(c) Neleha/Convvet
(d) Celeha/ResNet18
2.0
0.0
0.0
0.0
2200
+4.0
-4.0
.8.0
8.0
0.1
0.3
0.55
0.7
100
50
100.0
10.0
1.0
0.1
total_ total_budget
dp_epsilon_pp _er_tree tree
(c) Adult/FC
(f) Adult/DPBoost
(g) Adult/DPXGBoost
2.0
0.0
+4.0
-1.0
-4.0
-8.0
8.00
-6.0
100
100.0
10.0
1.0
0.1
total_ budget
So__pssllon_per...
(h) Texas/FC
(i) Texas/DPBoost
(i) Texas/DPXGBoost
Figure 8: Distribution of d;
15

---

Under review as a conference paper at ICLR 2023
Table 6: Changes in privacy risks when applying defenses. We calculated change for each user and show the means and standard deviations of the changes. The negative and positive values mean that privacy risks were decreased and increased by applying the defenses, respectively.
FEMNIST/CoIwNet
HEMNIST/RReNN111
Top1
10.58 + 0.55
-0.17
0.21
-009
0.111
-0.0 = 0.11
4.40 + 0.42
40.15 0.119
-0.0 +.11
-0.00-000 0.09
.000
Top2
Round1
-0.64 - 0.52
-0.19 = 0.19
-0.13
0.14
-0.07 = 0.10
40.39 1: 0.28
-0.16
0.17
-0.09
0.10
-0.06.0.000
-0.06
Round3
-042 - 0.47
0.17
-0.08
2:
-0.05
= 0.09
40.15
1: 0.21
-0.07
1 0.13
-0.05
0.09
.0.04 1:007
Temp5
0.00
0.06
40.01
0.19
0.10
0.01
-0.02
Temp15
0.13 1 0.26
0.002 = 0.10
-0.03
0.12
-0.01
0.10
40.12
1: 0.26
-0.04 1 0.15
-0.00 2: 0.15
-0.03
0.10
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
Roundl
0.27
Round2
.067 1 0.45
-038
0.41
-0.23
0.31
40.20
0.23
40.13 1: 0.15
-0.12 1 0.13
-0.11 1 0.18
.0.99 0.11
Temp5
-0.00
1 0.01
-0.00
0.01
0.01 0.00
0.00
0.01
0.00 1: 0.00
0.00 1 0.01
0.00
0.01
.0.00
1:000
Temp15
0.00
0.01
0.00
0.01
0.00
AdalDFC
TeassFC
Round
40.22 + 0.41
-0.08
= 0.16
-0.08 + 0.14
-0.05
0.09
4.20 + 0.30
-0.10 0.15
-0.07 0.12
.004 0.08
Round3
.0.08
-: 0.17
-0.04
- 0.10
-0.04
0.12
-0.03 = 0.97
40.08 1: 0.17
-0.04 1 0.10
-0.03 1 0.06
.0.03 0.06
Temp5
Temp15
0.00
1: 0.00
0.00
= 0.01
0.00
2: 0.00
0.00 0.00
0.00 1: 0.01
-0.00 1 0.01
.0.00 -: 0.00
.0.00 0.00
16


Reviews:
[{"review_id": "pm2WWo6WGAA", "reviewer": "Reviewer_fzWY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new definition of individual fairness (IF), which requires all users’ tradeoffs between gain and privacy risks are similar. Empirical analysis shows different findings of how the fairness of neural networks and tree-based algorithms change under different privacy protections.", "summary_of_the_review": "The contributions of the paper are limited by the motivation of the proposed IF, the lack of mitigation methods for unfairness, and the limited values provided by the empirical findings.", "strength_and_weaknesses": "Strengths:\n1. The new IF notion is novel in terms of its formulation.\n2. The empirical study is thorough regarding the diversity of model architecture, datasets, etc.\n\nWeaknesses:\n1. The contributions of the paper are limited by the motivation of the proposed IF. First of all, the authors do not justify the proposed IF well, and the requirement that similar gain corresponds to similar privacy is arguable. I suggest the author better justify the proposed IF notions by several practical use cases.\n2. Even though the definition makes sense, the paper does not provide any methods to mitigate the unfairness. It is not surprising that existing privacy protection methods do not provide any useful mitigation to the fairness problem. If the authors could provide some mitigation methods to the proposed fairness notion, even though it is only empirical, \n3. Lastly, the empirical findings do not provide additional insights to the community: for the tree model, the proposed IF notion is unnecessary; For the NN models, it has been well known that the majority of existing DP methods cause significant performance loss. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cF2BTT2w9fT", "reviewer": "Reviewer_77T3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers a timely topic: connection between fairness and privacy-preservation of an ML model. The angle is interesting: fairness is considered from the point of view that the model performance gains for user's data should correlate with the privacy leakage. It is well known that e.g. training with differential privacy leads to unequal individual privacy-preservation, see e.g. [1] and [2]. It is also known that the model performance correlates with privacy-preservation in a sense that the most accurate classes have best privacy-protection [2].\n\nThis paper consider empirical epsilons instead of a priori DP epsilons, computed as in [3] using a membership attack guessing game. These empirical epsilons are compared to empirical 'gain values' computed with models that contain and do not contain the particular sample. These (normalized) empirical values are used to determine the fairness and this fairness definition is tested on several experiments.\n\n\n[1] Feldman, V., & Zrnic, T. (2021). Individual privacy accounting via a renyi filter. Advances in Neural Information Processing Systems, 34, 28080-28091.\n\n[2] Yu, D., Kamath, G., Kulkarni, J., Yin, J., Liu, T. Y., & Zhang, H. (2022). Per-Instance Privacy Accounting for Differentially Private Stochastic Gradient Descent. arXiv preprint arXiv:2206.02617.\n\n[3] Nasr, M., Songi, S., Thakurta, A., Papernot, N., & Carlin, N. (2021, May). Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP) (pp. 866-882). IEEE.", "summary_of_the_review": "All in all I don't think the paper is mature for publication. The novelty and content is not enough and also the paper requires some rewriting in my opinion.\n\nInstead of only comping up with a new definition of fairness, it would be nice if some remedies were suggested. As for example shown already in [2], it happens to be so that there is a correlation between privacy preservation and model performance: the classes with the lowest average epsilons have also the best performance. And this seems to be inherent to DP-SGD trained models (and probably to DP-trained ML models in general). So instead of inventing new tools to study the individual fairness, it might be more interesting to try find remedies. And this, as far as I see, this paper is not able to do.", "strength_and_weaknesses": "Pros:\n\n- The problem the paper is studying is very interesting, this is a timely problem.\n\n- The main idea is interesting: \"We consider that a training algorithm is individually fair if all users obtain gains corresponding to their privacy risks.\"\n\n\nCons: \n\n- The content is a bit thin. There are only empirically computable quantities defined, which are tested in experiments. \n\n- The paper contains some errors. For example:\n\n\"A training algorithm is individually fair if the difference in privacy risks between any pair of users is small. This definition can be satisfied by reducing privacy risks with privacy-preserving ML because the differences are small if all users’ privacy risks are small. \"\n\nI would claim this is not true. As shown e.g. in [1] and [2], there can be very large differences in individual DP epsilons between\nusers. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RWwqI0GQO3", "reviewer": "Reviewer_bzAv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new notion of privacy, specifically targeted at individuals, rather than the entire training dataset as a whole. The core idea behind the concept of IF is that individuals should get gains (in privacy and performance) out of participation. Most results highlight disturbing trends in individual fairness- specifically for neural networks. Although most of the results (vis-a-vis tree algorithms and MIA defenses) are negative in nature, they raise the essential question of why users should participate in such model training routines, given their performance and privacy gains.", "summary_of_the_review": "The main issue with the paper in its current form is how it convinces the reader about the severity of this issue. Particularly, the approach of retraining models (even if shadow) for every potential participant is far from feasible, and it is unclear if relative gains should be the focus of the study, given the aspect of fairness (eliminating propagation of biases, in the form of privacy or performance gains). I think if the authors can address this main concern (discussed above) and add comparisons with some defenses (like causal learning, or something apart from Differential Privacy that is shown to work well in the literature), the paper would be in much better shape.", "strength_and_weaknesses": "## Strengths\n\n- This work introduces the user's perspective (which is the data owner, and should thus be at the center of decisions in an ideal scenario) to privacy-preserving learning- convincing users to participate in training, and in some way demonstrating benefits to privacy and/or utility. Although the results are negative, they show how MIA defenses and increasingly private learning do not necessarily lead to improved performance for 'all' users, which is what model trainers should strive for.\n\n- Evaluation is extensive, and the notion of $(\\epsilon, \\delta)$-IF (and its connection to IF) is well explained and intuitive to follow and understand. \n\n## Weaknesses\n\n- The authors acknowledge that there \"are differences in privacy risks between users\", and argue that a training algorithm is IF if all users gain gains corresponding to privacy risks. However, this seems to defeat the very purpose of privacy and algorithmic fairness in general. All users' privacy guarantees should be similar, and not unequally biased towards a specific sub-group of users; the same goes for model performance as well. From what I understand, IF says 'if you had poor privacy guarantees, you should be content with this gain, even though it may be less of a gain than some other user who already had better privacy guarantees. I am not convinced that this is the right approach to dealing with privacy, and is certainly not fair.\n\n- The proposed approach seems to be excessively computationally expensive. Even for a relatively small shadow model, re-training for each potential user is not feasible at all (and opens doors to other kinds of adversaries that may craft adversarial data and use the \"expected gains\" to leak sensitive information from the victim's models/training data). If a user is wary of the model trainer in the first place (which is why they want to look at potential gains), how can the same user be trusted with any 'metrics' that the model trainer may report as expected gains? One way could be to provide black-box API access to the trained model as proof, but that itself is a security/privacy risk.\n\n- Although this is a start, true overlook of \"individual\" fairness should really look at \"individuals\", not their individual records. A user, for instance, may have 5-10 images that they might upload to a cloud provider. Although the privacy/performance gains would be low when measured for a single data point (which is expected, given the relative scale of training data), they would be much higher if a collection of data is used instead. With the help of subject-level membership attacks [1, 2], similar gains can be measured for this case as well, which is much better aligned with real-world use cases. \n\n- I would like to see the use of some other defenses already proposed in the literature, in addition to the ones described in Section 4.3. For instance, causal learning is known to reduce inference risk significantly [3].\n\n### References\n\n[1] Hartmann, V., Meynent, L., Peyrard, M., Dimitriadis, D., Tople, S., & West, R. (2022). Distribution inference risks: Identifying and mitigating sources of leakage. arXiv preprint arXiv:2209.08541.\n\n[2] Suri, A., Kanani, P., Marathe, V. J., & Peterson, D. W. (2022). Subject Membership Inference Attacks in Federated Learning. arXiv preprint arXiv:2206.03317.\n\n[3] Tople, S., Sharma, A., & Nori, A. (2020, November). Alleviating privacy attacks via causal learning. In International Conference on Machine Learning (pp. 9537-9547). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3obyxeHniO", "reviewer": "Reviewer_MgQj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on the problem of individual fairness in terms of privacy risks and improved accuracy. More concretely, differential privacy has been shown to have a disparate impact on model accuracy for minorities, meaning groups with fewer members/outliers tend to lose more accuracy (by use of DP) than other larger groups. What this paper does is study this phenomenon, on an individual level, i.e. how much each individual loses, as opposed to each group. To enable this, the authors propose a new individual fairness metric that takes into account both the privacy risk and utility gain. They basically measure how much accuracy gain a single sample gets by participating in training, versus how much privacy they lose in terms of membership inference power.\nThey show that the shared trend is the more privacy risk, the better accuracy, and that similar to what was priorly shown on a group level, applying DP and other privacy mitigation attempts has a disparate impact on gains on an individual level.", "summary_of_the_review": "I have provided an extended review in the strengths and weaknesses box. To summarize, I think the problem the authors are studying is important and interesting, and the proposed individual fairness measure seems new. However, what I am skeptical about is the novelty of the findings, as it seems to mostly reflect the disparate impact of privacy-preserving methods which has been shown before on multiple different setups, and there are mitigations provided for it too [1-2]. Given all this, I find the contributions limited. However, there is a chance that there is something I am missing, so if the authors clarify their contributions and I realize I have misunderstood something I am willing to increase my score. ", "strength_and_weaknesses": "Strengths\n1. The paper is looking at individual fairness which is an understudied problem. They consider the interplay of individual accuracy increase and privacy gain which is new.\n\nWeaknesses\n1. I am not entirely sure about the use of individual training accuracy gain as part of the metric, I think we should technically look at user-level accuracy, where we have at least a single sample held out test for a given training sample thereby measuring generalization accuracy gains. I think measuring the gain on training samples is a bit meaningless as technically the default accuracy on those samples could be 100% for that user, if we do like a KNN classifier.\n\n2. The measurement of privacy expenditure is inconsistent, there should just be 4 privacy budget epsilons and all experiments done on those, as opposed to different epsilons and even different ways of reporting. Like for one method epsilon is reported,for another the standard deviation of the noise added is reported (which could be converted to epsilon and I think should be for presenting results). All this said, I find stacking all these inconsistent guarantees together like that in Figure 3 in appropriate.\n\n3. I am not sure I fully understand the two lines of reasoning for why the naive definitions of IF are bad, in section 3.2:\n•\tA training algorithm is individually fair if users having similar data face similar privacy risks. This definition is inadequate because a difference in privacy risks can be large if users’ data are dissimilar --> I am not sure how this relates to the former sentence. Doesn't really make sense.\n•\tA training algorithm is individually fair if the difference in privacy risks between any pair of users is small. This definition can be satisfied by reducing privacy risks with privacy-preserving ML because the differences are small if all users’ privacy risks are small. However, strong privacy protection with DP is known to degrade classification performance --> There are two issues with this: a) the only way to get similar privacy risks is not by applying privacy preserving methods, there could be other ways out there, so to say that this is achieved by strong DP guarantees is a bit inaccurate, and b) there are recent papers that show DP can be achieved with little loss to accuracy [3-5]\n\n\n4. I am not really sure what the conclusion  “necessity of the proposed IF for NNs'' really means. If it means we need improvements, there are some group level improvements, why not try them and then test? Improvments like [1-2]. I think the fact that the only conclusion from the paper is that individual level fairness is not good in privacy preserving methods is a bit repetitive gien prior work. \n\n[1]  Tran, Cuong, Ferdinando Fioretto, and Pascal Van Hentenryck. \"Differentially private and fair deep learning: A lagrangian dual approach.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 11. 2021.\n\n[2] Fioretto, Ferdinando, et al. \"Differential Privacy and Fairness in Decisions and Learning Tasks: A Survey.\" arXiv preprint arXiv:2202.08187 (2022).\n\n[3] Li X, Tramer F, Liang P, Hashimoto T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679. 2021 Oct 12. (ICLR 2022)\n\n[4] Yu D, Naik S, Backurs A, Gopi S, Inan HA, Kamath G, Kulkarni J, Lee YT, Manoel A, Wutschitz L, Yekhanin S. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500. 2021 Oct 13. (ICLR 2022)\n\n[5] Tramer, Florian, and Dan Boneh. \"Differentially Private Learning Needs Better Features (or Much More Data).\" International Conference on Learning Representations. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject