Bias Propagation in Federated Learning
Authors: Hongyan Chang
Reza Shokri
Venue: ICLR 2023 poster (2023)
Keywords: Fairness
Algorithmic Bias
Federated Learning

Abstract: We show that participating in federated learning can be detrimental to group fairness. In fact, the bias of a few parties against under-represented groups (identified by sensitive attributes such as gender or race) can propagate through the network to all the parties in the network. We analyze and explain bias propagation in federated learning on naturally partitioned real-world datasets. Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes. What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data. This indicates that the bias is due to the algorithm. Our work calls for auditing group fairness in federated learning and designing learning algorithms that are robust to bias propagation.


Published as a conference paper at ICLR 2023
BIAS PROPAGATION IN FEDERATED LEARNING
Hongyan Chang & Reza Shokri School of Computing
National University of Singapore
(hongyan, reza @comp nus. edu.sg
ABSTRACT
We show that participating in federated learning can be detrimental to group fair- ness. In fact, the bias of a few parties against under-represented groups (identified by sensitive attributes such as gender or race) can propagate through the network all the parties in the network. We analyze and explain bias propagation in federated learning on naturally partitioned real-world datasets. Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in small number of model parameters, and throughout the training, they steadily increase the depen- dence of the global model sensitive attributes. What is important highlight is that the experienced bias federated learning is higher than what parties would otherwise encounter in centralized training with model trained the union of all their data. This indicates that the bias is due to the algorithm. Our work calls for auditing group fairess in federated learning and designing learning algorithms that are robust to bias propagation.
INTRODUCTION
Machine learning models can exhibit bias against demographic groups. Previous research has extensively studied how machine learning algorithms can reflect and amplify bias in training data,
especially in centralized settings where data is held by a single party (Hardt et al., 2016 Dwork
ct al.] [0122] Calders et al. [2099] Hashimoto et al. 2018 Zhang et al., 2020 B Blum and Stangl 2020
Lakkaraju et 2017). However, practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns. Federated learning (FL.) provides a promising solution by enabling parties to collaboratively learn a global model without sharing their data. each round of FI., parties share their local model updates computed on their private datasets with a global server that aggregates them to update the global model. Despite the widespread adoption of FL in various applications such as healthcare, recruitment, and loan evaluation (Ricke et al. 2020 Yang et al. 2019), it is not yet fully understood how FL algorithms could magnify bias in training datasets.
Recent studies have investigated the problem of measuring and mitigating bias in federated learning with respect to a single global distribution (Chu et al. 2021 Zeng et al. [221aa] Hu et al. 2022] Du
et al. 2021 Abay et al. 2020 Papadaki et al. 2021 parties often have heterogeneous data distributions. Evaluating the model's bias with respect to the global distribution does not accurately reflect the fairness of the FL model with respect to parties' local data distributions, which are relevant to end-users. This is the critical problem that we address in this paper. Specifically, we investigate the following questions: How does participating in FL affect the bias and faimeess of the resulting models compared to models which are trained in a standalone setting? Does FL provide parties with the potential fairness benefits of centralized training on the union of their data? Can parties with biased datasets negatively impact the experienced fairness of other parties on their local distributions? How and why does the bias of a small number of parties affect the entire network? To the best of our knowledge, provide the first comprehensive analysis of how FL algorithms impact local fairness.
2022 Hu al. (022). However, practice,
We provide empirical analysis based on real-world datasets. We show that FL might not sustain the benefits of collaboration in terms of fairness, as compared to its accuracy benefit. Specifically, compared with the standalone models, on average, fairer on local data distributions. However, in those cases, the FL models, trained on the same dataset, can be more biased. This suggests that the FL algorithm itself can introduce new
find that the model trained in centralized setting can be,

---

Published as a conference paper at ICLR 2023
bias in the final model. Furthermore, we demonstrate that FL impacts different parties in different ways. Specifically, we find a strong correlation between parties' faimess gap in the standalone setting and the fairness benefit they obtain from joining FL: parties with a greater bias in the standalone setting (caused by their local data) would receive a fairer model from FI.. In contrast, FL has negative impacts on parties with less significant bias in the standalone setting, resulting in more biased model in FL. We further demonstrate that this is due to the fact that FL propagates bias: bias from a few parties can influence all parties in the network, hence aggravating the fairness problem globally.
Finally, we offer potential explanations for how bias is propagated in FL. Specifically, we show that local updates from biased parties increase the dependency between the model's 's predictions and the sensitive attributes. Such an increase is achieved by the norm increase in a small number of parameters (around 6% of the model parameters in some experiments). This increase then propagates to the global model through aggregation, subsequently impacting all other parties. In addition, we show that the fairness gap of the final model can be governed by adjusting the value of those parameters. Surprisingly, we find that scaling these parameters can either reduce the model's 's bias to small value of 0.05 (on a measurement scale to I) with only a 0.6% drop in accuracy or increase the bias to a large value of 0.96 with 11% drop in accuracy.
2 BACKGROUND
Federated Learning. In this paper, we consider the conventional federated learning (FL) setting. (McMahan et al. 2017). The FL. framework consists of a network of K parties, where each party E [K] holds a local dataset D2 of size ny, sampled from a local data distribution De. The objective of each party is to train model that minimizes the loss their local data distribution Da. To achieve this goal, FL trains a global model to minimize the average loss across parties, which is expressed as ming N Σ ny L(0,, Dx), where N is the sum of all local training dataset size. In each communication round 1,, global server sends the current global model to all parties. All parties train the global model locally on their local dataset and send the updated local model to the server. The server then aggregates those local models to obtain the new global model. We consider the case where all parties participate in the training in each round, which helps to mitigate potential biases that could arise from the non-umine non-uniform sampling of participating parties.
Group Fairness. Fairness has a wide variety of meanings in literature. Group fairness entails, in particular, that the model should perform comparably across groups defined by sensitive attributes (e.g., sex). It is now common practice to evaluate discrimination in model (or system) based on quantitative measurements of group fairness. In light of this, we focus on two widely fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al. 2012). To formally define those fairness notions, we assume each data point is associated with a sensitive attribute E A. and we use X and Y to denote the input features and the true label. To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, defined as
group
A E0 (0, D) ==
max A.
| PP((f((X) +|A a, Y y) Pr Pr(ff(X) = +|A - y) P
with an ideal value equal to zero (perfectly fair). Furthermore, in many applications, there exists a favorable prediction from the model (e.g.. grant the loan). We assume the positive prediction (-) as the favorable outcome. Demographic parity (Dwork et al. 2012) group fairness notion asks the model to give favorable label to groups with equal rates. Similarly, the fairness gap with respect to Demographic Parity is defined as follows:
DP (0, D) max [Pr(fo(X) +|A a) - ()(f((x)= +IA aa)l
a,a'EA D
with an ideal value equal to zero. In the rest of the paper, we use the fairness gap A (including AO and A DP) to measure the bias (fairness) of a model. The more significant fairness gap means the model is more biased (less fair). We will use bias or unfairness interchangeably.
Measuring the impact of FL. Federated learning aims to enhance model performance compared to standalone training and achieve comparable performance centralized training. Thus, in order to evaluate the impact of FL on local fairness, we use centralized training and standalone training as baselines. In standalone training, each party trains a model 0g independently to minimize the loss on its training data Dz. Note that the standalone model's fairness gap for party is contributed by
2

---

Published as a conference paper at ICLR 2023
herself. Therefore, we use the fairess gap of party's standalone model represent the party's bias. In contrast, in centralized training, all local training datasets from all parties are combined, and a centralized model 0g is trained using this global training dataset. We define the benefit of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines. The benefit of FI. for each party k is defined as the difference between the standalone model 02 and the FL model 9g. This difference reveals the extent to which a party benefits from participating in FL. Specifically, for a party k, we define the accuracy benefit of FL as Acc(O,, D2) - Acc(O,, Ds) and fairness benefit of FL as A6k, D&) - A0gg. Dz). Similarly, the benefit of collaboration is based on the difference between the standalone model and the centralized model. This difference indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset. We compute the average benefit of FL. or collaboration across all parties in the network. A positive benefit indicates that FL. (or collaboration) improves accuracy or fairesss, while a negative benefit implies that FL or collaboration has a detrimental effect.
Measuring bias aside fairness gap. Measuring the bias of model based on the fairesss gap reveals the model's performance disparity across groups but does not explain why the model biased. Specifically, it does not reveal whether the prediction of the model is attributed attribute or any other insensitive attributes whose distributions vary between groups. The former is typically referred to disparate treatment (direct discrimination) as the sensitive attribute directly influences the model prediction (Grabowicz et al., 2022] Zafar et al. (017). Towards measuring bias in this aspect, we employ the feature attribution method to explain the model's prediction on each data point. More precisely, given a model fe. and the input features I = (I1, T2, .... Td), the attribution of the prediction at input æ relative to a baseline input x' is vector (a),..., ag) where ag is the contribution of to the prediction of fo)).. We use Integrated Gradient (Sundararajan et al. 2017) to compute the feature attribution, which only requires a few calls to the gradient operation Following the suggestions by (Sundararajan et al. (2017), we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features. For the binary sensitive attribute. we use the opposite feature value the baseline. For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points). By employing this feature attribution, we are able to discover the bias associated with disparate treatment. For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al. 2017).
the sensitive
EMPIRICAL ANALYSIS
Our objective is understand how FL impacts fairness for parties. We start by comparing the average performance between FL and baselines (i.e., standalone training and centralized training) to answer the following question: does FL improve fairness for parties compared to standalone training, and does FL provide the same benefit as centralized training in terms of fairness? Our findings in Section 3.2 demonstrate that FL can exacerbate fairness issues for parties and does not retain the fairness benefit of collaboration, thus not achieving comparable performance to centralized training.
We further investigate how FL impacts performance for each party to answer the following questions: does FL provide the same benefit in terms of fairness for parties, and what causes the disparate impact of FL on parties' fairness? Our analysis in Section 3.3 shows that FL can propagate bias among parties. As result, FL improves fairness for parties with greater bias while worsening faimess issues for parties with less bias.
Finally, we investigate how the bias is propagated in FL. The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local updates, which are propagated to the aggregated model and ultimately to other parties via parameter aggregation.
3.1 SETUP
We use the two datasets with different tasks for our empirical analysis: US Census Data and CelebA (Liu et al. 2015). The US Census dataset consists of census data from different places in the US. We naturally partition the dataset based on the source of the data. Thus, we have 51 parties in total, representing 50 states in the US and Puerto Rico. We consider three tasks defined
3

---

Published as a conference paper at ICLR 2023
Table Average benefit of collaboration and FL. The lowest accuracy and highest fairness gap are bold. The standard deviation across five runs is indicated between parentheses. The green arrows (red arrows) represent the positive (negative) impacts of FL or centralized learning compared to standalone training (i.e., FL or centralized training increases the accuracy or decreases the fairness gap compared to standalone training).
Accuracy
A RD
AP
Dataset
Standalone
Centralized
FodAvg
Standalone
Centralized
FedArg
Standalone
Centralized
FedAvg
Income- Race
.738 (.01)
.787 (.00)] 1
7785 (.002) 1
.506 (.026)
.478 (.013)
.514 [.02))
321 (.011)
.307 (.012)
.322 (.004) 1
Income- Sex
.738 (.001)
.787 (003)
.785 (002)
.146.000)
.177 (,030)
.196 (007)
.667(00))
.00 (,0881
.217 (005)
Health-
(,002)
1
.17112 (013)
(021)
(00))
(,004)
Health-
.683 (1017)
(002)
.09 (001) f
(000))
(347))
029 [005)
(012)
(88()
((04)
Employment- Race
.759 (.001)
.241 (.002)
823 (.001) 1
(.23)
.21 (,010) 1
.237 [000)
2257 (.09)
(.07)
.211 (.010)
Empleyyent- Sex
.75M (.001)
824 (000))
823 (000) 1
.92 (.004)
(005) 1
1059 (000)
854 (003)
(.07))
0038 (.066
CelchA (Age)- Sex
.812 (1004)
852 (000))
863 (D02)
(01)
.222 (,024)
1
.264 (013)
226 (006)
(.051)
.238 (016)
in the folktables (Ding et al. 2021): Income (ACSIncome) | Health (ACSPublicCoverage)) and Employment (ACSEmployment). We train two-layer neural network models for all the tasks. We consider the binary notion of sex (i.e., male and female) and multi-value race the sensitive attribute for every task. Each party has 1, 1,00 training points and 2, 000 test points. Note that we use a large test. dataset ensure accurate estimation of the fairness gap with respect to parities' test distribution. For CelebA, an image dataset consisting of 200, 000 celebrity images, we focus on the age prediction task and use the binary attribute "Male" (male and non-male) as the sensitive attribute, which we refer to as the Sex attribute. We partition the data uniformly at random among all parties and train CNN models. We also consider non-IID partitioning on the CelebA dataset, and the results are presented in Appendix B In all experiments, we report the average results over five runs with different random seeds. We use FedML (He et al.] 2020p. a PyTorrhh (Paszke et al., 2019)-based library for FI., to train models in FL. We also use Captum (Kokhlikyan et al. (0200 to compute feature attribution. For further details on the datasets and models, please refer to Appendix
3.2 COLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE
Table I presents the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets. Centralized training observed to improve accuracy and fairness at the same time for parties. For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5.5% with respect to equalized odds and by 5.5% with respect to demographic parity. However, FL may not achieve the same benefit as centralized training in terms of fairness and can even exacerbate the fairness issue for parties. For example, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35%, and the DP fairness gap increases by 29.9%, compared to standalone models. We include results for other popular FL algorithms, including FedNova (Wang et al. 2020), lt al. (2020b). FedOpt (Reddi et al. 2020), FedProx [i et al. (2020), and Mime (Karimireddy et al. 2020a), Table 2 in Appendix [ which show similar pattern that of the FedAvg algorithm. Appendix B provides more detailed results on other FL algorithms. The centralized and FL models are trained on the same dataset. The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training. Therefore, the explanation of how bias is introduced during standard training in the centralized setting may not fully explain how FL. introduces bias. In the following, we further explore how FL impacts fairness from the party level.
3.3 FL PROPAGATES BIAS AMONG PARTIES
Disparate impact of FL on group fairness across parties. Figure I illlstraaes the benefit of FL. on fairness and accuracy for parties. We observe that FL improves accuracy for almost all parties, and the variance in accuracy improvement across parties is small. On the other hand, the fairesss benefit of FL negative for most parties. It implies that most parties obtain a more biased model in FL compared to standalone training. Furthermore, we notice that the variance in the fairness benefits across parties is large, suggesting that although all parties have the same global model in FL, they do not benefit from the FL model equally (cach party evaluates the model performance their local test dataset).
We use the Adult Reconstruction dataset Sarah et al. 2020
4

---

Published as a conference paper at ICLR 2023
Race (EO)
Race (DP)
Sex
Sex (DP)
Fairets benefit of
0:02
Coefficient:0.0 5
Coefficiept:07000 708..
Coefficient0.799
Coefficient:0.81 81
Better
6:02
-0.22
-03
-01
Worse
28 1.
022
0.2
8 (Standalone Model)
8 (Standalone Modell
8 (Standalone Modell
8 (Standalone Model
Figure 2: Correlation between the fairness gap of the standalone model and the benefit of FL Income. The x-axis shows the fairness gap of the standalone model, and the y-axis shows the faimess benefit of FI., which is the fairness gap of the standalone model subtracted by that of the FL model (defined in Section The Pearson correlation coefficients between the faimess gap of the parties' standalone models and the fairness benefit they obtain from FL are presented. The p-value for all settings is smaller than 0.0001.
Most biased party (A. 0.29)
Least biased party (4 0.04)
0.30
0.12
Local update Aggregation
0.10
0.36
0.08
0.34
0.06
0.32
0.04
0.30
0.00
Round
Round
Figure 3: Dynamic of fairness gap during the training Income (Sex, DP). Figure shows the fairness gap of the global model and locally updated model for most biased party and least biased party in the first 30 rounds (Figure in Appendix [ shows the results for all training rounds.). The most (least) biased party has the highest (lowest) fairness gap in the standalone setting. The fairness gap in the standalone setting for those parties is shown in the title.
To explore the impact of FL on fairness at the party level, Figure 2 shows strong correlations between the fairness benefit party obtains in FL and the fairness gap of the standalone model for the party, i.e., the bias level of the party. This finding highlights the disparate impact of FL fairness: FL can improve fairness for more biased parties
of
0.00
-0.05
but at the cost of worsening the issue for less biased parties.
-0.10
Contradiction between aggregation and local update. During the training process of FL, we observe that aggre- gation and local update contradict each other, shown in Figure Local update from the least biased party, whose standalone model has the lowest faimess gap, reduces the fairness gap the model. However, this reduction is elimi- nated by the aggregation step. Conversely, the local update from the most biased party increases the fairness gap of the model, which is then reduced by aggregation. This finding implies that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties but worsening the faimess for less biased parties compared to the standalone setting.
-0.15
Accuracy
DP
Figure Accuracy and Fairness Bene- fit of FL Income (Sex). The benefit of FL is the increase in accuracy reduc- tion in the fairness gap of the FL model compared to standalone training.
Biased parties negatively influence other parties via aggregation throughout the training. Why does aggregation have disparate impacts on local fairness for parties? Specifically, we ask which party's local update causes the increase (or decrease) of the fairness gap for other parties via
5

---

Published as a conference paper at ICLR 2023
Income (Sex)
Income (Race)
50
28
50
04
0.20
0.35
23
26
33
37
40
13
27
29
32
33
41
0.00
Figure 4: Inftuence graphs - Income (DP). The figure shows the most influential pairs of parties (with respect to demographic parity), with nodes the top representing parties that influence other parties and the nodes on the bottom representing parties that are influenced by others. The number inside the node represents the party index, and the color of the node represents the fairness gap in the standalone setting. Green edges (resp. red edges) connect pairs of parties where the top party positively (negatively) influences the bottom party. We show the top pairs with maximal positive influence and the top 5 pairs with maximal negative influence.
Race (DP)
Sex IDP)
Race (DP)
Sex (DP)
Comulativv wef infueence
Confficient.
Confriceett-..a
Positive
Top S Least Blased
Most Blased
Infuea
Positive
0.2
-029
Negatile
.050
Negative
(Sandda Modell
Sttandatooo Model)
Figure 5: Correlation between the influence and standalone bias Income (DP). The y-axis represents the average influence of each party. The Pearson correlation coefficient between the fairness gap party obtains in the standalone setting and the impact she has on local fairness for parties in FL. are shown in the figure.
Figure 6: Cumulative influence Income (DP). The y-axis shows the cumulative influence of each party on all parties up until the current round. The results for the top five most biased parties (parties with the highest fairness gap) and the top five least biased parties (parties with the lowest fairness gap) are presented.
aggregation. To answer this question, we look into the influence of a party's local update on other parties' fairness via aggregation. More precisely, we compute the influence of party party j as the fairness gap increase when party i's local update is removed from the aggregation in each round t and sum over all the training rounds. Formally, we define the influence of party i on party j as T. Ei= A(0,, Dÿ) - A0,, Dy). where 0g is the global model (i.e., the aggregated model overall local updated models) and Or. ; is the aggregated model over local updated models from parties excluding party i. If party improves fairness for party j. the influence is positive and vice versa. We compute the influence for all pairs of parties, and the most influential pairs are shown in Figure We observe that a less biased party has a positive influence on fairness for other parties, while more biased party has negative influence other parties. This result shows that biased party can negatively influence other parties' fairness via aggregation throughout the training.
Furthermore, we investigate the relationship between party's bias (i.e., the bias of the party's standalone model) and its average influence on all parties' fairness (i.e., ΣK 4,,//K K). The results are presented in Figure 5] which shows strong correlation between these two factors. This finding further supports our conclusion that the more biased parties have a stronger negative influence on other parties" fairness, while the less biased parties have a stronger positive influence.
Finally, we analyze the dynamics of the influence of the top most biased parties (i.e., parties with the largest fairness gap in the standalone setting) and the top 5 least biased parties, as shown in Figure 0 We observe that the influence of the most biased parties is monotonically increasing throughout the training, while the influence of the least biased parties is decreasing. In other words, biased parties consistently have a negative influence on others' fairness gaps, while less biased parties have positive influence. These results suggest that FL propagate bias among parties: the bias from biased parties negatively influence the faireess of other parties via aggregation throughout the training. Next, we will explore how the bias is propagated in FI..

---

Published as a conference paper at ICLR 2023
Centralized Model
Standalone Model
FL Model
Ma
Male:0.087
Male:0.161 Female:0.07
Male:0.161
Female
Female:-0.057
Female:-0.152
Proportion
0.06
0.02
0.00
-10
-0.5
0.5
-1.0
-0.5
0.0 0.5
1.0
-1.0
-0.5
0.0
1.0
Altribution Values
Attribution Values
Attribution Values
Figure 7: Histogram of feature attribution value for the sensitive attribute Income (Sex). The average attribution value for the female group and male group shown the figure. The results are computed on all test data points over five different runs. We show the results for the most biased party with the highest fairness gap in the standalone setting.
Party
Least Party
Most Blased Party
Least Blased Party
Attribution Value
FedAyg
Attribbttion Value
Before Agg Aggrogation After Aggregation
Standalone
Centralized
Rouuudo
EpochSRound
Round
Figure 8: Dynamic of absolute attribution value for the sensitive attribute Income (Sex). The results of different models are shown in different lines. We present the results for the most biased party and the least biased party.
Figure 9: Effect of local update and aggregation on the attribution value Income (Sex). We show the attribution value for sex attribute before and after aggregation for the most biased party and the least biased party during the training.
3.4 How BIAS PROPAGATED IN FL?
Disparate treatment causes large fairness gaps. Our first investigation explores what the bias represents. specifically whether the bias increase in FL is directly caused by the disparate treatment of the model among sensitive groups. To answer this question, we utilize Integrated Gradients (Sun- dararajan et al., (017)) to measure the attribution of each input feature to the models' predictions with respect to the positive class. Figure 7 shows the attribution value distribution for the sensitive attribute "Sex" over individual test points from the female and male groups. We notice that the sex attribute has large attribution value for the standalone model's predictions and the FL model's predictions. This finding implies that the predictions of those models are heavily dependent on the sensitive attribute of the test data. Moreover, the sensitive attribute affects the model predictions differently for the male and female groups, with the average attribution value being positive for the male group and negative for the female group. Furthermore, we find that there is minimal difference in the attribution value for other attributes with respect to the female and male groups (see Figure in Appendix B). This indicates that the large fairness gap in the models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it mainly caused by the models' disparate treatment of protected groups
FL model learns more biased patterns. In Figure 7 for different models and find that, while the predictions of the FL model depend less heavily on the sensitive attribute compared to those of the standalone model, the dependence is still stronger than that of the centralized model. This suggests that the FL. model learns more biased pattern compared to what could be learned in the centralized setting. Figure & shows the dynamic of the average absolute feature attribution for "Sex" during the training. We find that standalone training increases the model's dependence on the sensitive attribute throughout training for the most biased party. but centralized training decreases it. This suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features. In FL, however, the absolute attribution value barely changes after 100 rounds and remains significantly greater than
we compare the attribution value of "Sex"
7

---

Published as a conference paper at ICLR 2023
Local update vs. Aggregation
Parameter Norm Blas
(c) Effect of scaliigg
Appproption
0.205
Parameter Norm
0.76
0.8
Falmess Cap
0.200
Accuracy
0.21
0.74
Accuracy
0.0
0.185
0.73
4
Coef
4
0.190
0.50
50
100
150
200
0.0
0.3 0.4
10
301
Round
55 (Standalone Model))
Scaling factor
Figure 10: Effect of few parameters on fairness Income (Sex). (a) Effect of local update and aggregation on the parameter values: The y-axis shows the norm of parameters that are directly computed on the sensitive attribute, normalized by the parameter norm of the first layer. We show the dynamic of this parameter norm during FL for the aggregated model and locally updated model for the parties who benefit most from FL and suffer the most from FL (represented by the line with "Benefit" and "Suffer" respectively). (b) Correlation between standalone bias and parameter values: The x-axis shows the bias party receives in the standalone setting, and the y-axis shows the normalized norm of parameters (associated with sex attribute) for the locally updated model in the last round. (c) Effect of scaling the parameter values: The figure shows the model performance in terms of accuracy and fairness gap when the parameter values (associated with sex attribute) are multiplied by scaling factor.
that in the centralized setting. This implies that the FL algorithm may introduce bias to the final model by inhibiting the model from learning less biased features.
Biased parties increase the model dependence on the sensitive attribute. Figure 9 shows the attribution value of the aggregated model and locally updated model from the most biased party and least biased party. We observe that the biased party increases the global model dependence on sensitive attributes during the local update, and this increase persists throughout the training. In contrast, the least biased party reduces the dependence on the sensitive attribute, which is aligned with the trend of the fairness gap Figure 3 This suggests that the biased parties have a negative impact on the fairness of other parties by increasing the model's dependence on the sensitive attribute.
Bias is encoded in a few parameters. Biased parties increase the model dependence on the sensitive attribute. But how does this increase propagate to other parties in the network? Since parties share the model parameters of the local model with the server, the bias is likely encoded in the model parameters. Therefore, we investigate which parameters are related to the model's bias. Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction. If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will significantly affect the model's prediction. In our evaluation, the sensitive attribute is part of the input feature, so the parameters directly applied to the "Sex" attribute in the first layer of the neural network should contribute to the model's bias.
Figure 10 a) shows the normalized norm of the parameters associated with the sensitive attribute for the most and least biased parties in the aggregated and locally updated models. The normalized norm defined as the norm of parameters associated with the sensitive attribute divided by the parameter norm of the first layer. We observe that the least biased party reduces the parameter norm, hence decreasing the model's s sensitivity to the sensitive attribute. On the other hand, the biased party increases the norm for those parameters, amplifying the impact of the sensitive attribute on the model's 'ss prediction. Through aggregation, this amplification will be propagated to the global model. Figure| TT(b) reveals moderate correlation between the fairness gap party experiences in the standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information. This implies that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model's susceptibility to the sensitive attribute.
Controlling fairness gap by scaling a few parameters. To further investigate the impact of the parameters associated with the sensitive attribute (i.e., 104 parameters out of 1, 792) model fairness, we examine the effect of scaling these parameters on the fairness gap in Figure [) c). We find that the fairness gap can be greatly widened narrowed at the expense of moderate degree of
8

---

Published as a conference paper at ICLR 2023
accuracy. Specifically, by scaling the parameter value by 0.1 for the trained FL model, significantly reduce the EO gap by 74.6% (from 0.198 to 0.05) and the DP gap by 69.1% (from 0.217 to 0.067) with just moderate accuracy loss of 0.8% (from 0.785 to 0.779). In contrast, scaling the same set of parameters by a factor of 10 increases the EO gap to 0.96 (the maximal fairness gap is 1), which is almost five times larger, and increases the DP gap by 259% 0.78, while reducing the accuracy from 0.785 to 0.67. These findings explain how bias is propagated in FL: biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes. This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties. Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.
4 RELATED WORK
Fairness has received considerable attention due to the growing deployment of machine learning in decision-making processes. Various definitions of fairness have been presented (Hardt et al. |016|Dwork
Dwork ct al. 2012 Calders et al., 2009). Specifically, group fairness requires that the model behave similarly for groups defined by sensitive attribute (e.g., race). While how machine learning algorithms propagate data bias to the final model has been extensively investigated in a centralized setting (Blum and Stangl 2020 Lakkaraju et al. 2017 Rambachan and Roth 20200] [Friedler et al., 2019 Dullerud et al. (022), the effect of FL on model fairness is not yet fully understood.
The existing literature faimess in FL mainly focuses the performance disparity of FL models across parties, rather than demographic groups (Li et al. [021] Zhao and Joshi, 2022 Li et al. 2019 Mohri et al. [219;; Deng et al., 2020) Donahue and Kleinberg, [021]] Hao et al. 2021] Zhou et al. 2021; Yu et al. 2020 Lyu et al.) 2020). However, we focus on group fairness, which concerns performance disparity among groups In terms of group fairness, Abay et al. (2020) listed a few potential sources of bias in FL. Recently, considerable progress has been made in training group fair models in FL (Abay et al. 2020; Chu al., [021; Zeng et al., 2021a| Hu et al., 2022: Du et al. 2021 Ezzeldin et al. [2021) }. Nonetheless, the majority of these works suggest techniques for achieving fairness on a single test distribution. Instead, we focus on fairness issues for parties. Some studies (Cui et al., | Papadaki et al., (022) proposed algorithms to improve local fairness for partics. Our purpose, instead, is to gain a comprehensive understanding of how FL influences local fairness on its own, which we believe is equally crucial designing fair algorithms.
5 FUTURE WORK & CONCLUSION
Future Work. In this work, we have investigated how bias is propagated in FL when the sensitive attribute is included as an input feature. In practice, however, sensitive attributes may be prohibited from being included in input features. In such situations, the model may still be heavily biased due to variables that are correlated with the (unobserved) sensitive attribute. For instance, a person's 's zip code may be highly correlated with their race, phenomenon known as "redlining". A promising direction for future work is to identify which model parameters contribute to the bias and to audit the bias propagation in this setting. Another important direction is to design FL algorithms that are robust we briefly discuss a few potential ways to achieve this goal.
to bias propagation. In Appendix E
Conclusion. Federated learning has become increasingly popular in various applications with significant individual-lever consequences, making it essential to anticipate the possible bias introduced by FL. Our paper takes the first step in this direction by providing a comprehensive analysis of the impact of FL on local fairness for parties. We demonstrated that the FL. algorithm could introduce bias on its own which may exacerbate the issue of fairness for the involved parties. Moreover, we showed that this exacerbation is not evenly distributed among parties, FL can propagate bias among them. Finally, we explained how bias is propagated in FL: biased parties encode their bias into the local updates by increasing the signal of few parameters steadily throughout the training process, which is then propagated to the global model via aggregation and, ultimately, to other parties.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENT
The authors would like to thank Ergute Bao. Ta Duy Nguyen, and Martin Strobel for their valuable feedback on earlier versions of this paper, as well as the anonymous reviewers for their insightful comments. This research is supported by Google PDPO faculty research award, Intel within the www.private-ai.cm center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award num- ber NUS ECRA FY19 P16), and the National Research Foundation, Singapore under Strategic Capability Research Centres Funding Initiative. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Research Foundation, Singapore.
7 ETHICS STATEMENT
Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination. Auditing the bias with respect to other definitions of fairness, such as individual fairness, may result in different conclusions. Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute. We recognize that there are numerous protected groups outside those considered in the analysis, such as those defined by multiple sensitive attributes. The propagation of bias against fine-grained subgroups may be even more substantial than we found in the paper.
REPRODUCIBILITY STATEMENT
We provide details about the model, datasets, and implementations in Appendix A and the code for the paper is available at http:///////ttttoooooooootttrtra
in FL
REFERENCES
Annie Abay, Yi Zhou, Nathalie Baracaldo. Shashank Rajamoni, Ebube Chuba, and Heiko Ludwig. Mitigating bias in federated learning. arXiv preprint arXiv:2012.02447, 2020.
Muhammad Asad, Ahmed Moustafa, and Takayuki Ito. Fedopt: Towards communication efficiency and privacy preservation in federated learning. Applied Sciences, 10(8):2864, 2020.
Avrim Blum and Kevin Stangl. Recovering from biased data: Can fairness constraints improve accuracy? In /st Symposium on Foundations of Responsible Computing, 2020.
Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops, pages 13-18. IEEE, 2009.
Lingyang Chu, Lanjun Wang, Yanjie Dong, Jian Pei, Zirui Zhou, and Yong Zhang. Fedfair: Training fair models in cross-silo federated learning. arXiv preprint (tt.::/000000666. 2021.
Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, al. Ben20000: Dermoscopic lesions in the wild. arXiv preprint
2019.
Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. Fair and consistent federated learning. arXiv e-prints, pages arXiv-2108, 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Distributionally robust federated averaging. Advances Neural Information Processing Systems, 33:15111-15122 2020.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. arXiv preprint arXiv:2108. :0t0:::20000088884.. 2021.
Kate Donahue and Jon M. Kleinberg. Models of fairness in federated learning. CoRR, abs/21 12.00818, 2021. URL httpss:///aaat
10

---

Published as a conference paper at ICLR 2023
Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong. Faimess- agnostic federated learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 181-189.
SIAM, 2021.
Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, and Marzyeh Ghassemi. Is faimess only metric deep? evaluating and addressing subgroup gaps in deep metric learning. arXiv preprint arXiv:2203. 12748, 2022.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '12. page 214-226, New York. NY, USA, 2012. Association for Computing Machinery. ISBN 9781450311151. doi: 111452209023620002255 URL /ttp:///ooiiioo owo/o. 2090236. 2090255
Yahya H Ezzeldin, Shen Yan, Chaoyang He, Emilio Ferrara, and Salman Avestimehr. Fairfed: Enabling group fairness in federated learning. NeurlPS Workshop on New Frontiers Federated Learning (NFFL 2021), 2021.
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 329-338, 2019.
Przemyslaw A Grabowicz, Nicholas Perello, and Aarshee Mishra. Marrying fairness and explainabil- ity in supervised learning. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1905-1916, 2022.
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3310-3319, 2021.
Moritz HardT, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in su- pervised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL httpt: htttps::/prroceedingg wwwrips.u.... wtppscccpappeee wwcp//pappe//1///1... 112232288.3333000111144442771003
pdf
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938. PMLR, 2018.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. NewrIPS 2020 FL Workshop Best Paper Award, 2020.
Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Provably fair federated learning via bounded group loss. arXiv preprint 2rXiv:2203. 10/90, 2022.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowledge and information systems, 33(1):1-33, 2012.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in 2020a.
federated learning. arXiv preprint arXiv:2008.0336
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pages 5132-5143. PMLR, 2020b.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: unified and generic model interpretability library for pytorch, 2020.
11

---

Published as a conference paper at ICLR 2023
Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 275-284, 2017.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXiv: 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
Ji Liu, Zenan Li, Yuan Yao, Feng Xu, Xiaoxing Ma, Miao Xu, and Hanghang Tong. Fair representation learning: An alternative to mutual information. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1088-1097, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Lingjuan Lyu, Xinyi Xu, Qian Wang, and Han Yu. Collaborative faimess in federated learning. In Federated Learning, pages 189-204. Springer, 2020.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiee learning of deep networks from decentralized data, 2017.
Mchryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pages 4615-4625. PMLR, 2019.
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzellec, Constantin Philippenko, Santiago Silva, Maria Teleñczuk, Shadi Albarqouni, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi. Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Mare Tommasi, and Mathieu Andreux. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings, 2022.
Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Miguel Rodrigues. Federating for learning group fair models. arXiv preprint arXiv:2/ 10.0/999, 2021.
Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Miguel Rodrigues. Minimax demographic group fairness in federated learning. arXiv preprint arXiv:2201.00 08304, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Bucc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http:p//pppprrreerrrii....//rrrr cctppperrccppppp/// 9015 wwwpppyttochheee an- immmmetiiii ive styyleehigh-eeee hee-eii--oooooooe-eeeeeeeee deep- learning learnn library.. pdf
Ashesh Rambachan and Jonathan Roth. Bias in, bias out? evaluating the folk wisdom. In /st Symposium on Foundations of Responsible Computing, 2020.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush. Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2023
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):1-7, 2020.
Flood Sarah, King Miriam, Rodger Renae, Ruggles Steven, and Warren J. Robert. Integrated public use microdata series, current population survey: Version 8.0 [dataset], 2020. URL https
//wwwippmm http://pooojcttssppsssssee
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328. PMLR, 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 1 1000 dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 13:7611-7623, 2020.
Wensi Yang. Yuhang Zhang, Kejiang Ye, Li Li, and Cheng- Xu. Ffd: A federated learning based method for credit card fraud detection. In International conference on big data, pages 18-32. Springer, 2019.
Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A faimess--ware aware incentive scheme for federated learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 393-399, 2020.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreat- ment. In Proceedings of the 26th international conference on world wide web, pages 1171-1180, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pages 325-333. PMLR, 2013.
Yuchen Zeng, Hongxu Chen, and Kangwook Lee. Improving faimess via federated learning. arXiv preprint arXiv:21100 15.555, 2021a.
Ziqian Zeng, Rashidul Islam, Kamrun Naher Keya, James Foulds, Yangqiu Song, and Shimei Pan. Fair representation learning for heterogeneous information networks. In Proceedings of the International AAAI Conference on Weblogs and Social Media, volume 15, 2021b.
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? Advances in Neural Information Processing Systems, 33: -884,, 2020.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon. Conditional learning of fair representations. In International Conference on Learning Representations, 2019.
Zhiyuan Zhao and Gauri Joshi. A dynamic reweighting strategy for fair federated learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8772-8776. IEFF, 2022.
Zirui Zhou, Lingyang Chu, Changxin Liu, Lanjun Wang. Jian Pei, and Yong Zhang, Towards fair federated learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 4100-4101, 2021.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
This appendix is divided into five sections. Appendix A provides additional details about the experimental setups, including information about the models, datasets, and hyperparameters used. Appendix B presents further experimental results that support the claims made in the paper. In Appendix we discuss the results of an existing fair FL algorithm. AppendixDeext extends our analysis to real-world medical datasets. Finally, in Appendix bias in FI..
we discuss potential methods for mitigating
A DETAILS ABOUT EXPERIMENT SETUP
A.1 DATASETS AND MODELS
We explain the datasets and models used in the paper.
Census Dataset We use the datasets provided by folktables. In particular, we consider the ACSIn- come, ACSPublicCoverage. and ACSEmployment tasks defined in the forlktables. In the ACSIncome, the goal is to predict whether an individual's Income is above $50,000. In the ACSEmployment task, the goal is to predict whether an individual is employed. Similarly, in the Health (i.e.. ACSPublicCov- erage) task, the objective is to predict whether an individual is covered by public health insurance. We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks. For all the tasks, we use the RELU activation function. We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32. We train the NN models for 200 epochs. In FL, each client updates the global mdoel for epoch and shares it with the server. We encode the categorical features based on the encoding template provided in folktables. After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment. We consider Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black African American alone," " "American Indian alone," "Alaska Native alone," and "American Indian and Alaska Native tribes specified; or American Indian or Alaska Native, not specified and no other," " Assin alone," "Native Hawaiaan and Other Pacific Islander alone," "Some Other Race alone," "Two or More Races").
and race sensitive attributes.
CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1. We use 'same' padding for the CNN layer. Following this CNN layer, we have the Batch normalization layer and Max Pooling layer. After which, we have the connected layer. We train the model for 500 communication rounds or epochs with SGD optimizer. The learning rate is 0,1, and the batch size is 128. The train, test, and validation datasets ratio for each party 6:2:2.
A.2 HYPER-PARAMETER FOR FL ALGORITIIMS AND IMPLEMENTATION
In our paper, we also evaluate various popular FL algorithms. including FedNova /Wang et al. 2020), Scaffold Karimireddy et al. (2020b), FedOpt (Reddi et al.) 2020). FedProx Li et al. (2020), ) and Mime (Karimireddy et al. 2020a). We use the same local learning rate and the number of the local epoch as in FedAvg. We provided the detailed pypem-easameters for each of the algorithms in our code (See supplementary). We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs.
A.3 DATA HETEROGENEITY
Figure IT shows the fraction of samples for each subgroup across all parties, while Figure 12 shows the histogram of subgroup proportions for each party. We observe that the parties have different fractions of samples from each group in the Income dataset, indicating that their local data distributions are dissimilar. In contrast, for the Health and Employment datasets, parties have similar fractions of samples from each subgroup, suggesting that their data distributions are more alike than those of the Income dataset.
14

---

Published as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 BIAS PROPAGATION EFFECT
Census dataset We show the bias propagation effect of FL for the Health and Employment task in Figure and respectively.
CelebA We show the bias propagation effect of FL on CelebA dataset. Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the "Not Young" age label). In this way, we aim to change the data bias in the local training datasets for clients. In particular, in setting (ii), half the parties have more samples from the minority subgroup than another half of the parties. The ratio is 8:2. In setting (iii), single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties. the figure, we find that the in the non-iid setting, the more biased parties have larger and more positive benefit, while FL hurts the less biased parties. Figure shows the correlation between the fairness gap of the standalone model and the benefit party gets the FI.. We find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL. hurts the less biased parties.
B.2 AGGREGATION CONTRADICTS WITH LOCAL UPDATE
Figure shows the dynamic of the fairness gap for the aggregated model and locally updated model from the most biased party and least biased party during the training.
B.3 INFLUENCE SUB-GRAPHS
We show the top 10 maximal positive and top 10 maximal native influence client pairs in Figure 17 and the top 15 in Figure [8] We can see that the less biased client has a strong positive influence on other clients while the more biased client has a strong negative influence on other clients.
B.4 ATTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE
We show the attribution value for all input features for the most biased party and least biased party in Figure 19 and Figure 20 respectively. We observe that the attribution value for other features similar for female and male groups. However, the sex attribute has the largest attribution value and affects the groups differently. It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") )) for making predictions.
B.5 EFFECT OF FL ALGORITHM
Table shows results on other FL algorithms, including FedNova (Wang al. 2020). Scaffold Karim- ireddy et al. (20205), FedOpt (Asad et al., 2020), FedProx Li et al. (2020), and Mime (Karimireddy al.) 2020a). We find that when the FL model achieves higher accuracy than standalone models, the fairesss gap of the FL model can be higher than that of standalone models. This observation consistent across multiple FL algorithms. This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness. In addition, this is not unique to only the FedAvg algorithm.
B.6 GROUP PERFORMANCE
In Figure 21 we present the model ROC on groups to illustrate the performance disparity across groups. Surprisingly, there is no huge difference between the ROC between groups. In contrast, Figure 22 shows the noticeable difference between groups with respect to the reecision-recall curve, especially for the most biased party. The main reason is that the dataset is imbalanced. Thus. ROC is usually misleading. On the precision-recall curve, we find that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group)
15

---

Published as a conference paper at ICLR 2023
Table 2: Benefit of collaboration on local datasets Various FL algorithms (Income). The
standard deviation across five runs is indicated between parentheses.
Race
Sex
Algorithm
Accuracy
A AO
DP
A A EO
DP
Standalone
0.738 (0.001)
0.506 (0.026)
0.321 (0.011)
0.146 (0.007)
0.167 (0.005)
FedAvg
0.785 (0.002)
0.514 (0.021)
0.322 (0.004)
0.198 (0.007)
0.217 (0.005)
FedNova
0.785 (0.002)
0.517 (0.023)
0.322 (0.004)
0.203 (0.008)
0.22 (0.006)
Scaffold
0.695 (0.007)
0.406 (0.213)
0.233 (0.156)
0.099 (0.047)
0.077 (0.025)
FedOpt
0.78 (0.001)
0.525 (0.023)
0.322 (0.008)
0.201 (0.005)
0.218 (0.004)
FedProx
0.783 (0.002)
0.515 (0.024)
0.324 (0.006)
0.199 (0.011)
0.219 (0.009)
Mime
0.786 (0.002)
0.521 (0.027)
0.321 (0.01)
(0.00)
0.218 (0.007)
B.7 EFFECT OF SCALING OTHER PARAMETERS
Figure 23 shows comparison of the model's performance when scaling different model parameters. Specifically, we examine the impact of scaling the parameters in the first layer of the neural network that does have any computation on the Sex attribute (referred to as "Other parameters"), well as the parameter related to the Sex attribute (referred to "Related parameters"), The results indicate that up- up-scaling or down- scaling other parameters do not significantly affect the fairness gap while scaling the related parameters has considerable impact on model fairness. This finding supports our hypothesis that the model's bias is primarily encoded few model parameters. FL, biased parties introduce bias during local training by increasing the weights for those related parameters.
16

---

Published as a conference paper at ICLR 2023
Table Effect of Fair FL (Abay al., 2020) (Income). The standard deviation across five runs is indicated between parentheses. The "Sensitive Attribute" column indicates the sensitive attribute used in the reweighting algorithm.
Race
Scx
Algorithm
Sensitive Attribute
Accuracy
A 200
DP
A20
A
Standalone
0.738 (0.026)
0.506 (0.19)
0.321 (0.117)
0.146 (0.071)
0.167 (0.063)
FodAvg
0.785 (0.017)
0.514 (0.195)
0.322 (0.086)
0.198 (0.05)
0.217 (0.037)
Global reweighting
Race
0.783 (0.002)
0.353 (0.016)
0.212 (0.012)
0.204 (0.006)
0.219 (0.005)
Local reweighting
Race
0.783 (0.002)
0.367 (0.008)
0.222 (0.005)
0.196 (0.005)
0.214 (0.006)
Global reweighting
Sex
0.781 (0.001)
0.516 (0.031)
0.318 (0.01)
0.044 (0.005)
0.084 (0.003)
Local reweighting
Sex
0.781 (0.001)
0.521 (0.03)
0.323 (0.004)
0.045 (0.005)
0.08 (0.005)
C EVALUATION ON EXISTING FAIR FL
Table presents the results for fair FL algorithm proposed by Abay et al. (2020), which uses reweighting algorithm proposed in the centralized setting (Kamiran and Calders 2012). The algorithm involves assigning weights to each subgroup (defined by the label and a sensitive attribute) before FL, based on the local or global training dataset. This is called "local reweighting" or "global reweighting." respectively. During FL, parties update the FL model to minimize the weighted loss. The table presents the accuracy and fairness gap of the FL algorithm using local and global reweighting for the Income, Hcalth, and Employment datasets.
We found that applying reweighing algorithms in FL reduces the average fairness gap across parties. This is due to the fact that, after reweighing, the parties that were initially biased do not introduce significant bias to the model during local training, resulting in a reduction of the fairness gap in the FL model.
However, we also noted that the fairness gap in the model remains high for the Race sensitive attribute, indicating that the global and local reweighing algorithms do not entirely eliminate bias in FL. Furthermore, we would like to highlight some concerns regarding the fair FI. algorithm:
1. The fair FL. algorithm enhances the performance of the minority group the expense the majority group. Figure 24 illustrates that the minority group (i.e., the Female group) has a better performance after the reweighing, but at the cost of reduced performance for the majority.
2. When improving faireess with respect to one sensitive attribute, the fairness issue with respect to another sensitive attribute may worsen, as shown Table For instance, when applying local reweighing to reduce bias with respect to the "Sex" attribute, the fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521, indicating that reweighing improve fairness with respect to Sex may amplify the bias with respect to Race. This implics that the bias with respect to Race can still propagate in FL, and our analysis remains valid.
3. The reweighing algorithms assume that parties are interested in mitigating the bias with respect the same sensitive attribute, which may not be the case in practice. Figure 25 shows the fairness gap of standalone models with respect to different sensitive attributes. We observe that many parties have a low fairness gap with respect to one sensitive attribute and high fairness gap with respect to another sensitive attribute. Thus, parties may aim to enhance faimess with respect to different sensitive attributes, which renders this reweighing algorithm unsuitable. This problem more prevalent in FL, given the different data distributions of parties.
17

---

Published as a conference paper at ICLR 2023
Table 4: Efffect of FL on the real-world medical dataset (ISIC2019). The standard deviation across four runs is indicated between parentheses.
Setting
Party
Party
Party
Party
Algorithm
Accuracy
Adce
Accuracy
Acc
Accuracy
Acc
Accuracy
A
Standalone
658 (.008)
004 (.018)
972 (,009)
.013 (.007)
.77 (.024)
.06 (.04)
.717 (.057)
.019 (.006)
FedAvg
.546 (.117)
.056 (.044)
908 (.118)
.047 (.058)
.702 (.108)
.13 (.025)
.536 (.108)
.066 (.027)
Centralized
.672 (.056)
.033 (.016)
.973 (.007)
,011 (.01))
.782 (.02)
.06 (.032)
.725 (.019)
.056 (.049)
D EXTENDING TO A REAL-WORLD MEDICAL DATASET
We evaluate the effect of FL on local faimess real-world medical dataset, ISIC2019 (Abay et al. 2020; Tschandl et al. 2018: Combalia et al.) 2019), which contains dermoscopic images of skin lesions collected from six medical centers. The task is classify dermoscopic images among nine different diagnostic categories. We filter out medical centers with less than 1900 images and regard each medical center with over 1900 images as party (4 centers remain after filtering). For every party, we randomly select 1500 and 400 images for training and validation, respectively. We regard the binary notion of Sex as our sensitive attribute. In this multi-class and medical dataset setting, we measure the bias (fairness gap) based on the accuracy gap across groups. Following (Ogier du Terrail et al. 2022 we train EfficientNets (Tan and Le, 2019), the state-of-the-a model structure for medical data, under the centralized, FL, and standalone setting. The accuracy and fairness gap for each client in different settings are shown in Table
FedAvg does not improve accuracy, possibly due to data heterogeneity among parties (as shown in Figure 26). resulting in the model's instability and slow convergence, which known as 'client-drift' issue Karimireddy et al. 2020b
Morcover, the FedAvg model results in a higher accuracy gap between groups. worsening the fairness issue compared the standalone setting the centralized setting (see the faimess gaps for party 1, party 3, and party 4 as examples). Our results suggest that, in real-world settings, FedAvg may not improve accuracy compared to the standalone setting for the local data distribution and may even exacerbate the fairness issue, leading to a more biased model.
18

---

Published as a conference paper at ICLR 2023
E POTENTIAL MITIGATION
In this section, we suggest some potential solutions to reduce the effect of bias propagation in FL. Our hope is that this will inspire further research developing fair FL in the future.
Personalized FL Our study has revealed that aggregate updates, which involve the average local update from all parties, may result in higher bias than local updates for less biased parties (see Figure() 3). This can cause bias from other parties to propagate to the local model, thus increasing the local fairness gap. To mitigate the bias propagation effect, one possible approach to avoid completely overwriting the local model with the global model. Personalized Federated Learning (PFL) techniques (Li et al. 2021) have recently been proposed to address this issue by focusing on improving the model's accuracy. PFL can be promising direction for developing fairness- personalized federated learning approaches that can help reduce the bias propagation effect in FL.
Fair Representation Learning Our analysis demonstrated that bias is encoded small number of parameters in the first layer of the neural network, which is typically considered the feature extractor (as illustrated in Figure TO). This suggests that the FL model is biased because the learned representation is biased, heavily influenced by the sensitive attribute. To address this issue, one possible solution et al. 2019: Liu et al. 2022 Zeng al. 2021b) that aim to learn feature extractor which is fair, meaning that minimizes the dependence on the sensitive attribute while still retaining enough information about the task (or input features). Therefore, using fair representation learning FL can potentially mitigate the propagation of bias effects.
implement fair representation learning techniques (Zemel al. [2013 /Zhao
19

---

Published as a conference paper at ICLR 2023
Income (Sex)
Base
Employment (Sex)
(Sex)
0.32
0.38
6.21
0.10
0.26
0:33
0.21
0.20
0.30
0:43
0.11
0.36
0:31
6.31
0.24
8:15
0.31
0.25
6.21
0.20
0.31
0.32
0:19
0.30
0.32
0.35
0.20
0.13
0.27
9.31
0.22
0.20
0.31
0.37
0.14
0.38
0.34
0.27
0.25
0.16
9.27
0.23
0.41
0.36
0.23
0.90
0.25
0.20
0.24
3.28
0.25
0.23
0.28
0.39
0.16
0.37
0.27
0.34
0.24
0.16
0.23
0.29
0:25
0.23
0.26
0.39
0:15
0.39
0.32
0.38
6.20
0.13
0.20
0.30
0.23
0.20
0.35
0.44
0.10
0.12
0:30
0.22
0.45
0.09
0.11
12
11
0.31
0.19
0.27
0.25
0:15
0.23
9.29
0.25
0.23
0.42
0.17
0.31
0.37
0.22
0:10
0.23
0.29
0.24
0.22
0.29
0.43
0.13
0.15
0.33
0.57
0.20
0:10
0.25
0.27
0.42
0.14
0.30
13
0.32
0.30
0.21
0:10
0.33
0.47
0.10
0.00
16
0.32
0:30
0.20
0.50
9.22
18
0:32
0:39
0.28
0.28
0.2%
0.77
0.43
0.36
15
0:23
0.25
0.21
0.23
3.28
0.25
0.24
0.29
0.41
0.14
0.36
30
0:23
9.31
0.27
0.19
0.27
0.30
0.1.
0.18
22
0.31
0.35
0.22
0:12
22
0.27
0:38
0.16
0.39
25
0:28
0.24
9.26
0.41
0:15
0.:22
13
0.32
0.28
3.32
0.20
0.19
23
0.32
0.29
0.28
0.29
0:24
0.22
Cient-In ID
0.44
Ciient
12
0:35
0.25
0.27
9.27
0.24
0.22
0.60
0.37
0.32
0.36
0.21
0:11
0.33
0.47
0.10
0.09
0.32
0.34
0.21
0.13
0.31
0.41
0.15
0.35
0.26
0.35
0.25
0:14
0.28
0.24
0.30
0.45
0.12
0.34
23
0.23
0.24
3.27
0.25
0.23
50
0.29
0.31
0.20
0.19
0.27
0.23
R1
0.29
0.35
0.21
0.31
0.37
0.21
0.12
0.31
0.44
0.11
0.34
0:79
0:34
0.7%
0:10
0.27
0.22
0.10
0.47
0.09
0.17
0.30
0.36
0.23
0:22
0.29
0.25
9.23
0.28
0.40
0.15
0.38
0.34
0.32
0.77
0:31
0.2%
0.19
0.64
0:11
0.:2
0.26
3.29
0.23
0.21
0.29
0.22
0.42
0.36
0:249
0.25
0.60
0.16
0.38
0.32
0.38
0.20
0:11
0:30
0.21
0.32
0.43
0.12
0.34
33
0.33
0.40
0.18
0.08
0.25
3.27
0.24
0.24
0.32
0.47
0.09
0.11
41
0.33
0.37
0.20
0:1
0.25
0.30
9:25
5.22
0.31
0.02
0:12
0:30
42
0.30
0.24
0.30
0.24
0.21
48
0.31
0.19
0.25
0.31
0.27
0.17
0.28
0.24
0.32
0.48
0.10
0.11
0.26
0.31
0.27
0:16
0.25
3.29
0.24
0.22
0.26
0.43
0:1.6
0.32
0.33
0.38
0.20
0.09
0.29
0.33
0.20
0.19
0.25
0.34
0.19
0.22
8
0.30
3.26
8.23
0.39
0.23
Freaei Fraction of samples
Fraction of semples
Fraction of samples
Figure 11: Fraction of samples for each subgroup Sex
20

---

Published as a conference paper at ICLR 2023
Proportion over parties
Income
Health
Employment
Male (+)
Male (+)
Male (+)
0.4
Female (+)
Female (+)
Female (+)
0.3
0.1
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Fraction of samples
Fraction of samples
Fraction of samples
Figure 12: Histogram of the fraction of samples for each group with positive label Sex
Race (EO)
Race: (DP)
Sex (CO)
Sex (DP)
Falmess benaffs of
Coefficient:0.88
Coefficient:0.875
Coefficient:0.
Coefficient:0.991.
0.
value:0,0
value:0.411
valus:0.0
0.55
Better
0.00
0.00
183
-02
-200
-0.10
00
0.2
0.6
0.1
U.S
000
0155
0.30
0.00
0,020
0000
Actendacce
Accuntatone
Figure 13: Correlation between the fairness gap of the standalone model and the benefit obtained from FL Health
Fairess berreit of FL
Roce (FO)
Race (DP)
Sex (EDI
Sex (DP)
Coofficient:00.o 8:43
Coofficient:0.22 2,78
Coefficient:0.882
Coefficient00.801
value-0.0,
value: O4
value:0.8
511
P value:0.0.
Betteer
Worse
Accansstive
Accoosatie
Figure 14: Correlation between the fairness gap of the standalone model and the benefit obtained from FL Employment
Setting (i)
Setting (iil
Setting (lii)
0.10
0.2
Ramnes benelt of
Coefficient:0.211
Ramnor benette Fr
Coefficient:0.955 value:0.0
Farnass bondlt FL
Coefficient:0.855
value:0.139
9:1
value:0.0
Better
-0.05
-0.3
Worse
-0.10
-0.2
0.15
0.20
0.25
8.30
0.2
3.4
Setting (iI DP
Setting (ii)
Setting (ii8) DP
Falmers benent Ft
0.04
Coefficient:0.. .122 value:0.001
Faimuss bonettt of
0.05
Coefficient:0.992 value:0.0
Falmoss boncttt of Fz
A.1
Coefficient:0.82 value:0.0
0.00
0.00
0.0
0.02
Ascandionne
Figure 15: Correlation between the fairness gap of the standalone model and the benefit obtained from FL CelebA (Age)
21

---

Published as a conference paper at ICLR 2023
The most biased party (Age: 0.293)
0.36
0.34
0.30
Local update Aggregation
The least biased party (A 0.038)
0.12
Locall update Aggregation
0.10
0.
0.04
0.02
0.00
125
175
Figure 16: Dynamic of A DP during the training- Income (sex). Figure shows the faimeess gap for the aggregated model and local updated model from the most biased party and least biased party during the training. The most (least) biased party is the party with the highest (lowest) fairesss gap in the standalone setting. The fairness gap in the standalone setting for those parties is shown in the title.
(Sex)
50
0.25
0.20
13
18
22
24
25
26
34
40
47
(Race)
50
0.4
13
18
19
27
33
40
41
46
47
Figure 17: Influence subgraph- Income (A DP) maximal negative influential pair. The green edge and red edge represent the positive and negative influence, respectively. The color of the node represents the fairness gap the standalone setting.
Top 10 maximal positive influential pair and top 10
22

---

Published as a conference paper at ICLR 2023
Incoree (Sex)
50
0.15
0.10
0
1
3
11
13
16
18
22
23
24
25
26
34
40
48
Income (Race)
45
50
0.3
11
13
15
17
18
19
20
24
27
33
40
41
47
Figure 18: Influence subgraph Income (A PP) Top 15 maximal positive influential pair and top 15 maximal negative influential pair. The green edge and edge represent the positive and negative influence, respectively. The color of the node represents the fairness gap in the standalone setting.
Standalone training
Centralized training
Federated learning
Male
Termal
03
-0.2
Oe Recoration
Working four moudi
Kirth
Petaionnte
Citor mntter
Marial stats
Codo
Oetotation of bith
Worring how wnell Citos coloasional Martal taau
codo
Outonation bith Boaatonabi
nop snat Cits of woter
colcationll Hasitll Rinns
Roce cofo
Figure 19: Feature attribution value Income (Most biased party) We shows the feature attribution value for all the input features. The model trained FL and standalone setting has a large dependency on the sensitive attribute "Sex".
Standalone training
Centralized training
Federated learning
Hale
Foma
0.20
0.6
-0.55
-0:5
Age prossation
Modiang bniek
of buth
felectinal ollp
Enccatiinall atacttend
Clas of sookel
Rice code
Age cossoon
working bout for buss
of mucket
Marial alauo
ses Rlce coda
tge recoeation
working beer Hook Clas of wostie
at arainind
Mamell Rosuo
See
Rates codos
Figure 20: Feature attribution value- Income (Least biased party) We shows the feature attribution value for all the input features. The model trained in FL and standalone setting has a large dependency on the sensitive attribute "Sex".
23

---

Published as a conference paper at ICLR 2023
FedAvg Most biased party
Standalone Most biased party
Centralized Most biased party
1.0
Overall
True Positive Rate
0.8
Female
Overall AUC: 0.886
Overall AUC: 0.803
Overall AUC: 0.876
0.6
Male AUC: 0.872
Male AUC: 0.796
Male AUC: 0.862
Female AUC: 0.865
Female AUC: 0.743.
Female AUC: 0.854.
0.4
6.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
False Positive Rate
False Positive Rate
False Positive Rate
FedAvg
biased party
Standalone Least blased party
Centralized Least blased party
1.0
True Positive Rate
0.8
Overall AUC: 0.844
Overall AUC: 0.732
Overall AUC: 0.826
Male AUC: 0.841
Male AUC: 0.726
Male AUC: 0.82
Female AUC: 0.843
Female AUC: 0.737
Female AUC: 0.83
Overall
Female
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
False Positive Rate
Figure 21: Receiver operating characteristic (ROC) of different models on groups (Income, Sex)
FedAvg Most biased party
Standalone Most biased party
Centralized Most biased party
1.0
0.8
Precision
0.6
0.4
Overall
Female
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
10
Recall
Recall
Recall
FedAvg Least biased party
Standalone Least biased party
Centralized Least biased party
1.0
Overall
0.8
Fernale
Precision
0.2
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
Recall
Figure Precision-Reect curve (PRC) of different models on groups (Income, Sex)
24

---

Published as a conference paper at ICLR 2023
0.8
Other Farameters
Parameters
0.750
0.6
Accuracy
0.725
0.5
0.700
0.675
0.4
02
0.625
Parameters
Related Peremneters
0.600
0.0
0.0
10
Scaling factor
Scaling factor
Scaling factor
Figure 23: Effect of scaling model parameters (Income, Sex)
Positive Predicted Value
True Positive Rate
0.50
0.9
Male
Male
0.45
Female
0.8
Female
0.7
0.35
Reweighing
0.25
0.4
0.15
0.10
0.3
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.3
0.4
0.5
0.6
0.7
0.8
0.9
FedAvg
FedAvg
Figure 24: Effect of Fair FL algorithm (Abay al.
2020 group performance (Income, Sex)
DEO
AgF
1.0
0.6
AcalRace
ApeRRace)
0.4
0.2
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.2
Aco(Sex)
Auv(Sex)
Figure 25: Fairness gap of standalone models with respect to different sensitive attributes - (Income). Each point represents the result for single party in one run. We show the fairness gap with respect to different sensitive attributes differs a lot for each party.
25

---

Published as a conference paper at ICLR 2023
1400
Party
1
1200
2
3
1000
4
Count
800
600
400
200
0
1
2
3
4
5
6
7
Class
Figure 26: Number of samples from each class for each party ISIC2019
26


Reviews:
[{"review_id": "jwoQ7IF5OYP", "reviewer": "Reviewer_whH5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the bias propagation in federated learning, which is an important issue in terms of the fairness. The authors conducted a range of experiments to understand the potential mechanism of bias propagation and present several interesting insights.", "summary_of_the_review": "It is an interesting topic and I tend to accept this work, given the sufficient experimental evidence to understand the potential mechanisms of bias propagation in federated learning.", "strength_and_weaknesses": "Strength\n\n(1) It is a first attempt to systematically analyze the bias propagation problem in federated learning, which is in particular important considering the potential adverse impact of federated collaboration. \n\n(2) A range of experiments have been conducted to understand how bias propagates among parties based on the metrics equalized odds and demographic parity. Specifically, several interesting insights have been discovered like \"biased parties negatively influence other parties via aggregation throughout the training\", and \"large fairness gap is caused by disparate treatment\".\n\nWeakness\n\n(1) Given the insights about the bias propagation, it will be better to discuss some potential solutions to avoid the bias propagation.\n\n(2) The experiments that the bias is encoded in a few parameters are somewhat controversial, since it also possibly couples the heterogeneity and stochastic factors in the optimization. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "W_DVV0QCug", "reviewer": "Reviewer_RzLQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on measuring bias in federated learning on tabular (US Census) and image  (Celeb A) datasets. They measure fairness using demographic parity and equalized odds. They find that: \n1. Some federated averaging may increase the disparity in some datasets (Table 1)\n2. FL improves fairness for stand-alone models with more disparity at the cost of worsening fairness for less biased parties (Figure 2)\n3. Aggregation and local updates produce contradicting effects in fairness gap. \n4. Less biased parties have a stronger positive influence in terms of fairness on other parties via aggregation. A similar effect follows with more biased parties and negative influence. \n5. Model bias is caused by the models treating groups differently and the FL showing a more distinct separation in terms of sex feature attrition. \n6. Biased parties use just a few parameters to encode bias through local updates and are propagated through aggregation. \n", "summary_of_the_review": "The authors present several findings to measure biases in FL and investigate where they come from. There is nothing I strongly object to in this work but also nothing especially compelling. It is unclear what the take away message or the actionable insight is. I think the two main weaknesses are that most of the experiments are based on the Income dataset (the subset of the census that shows FL biases) and that there is no discussion about the potential unequal base rates between groups in these datasets. What would imposing demographic parity even achieve and is that desirable in an income dataset? \n\n*** Updating review to accept after authors replaced attractiveness task *** \n\nFuture papers evaluating and addressing biases in FL should follow suit in avoiding attractiveness prediction as a task. ", "strength_and_weaknesses": "Strengths: \n\n1. Authors go beyond observing disparity in FL and try to understand where model bias comes from. \n1. Discovering that bias is encoded in a few parameters is very interesting. This seems to suggest a direction for mitigating bias. \n1. Authors use an updated CPS dataset rather than UCI adult which is always a plus. \n\nWeaknesses: \n1. Demonstrations of disparities are dataset dependent and thus do not tell the most compelling story. Why don’t Health and Employment datasets experience increased bias due to federated averaging? \n2. The authors use demographic parity as a fairness metric without discussing base rates. For example, It is unclear to me whether men and women have the same base rates in Celeb A. If they do not, then an increase in accuracy from a random classifier will necessarily imply a decrease in demographic parity. \n3. Authors need to report p-values with (linear?) correlation coefficients \n4. In 3.4, the authors point out that the large fairness gap is caused mainly by disparate treatment of the model on protected groups. Here we see that the sensitive attribute to be protected in this dataset is a key attribute for predicting the output both for the centralized model and for the FL model. In the real world, we can imagine using different thresholds for different groups to predict the correct outcome. In this case, overall and group-specific AUC and AUC disparity may give a clearer picture of what is going on. \n\nNit: \n1. The language of “fairness benefit” vs “fairness gap” is confusing. My understanding is that fairness benefit is the difference between fairness gaps (Figure 2)\n2. The authors mention heterogeneous distributions in the introductions but then do not discuss how heterogenous their datasets are. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IMHujKmqbd2", "reviewer": "Reviewer_LgD7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates unfairness phenomena in federated learning when parties have heterogeneous data distributions. Empirical results are gathered from the UCI Census and CelebA datasets. As a result, the paper observes several interesting patterns, including 1) the bias of each party differently affects model fairness (e.g., demographic parity and equalized odds) and 2) the bias may aggregate during the training. In addition, the paper gives some intuitions on why such a bias propagation occurs. ", "summary_of_the_review": "The empirical studies of this paper give several nice intuitions on fairness issues in federated learning on heterogenous data distributions. However, there are also some weak points that make the contribution of this paper less significant. Overall, I vote that this paper is marginally below the acceptance threshold.\n\n========== Update after the rebuttal ==========\n\nMy major concerns have been mostly addressed, so I updated the score to 6.\n", "strength_and_weaknesses": "[Strength]\n\nS1: The paper shows several good empirical observations, which further reveal the importance of fairness-aware federated learning.\n\nS2: Also, the paper tries to give possible intuitions on why such unfair phenomena happen.\n\n[Weakness]\n\nW1: Although there are some interesting empirical results, the baselines are not enough to make the observations more reliable. Currently, the paper only focuses on one federated learning algorithm called FedAvg. Even though the paper writes the investigation on other algorithms is future work, it is currently hard to simply think that their results will be consistently observed in other standard federated learning algorithms. If the paper wants to focus on FedAvg, I would recommend clarifying the contribution limits of the paper.\n\nW2: Also, the paper does not evaluate any previous fair federated learning algorithms in their setting. I understand that the assumptions on the data distribution are different from the previous works and this work, as most previous works assume a single global distribution and this work focuses on heterogeneous data distributions among parties. However, it will be still worth analyzing how other fair federated learning algorithms work in this paper’s setting. If the previous algorithms still mitigate the bias propagation in the heterogenous setting, then we can get more hints on how we can solve this issue. Such results will enhance the empirical study.\n\nW3: Some of the observations are not very surprising. For example, the phenomenon that a more biased party negatively affects the fairness of other parties can be similarly observed in centralized learning (e.g., more biased subgroups will negatively affect the fairness of the entire model). Thus, it would be much helpful if the paper clarify why each observation is especially important in a federated learning setting compared to centralized training.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster