Can Fair Federated Learning reduce the need for personalization?
Authors: Alex Iacob
Pedro Porto Buarque de Gusmao
Nicholas Donald Lane
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Fair Federated Learning
FL
Fair FL
Local Adaptation
Personalization
Machine Learning
ML
Deep Learning
DL
Distributed Machine Learning

Abstract: Federated Learning (FL) allows edge devices to collaboratively train machine learning models without sharing local data. Since the data distribution varies across client partitions, the performance of the federated model on local data also varies. To solve this, fair FL approaches attempt to reduce the accuracy disparity between local partitions by emphasizing clients with larger losses during training; while local adaptation personalizes the federated model by re-training on local data to provide a device participation incentive in cases where a federated model underperforms relative to one trained locally---their accuracy difference is less than zero. This paper evaluates Q-Fair Federated Learning (Q-FFL) in this relative domain and determines whether it provides a better starting point for personalization or supplants it. Contrary to expectation, Q-FFL does not significantly reduce the number of underperforming clients in a language task while doubling them in an image recognition task. Furthermore, fairness levels which maintain average accuracy provide no benefit to relative accuracy in federated or adapted models. We postulate that Q-FFL is unsuitable for our goal since clients with highly accurate local models require the federated model to have a disproportionate local partition accuracy to receive a benefit. Instead, we propose using knowledge distillation during FL training to create models with a higher local accuracy floor without forfeiting the ceiling. Our preliminary evaluation shows a 50% reduction in underperforming clients in the language task with no increase in underperforming clients for the image task. Thus, we argue that this simple change represents a more promising avenue for reducing the need for personalization than fairness.

Under review as a conference paper at ICLR 2023
CAN FAIR FEDERATED LEARNING REDUCE THE NEED FOR PERSONALIZATION?
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) allows edge devices to collaboratively train machine learning models without sharing data. Since the data distribution varies across clients, the performance of the federated model on local data also varies. To solve this, fair FL. approaches attempt to reduce the accuracy disparity between local partitions by focusing clients with larger losses; while local adaptation per- sonalizes the federated model by re-training it on local data-providinn a device participation incentive when federated model underperforms relatively to trained locally. This paper evaluates two Fair Federated Learning (FFL) algo- rithms in this relative domain and determines if they provide better starting point for personalization or supplant it. Contrary to expectation, FFL does not reduce the number of underperforming clients in language task while doubling them in an image recognition task. Furthermore, faimness levels which maintain per- formance provide no benefit to relative accuracy in federated adapted models. We postulate that FFL is unsuitable for our goal since clients with highly accurate local models require the federated one have disproportionate local accuracy to receive benefits. Instead, we propose Personalization-awaar Federated learning (PaFL) as a paradigm which uses personalization objectives during FL training and allows them to vary across rounds. Our preliminary results show 50% reduc- tion in underperforming clients for the language task with knowledge distillation. For the image task, PaFL with elastic weight consolidation or knowledge distilla- tion avoids doubling the number of underperformers. Thus, we argue that PaFl. represents more promising means of reducing the need for personalization.
INTRODUCTION
Edge devices represent pool of computational power and data for ML tasks. To use such de- vices while minimizing communication costs, McMahan et al.) (2017) introduced Federated Learn- ing (FL). Federated Learning trains models directly clients devices without sharing data. As the data distribution differs across clients, FL must balance average performance and performance on specific clients. In some cases, federated model may perform worse than fully local one-thus lowering the incentive for FL participation.
The existing body of work on balancing global and local performance focuses on two primary means of improving the client accuracy distribution. Li et al. (2019a) and Li et al. (2020a) propose two Fair FL techniques, q-Fair Federated Learning (Q-FFL) and Tilted Empirical Risk Minimization (TERM), which raise the accuracy of the worst-performers by focusing on clients with large losses during global FL training. Alternatively, using local adaptation (personalization) methods such as Freezebase (FB), Multi-task Learning (MTL.) with Elastic Weight Consolidation (EWC), and Knowledge Distillation (KD) has been recommended by Yu et al. (2020) and Mansour et al. (2020) in order to construct effective local models from the global one. Since personalization is local, the natural baseline comparison is a local model trained only on the client. In this work, relative accuracy refers to the accuracy difference between federated and local model client test set.
While the sets of potential use cases for fairness and personalization are not identical e.g. person- alization would be inappropriate for very low-data clients -FL could potentially construct a fairer relative accuracy distribution without hurting average performance. For FFL to reduce the need for personalization it would have lower the number of underperforming clients improve their

---

Under review as a conference paper at ICLR 2023
erage relative accuracy enough to require less adaptation. This is not what observe in practice, as our experiments show FFL to have either a negative impact on relative accuracy or none at all.
Our contribution is threefold:
1. We construct an initial empirical evaluation of the relative accuracy distribution of models trained with FFL on the Reddit, CIFAR-10, and FEMNIST datasets for next word pre- diction and image recognition tasks. We then compare the number of underperforming clients for fair models to a FedAvg baseline. During our evaluation, we show that FFL does not significantly reduce the number of underperformers or improve the relative aceu- racy distribution on Reddit and brings little benefit over combination of FedAvg and local adaptation. Concerningly, it doubled the number of underperforming clients on FEMNIST.
2. We investigate any potential synergies between FFL and personalization by adapting fair federated models. Results show that the adapted models do not significantly outperform those initially trained with FedAvg in average accuracy number of underperformers.
3. We instead propose Personalization-awee local adaptation techniques during FL training. Preliminary experimental results the language task show a significant reduction in the number of underperforming clients over FFL when applying KD after model convergence without any downsides subsequent local adaptation. PaFL can also avoid the increase in the number of underperforming clients observed for image recognition on FEMNIST when using EWC or KD. We speculate that PaFL outperforms the loss- weighted averaging mechanism of FFL because it can take advantage of data from atypical clients without greatly harming average performance.
Federated Learning as a paradigm which uses
2 BACKGROUND AND RELATED WORK
Statistical (data) heterogeneity Data generation and accrual naturally vary across devices. Fac- tors such as sensor characteristics, geographic location, time, and user behaviour may influence the precise deviations in data distribution seen by client-e.g. feature, label, or quantity skew as reported by Kairouz al. 2019 sec.3.1) which in turn prevents treating client data as Indepen- dent and Identically Distributed (IID). Non-III data has been shown to impact both global accuracy (Zhao et al. 2018 Hsich et al. 2020) and theoretical convergence bounds (Li et al. 2019b).
System (hardware) heterogeneity Devices within the federated network may differ in computa- tional ability, network speed, reliability and data- gathering hardware. System heterogeneity creates barriers to achieving a fault and straggler-toleran algorithm. However, I et al. (2019b) indicate that it behaves similarly to data heterogeneity during training and benefits from similar solutions.
2.1
FAIR FEDERATED LEARNING
The standard objective function of FL is formulated by Li et al. (020b) as seen in Eq. ①
minf(w) Σ ()Fx(w),
(I)
where f is the federated loss, m is the client count, w the model at the start of a round, and F& is the loss of client k weighted by Pk- For total number of training examples n,, Pk is defined as the proportion of examples on the client 3*k or as the inverse of the number of clients 1. The Federated Averaging (FedAvg) algorithm introduced by McMahan et al. (017) optimizes the objective by training locally on clients and then summing the parameters of each model G3 weighted by pk into an update to the previous model G2 using an aggregation learning rate 1), as seen in Eq. (2)
G++ +
Σ
P&G
(2)
Li et al. (2019a) propose Fair Federated Learning (FFL.,, which attempts to train a model with
better accuracy distribution. They define q-FFL as a version of FFL with the objective from Eq. (3)
minf(w)= = Σ
Pk
F (w).
(3)
++1
k=
2

---

Under review as a conference paper at ICLR 2023
where q controls the degree of desired fairness. A value of q 0 corresponds to FedAvg while larger values prioritize clients with higher losses. As q approaches infinity, the objective function approaches optimizing solely for the highest-loss client.
Li ct al. 2020a) develop a more general technique shown in Eq. which behaves similarly to q-FFL when applied for FL. and can be tuned using It is important note that t and g do not scale fairness at the same rate and need to be optimized independently. While the two objectives show a comparable accuracy distribution improvements in the evaluations of Li al. (2020a), it is unclear how they would affect the relative accuracy distribution.
minf(w) log( (re(Fx(w))
(4)
w
The most relevant recent FFL work is Ditto published by Li et (021) means of constructing personalized models while encouraging faimesss for the global model. Ditto functions by training local personalized model in parallel with the global one. keeps them in syne by making the local model minimize the L2 distance to the federated model similarly to the personalization techniques discussed bellow. Li et al. (202I) show to be superior to TERM, however is not fit for our purposes as it replicates the costs and limitations of local adaptation on every round.
2.2 LOCAL ADAPTATION
The analysis of Yu et al. (2020) established that not only does the federated model perform worse on heterogeneous clients, as previously noted by al. (2019a): Kairouz et al. (2019), but may offer inferior performance to local models. Thus. certain clients may receive no benefit from participating in the FL process. Yu al.] (2020) recommend addressing this accuracy gap via personalization methods using the federated model as starting point for local training.
Multi-task Learning (MTL) The task of the global model is to perform well on the distributions of all clients while a local one must perform the distribution of single client. Equation (5) frames this as a Multi-task Learning (MTL) problem using the Elastic Weight Consolidation technique (EWC) introduced by Kirkpatrick et.l. (2017) to mitigate Catastrophic forgetting (Goodfellow et al., 2013
+
Σ & Mi] -G())
(5)
where L is the client loss, À determines the weighting between the two tasks and M is the Fisher information matrix.
Fine-tuning (FT) and Freezebase (FB) When client receives global model after the FL pro- cess, it can apply Fine-tuning (see Wang et al. (2019); Paulik et al.) (2022) and Mansour et al. (2020) Section D.2)) to retrain the model on its data. To avoid potential Catastrophic forgetting. Yuu et al. (2020) also opt to apply Freezebase (FB) as a variant of FT which retrains only the top layer.
Knowledge Distillation (KD) As an alternative to EWC and FT, Knowledge Distillation (Hinton et al. 2013) uses the global model as a teacher for client model. For the pure logit outputs of the federated model G(z)) and client model C(x)), the client minimizes the loss seen Eq. (6)
I(C,z)= = TK x)) + (1 a) (o((((())) (K)G)))))), (C(x)) K)),
(6)
where is the client loss, Kr, the Kullback-Leible divergence (Kullback & Leibler 1951), a is the softmax of the logit output, a is the weighting of the client loss and K is the temperature.
METHODS
3.1 PERSONALIZZATION-AAAAR FEDERATED LEARNING
As an alternative to FFL for reducing personalization costs, we present method based on modifying the local loss function during Fl. training. Federated learning and local adaptation have historically
3

---

Under review as a conference paper at ICLR 2023
been regarded largely separated phases. One means of combining them is to allow personalization methods which operate purely by modifying the local client objective function to be used at each round prior to aggregation. This work uses the term Personalization-awwe -ware Federated Learning (PaFI.) to refer to such a paradigm. The FedProx algorithm developed by Li et al. (2018) may be considered prototypical as it injects the 1.2 norm of the model weight differences into the loss func- tion of clients. However, their motivation was to improve convergence rather than local performance and their loss did not take into account the importance of each model weight.
PaFL extends the principle behind FedProx by allowing the personalization method and its weighting to vary across rounds. Beyond improved convergence, such a process may bring benefits to the final locally-trained models by providing continuity in the local objective between FL training and the final adaptation stage if the same loss function is used. Additionally, there is reason to believe that PaFl. could be more powerful conceptually than q-FFI. (Eq. (3) and TERM (Eq. ). Loss-based weighted averaging has no means beyond averaging of reconciling differences between models re- quired by clients with equally high losses and highly dissimilar data partitions. Additionally, q-FFL and TERM do not attempt to take the client's 's relation to the global distribution- round into account. By contrast, PaFL allows clients for whom the global model performs badly
the current to diverge in a manner which maintains accuracy on the whole federated distribution.
Formally, PaFL can be defined type of Federated Learning with a potential additional training round at the end which allows clients to keep their locally trained model rrapeesnipgnngnnnng the person- alization phase in standard FL. Each client has loss function obeying the following structure:
(C, x2 = v(t) L(C, x) + (1 u(t)) ((t) G*,
where t is the current round, L(C,,, function for the current round potentatyyl dependent on the data point z. The weight of cach term is set per round through the weighting function ((t)).
x. t) is the training loss and Dtt) returns a personalization loss
3.2 FFL MODIFICATIONS
Two implementaiion details of g-FedAvg-the q-FFL implementation proposed by Li et al. (2019a) are worth discussing. First, the choice of q impacts the optimal aggregation learning rate Rather than determining 1) for each q, the authors optimize it for q = 0 and then scale it for q > Since task training parameters are already optimized for FedAvg (i.e., q-FedAvg with y = 0) in all tasks, we do not change v). Second, the original publication (Li et al., 2019a) uses weighted sampling of devices based on their share of the total training examples, followed by uniform averag- ing. This methodology is untenable in many real-world scenarios, as it would require a server being aware of the amount of data available in each client a priori. Thus, we use uniform sampling and weighted averaging. The same considerations are applied to the TERM equivalent of g-FedAvg.
aare
4 EXPERIMENTAL DESIGN
4.1 TASKS
Following the lead of Yu et al. 20200) and McMahan al. (2017). we train models using FedAvg. q-FedAvg, TERM, or PaFL for next-word prediction on version of the Reddit (Caldas et al. 2018) dataset with 80 000 participants and for image recognition. For image recognition we use CIFAR-10 partitioned into 100 participants as well as the naturally heterogeneous Federated Extended MNIST (FEMNIST) (Caldas et al. 2018) dataset. For PaFL we choose simple proof-of-coneept training sequence where we apply KD or EWC with constant weightings and parameters after the model has converged at the halfway point of training denoted HEwe and HKD- In order to allow direct comparison against previous work, the training pipelines and model architectures of Yu et al. (2020) and Caldas et al. 2018) were adapted. As such, we only note necessary changes from the initial works and provide the full details necessary for reproducibility in Appendix A.2 For both FFL methods we tune their fairness hyperparameter and report performance for values which exhibit relevant behaviour. On FEMNIST we do not tune the value of for TERM and instead reuse the t = value chosen by Li et al. (202T). In the case of PaFL, we use the same parameters for the losses after the halfway point in the local adaptation setup (Appendix(A.2 A.2
meining-ddenoted
4

---

Under review as a conference paper at ICLR 2023
Next word prediction During the FL training process z 5% of the federated test set is used to track convergence, with the full test set being used for the final evaluation after training. Federated models are trained for 1 000 rounds using 20 clients per round, rather than the 5 000 rounds of 100 clients used by Yu et al. 2 This will be shown to be sufficient for the baseline FedAvg model reach accuracy of 17.8%, which is close the original optimum of 19.29%. Only 500 clients are evaluated and ~ 18 500 clients adapted due to resource constraints (Appendix A.1).
FEMNIST image classification Rather than subsampling 5% of the data, we keep the first 350 clients with more than 10 data points of the total 397 clients in the FEMINIST dataset. Since we require both local and federated testing sets, we use 70% of a client's 's data for training, 10% for local testing, and add the remaining 20% to the federated test set. For the FL process,
aggregation learning rate of n 1.0 with 10 clients per round for 500 rounds, instead of the clients per round used by Caldas al. (2018). For each client, we use SGD with a learning rate of 0.1 and batch size of 32. During adaptation, we lower the learning rate to 0.01. The CIFAR setup remained unchanged from Yu et al. (2020).
4.2 PERFORMANCE EVALUATION
Federated and absolute local performance The first set of experiments intends to compare the accuracy on the federated test and the local accuracy distribution of models trained with FFL meth- ods or PaFI.. Too large of a drop in federated performance may cause certain fairness level or PaFI. configuration to be considered overall unusable. For given model to perform well locally, it must not bring noticeable harm to average local accuracy while reducing variance when compared to underperforming client due to lacking
FedAvg. If local training and adaptation are unfeasible data or computational power, FFL or PaFL could still allow for improvements.
Relative local performance The second set involves assessing the difference accuracy between federated or adapted models and purely local ones when evaluated client data. The most im- portant factors for the relative utility such models are the number of clients which receive an improvement and the average population improvement. For local adaptation, FFL, or PaFL con- figuration to be worthwhile it must increase the number of clients which benefit while maintaining improving the average accuracy difference. If synergistic existed between FFL PaFL and local adaptation, models trained using such techniques would either receive larger improvement in average accuracy result in lower number of underperforming clients after adaptation.
RESULTS
5.1
FFL BASELINE PERFORMANCE
ven 0-01
ACCial (l
ACCna (Liteam)
1 8. B 12
200
300
1000
160
200
400
911
Round
RRundd
(a) Language task federated performance, TERM al- ways harms performance while F-FFL only does so for q 2 1.0 and only substantially at the shown q = 5 included to demonstrate the impact of increasing fair- ness. Both HEwc and HaD perform close to the Fe- dAvg baseline with HeD exceeding
(b) FEMNIST federated performance, g-FFL causes increased instability in training process and outright divergence past a certain round. Our proposed technique also diverges for both HEwe and HXD. however it reaches similar accuracy to FedAvg (g prior doing so. TERM per- forms acceptably does not diverge.
Figure 1: Federated accuracy of models across rounds Reddit and FEMNIST.
To establish a performance baseline, Fig. la provides an overview of the convergence process for next-word prediction on Reddit while Table showcases summary data for federated performance
5

---

Under review as a conference paper at ICLR 2023
and absolute local performance. From Fig. la it be seen that the impact of q-FFL accuracy is neutral to negative while that of TERM is highly negative for all t. The fairest q-FedAvg value (q 5) shows noticeable dip in accuracy. Fairness seems reliably reduce the accuracy variance for 2 1, however the performance cost too large. We were unable to find 1-value leading to an acceptable performance for TERM, such we excluded it from future Reddit experiments.
Objective
Aegroe (eC)ea(%)
(0%
(VarAug)
(Varp)
(Varw)
q =
17.826
18.645
24.572
14.815
9.177
22.114
1.072
q = 0.1
17.789
18.66
24.843
14.728
9.81
22.914
1.036
14.056
14.819
20.208
11.66
7.983
26.769
0.69
t = 0.1
14.299
16,476
28,985
11.806
39.584
176.42
0.369
f ==
14.373
16.438
28.981
11.766
39.642
175.96
0.382
HEwc
17.322
18.255
24.653
14.226
10.277
23.059
1.185
HKD
18.177
19.179
26.406
14.887
12.438
25.85
1.039
Local
NaN
4.456
10.227
1.204
8.777
31.11
0.893
Table 1: Results showing the federated and absolute local performance on Reddit. While fairness does decrease variance q 2 1.0, the harm accuracy is too great. The proposed HKD model improves accuracy across clients but increases variance for everyone except the worst performers.
Figure lb and Table 2 show FEMNIST image recognition models be highly sensitive q when trained with g-FedAvg or either PaFL version as their performance oscillates repeatedly or diverges. For such models. we test and adapt its last version prior divergence. Unlike Reddit, nothing re- sembling a trend emerges for any metric as faireess increases. Nevertheless, good balance between performance and average variance is struck by = except for the fairly high porst-performer variance. Unlike the language task, TERM is sufficiently well-behaved on this dataset for models trained using it to be used later adaptation experiments.
Objective ACC)ea(%%
Avgioc
Bioc 10%(%))
Wioe 10)(%)
(VarAvg)
(VarB)
(Verw)
4 84.739
75.341
99.435
35.717
432.507
1.151
73.747
q 10
84.19
76.591
99.013
42.055
320.385
1.714
64.049
q 15
78.634
69.749
96.681
34.988
374.637
5.177
47.761
t=
77.706
69.134
98.628
33.478
417.448
2.771
56.543
Hawc
82.825
73.964
99.321
33.457
465.605
1.376
71.481
HKD
84.51
75.243
99,491
34.34
443.015
1.07
62.91
Local
NaN
46.322
92.848
0.0
1006.77
18.144
0.0
Table 2: Results for FEMNIST. Given the much larger variability in performance compared to Reddit, the only acceptable fair model is the one trained with 10. Note that the average accuracy of the best-performing local clients is close that of the best-performing partitions for the federated model. Unlike the language task, using KD during FL seems to primarily help the best performers.
Table indicates the CIFAR-10 image classification task to be more resilient fairness than pre- vious tasks, with only small accuracy decrease being noticeable in models trained using > 10. The lower sensitivity of this task training parameters consistent with the previous findings of Yu et al. (2020) on differentialy private FL and robust FL. Due to the similarity in results across fairness levels, the convergence graph for CIFAR- 10 was relegated the appendix (Fig. 33.. Given the lack of insight, we chose not to expand the CIFAR- 10 experiments past q-FedAvg and H Overall these findings indicate that the dataset heterogeneity must meaningful rather than artificially imposed for significant effects to emerge for either FFL PaFL.
5.2 FFL FAILS TO IMPROVE RELATIVE PERFORMANCE
Having established baselines of accuracy for fair models, we can now evaluate the local relative performance of FFL, PaFL and their interactions with local adaptation. The CIFAR-10 data (Table unsuitable for multiple reasons including its artificial partitioning and the fact that the federated model does not benefit from local adaptation as it outperforms local one for all clients.

---

Under review as a conference paper at ICLR 2023
Objective AeC)ee(%%
Augioe (%6)
Bioc 10%)))
Wioe 10%((%)
(VarAve)
(VarB)
(Varw)
81.28
81.37
82.255
79.864
0.568
0.022
1.067
q
81.86
81.221
82.011
79.794
0.446
0.004
0.643
q
78.16
79.935
81.178
77.885
0.945
0.02
1.267
Local
NaN
31.718
38.297
24.649
16.3
1.543
0.906
Table Results for CIFAR-10. Unlike the language task []. = represents a clear optimum in terms of variance while maintaining performance. However, differences between all models are very small and cannot be guaranteed to be significant (see Appendix A.2
For q-FFL, the results for the language task showcased Table are less than satisfactory as fair models fail to provide benefits in terms of the number of underperforming clients, relative accuracy, or variance on average. Furthermore, fair models do not offer improved accuracy once adapted this
is directly visible in the Fig. 2a scatter plot of relative accuracy against fully local accuracy.
Objective
Adapt
Avg(%)
% <0
(10%(%)
W10%(%%)
(Var (VarAey) Aey)
(VarB)
(Varw)
q
q =
14.185
53
20.715
9.392
13.323
34.379
20,201
A_FB
15.87
25,849
11.311
29.216
149.736
3,246
A_MTL
16.046
27.558
11.304
36.067
178.387
3.337
A_KD
15.538
24.376
11.209
23.112
115.016
3.183
4 0.1
4 0.1
14.208
50
20.907
9.359
13.742
35,005
20.733
15.827
25.964
11.261
29,505
149.011
3.212
A.MTL
15.839
27.692
11.024
37.108
179.066
3,336
A.KD
15.546
0
24.614
11.19
23,95
118.471
3.166
HgwC
Hewe
13.807
108
20.723
8.681
14.994
38.119
24.938
A.FB
15.423
0
25.709
10.88
29.795
149.308
3.02
A.M.L
15.561
27.482
10.762
37.251
179.528
3.266
A KD
15.157
24.336
10.823
23.996
117.427
3.041
HKD
HKD
14.729
27
22.038
9.971
14.969
41.225
18.409
FB
15.772
26.533
11.154
31.068
150.552
3.156
A MTL 15.824
28.217
10.966
38.439
178.543
3.469
A KD
15.698
25.358
11.214
25.367
119.418
3.241
Table 4: Relative performance on the Reddit dataset of the acceptable fair models and our PaFL variants. The best value in column is in bold while the best in group is underlined. The chosen optimal fair model does not significantly reduce the number of underperformers. Alternatively, HKD lowers it to half. Local adaptation provides simillar results regardless of starting point.
The results for image recognition on FEMNIST are more unusual yet similarly discouraging for both q-FFL and TERM. Table it clear that the fair model actually achieves higher relative accuracy on average and amongst the top 10% of performers at the cost of obtaining a negative average accuracy the relative worst performers. Additionally, has more than twice as many clients with negative relative accuracies. We speculate that since the federated model has lower absolute accuracy variance, it cannot obtain good enough level of performance on clients which are able to train high-quality local models. This is corroborated by the final distribution shown in Fig. 2b) as nearly all the underperforming clients have high local model accuracy. Another factor to consider is the atypical behaviour of personalization
FEMNIST and CIFAR-10. 10. Models trained with FedAvg and then adapted tend to converge to nearly the same relative accuracy regardless of the specific adaptation technique. Thus, FedAvg is potentially near-optimal for the dataset model.
5.3 PAFL AS AN ALTERNATIVE
Having shown the inability of FFL to replace enhance local adaptation, we argue that it is not the right approach for this domain. In principle, for an FL algorithm to provide benefits in terms of relative accuracy it must achieve two goals. First, it must make that the worst worst- persstperforming clients receive a sufficient level accuracy to match or exceed local models. Second, for the clients with the best local models, it must provide disproportionately high accuracy. While FFL may help

---

Under review as a conference paper at ICLR 2023
Objective
Adapt
(%)(%)
<0
(10%(%)
W10%(%)
(Var Ang)
(Varn)
(Varw)
q
q 0 A_FB
29.02
16
65.768
2.729
338.387
137.366
12.33
28.954
17
65.463
2.72
334.754
134.824
11.853
A MTL
28.994
16
65,672
2.802
336,25
137.507
11.325
A_KD
28.986
16
65.684
2.788
337.14
137.796
11.594
q 10
10
30.269
39
79.687
-14.844
673.351
111.144
206.995
A FB
28.613
12
64.818
2.699
320.729
123.679
15.552
A.MTL
28.612
14
64,818
2.516
321.3
123.679
15.869
A.KD
28.563
14
64.645
2.52
320.957
127.618
15.934
t=1
t=1 22.812
56
73.593
-17.069
627.167
215.261
91.294
A FB
21.261
15
50.057
0.806
201.35
127.802
8.417
A.MTL
21.362
15
50.218
0.765
201.703
123.192
8.102
A KD
21.202
15
50.1
0.706
202.488
125.153
7.71
HEWC
Hgwc 27.642
13
62.157
1.522
315.388
123,438
19.225
A.FB
27.558
14
62.431
1.404
316.622
124.092
18.888
A.MTL
27.603
13
62.588
1.349
315.619
122.717
20.502
A.KD
27.611
13
62.588
1.542
314.112
122.717
19.176
HKD
HKD 28.921
16
66.178
1,65
353.698
154,217
16.231
A.FB
28.916
17
66.605
1.644
350.631
152.111
16.922
A ATT
28.871
16
66.605
1.719
350.742
152.111
16.865
A.KD 28.967
15
66.329
1.869
351.991
150.583
17.124
Table FEMNIST performance of the best fair model and our proposed alternative FFI.. Despite providing the highest average accuracy and accuracy amongst the best 10%. the model trained using q = has more than double the number of underperformers FedAvg (q = 0)-as as does TERM with 1. For PaFL., HKD is close FedAvg while HEwc brings trivial improvement.
(a) Language task results, fairness shows no benefit while HKD reduces the number of underperformers.
q- 0.01
Hi
PH
A rd A.TTL Kn
ALFR
A.MIL
A MT..
20-
A,KD
ADKD
ACC
ACCus (Lincar)
ACCuc (Hi(iiner))
ACCus (lincar)
(b) FEMNIST results, clients with highly accurate local models are underserved by federated models trained
using 4 = 10. Alternatiely, those trained using FedAvg HEwe achieve nearly identical performance.
q-10
A.KD
A_ED
60
20
Figure 2: Federated model relative accuracy on a client plotted against local client model accuracy,
fulfil the first requirement, its inability to raise the floor of the worst performers without hurting the ceiling of those that might have good local model makes it incapable of fulfilling the second.
8

---

Under review as a conference paper at ICLR 2023
PaFL in the most general case offers an alternative where models can be kept closer to another during training and only allowed to diverge in ways which hurt federated performance the least. Unlike blind regularization based on the norm of the distance between model parameters (e.g. Fed- Prox), EWC and KD offer the distinct advantage of determining how a parameter may diverge based on its importance to federated performance. Thus, the model can learn from highly heterogeneous data and raise its accuracy floor for the worst-performers without hurting the accuracy ceiling of the best or even improving The models trained with either EWC or KD past the halfway point, H Hgwe and HKD, have already been included in previous tables and graphs to allow for comparison.
Preliminary results for the language task are promising in the case of HKD, Fig. Ia and Table indicate that it performs better than FedAvg and FFL models in every metric except average and best-performer variance. Importantly, variance not increased for the worst performers. While H HEwC is not far below the FedAvg baseline, it fails to provide any obvious improvements. In terms of relative accuracy, Table|4]shows that HKD halves the number of underperformers and provides the best average relative accuracy. However, this higher bascline does not translate to improved relative accuracy for adapted models. Overall, by lowering the number of clients which require adaptation in order receive an incentive to participate H D successfully reduces the need for personalization on Reddit. On the other hand. HEwe seems to double the number of underperforming clients for the fixed chosen À although different value or scheduling across rounds may change results.
For image recognition on FEMNIST, HKD D and HEWC are satisfactory in terms of federated and average accuracy according to Fig. ab] Table On the other hand, the results related to relative accuracy shown in Table55jrr Jre mixed. While they both avoid the doubling in underperforming clients that fair models suffer, locally adapted models starting from HDD as a baseline do not seem to out- perform those adapted from FedAvg. Perhaps surprisingly given its failure on the language task, HEwe brings a very small reduction to the number of underperformers for baseline and adapted models. While this is not sufficient to draw strong conclusions, it does indicate that PaFL configu- rations behave differently across domains.
Overall, for both tasks PaFL variants have brought small to medium improvements to the number of underperforming clients, average relative accuracy and associated metrics without clear downsides. Nonetheless, more experimentation is required on the specific parameters and other techniques originating from Multi-task Learning and its associated fields.
CONCLUSION
This paper set out to find a means of reducing the amount of personalization needed to incentivize FL participation for clients whose local model outperforms federated one. Such a reduction would be relevant for federated networks containing devices with limited capabilities for retraining or little data. Our investigation began with Fair Federated Learning due to the possibility that reducing dis- parity in the local accuracy distribution would translate to reducing disparity in the relative accuracy distribution. Our experimental results indicate that FFL is unlikely to provide the desired properties as it has been shown to maintain the number of underperformers on our language task while increas- ing it by more than 100% for image recognition on FEMNIST. We speculate that the reason for this is that although it can help clients for which the global model is highly inaccurate, it cannot help those for whom relative underperformance caused by an extremely accurate local model. As an al- ternative, Personalization-awar Federated Learning allows loss functions historically used for local adaptation to applied during FL. and vary across rounds. We tested applying EWC or KD after the model had partially converged in the hopes that it would allow it to learn from worst-performet data without sacrificing performance on the federated distribution or best performers. While our chosen EWC configuration did not bring a meaningful improvement over FedAvg the language task, KD showed promising results by reducing the number of underperformers by up to 50%. Both of them avoided increasing the number of underpeformers the FEMNIST image recognition task. Unlike more complex systems which simultaneously train local and federated models during FL, this approach has little computational overhead. Consequently, we recommend using PaFl. to incentivize FL participation without explicit local adaptation and advise further research adapting research directions from the field of Multi-task learning to FL.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: benchmark for federated settings. CoRR, abs/1812.01097,, 2018.
URL WtRLhttp://axxiii org/ abs 1 1818811111 01097
lan J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2013. URL https:
//axxiv.orgaab//11...
6211
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL
02531
Kevin Hsich, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-lID data quagmire of decentralized machine learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4387-4398, PMLR, 13-18 Jul 2020. URL http::ttttocceeeeeeeeeee
mlr. www.sresssss...ee wt..ppres/vil. https//v19/hssiiieee
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects nomidenticaal data dis- tribution for federated visual classification, 2019. URL htt.tts://axxx...rrr//b/////////
06335
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv:191..0497, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521335) -32223266..11 2017.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 7:7 86, 1951. doi: 10. 10.1214aamm/117779996 10.1214/aoms 11777296941
URL httthts:://dor /rtp::////oooorg/
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. arXiv preprint arXiv:1812.0612 2018
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint /90. 2019a.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization.
CoRR, abs/2007.01 162, 2020a. URL
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6357-6368. PMLR, 18-24 Jul 2021. URL httt::::/proceedime ngs. mlr. www.press/ae v 1.3 /9/112//1hhh... html
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg non-iid data. arXiv preprint arXiv:1907.021190 2019b.
Y. Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personal- ization with applications to federated learning. ArXiv, abs/2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, Sudeep Agarwal, Julien Freudi- ger, Andrew Byde, Abhishek Bhowmick, Gaurav Kapoor. Si Beaumont, Áine Cahill, Do- minic Hughes, Omid Javidbakht, Fei Dong, Rehan Rishi, and Stanley Hung. Federated eval- uation and tuning for on-device personalization: System design & applications, 2022. URL pdf
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage, Federated evaluation of on-device personalization. CoRR, abs/1910.10252, 2019. URL
http://axxxvv.rrr//////// http://bs/111100 10252
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adapta- tion. arXiv preprint arXiv:2002.04758, 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, 2018.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 HARDWARE
Each node of the cluster that the experiments were run holds four Nvidia A100 GPUs with 80GiB VRAM and 2 x AMD EPYC 7763 64-Core processors alongside a 1000GiB shared memory. Given the quotas and service levels of the cluster, the number of clients on which the federated model could be tested locally for the language task was limited to 65 500 while the number that could be adapted was limited to (~ 185 entire client set was available during FL training. To compensate for this fact, all charts and tables comparing local model performance adaptation performance only use data from the client-set common to all results. This limitation did not impact any other sets of experiments.
A.2 FULL EXPERIMENTAL DETAILS
During local adaptation we reuse the parameters recommended by Yu et al. (2020). MTL uses a weighting of 5000 while KD uses temperature l = 6 and weighting À 0.95,
Next-word prediction A standard LSTM with 2 layers, 200 hidden units and 10 million param- eters used to predict the next word in a sentence for each client used during training or testing. We reuse the dictionary of the 50 000 most frequent words compilled by Yu et al. (2020), all other words are replaced with a placeholder. The first 90% of a users posts, chronologically. are used as training with the final 10% being reserved for local testing. A separate federated testing set is maintained for evaluating global task performance, during the Fl. training process s 5% of it is used track convergence with the full test-set being used for the final evaluation after training. Federated models are trained for 000 rounds using 20 clients per round rather than the 5 000 rounds of 1 000 clients used by Yu et (2020). On the client side, each model trained for internal epochs with a batch size of 20 using Stochastic Gradient Descent with the learning rate set to 40. For adaptation,
we use a learning rate of and batch size of 20 for 100 epochs of retraining. Only 18 500 clients are adapted due to resource constraints (Appendix A.1)
CIFAR-10 image classification Since CIFAR-10 is not a naturally federated dataset, a Dirichlet distribution (a 0.9) is used to simulate non-iid partitioning (Hsu et al., (2019] Yu et al.) 2020). A ResNet-18 18 model is trained over 1, 000 rounds with 10 clients per round. Clients are trained using batch size of 32 with 2 internal epochs and learning rate of 0.1. Test accuracy computed by multiplying a client's per-class accuracy on the CIFAR-10 test set with its proportion of the local device data. For adaptation, we use learning rate of 10-3 and batch size of 32 for 200 epochs of retraining. Training uses SGS with momentum 0.9 and weight decay 0.0005.
FEMNIST image classification We use a similar experimental setup to Caldas et al. (2018) with a simple two-layer CNN while changing the way the dataset is divided and the FL training parameters. Rather than subsampling 5% of the data, we keep the first 350 with more than 10 data points out of the total 3 597 clients in the FEMINIST dataset. Since require both local and federated testing sets we keep 70% of a clients' data for training, 10% for local testing and add the remaining 20% to the federated test set. For the FL process, 10 clients per round for 1 000 rounds instead of the 3 clients per round used by Caldas et al. (2018) For each client, we use SGD with learning rate of 0.1 and batch size of 32.
aggregation learning rate of 1) 1.0 with
A.3 FULL Q-FEDAVG RESULTS
12

---

Under review as a conference paper at ICLR 2023
Objective
Adapt
A%)(%)
Acc 0
(1)%
W (10%(%%)
AVarAva)
(Varn)
(Varw)
q FB
14.185
53
20.715
9.392
13.323
34.379
20.201
15.87
25.849
11.311
29.216
149.736
3.246
A MTL
16.046
27.558
11.304
36,067
178.387
3.337
A.KD
15,538
24.376
11.209
23.112
115.016
3.183
q = 0.01
q = 0.01
13.989
106
20.458
8,793
14.421
33.874
28.638
A.FB
15.799
25.7
11.238
29.093
149.929
3,228
A.MTL
15,833
27.419
11.057
36.547
180.574
3.326
A_KD
15.495
24.308
11.175
23.343
117.768
3.173
q 0.1
g == 0.1
14.208
50
20,907
9.359
13.742
35.005
20.733
A FB
15,827
25,964
11.261
29.505
149.011
3,212
15,839
27.692
11.024
37.108
179.066
3.336
A.KD
15.546
24.614
11.19
23,95
118.471
3.166
9 0.5
q = 0.5
14.097
53
20.705
9,287
13.514
34.546
20.655
A FB
15.837
25.838
11.285
29.2
148.925
3.232
A.MTL
15,892
27.555
11.133
36,531
179.11
3.26
A.KD
15.52
24.42
11.186
23.402
116.556
3.199
==1
==1
11.397
70
17.618
7.047
12.546
37.951
19.031
A FB
13,463
22,632
9.256
25.775
136.853
1.987
A.MTL
13,364
24.519
8,993
35,506
187.701
2.376
A.KD
13.238
21.567
9.245
20.821
105.756
2.016
==5
1 1=
10.384
121
16,583
5,904
12.952
39.358
21.502
FB 12.771
21.703
8.678
24.874
133.58
1.694
MTL
12,838
23.37
8,548
33.262
181.204
2.146
A.
12.492
20.608
8.614
19.704
99.431
1.695
Table 6: Reddit full q-FedAvg results.
Objective
Adapt
Aug(%)
< (10%(%)
W10%(%%)
(VarAvg)
(Varp)
(Varw)
q
9 ==
49.652
56.54
43,045
15.557
0.79
2.563
A_FB
49.669
56.522
43.145
15.364
0.839
2.165
A.MTL
49.647
56.489
43.064
15.468
0.829
2.267
A KD
49.647
0
56.527
43.066
15.545
0.789
2.076
= 0.1
== 0.1
49.254
0
56.078
42,839
15.408
0.858
1.94
A FB
49.23
0
55.997
42.758
15.492
0.987
1.923
MTL.
49.257
55.925
42.933
15.569
1.129
2.077
KD
49.248
56.047
42.82
15.486
0.996
1.973
@=1
49.442
0
56.297
42,865
15.614
1.053
1,991
FB
49.448
56.308
42.812
15.729
0.98
2.423
A M.T
49.425
56.352
42.786
15.728
0.966
2.287
A.KD
49.439
56.314
42.78
15.742
0.981
2.307
== 5
g /=
49.503
0
56.25
42,864
15.583
0.755
2.07
Á FB
49.517
0
56.315
42.974
15.617
0.77
2.091
MTL.
49.522
56.404
42.909
15.651
0.603
1.874
49.548
56.416
42.991
15.531
0.678
1.902
q 10
g= 10 FB
48.286
54.949
42,033
14.417
0,608
1.956
48.3
54.948
42.096
14.275
0,424
1.847
MTL
48,306
55.025
42,087
14.468
0,565
1.71
A KD
48.297
54.987
42,021
14.504
0.507
1.873
q 15
== 15
48,217
54.745
41.842
14.449
0.585
2.434
A.FB 48.22
54.718
41.883
14.482
0.619
1.859
A.MTL 48.202
54.761
41.824
14.444
0,614
2.191
K.
48.193
54.779
41.713
14.763
0.564
2.473
Table 7: CIFAR-10 full q-FedAvg results.
13

---

Under review as a conference paper at ICLR 2023
q
9-0.1
q=1
3.0
ACCred (%) (Linear)
90
Lie (Linear)
2.5
75
2.0
60
45
30
0.5
15
0.0
200
400
600
800
1000
200
600
800
1000
# Round
Round
Figure 3: Federated performance fair models on CIFAR-10. Avg performs marginally worse for q 10.0, however, it must be concluded that the accuracy of the given task is not sensitive enough to fairness to draw strong conclusions.
Objective
Adapt
Avg(%)
Acc
B10%(%)
W10%(%)
(Var
(Vara)
(Varw)
q
g A_FB
29.02
16
65.768
2.729
338.387
137.366
12.33
28.954
65.463
2.72
334.754
134.824
11.853
A.MTL
28.994
65.672
2,802
336.25
137.507
11.325
A_KD
28.986
16
65.684
2.788
337.14
137.796
11,594
q = 0.1
g == 0.1
27.427
45
80.262
-23.681
761.163
117.165
165.003
A.FB
26.174
14
59.483
2,09
282.14
128.526
14,212
A.MTL
26.205
14
59.567
2.345
281,689
128,659
15.377
ALKD
26.18
59.613
1,997
282.946
127.796
14.284
1=1
27.011
48
78.722
-18.92
699.58
131.927
150.33
A_FB
24.999
57.014
0.513
270.651
140.273
24.641
A.MTL
25.022
57.179
0.76
271.681
136.18
23.188
ALKD
25.053
57.165
0.859
270.926
135.297
22.658
0= 5
25.048
73.939
-23.702
683.322
129.091
177.272
A.FB
25.385
18
56.625
1.758
264.374
97.084
8.057
A.MTL
25,468
16
56.64
2.031
261.138
96.776
8,409
ALKD
25.437
57.116
1.695
267.606
91.34
10.717
9 10
== 10
30.269
39
79.687
-14.844
673.351
111.144
206.995
A.FB
28.613
64,818
2.699
320.729
123.679
15.552
A.MTL
28.612
64.818
2,516
321.3
123.679
15.869
A_KD
28.563
64.645
2.52
320.957
127.618
15.934
==
g = 15
23.427
74.969
-23.34
699.404
145.773
170.676
A FB
22.008
51.138
1.039
214.214
115.322
7.717
A MTL. 21.848
21
51.167
0.149
222.45
115.582
9.945
KD
21.935
51.467
0.84
216.567
112.101
8.538
Table FEMNIST full FedeAA results.
14


Reviews:
[{"review_id": "x0K_6dMyyF", "reviewer": "Reviewer_j5Ef", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors talk about the idea of utilizing fairness in reducing the need for personalization. The authors have shown experimentally that the fairness doesn't always help in providing a better starting point for personalization. The authors have proposed a knowledge distillation during FL training to improve the local accuracy. Experimental evidences are provided. ", "summary_of_the_review": "Overall, the idea is interesting but it is not executed well. The paper requires major revision before publications. The main idea is to show that fairness it not suitable for personalization and authors have proposed to change the loss function with knowledge distillation. ", "strength_and_weaknesses": "*Strength*\n- The idea is interesting and empirical resutls are provided. \n\n*Weakness*\n\n- The paper is not well organized and difficult to follow. \n\n- The contributions are not clear and properly placed in comparison to related literature. \n\n- What is G in equation (2) ? \n\n- What is he proposed/new part in the paper is not clearly mentioned. \n\n- There is no theoretical justification. \n\n- It looks like the main contribution is in Sec 4.3 but that is also not clearly discussed. The paper is not publishable in its current form. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jU4jpc58v4", "reviewer": "Reviewer_AD53", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluates Q-Fair Federated Learning (Q-FFL) in personalized federated learning with its underperforming clients and proposes using knowledge distillation during FL training. And experiments with different datasets show a 50% reduction in underperforming clients in the language task with no increase for the image task. ", "summary_of_the_review": "Although it is a good effort to evaluate Q-Fair Federated Learning (Q-FFL) with comprehensive and sufficient analysis. But this paper needs to use more words to describe the author’s ideas, not the other’s.", "strength_and_weaknesses": "Strength:\n1. There is a good effort to evaluate Q-Fair Federated Learning (Q-FFL) with comprehensive and sufficient analysis. And all of the formulas in the paper are derived from others' papers, rather than serving the methods proposed in the paper.\n\nWeakness:\n1. Use too many words to describe relevant knowledge, but not enough to the author's ideas.\n2. The whole ‘Method’ part seems to be the details of the experiment and the analysis of the results. I think they should be put in the experimental position rather than the method.\n3. The baseline performance occupies too much space and compresses the space for other experiments.\n4. Suggest the author apply the knowledge distillation to other federated learning methods to illustrate the universality.\n5. Inadequate citation of papers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h_Zz3VUWLP", "reviewer": "Reviewer_Wm3o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the performance fairness problem of federated learning. The authors empirically show that Q-Fair Federated Learning fails to improve performance fairness. They also show that knowledge distillation is a better way for fair federated learning without additional personalization mechanisms.", "summary_of_the_review": "This paper has limited contribution, flawed evaluation, and insufficient literature review. Thus, my recommendation is rejection.", "strength_and_weaknesses": "Strengths:\n- The problem studied in this paper is interesting.\n- The analysis in this paper can provide some reference for future fair federated learning studies.\n\nWeaknesses:\n- The scope of this paper is quite narrow and the findings are not significant. The study is limited to Q-FFL and only one baseline method is used. Thus, the conclusion drawn from the results may not be universally valid. In addition, the advantage of knowledge distillation over other types of techniques is not verified.\n- Using knowledge distillation to teach local models is a quite common strategy (e.g., [1]), thereby the technical contribution of this paper is very limited. \n- The coverage of related work is quite limited. The authors should present a more comprehensive review of fair federated learning and personalized federated learning.\n- The experimental setting is unclear. It would be a hard task to reproduce the results without sufficient details.\n\n[1] Yu, T., Bagdasaryan, E., & Shmatikov, V. (2020). Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "besexQwLggR", "reviewer": "Reviewer_C1c4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper asks the question of whether the Q-Fair Federated Learning (Q-FFL) algorithm can help or obviate personalization. This federated learning algorithm attempts to instill fairness by weighting losses more heavily for clients with large losses, controlled via an exponential weighting parameter q. By equalizing the loss for under-performing clients, the authors hypothesize that Q-FFL may reduce the need to use personalization to boost individual client performance. However, they find through an offline study that Q-FFL does not reduce and can even increase the number of underperforming clients. They instead propose to use distillation to train local models that still benefit from global learnings. Preliminary results show this helps reduce the number of underperforming clients, indicating this as a promising new direction.", "summary_of_the_review": "I am marginally uninclined to accept this paper. I believe there is a very promising piece of work here. The authors present a strong and clear background for the problem of fairness in federate search and explore an interesting question about whether fairness can reduce the need for personalization. This is an important topic, and the authors have made progress by showing one existing algorithm q-FFL does not meet this criteria, and they propose an alternative approach that performs better. However, the work is quite preliminary, meaning the authors focus most of the paper on the under-performing baseline. This paper would benefit by giving the authors more time to flesh out their new approach.", "strength_and_weaknesses": "Strengths:\n1. Presents detailed investigation of whether a fairness algorithm can also reduce the need for personalization, yielding the surprising finding that it does not reduce disparities between clients overall.\n2. The background information is well written and provides a clear overview of fairness and federate learning.\n3. Authors present results from a new algorithm based on distillation that can reduce the number of under-performing clients.\n\nWeaknesses:\n1. The main result is a negative result -- that Q-FFL did not reduce the need for personalization. This is unsurprising since Q-FFL is not a personalization algorithm. Since the authors present an alternative algorithm that does use fairness to reduce the need for personalization, it would be better if they led with this and used Q-FFL as a baseline.\n2. Along the same lines, the results with their new approach seem too preliminary and under-developed. It is likely the authors will need to follow-up with another publication to present this approach in more detail. I strongly suggest to the authors to develop this work further and make it the main subject of their publication.\n3. Color scheme in figure 3 makes it hard to distinguish red and orange dots, especially on such a small figure. Maybe use black for one of these symbols.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject