When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning
Authors: Wenkai Yang
Yankai Lin
Guangxiang Zhao
Peng Li
Jie Zhou
Xu Sun
Venue: Accepted by TMLR ()

Abstract: Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy. However, federated learning faces severe optimization difficulty when training samples are not independently and identically distributed (non-i.i.d.). In this paper, we point out that the client sampling practice plays a decisive role in the aforementioned optimization difficulty. We find that the negative client sampling will cause the merged data distribution of currently sampled clients heavily inconsistent with that of all available clients, and further make the aggregated gradient unreliable. To address this issue, we propose a novel learning rate adaptation mechanism to adaptively adjust the server learning rate for the aggregated gradient in each round, according to the consistency between the merged data distribution of currently sampled clients and that of all available clients. Specifically, we make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate, which is supposed to minimize the Euclidean distance between the aggregated gradient given currently sampled clients and that if all clients could participate in the current round. We show that our proposed indicator can effectively reflect the merged data distribution of sampled clients, thus we utilize it for the server learning rate adaptation. Extensive experiments on multiple image and text classification tasks validate the great effectiveness of our method in various settings. Our code is available at https://github.com/lancopku/FedGLAD.

Published in Transactions on Machine Learning Research (05/2023)
When to Trust Aggregated Gradients: Addressing Nega- tive Client Sampling in Federated Learning
Wenkai Yang
wkyang @ plu. pku.uuu.uu.c edu.cn
Center for Data Science Peking University
Yankai Lin'
yankailin @ ruc. mcc.ed..
Gaoling School of Artificial Intelligence Renmin University of China
Guangxiang Zhao
guangxiangzhuo gmail.com
Shanghai Al Lab
Peng Li
lipeng aigeng@@iiittiinguuu.eeee
Institute for Al Industry Research (AIR) Tsinghua University
Jie Zhou
withtomzhoo tencent.com
Pattern Recognition Center, WeChat Al Tencent Inc.
Xu Sun
xusun 2 wwt.pppkuuucccoo
National Key Laboratory for Multimedia Information Procexsing, School of Computer Science Peking University
Reviewed OpenReview: http://///eeeeeeeeeeaa///////aaa
Abstract
Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy. However, federated learning faces severe optimization difficulty when training samples are not independently and iden- tically distributed (non-i.i.d.). In this paper, we point out that the client sampling practice plays decisive role the aforementioned optimization difficulty. We find that the negative client sampling will cause the merged data distribution of currently sampled clients heavily inconsistent with that of all available clients, and further make the aggregated gradient unreliable. To address this issue, we propose novel learning rate adaptation mechanism to adaptively adjust the server learning rate for the aggregated gradient in each round, according to the consistency between the merged data distribution of currently sampled clients and that all available clients. Specifically, we make the- oretical deductions to find meaningful and robust indicator that positively related to the optimal learning rate, which is supposed to minimize the Euclidean distance between the aggregated gradient given currently sampled clients and that if all clients could participate in the current round. We show that our proposed indicator can effectively reflect the merged data distribution of sampled clients, thus we utilize it for the server learning rate adaptation. Extensive experiments on multiple image and text classification tasks validate the great effectiveness of our method in various settings. Our code is available pletps://gitaue htt.aaa...c.c.aaooooo//aaaaaaaaa
Part of the work done while Yankai Lin and Peng Tencent. Part of the work while Guangxiang Zhao Peking University.

---

Published in Transactions on Machine Learning Research (05/2023)
Glient
Sample Client Client
Client Client
0
Sample Client and Clicat N
Client + Clien N
Client N
o
uploaded gradient
averaged server gradient
ideal server gradient
Figure I: An illustration of the impact of client sampling through a binary (cats v.s. dogs) image classification task. (Right Top): If Client and Client are selected, their merged local samples are almost all cat images. Then the averaged gradient deviates far away from the ideal gradient that would be averaged by all clients' local gradients. (Right Bottom): If Client and Client N are selected, their merged data distribution matches well with the global data distribution merged by all clients' data, and the averaged gradient has more reliable direction.
1
Introduction
As tremendous data is produced in various edge devices (e.g., mobile phones) every day, it becomes important to study how to effectively utilize the data without revealing personal information and privacy. Federated Learning (Konečný) et al. 2016 McMahan et al. 2017) is then proposed to allow many clients to jointly train a well-behaved global model without exposing their private data. In each communication round, clients get the global model from a server and train the model locally on their own data for multiple steps. Then they upload the accumulated gradients only to the server, which is responsible to aggregate (average) the collected gradients and update the global model. By doing so, the training data never leaves the local devices.
It has been shown that the federated learning algorithms perform poorly when training samples are not independently and identically distributed (non-i.i.d.) across clients (McMahan et al. 2017 Li et al. [002]. which is the common
case in reality. Previous studies (Zhao et al. 2018 Karimireddy et al. 2020) mainly attribute this problem to the fact that the non-i.i.d. data distribution leads to the divergence of the directions of the local gradients. Thus, they aim to solve this issue by making the local gradients have more consistent directions eLi al.] 2018 Sattler et al. 202T: Acar et al. 2020). However, we point out that the above studies overlook the negative impact brought by the client sampling procedure (McMahan et al.) 2017 Fraboni et al., 2021b), whose existence we think should be the main cause of the optimization difficulty of the federated learning non-i.idd data.
Client sampling is widely applied in the server to solve the communication difficulty between the great number of total clients and the server with limited communication capability, by only sampling a small part of clients to participate in each round. We find that the client sampling induces the negative effect of the non-i.i.d. data distribution on federated learning. For example, assume each client performs one full-batch gradient descent and uploads the local full-batch gradient immediately (i.e., FedSGD in McMahan et al. (201)). ), (1) if all clients participate in the current round, it is equivalent to performing global full-batch gradient descent on all training samples, and the aggregated server gradient is accurate regardless of whether the data is i..dd or not; (2) if only a part of clients is sampled for participation, the aggregated server gradient will deviate from the above global full-batch gradient and its direction depends on the merged data distribution of the currently sampled clients (i.e., the label distribution of the dataset constructed by data points from all sampled clients). The analysis is similar in the scenario where clients perform multiple local updates before uploading the gradients, and we have a detailed discussion in Appendix
The above analysis indicates that the reliability of the averaged gradients depends on the consistency between the merged data distribution of currently sampled clients and that of all available clients. Specifically, take the image
2

---

Published in Transactions on Machine Learning Research (05/2023)
classification task as an example (refer to Figure I (1) if local samples from all selected clients are almost all cat images, the averaged gradient's direction deviates far away from the ideal server gradient averaged by all clients' local gradients. In this case, we may not trust the aggregated gradient, and decrease the server learning rate. (2) However, if the merged data distribution of selected clients matches well with the merged data distribution of all clients, the averaged gradient's direction more reliable. Thus, we should relatively enlarge the server learning rate. This example motivates us to set dynamic server learning rates across rounds instead of fixed one as previous methods do.
In this paper, we first analyze the impact of client sampling and are motivated to mitigate its negative impact by dynam- ically adjusting the server learning rates based on the reliability of the aggregated server gradients. We theoretically show that the optimal server learning rate in each round, which is defined to minimize the minimize the Euclidean distance between the aggregated gradient given currently sampled clients and that under the situation if all clients are selected, is positively related to an indicator called the Gradient Simae Indicator (GSD) that can reflect the merged data distribution of sampled clients by measuring the dissimilarity between uploaded gradients. Based on this indicator, we propose a gradient limilarity-awar learning rate adaptation mechanism to adaptively adjust the server learning rates. Furthermore, our method will adjust the learning rate for each parameter group (i.e., weight matrix) individually based on its own GSI, in order to solve the issue of the inconsistent fluctuation patterns of GSL across parameter groups and achieve more precise adjustments.
In summary, our contributions can be summarized as:
We systematically analyze the negative impact of the client sampling practice federated learning on non-i.i.d. data, and are motivated to dynamically adjust the server learning rates based on the client sampling results.
We make theoretical derivations to find a meaningful indicator that can reflect the reliability of current sam- pling results, and utilize it to adjust the server learning rates.
Furthermore, we make preliminary explorations about our proposed indicator, and propose a parameter group-wise adaptation with dynamic bounds mechanism to boost the performance of our method.
Extensive experiments on multiple benchmarks show that our method consistently brings improvement to various state-of-the-art federated optimization methods in various settings.
2
Related work
Federated Averaging (FedArg) is first proposed by McMahan et al. (2017) for the federated learning, and its conver- gence property is then widely studied (Li et al. 2019 Karimireddy et al. 2020). While FedAvg behaves well the i..dd data with client sampling, further studies (Karimireddy et al. 2020) reveal that its performance on the non-i.i.d. data degrades greatly. Therefore. existing studies focus on improving the model's performance when the training samples are non-i.i.d. under the partial client participation setting. They can be divided into the following categories:
Advancing the optimizer used in the server's updating: Compared with FedAvg that is equivalent to applying SGD during the server's updating, other studies take steps further to use advanced server optimizers, such as using SGDM (Hsu et al., 2019 Wang al. 2019b) and adaptive optimizers (Reddi al. (020)) like Adam (Kingma & Ba 2014). However, their improvement comes from the effects of using advanced server optimizers, and they do not truly target on addressing the problems of the non-i.i.d. data distribution and the client sampling. We show that our method can be effectively applied with these federated optimizers.
Improving the directions of the averaged server gradients: (1) Some studies (Wang et al. 2020 Yeganeh et al. 20205; Wu & Wang 202T) manage to introduce better gradients' re-weighting procedures during server's aggregation than the naive averaging (McMahan et al. 2017). However, they either require an extra global dataset for validation or fail tackle the situation where most of the sampled clients have similar but skewed data distributions.
(2) Instead of focusing on the server side, other studies choose improve the local training the client side. For example, Fed- Prox (Li et al. 2018) and FedDyn (Acar et (020)) choose to add the regularization terms to make local gradients' directions more consistent; SCAFFOLD (Karimireddy et al., 2020) and VRLSGD (Liang et al.) 2019) send the his- torical aggregated gradients to each client for helping correct its local gradient's 'ss direction. In this paper, we consider
I Though in our example Figure for simplicity, we assume the merged data distribution of all available clients balanced, we do not have this assumption our later analysis.
3

---

Published in Transactions on Machine Learning Research (05/2023)
from orthogonal angle that aims to find optimal server learning rate after the averaged gradient is determined. Thus, our method can be naturally combined with them.
Introducing new client sampling strategies: Being aware of the important role the client sampling plays in federated learning, some studies (Ribero & Vikalo 2020 Chen et al. 2020 Cho et al., 2020; Fraboni et al.) 2021a) try to propose better client sampling schemes than the default uniform sampling without replacement strategy (McMahn et 2017). They tend select the most important clients based on some statisties of their local gradients (e.g., the gradient's s norm). Our work is also orthogonal to them and bring additional improvement when applied with them.
3 Methodology
3.1 Problem definition
The optimization target of the federated learning (McMahan et al. 2017) can be formalized
min L())= = min nk (0),
Σ N
(1)
0
0
n
where N is the total number of clients, is the number of local training samples on client k. n Ex ni is the total number of training samples, and (x = [ (k) yy (*)) is the local training objective for client
The most popular federated learning framework is the FedAvg (McMahan et al., (017). which broadcasts the current global model 0c to the available clients when the II+h) round begins and allows each client perform the multi-step centralized training on its own data. Then each client only sends the accumulated local gradient g (k 1,... .N) to the server, and the server will aggregate (average) the local gradients and update the global model as the following:
r++l = =² 17 Σ
Jk
12
where 7/s is the server learning rate. The server will send the 0++1 clients again for the next round's training.
In most real cases, due to the great number of clients and the server's is limited communication capability, the server will perform client sampling strategy to sample small portion of clients only in each round. Also, since we assume the number of local updates performed during each client's local training is the same in each training round following Reddi et al. 2020 then all 7g are assumed to the same. Therefore,
previous studies (McMahan et al. 2017 Eq. can be re-written
0++ 0' 1/s 1
Σ KES(N, gh.
(3)
where r is the number of sampled clients, S(N, r) represents a sampling procedure of sampling r clients from clients in the (t 1)-th round, and we abbreviate it St in the following paper.
3.2 Analysis of the impact of client sampling
As except, in a given round, the averaged gradient under partial client participation will deviate from the ideal gradient under full client participation, which we think has a reliable direction since it contains the data information of all clients. Thus, the motivation of most current studies can be summarized into optimizing the following problem:
max(cosSim(g', ,9)},
g'
where = and g: = =
gi. They achieve this goal indirectly from the perspective of improving
the local gradients, either by adjusting the direction of g (Li et al. 2018 Karimireddy et al. 2020 Acar et al. 2020).
or by sampling better groups of clients (Chen et al. 2020 2021).
"Our later proposed method Section 3.3can can also be applied the setting where the number of local updates training samples varies clients, combining with specific optimization methods like FedNova (Wang al. (0200) IDA Yeganch et al. 2020a). also conduct experiments on combining our method with these weighted averaging-base based methods Appendix
4

---

Published in Transactions on Machine Learning Research (05/2023)
averaged server gradient with fixed learning rate
-1
averaged server gradient with optimal learning rate
0 0¹ n.gt
0r
ideal server gradient with fixed learning rate
Figure 2:
Our motivation is to find an optimal server learning rate multiplied with which the averaged server gradient achieves the nearest distance to the ideal server gradient multiplied by the global fixed server learning rate.
However, based on the analysis in Figure 1 we motivated to mitigate the inconsistent between g and g from different perspective, that is scaling g' to achieve the nearest distance between it and g (refer to Figure 2 By doing so, a great advantage is to avoid updating the model towards a skewed direction overly when the uploaded gradients are similar but skewed (refer to the right top of Figure (.. Then our optimization target can be written as
T = arg min | "" n=g'!1?
where 7) is the global learning rate used under the full client participation setting.
Moreover, we point out that since our method orthogonal to previous studies, we can get a more comprehensive optimization target by combining Eq. (5) with Eq. as: min || "n'gg - -ngg"1?
S.L. g' arg max mnasSCoosimmg...
(6)
However, in the scope of this paper, we only focus on the first part of Eq. 66) that is overlooked by previous studies, while assuming g' is already obtained or optimized by existing methods. Then, by solving Eq. (5) we can get
r n ((g', /lgl'
where < <,, >> is an clement-wise dot product operation.
In the (t - 1) round given 0', g¹. is the determined but unknown parameters, which means it is impossible to calculate < g g. > precisely. However, we can still find the factors that will affect v by analyzing its upper bound:
L g
ns |ل
E |1
v < 1
(8)
V
Eves
1g
|gg
In the first equality of Eq. (8) we make the further derivation to separate the scale component / / ll that
is decided by the scales of the local gradients, and the direction component l|g /1/ V ERES' l1 112 that depends on the similarity between the directions of local gradients, from ||g'll ||. Now in the first term of the final equation of Eq. 8 |لgولl can be considered as an unknown constant in this round given 0% and we will introduce how to estimate it in the following section but consider it as an available constant for now. Also, the scale component can be regarded as a constant that is barely affected by the sampling results in current round, since we assume each client performs the same number of local updates, so the difference between the norms of all local gradients can be negligible (especially when the local optimizer is Adam) ( with the difference of local gradients' directions.
EKES: ||
However, the second term (v EkES' ll |2)/|lg'|| indeed depends the merged data distribution of currently sampled clients, as the directions of the local gradients depends on the sampled clients' local data distributions. Since it measures the normalized similarity between local gradients, we name it the Gradient Similarity- aware Indicator (GSD,, and aim to utilize it to adjust the server learning rates:
GSI = -V E AESI (1a1)))/(rrll()1)))
(9)
*In the following paper. || represents the Frobenius We make the detailed discussion provide the empirical evidence Appendix
5

---

Published in Transactions on Machine Learning Research (05/2023)
The reason why do not choose 1/|lg'|| as the indicator and make the further derivation in Eq. (8) is. the scales of the gradients will naturally decreases when training goes on (refer to Figure 8 in the Appendix), and 1/llg'|l will be in an increasing trend. This leads to consistently increasing server learning rates, making SGD based optimization (happens in the server aggregation phase) fail at the end of training. However, GSI is a normalized indicator whose scale is stable across rounds.
We surprisingly find that the GSI happens have the similar form the gradient diversity concept proposed in the distributed learning field (Yin al., 2018), however, they have different functions: the gradient diversity concept is originally used to illustrate that allocating heterogencous data to different nodes helps distributed learning; while in federated learning, the local data distributions and the sampling results in each round have been decided, but GST can reflect the reliability of the sampling results and be further used to mitigate the negative impacts of bad sampling results. Specifically, we can re- write Eq. (9) as
GSI = -V+ e RES: ll g /
V 1+ EW- Var S*))) E SHI, (10)
where EW-Var(-) represents the sum of the element- variance of random vectors. Then, we can interpret that, (I) if the training data is non-i.i.d., and if the sampled clients have more similar local data distributions, which will be reflected in more similar local gradients and smaller GST, the merged data distribution of sampled clients is more likely to be skewed from the global data distribution. Then the direction of the aggregated server gradient may not be fully trusted, and the server learning rate should be relatively smaller. Otherwise, the variance between local gradients is larger and the GST is larger, thus the server learning rate should larger. (2) If the data i.i.d., client sampling has little effect on the directions of averaged gradients, and GSI will be stable during training. For example, when all clients have the same data, GSI always equals 1. We display the empirical evidence in Section 5.1 show that GSI can indeed reflect the similarity between merged data distribution of sampled clients and that of all clients.
3.3 Gradient similarity-aware learning rate adaptation for federated learning
The above analysis motivates us to use GST as an indicator to relatively adjust the current round's server learning rate. However, the problem that || is unknown still remains. To tackle this problem, we further derive the scale component and direction component of [[ghll, and extend Eq. (8) as
N D
- 9 V
Σ EES 11 |19'11
J N
E
l GST
GST GSI
th S n7
EEESs 19
N
>k=1 112
N
77s
ERESt l1g ||2 GSI
77
(11)
We point put that, the first term EN |egh according to our assumption on the scale component mentioned above, in which the norm of each local gradient is assmued to be similar. Now, the second term includes GSL that is exactly the GSI under the full client participation assumption, and we choose to estimate it with a baseline chosen as the exponential average of historical GSIs:
Exes: l9 2 in Eq. (1)) can be regarded as an constant 1
GSI ~ B' = 3B + (1 - B)S t= 1,2,
Bd GSI..
(12)
Therefore, the adjusted learning rate ît can be calculated by multiplying the initial global learning rate nd with the ratio of GST in the current round and the estimator B¹
タ' = /GSI
(13)
We put the corresponding discussions on the effect of using Ba to estimate GSL in Appendix O
Parameter group-wise adaptation Based on the above analysis, a direct solution is to regard all parameters as a whole and universally adjust the global learning rate based on Eq. (13.. However, in our preliminary explorations, we visualize the fluctuation patterns of GSI in different parameter groups (i.e., weight matrices) when using FedAvg to
"g hryper-paramed that can be tuned the way as previous methods did to get their optimal server learning rate ne. Thus, this an hyper- haper-parameter we introduced our work.

---

Published in Transactions on Machine Learning Research (05/2023)
Gradiemt Similanity aware Inileatos
Indicator
2.6-
3.3
3.0-
Gradien Similarity Similriit--avrr
2.8
2.6
2.3
2.-
1.8
1.
Loyllbertloyyell. w
layert Mobcdti bas weightt
disiiberr layert, alteation linear.wiighl
layeri 2lock0 canv weight
distibert layed attestonn linear, weight
layers aloct conv3
10
10
50
100
10
20
30
40
50
70
50
50
100
Commumication Round
Communication Round
(a) Oluctuation patterns of GSI different parameter groups (b) fluctuation patierns of GSI different parameter groups when
when federated training on 20NewsGruups [Lang 1995)
with Distil- federated training on CIFAR-IO (Krizhevsky et al. 2009) with ResNet- (Ile el al. 2016).
BERT (Sanh et al. 2019).
Figure 3: Visualizations of fluctuation patterns of GSI in different parameter groups during federated training with client sampling.
train on 20NewsGroups (Lang 1995) and CIFAR- 10 (Krizhevsky et al. 2009 in Figure and find that the GSIs have very different fluctuation degrees in different parameter groups whether from the same layer or not. Thus, to achieve more precise adaptations, we give each parameter group 0p an independent adaptive learning rate based on its own GST in each round:
q/- nu (GSIP/BB). =
(14)
where GST; and By, are calculated based on the averaged gradient of the 0% only from Eq. 9
Adaptation with dynamic bounds Moreover, as we can see from Figure the GST varies much more greatly at the beginning than does when the training becomes stable. The reason that, when the global model is newly initialized, the local gradients are not stable, leading to the great variability of GSIs. In addition, there are not enough historical GSIs for estimating B at the beginning. which will further cause the GST./ BBB and fipp to vary greatly. Therefore, to avoid doing harm to the initial training caused by the great variations of the learning rates, set dynamic bounds which will restrict the parameter grsep-wise adaptive learning rates at the initial stage but gradually relax restrictions as the training becomes stable. The finally used learning rate for 0 is:
f = 7o min max (fpp/no. 7t), 14,1++00 *t), t 0,0,
(15)
where y is the bounding factor that controls the steepness of dynamic bounds. We visualize the patterns of the adapted server learning rates by our method and show their connections to the lower and upper bounds (i.e., how frequently i is set to (1 - 7() or no + 7() in Appendix P
We name our method the Gradient Similarity-Aware Learning RAte ADaptation for Federated learning (FedGLAD).
3.4 Combining with other optimization methods
Though the previous analysis based on the FedAvg framework, our proposed algorithm can also be applied with other federated optimization methods. For example, (1) optimization methods such FedProx (Li et al. 2018) and SCAFFOLD (Karimireddy et al., 2020) follow the same server aggregation mechanism that of FedAvg. thus our method is naturally applicable to them; (2) some methods like FedAvgM (Hsu al. 2019 Wang al. 2019b) add the momentum term in the server, so it is feasible to first adjust the scale of the current aggregated gradient by our method, then update the momentum and the global model; (3) if the optimization method takes an adaptive server optimizer such as FedAdam (Reddi et al. 2020), our solution is to adjust the scale of the server gradient in the numerator only while keeping the squared gradient used in the denominator unchanged. We summarize all above variations in detail in Algorithm [ (the specific procedure of implementnng our method is marked in green), and put the detailed discussion about these variations in Appendix
7

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 1 Gradient Similarity-Awarr Learning Rate Adaptation for Federated Learning
1: Server Input: initial global model 0% global learning rate 7/- prraaeeprs / = 0.9, = 0.02 and 31, 82
2: for each round
to do Sample clients from N clients: S C (1, (1,,,,,) ,N
for k E St in parallel do Broadcast current global model O to client it 6: g < Local ArocalTraining(.. 0', D2)
7: end for
8:
g' - + gi
9:
wt+1 + B22' + (1 32)g' @ g' (FedAdam)
10:
for each parameter group 0 's gradient g/ do
11:
GSI + (Enesi 11 (r)lg))))))))))))
/*|Calculate current GSII, based on Eq. 9))
12:
if then
13:
Bd t GST; [*Updatte estimated GST based on Eq. 12
14:
end if
15:
nip min max Bpt + BBp + (1 B)GSTP / (Uppate estimated GST c.P based on Eq. (12) )
-tt, |(Cacullate adapted learning rate based on Eq. 15 |*
16:
17:
gp Wpg'' 1 (Re-scale the aggregated gradient end for
18:
19:
g + 0 - Nugg nug' (FedAvg, FedProx, SCAFFOLD)
20:
m²
m 1+ t 3.m + (1 111--1)6 A1
21:
@ + SGDM m ++1 no) (FedAvgM)
22:
g Adam(0' m w-- no) (FedAdam)
23: end for
24:
25: Client k's Input: current global model 0', local dataset DR, local training hyper-parameters
26: LocalTraining(A: 0¹, Dx):
27: Client l performs centralized training to on D₂ with tuned training hyper-parameters, get final model 0++1
28: Return g f g++1
k
4 Experiments and analysis
4.1 Datasets, models, data partitioning and client sampling
We perform main experiments on two image classification tasks: CIFAR- Krizhevsky et al. 2009) and MNIST Le-
Cun et al., 1998), and two text classification tasks: 20NewsGroupp Lang 1995) and AGNews (Zhang et al. 2015).
The statistics all datasets can be found in Appendix F We use ResNet-56 (He et al., 2016 the global model for CIFAR-10 10 and the same convolutional neural network (CNN) as that used in Reddi et al. (2020) for MNIST. We adopt DistilBERT (Sanh et al. 2019) the backbone model for the two text classification datasets following recent studies (Hilmkil et al. 2021 Lin et al. 202I). We also explore the performance of using BiLSTM (Hochreiter & Schmidhuber 1997) as the backbone in Appendix M to show that our method can also work well when the global model is not pre-trained. Furthermore, we conduct extra experiments on some realistic federated datasets to show that our method can also work well these realistic scenarios, and the results are in Appendix
To simulate realistic non-i.i.d. data partitions, we follow previous studies (Wang et al. 2019a Lin et al. 202 by taking the advantage of the Dirichlet distribution Dir(a). In our main experiments, we choose the non-i.i.d degree hyper-parameter a 0.1. We also conduct extra experiments a = 1.0. The results are in Appendix K and the conclusions remain the same.
The total number of clients for each dataset is all 100, and the number clients sampled in each round is 10. We adopt the widely-used uniform sampling without replacement strategy for client sampling (McMahan et al.] [017]] Fraboni] et al. 2021b). We want show that though our motivation is based on the extreme scenario in which most sampled clients have similar but skewed gradients, our method basically general approach that aims to address the negative
8

---

Published in Transactions on Machine Learning Research (05/2023)
effect of client sampling and can be applied under realistic situations. We further explore the impacts of using different sampling ratios and sampling strategies in Section 5.4 and Section 5.5 respectively.
4.2 Baseline methods and training details
We choose 4 popular federated optimization methods as baselines the server optimizer: FedAvg (McMahan et al. (017)) and FedProx (Li et al. 2018); (2) Using advanced server
0 which can be divided as: (1) Using SGD as
optimizers: FedAvgM (Hsu al. 2019. Wang et al. (019b) and FedAdam (Reddi et al. 2020). The detailed descrip-
tions of them are in the Appendix H We show that our method can be applied to any of them to further improve the performance.
As for the local training, we choose the client optimizers for image and text tasks as SGDM and Adam respectively, For each dataset, we adopt the optimal local learning rate t and batch size which are optimal in the centralized training as the local training syper-paramette tune its own best server learning rate v/s on each dataset. The number of local epochs is 5 for two image classification datasets, and 1 for two text classification datasets. We also conduct the experiments to explore the impact of the number of local epochs E, the results and analysis are in Appendix N The optimal training byper-parameters (e.g., server learning rates) will be tuned based on the training loss of the previous 20% rounds (Reddi et al. (2200). Details of all the training hyper-parameter (including the total number of communication rounds for each dataset) and further explanations are in Appendix[[
for all federated baselines. Then we allow each baseline method to
When applying our method with each baseline method in each setting, we use the original training hyper- parameters tuned before in that experiment directly without extra tuning. In Section 5.3 explore the sensi- tivity to the bounding factor y of our method, and find 0.02 can be generally good choice in all cases. Thus, in our main experiments, we fix 7 as 0.02. The exponential decay factor /3 is fixed as 0.9. We run each experiment on 3 different seeds, and on each seed we save the average test accuracy (%) over the last 10 rounds (Reddi et al. (2020). Finally, we report the mean and the standard deviation of the average accuracy over the last 10 rounds on all seeds. Our code mainly based on FedNLP (Lin et al. 202T) and FedML (He et al. 2020).
4.3
Main results
The results of our main experiments are displayed in Table [ We also display the results of the centralized training on each dataset in Appendix D The main conclusion is, our method can consistently bring improvement to all baselines on both image and text classification tasks in almost all settings. Note that though the standard deviation in each experiment in Table T may seems a bit large compared with the improvement gap, we point out that actu- ally on almost each seed of one specific experiment, applying our method can gain the improvement over the corresponding baseline. We put the result on each seed Appendix Tor comparison.
(1) First of all, our method effectively improves the performance of FedAvg in all settings, which directly validates our analysis in Section 3.3 that it is better to set adaptive server learning rates based on the client sampling results. FedProx also uses SGD as the server optimizer but tries to improve the local gradients by making them have more consistent directions. However, FedProx requires the carefully tuning of its hyper-parameter to achieve better performance than FedAvg (refer to Appendix [.4), and there is no guarantce that FedProx can outperform FedAvg in any setting (same conclusion for SCAFFOLD). This means that empirically adjusting the directions of local gradients may bring negative effects. Additionally, we find that FedAvg applied with FedGLAD can achieve comparable performance with FedProx in some cases. This result indicates that optimizing the scales of the averaged gradients by adjusting the server learning rates is equally as important as optimizing the directions of local gradients before aggregation, while the former is overlooked by previous studies. Also, our method can further help FedProx to gain more improvement, since they are two complementary aspects for optimizing the full target in Eq. 66)
(2) MedAvgM and FedAdam belong to another type of baselines, which uses advanced server optimizers. For these two methods, the averaged server gradient in the current round will continuously affect the training in the future rounds since there is momentum term to store it. Our method aims to enlarge the scales of the relatively reliable averaged gradients, while down-scaling the averaged gradients that deviate greatly from the global optimization direction in
We also perform experiments with SCAFFOLDD Karimireddy et The detailed discussions can he found Appendix
2020 hut we find can not work well in main experiments' settings.

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1: The main results of all baselines with (+FedGIAAD) without our method.
Method
CIFAR-10
MNIST
20NewsGroups
AGNews
SGD Server Optimizer
FedAvg
35,30 (# L62)
78.17 (#b 0.35)
72.84 (# 0.12)
79.48 (#: 0.46)
FodGLAD
38.44 (# 0.91)
79.71 (#: 0.49)
73.14 (#: 0.08)
80.74 (#: 0.79)
FedProx
43.48 0.79)
82.72 (1: 0.25)
72.78 (#: 0.20)
82.61 (:: 0.56)
+ FedGLAD
44.52 (# 0.72)
83.54 (# 0.34)
73.19 (# 0.08)
83.44 (+: 0.44)
Advanced Server Optimizer
FedAvgM
48.79 (# 1,35)
81.71 (# 0.99)
75.44 (# 0.10)
80.28 (+: 1.75)
FedGLAD
50.91 (# 1.33)
82.11 (#b 0.64)
75.91 (#: 0.29)
83.68 (+: 0.87)
FedAdam
56.82 L12)
96.94 (# 0.14)
76.76 (# 1.17)
86.72 (# 1.02)
FedGLAD
57.69 (#: 1.23)
97.11 (# 0.11)
76.59 (#: 0.81)
88.47 (#: 0.51)
specific rounds. Thus, our method can improve the learning when server optimizer includes a momentum term to stabilize training.
(3) When comparing the results in Table I with the results on a = 1.0 in Table 7 Appendix we can get that on each dataset, the trend is the improvement brought by our method is greater if the non-ii.d. degree is larger (i.e., smaller a). Our explanation is when the label distribution is more skewed, the impact of client sampling becomes larger. More analysis is in Appendix K
4.4
The positive effect of FedGLAD on stabilize training under extremely bad client sampling scenarios
In Figure we draw example under the extremely bad client sampling scenario to illustrate the necessity of adapting server learning rates according to the client sampling results. However, in our main experiments, we have conducted experiments to show that our method is a general approach that can gain improvement under realistic scenarios. Here, in order to validate our motivation and explore the superior performance our method in the extremely bad client sampling cases, we further conduct a special experiment on MNIST with FedAvg. Specifically, while keeping other settings unchanged, we make sure that clients sampled in the 40-th round only have almost one same category of samples. Figure 4 displays the curves of the evaluation accuracy and the GSI calculated by regarding all parameters a whole after each round.
We notice that in the 40-th round, the accuracy of FedAvg drops greatly due to the bad client sampling results, and the corresponding GSI is also the smallest (i.c., close to 1) during the training. After applying FedGLAD, the accuracy degradation problem is effectively mitigated, and the accuracy curve is much more stable. The indicates that, our method can help to stabilize the training when extremely bad client sampling scenarios happen. This also helps to verify our proposal to utilize GSI as the indicator to reveal the reliability of current sampling results.
5 Deep explorations
In this section, we take steps further to make more explorations of our method. We choose FedAvg and FedAvgM as baseline methods. and perform experiments on CIFAR-I0 and 20NewsGroups (with the backbone DistilBERT) datasets with the non-i.i.d. degree a 0.1. The bounding factor y of our method is fixed 0.02.
5.1
GSI can effectively reflect the merged data distribution similarity between the sampled clients and all clients
Here, we make deep analysis about the correlation between our proposed GSI and the merged data distribution similarity between the sampled clients and all clients. First, we calculate the GST in each round considering all
10

---

Published in Transactions on Machine Learning Research (05/2023)
3.6
Pearson Corr. 0.4753
Evaluation Accuracy (%)
3.3
GSI All Paramelen
70
3.0
GSI GSI w.rk All Parameters
4.0-
60
2.7
3.5
2.4
40
2.1
3,0
30
1.8
Bad Sampling Happens
2.5
PodAvg
1.5
10
TodArg TedGLAD
1.2
2.0
FSS
0.9
0
10
20
30
35
40
45
0.60
0.65
0.50
0.75
0,50
0.85
0.90
0.95
Communication Round
Merged
Distribution Similarity biw Selected and All Clients
Figure The good effect of FedGLAD on stabilizing training when extremely bad client sampling scenarios happen.
Figure 5: GSI effectively reflects the merged data dis- tribution similarity between the sampled clients and all clients.
parameters as a group. At the same time, we calculate the similarity score between the merged data distribution of currently sampled clients and that all clients. The similarity score is based on the cosine similarity between two distributions: denote I, as the total number of classes, nig as the number of samples belonging to label in client k's local data, n2g -'' n then the merged data distribution of sampled clients is
Tie EnSs: (L)
d' ERES: nhk EkEST n2
(16)
Evest 7kk
Similarly, the merged data distribution of all clients is written as
(2) k= nk
N
de k=
(17)
2k=1 7
k= nik
Thus, the similarity score can be calculated as
sim_score' CosSim(d' de).
(18)
We then visualize the scatter plot of the corresponding GSI* and sim_score' on CIFAR-10 in Figure 5 The red line corresponds to the linear regression result. The Pearson correlation coefficient hetween two variables is 0.4753, revealing that GSI is positively related to the the merged data distribution similarity between the sampled clients and all clients. This indicates that the GSI can effectively reflect the sampling results, which also helps to verify our original motivation use GSI to optimize the server learning rates.
5.2 The benefits of adjusting learning rates individually for different parameter groups
Here, we explore the benefits of using parameter group-wise adaptation compared with the direct solution that con- siders all parameters whole and adjusts the server learning rate universally (universal adaptation, mentioned at the beginning of Section 3.3) The results are in Figure
Firstly, the universal adaptation strategy can already bring improvement to the baselines without adaptation. This indi- cates that dynamically adjusting the server learning rates based on the dissimilarity between local gradients can indeed help federated learning. Moreover, the performance parameter group-wise adaptation strategy is consistently better than the universal adaptation strategy, which validates our motivation based on the findings Figure 3 that the non-i.i.d. data distribution has different impacts on the different parameter groups, so we should deal with them individually.
5.3 Setting dynamic bounds helps to stabilize the training and achieve better performance
In our method, we introduce a dynamic bound to restrict the adapted server learning rates the beginning of training. in order to avoid the greatly fluctuated server learning rates to cause harm to the training at the initial stage. Therefore,
11

---

Published in Transactions on Machine Learning Research (05/2023)
77
without adaptation
50.40
500.1
without adaptation
adaptation
Test Accuracy (%)
76
adaptation
75. 15.44
75.82
75.91
Accuracy
group- adaptation
group-wiis -wise adaptation
45
75
80
74
35.30
37,05
72.84
72.95
73.14
35
FedAvg
FodAvgM
FedAvg
FedAvgM
(a) Results CIFAR-IO.
(b) Results on 20NewsGroups..
Figure 6: Comparisons between two learning rate adaptation strategies of FedGLAD, Parameter group-wise adapta- tion brings consistent improvement.
50.91
Test Accuracy (%6
+8:09
48.92
49.09
68/41
49/67
Test Accuracy (%)
75.44
73,50
75.91
75/72
75:56
73.48
45
FedAyg
FodAvgM
75
FedAvg FodAvgM
30.03
38,44
39.52
37.73
74
3689
55.30
72,84
73.14
73:05
73/13
73,05
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
Bounding Factor y
Bounding Factor y
(a) Results on CIFAR-IO.
(b) Results on 20NewsGroups..
Figure 7: The results of using different y for FedGLAD, 7 = 0 represents original baseline and ? = oo represents removing the bounds from the beginning. 7 = 0.02 can generally good choice.
we want to explore our method's sensitivity on the bounding factor 7 here. The results are displayed in Figure Generally speaking, the best y in each setting depends on many factors, such the dataset and the non-i.i.d. degree that there exists general value (i.e., 7 0.02) which
of data distribution across clients, but from figures works well in most cases. Moreover, (1) though the performance of removing the bounds is better than the original baseline in some cases, is consistently worse than that of setting the bounds. (2) The standard deviation in each experiment of using larger increases. This validates our motivation that without the restrictions, the learning rates at the beginning will vary greatly, which will cause harm to the training.
5.4 Our method consistently brings improvement under different sampling ratios
In the main experiments, we fix the sampling ratio as 0.1. Here, we explore our method's performance in the settings where 5, 10, 20 out of 100 clients are sampled in each round. The optimal server learning rates for baselines are re-tuned here. The results are in Table 2 The overall trend is the improvement brought by our method is more significant when the sampling ratio is smaller. That because when the number sampled clients increases, the merged data distribution of sampled clients is more likely be consistent with the merged data distribution of all clients, thus the impact of client sampling decreases.
5.5 Our method can be effectively combined with various client sampling strategies
In this subsection, we want to show that our method can also be combined with other sampling mechanisms besides the normal one used our main experiments. We conduct experiments with two additional sampling strategies: (1) Multinomial Distribution-based Sampling (MD) (Fraboni et al., 2021b): considering sampled clients as i.idd
samples from Multinomial Distribution; (2) Dynamic ittention-based Sampling (AdaFL.) (Chen al. 2021
12

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: Results under different sampling ratios.
Table 3: Results of different sampling strategies.
Sampling Ratio
Sampling Strategy
Datasel
Method
Dataset
Method
10%
20%
MD
AdaFL
FedAvg
29.91
35,30
52.05
FedAvg
41.44
39.85
FodGLAD
33.35
38.44
53.40
FedGLAD
42.20
41.39
CIFAR-I0
CIFAR-10
FedAvgM
34.72
48.79
63.56
FedAvgM
51.77
52.76
FodGLAD
36.48
50.91
63.84
FedGLAD
53.85
53.68
FedAvg
63.09
72.84
76.31
FedAvg
73.36
72.14
FedGLAD
63.49
73.14
76.35
FeuGLAD
74.04
72.92
20NewsGroups
20NewsGroups
FedAvgM
61.51
75.44
80.08
FedAvgM
75.22
76.18
FedGLAD
62.48
75.91
79.98
FedGLAD
75.60
76.40
dynamically adjusting the sampling weights of clients during the training. The sampling ratio is 0.1. The results in Table that our method can also work well with these sampling strategies, since they aim sample better groups of clients in each round while our method improves the learning after the sampling results are determined.
6 Conclusion
In this paper, we study and mitigate the difficulty of federated learning on non-i.i.d data caused by the client sampling practice. We point that the negative client sampling will make the aggregated gradients unreliable, thus the server learning rates should be dynamically adjusted according to the situations of the local data distributions of sampled clients. We then theoretically find that the optimal server learning rate in each round is positively related to an indicator, which can reflect the merged data distribution of sampled clients in that round. Based on this indicator, we design a novel learning rate adaptation mechanism to mitigate the negative impact client sampling. Thorough experiments validate the effectiveness of our method.
Broader impact statement
Though the purpose of our work is positive as we aim to improve the performance of improve federated learning on non-iidd. data, we admit that some adversarial clients may utilize the characteristics of our method enforce the server to focus on their outlier local gradients, since our method tends to give larger server learning rates when local gradients have more divergent directions. However, safety problems are in the scope of this paper and belong another field, One possible solution is applying existing detecting techniques filter extremely abnormal gradients first before aggregation. Take Krum (Blanchard et al. 2017] as example, when the local clients upload gradients as {g} l E 5*). Krum may first select part of local gradients based on its mechanism as {g Ik E S Kram }. Sk Kram C S*. Then, the aggregated gradient will be calculated based on 5% as Ikrum
ΣNEST ExEskrum Krum g and our proposed
1St Nrum
GSI and learning rate adaptation are also based on St krrw as GSI
EKESY
[9 krrmm 12).
Krum
Acknowledgments
We sincerely thank all the anonymous reviewers and the Action Editors for their constructive comments and valuable suggestions. This work was supported by Tencent Research Grant. Xu Sun is the corresponding author of this paper.
13

---

Published in Transactions on Machine Learning Research (05/2023)
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations. 2020.
Peva Blanchard, EI Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprint 1rX1v:2010.13723, 2020.
Zihan Chen, Kai Fong Ernsst Chong, and Tony QS Quek. Dynamic attention- based communication--fficieee federated learning. arXiv preprint arXiv:2108.05763 2021.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-
of-choice selection strategies. arXiv preprint arXiv:2010.01243,, 2020.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten
letters. In 2017 international joint conference on neural networks (IJCNN), pp. 2921-2926. IEEE, 2017.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/
transformers for language understanding. In Proceedings of the 2019 Conference the
N19- 1423. URL www.eb..llbb....... www..//ww...//wwwooeeooooo http://aaatooooogg/111111./..
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic opti- mization. Journal of machine learning research, 12(7), 2011.
Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. arXiv preprint arXiv:2105.05885 2021a.
Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. On the impact of client sampling federated learning convergence. arXiv preprint 200.1221.......11
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13511 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings ofthe the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
Agrin Hilmkil, Sebastian Callh, Matteo Barbieri, Lcon René Sütfeld, Edvin Listo Zec, and Olof Mogren. Scaling federated learning for fine-tuning of large language models. International Conference on Applications of Natural Language to Information Systems, pp. 15-23. Springer, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1997.
Tzu-Ming Harry Hsu, Hang Qi, and Mathew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint 19Xiv:1909006335. 2019.
Sai Pranceth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980. 2014.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint 100.021610 2016.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Alex Krizhevsky al. Learning multiple layers of features from tiny images. 2009.
Ken Lang. Newsweeder: learning to filter netnews. In Proceedings of the Twelfth International Conference on International Conference on Machine Learning, pp. 331-339, 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 11:22)8-2232222 2324, 1998.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study, arXiv preprint arXiv:2102.02079 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimiza- tion in heterogeneous networks. arXiv preprint arXiv: 18/2.0677, 2018.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019.
Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance reduced local sgd with lower communication complexity. arXiv preprint arXiv: 12844, 2019.
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi. Xiang Ren, and Salman Avestimehr. Fednlp: A research platform for federated learning in natural language processing. arXiv preprint arXiv:2104.08815 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282.
PMLR, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532- 1543, 2014.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representa- tions, 2020.
Monica Ribero and Haris Vikalo. Communication-eficiienn federated learning via optimal client sampling. arXiv preprint 15X1 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint wwrXivv/990..0m 108, 2019.
Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging unlabeled auxiliary data in federated learning. arXiv preprint
2021.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2019a.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-eficen distributed sgd with slow momentum. In International Conference on Learning Representations, 2019b.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33, 2020.
Hongda Wu and Ping Wang. Fast-convergent federated learning with adaptive weighting. IEEE Transactions on Cognitive Communications and Networking, 2021.
Yousef Yeganch, Azade Farshad, Nassir Navab, and Shadi Albarqouni. Inverse distance aggregation for federated learning with non-iid data. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop. DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2, pp. 150-159. Springer, 2020a.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Yousef Yeganch, Azade Farshad, Nassir Navab, and Shadi Albarqouni. Inverse distance aggregation for federated learning with non-iid data. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 150-159. Springer, 2020b.
Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: key ingredient for scalable distributed learning. In International Conference Artificial Intelligence and Statistics, pp. 1998-2007. PMLR, 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systemx, 28:649-657, 2015.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00588, 2018.
16

---

Published in Transactions on Machine Learning Research (05/2023)
A Limitations
There are also some limitations of our method: (1) First, we do not provide the convergence analysis of our algorithm. However, our algorithm's 's convergence can be guaranteed by the convergence results of existing federated learning algorithms (Li et al. 2019 Karimireddy et al. 2020), since our approach is directly plugged into these methods without changing their main frameworks and our adaptive learning rate is well-bounded by the common assumption
that the heterogeneity of local gradients is bounded. (2) Second, in Eq. (8) and Eq. [TI). we make the assumption that
the term / ERES | is barely affected by client sampling, while this assumption only strictly holds when the local optimizer is Adam. However, we provide the empirical evidence in the following Appendix [[ show that this assumption can still roughly holds even when the local optimizer is SGD(M). (3) Finally, in order to tackle the problem of unknown g¹ when solving 7/. in Eq. ( we make a series of mathematical deductions and approximations to finally get GSI in Eq. 9 for adapting server learning rates. As we can find, GSL has a limitation that will be different under two situations in which the sum of the glement-wise variances between local gradients (EKEs: llg. -3' |13 are different while the mean ge is the same. This causes some inconsistency with the idea optimization goal in Eq (5). as in Eq (5) same g' leads to same GSI However, we think it is the artifact and the cost of the deductions made to tackle with unknown g::
B The impact of client sampling when clients perform multiple local updates
In our main paper, we take the example of FedSGD (McMahan et al. 2017 to illustrate our claim that, client sampling is the main root cause of the performance degradation of federated learning non-ii.d. data. In real cases, clients usually perform multiple local updates before uploading the accumulated gradients (McMahan et al. 2017), in order to accelerate the convergence. In this case, previous studies (Karimireddy et al. (2020) point out that even under the full client participation setting, there still exists the optimization difficulty of FedAvg. The reason is, when the number of local steps increases, the local gradients are accumulated to have more divergent directions, which leads to the unstable aggregation of local gradients in the server. However, this is not contradictory to our claim that, in this scenario, the directions of the averaged server gradients should still be more reliable and accurate under the full client participation setting than that under the partial participation setting.
C Discussion about the impact of client sampling on the scale component
In Eq. and Eq. we made the further derivation by claiming that compared with the GSI, the scale component V E g 2 is barely affected by the client sampling results. We admit that this claim can hold under some certain assumptions. (I) First, if the local optimizer is chosen as Adam, the norm of each local gradient is almost the same. That is because the update scale by using Adam is independent of the scales of gradients, but only depends on the learning rate and the number of updates, which are assumed to be the same across clients. (2) Second, if SGD(M) is the local optimizer, it true that the scales of local gradients are different across clients, especially under the non-i.i.d. data scenario. However, if the number of local epochs is small, the difference between the norms of local gradients can be negligible compared with the divergence between their directions. This assumption can hold in ours setting since previous studies (L.i et al. 2018) suggest to use relatively small number of local epochs under the non-i.i.d. data situation with large number of clients and the small sampling ratio. The reason is, in this case, large local epochs may cause the local model to overfit the local data severely and make the aggregation difficult.
In order to explore the impact of client sampling on the scale component, we conduct two validation experiments:
The first experiment is, we calculate the mean and the standard deviation of the squared norms of the uploaded local gradients in each round and then calculate the ratio between the standard deviation and the mean value. We report the averaged value of the ratios in each run. If the ratio is small during the training, component is indeed barely affected by the client sampling. The results are in Table 4 As we can see, (1) when the local optimizer is Adam (i.e., experiments on 20NewsGrroups), the ratios are all less than 0.2 in all settings, and the ratios are smaller when a is smaller or the number of epochs decreases. This indicates that when Adam is the local optimizer, our assumption that the squared norms of local gradients are almost the same across clients can hold
can have that the scale
Thee optimization method is FedAvg and the settings kept as the same as that the main experiments.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Grad, Norm Cone. Layer
Average Gead Noma of Local Gradicus
|1oE1l
Table 4: The averaged ratio values between the mean and standard deviation of uploaded local gradients in differ- ent settings.
20.5
Avenge Grai Norm of All Local Cratiians n ||s||p
Local Epochs
Dataset
3
5
0.1
0.24
0.29
0.33
CIFAR-10
00
1.0
0.23
0.19
0.15
Communication Round
0.1
0.12
0.15
0.16
20NewsGroops
Figure 8: The visualizations of the values of
1.0
0.08
0.15
0.15
$ Ekes |! and N E llok during training.
well. (2) When the local optimizer is SGDM (i.e., experiments on CIFAR-10), the ratios become larger as expected. However, even when the number of local epochs and a 0.1, the ratio smaller than 1/3, so our assumption that the scale component is barely affected by the client sampling compared with the direction component still roughly holds, and the experimental results also validate the effectiveness of our method even when choose SGDM as the local optimizer. An interesting phenomenon in CIFAR-10 is, when 1.0, the ratio becomes smaller as the number of local epochs increases, but is still smaller than that when = 0.1. Our explanation is, when the number of local epochs is small, the randomness (e.g., data random shuffle, dropout) during the local training plays great role; and when the number of local epochs is large enough, the scale of each local gradient stable and only depends on the local training objective.
The above experiment is based on a stronger assumption that any local gradient's norm is similar to each other. However, in Eq. (II), the more relax assumption need is the averaged gradient norm of currently sampled local gradients EEES:: || is equal to that of all local gradients under the full client participation setting N Ex= |l Therefore, we then visualize the values of E>es: ll and N E llog I| during the training on MNIST with FedAvg and a 0.1 (results in other settings are same). The results are Figure & As we can see, in each round, 1 Eest l1 2 is very close to N ا |l ||2, indicating that assumption on the scale component used in Eq. 111) can be satisfied.
D Why choose parameter group-wise adaptation
we point out that giving different adaptive learning rates to different parameter groups (i.e., weight matrices) can achieve more precise adjustments compared with the universal adaptation strategy, which considers all parameters as a whole group. We verify the benefit of using the parameter group-wise adaptation strategy in Section [522 Researchers can also choose other adaptation strategies, such as adjusting the learning rates layer-wisely that consider all parameters the same layer as group. However, we find the layer-wise adaptation performs worse than the parameter group-wise adaptation. The reason is, even in the same layer, GSI has different fluctuation patterns in different kinds of weight matrices (refer Figure 3 Thus, the parameter group-wise adaptation strategy should be better. On the other hand, we point out that individual group to be adjusted should contain certain number of parameters, in order to make the GSI more reliable due to the fact that the neurons in the same global model that are dropped out during different clients' local training are different. This indicates that the element-wiss adaptation (i.e., each element in a weight matrix is re-scaled independently based on its own GSI) may not work well.
In Section 3.3
E Combining FedGLAD with various federated optimization methods
If the optimization method aims to improve the local training while following the same aggregation mechanism as that of FedAvg, such as FedProx (Li al., 20118). SCAFFOLD (Karimireddy al., 2020) or FedDyn (Acar et al. 2020). adapt the server learning rates following the way that in FedAvg.
18

---

Published in Transactions on Machine Learning Research (05/2023)
There some other methods (Wang al. 2019b; Hsu et al. 2019) which try to utilize momentum term of the server gradient to help aggregating. In this case, we reformulate the original optimization target in Eq. 5 as:
To = arg min (I(n'g" + 3mmmm (g: + B. 8m mI2 arg min 11 g''g" - g'll?
n1¹
of
g¹ - 1 Σ
(19)
m = "0----1 + mm m gi
RES
Thus, the solution can be written as
< g g. |
%
(20)
which is the same as that in Eq. 77) with 7/ = 1.0. Therefore, in this case, we can first adjust the current aggregated gradient's scale following the same mechanism discussed through Section 3.3] Then we use the adjusted server gradient update the momentum, and update the global model with the global learning rate.
If the federated optimization framework (Reddi et al., 2020) utilizes an adaptive optimizer such Adam (Kingma & Ba 2014) or AdaGrad (Duchi et al. 2011) to update the global model in the server, our solution is to only adjust the scale of the server gradient used in the numerator while keeping the squared gradient in the denominator unchanged. There are two reasons: (1) The scale of the server gradient will not only affect the first moment estimate mn in the numerator, but also the second moment estimate of in the denominator. For example, we decrease the scale of current server gradient, then if the squared gradient g @ g' also correspondingly down-scaled, the second moment estimate ot in the denominator will also decrease, which may lead to larger adaptive learning rate, and this is not we expect happen. (2) The second reason is, the purpose of using the second moment estimate in Adam is to estimate and adjust the scale of current gradient based on the previous steps' gradients as the training goes on, while the purpose of our method is to optimize the scale of current gradient to mitigate the negative effect brought by the randomness of client sampling at a given round. Thus, we choose to keep the scale of the squared gradient unchanged.
We summarize all above variations in detail in Algorithm I in the main paper.
F
Datasets and models
We Here we provide the detailed descriptions of the datasets and models we used in our experiments in Table [ get data from their public releases. Since these datasets are widely used, they do not contain any personally private information. Some of them do not have the public licenses, so we give all the public links to the datasets we use in the following. Following previous studies (Reddi et al.) 2020). we use all original training samples for federated training and use test examples for final evaluation. We tune the -erra-pprreeeers based the averaged training loss of the previous 20% communication rounds (Reddi al., 2020).
We use the ResNet-56 (He et al. 2016) as the global model for CIFAR- 1 C Krizhevsky et al. 2009). and the same CNN used in Reddi et al. (2020) for MNIST" (LeCun et al. 1998). For 20NewsGroups (Lang 1995) and AG- New! Zhang et al. 2015), we choose the DistilBERT (Sanh et al., 2019) as the global model, since previous work (Hilmkil et al. 202T) validates the great effectiveness of utilizing pre-trained language models for federated fine-tuning, and DiTtilBERT is a powerful lite version of BERT (Devlin al. 2019) which makes more suitable to local devices. The max max_seq________e seq_lengt is set 128. We also conduct experiments BiLSTM (Hochreiter & Schmidhuber 1997), and display the results in Appendix M
G Data partitioning strategy
To simulate the non-i.i.d. data partitions, follow FedNLP (Lin et al. 2021] by taking the advantage of the Dirichlet
distribution. Specifically, assume there are totally C classes in the dataset, the vector qk (9k.i 2 0, [1, ||q |l
1) represents the label distribution in client k, then we draw q from Dirichlet distribution qk Dir(op) where
*CIFAR-10 can be obtained from http::/wwwwww./c.. es toronto. kriz/cifar.htm)
MNNIST can he obtained from http://yann..... lecun. wwwun.com/ex.t//nee..... com/extl//mnss//
"22NewsSrroups can be obtained from http://wwon.....m com 2uson/2UNewsrrougg
11 The source page of AGNeww corpus http://grous...i...o uuiiii gull:/Al corrpus_________e of news_articles.hmm
19

---

Published in Transactions on Machine Learning Research (05/2023)
Table 5: The statistics of the datasets used in our experiments.
Dataset
# Training Samples
# Testing Samples
# Labels
Model
of Rounds
CIFAR-10
50,000
10,000
10
ResNet-56
500
MNIST
60,000
10,000
CNN
50
20NewsGroups
11,314
7,532
20
DistilBERT DistilBERT
100
AGNews
120,000
7,600
100
P represents the label distribution in the whole dataset and controls the degree of skewness of label distribution: when becomes smaller, the label distribution becomes more skewed; when 0o, it becomes the i.i.d. setting. We assume each client has the same number of local training samples [2] so we can allocate samples to client l once qk is determined. In order to solve the problem that some clients may not have enough examples to sample from specific classes as the sampling procedure goes, FedNLP provides a dynamic reassigning strategy that can use samples from other classes which still have remainings to fill the vacancy of the chosen class whose samples are all used out. More details can be found in Lin al. (202T).
H Baseline methods
We choose 4 widely-used federated optimization methods baselines in our main experiments, and they are: (1) FedAvg (McMaaan et al.) (017): the most popular baseline in federated learning; (2) FedProx (Li et al.) 2018): the widely-adopted baseline which aims to optimize the directions of local gradients, while keeping as the same aggregation mechanism as that of FedAvg in the server; (3) FedAvgM (Hsu et al., 2019 Wang et al., 2019b): the method that adds the momentum term during the server's updates; (4) FedAdam (Reddi et al. 2020): the method that utilizes the adaptive optimizer Adam (Kingma & Ba 2014) to update the global model with the averaged server gradient. We also conduct experiments on another baseline SCAFFOLD (Karimireddy al.] 2202)), while during experiments find SCAFFOLD itself not converge well in our main experiments' settings. Thus, make separate discussion about this part in Appendix D
Training details
1.1 Local training
Firstly, we find SGDM is more suitable to train CNNs and Adam is more suitable to train Transformer-baat models. Thus we choose the client optimizer as SGDM when training on two image classification datasets, and choose the client optimizer as Adam for two text classification datasets. Then we perform centralized training on each dataset in order to find the optimal client learning rate тι and batch size B, and use that learning rate and batch size as the local training hyper-parameters for all federated optimization baselines in the federated training setting. The optimal local training typer-parametens for each dataset listed in Table 16 To decide the number of local training epoches for each dataset, we tune it from (1, 5, 10) by using FedAvg, and use the tuned one for other optimization baselines.
1.2 Server aggregating
We allow each baseline to tune its own optimal server learning rate in each data partition setting. The server learning rate search grids for all baseline methods are displayed in Table 17 We also display the server learning rates we have tuned and used in our main experiments in Table T8 The server learning rates used for the baseline methods in Section 5.4 and Section 5.5 are re-tuned, since we find the optimal server learning rates under different sampling ratios or different sampling strategies are different.
There are several reasons to use the above optimal hyper-parameters in the centralized training setting should also work well for clients' local training in the
tuning strategy: (1) Since clients have local data, the
I2 this paper, we mainly focus the label distribution shift problem, rather than quantity shift problem. Also, since we assume each client performs the same number of local updates following previous studies, this practice reasonable.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Table 6: The mean value and the standard deviation of the test accuracy for each dataset in the traditional centralized training setting.
CIFAR-10
MNIST
20NewsGroups
AGNews
91.86 (# 0.49)
99.26 0.04)
84. 12 (# 0.05)
93.43 0.07)
federated training. (2) Using the same local training hyper-parameters for each baseline method allows us to make fair and clear comparison between them.
1.3
Infrastructure and training costs
All experiments are conducted 4+TITAN RTX. The running time of each method CIFAR-I0, MNIST, 20News- Groups and AGNews is about 12h, 0.5h, 6h and 6h respectively.
1.4
Hyper-parameter's tuning of FedProx
For FedProx, we allow it to tune its own syper-parameter μ from {1.0, 0.10 0.01, 0.001] in each data partition setting, and we display the best in each setting Table [5] As we can see, there no general μ for settings, and researchers may carefully tune this hyper-parameter each setting.
1.5 Hyper-parameter's tuning of FedGLAD
As for our method, the key its dataset, the label skewness degree and the backbone model, in our main experiments we show that there exists a general value (i.e., 7 0.02) which works well in most settings. If researchers want to tune the optimal y in each setting achieve better performance, the recommended grid is {0.01, 0.0................... 0.055.
is the bounding factor y. Though the best in each setting depends on
Since method aims to adjust the server learning rate relatively, it is also important to choose proper global learning rate dou (refer to Algorithm 1 During our experiments, when applying server learning rate used in the original baseline without extra tuning, since we assume the optimal learning rate for the original baseline is the right choice for the global learning rate after applying our method, and we want to show our method is easy to apply. However, is expectable that re- re-tuning the global learning rate for our method may bring more improvement.
method, we directly use the
J Results of centralized training
For comparison with the results in the federated training setting, we display the results on all datasets in the traditional centralized training setting in Table The training settings and hyper-parameters on each dataset are kept as the same as that in the local training of the federated learning.
K Experiments on 1.0
Besides the non-ii.d. degrees we set in our main paper conduct the main experiments, we also explore the perfor- mance of our method when = 1.0. The baselines and the settings are the same as that in the main experiments, and the results are in Table
As we can see, FedGLAD can also consistently bring improvement in this setting, which validates the effectiveness of our method. However, compared with the results Table | in the main paper, we find that on each dataset, the overall trend is the improvement brought by our method are greater if the non-i.id. degree larger (i.e., smaller a). Our explanation is when the label distribution is more skewed, the impact of client sampling becomes larger. For
This important paper-parameter if the optimization method follows the same aggregation strategy that of FedAvg. However, using FedAvgM FedAdam, no 1.0.
21

---

Published in Transactions on Machine Learning Research (05/2023)
Table 7: The results of all baselines with (+FedGLADD or without our method on 1.0.
Method
CIFAR-10
MNIST
20NewsGroups
AGNews
SGD Server Optimizer
FedAvg
72.09
1.27)
91.42 (#: 0.11)
76.66 (# 0.36)
91.27 (# 0.04)
FedGLAD 72.79 (= 1.13)
91.63 (d: 0.08)
77.21 (# 0.42)
91.38 (#: 0.04)
FedProx
71.22 (1: 0.83)
91.68 (1 0.05)
76.51 (1 0.47)
91.24 (1 0.09)
+ FedGLAD
71.52 (= 0.61)
91.86 (#: 0.06)
77.13 (= 0.17)
91.35 (# 0.09)
Advanced Server Optimizer
FedAvgM
72.63 ( 0.56)
93.37 (dt: 0.29)
79.51 (# 0.31)
90.07 (1 0.27)
FedGLAD 73.07 (= 0.33)
93.97 (# 0.26)
79.77 (# 0.08)
90.45 (# 0.42)
FedAdam
79.97 (= 1.30)
98.42 (# 0.08)
79.96 (= 0.12)
91.72 (# 0.12)
+ FedGLAD 80.75 (= 1.35)
98.53 (1: 0.04)
80.13 (# 0.20)
91.81 (1: 0.10)
Table 8: The results of SCAFFOLD with (+FedGLADD or without our method in different non-i.i.d. degree settings.
20oups
AGNews
MNIST
Method
0.1
1.0
0.1
1.0
0.1
1.0
SCAFFOLD
76.22 (#: 0.63)
77.14 (d: 0.39)
77.43 (t: 0.99)
91.08 (=) 0.15)
86.02 (d: 0.64)
87.93 (d: 0.30)
FedGLAD 76.88 (#: 0.48)
78.46 (:: 0.30)
81.42 (t: 1.48)
91.17 0.16)
86.06 (+: 0.88)
87.91 (d: 0.29)
example, when data is more non-i.i.d. and the sampled clients have the similar but skewed local data distributions, the averaged server gradient's direction will deviate far more away from the ideal direction that we assume is averaged by all clients' local gradients. Thus, it becomes more important to adjust the server learning rates in these scenarios.
L Experiments on SCAFFOLD
Besides the baselines we choose in the main experiments, we also conduct experiments with another baseline SCAF- FOLD (Karimireddy et al. 2020). SCAFFOLD aims to optimize the directions of the averaged server gradients by improving the local training with the help of local control variates. The original version of SCAFFOLD assumes the client optimizer is SGD, while we make some adaptations make applicable when the client optimizer is SGDM or Adam. However, we find the SCAFFOLD can not converge well on two image classification tasks if we follow the same training settings in our main experiments. We find the key factor which will affect the performance of SCAF. FOLD is the sampling ratio and the number of local training epochs E. That is, if the sampling ratio is very small and when E is very large, SCAFFOLD may not converge well. We guess the reason may be that, when the sampling ratio is small, the number between two consecutive rounds of participation of client k is large. Thus, the local control variate of client & in the latter round may no longer be reliable, and using this control variate to adjust the directions of local gradients for many local update steps will cause harm to the local training. While for two text classification tasks, we choose E I and also use pre-trained backbone that already mitigates the problem caused by the non- i..d. data distribution and the client sampling at a certain degree, the SCAFFOLD can work well on these two text classification datasets.
We keep the training hyper-parameters the same as that in our main experiments for two text classification tasks. After re-tuning, we choose E for two image classification tasks. Other settings are the same as that in our main experiments. We find SCAFFOLD itself can still not converge well on CIFAR-10, 10, so we only report the results on the other three datasets in Table 8 The conclusions are the same as those in the main experiments: FedGLAD is also applicable with SCAFFOLD and tends to bring more improvement when non-i.i.d. degree is larger.
22

---

Published in Transactions on Machine Learning Research (05/2023)
Table 9: The results of BiL.TM..
Method
20NewsGroups
AGNews
Centralized
78.28 (#: 0.94)
91.47 (=011))
FedAvg
0.1
36.25 (1 0.46)
69.84 (# 0.29)
FedGLAD
0.1
37.37 (# 0.40)
70.64 (= 0.24)
FedAvgM
0.1
35.54 (#: 0.64)
71.36 (# 1.29)
FedGLAD
0.1
35.91 (d: 0.67)
73.50 (# 1.40)
Table 10: The results of different numbers of local epochs.
CIFAR-10
MNIST
Method
E = 10
E = 20
E = 10
E = 20
FedAvg
52.23 (#: 1.78)
58.31 (# 1.59)
82.71
(# 0.68)
86. 13 (:: 0.40)
+ FedGLAD
53.81 0.56)
59.02
(1: 1.13)
84.25
(1 0.33)
87.47 (1. 0.35)
FedAvgM
53.28 (#: 1866
56.59 (#: 1.50)
83.15
(# 2.63)
86.20 (1: 2.55) 86.68 (1: 2.68)
+ FedGLAD
55.55 (#: 0.24)
58.35 (#: 1.04)
82.71
( 2.64)
M Experiments on BiLSTM
In our main experiments, we use pre-trained backbone DistiBBERT for two text classification tasks. In this sec- tion, we conduct extra experiments with a not pre-trained backbone BiLSTM (Hochreiter & Schmidhuber 1997) on 20NewsGroups and AGNews. The model a I-layer BiLSTM with the hidden size of 300. We use the pre-trained Pennington et al. 2014) for the embedding layer. The max_seq_lengte
word embeddings glove. 6b. 300d 14 128. The dropout ratio for the LSTM layer 0.2.
We use FedAvg and FedAvgM baselines, and the non-i.i.d. degree a 0.1. Other details can be found in Table 19 The results are in Table As we can see, since BiLSTM is not pre-trained, the performance gap between the federated training and the centralized training is larger than that of using DistilBERT as the backbone under the same number of communication rounds. Secondly, after applying our learning rate adaptation mechanism, the performance consistently becomes better. The experimental results validate the effectiveness of our method on BiLSTM.
N Experiments with different numbers of local epochs
In our main experiments, we set local epochs in the main paper and Appendix C when the local optimizer is SGD(M) and when the number of epochs increases, our assumption about the scale component and the direction component may generally holds. However, also explained, increasing local epochs will casily make the local model overfit to the local dataset severely, and the choices of local epochs in our experiments are also widely-adopte choices in previous studies (Reddi et al. 2020 Lin
for image classification tasks. As we discussed in Section 3.2
et al. 202T).
Here, we conduct further experiments to explore the impact of the large number of local epochs on FedGLAD. We perform experiments on CIFAR-10 and MNIST with a 0.1 and the local epochs also bring improvement almost all settings. Our explanation is, in this case,
10 and 20. As we can
see from the Table 10 FedGLAD the negative effect of bad sampling results will be further amplified, and GSI can detect this anomaly and decrease the learning rate in time. Also, during training we find that the accuracy curves (also the loss curves) vary greatly when E' = 10 or E = 20, indicates that they may not be proper choices in ours settings, and E = 5 can guarantee the stable training.
Downloaded from http://hhpssanfcccleeu/pppsggggs
23

---

Published in Transactions on Machine Learning Research (05/2023)
B' Conv. Layer with å 0,9 8 Layer with f 0.99 GSE Cam: Layer
B Loyerwith B 0.9
B Carv. Layerwoth B 0.99
(.51) Canv. Layer
Value
Value
10
15
20
30
45
50
10
13
20
30
50
Communication Round
Communication Round
(4) = 0.1.
(h) a 1.0.
Figure 9: The values of the true GSI and the estimator B in each round under different settings on MNIST with FedAvg.
O The effect of using B to estimate GSI
In Eq. [2). in order to deal with unknown GSL we choose to estimate with the exponential average of historical GSI Here, we conduct extra experiments on MNIST explore the accuracy of this estimation. Specifically, on MNIST dataset, calculate and display the true GSI and the estimator B randomly selected weight matrix in the convolutional layer under different non-i.i.d. degrees and different B. The results in Figure We can draw following conclusions:
(I) First, the estimation in Eq. 12 with 3 = 0.9 can properly refeect the patterns of true GSI under different data distributions, validating the effectiveness our solution. (2) Second, the overall trend of GST is increasing and then becoming stable around Choosing 33 closer to (i.e., B = 0.99) will make the estimator fall much behind in fitting the pattern of GST Also, in our preliminary experiments, we find choosing 3 = 0.9 brings more improvement than choosing 3 0.99.
P The visualization of the patterns of server learning rates adapted by FedGLAD
Here, we give an example on MNIST and CIFAR-10 to show the patterns of server learning rates adapted by FedGLAD. For simplicity, we visualize the patterns of GST / while ignoring the constant 7/-- We also plot the lower bound line - yt and upper bound line + yt for comparisons. The results are Figure[1O] (1) First, as we can see, the adapted server rates fluctuate more greatly on a 0.1 than a 1.0, which is because when the data is more non-i.i.d. the impact of client sampling becomes greater. (2) Second, at the beginning of training, the adapted learning rate will be more likely to be set to lower upper bounds than that when the training becomes stable (this phenomenon is more obvious when non-i.i.d. degree is greater). This is because at the initial training stage, B is not stable due to limited samples of historical GSI. This also verifies the necessity of setting dynamic bounds to stabilize training. (3) Finally, GST B' always fluctuates around its expectation 1.0, indicating the server learning rates will not appear an increasing trend during training, which guarantees convergence our method.
Q Combining FedGLAD with weighted averaging-based methods
In Eq. assume the local gradients aggregated in an arithmetic averaging manner. That is because
stated in Section 3.2] this paper focuses on the first optimization target of Eq. ( while assuming the direction of g is
decided by baseline methods. Thus, we follow existing studies (Karimireddy et al., 2020 Reddi et al. 2020) to assume the local clients will perform the same number local updates before uploading local gradients, then the weighted averaging is equal to the arithmetic averaging.
However, our method will not modify the aggregation phase, because our method only adapts the learning rates after the aggregated gradients obtained. Thus, method is still applicable when the aggregation is weighted
24

---

Published in Transactions on Machine Learning Research (05/2023)
40
40
Mices1R Crere Waggeeeden vieu
Adguoe Weight or Adgiaal LK Wiggle vice ot 1.0
Ai Ohes Wegis vae dhe
25
Boandl
25
20
Ne.dlepaiee
2.0
No Ne.Aleiatto
Vallo
Value
0.5
0.5
0.0
0.0
-1.
-15
-20
-20
Communication Round
Communic ation Round
(a) Results random weight matrix in convolutional layer MNIST. (b) Results the random batch normalization layer CIFAR-
Figure 10: The visualizations of the patterns of GST and how frequently it is set the lower upper bounds.
Table 11: The results of combining FedGLAD with IDA Yeganch et (2020a) on different datasets with a 0.1.
Table 12: Results (Accuracy) on FEMNIST and StackOverflow with a 0.1.
Method
CIFAR-10
MNIST
20NewsGroups
AGNews
Method
FEMNIST
StackOverflww
IDA
31.47
78.28
69.02
76.61
FedAvg
81.14 81.47
19.27
+ FedGLAD
33.39
79.36
69.34
79.28
FedGLAD
19.36
averaging. For example, the weighted averaging methods aim to calculate g' Σ oig' where Σ 0r =
That
kESt
kESt
is, setting larger alpha for more reliable local gradient g In this case, also calculate the corresponding GSI
GSI* (kESS |la [?). Then, we can successfully apply FedGLAD.
We then conduct an extra experiment based on weighted averaging baseline IDA Yeganch et al. (2020a) which give dynamic aggregation weight for each local gradient based on its distance to the arithmetic averaged server gradient. The results are in Table II As we can see,our method can also bring improvement when the aggregation is weighted averaging. Also, find that IDA underperforms FedAvg on some datasets, indicating that empirically adjusting the directions of server gradients information may bring benefits.
R Evaluation on the model's fairness
As mentioned above, our method will not affect the process before the aggregation procedure, which means the local gradients will be aggregated following the same way as the baseline method does, while our method only adapt the learning rates for the aggregated gradients. This ensures that the weight for each local gradient in the gradient averag- ing is calculated exactly following previous basclines, and our method itself will not further decrease the aggregation weights for local gradients that are not aligned with the majority. We then report the model's performance on MNIST each client's local dataset to explore the model's faimess. The baseline method is FedAvg, and we report the per- formance improvement (%) after applying FedGLAD. The results are in Figure IT The conclusion is, our method can consistently improve model's performance on most clients" local data, while only cause a small degree of performance degradation on few clients.
S Experiments on realistic federated datasets
Here, we conduct experiments on two realistic federated datasets: Federated EMNIST (Cohen et al. 2017 Reddi et al. 2020) (FEMNIST) and StackOverflow FEMNIST is federated and extended MNIST dataset collected from totally
Downloaded from here
25

---

Published in Transactions on Machine Learning Research (05/2023)
Test Impovement
91
Client Index
Figure 11: The improvement brought by applying FedGLAD over FedAvg on each client's local data MNIST. Our method consistently brings improvement most clients' local data.
Table 13: Results on each seed under a = 0.1.
Table 14: Results on each seed under a 1.0.
Random Sced
Random Sced
Dataset
Method
Datasel
Method
Seed1
Seed2
Seed3
Seed1
Seed2
Seed3
FedAvg
37.08
33,90
34.91
FedAvg
73.20
70.70
72.36
FadGLAD
39.49
38.02
37.82
FddGLAD
74.07
71.91
72.39
CIFAR-10
CIFAR-10
FedAvgM
47.24
49.38
FedAvgM
72.18
72.44
73.26
FedGLAD
49.40
51.91
FedGLAD
72.70
73.18
73.32
FedAvg
78.29
77.77
78.44
FedAvg
91.48
91.30
91.49
FedGLAD
79.20
79.75
80.19
FedGLAD
91.70
91.55
91.65
MNIST
MNIST
FedAvgM
80.68
82.65
81,79
FedAvgM
93.55
93.52
93.04
FedGLAD
81.68
82.84
81.81
TedGLAD
94.06
94.17
93.67
FedAvg
72.72
72.96
72.84
FedAvg
76.57
76.35
77.06
FedGLAD
73.13
73.22
73.06
FedGLAD
77.53
76.74
77.37
20NewsGroups
20NewsCroups
FedAvgM
75.39
75.38
75.56
FedAvgM
79.51
79.20
79.82
FedGLAD
76.02
76.12
75.58
FedGLAD
79.80
79.68
79.83
FodAvg
78.95
79.78
79.72
FedAvg
91.26
91.31
91.23
FedGLAD
79.83
81.29
81.11
FedGLAD
91.38
91.41
91.34
AGNews
AGNews
FedAvgM
82.25
79.67
78.91
FedAvgM
90.28
89.77
90.17
FaGGLAD
84.22
84.14
82.67
FeGGLAD
90.82
89.99
90.53
3400 clients. It has 62 labels, and contains 671,585 training samples and 77,483 testing samples. StackOverflow is
Next Sentence Prediction task in which 135,818, 18, training samples are collected from 342,477 users and 16.586,035 testing samples are collected from 204,088 users. They are widely used in previous studies (Reddi et al.) 2020 He
ct al. 2020), and more detailed descriptions them can be found in Reddi et al. (020)) We adopt the same neural network used in MNIST dataset for FEMNIST, and the same LSTM network in Reddi et al (2020) for StackOverflow For two datasets, we randomly sampled 20 clients each round, the local batch size is 16, and the local optimizers are both SGDM. The local learning rate is 0.01 for FEMNIST and 0.1 for StackOverflow. The server learning rate is 1.0 for FEMNIST and 0.5 for StackOverflow. The number of local epochs is 5 for FEMNIST and for StackOverflow. The total number of communication rounds are 500 and 2500 for FEMNIST and StackOverflow respectively. The results are in Table 12. As we can see, our method can also improve the global model's performance on the realistic partitioned datasets, which further strengthens the empirical effectiveness of our method.
T Results on each seed of each experiment
As we mentioned in our paper in the experiment analysis section (Section the standard deviation in each ex- periment in Table and Table 7 may seems a bit large compared with the improvement gap. However, we display the result on each seed of our method and basclines (take FedAvg and FedAvgM as examples) in the Table [ and Table [4] we point out that actually on almost each seed of one specific experiment, applying our method can gain the improvement over the corresponding baseline. The large standard deviation the baseline results are mainly caused by the fact that the large non-i.i.d. degree makes the baseline training sensitive the random seeds.
26

---

Published in Transactions on Machine Learning Research (05/2023)
Table 13: The best for FedProx we tuned in each data partition setting.
CIFAR-10
MNIST
20NewsGroups
AGNews
Method
0.1
1.0
0.1
1.0
0.1
1.0
0.1
1.0
FedProx
0.1
0.1
1.0
0.1
0.001
0.001
1.0
0.001
Table 16: Local training parameeas for each dataset.
Dataset
Learning Rate
Batch Size
Client Optimizer
Local Epachs
CIFAR-10
0.1
64
SGDM
5
MNIST
0.01
64
SGDM
20NewsGroups
5e-5
32
Adam
AGNews
2e-5
32
Adam
Table 17: The server learning rate search grids for each baseline method.
Methods
Datasets
Search Grids
(CIFAR-10, MNIST)
{0.5, 1.0, 2.0, ... 10.03
[FedAvg, FedProx, SCAFFOLD) (20NewsGroups, AGNews]
{0.5, 1,0, 2.0, 10.0)
(CIFAR-IO, MNIST)
(0.05, 1.05,0.10..2.-..... 0.5, 1.0, ..0. 550)
FedAvgM
(20NewsGroups, AGNews)
0,05,0.1.0.2.10
0.5, 1.0, 5.0}
(CIFAR-10, MNIST)
(1e-32e-3, Se-3,1e-2222222.. 5e-2]
FedAdam
(20NewsGroups, AGNews)
(le-5,ee55, (..52226...,5555...44...26
,5e-5.1c-422-4.
5c-5)
Table 18: The optimal server learning rates for each baseline method in our experiments.
CIFAR-10
MNIST
20NewsGroups
AGNews
Method
0.1
a
1.0
0.1
a
1.0
0.1
a
1.0
0.1
1.0
FedAvg
0.5
0.5
1.0
1.0
7.0
6.0
1.0
1.0
FedProx
0.5
0.5
1.0
1.0
7.0
6.0
1.0
1.0
SCAFFOLD
0.5
0.5
1.0
1.0
7.0
6.0
1.0
1.0
FedAvgM
0.2
0.1
0.5
0.5
2.0
2.0
0.5
0.5
FedAdam
Se-3
5e-3
le-2
1e-2
3e-4
3e-4
2e-4
le-4
Table 19; Training details experiments on BiLSTM.
Local Training
Server Aggregating
Dataset
Optimizer
I.R
Batch Size
Local Epochs
Method
Server LR
Total Rounds
FedAvg FedAvgM
1.0
200
0.01
20NewsGroups
Adam
0.001
32
200
FedAvg
0.5
100
0.02
AGNews
Adam
0.001
32
FedAvgM
0.05
100
0.02
27


Reviews:
[{"review_id": "wvCc5TQKLY", "reviewer": "Reviewer_pBTK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of federated learning for non-iid clients is very important and practical.\n\nThe paper is written well, the idea is stated clearly and experiments seem to demonstrate the effect of combining gradients.\n\nHowever, there are a couple of concerns that complicate the paper:\n\n1. Framing: \"when to trust\" assumes some trust concerns with the gradients. However, there is no discussion on types of issues that could cause a lack of trust (broader impact section mentions that attacker can abuse your method but not trust issues with gradients under existing FL). I doubt that example: `local samples from all selected clients are almost all cat images, the averaged gradient’s direction deviates far away from the ideal server gradient averaged by all clients’ local gradients. In this case, we may not trust the aggregated gradient` is the right definition of the word \"trust\" as the gradients are themselves \"trustworthy\", e.g. produced by trusted parties (per your threat model), and aggregation is also performed correctly.  Overall, if the authors aim to explore \"trust\" in FL there should be more discussion on security aspects of FL. \n\n2.  The biggest problem I suspect is the lack of discussion on the model's fairness --> specifically, how the co-aligned gradients affect participants with gradients that are not aligned with the majority? I suspect that the proposed method will discriminate underrepresented participants with dissimilar weights greatly affecting usability of the model. \n\n3. I suspect that the used datasets might be the reason for the second issue: I don't think they are \"non-iid\" and are simply derived by taking iid dataset and sampling classes with Dirichlet. CIFAR-10 was used extensively in FL before, however I doubt that it can be used to address \"non-iid\"-specific issues. As the field of FL is already mature there exist multiple datasets that are non-iid, i.e. created by different users, for example StackOverflow or Reddit. Seeing more performance on those datasets and fairness metrics, i.e. performance on individual users, should greatly help. \n\n4. Lastly, the major practical shortcoming of the method is the reliance on computing the GSI on the server. Access to user's gradients can reveal user data. Therefore, user privacy depends on the server not accessing user gradients, e.g. Secure Aggregation[1] is de-facto a standard in practical deployments to protect individual contributions. Proposed method directly violates this privacy premise but does not present any reasoning on why it might be acceptable to users.\n\n\n\n[1]Bonawitz, Keith, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. \"Practical secure aggregation for privacy-preserving machine learning.\" In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191. 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method to mitigate distribution differences between clients in federated learning case. It demonstrates the effect of ensuring similarity between weight updates by using Gradient Similarity–aware Indicator and being able to achieve the significantly better performance. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above points.", "broader_impact_concerns": "The broader impact section mentions method[1] that will go directly against the proposed GSI. The Krum method in that paper [1] does not \"filter\" but select individual vector that is aligned with majority. This approach would need a more complex discussion on combining with GSI method and I wonder if mentioning [1] is necessary at all in that section. \n\n[1] Blanchard, Peva, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. \"Machine learning with adversaries: Byzantine tolerant gradient descent.\" Advances in neural information processing systems 30 (2017).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e0ZAH3Qhyc", "reviewer": "Reviewer_JJXF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper studies an interesting perspective on tackling heterogeneity in federated learning.\n- The per-layer adaptive learning rate looks interesting.\n- The empirical performance seems strong.\n\nWeaknesses:\n- I'm concerned with the convergence of the proposed method. It seems that $B^t$ is exponential in $t$ and $\\beta$ is a positive constant smaller than 0, which indicates $B^t$ is diminishing given that GSI does not change too much as $t$ grows (which seems to be the case given the observation in Figure 4). However, this implies $\\tilde{\\eta}^t$ is increasing in $t$ unless GSI grows at the same rate as $B^t$ does w.r.t $t$, which does not seem to be the case from Figure 4. Hence, Algorithm 1 is essentially saying that we should use increasing global learning rate in order to get better performance. I'm curious on how this lead to convergence of the algorithm. In addition, it is also not clear to me that in practice, how frequently is $\\tilde{\\eta}^t$ set to GSI/B^t and how frequently it is set to $1+\\gamma t$, both of which are increasing in $t$.\n- The assumption of average grad norm over a mini-batch is approximately the same as the average grad norm over the full batch seems to strong to me. Could the authors provide some justifications on why this is true?\n- Figure 5 presents a positive correlation between GSI and the cosine similarity between full batch data proportion distribution and mini batch data proportion distribution. This makes me wonder why the authors choose to take the average rather than taking the weighted average in Line 8 of the algorithm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose a new method to address negative client sampling in federated learning based on adaptively scaling the server learning rate. Empirical results on four real world datasets have shown that the proposed FedGlad is able to achieve better performance compared to baseline methods like FedAvg and FedAdam.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the weaknesses part, I would like to see the following explanations / revisions:\n- Convergence analysis of the proposed solver and how frequently the learning rate is set to be $1+\\gamma t$. Explanation on why increasing learning rate makes sense here.\n- Justification on the assumption mentioned in the second weakness point.\n- Additional experiments where aggregation rule is weighted average.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VoF6AcPiNS", "reviewer": "Reviewer_zTsX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1) The paper provides sufficient theoretical derivation. The proposed indicator is meaningful that reflects the similarity between client gradients.\n\n2) The experiment results show that FedGLAD improves the accuracy compared to the baseline methods. \n\n3) This paper is well written.\n\nWeaknesses\n1) The best bounding factor γ is unknown for different training settings. The uncertainty of γ  weakens the practicality of the proposed method.\n\n2) In section 3.3, is the estimate of equation 11 solid in all cases? Will the quality and distribution of data have a greater impact on it? How to choose a trade-off parameter β?\n\n3) This method seems weak against perturbed client data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a learning rate adaptation mechanism to mitigate the negative impact of client sampling of non-i.i.d. in federated learning (FedGLAD). The authors find a Gradient Similarity-aware Indicator (GSI) indicator to reflect the similarity between client gradients. The FedGLAD utilized GSI to dynamically adjust the server learning rates. \nThe proposed FedGLAD is supported by theoretical derivations and is evaluated in various experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness above. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed a server-side learning rate adaptation scheme for federated learning and conducted extensive experiments to verify their claims. The reviewers posed some excellent questions that the authors were able to largely address during the revision. Overall, the proposed server-side learning rate adaptation seems to be a simple implementation trick that many in FL may find useful. 

In the final revision, please consider proofreading the entire manuscript and incorporating the following further comments.

-- Abstract: refrain from using "optimal server learning rate" when the precise meaning or significance of optimality is not clear. 

-- Eq (9) can be rewritten as sqrt{ var(g) / mean(g)^2 + 1} (recall that EX^2 = Var(X) - [EX]^2), which is lower bounded by 1 (attained when gradient variance is 0, e.g., sampled clients are homogeneous). Rewriting GSI in terms of variance (instead of second moment) might be more self-explanatory and I would encourage the authors to consider adding some discussions along this line. 

-- Looking back at Eq (5) and consider two scenarios where the mean g^t is fixed but the variance (among the sampled clients) is different. Eq (5) would not distinguish the two scenarios while the authors scheme would assign bigger eta_t for the more noisy scenario. Is this difference necessary, or simply an artifact of the approximations made in the following analysis? 

-- There is also concern that FEMNIST may not be (sufficiently) representative for the non-iid setting. If the authors could add some experiments on StackOverflow or Reddit, that'll further strengthen the empirical results. 

Decision: Accept (poster)