Year: 2020

Abstract:
Federated averaging (FedAvg), despite its simplicity, has been the main approach in training neural networks in the federated learning setting. In this work, we show that the algorithmic choices of the FedAvg algorithm correspond to optimizing a single objective function that involves the global and all of the shard specific models using a hard version of the well known Expectation-Maximization (EM) algorithm. As a result, we gain a better understanding of the behavior and design choices of federated averaging while being able to provide interesting connections to recent literature. Based on this view, we further propose FedSparse, a version of federated averaging that employs prior distributions to promote model sparsity. In this way, we obtain a procedure that leads to reductions in both server-client and client-server communication costs as well as more efficient models.

heading: 1 INTRODUCTION
text: Smart devices have become ubiquitous in today’s world and are generating large amounts of potentially sensitive data. Traditionally, such data is transmitted and stored in a central location for training machine learning models. Such methods rightly raise privacy concerns and we seek the means for training powerful models, such as neural networks, without the need to transmit potentially sensitive data. To this end, Federated Learning (FL) (McMahan et al., 2016) has been proposed to train global machine learning models without the need for participating devices to transmit their data to the server. The Federated Averaging (FedAvg) (McMahan et al., 2016) algorithm communicates the parameters of the machine learning model instead of the data itself, which is a more private means of communication.
The FedAvg algorithm was originally proposed through empirical observations. While it can be shown that it converges (Li et al., 2019), its theoretical understanding in terms of the model assumptions as well as the underlying objective function is still not well understood. The first contribution of this work improves our understanding of FedAvg; we show that FedAvg can be derived by applying the general Expectation-Maximization (EM) framework to a simple hierarchical model. This novel view has several interesting consequences: it sheds light on the algorithmic choices of FedAvg, bridges FedAvg with meta-learning, connects several extensions of FedAvg and provides fruitful ground for future extensions.
Apart from theoretical grounding, the FL scenario poses several practical challenges, especially in the “cross-device” setting (Kairouz et al., 2019) that we consider in this work. In particular, communicating model updates over multiple rounds across a large amount of devices can incur significant communication costs. Communication via the public internet infrastructure and mobile networks is potentially slow and not for free. Equally important, training (and inference) takes place on-device and is therefore restricted by the edge-devices’ hardware constraints on memory, speed and heat dissipation capabilities. Therefore, jointly addressing both of these issues is an important step towards building practical FL systems, as also discussed in Kairouz et al. (2019).
Through the novel EM view of FedAvg that we introduce, we develop our second contribution, FedSparse. FedSparse allows for learning sparse models at the client and server via a careful choice of priors within the hierarchical model. As a result, it tackles the aforementioned challenges, since it can simultaneously reduce the overall communication and computation at the client devices. Empirically, FedSparse provides better communication-accuracy trade-offs compared to both FedAvg as well as methods proposed for similar reasons (Caldas et al., 2018).
2 FEDAVG THROUGH THE LENS OF EM
The FedAvg algorithm is a simple iterative procedure realized in four simple steps. At the beginning of each round t, the server communicates the model parameters, let them be w, to a subset of the devices. The devices then proceed to optimize w, e.g., via stochastic gradient descent, on their respective dataset via a given loss function
Ls(Ds,w) := 1
Ns Ns∑ i=1 L(Dsi,w) (1)
where s indexes the device, Ds corresponds to the dataset at device s and Ns corresponds to its size. After a specific amount of epochs of optimization on Ls is performed, denoted as E, the devices communicate the current state of their parameters, let it be φs, to the server. The server then performs an update to its own model by simply averaging the client specific parameters wt = 1S ∑ s φs.

heading: 2.1 THE CONNECTION TO EM
text: We now ask the following question; does the overall algorithm correspond to a specific optimization procedure on a given objective function? Let us consider the following objective function:
argmax w
1
S S∑ s=1 log p(Ds|w), (2)
where Ds corresponds to the shard specific dataset that has Ns datapoints, p(Ds|w) corresponds to the likelihood of Ds under the server parameters w. Now consider decomposing each of the shard specific likelihoods as follows:
p(Ds|w) = ∫ p(Ds|φs)p(φs|w)dφs, p(φs|w) ∝ exp ( −λ 2 ‖φs −w‖2 ) , (3)
where we introduced the auxiliary latent variables φs, which are the parameters of the local model at shard s. The server parameters w act as “hyperparameters” for the prior over the shard specific parameters and λ acts as a regularization strength that prevents φs from moving too far from w.
How can we then optimize the resulting objective in the presence of these latent variables φs? The traditional way to optimize such objectives is through Expectation-Maximization (EM). EM consists of two steps, the E-step where we form the posterior distribution over these latent variables
p(φs|Ds,w) = p(Ds|φs)p(φs|w)
p(Ds|w) , (4)
and the M-step where we maximize the probability of Ds w.r.t. the parameters of the model w by marginalizing over this posterior
argmax w
1
S ∑ s Ep(φs|Ds,wold)[log p(Ds|φs) + log p(φs|w)]
= argmax w
1
S ∑ s Ep(φs|Ds,wold)[log p(φs|w)]. (5)
If we perform a single gradient step for w in the M-step, this procedure corresponds to doing gradient ascent on the original objective, a fact we show in Appendix D.
When posterior inference is intractable, hard-EM is sometimes employed. In this case we make “hard” assignment for the latent variables φs in the E-step by approximating p(φs|Ds,w) with its most probable point, i.e.
φ∗s = argmax φs p(Ds|φs)p(φs|w) p(Ds|w) = argmax φs log p(Ds|φs) + log p(φs|w). (6)
This is usually easier to do as we can use techniques such as stochastic gradient ascent. Given these hard assignments, the M-step then corresponds to another simple maximization
argmaxw 1 S ∑ s log p(φ ∗ s|w). As a result, hard-EM corresponds to a block coordinate ascent type of algorithm on the following objective function
argmax φ1:S ,w
1
S ∑ s (log p(Ds|φs) + log p(φs|w)), (7)
where we alternate between optimizing φ1:S and w while keeping the other fixed.
How is this framework the same as FedAvg? By letting λ → 0 in Eq. 3 it is clear that the hard assignments in the E-step mimic the process of optimizing a local model on the data of each shard. In fact, even by optimizing the model locally with stochastic gradient ascent for a fixed number of iterations with a given learning rate we implicitly assume a specific prior over the parameters; for linear regression, this prior is a Gaussian centered at the initial value of the parameters (Santos, 1996) whereas for non-linear models it bounds the distance from the initial point. After obtaining φ∗s the M-step then corresponds to argmaxw Lr := 1S ∑ s− λ 2 ‖φ ∗ s −w‖2, and we can easily find a closed form solution by setting the derivative of the objective w.r.t. w to zero and solving for w: ∂Lr ∂w = 0⇒ λ S ∑ s (φ∗s −w) = 0⇒ w = 1 S ∑ s φ∗s. (8) It is easy to see that the optimal solution for w given φ∗1:S is the same as the one from FedAvg. Of course, FedAvg does not optimize the local parameters φs to convergence at each round, so one might wonder whether the correspondence to EM is still valid. It turns out that the alternating procedure of EM corresponds to block coordinate ascent on a single objective function, the variational lower bound of the marginal log-likelihood (Neal & Hinton, 1998) of a given model. More specifically for our setting, we can see that the EM iterations perform block coordinate ascent to optimize:
argmax w1:S ,w
1
S ∑ s Eqws (φs) [ log p(Ds|φs) + log p(φs|w)] +H[qws(φs)] (9)
where ws are the parameters of the variational approximation to the posterior distribution p(φs|Ds,w) and H[q] corresponds to the entropy of the q distribution. To obtain the procedure of FedAvg we can use a (numerically) deterministic distribution for φs, qws(φs) := N (ws, I). This leads us to the same objective as in Eq. 7, since the expectation concentrates on a single term and the entropy of qws(φs) becomes a constant independent of the optimization. In this case, the optimized value for φs after a fixed number of steps corresponds to the ws of the variational approximation.
It is interesting to contrast recent literature under the lens of this framework. Optimizing the same hierarchical model with hard-EM but with a non-trivial λ results into the same procedure that was proposed by Li et al. (2018). Furthermore, using the difference of the local parameters to the global parameters as a “gradient” (Reddi et al., 2020) is equivalent to hard-EM on the same model where in the M-step we take a single gradient step. In addition, this view makes precise the idea that FedAvg is a meta-learning algorithm (Jiang et al., 2019); the underlying hierarchical model we optimize is similar to the ones used in meta-learning (Grant et al., 2018; Chen et al., 2019).
How can we then use this novel view of FedAvg to our advantage? The most straightforward way is to use an alternative prior which would result into different behaviours in local training and server side updating. For example, one could use a Laplace prior, which would result into the server selecting the median instead of averaging, or a mixture of Gaussians prior, which would result into training an ensemble of models at the server. In order to tackle the communication and computational costs, which is important for “cross-device” FL, we chose a sparsity inducing prior, namely the spike and slab prior. We describe the resulting algorithm, FedSparse, in the next section.
3 THE FEDSPARSE ALGORITHM: SPARSITY IN FEDERATED LEARNING
Encouraging sparsity in FL has two main advantages; the model becomes smaller and less resource intensive to train and furthermore, it cuts down on communication costs as the pruned parameters do not need to be communicated. The golden standard for sparsity in probabilistic models is the spike and slab (Mitchell & Beauchamp, 1988) prior. It is a mixture of two components, a delta spike at zero, δ(0), and a continuous distribution over the real line, i.e. the slab. More specifically, by adopting a Gaussian slab for each local parameter φsi we have that
p(φsi|θi,wi) = (1− θi)δ(0) + θiN (φsi|wi, 1/λ), (10)
or equivalently as a hierarchical model p(φsi|θi,wi) = ∑ zsi p(zsi|θi)p(φsi|zsi,wi), p(zsi) = Bern(θi), (11)
p(φsi|zsi = 1,wi) = N (φsi|wi, 1/λ), p(φsi|zsi = 0) = δ(0), (12) where zsi plays the role of a “gating” variable that switches on or off the parameter φsi. Now consider using this distribution for the prior over the parameters in the federated setting. w, θ will be the server side model weights and probabilities of the binary gates. In order to stay close to the FedAvg paradigm of simple point estimation and since approximate inference for complex posteriors, such as those that arise in neural networks, is still an open problem, we will perform hard-EM in order to optimize w,θ. By using approximate distributions qws(φs|zs), qπs(zs), the variational lower bound for this model becomes
argmax w1:S ,w,π1:S ,θ
1
S ∑ s Eqπs (zs)qws (φs|zs) [ log p(Ds|φs) + log p(φs|w, zs) + log p(zs|θ)
− log qws(φs|zs) ] +H[qπs(zs)]. (13)
For the shard specific weight distributions, as they are continuous, we will use qws(φsi|zsi = 1) := N (wsi, ), q(φsi|zsi = 1) := N (0, ) with ≈ 0 which will be, numerically speaking, deterministic. For the gating variables, as they are binary, we will use qπsi(zsi) := Bern(πsi) with πsi being the probability of activating local gate zsi. In order to do hard-EM for the binary variables, we will remove the entropy term for the qπs(zs) from the aforementioned bound as this will encourage the approximate distribution to move towards the most probable value for zs. Furthermore, by relaxing the spike at zero to a Gaussian with precision λ2, i.e., p(φsi|zsi = 0) = N (0, 1/λ2), and by plugging in the appropriate expressions into Eq. 13 we can show that the local and global objectives will be
argmax ws,πs Ls(Ds,w,θ,ws,πs) := Eqπs (zs) [ Ns∑ i L(Dsi,ws zs) ] − λ 2 ∑ j πsj(wsj −wj)2
− λ0 ∑ j πsj + ∑ j (πsj log θj + (1− πsj) log(1− θj)) + C, (14)
argmax w,θ L := 1 S S∑ s=1 Ls(Ds,w,θ,ws,πs) (15)
respectively, where λ0 = 12 log λ2 λ and C is a constant independent of the variables to be optimized. The derivation can be found at Appendix E. It is interesting to see that the final objective at each shard intuitively tries to find a trade-off between four things: 1) explaining the local dataset Ds, 2) having the local weights close to the server weights (regulated by λ), 3) having the local gate probabilities close to the server probabilities and 4) reducing the local gate activation probabilities so as to prune away a parameter (regulated by λ0). The latter is an L0 regularization term, akin to the one proposed by Louizos et al. (2017).
Now let us consider what happens at the server after the local shard, through some procedure, optimized ws and πs. Since the server loss for w,θ is the sum of all local losses, the gradient for each of the parameters will be
∂L ∂w = ∑ s λπs(ws −w), ∂L ∂θ = ∑ s ( πs θ − 1− πs 1− θ ) . (16)
Setting these derivatives to zero, we see that the stationary points are
w = 1∑ j πj ∑ s πsws, θ = 1 S ∑ s πs (17)
i.e., a weighted average of the local weights and an average of the local probabilities of keeping these weights. Therefore, since the πs are being optimized to be sparse through the L0 penalty, the server probabilities θ will also become small for the weights that are used by only a small fraction of the shards. As a result, to obtain the final sparse architecture, we can prune the weights whose corresponding server inclusion probabilities θ are less than a threshold, e.g., 0.1. It should be noted that the sums and averages of Eq. 16, 17 respectively can be easily approximated by subsampling a small set of clients S′ from S. Therefore we do not have to consider all of the clients at each round, which would be prohibitive for the “cross-device” setting of FL.

heading: 3.1 REDUCING THE COMMUNICATION COST
text: The framework described so far allow us to learn a more efficient model. We now discuss how we can use it in order to cut down both download and upload communication costs during training.
Reducing client to server communication cost In order to reduce the client to server cost we will communicate sparse samples from the local distributions instead of the distributions themselves; in this way we do not have to communicate the zero values of the parameter vector. This leads to large savings, while still keeping the server gradient unbiased. More specifically, we can express the gradients and stationary points for the server weights and probabilities as follows
∂L ∂w = ∑ s λEqπs (zs) [zs(ws −w)] , w = Eqπ1:S (z1:S) [ 1∑ j zj ∑ s zsws ] , (18)
∂L ∂θ = ∑ s Eqπs (zs) [ zs θ − 1− zs 1− θ ] , θ = 1 S ∑ s Eqπs (zs) [zs] . (19)
As a result, we can then communicate from the client only the subset of the local weights ŵs that are non-zero in zs ∼ qπs(zs), ŵs = ws zs, and the server can infer the state of zs by inspecting which parameters were omitted. Having access to those samples, the server can then form 1-sample stochastic estimates of either the gradients or the stationary points for w,θ. Notice that this is a way to reduce communication without adding bias in the gradients of the original objective. In case that we are willing to incur extra bias, we can further use techniques such as quantization (Amiri et al., 2020) and top-k gradient selection (Lin et al., 2017) to reduce communication even further. Such approaches are left for future work.
Reducing the server to client communication cost The server needs to communicate to the clients the updated distributions at each round. Unfortunately, for simple unstructured pruning, this doubles the communication cost as for each weight wi there is an associated θi that needs to be sent to the client. To mitigate this effect we will employ structured pruning, which introduces a single additional parameter for each group of weights. For groups of moderate sizes, e.g., the set of weights of a given convolutional filter, the extra overhead is small. We can also take the communication cost reductions one step further if we allow for some bias in the optimization procedure; we can prune the global model during training after every round and thus send to each of the clients only the subset of the model that has survived. Notice that this is easy to do and does not require any data at the server. The inclusion probabilities θ are available at the server, so we can remove the parameters that have θ less than a threshold, e.g. 0.1. This can lead to large reductions in communication costs, especially once the model becomes sufficiently sparse.
3.2 FEDSPARSE IN PRACTICE
Local optimization While optimizing for ws locally is straightforward to do with gradient based optimizers, πs is more tricky, as the expectation over the binary variables zs in Eq. 14 is intractable to compute in closed form and using Monte-Carlo integration does not yield reparametrizable samples. To circumvent these issues, we rewrite the objective in an equivalent form and use the hard-concrete relaxation from (Louizos et al., 2017), which can allow for the straightforward application of gradient ascent. We provide the details in Appendix F. When the client has to communicate to the server, we propose to form ŵs by sampling from the zero-temperature relaxation, which yields exact binary samples. Furthermore, at the beginning of each round, following the practice of FedAvg, the participating clients initialize their approximate posteriors to be equal to the priors that were communicated from the server. Empirically, we found that this resulted in better global accuracy.
Parametrization of the probabilities Since there have been evidence that such optimization based pruning can be inferior to simple magnitude based pruning (Gale et al., 2019), we take an approach that combines the two and reminisces the recent work of Azarian et al. (2020). We parametrize the probabilities θ,πs as a function of the model weights and magnitude based thresholds that regulate how active a parameter can be. More specifically, we use the following parametrization
θg := σ
( ‖wg‖2 − τg
T
) , πsg := σ ( ‖wsg‖2 − τsg
T
) , (20)
where the subscript g denotes the group, σ(·) is the sigmoid function, τg, τsg are the global and client specific thresholds for a given group g and T is a temperature hyperparameter. Following Azarian et al. (2020) we also “detach” the gradient of the weights through θ,πs, to avoid decreasing the probabilities by just shrinking the weights. With this parametrization we lose the ability to get a closed form solution for the server thresholds, but nonetheless we can still perform gradient based optimization at the server by using the chain rule. For a positive threshold, we use a parametrization in terms of a softplus function, i.e., τ = log(1 + exp(v)) where v is the learnable parameter.
Algorithm 1 The server side algorithm for FedSparse (assuming weight sparsity for simplicity). σ(·) is the sigmoid function, is the threshold for pruning.
Initialize v and w for round t in 1, . . . T do
τ ← log(1 + exp(v)) θ ← σ ((|w| − τ )/T ) w← I[θ > ]w . prune global model Initialize∇tw = 0,∇tv = 0 for s in random subset of the clients do
ŵts ← CLIENT(s,w,v) zs ← I[ŵts 6= 0] ∇tw+ = zs(ŵts −w) ∇tv+ = − (zs(1− θ)− (1− zs)θ)σ(v)/T
end for wt+1,vt+1 ← ADAM(∇tw), ADAMAX(∇tv)
end for
Algorithm 2 The client side algorithm for FedSparse.
Get w,v from the server θ ← σ ((|w| − τ )/T ) ws,vs ← w,v for epoch e in 1, . . . , E do
for batch b ∈ B do τs ← log(1 + exp(vs)) πs ← σ ((|ws| − τs)/T ) Ls ← Ls(b,w,θ,ws,πs) ws ← SGD(∇wsLs) vs ← ADAMAX(∇vsLs)
end for end for πs ← σ ((|ws| − τs)/T ) zs ∼ qπs(zs) return zs ws

heading: 4 RELATED WORK
text: Both contributions of this work share similarities with several recent works in federated learning. FedProx (Li et al., 2018) proposed to add a proximal term to the local objective at each shard, so that it prevents the local models from drifting too far from the global model. Through the EM view of FedAvg we show how such a local objective arises if we use a non-trivial precision for the Gaussian prior over the local parameters. Furthermore, FedAvg has been advocated to be a meta-learning algorithm in Jiang et al. (2019); with the EM view we make this claim precise and show that the underlying hierarchical model that FedAvg optimizes is the same as the models used in several meta-learning works (Grant et al., 2018; Chen et al., 2019). Furthermore, by performing a single gradient step for the M-step in the EM view of FedAvg we see that we arrive at a procedure that has been previously explored both in a meta-learning context with the Reptile algorithm (Nichol et al., 2018), as well as the federated learning context with the “generalized” FedAvg (Reddi et al., 2020). One important difference between meta-learning and FedAvg is that the latter maximizes the average, across shards, marginal-likelihood in order to find the server / global parameters whereas meta-learning methods usually optimize the global parameters such that the finetuned model perform well on the local validation set. Exploring such parameter estimation methods, as, e.g., in Chen et al. (2019), in the federated scenario and how these relate to existing approaches that merge meta-learning with federated learning, e.g., Fallah et al. (2020), is an interesting avenue for future work. Finally, the EM view we provide here for FedAvg can also provide a novel perspective for optimization works that improve model performance via model replicas (Zhang et al., 2019; Pittorino et al., 2020).
Reducing the communication costs is a well known and explored topic in federated learning. FedSparse has close connections to federated dropout (Caldas et al., 2018), as the latter can be understood via a similar hierarchical model, where gates z are global and have a fixed probability θ for both the prior and the approximate posterior. Then at each round and for each client, the server samples these gates (due to the expectation), collects the submodels and sends those to be optimized at the shard. Compared to federated dropout, FedSparse allow us to optimize the dropout rates to the data, such that they satisfy a given accuracy / sparsity trade-off, dictated by the hyperparameter λ0. Another benefit we gain from the EM view of FedAvg is that it makes it clear that the server
can perform gradient based optimization. As a result, we can harvest the large literature on efficient distributed optimization (Lin et al., 2017; Bernstein et al., 2018; Wangni et al., 2018; Yu et al., 2019), which involves gradient quantization, sparsification and more general compression. On this front, there have been also other works that aim to reduce the communication cost in FL via such approaches (Sattler et al., 2019; Han et al., 2020). In general, such approaches can be orthogonal to FedSparse and exploring how they can be incorporated is a promising avenue for future research.

heading: 5 EXPERIMENTS
text: We verify in three tasks whether the FedSparse procedure leads to similar or better global models compared to FedAvg while providing reductions in communication costs and efficient models. As a baseline that also reduces communication costs by sparsifying the model, we consider the federated dropout procedure from Caldas et al. (2018), which we refer to as FedDrop. For each of the three tasks, we present the results for FedSparse with regularization strengths that target three sparsity levels: low, mid and high. For the FedDrop baseline, we experiment with multiple combinations of dropout probabilities for the convolutional and fully connected layers. For each of these tasks we report the setting that performs best in term of accuracy / communication trade-off.
The first task we consider is a federated version of CIFAR10 classification; we partition the data among 100 shards in a non-i.i.d. way by following Hsu et al. (2019). For the model we employ a LeNet-5 convolutional architecture (LeCun et al., 1998) with the addition of dropout(0.1) for the second convolutional layer and dropout(0.3) for the first fully connected layer in order to prevent overfitting locally at the shard. We optimize the model for 1k communication rounds. For the second task we consider the 500 shard federated version of CIFAR100 classification from Reddi et al. (2020). For the model we use a ResNet20, where we replace the batch normalization layers with group normalization, following Reddi et al. (2020), and we optimize for 6k rounds. For the final task we considered the non-i.i.d. Femnist classification and we use the same configuration as CIFAR10 but we optimize the model for 6k rounds. More details can be found at Appendix A.
We evaluate FedSparse and the baselines on two metrics that highlight the tradeoffs between accuracy and communication costs. On both metrics the x-axis represents the total communication cost incurred up until that point and the y-axis represents two distinct model accuracies. The first one corresponds to the accuracy of the global model on the union of the shard test sets, whereas the second one corresponds to the average accuracy of the shard specific “local models” on the shard specific test sets. The “local model” on each shard is the model configuration that the shard last communicated to the server, and serves as a proxy for the personalized model performance on each shard. The later metric is motivated from the meta-learning (Jiang et al., 2019) and EM view of federated averaging, and corresponds to using the local posteriors for prediction on the local test set instead of the server side priors.

heading: 5.1 EXPERIMENTAL RESULTS
text: The results from our experiments can be found in the following table and figures. Overall, we observed that the FedSparse models achieve their final sparsity ratios early in training, i.e., after 30-50 rounds, which quickly reduces the communication costs for each round (Appendix B).
(a) CIFAR 10 global acc. (b) CIFAR 10 local acc. We can see that for CIFAR 10, the FedSparse models with medium (∼45%) and high (∼62%) sparsity outperform all other methods for small communications budgets on the global accuracy front, but are eventually surpassed by FedDrop on higher budgets. However, on the local accuracy front, we see that the FedSparse models Pareto dominate both baselines, achieving, e.g., 87% local accuracy with 43% less communication compared to FedAvg. Overall, judging the final performance only, we see that FedDrop reaches the best accuracy on the global model, but FedSparse reaches the best accuracy in the local models.
(a) CIFAR 100 global acc. (b) CIFAR100 local acc.
On CIFAR 100, the differences are less pronounced, as the models did not fully converge for the maximum number of rounds we use. Nevertheless, we still observe similar patterns; for small communication budgets, the sparser models are better for both the global and local accuracy as, e.g., they can reach 32% global accuracy while requiring 13% less communication than
FedAvg.
(a) Femnist global acc. (b) Femnist local acc. Finally for Femnist we observe the most differences, as the Femnist task is also more communication intensive due to having 3.5k shards. We see that the FedSparse algorithm Pareto dominates both FedDrop and FedAvg and, more specifically, in the high sparsification setting, it can reach 84% global accuracy and 89% local accuracy while requiring 41% and 51% less communication compared to FedAvg respectively. Judging by the final accuracy; both FedAvg and FedSparse with the low setting reached similar global and local model performance, which is to be expected given that that particular FedSparse setting lead to only 1% sparsity.

heading: 6 CONCLUSION
text: In this work, we showed how the FedAvg algorithm, the standard in federated learning, corresponds to applying a variant of the well known EM algorithm to a simple hierarchical model. Through this perspective, we bridge several recent works on federated learning as well as connect FedAvg to meta-learning. As a straightforward extension stemming from this view, we proposed FedSparse, a generalization of FedAvg with sparsity inducing priors. Empirically, we showed that FedSparse can learn sparse neural networks which, besides being more efficient, can also significantly reduce the communication costs without decreasing performance.

heading: A EXPERIMENTAL DETAILS
text: For all of the three tasks we randomly select 10 clients without replacement in a given round but with replacement across rounds. For the local optimizer of the weights we use stochastic gradient descent with a learning rate of 0.05, whereas for the global optimizer we use Adam (Kingma & Ba, 2014) with the default hyperparameters provided in (Kingma & Ba, 2014). For the pruning thresholds in FedSparse we used the Adamax (Kingma & Ba, 2014) optimizer with 1e− 3 learning rate at the shard level and the Adamax optimizer with 1e− 2 learning rate at the server. For all three of the tasks we used E = 1 with a batch size of 64 for CIFAR10 and 20 for CIFAR100 and Femnist. It should be noted that for all the methods we performed gradient based optimization using the difference gradient for the weights (Reddi et al., 2020) instead of averaging.
For the FedDrop baseline, we used a very small dropout rate of 0.01 for the input and output layer and tuned the dropout rates for convolutional and fully connected layers separately in order to optimize the accuracy / communication tradeoff. For convolutional layers we considered rates in {0.1, 0.2, 0.3} whereas for the fully connected layers we considered rates in {0.1, 0.2, 0.3, 0.4, 0.5}. For CIFAR10 we did not employ the additional dropout noise at the shard level, since we found that it was detrimental for the FedDrop performance. Furthermore, for Resnet20 on CIFAR100 we did not apply federated dropout at the output layer. For CIFAR10 the best performing dropout rates were 0.1 for the convolutional and 0.5 for the fully connected, whereas for CIFAR100 it was 0.1 for the convolutional. For Femnist, we saw that a rate of 0.2 for the convolutional and a rate of 0.4 for the fully connected performed better.
For FedSparse, we initialized v such that the thresholds τ lead to θ = 0.99 initially, i.e. we started from a dense model. The temperature for the sigmoid in the parameterization of the probabilities was set to T = 0.001. Furthermore, we downscaled the cross-entropy term between the client side
probabilities, πs, and the server side probabilities, θ by mutltiplying it with 1e − 4. Since at the beginning of each round we were always initializing πS = θ and we were only optimizing for a small number of steps before synchronizing, we found that the full strength of the cross-entropy was not necessary. Furthermore, for similar reasons, i.e. we set ws = w at the beginning of each round, we also used λ = 0 for the drift term λ2πsj(ws −w)
2. The remaining hyperparameter λ0 dictates how sparse the final model will be. For the LeNet-5 model the λ0’s we report are {5e− 7, 5e− 6, 5e− 5} for the “low”, “mid” and “high” settings respectively, which were optimized for CIFAR10 and used as-is for Femnist. For CIFAR100 and Resnet20, we did not perform any pruning for the output layer and the λ0’s for the “low”, “mid” and “high” settings were {5e − 7, 5e − 6, 5e − 5} respectively. These were chosen so that we obtain models with comparable sparsity ratios as the one on CIFAR10.

heading: B EVOLUTION OF SPARSITY
text: We show the evolution of the sparsity ratios for all tasks and configurations in the following plot. We can see that in all settings the model attains its final sparsity quite early in training.
(a) CIFAR10 sparsity (b) CIFAR100 sparsity (c) Femnist sparsity

heading: C ADDITIONAL RESULTS
text: Convergence plots in terms of communication rounds. In order to understand whether the extra noise is detrimental to the convergence speed of FedSparse, we plot the validation accuracy in terms of communication rounds for all tasks and baselines. As it can be seen, there is no inherent difference before FedSparse starts pruning. This happens quite early in training for CIFAR 100 thus it is there where we observe the most differences.
Impact of server side pruning. In order to understand whether server side pruning is harmful for convergence, we plot both the global and average local validation accuracy on CIFAR 10 for the “mid” setting of FedSparse with and without server side pruning enabled. As we can see, there are no noticeable differences and in fact, pruning results into a slightly better overall performance.

heading: D CORRESPONDENCE BETWEEN SINGLE STEP EM AND GRADIENT ASCENT
text: With the addition of the auxiliary variables φs we have that the overall objective for the server becomes
argmax w
1
S S∑ s=1 log ∫ p(Ds|φs)p(φs|w)dφs. (21)
By performing EM with a single gradient step for w in the M-step (instead of full maximization), we are essentially doing gradient ascent on the original objective at 21. To see this, we can take the gradient of Eq. 21 w.r.t. w where Zs = ∫ p(Ds|φs)p(φs|w)dφs
1
S ∑ s 1 Zs ∫ p(Ds|φs) ∂p(φs|w) ∂w dφs = (22)
1
S ∑ s ∫ p(Ds|φs)p(φs|w) Zs ∂ log p(φs|w) ∂w dφs = (23)
1
S ∑ s ∫ p(φs|Ds,w) ∂ log p(φs|w) ∂w dφs (24)
where to compute Eq. 24 we see that we first have to obtain the posterior distribution of the local variables φs and then estimate the gradient for w by marginalizing over this posterior.

heading: E DERIVATION OF THE LOCAL LOSS
text: Let p(φsi|wi, zsi = 1) = N (wi, 1/λ), p(φsi|wi, zsi = 0) = N (0, 1/λ2) and q(φsi|zsi = 1) = N (wsi, 2), q(φsi|zsi = 0) = N (0, 2). Furthermore, let q(zsi) = Bern(πsi). The local objective that stems from 13 can be rewritten as:
argmax w1:S ,π1:S
Eqπs (zs)qws (φs|zs) [ log p(Ds|φs) ] − Eqπs (zs) [ KL(qws(φs|zs)||p(φs|w, zs)) ] + Eqπs (zs)[log p(zs|θ)], (25)
where we omitted from the objective the entropy of the distribution over the local gates.
One of the quantities that we are after is Eq(zsi)[KL(q(φsi|zsi)||p(φsi|zsi))] =
πsiKL(N (wsi, 2)||N (wi, 1/λ)) + (1− πsi)KL(N (0, 2)||N (0, 1/λ2)). (26) The KL term for when zsi = 1 can be written as
KL(N (wsi, 2)||N (wi, 1/λ)) = − 1 2 log λ− log + λ
2
2 − 1 2 + λ 2 (wsi −wi)2. (27)
The KL term for when zsi = 0 can be written as
KL(N (0, 2)||N (0, 1/λ2)) = − 1
2 log λ2 − log +
λ2 2
2 − 1 2 . (28)
Taking everything together we thus have
Eq(zsi)[KL(q(φsi|zsi)||p(φsi|zsi))] = λπsi 2 (wsi −wi)2 + πsi(− 1 2 log λ− log + λ
2
2 − 1 2 )+
(1− πsi)(− 1
2 log λ2 − log +
λ2 2
2 − 1 2 ) (29)
= λπsi 2
(wsi −wi)2 + πsi ( 1
2 log λ2 λ + 2 2 (λ− λ2)
) + C
(30)
≈ λπsi 2 (wsi −wi)2 + λ0πsi + C, (31)
where λ0 = 12 log λ2 λ and
2 2 (λ− λ2) was omitted due to 2 ≈ 0. In the appendix of Louizos et al.
(2017), the authors argue about a hypothetical prior that results into needing λ nats to transform that prior to the approximate posterior. Here we make this claim more precise and show that this prior is approximately equivalent to a mixture of Gaussians prior where the precision of the non-zero prior component λ→ (in order to avoid the L2 regularization term) and the precision of the zeroth component λ2 is equivalent to λ exp(2λ0), where λ0 is the desired L0 regularization strength.
Furthermore, the cross-entropy from qπs(zs) to p(zs|θ) is straightforward to compute as Eqπs (zs)[log p(zs|θ)] = ∑ j (πsj log θj + (1− πsj) log(1− θj)) . (32)
By putting everything together we have that the local objective becomes
argmax ws,πs Eqπs (zs) [ Ns∑ i L(Dsi,ws zs) ] − λ 2 ∑ j πsj(wsj −wj)2 − λ0 ∑ j πsj
+ ∑ j (πsj log θj + (1− πsj) log(1− θj)) + C. (33)

heading: F LOCAL OPTIMIZATION OF THE BINARY GATES
text: We propose to rewrite the local loss in Eq. 13 to
Ls(Ds,w,θ,φs,πs) := Eqπs (zs) [ Ns∑
i L(Dsi,ws zs)− λ ∑ j I[zsj 6= 0](wsj −w)2
− λ0 ∑ j I[zsj 6= 0] + ∑ j ( I[zsj 6= 0] log θj 1− θj + log(1− θj) )] , (34)
and then replace the Bernoulli distribution qπs(zs) with a continuous relaxation, the hard-concrete distribution (Louizos et al., 2017). Let the continuous relaxation be rus(zs), where us are the parameters of the surrogate distribution. In this case the local objective becomes
Ls(Ds,w,θ,φs,us) := Erus (zs) [ Ns∑
i
L(Dsi,ws zs) ] − λ ∑ j Rusj (zsj > 0)(wsj −w)2
− λ0 ∑ j Rusj (zsj > 0) + ∑ j ( Rusj (zsj > 0) log θj 1− θj + log(1− θj) ) , (35)
where Rus(·) is the cumulative distribution function (CDF) of the continuous relaxation rus(·). We can now straightforwardly optimize the surrogate objective with gradient ascent.

Review summaries:
Federated learning has emerged as a promising approach to training models at the edge devices. This paper makes an observation that most algorithms used within federated learning, including the popular FedAvg, could be cast as instances of EM methods. The paper then continues to propose FedSparse, a federated learning framework imposing sparse priors (specifically Bernoulli-Gaussian priors), and concludes with some experimental results on FedSparse.

The paper proposes to re-interpret federated averaging (FedAvg) as a version of the expectation-maximization (EM) algorithm under a particular probabilistic model. Further, the authors propose to use spike-and-slab sparsity inducing priors over the local model parameters to sparsify the learned models (the corresponding method is called FedSparse), which naturally reduces the communication cost (only non-zero parameters need to be sent over the network). Improvement in communication efficiency is showcased on a few standard federated datasets.