Title: Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Optimization

Deep Learning
Abstract: Cross-device Federated Learning (FL) faces significant challenges where low-end clients that could potentially make unique contributions are excluded from training large models due to their resource bottlenecks. Recent research efforts have focused on model-heterogeneous FL, by extracting reduced-size models from the global model and applying them to local clients accordingly. Despite the empirical success, general theoretical guarantees of convergence on this method remain an open question. 
This paper presents a unifying framework for heterogeneous FL algorithms with online model extraction and provides a general convergence analysis for the first time. 
In particular, we prove that under certain sufficient conditions and for both IID and non-IID data, these algorithms converge to a stationary point of standard FL for general smooth cost functions. Moreover, we introduce the concept of minimum coverage index, together with model reduction noise, which will determine the convergence of heterogeneous federated learning, and therefore we advocate for a holistic approach that considers both factors to enhance the efficiency of heterogeneous federated learning.

Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction

Abstract

Cross-device Federated Learning (FL) faces significant challenges where lowend clients that could potentially make unique contributions are excluded from training large models due to their resource bottlenecks. Recent research efforts have focused on model-heterogeneous FL, by extracting reduced-size models from the global model and applying them to local clients accordingly. Despite the empirical success, general theoretical guarantees of convergence on this method remain an open question. This paper presents a unifying framework for heterogeneous FL algorithms with online model extraction and provides a general convergence analysis for the first time. In particular, we prove that under certain sufficient conditions and for both IID and non-IID data, these algorithms converge to a stationary point of standard FL for general smooth cost functions. Moreover, we introduce the concept of minimum coverage index, together with model reduction noise, which will determine the convergence of heterogeneous federated learning, and therefore we advocate for a holistic approach that considers both factors to enhance the efficiency of heterogeneous federated learning.

1 Introduction

Federated Learning (FL) is a machine learning paradigm that enables a massive number of distributed clients to collaborate and train a centralized global model without exposing their local data [1]. Heterogeneous FL is confronted with two fundamental challenges: (1) mobile and edge devices that are equipped with drastically different computation and communication capabilities are becoming the dominant source for FL [2], also known as device heterogeneity; (2) state-of-the-art machine learning model sizes have grown significantly over the years, limiting the participation of certain devices in training. This has prompted significant recent attention to a family of FL algorithms relying on training reduced-size heterogeneous local models (often obtained through extracting a subnet or pruning a shared global model) for global aggregation. It includes algorithms such as HeteroFL [3] that employ fixed heterogeneous local models, as well as algorithms like PruneFL [4] and FedRolex [5] that adaptively select and train pruned or partial models dynamically during training. However, the success of these algorithms has only been demonstrated empirically (e.g., [2, 4, 3]). Unlike standard FL that has received rigorous analysis [6, 7, 8, 9], the convergence of heterogeneous FL algorithms is still an open question.

This paper aims to answer the following questions: Given a heterogeneous FL algorithm that trains a shared global model through a sequence of time-varying and client-dependent local models, what conditions can guarantee its convergence ? And intrinsically how do the resulting models compare to that of standard FL ? There have been many existing efforts in establishing convergence guarantees for FL algorithms, such as the popular FedAvg [1], on both IID (independent and identically distributed data) and non-IID[9] data distributions, but all rely on the assumption that local models share the same uniform structure as the global model 1 . Training heterogeneous local models, which could change both over time and across clients in FL is desirable due to its ability to adapt to resource constraints and training outcomes[10].

For general smooth cost functions and under standard FL assumptions, we prove that heterogeneous FL algorithms satisfying certain sufficient conditions can indeed converge to a neighborhood of a stationary point of standard FL (with a small optimality gap that is characterized in our analysis), at a rate of O ( 1 √ Q ) in Q communication rounds. Moreover, we show not only that FL algorithms involving local clients training different subnets (pruned or extracted from the global model) will converge, but also that the more they cover the parameters space in the global model, the faster the training will converge. Thus, local clients should be encouraged to train with reduced-size models that are of different subnets of the global model rather than pruning greedily. The work extends previous analysis on single-model adaptive pruning and subnetwork training[11, 12] to the FL context, where a fundamental challenge arises from FL's local update steps that cause heterogeneous local models (obtained by pruning the same global model or extracting a submodel) to diverge before the next aggregation. We prove

b

C

d

b

C

b

P

Round Q

C

d

a

b

C

d

Round Q+1

3x Models(1) for

3x Models(2)for

4x Models for

Small-capacity Client

Small-capacity Client

Large-capacity Client

a new upperbound and show that the optimality gap (between heterogeneous and standard FL) is affected by both model-reduction noise and a new notion of minimum coverage index in FL (i.e., any parameters in the global model are included in at least Γ min local models).

The key contribution of this paper is to establish convergence conditions for federated learning algorithms that employ heterogeneous arbitrarily-pruned, time-varying, and client-dependent local models to converge to a stationary point of standard FL. Numerical evaluations validate the sufficient conditions established in our analysis. The results demonstrate the benefit of designing new model reduction strategies with respect to both model reduction noise and minimum coverage index.

2 Background

Standard Federated Learning A standard FL problem considers a distributed optimization for N clients:

where θ is as set of trainable weights/parameters, F n ( θ ) is a cost function defined on data set D i with respect to a user specified loss function l ( x, θ ) , and p i is the weight for the i -th client such that p i ≥ 0 and ∑ N i =1 p i = 1 .

The FL procedure, e.g., FedAvg [1], typically consists of a sequence of stochastic gradient descent steps performed distributedly on each local objective, followed by a central step collecting the workers' updated local parameters and computing an aggregated global parameter. For the q -th round of training, first, the central server broadcasts the latest global model parameters θ q to clients n = 1 , . . . , N , who perform local updates as follows:

where γ is the local learning rate. After all available clients have concluded their local updates (in T epochs), the server will aggregate parameters from them and generate the new global model for the next round, i.e., θ q +1 = ∑ N n =1 p i θ q,n,T The formulation captures FL with both IID and non-IID data distributions.

3 Related Work

Federated Averaging and Related Convergence Analysis . FedAvg [1] is consi dered the first and the most commonly used federated learning algorithm . Several works have shown the convergence of FedAvg under several different settings with both homogeneous (IID) data [6, 13] and heterogeneous (non-IID) data [9, 7, 8] even with partial clients participation [14]. Specifically, [8] demonstrated LocalSGD achieves O ( 1 √ NQ ) convergence for non-convex optimization and [9] established a conver- gence rate of O ( 1 Q ) for strongly convex problems on FedAvg, where Q is the number of SGDs and N is the number of participated clients.

Efficient and Heterogeneous FL through Neural Network Pruning and Sparsification. Several works [15, 16, 17, 18, 19, 20] are proposed to further reduce communication costs in FL. One direction is to use data compression such as quantization [21, 7, 22, 23], sketching [24, 25], split learning [26], learning with gradient sparsity [27] and sending the parameters selectively[28]. This type of work does not consider computation efficiency. There are also works that address the reduction of both computation and communication costs, including one way to utilize lossy compression and dropout techniques[29, 30]. Although early works mainly assume that all local models share the same architecture as the global model [31], recent works have empirically demonstrated that federated learning with heterogeneous client models to save both computation and communication is feasible. PruneFL[4] proposed an approach with adaptive parameter pruning during FL. [32] proposed FL with a personalized and structured sparse mask. FjORD[33] and HetroFL[3] proposed to generate heterogeneous local models as a subnet of the global network by extracting a static sub-models, Hermes[34] finds the small sub-network by applying the structured pruning. There are also researches on extracting a subnetwork dynamically, e.g. Federated Dropout[29] extracts submodels randomly and FedRolex[5] applies a rolling sub-model extraction.

Despite their empirical success, they either lack theoretical convergence analysis or are specific to their own work. PruneFL only shows a convergence of the proposed algorithm and does not ensure convergence to a solution of standard FL. Meanwhile, static subnet extraction like Hermes does not allow the pruned local networks to change over time nor develop general convergence conditions. Following Theorem 1 works like Hermes can now employ time-varying subnet extractions, rather than static subnets, while still guaranteeing the convergence to standard FL. The convergence of HeteroFL and FedRolex- which was not available - now follows directly from Theorem 1. Once PruneFL satisfies the conditions established in our Theorem 1, convergence to a solution of standard FL can be achieved, rather than simply converging to some point. In summary, our general convergence conditions in Theorem 1 can provide support to existing FL algorithms that employ heterogeneous local models, ensuring convergence to standard FL. It also enables the design of optimized pruning masks/models to improve the minimum coverage index and thus the resulting gap to standard FL.

4 Methodology

4.1 Problem Formulation for FL with Heterogeneous Local models

Given an FL algorithm that trains heterogeneous local models for global aggregation, our goal is to analyze its convergence with respect to a stationary point of standard FL. We consider a general formulation where the heterogeneous local models can be obtained using any model reduction

strategies that are both (i) time-varying to enable online adjustment of reduced local models during the entire training process and (ii) different across FL clients with respect to their individual heterogeneous computing resource and network conditions. More formally, we denote the sequence of local models used by a heterogeneous FL algorithm by masks m q,n ∈ { 0 , 1 } | θ | , which can vary at any round q and for any client n . Let θ q denote the global model at the beginning of round q and ⊙ be the element-wise product. Thus, θ q ⊙ m q,n defines the trainable parameters of the reduced local model 2 for client n in round q . Our goal is to find sufficient conditions on such masks m q,n ∀ q, n for the convergence of heterogeneous FL.

Here, we describe one around (say the q th) of the heterogeneous FL algorithm. First, the central server employs a given model reduction strategy P ( · ) to reduce the latest global model θ q and broadcast the resulting local models to clients:

We note that the model reduction strategy P ( θ q , n, q ) can vary over time q and across clients n in heterogeneous FL. Each client n then trains the reduced local model by performing T local updates (in T epochs):

where γ is the learning rate and ξ n,t -1 are independent samples uniformly drawn from local data D n at client n . We note that ∇ F n ( θ q,n,t -1 , ξ n,t -1 ) ⊙ m q,n is a local stochastic gradient evaluated using only local parameters in θ q,n,t -1 (available to the heterogeneous local model) and that only locally trainable parameters are updated by the stochastic gradient (via an element-wise product with m q,n ).

Finally, the central server aggregates the local models θ n,q,T ∀ n and produces an updated global model θ q +1 . Due to the use of heterogeneous local models, each global parameter is included in a (potentially) different subset of the local models. Let N ( i ) q be the set of clients, whose local models contain the i th modeling parameter in round q . That is n ∈ N ( i ) q if m ( i ) q,n = 1 and n / ∈ N ( i ) q if m ( i ) q,n = 0 . Global update of the i th parameter is performed by aggregating local models with the parameter available, i.e.,

where |N ( i ) q | is the number of local models containing the i th parameter. We summarize the algorithm details in Algorithm 1 and the explanation of the notations in the Appendix.

The fundamental challenge of convergence analysis mainly stems from the local updates in Eq.(3). While the heterogeneous models θ q,n, 0 provided to local clients at the beginning of round q are obtained from the same global model θ q , performing T local updates causes these heterogeneous local models to diverge before the next aggregation. In addition, each parameter is (potentially) aggregated over a different subset of local models in Eq.(3). These make existing convergence analysis intended for single-model adaptive pruning [11, 12] non-applicable to heterogeneous FL. The impact of local model divergence and the global aggregation of heterogeneous models must be characterized in order to establish convergence.

The formulation proposed above captures heterogeneous FL with any model pruning or sub-model extraction strategies since the resulting masks m q,n ∀ q, n can change over time q and across clients n . It incorporates many model reduction strategies (such as pruning, sparsification, and sub-model extraction) into heterogeneous FL, allowing the convergence results to be broadly applicable. It recovers many important FL algorithms recently proposed, including HeteroFL [3] that uses fixed masks m n , PruneFL [4] that periodically trains a full-size local model with m n,q = 1 , Prune-andGrow [12] that can be viewed as a single-client version, as well as FedAvg [1] that employs full-size local models with m n,q = 1 at all clients. Our analysis establishes general conditions for any heterogeneous FL with arbitrary online model reduction that may vary over communication rounds to converge to standard FL.

Our paper establishes convergence conditions in the general form, which apply to both static and dynamically changing masks. With dynamically changing masks, we denote the reduced models/networks as θ q,n,t , which means that the model structure (with its corresponding mask) can change between each round of communications and during each local training epoch, and can be different from other local clients. We show that as long as the dynamic heterogenous FL framework can be framed as the setting above, our convergence analysis in this paper applies.

4.2 Notations and Assumptions

We make the following assumptions that are routinely employed in FL convergence analysis. In particular, Assumption 1 is a standard and common setting assuming Lipschitz continuous gradients. Assumption 2 follows from [12] (which is for a single-worker case) and implies the noise introduced by model reduction is bounded and quantified. This assumption is required for heterogeneous FL to converge to a stationary point of standard FL. Assumptions 3 and 4 are standard for FL convergence analysis following from [36, 37, 8, 9] and assume the stochastic gradients to be bounded and unbiased.

Assumption 1. (Smoothness). Cost functions F 1 , . . . , F N are all L-smooth: ∀ θ, ϕ ∈ R d and any n , we assume that there exists L > 0 :

Assumption 2. (Model Reduction Noise). We assume that for some δ 2 ∈ [0 , 1) and any q, n , the model reduction error is bounded by

Assumption 3. (Bounded Gradient). The expected squared norm of stochastic gradients is bounded uniformly, i.e., for constant G > 0 and any n, q, t :

Assumption 4. (Gradient Noise for IID data). Under IID data distribution, ∀ q, n, t , we assume a gradient estimate with bounded variance:

4.3 Convergence Analysis

We now analyze the convergence of heterogeneous FL for general smooth cost functions. We begin with introducing a new notion of minimum covering index , defined in this paper by

where Γ min 3 measures the minimum occurrence of the parameter in the local models in all rounds, considering |N ( i ) q | is the number of heterogeneous local models containing the i th parameter. Intuitively, if a parameter is never included in any local models, it is impossible to update it. Thus conditions based on the covering index would be necessary for the convergence toward standard FL (with the same global model). All proofs for theorems and lemmas are collected in the Appendix with a brief proof outline provided here.

Theorem 1. Under Assumptions 1-4 and for arbitrary masks satisfying Γ min ≥ 1 , when choosing γ ≤ 1 / (6 LT ) ∧ γ ≤ 1 / ( T √ Q ) , heterogeneous FL converges to a small neighborhood of a stationary point of standard FL as follows:

where G 0 = 4 E [ F ( θ 0 )] , V 0 = 6 LNσ 2 / (Γ ∗ ) 2 , H 0 = 2 L 2 NG/ Γ ∗ , and I 0 = 3 L 2 δ 2 N are constants depending on the initial model parameters and the gradient noise.

An obvious case here is that when Γ min = 0 , where there exists at least one parameter that is not covered by any of the local clients and all the client models can not cover the entire global model, we can consider the union of all local model parameters, the 'largest common model" among them, as a new equivalent global model ˆ θ (which have a smaller size than θ ). Then, each parameter in ˆ θ is covered in at least one local model. Thus Theorem 1 holds for ˆ θ instead and the convergence is proven - to a stationary point of ˆ θ rather than θ . 4

Assumption 5. (Gradient Noise for non-IID data). Let ˆ g ( i ) q,t = 1 |N ( i ) q | ∑ n ∈N ( i ) q ∇ F ( i ) n ( θ q,n,t , ξ n,t ) . Under non-IID data distribution, we assume ∀ i, q, t a gradient estimate with bounded variance:

Theorem 2. Under Assumptions 1-3 and 5, heterogeneous FL satisfying Γ min ≥ 1 , when choosing γ ≤ 1 / √ TQ and γ ≤ 1 / (6 LT ) , heterogeneous FL converges to a small neighborhood of a stationary point of standard FL as follows:

where G 1 = 4 E [ F ( θ 0 )] + 6 LKσ 2 .

Proof outline. There are a number of challenges in delivering main theorems. We begin the proof by analyzing the change of loss function in one round as the model goes from θ q to θ q +1 , i.e., F ( θ q +1 ) -F ( θ 1 ) . It includes three major steps: reducing the global model to obtain heterogeneous local models θ q,n, 0 = θ q ⊙ m q,n , training local models in a distributed fashion to update θ q,n,t , and parameter aggregation to update the global model θ q +1 .

Due to the use of heterogeneous local models whose masks m q,n both vary over rounds and change for different workers, we first characterize the difference between local model θ q,n,t at any epoch t and global model θ q at the beginning of the current round. It is easy to see that this can be factorized into two parts: model reduction error ∥ θ q,n, 0 -θ q ∥ 2 and local training ∥ θ q,n,t -θ q,n, 0 ∥ 2 , which will be analyzed in Lemma 1.

Lemma 1. Under Assumption 2 and Assumption 3, for any q , we have:

We characterize the impact of heterogeneous local models on global parameter updates. Specifically, we use an ideal local gradient ∇ F n ( θ q ) as a reference point and quantify the difference between aggregated local gradients and the ideal gradient. This will be presented in Lemma 2.

Lemma 2. Under Assumptions 1-3, for any q , we have:

where we relax the inequality by choosing the smallest Γ ∗ = min q,i Γ ( i ) q . We also quantify the norm difference between a gradient and a stochastic gradient (with respect to the global update step) using the gradient noise assumptions, in Lemma 3.

Since IID and non-IID data distributions in our model differ in the gradient noise assumption (i.e., Assumption 4 and Assumption 5), we present a unified proof for both cases. We will explicitly state IID and non-IID data distributions only if the two cases require different treatments (when the gradient noise assumptions are needed). Otherwise, the derivations and proofs are identical for both cases.

Lemma 3. For IID data distribution under Assumptions 4, for any q , we have:

For non-IID data distribution under Assumption 5, for any q , we have:

Finally, under assumption 1, we have F ( θ q +1 ) -F ( θ q ) ≤ ⟨∇ F ( θ q ) , θ q +1 -θ q ⟩ + L 2 ∥ θ q +1 -θ q ∥ 2 and use the preceding lemmas to obtain two upperbounds for the two terms. Combining these results we prove the desired convergence result in theorem 1 and theorem 2.

Theorem 1 shows the convergence of heterogenous FL to a neighborhood of a stationary point of standard FL albeit a small optimality gap due to model reduction noise, as long as Γ min ≥ 1 . The result is a bit surprising since Γ min ≥ 1 only requires each parameter to be included in at least one local model - which is obviously necessary for all parameters to be updated during training. But we show that this is also a sufficient condition for convergence. Moreover, we also establish a convergence rate of O ( 1 √ Q ) for arbitrary model reduction strategies satisfying the condition. When the cost function is strongly convex (e.g., for softmax classifier, logistic regression, and linear regression with l 2 -normalization), the stationary point becomes the global optimum. Thus, Theorem 1 shows convergence to a small neighborhood of the global optimum of standard FL for strongly convex cost functions.

5 Interpreting and Applying the Unified Framework

Discussion on the Impact of model reduction noise. In Assumption 2, we assume the model reduction noise is relatively small and bounded with respect to the global model: ∥ θ q -θ q ⊙ m q,n ∥ 2 ≤ δ 2 ∥ θ q ∥ 2 . This is satisfied in practice since most pruning strategies tend to focus on eliminating weights/neurons that are insignificant, therefore keeping δ 2 indeed small. We note that similar observations are made on the convergence of single-model adaptive pruning [11, 12], but the analysis does not extend to FL problems where the fundamental challenge comes from local updates causing heterogeneous local models to diverge before the next global aggregation. We note that for heterogeneous FL, reducing a model will incur an optimality gap δ 2 1 Q ∑ Q q =1 E ∥ θ q ∥ 2 in our convergence analysis, which is proportional to δ 2 and the average model norm (averaged over Q ). It implies that a more aggressive model reduction in heterogeneous FL may lead to a larger error, deviating from standard FL at a speed quantified by δ 2 . We note that this error is affected by both δ 2 and Γ min .

Discussion on the Impact of minimum covering index Γ min . The minimum number of occurrences of any parameter in the local models is another key factor in deciding convergence in heterogeneous FL. As Γ min increases, both constants G 0 , V 0 , and the optimality gap decrease. Recall that our analysis shows the convergence of all parameters in θ q with respect to a stationary point of standard FL (rather than for a subset of parameters or to a random point). The more times a parameter is covered by local models, the sooner it gets updated and convergences to the desired target. This is quantified in our analysis by showing that the optimality gap due to model reduction noise decreases at the rate of Γ min .

Connections between model reduction noise and minimum covering index. In this paper, we introduced the concept of minimum coverage index for the first time, where we show that only model compression alone is not enough to allow a unified convergence analysis/framework for

heterogeneous federated learning. The minimum coverage index, together with pruning/compression noises, determines convergence in heterogeneous FL. Our results show that heterogeneous FL algorithms satisfying certain sufficient conditions can indeed converge to a neighborhood of a stationary point of standard FL. This is a stronger result as it shows convergence to standard FL, rather than simply converging somewhere. A minimum coverage index of Γ min = 0 means that the model would never be updated, which is meaningless even if it still converges.

Discussion for non-IID case. We note that Assumption 5 is required to show convergence with respect to standard FL and general convergence may reply on weaker conditions. We also notice that Γ min no longer plays a role in the optimality gap. This is because the stochastic gradients computed by different clients in N ( i ) q now are based on different datasets and jointly provide an unbiased estimate, no longer resulting in smaller statistical noise.

Applying the main theoretical findings. Theorem 1 also inspires new design criteria for designing adaptive model-reducing strategies in heterogeneous FL. Since the optimality gap is affected by both model-reduction noise δ 2 and minimum covering index Γ min , we may prefer strategies with small δ 2 and large Γ min , in order to minimize the optimality gap to standard FL.

The example shown in Figure 1 illustrates alternative model reduction strategies in heterogeneous FL for N = 10 clients. Suppose all 6 low-capacities clients are using the reduced-size model by pruning greedily, which covers the same region of the global model, scenarios like this will only produce a maximum Γ min = 4 ; however when applying low-capacities local clients with models covering different regions of the global model, Γ min can be increased, as an example we show how to design local models so Γ min is increased to 7 without increasing any computation and communication cost. The optimal strategy corresponds to lower noise δ 2 while reaching a higher covering index. Using these insights, We present numerical examples with optimized designs in Section 6.

6 Experiments

6.1 Experiment settings

In this section, we evaluate heterogeneous FL with different model reduction strategies and aim to validate our theory. We focus on two key points in our experiments: (i) whether heterogeneous FL will converge with different local models and (ii) the impacts of key factors to the FL performances including minimum coverage index Γ min and model-reduction noise δ 2 .

0

20

40

60

80

100

Communication Round

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Global Loss

FullModel

Pruning-Medium

Pruning-Medium-Opt

Subnet-Medium

Subnet-Medium-Opt

(a) MLP trained on MNIST with IID data

0

20

40

60

80

100

Communication Round

0.0

0.2

0.4

0.6

0.8

Global Loss

FullModel

Pruning-Medium

Pruning-Medium-Opt

Subnet-Medium

Subnet-Medium-Opt

(b) MLP trained on MNIST with non-IID data

Datasets and Models. We examine the theoretical results on the following three commonly-used image classification datasets: MNIST [38] with a shallow multilayer perception (MLP), CIFAR-10 with Wide ResNet28x2 [39], and CIFAR100 [40] with Wide ResNet28x8[39]. The first setting where using MLP models is closer to the theoretical assumptions and settings, and the latter two settings are closer to the real-world application scenarios. We prepare N = 100 workers with IID and non-IID

data with participation ratio c = 0 . 1 which will include 10 random active clients per communication round. Please see the appendix for other experiment details.

Data Heterogeneity. We follow previous works [3, 5] to model non-IID data distribution by limiting the maximum number of labels L as each client is accessing. We consider two levels of data heterogeneity: for MNIST and CIFAR-10 we consider L = 2 as high data heterogeneity and L = 5 as low data heterogeneity as used in [9]. For CIFAR-100 we consider L = 20 as high data heterogeneity and L = 50 as low data heterogeneity. This will correspond to an approximate setting of Dir K ( α ) with α = 0 . 1 for MNIST, α = 0 . 1 for CIFAR-10, and α = 0 . 5 for CIFAR-100 respectively in Dirichlet-distribution-based data heterogeneity.

Model Heterogeneity. In our evaluation, we consider the following client model reduction levels: β = { 1 , 3 4 , 1 2 , 1 4 } for MLP and β = { 1 , 1 2 , 1 4 , 1 8 } for ResNet, where each fraction represents its model capacity ratio to the largest client model (full model). To generate these client models, for MLP we reduce the number of nodes in each hidden layer, for WResNet we reduce the number of kernels in convolution layers while keeping the nodes in the output layer as the original.

Baselines and Testcase Notations. As this experiment is mainly to validate the proposed theory and gather empirical findings, we choose the standard federated learning algorithm, i.e. FedAvg [1], with several different heterogeneous FL model settings. Since this experiment section is to verify the impact of our proposed theory rather than chasing a SOTA accuracy, no further tuning or tricks for training were used to demonstrate the impacts of key factors from the main theorems.

We consider 3 levels of model reduction through pruning and static subnet Extraction: which will reduce the model by only keeping the largest or leading β percentile of the parameters per layer. We show 4 homogeneous settings with the full model and the models with 3 levels of model reduction, each with at least one full model so that Γ min > 1 is achieved. Finally, we consider manually increasing the minimum coverage index and present one possible case denoted as "optimized", by applying local models covering different regions of the global model as illustrated in Fig 1.

Note that even when given a specific model reduction level and the minimum coverage index, there could be infinite combinations of local model reduction solutions; at the same time model reduction will inevitably lead to an increased model reduction noise, by conducting only weights pruning will bring the lowest model reduction noise for a certain model reduction level. How to manage the trade-off between increasing Γ min while keeping δ 2 low is non-trivial and will be left for future works on designing effective model reduction policies for heterogeneous FL.

6.2 Numerical Results and Further Discussion

We summarize the testing results with one optimized version for comparison in Table 1. We plot the training results of Heterogeneous FL with IID and non-IID data on the MNIST dataset in Figure 2a and Figure 2b, since the model and its reduction are closer to the theoretical setup and its assumptions. We only present training results of medium-level model reduction (where we deploy 4 clients with

fullmodel and 6 clients with 3 4 models) in the figure at the main paper due to page limit and simplicity. We leave further details and more results in the appendix.

General Results. Overall, we observe improved performance on almost all optimized results, especially on subnet-extraction-based methods on high model reduction levels. In most cases, performances will be lower compared to the global model due to model-reduction noise.

Impact of model-reduction noise. As our analysis suggests, one key factor affecting convergence is model-reduction noise δ 2 . When a model is reduced, inevitably the model-reduction noise δ 2 will affect convergence and model accuracy. Yet, our analysis shows that increasing local epochs or communication rounds cannot mitigate such noise. To minimize the convergence gap in the upperbounds, it is necessary to design model reduction strategies in heterogeneous FL with respect to both model-reduction noise and minimum coverage index, e.g., by considering a joint objective of preserving large parameters while sufficiently covering all parameters.

Impact of minimum coverage index . Our theory suggests that for a given model reduction noise, the minimum coverage index Γ min is inversely proportional to the convergence gap as the bound in Theorem 1 indicates. Then for a given model reduction level, a model reduction strategy in heterogeneous FL with a higher minimum coverage index may result in better training performance. Note that existing heterogeneous FL algorithms with pruning often focus on removing the small model parameters that are believed to have an insignificant impact on model performance, while being oblivious to the coverage of parameters in pruned local models, and the model-extraction-based method will only keep the leading subnet. Our analysis in this paper highlights this important design for model reduction strategies in heterogeneous FL that parameter coverages matter.

More discussions and empirical findings . For the trade-off between minimum coverage index and model reduction noise, it's nearly impossible to fix one and investigate the impact of the other. In addition, we found: (1) Large models hold more potential to be reduced while maintaining generally acceptable accuracy. (2) Smaller models tend to be affected more by δ 2 while the larger model is more influenced by Γ min , which suggests that it's more suitable to apply pruning on small networks and apply subnet extraction on large networks.

Limitations In this work we consider full device participation, where arbitrary partial participation scenario is not considered. Also, the optimal design of model extraction maintaining a balance between a low δ 2 and a high Γ min is highly non-trivial which would be left for future work.

7 Conclusion

In this paper, we present a unifying framework and establish sufficient conditions for FL with dynamic heterogeneous client-dependent local models to converge to a small neighborhood of a stationary point of standard FL. The optimality gap is characterized and depends on model reduction noise and a new concept of minimum coverage index. The result recovers a number of state-of-the-art FL algorithms as special cases. It also provides new insights on designing optimized model reduction strategies in heterogeneous FL, with respect to both minimum coverage index Γ min and model reduction noise δ 2 . We empirically demonstrated the correctness of the theory and the design insights. Our work contributes to a deeper theoretical comprehension of heterogeneous FL with adaptive local model reduction and offers valuable insights for the development of new FL algorithms in future research.

References

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials , 22(3):2031-2063, 2020.

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2021.

Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K. Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices, 2020.

Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. arXiv preprint arXiv:2212.01548 , 2022.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of communication-efficient sgd algorithms. arXiv preprint arXiv:1808.07576 , 2018.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046 , 2019.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5693-5700, 2019.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data, 2020.

Hanhan Zhou, Tian Lan, Guru Prasadh Venkataramani, and Wenbo Ding. Federated learning with online adaptive heterogeneous local models. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022) , 2022.

Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint arXiv:2006.07253 , 2020.

Xiaolong Ma, Minghai Qin, Fei Sun, Zejiang Hou, Kun Yuan, Yi Xu, Yanzhi Wang, Yen-Kuang Chen, Rong Jin, and Yuan Xie. Effective model sparsification by scheduled grow-and-prune methods. arXiv preprint arXiv:2106.09857 , 2021.

Blake Woodworth, Jialei Wang, Adam Smith, Brendan McMahan, and Nathan Srebro. Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. arXiv preprint arXiv:1805.10222 , 2018.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. arXiv preprint arXiv:2205.13648 , 2022.

Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K. Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems, 2019.

Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd, 2019.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Bing Luo, Xiang Li, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Cost-effective federated learning design. In IEEE INFOCOM 2021 - IEEE Conference on Computer Communications , pages 1-10, 2021.

Runxue Bao, Xidong Wu, Wenhan Xian, and Heng Huang. Doubly sparse asynchronous learning for stochastic composite optimization. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI , pages 1916-1922, 2022.

Xidong Wu, Feihu Huang, Hu Zhengmian, and Huang Heng. Faster adaptive federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016.

Yuzhu Mao, Zihao Zhao, Guangfeng Yan, Yang Liu, Tian Lan, Linqi Song, and Wenbo Ding. Communication efficient federated learning with adaptive quantization, 2021.

Dezhong Yao, Wanning Pan, Yao Wan, Hai Jin, and Lichao Sun. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization, 2021.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communicationefficient sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems , 30:1709-1720, 2017.

Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Vladimir Braverman, Ion Stoica, and Raman Arora. Communication-efficient distributed sgd with sketching. arXiv preprint arXiv:1903.04488 , 2019.

Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004.12088 , 2020.

Pengchao Han, Shiqiang Wang, and Kin K Leung. Adaptive gradient sparsification for efficient federated learning: An online learning approach. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS) , pages 300-310. IEEE, 2020.

Zachary Charles, Kallista Bonawitz, Stanislav Chiknavaryan, Brendan McMahan, et al. Federated select: A primitive for communication-and memory-efficient federated learning. arXiv preprint arXiv:2208.09432 , 2022.

Sebastian Caldas, Jakub Koneˇ cny, H Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210 , 2018.

Zirui Xu, Zhao Yang, Jinjun Xiong, Janlei Yang, and Xiang Chen. Elfish: Resource-aware federated learning on heterogeneous edge devices. Ratio , 2(r1):r2, 2019.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems , pages 42-55, 2021.

Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems , 34:12876-12889, 2021.

Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking , pages 420-437, 2021.

Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning?, 2020.

Yuchen Zhang, John C Duchi, and Martin J Wainwright. Communication-efficient algorithms for statistical optimization. The Journal of Machine Learning Research , 14(1):3321-3363, 2013.

Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767 , 2018.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work establishes a general convergence analysis for a heterogeneous federated learning (FL) algorithm that trains a shared global model using a sequence of time-varying and client-dependent local models. In particular, this work establishes sufficient conditions for the convergence of such a heterogeneous FL algorithm to the neighborhood of a stationary point of the standard FL. Based on the theoretical results, the authors propose practical suggestions for designing heterogeneous FL algorithms and conduct thorough experiments to support their claims.\n\nSTRENGTHS:\nOverall, this paper is well-written. It is the first to provide a general convergence result to the neighborhood of a stationary point for the standard federated learning (FL) framework, specifically for a heterogeneous FL algorithm that trains a shared global model through a sequence of time-varying and client-dependent local models. This result provides a convergence guarantee for several previously proposed heterogeneous FL algorithms. The optimality gap in the result depends on two important parameters: the minimum coverage index and the model reduction noise. These parameters offer valuable insights for designing practical heterogeneous FL algorithms. The experimental results align well with the theoretical findings presented in the paper.\n\nWEAKNESSES:\nThe last term on the right-hand side of the inequality in Theorem 1 (and similarly for Theorem 2) corresponds to the average of the norms of the global parameters encountered during the optimization process. This term is not necessarily small in a straightforward manner.\n\nQUESTIONS:\nCan the authors provide additional discussions regarding the last term in Theorem 1 (and Theorem 2)? Specifically, it would be helpful to explore whether this term is tight or under what circumstances it can be considered small.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies federated learning with heterogeneous client models and non-iid client data. By assuming that the client models are pruned versions of a common global model and using the notion of minimum covering index, the paper provides the convergence of FedAvg under client model pruning. Theoretical results show that pruning techniques that more evenly update the parameters and result in smaller model distortion are preferable to aggressive pruning. Such result is also verified through the numerical results on MNIST and Cifar-10, and Cifar-100 datasets.\n\nSTRENGTHS:\nNovelty: this paper provides a first convergence analysis to FedAvg with model pruning. The theoretical proof is sound and clear. \n\nClarity: the paper thoroughly discusses the convergence result in theorems 1 and 2 and verifies the results through numerical experiments.\n\nWEAKNESSES:\n1. Term $E\\Vert \\theta_q\\Vert^2$ in the bound. This term appears in all theorems and lemmas in the main paper. However, it is unclear how large this term can be. Neither theoretical discussion nor numerical justification of this term is provided. This undermines the strength of the theoretical results.\n\n2. The connection between the theorem to the general convergence result of FedAvg under the same assumption. When $\\delta = 0, \\Gamma_{min} = T$, the convergence should recover the rate of standard FedAvg. However, such a discussion is missing, weakening the paper's clarity.\n\n3. The connection between the proof of pruning and lossy compression. Assumption 2 takes the standard assumption of model compression in compressed FL (e.g., [19, 21, 22]). Yet there is a critical difference in the minimum covering index. The connection and difference between the proof technique should be discussed.\n\nQUESTIONS:\nPlease address the above weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitation of the paper, that partial client participation is not considered, and how the theoretical result guides the design of an optimal model pruning strategy is also limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the cross-device federated learning setting. The authors introduce a general theoretical framework for analyzing FedAvg with masks on local pruned models. This analysis is particularly valuable in establishing the convergence of federated schemes when the global model is distributed across multiple edge clients. Furthermore, the paper includes numerical illustrations of their algorithm, providing practical insights on its performance.\n\nSTRENGTHS:\n* The paper is well-written and easy to understand, but with few points that might be improved (see \"Questions\" part).\n\n* The paper conducts a thorough theoretical investigation on federated learning with reduced-size models. Specifically, the authors provide novel tools to study the convergence of FL pruned models.\n\nWEAKNESSES:\n* Although interesting, some parts may seem to have been rushed. For example, assumptions and lemmas should be referred with ref{}. Further, I also spotted a few errors on the supplement, even though I haven't read the supplementary in detail. It's important to review and correct these typos.\n\nQUESTIONS:\nThe article is interesting, could you address the following points so that I can raise my score?\n\n* It is unclear whether $\\Gamma_q^{(i)}$ is defined or if it refers to $N_{q}^{(i)}$ mentioned on line 177. Similarly, does $\\Gamma^*$ denote the same quantity as $\\Gamma_{min}$?\n\n* The proposed update in the equation on line 134 is not exactly the same implemented in the \"Update.py\" script. I think the mask is not applied to the gradient.\n\n* I haven't looked at the supplementary paper, but it seems to me that an expectation is missing on Eq.(20).\n\n* Is it necessary to use $\\delta<1$ in assumption 2? Since I don't see $1/(1-\\delta)$ in your bounds, I wonder if this constraint is useful.\n\n**Requested Changes:**\n\n* Line 59: \"will it converge\" inside Figure 1 legend.\n\n* Line 75: client is denoted by $i$, then latter by $n$ at the end of page 2.\n\n* Line 95: \"models are to share\".\n\n* Lines 108: \"works like Hermes can\".\n\n* Definition of $I_0$ in Theorem 1 and Theorem 2 is different, there is maybe an additional $\\delta^2$.\n\n* Some punctuation is missing, e.g., Eq. (9).\n\n* Conclusion seems little heavy.\n\nLIMITATIONS:\nThe authors did not address the potential negative societal implications of their research, but this does not seem critical for this particular theoretical and numerical study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors provide a general theoretical framework to analyze the convergence of Federated training schemes conducted over local models of heterogeneous network structures. Such structures can usually be obtained through different model reduction methods, such as model pruning/sparsification or model extraction. The proposed framework introduces the minimum covering index concept to conduct the analysis, representing the number of local models concurrently updating the same set of parameter indices. The paper is well written, but a couple of clarifications are necessary.\n\nSTRENGTHS:\nVery well-written paper with clear objectives and contributions.\n\nGeneralized framework to encapsulate different model reduction algorithms.\n\nThe introduction of the minimum coverage index concept and its interplay with model reduction noise can lead to very promising and interesting insights.\n\nWEAKNESSES:\nFurther elaboration is needed on terms and concepts used during the theoretical analysis.\n\nEmpirical evaluation needs improvement.\n\nQUESTIONS:\nFrom my understanding, your framework seems to consider reduced model structures when these structures are obtained when the global model is reduced (pruned). Will your analysis hold even in the case of local model reduction, i.e., client-side reduction, not server-side? Will the model reduction noise (assumption 2) and bounded gradient (assumption 3) still hold? Your analysis shows that local model training is performed over the entire network. How would that affect your analysis if you were to enforce training over the parameters of the reduced model (enforce masks during training)?\n\nThe proof outline does not clarify how the mask is constructed. Is it always static, or can it be dynamically changing? In PruneFL, masks can periodically increase to accommodate model regrowth. Similarly, FedDST[1] performs local model regrowth with a fixed pruning degree during model communication. Analogously, FedSparsify[2] performs progressive model reduction (progressive sparsification), where the global or local model is progressively pruned. Can your framework capture such dynamic mask construction? Please expand your related work and discuss whether your framework can also accommodate such dynamic pruning approaches. \n\nThe model settings through which the model reduction level is obtained in Table 1 are not clear. Can you please elaborate on what are the methods you used to perform the greedy pruning, pruning-optimized, static subnet subtraction, and homogeneous methods? Did you use structured or unstructured pruning? Random weight magnitude or based on magnitude? Can you report the noise reduction (percentage-wise) in the models learned at every model reduction level? Such an analysis could lead to key insights with respect to the index coverage vs. noise reduction tradeoff.\n\nCan your analysis be extended to activations and/or neuron pruning as part of your future work? Also, have you considered extending your framework for asynchronous federated settings?\n\n[1] Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better Sameer Bibikar, Haris Vikalo, Zhangyang Wang, Xiaohan Chen. https://ojs.aaai.org/index.php/AAAI/article/view/20555/20314\n\n[2] Federated Progressive Sparsification (Purge-Merge-Tune)+. Dimitris Stripelis, Umang Gupta, Greg Ver Steeg, Jose Luis Ambite. https://openreview.net/pdf?id=GLQqPTRrQMx\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}