SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we relax the inequality by choosing the smallest  $\Gamma^{*} = \min_{q,i}\Gamma_{q}^{(i)}$ . We also quantify the norm difference between a gradient and a stochastic gradient (with respect to the global update step) using the gradient noise assumptions, in Lemma 3.
4To better illustrate this scenario of  $\Gamma_{min} = 0$  , we will introduce an illustrative simplified example as follows: A global model  $\theta = <  \theta_1,\theta_2,\theta_3 >$  where there will be two local models  $\theta_{a} = <  \theta_{1}>$  and  $\theta_{b} = <  \theta_{3}>$  Although  $\Gamma_{min} = 0$  regarding the global model  $\theta$  , but for their largest common model, the union of  $\theta_{a}$  and  $\theta_{b}$  which is  $<  \theta_{1},\theta_{3}>$  will become the new conceptual global model  $\hat{\theta}$  , where  $\Gamma_{min} = 1$  regarding this conceptual global model. Thus the convergence still stands, but it will converge to a stationary point of FL with a different global model.
Since IID and non-IID data distributions in our model differ in the gradient noise assumption (i.e., Assumption 4 and Assumption 5), we present a unified proof for both cases. We will explicitly state IID and non-IID data distributions only if the two cases require different treatments (when the gradient noise assumptions are needed). Otherwise, the derivations and proofs are identical for both cases.
Lemma 3. For IID data distribution under Assumptions 4, for any  $q$ , we have:
$$
\sum_ {i = 1} ^ {K} \mathbb {E} \| \frac {1}{\Gamma_ {q} ^ {(i)} T} \sum_ {t = 1} ^ {T} \sum_ {n \in \mathcal {N} _ {q} ^ {(i)}} \nabla F _ {n} ^ {(i)} (\theta_ {q, n, t - 1}, \xi_ {n, t - 1}) - \nabla F ^ {(i)} (\theta_ {q, n, t - 1}) \| ^ {2} \leq \frac {N \sigma^ {2}}{T (\Gamma^ {*}) ^ {2}}
$$
For non-IID data distribution under Assumption 5, for any  $q$ , we have:
$$
\sum_ {i = 1} ^ {K} \mathbb {E} \| \frac {1}{\Gamma_ {q} ^ {(i)} T} \sum_ {t = 1} ^ {T} \sum_ {n \in \mathcal {N} _ {q} ^ {(i)}} \nabla F _ {n} ^ {(i)} (\theta_ {q, n, t - 1}, \xi_ {n, t - 1}) - \nabla F ^ {(i)} (\theta_ {q, n, t - 1}) \| ^ {2} \leq \frac {K \sigma^ {2}}{T}
$$
Finally, under assumption 1, we have  $F(\theta_{q + 1}) - F(\theta_q) \leq \langle \nabla F(\theta_q), \theta_{q + 1} - \theta_q \rangle + \frac{L}{2} \| \theta_{q + 1} - \theta_q \|^2$  and use the preceding lemmas to obtain two upper bounds for the two terms. Combining these results we prove the desired convergence result in theorem 1 and theorem 2.
Theorem 1 shows the convergence of heterogenous FL to a neighborhood of a stationary point of standard FL albeit a small optimality gap due to model reduction noise, as long as  $\Gamma_{\mathrm{min}} \geq 1$ . The result is a bit surprising since  $\Gamma_{\mathrm{min}} \geq 1$  only requires each parameter to be included in at least one local model - which is obviously necessary for all parameters to be updated during training. But we show that this is also a sufficient condition for convergence. Moreover, we also establish a convergence rate of  $O\left(\frac{1}{\sqrt{Q}}\right)$  for arbitrary model reduction strategies satisfying the condition. When the cost function is strongly convex (e.g., for softmax classifier, logistic regression, and linear regression with  $l_2$ -normalization), the stationary point becomes the global optimum. Thus, Theorem 1 shows convergence to a small neighborhood of the global optimum of standard FL for strongly convex cost functions.
5 Interpreting and Applying the Unified Framework
Discussion on the Impact of model reduction noise. In Assumption 2, we assume the model reduction noise is relatively small and bounded with respect to the global model:  $\| \theta_q - \theta_q\odot m_{q,n}\| ^2\leq \delta^2\| \theta_q\| ^2$ . This is satisfied in practice since most pruning strategies tend to focus on eliminating weights/neurons that are insignificant, therefore keeping  $\delta^2$  indeed small. We note that similar observations are made on the convergence of single-model adaptive pruning [11, 12], but the analysis does not extend to FL problems where the fundamental challenge comes from local updates causing heterogeneous local models to diverge before the next global aggregation. We note that for heterogeneous FL, reducing a model will incur an optimality gap  $\delta^2\frac{1}{Q}\sum_{q = 1}^{Q}\mathbb{E}\| \theta_q\| ^2$  in our convergence analysis, which is proportional to  $\delta^2$  and the average model norm (averaged over  $Q$ ). It implies that a more aggressive model reduction in heterogeneous FL may lead to a larger error, deviating from standard FL at a speed quantified by  $\delta^2$ . We note that this error is affected by both  $\delta^2$  and  $\Gamma_{\mathrm{min}}$ .
Discussion on the Impact of minimum covering index  $\Gamma_{\mathrm{min}}$ . The minimum number of occurrences of any parameter in the local models is another key factor in deciding convergence in heterogeneous FL. As  $\Gamma_{\mathrm{min}}$  increases, both constants  $G_0, V_0$ , and the optimality gap decrease. Recall that our analysis shows the convergence of all parameters in  $\theta_q$  with respect to a stationary point of standard FL (rather than for a subset of parameters or to a random point). The more times a parameter is covered by local models, the sooner it gets updated and convergences to the desired target. This is quantified in our analysis by showing that the optimality gap due to model reduction noise decreases at the rate of  $\Gamma_{\mathrm{min}}$ .
Connections between model reduction noise and minimum covering index. In this paper, we introduced the concept of minimum coverage index for the first time, where we show that only model compression alone is not enough to allow a unified convergence analysis/framework for
heterogeneous federated learning. The minimum coverage index, together with pruning/compression noises, determines convergence in heterogeneous FL. Our results show that heterogeneous FL algorithms satisfying certain sufficient conditions can indeed converge to a neighborhood of a stationary point of standard FL. This is a stronger result as it shows convergence to standard FL, rather than simply converging somewhere. A minimum coverage index of  $\Gamma_{min} = 0$  means that the model would never be updated, which is meaningless even if it still converges.
Discussion for non-III case. We note that Assumption 5 is required to show convergence with respect to standard FL and general convergence may reply on weaker conditions. We also notice that  $\Gamma_{\mathrm{min}}$  no longer plays a role in the optimality gap. This is because the stochastic gradients computed by different clients in  $\mathcal{N}_q^{(i)}$  now are based on different datasets and jointly provide an unbiased estimate, no longer resulting in smaller statistical noise.
Applying the main theoretical findings. Theorem 1 also inspires new design criteria for designing adaptive model-reducing strategies in heterogeneous FL. Since the optimality gap is affected by both model-reduction noise  $\delta^2$  and minimum covering index  $\Gamma_{\mathrm{min}}$ , we may prefer strategies with small  $\delta^2$  and large  $\Gamma_{\mathrm{min}}$ , in order to minimize the optimality gap to standard FL.
The example shown in Figure 1 illustrates alternative model reduction strategies in heterogeneous FL for  $N = 10$  clients. Suppose all 6 low-capacities clients are using the reduced-size model by pruning greedily, which covers the same region of the global model, scenarios like this will only produce a maximum  $\Gamma_{min} = 4$ ; however when applying low-capacities local clients with models covering different regions of the global model,  $\Gamma_{min}$  can be increased, as an example we show how to design local models so  $\Gamma_{min}$  is increased to 7 without increasing any computation and communication cost. The optimal strategy corresponds to lower noise  $\delta^2$  while reaching a higher covering index. Using these insights, We present numerical examples with optimized designs in Section 6.
6.1 Experiment settings
In this section, we evaluate heterogeneous FL with different model reduction strategies and aim to validate our theory. We focus on two key points in our experiments: (i) whether heterogeneous FL will converge with different local models and (ii) the impacts of key factors to the FL performances including minimum coverage index  $\Gamma_{\mathrm{min}}$  and model-reduction noise  $\delta^2$ .
(a) MLP trained on MNIST with IID data
(b) MLP trained on MNIST with non-IID data
Figure 2: Selected experimental results for MNIST with IID (a) and Non-IID (b) with high data heterogeneity data on medium model reduction level. "Opt" stands for optimized local model distribution covering more regions for a higher  $\Gamma_{min}$ , others do pruning greedily. As the shallow MLP is already at a small size, applying a medium level of model reduction will bring a high model reduction loss for subnet extraction method.
Datasets and Models. We examine the theoretical results on the following three commonly-used image classification datasets: MNIST [38] with a shallow multilayer perception (MLP), CIFAR-10 with Wide ResNet28x2 [39], and CIFAR100 [40] with Wide ResNet28x8[39]. The first setting where using MLP models is closer to the theoretical assumptions and settings, and the latter two settings are closer to the real-world application scenarios. We prepare  $N = 100$  workers with IID and non-IID
Model Reduction Level	Model Setting	Γmin	MNIST	CIFAR-10	CIFAR100
IID	Non-IID	IID	Non-IID	IID	Non-IID
L=5	L=2	L=5	L=2	L=50	L=20
FullModel	Homogenous-Full	10	98.08	97.70	93.59	70.63	65.12	61.08	67.34	66.74	64.38
Low Model Reduction	Pruning-Greedy	6	98.18	97.60	93.15	72.66	62.49	57.17	67.41	65.67	65.06
Pruning-Optimised	8	98.53	98.25	95.85	76.26	66.25	59.98	67.47	66.56	67.47
Static Subnet Subtraction	6	97.62	95.12	92.33	73.20	61.25	56.09	66.60	65.24	65.79
Subnet Subtraction - Optimised	8	97.76	94.41	93.60	73.78	64.17	58.09	67.81	66.66	67.23
Homogenous-Large	10	97.52	96.08	93.23	69.05	63.72	57.42	66.81	65.57	63.90
Medium Model Reduction	Pruning-Greedy	4	97.51	95.05	91.86	66.85	60.93	56.98	52.92	45.88	45.68
Pruning-Optimised	8	98.39	98.02	95.48	71.43	66.94	56.93	55.32	46.81	45.74
Static Subnet Subtraction	4	95.56	92.33	92.05	61.87	58.08	46.03	50.59	44.22	45.25
Subnet Subtraction - Optimised	8	97.96	94.05	93.36	63.96	62.65	47.44	52.95	46.23	46.15
Homogenous-Medium	10	97.05	92.71	90.82	59.21	57.61	53.43	52.19	36.08	34.06
High Model Reduction	Pruning-Greedy	3	95.01	86.83	76.64	67.35	56.75	22.55	39.29	26.14	25.97
Pruning-Optimised	5	95.32	91.98	81.66	67.74	57.33	27.97	40.78	29.63	26.63
Static Subnet Subtraction	3	95.88	81.64	71.64	68.78	56.88	30.61	41.18	27.55	26.23
Subnet Subtraction - Optimised	5	94.41	90.70	85.82	69.15	57.98	33.46	37.42	24.98	22.40
Homogenous-Small	10	93.79	85.66	75.23	66.87	51.90	30.61	37.40	27.16	26.20
Table 1: Global model accuracy comparison between baselines and their optimized versions suggested by our theory. We observe improved performance on almost all optimized results, especially on subnet-extraction-based methods on high model reduction levels.
data with participation ratio  $c = 0.1$  which will include 10 random active clients per communication round. Please see the appendix for other experiment details.
Data Heterogeneity. We follow previous works [3, 5] to model non-IID data distribution by limiting the maximum number of labels  $L$  as each client is accessing. We consider two levels of data heterogeneity: for MNIST and CIFAR-10 we consider  $L = 2$  as high data heterogeneity and  $L = 5$  as low data heterogeneity as used in [9]. For CIFAR-100 we consider  $L = 20$  as high data heterogeneity and  $L = 50$  as low data heterogeneity. This will correspond to an approximate setting of  $Dir_{K}(\alpha)$  with  $\alpha = 0.1$  for MNIST,  $\alpha = 0.1$  for CIFAR-10, and  $\alpha = 0.5$  for CIFAR-100 respectively in Dirichlet-distribution-based data heterogeneity.
Model Heterogeneity. In our evaluation, we consider the following client model reduction levels:  $\beta = \{1,\frac{3}{4},\frac{1}{2},\frac{1}{4}\}$  for MLP and  $\beta = \{1,\frac{1}{2},\frac{1}{4},\frac{1}{8}\}$  for ResNet, where each fraction represents its model capacity ratio to the largest client model (full model). To generate these client models, for MLP we reduce the number of nodes in each hidden layer, for WResNet we reduce the number of kernels in convolution layers while keeping the nodes in the output layer as the original.
Baselines and Testcase Notations. As this experiment is mainly to validate the proposed theory and gather empirical findings, we choose the standard federated learning algorithm, i.e. FedAvg [1], with several different heterogeneous FL model settings. Since this experiment section is to verify the impact of our proposed theory rather than chasing a SOTA accuracy, no further tuning or tricks for training were used to demonstrate the impacts of key factors from the main theorems.
We consider 3 levels of model reduction through pruning and static subnet Extraction: which will reduce the model by only keeping the largest or leading  $\beta$  percentile of the parameters per layer. We show 4 homogeneous settings with the full model and the models with 3 levels of model reduction, each with at least one full model so that  $\Gamma_{min} > 1$  is achieved. Finally, we consider manually increasing the minimum coverage index and present one possible case denoted as "optimized", by applying local models covering different regions of the global model as illustrated in Fig 1.
Note that even when given a specific model reduction level and the minimum coverage index, there could be infinite combinations of local model reduction solutions; at the same time model reduction will inevitably lead to an increased model reduction noise, by conducting only weights pruning will bring the lowest model reduction noise for a certain model reduction level. How to manage the trade-off between increasing  $\Gamma_{min}$  while keeping  $\delta^2$  low is non-trivial and will be left for future works on designing effective model reduction policies for heterogeneous FL.
6.2 Numerical Results and Further Discussion
We summarize the testing results with one optimized version for comparison in Table 1. We plot the training results of Heterogeneous FL with IID and non-IID data on the MNIST dataset in Figure 2a and Figure 2b, since the model and its reduction are closer to the theoretical setup and its assumptions. We only present training results of medium-level model reduction (where we deploy 4 clients with
fullmodel and 6 clients with  $\frac{3}{4}$  models) in the figure at the main paper due to page limit and simplicity. We leave further details and more results in the appendix.
General Results. Overall, we observe improved performance on almost all optimized results, especially on subnet-extraction-based methods on high model reduction levels. In most cases, performances will be lower compared to the global model due to model-reduction noise.
Impact of model-reduction noise. As our analysis suggests, one key factor affecting convergence is model-reduction noise  $\delta^2$ . When a model is reduced, inevitably the model-reduction noise  $\delta^2$  will affect convergence and model accuracy. Yet, our analysis shows that increasing local epochs or communication rounds cannot mitigate such noise. To minimize the convergence gap in the upper bounds, it is necessary to design model reduction strategies in heterogeneous FL with respect to both model-reduction noise and minimum coverage index, e.g., by considering a joint objective of preserving large parameters while sufficiently covering all parameters.
Impact of minimum coverage index. Our theory suggests that for a given model reduction noise, the minimum coverage index  $\Gamma_{\mathrm{min}}$  is inversely proportional to the convergence gap as the bound in Theorem 1 indicates. Then for a given model reduction level, a model reduction strategy in heterogeneous FL with a higher minimum coverage index may result in better training performance. Note that existing heterogeneous FL algorithms with pruning often focus on removing the small model parameters that are believed to have an insignificant impact on model performance, while being oblivious to the coverage of parameters in pruned local models, and the model-extraction-based method will only keep the leading subnet. Our analysis in this paper highlights this important design for model reduction strategies in heterogeneous FL that parameter coverages matter.
More discussions and empirical findings. For the trade-off between minimum coverage index and model reduction noise, it's nearly impossible to fix one and investigate the impact of the other. In addition, we found: (1) Large models hold more potential to be reduced while maintaining generally acceptable accuracy. (2) Smaller models tend to be affected more by  $\delta^2$  while the larger model is more influenced by  $\Gamma_{min}$ , which suggests that it's more suitable to apply pruning on small networks and apply subnet extraction on large networks.
Limitations In this work we consider full device participation, where arbitrary partial participation scenario is not considered. Also, the optimal design of model extraction maintaining a balance between a low  $\delta^2$  and a high  $\Gamma_{\mathrm{min}}$  is highly non-trivial which would be left for future work.
In this paper, we present a unifying framework and establish sufficient conditions for FL with dynamic heterogeneous client-dependent local models to converge to a small neighborhood of a stationary point of standard FL. The optimality gap is characterized and depends on model reduction noise and a new concept of minimum coverage index. The result recovers a number of state-of-the-art FL algorithms as special cases. It also provides new insights on designing optimized model reduction strategies in heterogeneous FL, with respect to both minimum coverage index  $\Gamma_{\mathrm{min}}$  and model reduction noise  $\delta^2$ . We empirically demonstrated the correctness of the theory and the design insights. Our work contributes to a deeper theoretical comprehension of heterogeneous FL with adaptive local model reduction and offers valuable insights for the development of new FL algorithms in future research.
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22(3):2031-2063, 2020.
[3] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2021.